Open Source Analytics Platform


Re-architecting a leading Opensource Analytics Platform for better
experience & performance.

IndustryTechnology Platform

Year2012 – Ongoing

RoleProduct Development, Advanced Analytics

TechnologyRuby on Rails – 2.3, Ruby – 1.9, PostgreSQL – 9.2, Ngnix with Passenger, AWS S3 for Cloud Storage

Re-architect a leading Open Source platform

Our client is a leading Open Source management platform.  The platform has over 670,000 open source projects listed with a strong community of 39,000 members. It indexes thousands of open source projects, providing detailed analytics on code history, ongoing updates, attributes, contributors etc. With growing adoption, the customer felt the need to re-architect the entire platform for better user experience, performance & stability.

Data crawlers fetch details from online sources

We adopted a Cloud native architecture in order to improve the overall performance. We upgraded the code to the latest stack for stability – Ruby 2.2 to Ruby 4.2X & Rails 2.3 to Rails 4.2. We used a scalable Private Cloud framework built on top of AWS for storage to save the overall storage cost. Our crawlers fetched the open source code from open source directories along with detailed project metrics such as commits, contributors, languages used, commit time etc. 18 web crawlers were used with dynamic crawl based on frequency of code updates. Once the codes was fetched, it was then pushed into a private cloud for storage.


Search engine that understands context

The platform is now an open source mansion for enterprises. The cloud-native architecture & private-cloud adoption has significantly enhanced the platform’s performance, scalability & future expansion. In addition, the search is more contextual now, with wide search criteria based on popularity, relevance, activity level, newness & more. The new platform also provides real-time opensource project comparison & deep dive analytics.