Business Intelligence (at scale) in Microservice architecture

06/06/2016 - 16:30 to 17:10
Frannz Salon
long talk (40 min)

Session abstract: 

"Business intelligence is the part of the company which is affected the most when we a company under goes microservice transformation". is the largest online retailer in the Netherlands and Belgium and is still growing at a staggering rate. is a fact based decision making company hence business intelligence plays a key role in the organization. For gathering business intelligence (BI) insights we have a team of twelve dedicated BI engineers. However, it is not just the scale of data, but also the complexity of these services that can makes it difficult to gather all the business insights. The growth of data, users and complexity have forced our team of to rethink our traditional relational data warehouse structure. Currently we are moving towards a more hybrid BI solution based on Microservices at big data scale, using Hadoop, Hbase, relational databases, etc.  In this presentation, we will discuss key concepts that govern our datawarehousing, the unique challenges we have faced and are still facing with petabytes of data along with design decisions, tooling landscape and architectural choices. We have created and explored various toolings and concepts for solving our problems.  We discuss concepts pertaining to Data quality at big data scale, large scale batch scheduling, job monitoring, ETL processing, reporting, continuous deployment, shift in mindset of traditional datawarehouse developers and team autonomy for BI at scale. Essentially, all the hacks to get our systems towards awesomeness. The presentation will guide audience through transformation of legacy (vintage) BI towards more scalable BI setup as well as mental and techincal block that came along the way. We believe it is important to educate impact of microservices on datawarehouse and we have learned how we could do that at petabyte scale.