Get More Out of Your Data With Analytics Microservices
Updated: Jan 4, 2022
After this article, please visit here where I showcase a working series of microservices supporting reports for the City of Chicago.
The velocity, variety and volume of data is expanding quickly. According to recent studies by IDC, the global creation of data will balloon to 181 zettabytes by 2025 while corporations will capture about 16 zettabytes (~9% of global data created is captured by corporations.) And this gap is expected to widen. Using current practices, some corporations may be less capable and unable to take advantage of the growth in data.
The digitization of consumer experiences is a driver of the data deluge.
Some companies are looking to close the gap using agile development methodologies that allow them to try new ideas, pivot quickly, and respond to consumer needs by tapping into consumer behavioral data.
Companies that successfully close this gap will create a competitive advantage for themselves. They are able to collect data, and leverage it in ways that give them a unique consumer insight. Those insights can become new innovations and experiences for their customers, and ultimately new business opportunities for themselves.
Just like the clients they service, business service companies need to close the gap in data creation and usage. Service companies like marketing agencies and consultancies would benefit from adopting new design patterns that facilitate innovations in data practice on behalf of their clients. These companies should consider analytics microservices.
What are microservices?
Sam Newman, author of "Building Microservices" and "Monolith to Microservices" describes microservices as a development pattern that "focuses on giving you many options for solving the problems you may face." That's an extremely broad definition, but extremely enticing for tackling analytics problems where patterns are evolving continuously. Let's dive deeper...
He further describes microservices as a collection of independently deployable, technology agnostic application services that are built around a specific business domain.
That's very helpful; and appealing for an analytics application...
Chris Richardson, author of Microservices Patterns, describes these services as a set of loosely coupled, collaborating services. He also agrees they are independently deployable and he expands on the idea by describing the teams that work on them. He explains that microservices teams are smaller teams who do independent work, design and architect their service independent of downstream services/users/requirements - but keeping those end users in mind as they develop their service.
When we think about analytic services, in a rapidly evolving big data environment, microservice development architectures are tempting. Let's explore their pros and cons to understand if they truly fit an analytics use case.
Pros and cons of microservices.
When considering deploying a microservice development pattern within your organization, you'll want to identify if the problems are worth the additional work required to keep microservices running smoothly. Let's first take a look at the benefits of this architectural pattern and compare it to the additional work needed to support it.
Pros of Analytics Microservices
Speed to market
The microservices approach allows for agility in that you can design around new features / services - meeting the needs of the consumers of that service - rather than thinking about how it will impact other applications.
With agility comes the ability to meet the new requirements of technologies introduced in the rapidly developing data storage/analytics space. In addition, we can begin to mine for insight using new data as it comes online - allowing firms to innovate and capitalize on business opportunities that arise from additional data availability.
Independence of Components
In a typical analytics approach with a single database, a single new line of code may require the complete redeployment of an entire application to facilitate the change. Downstream processes will need to account for any changes to the schema; while upstream processes will need to facilitate the newly introduce part of the overarching analytics application.
In a microservice architecture, we separate components and build them independent of each other. While they work together, we do not make them reliant on other services or technology. This makes microservices a useful way to explore new ideas and develop new services that work together with existing services.
Replication / Scale Services for B2B
Because of their independence, microservices can be replicated to facilitate a client / agency relationship. As new clients require the microservice, a service provider can initiate the microservice in the client's environment - separate from other clients' services. This makes billing and accounting easier to manage. Extra security is also a beneficial side effect of this change.
Pricing favors the customer as well as they only need to pay for what they use - which can be an appealing model in a client / agency relationship.
In addition, DevOps and data engineering teams can work on new features and enhancements to the microservice and roll out updates to all existing users - expediting the adoption of the latest updates across the customer base - without the need to re-platform or re-architect an entire monolithic solution. Updates, patches and features become available and deployable across the analytics client footprint.
Technology Flexibility
Another important feature of a microservice architectural pattern is the ability to quickly explore and adopt new technologies. Because each microservice is independent, we can choose the best tools to deliver the service.
For example, we can design an ETL service by integrating with a partner service that does a good job of giving us access to data. Perhaps we build on top of that service using Go as our object oriented programming language to transform and load data into our database.
Another service could be written in Python to take advantage of its strong data science capability and out of the box machine learning libraries. Perhaps this microservice establishes a predictive model for forecasting an important business result for clients who leverage it - and adds another layer to their available analytics service.
A third service may require a distributed compute environment to process extremely large amounts of data (maybe this is in the form of web logs with billions of lines of data each day.) That service could leverage Scala and Apache Spark to handle extremely large data and transform it into a more manageable report for a downstream process.
Microservices give you the flexibility to choose the appropriate software, hardware, containers, and cloud services for a specific business application - and keep it separate from other services in your stack for maintenance, enhancement, and scale.
Cons of Analytics Microservices
Complexity in Connecting Services
Microservices are independent and connecting them is a challenge that this design pattern faces. A microservice usually doesn't stand on its own from a total value to end user perspective. It's integrated into an offering and ultimately delivers its result to another microservice in a downstream process or is part of a broader deliverable or experience.
Teams that build microservices must think about the handoff and make sure that it is tested, efficient, and delivers the value expected by the users of the service. They must also think about their release of new features and how that may affect a downstream process that isn't ready for that new feature. An increasingly common way to address this issue is the use of feature flags, a way to toggle features on and off.
Let's demonstrate this complexity using an analytics example. Consider a process that geocodes data and delivers a latitude and longitude to reports and other analytics services. If, in the future, the dev team decides to move to point coordinates (combining latitude and longitude into a single data point) they may need to support point coordinates AND lat/long for period of time until downstream microservices (or customers) can switch over to use point coordinates. Only at that time can they deprecate the latitude / longitude data points. This requires strong communication - which leads me to the next con of microservices.
Coordination
Communication and coordination is key for microservice design patterns. Each service will need careful documentation available to internal and external users of the service. They will need to communicate changes; and facilitate training on how to leverage the change. Teams adopting microservice design patterns WILL spend time on the communication layer between services.
Investment in the communication interface between microservices is a non-trivial cost of the microservice design pattern.
A team with communication challenges will struggle with microservices. Due to this, I would not recommend microservices in a cross-agency or holding company environment where clients may work with more than one agency. Microservices may be best adopted by a fully integrated agency that handles most (if not all) of the client work. In fact, microservices may make the fully integrated agency model stronger and stickier with clients as coordination is emphasized as a key to success.
Staffing and DevOps Support
Due to the variety of technology that can be leveraged by DevOps teams and the emphasis on agility, microservice design patterns will work best in a dev ops culture. Team members and staffing should be at levels required to support effective DevOps. Each microservice should incorporate all of the staff required to build the service. Each microservice is bound to a specific business operation and those organizations will need to deploy a team that works around that operation - understanding its processes, integration points and technologies.
For example, an analytics microservice may have an engineer designing data integrations, a solution consultant responsible for the business logic, and an infrastructure team member responsible for the underlying hardware / technologies. That team would be assigned to and responsible for the microservice throughout the product lifecycle managing new feature development, maintenance and deployments.
Analytics teams adopting microservice design patterns should also have access to the tools required to support automation. Continuous integration and delivery is emphasized in microservice design - so those servers will need to be up and running. Cloud providers like AWS, Azure and GCP can facilitate the adoption of these tools.
Comparison to monolithic analytics applications.
In contrast to microservices, monolithic analytics applications put all of the functionality into a single process and replicate that entire process for each deployment. For agency relationships consider a complete application that is built to service all clients. Every client is
deployed in the same way and takes advantage of all of the monolithic functionality as they are forced to take on the entire application.
Monoliths do have a place, but they can become frustrating when looking to build new features. A change made to a small part of the application requires the entire monolith to be rebuilt and deployed for all clients.
In addition, monoliths can stifle innovation as analytics teams are unable to explore new technologies within their analytics product without re-designing and re-deploying the entire service.
Microservices adoption recommendations.
For organizations thinking about adopting microservices within their analytics function, it's important for them to remember that analytics products are not projects. Products do not have a start and end date. They continue until they are replaced with a new product or retired due to a lack of need. They must be invested in. They must evolve to solve the needs of their users. A microservice approach is also a product approach with each service solving an analytics problem for end users / clients.
Organizations may consider MonolithFirst (see Martin Fowler's recommendation.) For an analytics team, this could start with a monolithic application powering an enterprise data warehouse for all clients to use. Perhaps that solution solves some of the most common questions asked by the organization and its clients. As analytics services evolve, more questions are asked of analytics users - testing the capability of the Monolithic application.
I think the next logical step - in this case - is to develop microservices to extend the monolith. Perhaps a new analytics service splits off as a microservice powering a specific business need. That product evolves on its own. It takes input from other services (or the monolith itself) upstream, and communicates to other services and/or back to the monolithic application downstream.
In summary, microservices enable businesses to close the gap in data usage / data creation.
The data deluge creates new opportunities for businesses to find innovative new ways to create data-driven applications powering new experiences for their customers. These new opportunities will manifest themselves from teams that are capable enough to deploy new data engineering and development to support new analytics services.
The microservices design pattern enables teams to explore new technology, fail fast, and deliver new services without re-architecting their entire analytics infrastructure. Businesses should consider microservices as a process to assisting data and analytics innovation.
After this article, please visit here where I showcase a working series of microservices supporting reports for the City of Chicago.
Resources for further exploration on this topic:
Newman, S. (2021). Building microservices: Designing fine-grained systems. O'Reilly Media.
Newman, S. (2020). Monolith to microservices: Evolutionary patterns to transform your monolith. O'Reilly Media, Inc.
Richardson, C. (2018). Microservices patterns: With examples in Java. Manning.
Kleppmann, M. (2021). Designing data-intensive applications: The big ideas behind reliable, scalable, and maintainable systems. O'Reilly.
Humble, J., & Farley, D. (2015). Continuous delivery: Reliable software releases through build, test, and Deployment Automation. Addison-Wesley.