Optimize product engineering efforts with unified log management

Engineering and developing a world-class product in a distributed environment is challenging. When said engineering needs to be carried out by a team that is distributed geographically and spans organizational boundaries, you’d have to stay on top of a lot of things. You’d have to ensure proper visibility of development data and project statuses, invest in adequate tooling that promotes collaboration, distribute work appropriately, and maintain code quality, all while making sure your development goals are being achieved. You may also look at outsourcing some of your development to focus on incrementally improving your product as it is being developed. 

While outsourcing is genuinely a fantastic solution, your vendor teams can be productive only if they have ready access to the data they need for development. A distributed development environment can be quite a complex machine. It is likely to have services from different service providers, each playing an integral role within the environment. For instance, your development environment could use Fargate or Lambda for serverless compute, S3 or MinIO for storage, and Cloudwatch for monitoring. All of these services will generate a large amount of data that your outsourced team also needs access to. Reigning data from all these sources can be quite a mammoth task, let alone analyzing that data and managing RBAC across these services. 

Apica recently helped a customer solve these exact problems. 

About our customer

Our customer is an established fintech company that invests in and partners with financial institutions worldwide to foster financial inclusion for unserved and underserved communities. They facilitate management consulting and technology to support their partners and new business initiatives. 

The customer’s development stack predominantly comprised of the following services:

• AWS CloudWatch for monitoring and log analysis
• AWS ECS, AWS Lambda, and AWS Fargate for serverless compute

Their challenges

Our customer followed a product development strategy that depended on collaboration from external partners in a distributed environment. Product development in such environments requires every collaborator to be on the same page for effective collaboration constantly. The need to unify data sources, derive and analyze performance metrics, and display them in a well-organized, searchable, and shareable manner became even more crucial. 

The company wanted to negate the hassles of developing in distributed environments by using the benefits of unified and centralized logging and analysis to enable internal and external development teams to accelerate their development efforts. 

The complex documentation and inaccuracies across the services that made up their development stack made it tougher to analyze their logs efficiently and derive operational visibility and actionable insights. The complexity of their monitoring stack and relative inaccessibility of log data also meant that they couldn’t improve operational efficiency and capitalize on optimal resource utilization. 

The company also wanted to ensure that external teams involved in development and monitoring tasks would only have access to performance metrics and information relevant to them.

How Apica helped

Apica was employed for the following purposes:

• DevOps: Exporting CloudWatch logs to Apica using an AWS Lambda exporter
• Infrastructure Monitoring: Unified monitoring for AWS ECS, Fargate, and Lambda so that developers can rapidly iterate and accelerate their product development. 
• Secure Collaboration: Enabling distributed teams to monitor and analyze logs from ECS and Lambda collaboratively. 

Solutions brief

We set Apica up to ingest log data from the customer’s AWS ECS, Fargate, Lambda, and CloudWatch instances. Connecting CloudWatch with Apica was quickly taken care of using our AWS CloudWatch Exporter Lambda function. With these integrations in places, our customer could unify log data from distributed and disparate data sources and access them through a single window.

The unification of log data through Apica enabled our customer to gain complete visibility into their AWS-based development environment. Apica also provided quick turnarounds for dealing with the inconsistencies across AWS service offerings and their documentation. Apica’s RBAC feature also helped our customer set up fine-grained access policies that enabled them to govern data access based on their levels of authorization.

The results

Apica made it easy to bring critical application data in the form of logs from their development stack and monitoring stacks in one place. Apica’s quick and seamless integration with AWS ECS, Fargate, and CloudWatch made it easier to view critical information from all of these distributed sources in a single pane. 

Apica’s LOG2Metrics feature also made it phenomenally simple to extract robust metrics from log data that was otherwise tough to access using their existing monitoring solution. LOG2Metrics also made it easier to visualize these metrics in time-series graphs that enabled them to identify patterns in their application performance, derive actionable insights, and identify threats and weaknesses even before they caused real damage.

Moreover, Apica also made it easier to map application logs from ECS tasks and CloudWatch to RBAC policies, thereby ensuring that every member of their distributed teams would only have access to data relevant to their tasks.

Key statistics

Along with seamless data sharing across their distributed team, simpler RBAC governance, and greater analytical capabilities, our customer also witnessed:

• Faster querying with queries run on historical data as old as 30 days returning results in under 5 seconds. 
• Unification of logs from 3 distributed systems: AWS ECS + Fargate + Lambda

Why you should use Apica, too

Apica is designed to be the perfect data platform for DevOps, SRE, and DevSecOps teams, alike. While unification of distributed data sources and real-time monitoring and data analytics are a given, Apica is the only platform that lets you perform Log Management, APM, SIEM, AIOps, and Infrastructure & Event Monitoring through a single interface. No software lock-in, complicated pricing, or unnecessary strings attached.

See how Apica lets you empower the people behind your great software with actionable insights derived from machine data, continuous intelligence, and total visibility over your applications and infrastructure. Get started with the free-forever Community Edition of Apica PaaS or contact us for a demo.