Modernize streaming solutions with Amazon MSK

A blog post by Sasha Tulchinskiy, specialist leader, Cloud Engineering, Deloitte Consulting LLP; Tony Witherspoon, principal, Cloud Architecture Talent Group Lead, Deloitte Consulting LLP; Samuel Lefki, senior consultant, Cloud Engineering, Deloitte Consulting LLP

Running Apache Kafka to support streaming data solutions can pose a challenge for developers and DevOps teams. This, at times, introduces the need for development teams to also function as experts in operating Apache Kafka. Amazon Managed Streaming for Apache Kafka (MSK) provides a fully managed service for Apache Kafka clusters. With MSK, you can let AWS run and manage your Apache Kafka clusters for you in the cloud, enhancing your streaming data applications and making it easy for you to collect, process, and deliver streaming data in real time.

Challenges with a conventional Apache Kafka setup

• Difficult setup: Apache Kafka clusters are difficult to set up and often require an expert level of knowledge in Kafka.
• Operational overhead: Maintaining, operating, and scaling Kafka clusters is a challenge and time consuming, with a steep learning curve.
• Limited out-of-the-box integration: Integrations with other AWS services will often require custom development on your part.

Benefits of using Amazon MSK

• Managed service: Amazon MSK manages Kafka infrastructure, reducing the need to staff or create expertise in Apache Kafka setup. Amazon MSK operates, maintains, and scales Apache Kafka clusters, reducing load on your developer or infrastructure teams.
• AWS integrations: Amazon MSK lets you easily deploy secure, compliant, and production-ready applications using native AWS integrations. 
• Highly available: Clusters are distributed across multiple availability zones (AZs) and are supported by Amazon MSK’s service level agreement (SLA). Automation detects and responds to issues within the cluster and Apache Kafka software. Data is replicated at no additional cost.
• Secure: Kafka clusters run in an AWS Virtual Private Cloud (VPC). The clusters are available to VPCs, subnets, and security groups based on network configurations you control. MSK also encrypts data at rest with AWS Key Management Service (KMS). You can use your own key to encrypt data at rest. Data in transit between clients and brokers on the Kafka cluster can also be encrypted with TLS.
• Cost-effective: MSK follows a pay-per-use model. You do not pay for nodes that Amazon MSK provisions for you, or for data transfers within your clusters. You pay for broker instance and storage usage and for data transfer in and out of your cluster.
• Granular access control: AWS IAM access controls help you with cluster authentication and authorization. Using IAM roles or policies to control access helps you secure your MSK resources without the need to build other access management systems.
• Scalable: MSK clusters can be scaled easily. You can change the size or family of your Apache Kafka brokers without downtime. If leveraging serverless clusters, Amazon MSK will automatically handle scaling based on your application’s needs.
• Open source: Because Amazon MSK deploys native versions of Apache Kafka, tools built for Apache Kafka can work easily. Additionally, as MSK makes newer versions of Kafka available, version upgrades are seamless.
• Observability: Get insights into your streaming solution performance with Amazon CloudWatch by default. CloudWatch enables you to monitor metrics you’re interested in for your streaming solution.

How can Deloitte help?

Deloitte has extensive experience with implementation of Amazon MSK and has helped many clients to modernize their streaming data applications leveraging industry-leading practices. Deloitte has teamed with AWS on Amazon MSK service to build sophisticated data streaming solutions.

As an AWS Partner Network (APN) Premier Consulting Partner, Deloitte provides specialization in a host of AWS-enabled business solutions, each of which can be efficiently integrated to meet clients’ needs when and where they need it most.

No matter where you are on your cloud transformation journey, Deloitte can help you discover and accelerate toward what’s truly possible for your people, customers, and business.