System Architecture Overview
Producers: Lambda functions with HTTP input triggers, invoked via AWS API Gateway. These functions publish the appropriate messages to Kafka topics.
Consumers: EKS-deployed services that consume from Kafka topics and may produce new messages as needed. Each consumer handles specific business logic and often calls external/internal APIs.
Secrets: Pulled from AWS Secrets Manager. Used by consumers for DB credentials, PAT tokens, API keys/secrets, and client credentials.
Topics: Kafka topics hosted on a managed Kafka service. All producer and consumer communication flows through these topics.
Architecture Diagram
