SYSTEM Cited by 1 source
Amazon Kinesis Data Streams¶
Amazon Kinesis Data Streams is AWS's managed durable streaming substrate — a distributed, sharded, append-only log for real-time data ingestion. Comparable in architectural role to Apache Kafka (see systems/kafka): producers append records, consumers read by shard + sequence number, records are retained for a configurable TTL (24 h default, up to 365 days). Scales by shards; each shard supports a fixed ingest rate.
Stub page — minimal viable for the 2025-12-11 conversational- observability ingest. Expand as future Kinesis-specific sources land.
Role in the conversational-observability blueprint (2025-12-11)¶
Kinesis Data Streams is the buffer tier in AWS's telemetry-to-RAG pipeline on EKS. It sits between Fluent Bit's in- cluster telemetry collection and Lambda's embedding-generation compute. Responsibilities:
- Absorb spikes — ingest bursts from the cluster are smoothed before hitting Lambda concurrency.
- Durability during downstream outages — if Lambda / Bedrock / OpenSearch hiccup, records remain in Kinesis for their retention window and are re-processed on recovery.
- Enable batching — Lambda's Kinesis event source supports batch sizes, which AWS's "Pro tip" explicitly recommends for cost-efficient embedding generation.
Kinesis vs SQS vs Kafka (positioning)¶
- SQS (systems/aws-sqs) — message queue with at-least-once delivery and no replay. Good for task fanout, not for ordered replay or multi-consumer broadcast.
- Kafka (systems/kafka) — self-managed / MSK; richer ecosystem (Kafka Streams, Connect); more operational work; higher theoretical throughput ceiling.
- Kinesis Data Streams — AWS-managed Kafka-shaped primitive; tight integration with AWS-native consumers (Lambda, Firehose); fewer operational levers; sufficient for most AWS-resident streaming workloads.
Caveats¶
- Stub page based on one source. Production design considerations (shard count, resharding, consumer fanout limits, enhanced fan-out consumers, on-demand vs provisioned capacity, KCL client library) not covered.
- Encryption at rest / in transit via KMS is mentioned in the source but not expanded here.
Seen in¶
- sources/2025-12-11-aws-architecting-conversational-observability-for-cloud-applications — Kinesis Data Streams as the ingest buffer between Fluent Bit (producer) and Lambda (consumer). Lambda batching from Kinesis called out as the canonical cost / performance optimization.