I'm designing a system that should be able to listen to over 1000 different event streams. Those are all HTTP streams, around 80% of the time the events are merely a heartbeat ( < 1Kb). The other 20% are actual events ( < 10Kb) and each event will trigger a different behavior.
It's important that the latency of the chain represented belowbe minimal, near real-time, but not necessarily so. To put into numbers a 5 to 10 seconds latency is perfectly fine for my purposes.
Event occurred > event streamed > event ingested > event processed > behavior triggered
No long term storage needed
I'm unsure what are the appropriate AWS services here, AWS Kinesis Data Streams or AWS Kinesis Firehose. I've never used any of them and I don't fully understand the actual difference between them.
However Data Streams seems to be considerably more expensive than Firehose.
From what I understand in Data Streams I'd need to pay
Per stream/per hour,so 1000 streams * 24 hours * 30 days * price ($0.04) = 28.8k USD / month just to keep the streams running. That is added of +$0.08 per GB ingested. Is that correct?While on Firehose I'm charged only on the data ingested, at an immensely lower rate: $0.029 per 500 TB / month
Admittedly I don't know how Ingestion Direct PUT and KDS as a source and Ingestion Pricing for MSK as a source actually work, so they may not fit my needs.
ATM, I'm looking to understand if AWS Firehose can be used for my purposes, and if so, why Data Streams is so much more expensive?