It's been a month now and I am happy to share my POC experience with the Pulsar Community. Also have some scaling, stability, and configs-related concerns to resolve.
With Standalone Setup, I got familiar with the basic concepts/terms and component-wise flow like broker, bookKeeper, and ZooKeeper, etc.
With Bare Metal setup, I learned to run Apache Pulsar Cluster on multiple Nodes. Which results in an increase in TPS too. My Major concern is to achieve maximum TPS with Stable infra, which is missing in my current setup.
So basically I am doing a load test with a 52kb payload size with 100 000 records in a single request. To process all 100k batches it is taking around 46 sec.
So in short it gives me 4.6k TPS to process 100k records with 52kb payload size each. Where I wanted to achieve min TPS of 15k with a payload size of 52kb.
Cluster Stability Concern - Benchmark/Load Test Observations (On running load test multiple times)
Brokers get down randomly (sometimes OOM issue, sometimes another a random issue like System load, etc)
Bookie gets down randomly (sometimes OOM issue, sometimes another random issue like System load, etc)
The data folder does not get a decrease in size even if messages got consumed (persistent storage)
Broker does not get graceful shutdown on stopping it using CLI (to scale down the number of brokers and to reduce the load on a machine) All above-mentioned issues are recurring and hence it is impacting POC results.
Questions.
Kindly suggest the best architecture setup so that setup will be scalable and easy to troubleshoot
- A basic checklist to maintain while setting up the Pulsar Cluster
- Basic Server Configuration to achieve mentioned TPS
- Cluster tunning & troubleshooting tips to avoid maintenance time
Sharing the current setup design on which I am running load the test ( All machines are in GCP)
Please go through the above-detailed information and suggest to me the best solution to resolve my requirement
Original Issue Raised - Apache Pulsar Discussion
Thanks in advance!