Linked Questions

Popular Questions

How many executors should I use for spark streaming

Asked by At

I have to write spark streaming(createDirectStream API) code. I will be receiving around 90K messages per second so though of using 100 partitions for kafka topic to improve the performance.

Could you please let me know how many executors should I use? Can I use 50 executors and 2 cores per executor?

Also, consider if the batch interval is 10seconds and number of partitions of kafka topic is 100, will I receive 100 RDDs i.e. 1 RDD from each kafka partition? Will there be only 1 RDD from each partition for the 10second batch interval.

Thanks

Related Questions