what is topic count parameter for Kafka ConsumerConnector

Question

I am new to apache kafka, and try with the examples given.

The following code snippet is used to initialize a ConsumerConnector, I am confused by the topic count parameter; it seems it will cause kafka hands out corresponding number of streams for that topic. however, I tried several times, only the first stream produces messages. So, two questions: 1. how can I determine the count number for a topic? 2. how does the messages split cross over the streams?

thanks in advance.

    Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
    **topicCountMap.put(topic, new Integer(a_numThreads));**
    Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer
            .createMessageStreams(topicCountMap);
    List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);

Answer 1

Another user may feel free to correct me if I am slightly incorrect but:

I have looked at this example before and the code is typically initialized as

topicCountMap.put(topic, new Integer(1));

With one being the integer because for the examples given the sample topic is usually created with only a single partition. When your topic only has a single partition, there is no capability of parallelism (as if you had multiple consumers in the same group there will be some consumer instances that do not get any data), and thus, no reason to create multiple streams. As such, for this single partition only a single stream is created and that is why you only have the first (and only) stream producing messages.