Kinesis GetShardIterator... invalid because it did not come from this stream

Question

I have build a KCL plus spark base on https://spark.apache.org/docs/1.2.0/streaming-kinesis-integration.html

I am running this on EMR (spark installed via bootstrap). I have created on stream sparkTest and did my testing its working fine . I have observed no DynamoDB was created . I have deleted stream and cluster . Next day again i have created Kinesis Steam with same name and deployed my code with newly launched cluster. Now i am getting

5/06/12 08:17:28 ERROR worker.InitializeTask: Caught exception:
com.amazonaws.services.kinesis.model.InvalidArgumentException: StartingSequenceNumber 49551532098093284204238000035066183240246145871536717826 used in GetShardIterator on shard shardId-000000000000 in stream sparkTest under account 618673372431 is invalid because it did not come from this stream. (Service: AmazonKinesis; Status Code: 400; Error Code: InvalidArgumentException; Request ID: 770ef875-10db-11e5-b24b-af6f372168ae)
        at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1078)
        at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:726)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:461)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClie

I am not able to understand why this is coming. if i create new kinesis stream and then do work it works again. is it issue with Kinesis ?

One more thread is going into this is with https://github.com/awslabs/amazon-kinesis-connectors/issues/8 however i am not using kinesis app name and creating stream with

KinesisUtils.createStream(
                        jssc, streamName, endpointUrl, kinesisCheckpointInterval, InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2())

Answer 1

SparkConf sparkConfig = new SparkConf().setAppName("arbitraryName").setMaster("local[2]");

KinesisUtils.createStream(
                        jssc, streamName, endpointUrl, kinesisCheckpointInterval, InitialPositionInStream.LATEST, StorageLevel.MEMORY_AND_DISK_2()));

If i change the name "arbitraryName". it works fine . i found this hint from the https://spark.apache.org/docs/1.2.0/streaming-kinesis-integration.html

key points:
The application name used in the streaming context becomes the Kinesis application name
The application name must be unique for a given account and region.

Answer 2

stack_trace: com.amazonaws.services.kinesis.model.InvalidArgumentException: 
StartingSequenceNumber 49629849211345904785630184136577730155367153937946771474 used in GetShardIterator on 
shard shardId-000000000001 in stream Stream-abc-events-stream-Stream under 
account 186088486654 is invalid because it did not come from this stream. 
(Service: AmazonKinesis; Status Code: 400; Error Code: InvalidArgumentException;

I also faced this issue, because of changing Stream alone suddenly with existing Dynamo DB. just down the dynamo db which allow the new stream with new dyano db sync, That will resolve the issue.