How to connect a kinesis stream with a firehose delivery stream using terraform

Question

My logging system uses a kinesis stream attached to a firehose delivery stream which writes to an S3 bucket. This can be manually configured through the AWS console by setting the "Source" attribute of the firehose stream to the kinesis stream. I want to use terraform and capture this setup in code.

Neither the aws_kinesis_firehose_delivery_stream nor the aws_kinesis_stream terraform resources have attributes (that I can find) which set the Source attribute. I cloned the terraform source and looking through it and I see this:

createInput := &firehose.CreateDeliveryStreamInput{
    DeliveryStreamName: aws.String(sn),
}

My next thought was to see if I could edit the code to set the Source attribute. So I looked through the AWS Firehose API to find the attribute name and I found this:

DeliveryStreamType

The delivery stream type. This parameter can be one of the following values:

DirectPut: Provider applications access the delivery stream directly.

KinesisStreamAsSource: The delivery stream uses a Kinesis stream as a source. Type: String

Valid Values: DirectPut | KinesisStreamAsSource

Given this I thought I would just edit the terraform code to set DeliveryStreamType with the necessary configuration, "KinesisStreamSourceConfiguration." However grepping around I find no reference to DeliveryStreamType in the aws sdk code present in the terraform repo. I do see DeliveryStreamName however.

Is it possible to connect kinesis and firehose streams using terraform? If not, is this a feature that's on the horizon?

Thanks in advance.

Answer 1

I've attempted to implement this functionality & have raised a PR here

Answer 2

I cloned the latest from https://github.com/aws/aws-sdk-go and confirmed that terraform is simply using an older version of the go aws API that doesn't support DeliveryStreamType. Terraform code:

type CreateDeliveryStreamInput struct {
_ struct{} `type:"structure"`

    // The name of the delivery stream. This name must be unique per AWS account
    // in the same region. You can have multiple delivery streams with the same
    // name if they are in different accounts or different regions.
    //
    // DeliveryStreamName is a required field
    DeliveryStreamName *string `min:"1" type:"string" required:"true"`
    ...
}

Current aws-sdk-go repo:

type CreateDeliveryStreamInput struct {
_ struct{} `type:"structure"`

    // The name of the delivery stream. This name must be unique per AWS account
    // in the same region. If the delivery streams are in different accounts or
    // different regions, you can have multiple delivery streams with the same name.
    //
    // DeliveryStreamName is a required field
    DeliveryStreamName *string `min:"1" type:"string" required:"true"`

    // The delivery stream type. This parameter can be one of the following values:
    //
    //    * DirectPut: Provider applications access the delivery stream directly.
    //
    //    * KinesisStreamAsSource: The delivery stream uses a Kinesis stream as
    //    a source.
    DeliveryStreamType *string `type:"string" enum:"DeliveryStreamType"`
    ...
    // When a Kinesis stream is used as the source for the delivery stream, a KinesisStreamSourceConfiguration
    // containing the Kinesis stream ARN and the role ARN for the source stream.
    KinesisStreamSourceConfiguration *KinesisStreamSourceConfiguration `type:"structure"`
    ...
}

So this answers my question, basically the terraform repo needs to be updated to use the current aws-sdk-go code.