How to configure AWS CDK ApplicationLoadBalancedFargateService to log parsed JSON lines with Firelens and Firebit

Question

When I create an ApplicationLoadBalancedFargateService with a Firelens logdriver, and the application writes JSON lines as the log message, such as when using net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder with Logback, the log messages are displayed in my logging repository (ex. Sumo Logic), as an escaped string, like:

How can I get the log messages to save as parsed JSON?

Answer 1

After scanning CDK source code, browsing several related references (which I will provide links for to help direct appropriate traffic here), and using cdk diff until the only change was to enable json parsing, I was able to make is work as shown in the following code. The key here is the use of the addFirelensLogRouter method and the Firelens config contained therein.

TaskDefinition code does not automatically create a LogRouter container, if the task definition already contains one, which is what allows us to override the default behavior.

protected _createFargateService() {
    const logDriver = LogDrivers.firelens({
        options: {
            Name: 'http',
            Host: this._props.containerLogging.endpoint,
            URI: this._props.containerLogging.uri,
            Port: '443',
            tls: 'on',
            'tls.verify': 'off',
            Format: 'json_lines'
        }
    });
    const fargateService = new ApplicationLoadBalancedFargateService(this, this._props.serviceName, {
        cluster: this._accountEnvironmentLookups.getComputeCluster(),
        cpu: this._props.cpu, // Default is 256
        desiredCount: this._props.desiredCount, // Default is 1
        taskImageOptions: {
            image: ContainerImage.fromEcrRepository(this._props.serviceRepository, this._props.imageVersion),
            environment: this._props.environment,
            containerPort: this._props.containerPort,
            logDriver
        },
        memoryLimitMiB: this._props.memoryLimitMiB, // Default is 512
        publicLoadBalancer: this._props.publicLoadBalancer, // Default is false
        domainName: this._props.domainName,
        domainZone: !!this._props.hostedZoneDomain ? HostedZone.fromLookup(this, 'ZoneFromLookup', {
            domainName: this._props.hostedZoneDomain
        }) : undefined,
        certificate: !!this._props.certificateArn ? Certificate.fromCertificateArn(this, 'CertificateFromArn', this._props.certificateArn) : undefined,
        serviceName: `${this._props.accountShortName}-${this._props.deploymentEnvironment}-${this._props.serviceName}`,
        // The new ARN and resource ID format must be enabled to work with ECS managed tags.
        //enableECSManagedTags: true,
        //propagateTags: PropagatedTagSource.SERVICE,
        // CloudMap properties cannot be set from a stack separate from the stack where the cluster is created.
        // see https://github.com/aws/aws-cdk/issues/7825
    });
    if (this._props.logMessagesAreJsonLines) {
        // The default log driver setup doesn't enable json line parsing.
        const firelensLogRouter = fargateService.service.taskDefinition.addFirelensLogRouter('log-router', {
            // Figured out how get the default fluent bit ECR image from here https://github.com/aws/aws-cdk/blob/60c782fe173449ebf912f509de7db6df89985915/packages/%40aws-cdk/aws-ecs/lib/base/task-definition.ts#L509
            image: obtainDefaultFluentBitECRImage(fargateService.service.taskDefinition, fargateService.service.taskDefinition.defaultContainer?.logDriverConfig),
            essential: true,
            firelensConfig: {
                type: FirelensLogRouterType.FLUENTBIT,
                options: {
                    enableECSLogMetadata: true,
                    configFileType: FirelensConfigFileType.FILE,
                    // This enables parsing of log messages that are json lines
                    configFileValue: '/fluent-bit/configs/parse-json.conf'
                }
            },
            memoryReservationMiB: 50,
            logging: new AwsLogDriver({streamPrefix: 'firelens'})
        });
        firelensLogRouter.logDriverConfig;
    }
    fargateService.targetGroup.configureHealthCheck({
        path: this._props.healthUrlPath,
        port: this._props.containerPort.toString(),
        interval: Duration.seconds(120),
        unhealthyThresholdCount: 5
    });
    const scalableTaskCount = fargateService.service.autoScaleTaskCount({
        minCapacity: this._props.desiredCount,
        maxCapacity: this._props.maxCapacity
    });
    scalableTaskCount.scaleOnCpuUtilization(`ScaleOnCpuUtilization${this._props.cpuTargetUtilization}`, {
        targetUtilizationPercent: this._props.cpuTargetUtilization
    });
    scalableTaskCount.scaleOnMemoryUtilization(`ScaleOnMemoryUtilization${this._props.memoryTargetUtilization}`, {
        targetUtilizationPercent: this._props.memoryTargetUtilization
    });
    this.fargateService = fargateService;
}

Resources:

• How I first discovered it might be possible. https://github.com/aws-samples/amazon-ecs-firelens-examples/tree/master/examples/fluent-bit/parse-json
• How I discovered it might be possible with CDK https://github.com/aws/aws-cdk/pull/6322
• Understanding it from an AWS service standpoint https://docs.aws.amazon.com/AmazonECS/latest/userguide/using_firelens.html
• Narrowing in on where it resides in CDK source. https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ecs.FirelensLogRouter.html
• Eventually I landed here and figured out https://github.com/aws/aws-cdk/blob/60c782fe173449ebf912f509de7db6df89985915/packages/%40aws-cdk/aws-ecs/lib/base/task-definition.ts#L509