Flume agent appending extra '\n' after every line write

109 views Asked by At

I am using my custom configuration as below for flume agent

# Name the components on this agent
agent.sources = r1
agent.sinks = k1
agent.channels = c1

# Describe the source
agent.sources.r1.type = org.apache.flume.source.AvroSource
agent.sources.r1.bind = 192.168.1.31
agent.sources.r1.port = 43999

# Describe the sink
agent.sinks.k1.type = com.zaloni.bedrock.collection.flume.sink.BedrockAvroHDFSEventSink
agent.sinks.k1.hdfs.path = /user/bedrock/sentimentAnalysis/TweetData
agent.sinks.k1.hdfs.rollInterval = 300
agent.sinks.k1.hdfs.rollSize = 1000
agent.sinks.k1.hdfs.rollCount = 100
agent.sinks.k1.hdfs.fileType = DataStream
agent.sinks.k1.hdfs.writeFormat = Text

# Describe the channel
agent.channels.c1.type = org.apache.flume.channel.MemoryChannel

# bind the source and sink to the channel
agent.sources.r1.channels = c1
agent.sinks.k1.channel = c1

Using the above configurations, I am sending my Stream Data from Java Program to AVRO source. When Flume agent writes the output to HDFS then it appends one extra '\n' character at the end of each line.

Below is the output sample

@VermaAmrutaRT @AnjneyaParashar: IBM Watson can now transcribe speech perfectly #ibm #watson #transcription http://t.co/pm5iyLXOOe06-17-2015 13:35:00 +0530 #IBM1

@ThomasLaceyEire#IBM @IBM_DS_Europe  https://t.co/c3ybimNkc606-17-2015 13:35:00 +0530#CSCO1

@INQRT @IBMPowerSystems: #IBM and @OpenPOWERorg encouraging #OpenSource all around the world: http://t.co/duyPrzaZL6 via @ChrisTheDJ @INQ06-17-2015 13:35:00 +0530 #IBM1

In the above output each extra line with blank-space is having '\n' character.

Conclusion: Why am I getting extra '\n' character? what might be possible solutions for it?

0

There are 0 answers