ERROR flume.SinkRunner: Unable to deliver event

12 views Asked by At

I am trying to transfer data from flume to hdfs. My flume.conf files looks like:


agent.sources = tail
agent.channels = channel1
agent.sinks = sink1

agent.sources.tail.type = exec
agent.sources.tail.command = cat /home/hadoop/data.csv
agent.sources.tail.channels = channel1
agent.sources.tail.fileHeader = true

agent.sinks.sink1.channel = channel1
agent.sinks.sink1.type = hdfs
agent.sinks.sink1.hdfs.path = /user/hadoop/flume_data
agent.sinks.sink1.hdfs.fileType = DataStream
agent.sinks.sink1.hdfs.rollInterval = 10
agent.sinks.sink1.hdfs.rollSize = 0
agent.sinks.sink1.hdfs.rollCount = 0
agent.sinks.sink1.hdfs.format = text

agent.channels.channel1.type = memory
agent.channels.channel1.capacity = 50000000
agent.channels.channel1.transactionCapacity = 500000

I am using below command in apache flume conf directory:

flume-ng agent --conf conf --conf-file flume.conf --name agent -Dflume.root.logger=INFO,console -Xmx2g 

Getting below:

 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:451)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
        at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
        at org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$FsPermissionProto.getSerializedSize(HdfsProtos.java:5407)
        at com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:749)
        at com.google.protobuf.CodedOutputStream.computeMessageSize(CodedOutputStream.java:530)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$CreateRequestProto.getSerializedSize(ClientNamenodeProtocolProtos.java:2371)
        at com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.constructRpcRequest(ProtobufRpcEngine.java:149)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:193)
        at com.sun.proxy.$Proxy11.create(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
        at com.sun.proxy.$Proxy11.create(Unknown Source)
 ...

I am using Hadoop 2.0.0, Flume 1.7.0 and Java 1.7.0_25. How do I fix this issue? Appreciate any help on this

0

There are 0 answers