Apache flume Regex Extractor Interceptor

2.2k views Asked by At

I am trying to append the header using Regex Extractor Interceptor but it is not working. I am not sure where I am going wrong. Below is my config file content :

agent.channels = CH
agent.channels.CH.type = memory
agent.channels.CH.capacity = 1000

agent.sources = SRC
agent.sources.SRC.type = avro
agent.sources.SRC.bind = localhost.localdomain
agent.sources.SRC.port = 2222
agent.sources.SRC.channels = CH

agent.sources.SRC.interceptors = i1
agent.sources.SRC.interceptors.i1.type = regex_extractor
agent.sources.SRC.interceptors.i1.regex = ^.*(aa).*$
agent.sources.SRC.interceptors.i1.serializers = t1
agent.sources.SRC.interceptors.i1.serializers.t1.name = A:

agent.sinks = SNK
agent.sinks.SNK.type = hdfs
agent.sinks.SNK.hdfs.fileType = DataStream
agent.sinks.SNK.channel = CH
agent.sinks.SNK.hdfs.path = hdfs://localhost.localdomain:8020/user/cloudera/test/fl1/

When i tried Regex Extractor Interceptor with multiplexing channel selector,the events containing "aa" were routed to "/user/cloudera/test/fl1/" and events containing "bb" were routed to "/user/cloudera/test/fl2/" but headers were not appended in this case too. Below is my config file content :

agent.channels = CH1 CH2
agent.channels.CH1.type = CHory
agent.channels.CH1.capacity = 100
agent.channels.CH2.type = CHory
agent.channels.CH2.capacity = 100

agent.sources = SRC
agent.sources.SRC.channels = ch1
agent.sources.SRC.type = avro
agent.sources.SRC.bind = localhost.localdomain
agent.sources.SRC.port = 2222

agent.sources.SRC.channels = CH1 CH2
agent.sources.SRC.interceptors = i1



agent.sources.SRC.interceptors.i1.type = regex_extractor
agent.sources.SRC.interceptors.i1.regex = ^.*(aa|bb).*$
agent.sources.SRC.interceptors.i1.serializers = t1
agent.sources.SRC.interceptors.i1.serializers.t1.name = type1
agent.sources.SRC.selector.type = multiplexing
agent.sources.SRC.selector.header = type1
agent.sources.SRC.selector.mapping.aa = CH1
agent.sources.SRC.selector.default = CH2

agent.sinks = SNK1 SNK2
agent.sinks.SNK1.type = hdfs
agent.sinks.SNK1.hdfs.fileType = DataStream
agent.sinks.SNK1.channel = CH1
agent.sinks.SNK1.hdfs.path = hdfs://localhost.localdomain:8020/user/cloudera/test/fl1/

agent.sinks.SNK2.type = hdfs
agent.sinks.SNK2.hdfs.fileType = DataStream
agent.sinks.SNK2.channel = CH2
agent.sinks.SNK2.hdfs.path = hdfs://localhost.localdomain:8020/user/cloudera/test/fl2/

Any help is appreciated. With Regards

0

There are 0 answers