보물창고/Big Data2015.08.23 23:30





flume error 해결 org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable



해결에 참고한 내용



입력되는 값
문자열이 탭으로 구분된 데이터

sfae a;ldfja;sldj alwkejhfalskdhf a;slekjafabcd def de
ab d eef ddefsef
sfae a;ldfja;sldj alwkejhfalskdhf a;slekjafabcd def de
ab d eef ddefsef
sfae a;ldfja;sldj alwkejhfalskdhf a;slekjafabcd def de
ab d eef ddefsef
sfae a;ldfja;sldj alwkejhfalskdhf a;slekjaabcd def de
ab d eef ddefsef
sfae a;ldfja;sldj alwkejhfalskdhf a;slekjafabcd def de
ab d eef ddefsef
sfae a;ldfja;sldj alwkejhfalskdhf a;slekjaf
sfae a;ldfja;sldj alwkejhfalskdhf a;slekjafabcd def de
ab d eef ddefsef
sfae a;ldfja;sldj alwkejhfalskdhf a;slekjafabcd def de
ab d eef ddefsef
sfae a;ldfja;sldj alwkejhfalskdhf a;slekjafabcd def de
ab d eef ddefsef
sfae a;ldfja;sldj alwkejhfalskdhf a;slekjaabcd def de
ab d eef ddefsef
sfae a;ldfja;sldj alwkejhfalskdhf a;slekjafabcd def de





 에러로 출력되는 결과
SEQ!org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable      €藪???pR??      NC+?   abcd def de      NC+?   ab d eef ddefsef   8    NC+?   ,sfae a;ldfja;sldj alwkejhfalskdhf a;slekjaf      NC+?   abcd    def     de   +    NC+?   ab      d       eef     ddefsef   O    NC+?   Csfae    a;ldfja;sldj    alwkejhfalskdhf a;slekjafabcd    def     de   +    NC+?   ab      d       eef     ddefsef   O    NC+?   Csfae    a;ldfja;sldj    alwkejhfalskdhf a;slekjafabcd    def     de   +    NC+?   ab      d       eef     ddefsef   O    NC+?   Csfae    a;ldfja;sldj    alwkejhfalskdhf a;slekjafabcd    def     de








아래 설정에서 파란색 라인을 추가 하는 것으로 해결됨

agent.sources = logfilereadSource
agent.channels = memoryChannel
agent.sinks = loggerSink

# source
# For each one of the sources, the type is defined
agent.sources.logfilereadSource.type = exec
agent.sources.logfilereadSource.command = tail -F /home/hadoop/download/flumeTestfile.log
agent.sources.logfilereadSource.batchSize = 10
agent.sources.logfilereadSource.channels = memoryChannel

# channel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 20480


# Sink
#Specify the channel the sink should use
agent.sinks.loggerSink.type = hdfs
agent.sinks.loggerSink.channel = memoryChannel
agent.sinks.loggerSink.hdfs.path = hdfs://localhost:8020/user/admin/flumetest/log
agent.sinks.loggerSink.hdfs.fileType = DataStream
agent.sinks.loggerSink.serializer = com.adaltas.flume.serialization.HeaderAndBodyTextEventSerializer$Builder
agent.sinks.loggerSink.serializer.format = CSV
agent.sinks.loggerSink.serializer.appendNewline = true
agent.sinks.loggerSink.serializer.delimiter = '\t'


위 설정 추가후

ab d eef ddefsef sfae a;ldfja;sldj alwkejhfalskdhf a;slekjafabcd def de ab d eef ddefsef sfae a;ldfja;sldj alwkejhfalskdhf a;slekjafabcd def de ab d eef ddefsef sfae a;ldfja;sldj alwkejhfalskdhf a;slekjaabcd def de ab d eef ddefsef sfae a;ldfja;sldj alwkejhfalskdhf a;slekjafabcd def de ab d eef ddefsef sfae a;ldfja;sldj alwkejhfalskdhf a;slekjaf





신고





Posted by 파란물

티스토리 툴바