context.getConfiguration(); java.lang.NullPointerException 문제 해결

context.getConfiguration(); java.lang.NullPointerException 문제 해결

hadoop MRjob을 작성하는중 나타난 트러블 슈팅 내용입니다

환경은 CDH5.X버전이고 ubuntu14.04입니다

---

15/08/07 10:53:04 INFO input.FileInputFormat: Total input paths to process : 1 15/08/07 10:53:04 INFO mapreduce.JobSubmitter: number of splits:1 15/08/07 10:53:04 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1197195623_0001 15/08/07 10:53:04 INFO mapreduce.Job: The url to track the job: http://localhost:8080/ 15/08/07 10:53:04 INFO mapreduce.Job: Running job: job_local1197195623_0001 15/08/07 10:53:04 INFO mapred.LocalJobRunner: OutputCommitter set in config null 15/08/07 10:53:04 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1 15/08/07 10:53:04 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 15/08/07 10:53:04 INFO mapred.LocalJobRunner: Waiting for map tasks 15/08/07 10:53:04 INFO mapred.LocalJobRunner: Starting task: attempt_local1197195623_0001_m_000000_0 15/08/07 10:53:04 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1 15/08/07 10:53:04 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 15/08/07 10:53:04 INFO mapred.MapTask: Processing split: hdfs://Big4:8020/user/hadoop/k/20150804/s0104 15/08/07 10:53:05 INFO mapreduce.Job: Job job_local1197195623_0001 running in uber mode : false 15/08/07 10:53:05 INFO mapreduce.Job: map 0% reduce 0% 15/08/07 10:53:05 INFO mapred.LocalJobRunner: map task executor complete. 15/08/07 10:53:05 WARN mapred.LocalJobRunner: job_local 1197195623_0001 java.lang.Exception: java.lang.NullPointerException at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) Caused by: java.lang.NullPointerException at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1017) at java.lang.Double.parseDouble(Double.java:540) at kstar_mr_get_idl_data.K$AMapper.map(K_mr.java:89) at kstar_mr_get_idl_data.K$AMapper.map(Kr_mr.java:1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 15/08/07 10:53:06 INFO mapreduce.Job: Job job_local1197195623_0001 failed with state FAILED due to: NA 15/08/07 10:53:06 INFO mapreduce.Job: Counters: 0 at ChildProcess.exithandler (child_process.js:751:12) at ChildProcess.emit (events.js:110:17) at maybeClose (child_process.js:1015:16) at Process.ChildProcess._handle.onexit (child_process.js:1087:5) hadoop@B:~/DEVS/WORK/w/kev$

java Map 클래스안에서 아래와 같은 부분에서 에러가 났습니다

89번째라인이 문제였는데

context.getConfiguration()에서 가져온값중에 from, to 값이 없는것이 문제였습니다

---

구글링을 해보니 아래와 같은 링크가 있어 참고해서 해결 했습니다

http://stackoverflow.com/questions/13214914/configuration-object-is-null-in-hadoop-mapper

위에 내용중 참고한 부분은 아래 내용입니다

And also try this JobConf job = new JobConf(new Configuration());

MapReduce java 코드 main에서 conf를 아래처럼 만들어 쓰고 있었는데요

JobConf conf = new JobConf();

이렇게 써서 특정 시스템에서는 잘 동작으했으나

시스템을 옮기기 안되는 문제가 있어

JobConf conf = new JobConf(new Configuration());

이런식으로 변경했더니 잘 동작을 했습니다