보물창고/Programming2015.07.15 23:30





FFT DFT java example 이론 정리와 java 예제 코드 구성


FFT를 알아보기 위해 찾아본 자료를 정리 했습니다



FFT는 신호를 바꾸는 알고리즘인데요 이론적인 기초를 파악할 수 있는 링크 입니다

  1. signal
    1. http://dmusiclab.com/rb/?c=4/14&p=3&uid=3
    2. 시간축 표현과 주파수측 표현에 대한 이해

  2. Fourier Transform (1)
    1. http://dmusiclab.com/rb/?c=4/14&p=2&uid=10
    2. 푸리에 이론에 대한 설명
    3. 모든 주기 신호는 정현팓르의 합으로 이루어질 수 있다

  3. Fourier Transform (2)
    1. http://dmusiclab.com/rb/?c=4/14&p=2&uid=11
    2. 시각축 그래프를 주파수측 그래프로 바꿔지는 과정을 하나하나 진행

  4. Discrete Fourier Transform
    1. http://dmusiclab.com/rb/?c=4/14&p=2&uid=12
    2. 이산 푸리에 변환에 대한 설명
    3. 단점 설명 Fast Fourier Transform

  5. Fast Fourier Transform (1)
    1. http://dmusiclab.com/rb/?c=4/14&p=2&uid=13
    2. 쿨리-튜키 알고리즘 설명

  6. Fast Fourier Transform (2)
    1. http://dmusiclab.com/rb/?c=4/14&p=2&uid=14
    2. 빠르게 계산되는 이유 설명


위에 링크로 정리한 글이 아닌 다른 글도 유용한 글이 많으니 개념 잡으시는데 참고하세요~







FFT java example 내용입니다


참고한 github

https://github.com/bytedeco/javacpp-presets/tree/master/fftw


java maven 프로젝트에서 pom.xml에  dependency만 추가해주면 바로 FFT를 실험 할 수 있습니다

예제 코드도 위 github에 있으니 바로 적용해서 실행해 볼 수 있습니다

README 파일 그대로 따라하면 아래처럼 실행시켜볼 수 있습니다



위 프로젝트 압축파일 추가합니다

mavne 프로젝트 입니다

fft2 (2).zip

신고





Posted by 파란물
보물창고/Big Data2014.07.24 21:00






Spark WordCount MapReduce 샘플 코드 분석 중입니다

코드 원본 중간중간에 데이터를 출력해서 확인하고 있습니다









실행 환경

CentOs 6.5, CDH5(hadoop2.3.0 CDH 5.0.3, Spark1.0.0 CDH5.1.0 )

Spark on YARN - client mode

한대의 노드에서 테스트


실행 스크립트

spark-submit --class com.cloudera.sparkwordcount.JavaWordCount --deploy-mode client --master yarn /data01/SparkExample/sparkwordcount-0.0.1-SNAPSHOT.jar SparkExample/inputfile.txt


코드 원본위치

https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/JavaWordCount.java


Input file data (inputfile.txt)

apple

banana counter

counter one two three

three one

five seven eight

twenty one three five counter six

one siz helga

apple banana fiver 



코드에 주석

package com.cloudera.sparkwordcount;

/*

* Licensed to the Apache Software Foundation (ASF) under one or more

* contributor license agreements. See the NOTICE file distributed with

* this work for additional information regarding copyright ownership.

* The ASF licenses this file to You under the Apache License, Version 2.0

* (the "License"); you may not use this file except in compliance with

* the License. You may obtain a copy of the License at

*

* http://www.apache.org/licenses/LICENSE-2.0

*

* Unless required by applicable law or agreed to in writing, software

* distributed under the License is distributed on an "AS IS" BASIS,

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

* See the License for the specific language governing permissions and

* limitations under the License.

*/


import java.util.Arrays;

import java.util.List;

import java.util.regex.Pattern;


import org.apache.spark.SparkConf;

import org.apache.spark.api.java.JavaPairRDD;

import org.apache.spark.api.java.JavaRDD;

import org.apache.spark.api.java.JavaSparkContext;

import org.apache.spark.api.java.function.FlatMapFunction;

import org.apache.spark.api.java.function.Function2;

import org.apache.spark.api.java.function.PairFunction;


import scala.Tuple2;


public final class JavaWordCount {

private static final Pattern SPACE = Pattern.compile(" ");


public static void main(String[] args) throws Exception {


if (args.length < 1) {

System.err.println("Usage: JavaWordCount <file>");

System.exit(1);

}


SparkConf sparkConf = new SparkConf().setAppName("JavaWordCount");

JavaSparkContext ctx = new JavaSparkContext(sparkConf);

JavaRDD<String> lines = ctx.textFile(args[0], 1);


JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {

@Override

public Iterable<String> call(String s) {

System.out.println("words");

System.out.println("String s : "+ s); // String s : apple, String s : banana counter, String s : counter one two three, String s : three one , ...

Iterable<String> a = Arrays.asList(SPACE.split(s));

System.out.println("Arrays.asList(SPACE.split(s) : " + a); //Arrays.asList(SPACE.split(s) : [apple], Arrays.asList(SPACE.split(s) : [banana, counter], Arrays.asList(SPACE.split(s) : [counter, one, two, three], ...

return a;

}

});


JavaPairRDD<String, Integer> ones = words.mapToPair(new PairFunction<String, String, Integer>() {

@Override

public Tuple2<String, Integer> call(String s) {

System.out.println("ones");

System.out.println("String s : "+ s); // String s : apple, String s : banana, String s : counter, String s : counter, String s : one, String s : two, String s : three, ...

Tuple2<String, Integer> a = new Tuple2<String, Integer>(s, 1);

System.out.println("new Tuple2<String, Integer>(s, 1) : " + a); // new Tuple2<String, Integer>(s, 1) : (apple,1), (banana,1), (counter,1), (counter,1)

return a;

}

});


// 하둡 reduce와 마찬가지로 정렬되어옮?

JavaPairRDD<String, Integer> counts = ones.reduceByKey(new Function2<Integer, Integer, Integer>() {

// :1, :1, :1의 값 부분을 계속 더하는 구조?

@Override

public Integer call(Integer i1, Integer i2) {

System.out.println("counts");

System.out.println("Integer i1 : "+i1+ " Integer i2 : "+i2); // Integer i1 : 1 Integer i2 : 1, Integer i1 : 2 Integer i2 : 1, Integer i1 : 1 Integer i2 : 1, Integer i1 : 3 Integer i2 : 1

System.out.println("i1 + i2 : "+(i1 + i2)); // i1 + i2 : 2, i1 + i2 : 2, i1 + i2 : 3, i1 + i2 : 4

return i1 + i2;

}

});


//여기서 부터는 클라이언트 드라이버에서 동작 (위에는 YARN내부에서 동작, logHDFS에 있음)

List<Tuple2<String, Integer>> output = counts.collect();

System.out.println("output");

System.out.println("output : "+ output); // output : [(counter,3), (banana,2), (two,1), (seven,1), (eight,1), (one,4), (fiver,1), (siz,1), (,1), (six,1), (apple,2), (three,3), (helga,1), (five,2), (twenty,1)]

for (Tuple2<?,?> tuple : output) {

System.out.println(tuple._1() + ": " + tuple._2());

}

ctx.stop();

}

}


log 결과1 - HDFS

/var/log/hadoop-yarn/apps/inbrein/logs/application_1405498888069_0217

words

String s : apple

Arrays.asList(SPACE.split(s) : [apple]

ones

String s : apple

new Tuple2<String, Integer>(s, 1) : (apple,1)

words

String s : banana counter

Arrays.asList(SPACE.split(s) : [banana, counter]

ones

String s : banana

new Tuple2<String, Integer>(s, 1) : (banana,1)

ones

String s : counter

new Tuple2<String, Integer>(s, 1) : (counter,1)

words

String s : counter one two three

Arrays.asList(SPACE.split(s) : [counter, one, two, three]

ones

String s : counter

new Tuple2<String, Integer>(s, 1) : (counter,1)

counts

Integer i1 : 1 Integer i2 : 1

i1 + i2 : 2

ones

String s : one

new Tuple2<String, Integer>(s, 1) : (one,1)

ones

String s : two

new Tuple2<String, Integer>(s, 1) : (two,1)

ones

String s : three

new Tuple2<String, Integer>(s, 1) : (three,1)

words

String s : three one

Arrays.asList(SPACE.split(s) : [three, one]

ones

String s : three

new Tuple2<String, Integer>(s, 1) : (three,1)

counts

Integer i1 : 1 Integer i2 : 1

i1 + i2 : 2

ones

String s : one

new Tuple2<String, Integer>(s, 1) : (one,1)

counts

Integer i1 : 1 Integer i2 : 1

i1 + i2 : 2

words

String s : five seven eight

Arrays.asList(SPACE.split(s) : [five, seven, eight]

ones

String s : five

new Tuple2<String, Integer>(s, 1) : (five,1)

ones

String s : seven

new Tuple2<String, Integer>(s, 1) : (seven,1)

ones

String s : eight

new Tuple2<String, Integer>(s, 1) : (eight,1)

words

String s : twenty one three five counter six

Arrays.asList(SPACE.split(s) : [twenty, one, three, five, counter, six]

ones

String s : twenty

new Tuple2<String, Integer>(s, 1) : (twenty,1)

ones

String s : one

new Tuple2<String, Integer>(s, 1) : (one,1)

counts

Integer i1 : 2 Integer i2 : 1

i1 + i2 : 3

ones

String s : three

new Tuple2<String, Integer>(s, 1) : (three,1)

counts

Integer i1 : 2 Integer i2 : 1

i1 + i2 : 3

ones

String s : five

new Tuple2<String, Integer>(s, 1) : (five,1)

counts

Integer i1 : 1 Integer i2 : 1

i1 + i2 : 2

ones

String s : counter

new Tuple2<String, Integer>(s, 1) : (counter,1)

counts

Integer i1 : 2 Integer i2 : 1

i1 + i2 : 3

ones

String s : six

new Tuple2<String, Integer>(s, 1) : (six,1)

words

String s : one siz helga

Arrays.asList(SPACE.split(s) : [one, siz, helga]

ones

String s : one

new Tuple2<String, Integer>(s, 1) : (one,1)

counts

Integer i1 : 3 Integer i2 : 1

i1 + i2 : 4

ones

String s : siz

new Tuple2<String, Integer>(s, 1) : (siz,1)

ones

String s : helga

new Tuple2<String, Integer>(s, 1) : (helga,1)

words

String s : apple banana fiver

Arrays.asList(SPACE.split(s) : [apple, banana, fiver]

ones

String s : apple

new Tuple2<String, Integer>(s, 1) : (apple,1)

counts

Integer i1 : 1 Integer i2 : 1

i1 + i2 : 2

ones

String s : banana

new Tuple2<String, Integer>(s, 1) : (banana,1)

counts

Integer i1 : 1 Integer i2 : 1

i1 + i2 : 2

ones

String s : fiver

new Tuple2<String, Integer>(s, 1) : (fiver,1)

words

String s : 

Arrays.asList(SPACE.split(s) : []

ones

String s : 

new Tuple2<String, Integer>(s, 1) : (,1)


log 결과2 - console
stdout

output

output : [(counter,3), (banana,2), (two,1), (seven,1), (eight,1), (one,4), (fiver,1), (siz,1), (,1), (six,1), (apple,2), (three,3), (helga,1), (five,2), (twenty,1)]

counter: 3

banana: 2

two: 1

seven: 1

eight: 1

one: 4

fiver: 1

siz: 1

: 1

six: 1

apple: 2

three: 3

helga: 1

five: 2

twenty: 1


신고





Posted by 파란물