Hadoop2.2.0源代码编译

一、环境说明

虚拟软件:VMware Workstation 10

虚拟机配置:

RHEL Server release 6.5 (Santiago) 2.6.32-431.el6.x86_64

cpu:4核心,内存:4G,硬盘:50G

二、前提条件:

1:将rhel6.5的iso文件作为yum源

2:hadoop-2.2.0-src.tar.gz

3:安装JDK 1.6.0_43

4:安装并配置apache-maven 3.0.5(apache-maven-3.0.5-bin.tar.gz)

源码中BUILDING.txt中要求使用3.0，从hadoop2.0版本以后使用maven编译，之前用Ant)

解压并配置环境变量

 

mvn -version

5:安装并配置apache-ant-1.9.3-bin.zip（下载二进制版本的,这个需要编译安装findbugs)

解压并配置环境变量

 

ant -version

6:下载并安装cmake cmake-2.8.12.1,安装命令如下:

  tar -zxvf cmake-2.8.12.1.tar.gz

  cd cmake-2.8.12.1

  ./bootstrap

  make

  make install

 

  检查安装是否正确

cmake --version(如果能正确显示版本号,则说明安装正确)

7:下载并安装配置findbugs-2.0.2-source.zip

http://sourceforge.jp/projects/sfnet_findbugs/releases/

使用ant编译安装。如果不编译安装则编译的时候会报

hadoop-common-project/hadoop-common/${env.FINDBUGS_HOME}/src/xsl/default.xsl doesn’t exist. -> [Help 1]

进入到解压后的目录，直接运行ant命令

 

如果不安装,则在编译时会报如下错误:

Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (site) on project hadoop-common: An Ant BuildException has occured

8:安装zlib-devel

默认情况下,系统没有安装zlib-devel

yum install zlib-devel

如果不安装,则在编译时会报如下错误:

Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-common

9: protobuf-2.5.0

yum install gcc-c++ (如果不安装,则cmake configure失败)

./configure

make

make check

make install

检查安装是否正确

protoc --version((如果能正确显示版本号,则说明安装正确)

三、hadoop2.2源码编译

1：进入hadoop2.2.0解压后的源码目录

2：执行mvn命令编译，此过程需要连接网络，编译的速度取决于你的网速

mvn clean package -Pdist,native -DskipTests -Dtar

 

2.1.Create binary distribution without native code and without documentation:

$ mvn package -Pdist -DskipTests -Dtar

2.2.Create binary distribution with native code and with documentation:

$ mvn package -Pdist,native,docs -DskipTests -Dtar

2.3.Create source distribution:

$ mvn package -Psrc -DskipTests

2.4.Create source and binary distributions with native code and documentation:

$ mvn package -Pdist,native,docs,src -DskipTests -Dtar

2.5.Create a local staging version of the website (in /tmp/hadoop-site)

$ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site

3：编译后的项目发布版本在hadoop-2.2.0-src/hadoop-dist/target/目录下

hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/

 

四、安装hadoop 单点伪分布式模式

1：配置ssh互信

ssh-keygen -t dsa （或行ssh-keygen -t rsa -P "" 加上-P ""参数只需要一次回车就可以执行完毕。不加需要三次回车）

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys 

如果仍然需要输入密码，则执行如下命令

chmod 600 ~/.ssh/authorized_keys

2：将编译后的hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0复制到/data/hadoop/目录下

3：建议软链接 ln -s hadoop-2.2.0 hadoop2

4: 在用户的.bash_profile增加如下变量

export HADOOP_HOME=/data/hadoop/hadoop2

export PATH=$PATH:$HADOOP_HOME/bin

export PATH=$PATH:$HADOOP_HOME/sbin

export HADOOP_MAPRED_HOME=${HADOOP_HOME}

export HADOOP_COMMON_HOME=${HADOOP_HOME}

export HADOOP_HDFS_HOME=${HADOOP_HOME}

export YARN_HOME=${HADOOP_HOME}

5:建议data.dir及namenode.dir目录

mkdir hdfs

mkdir namenode

chmod -R 755 hdfs

6:修改hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.6.0_43

export HADOOP_HOME=/data/hadoop/hadoop2

7:修改core-site.xml,hdfs-site.xml,mapred-site.xml,yarn-site.xml，内容见附录2

8：格式化HDFS文件系统

执行命令 hadoop namenode -format

9：启动hdfs,yarn

start-dfs.sh

start-yarn.sh

10:验证启动是否成功

jps

如果显示如下6个进程，则启动成功

53244 ResourceManager

53083 SecondaryNameNode

52928 DataNode

53640 Jps

52810 NameNode

53348 NodeManager

五、运行自带的wordcount例子

hadoop fs -mkdir /tmp

hadoop fs -mkdir /tmp/input

hadoop fs -put /usr/hadoop/test.txt /tmp/input

cd /data/hadoop/hadoop2/share/hadoop/mapreduce

hadoop jar hadoop-mapreduce-examples-2.2.0.jar wordcount /tmp/input /tmp/output

如果能正确运行，则hadoop安装配置正确

 

 

六、附录1：

设置的环境变量（/etc/profile,编辑后运行source /etc/profile/ 使配置生效）

#java set

export JAVA_HOME=/usr/java/jdk1.6.0_43

export JRE_HOME=/usr/java/jdk1.6.0_43/jre

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$PATH:$JAVA_HOME/bin

 

#maven set

export M2_HOME=/home/soft/maven

export PATH=$PATH:$M2_HOME/bin

 

#ant

export ANT_HOME=/home/soft/apache-ant-1.9.3

export PATH=$PATH:$ANT_HOME/bin

 

#findbugs

export FINDBUGS_HOME=/home/soft/findbugs-2.0.2

export PATH=$PATH:$FINDBUGS_HOME/bin

 

   附录2：core-site.xml,hdfs-site.xml,mapred-site.xml,yarn-site.xml的文件内容

core-site.xml内容：

<configuration>

   <property>

<name>fs.defaultFS</name>

<value>hdfs://vdata.kt:8020</value>

   </property>

</configuration>

 

hdfs-site.xml内容：

<configuration>

   <property>

<name>dfs.name.dir</name>

<value>/data/hadoop/namenode</value>

   </property>

   <property>

<name>dfs.data.dir</name>

<value>/data/hadoop/hdfs</value>

   </property>

   <property>

<name>dfs.replication</name>

<value>1</value>

   </property>  

   <property>

<name>dfs.permissions</name>

<value>false</value>

   </property>

</configuration>

 

mapred-site.xml内容：

<configuration>

   <property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

   </property> 

</configuration>

 

yarn-site.xml内容：

<configuration>

   <property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

    </property>

</configuration>