Hadoop编译支持5种压缩格式

Hadoop-2.6.0-CDH-5.7.0版本编译：

前置环境：1. JDK为jdk-7u80-linux-x64.tar.gz

2. Maven为apache-maven-3.3.9-bin.zip

3. findbugs为findbugs-1.3.9.zip

4. protoc为protobuf-2.5.0.tar.gz

5. hadoop-2.6.0-cdh5.7.0-src.tar.gzprotoc为protobuf-2.5.0.tar.gz

组件名称	组件版本	百度网盘链接
vm	vm10	链接：https://pan.baidu.com/s/1N5i8p8htXz9H_v__YNV1lA 提取码：yasn
centos	centos6.7	链接：https://pan.baidu.com/s/1Z_6AcQ_WnvKz1ga_VCSI9Q 提取码：a24x
Hadoop	Hadoop-2.6.0-cdh5.7.0-src.tar.gz	链接：https://pan.baidu.com/s/1uRMGIhLSL9QHT-Ee4F16jw 提取码：jb1d
jdk	jdk-7u80-linux-x64.tar.gz	链接：https://pan.baidu.com/s/1xSCQ8rjABVI-zDFQS5nCPA 提取码：lfze
maven	apache-maven-3.3.9-bin.tar.gz	链接：https://pan.baidu.com/s/1ddkdkLW7r7ahFZmgACGkVw 提取码：fdfz
protobuf	protobuf-2.5.0.tar.gz	链接：https://pan.baidu.com/s/1RSNZGd_ThwknMB3vDkEfhQ 提取码：hvc2

1)下载：

下载hadoop-2.6.0-cdh5.7.0-src.tar.gz

1	下载地址：http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0-src.tar.gz

下载jdk-7u80-linux-x64.tar.gz

1	下载地址：http://ghaffarian.net/downloads/Java/

2）上传：

上传hadoop-2.6.0-cdh5.7.0-src.tar.gz至/source下

上传jdk-7u80-linux-x64.tar.gz、apache-maven-3.3.9-bin.zip、findbugs-1.3.9.zip、protobuf-2.5.0.tar.gz至/software

解压文件

3）查看环境要求：

1
2
3

[root@hadoop001 ~]$ cd source/hadoop-2.6.0-cdh5.7.0

[root@hadoop001 hadoop-2.6.0-cdh5.7.0]$ cat BUILDING.txt

Requirements:

* Windows System

* JDK 1.7+

* Maven 3.0 or later

* Findbugs 1.3.9 (if running findbugs)

* ProtocolBuffer 2.5.0

* CMake 2.6 or newer

* Windows SDK or Visual Studio 2010 Professional

* Unix command-line tools from GnuWin32 or Cygwin: sh, mkdir, rm, cp, tar, gzip

* zlib headers (if building native code bindings for zlib)

* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).

Do not use Visual Studio Express.  It does not support compiling for 64-bit,
which is problematic if running a 64-bit system.  The Windows SDK is free to
download here:

http://www.microsoft.com/en-us/download/details.aspx?id=8279

4）配置maven目录：

1
2
3

[root@hadoop001 ~]$ cd /usr/local/apache-maven-3.3.9/conf

[root@hadoop001 conf]$ vi settings.xml

配置本地仓库：

配置阿里源：

5）预编译安装：

[root@hadoop001 protobuf-2.5.0]# yum install -y gcc gcc-c++ make cmake

[root@hadoop001 protobuf-2.5.0]# ./configure --prefix=/usr/local/protobuf

[root@hadoop001 protobuf-2.5.0]# make && make install

6）配置环境变量：

1	[hadoop@hadoop001 ~]$ vi .bash_profile

PATH=$PATH:$HOME/bin

export PATH

export JAVA_HOME=/usr/java/jdk1.7.0_80

export MVN_HOME=/home/hadoop/app/apache-maven-3.3.9

export FINDBUGS_HOME=/home/hadoop/app/findbugs-1.3.9

export PROTOC_HOME=/usr/local/protobuf

export PATH=$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MVN_HOME/bin:$JAVA_HOME/bin:$PATH

7）yum源安装其他组件

1
2
3

[root@hadoop001 ~]# yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool

[root@hadoop001 ~]# yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake

9）开始编译

[root@hadoop001 ~]$ cd source/hadoop-2.6.0-cdh5.7.0

[root@hadoop001 hadoop-2.6.0-cdh5.7.0]$ pwd

/source/hadoop-2.6.0-cdh5.7.0

[root@hadoop001 hadoop-2.6.0-cdh5.7.0]$ mvn clean package -Pdist,native -DskipTests -Dtar

10）开始编译

如果看到BUILD SUCCESS，且没有异常信息，说明hadoop已经编译成功

编译完成后可以看到hadoop的压缩包

通过日志可以知道编译好的压缩包所在位置:/source/hadoop-2.6.0-cdh5.7.0/hadoop-dish/target下

编译前：

编译后：