2.hadoop伪分布式部署-hdfs进程启动的机器修改-yarn部署

发表于
字数统计: 1.1k | 阅读时长 ≈ 6

1.hdfs三个进程要以hadoop002启动：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
	 这里的 hadoop001指的是你的主机名，
	 配置文件在 etc/hadoop目录下：
[hadoop@hadoop002 hadoop]$ ll
total 140
-rw-r--r-- 1 hadoop hadoop   884 Feb 13 22:34 core-site.xml  （存放hdfs，mapredurce，yarn的公共配置信息）
-rw-r--r-- 1 hadoop hadoop  4294 Feb 13 22:30 hadoop-env.sh  JDK目录 hadoop家目录

-rw-r--r-- 1 hadoop hadoop   867 Feb 13 22:34 hdfs-site.xml（存放hdfs独有的配置信息）

-rw-r--r-- 1 hadoop hadoop 11291 Mar 24  2016 log4j.properties

-rw-r--r-- 1 hadoop hadoop   758 Mar 24  2016 mapred-site.xml.template（存放mapreduce独有的配置信息）
-rw-r--r-- 1 hadoop hadoop    10 Mar 24  2016 slaves（这里修改 datanode的机器名称）

-rw-r--r-- 1 hadoop hadoop   690 Mar 24  2016 yarn-site.xml（存放yarn独有的配置信息）

生产 学习: 不用ip部署，统一机器名称hostname部署，如果有一天公司修改了网段，
我们只需要/etc/hosts 修改ip即可即可(第一行 第二行不要删除)

2.修改配置文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
namenode进程:（namenode配置信息在core-site.xml 中 core中配置的是共有的重要的信息）
[hadoop@hadoop002 hadoop]$ vi core-site.xml 

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop002:9000</value>
    </property>
</configuration>

datanode进程:（datanode配置信息在 slaves中）
[hadoop@hadoop002 hadoop]$ vi slaves 
hadoop002

secondarynamenode进程:（secondarynamenode的配置在 hdfs-size.xml中）
	vi hdfs-size.xml
<property>
	<name>dfs.namenode.secondary.http-address</name>
	<value>hadoop002:50090</value>
</property>
<property>
	<name>dfs.namenode.secondary.https-address</name>
	<value>hadoop002:50091</value>
</property>

在修改配置文件信息的失手，我们应该学会查看官网，根据官网的信息来进行修改:

这里有是配置信息的参数名称和信息。

3.重新构建ssh信任关系：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
	 之前的配置也是可以的，但是我们为了统一 重新再来一次配置 [hadoop@hadoop002 ~]$ rm -rf .ssh
[hadoop@hadoop002 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
ca:e4:a1:fc:9f:e2:86:e7:9c:ab:f2:19:7a:70:c5:3d hadoop@hadoop002
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|     . .         |
|      o E        |
|     .o S.       |
|  ...= o         |
|   o+.+          |
|  ..o=+. .       |
|  .++*B+o        |
+-----------------+
[hadoop@hadoop002 ~]$ cd .ssh
[hadoop@hadoop002 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1675 Feb 16 20:27 id_rsa
-rw-r--r-- 1 hadoop hadoop  398 Feb 16 20:27 id_rsa.pub

[hadoop@hadoop002 .ssh]$ cat id_rsa.pub >>  authorized_keys

[hadoop@hadoop002 .ssh]$ ll
total 12
-rw-rw-r-- 1 hadoop hadoop  398 Feb 16 20:27 authorized_keys
-rw------- 1 hadoop hadoop 1675 Feb 16 20:27 id_rsa
-rw-r--r-- 1 hadoop hadoop  398 Feb 16 20:27 id_rsa.pub

[hadoop@hadoop002 .ssh]$ cd

[hadoop@hadoop002 ~]$ cd app/hadoop-2.6.0-cdh5.7.0

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh 
19/02/16 20:28:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [hadoop002]
The authenticity of host 'hadoop002 (172.31.236.240)' can't be established.
RSA key fingerprint is b1:94:33:ec:95:89:bf:06:3b:ef:30:2f:d7:8e:d2:4c.
Are you sure you want to continue connecting (yes/no)? yes
hadoop002: Warning: Permanently added 'hadoop002,172.31.236.240' (RSA) to the list of known hosts.
hadoop@hadoop002's password: 

[2]+  Stopped                 sbin/start-dfs.sh

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ jps

962 Jps
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ ps -ef|grep start-dfs.sh

hadoop     790   349  0 20:25 pts/0    00:00:00 bash sbin/start-dfs.sh
hadoop     887   349  0 20:28 pts/0    00:00:00 bash sbin/start-dfs.sh
hadoop     977   349  0 20:28 pts/0    00:00:00 grep start-dfs.sh

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ kill -9 790 887

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ cd
[1]-  Killed                  sbin/start-dfs.sh  (wd: ~/app/hadoop-2.6.0-cdh5.7.0)
(wd now: ~)
[2]+  Killed                  sbin/start-dfs.sh  (wd: ~/app/hadoop-2.6.0-cdh5.7.0)
(wd now: ~)

[hadoop@hadoop002 ~]$ cd .ssh

[hadoop@hadoop002 .ssh]$ ll
total 16
-rw-rw-r-- 1 hadoop hadoop  398 Feb 16 20:27 authorized_keys
-rw------- 1 hadoop hadoop 1675 Feb 16 20:27 id_rsa
-rw-r--r-- 1 hadoop hadoop  398 Feb 16 20:27 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop  406 Feb 16 20:28 known_hosts

[hadoop@hadoop002 .ssh]$ chmod 600 authorized_keys

[hadoop@hadoop002 .ssh]$ 
[hadoop@hadoop002 .ssh]$ cd -
/home/hadoop
[hadoop@hadoop002 ~]$ cd app/hadoop-2.6.0-cdh5.7.0

启动hdfs 发现三个进程都以hadoop002启动
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh 
19/02/16 20:29:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [hadoop002]
hadoop002: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop002.out
hadoop002: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop002.out
Starting secondary namenodes [hadoop002]
hadoop002: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop002.out
19/02/16 20:29:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

本文标题:2.hadoop伪分布式部署-hdfs进程启动的机器修改-yarn部署

文章作者:skygzx

发布时间:2019年04月07日 - 09:07

最后更新:2019年04月07日 - 15:04

原始链接:http://yoursite.com/2019/04/07/2.Hadoop重新部署伪分布式hdfs/

许可协议: 署名-非商业性使用-禁止演绎 4.0 国际 转载请保留原文链接及作者。

-------------本文结束感谢您的阅读-------------
0%