Hadoop3.2在Ubuntu18.04上的安装

背景与机器

发现大数据管理平台HDP和CDH合并了，并且开始收费了。从头安装还是比较麻烦的，于是胡乱记录下Hadoop3.2在Ubuntu18.04上的安装笔记。

准备了3台ubuntu 18.04
server181 - 192.168.0.121 (master)
server182 - 192.168.0.122 (slave1)
server183 - 192.168.0.123 (slave2)

准备与基础环境

1. 下载jdk1.8和hadoop3.2
2. 免密登录配置
3. 设置hosts (注意应该把前面两行注释掉，否则可能datanode可能会不显示在UI上)

#vi /etc/hosts
#127.0.0.1   localhost
#::1         localhost
192.168.0.121 server181
192.168.0.122 server182
192.168.0.123 server183

4. 复制文件到目录并修改权限

sudo tar -zxvf hadoop-3.2.2.tar.gz -C /usr/local/
sudo tar -zxvf jdk-8u291-linux-x64.tar.gz -C /usr/local/
sudo chown -R winfred hadoop-3.2.2/
sudo chown -R winfred /usr/local/jdk1.8.0_291

5. 设置环境

# vi .bashrc
# source .bashrc     添加完成之后执行这个条命令，让其生效。
export JAVA_HOME=/usr/local/jdk1.8.0_291
export JRE_HOME={JAVA_HOME}/jre
export CLASSPATH=.:{JAVA_HOME}/lib:{JRE_HOME}/lib
export PATH=.:{JAVA_HOME}/bin:PATH

export HADOOP_HOME=/usr/local/hadoop-3.2.2
export HADOOP_LOG_DIR={HADOOP_HOME}/logs
export PATH={HADOOP_HOME}/sbin:{HADOOP_HOME}/bin:$PATH

hadoop配置

1. core-site.xml

# vi core-site.xml    打开文件，添加以下内容
# 注意以下ip和目录的修改

<property>
    <name>fs.defaultFS</name>
    <value>hdfs://192.168.0.121:9000</value>
</property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/home/winfred/hadoopdata</value>
</property>

2. hadoop-env.sh和yarn-env.sh

# vi hadoop-env.sh  添加JAVA_HOME
# vi yarn-env.sh 同样添加JAVA_HOME
export JAVA_HOME=/usr/local/jdk1.8.0_291

3. 配置hdfs-site.xml

# vi hdfs-site.xml

<property>
    <name>dfs.replication</name>
    <value>3</value>
</property>
<property>
     <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
     <value>false</value>
</property>

4. yarn-site.xml

# vi yarn-site.xml
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.resourcemanager.address</name>
    <value>192.168.0.121:18040</value>
</property>                                     
<property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>192.168.0.121:18030</value>
</property>
<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>192.168.0.121:18025</value>
</property>
<property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>192.168.0.121:18141</value>
</property>
<property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>192.168.0.121:18088</value>
</property>

5. mapred-site.xml

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

6. works

# vi bash 配置datanode
server182
server183

7. 将配置好的内容复制到server182, server183中

在server181上运行
sudo scp -r /usr/local/hadoop-3.2.2 winfred@192.168.0.122:~/
sudo scp -r /usr/local/hadoop-3.2.2 winfred@192.168.0.123:~/
sudo scp -r /usr/local/jdk1.8.0_291 winfred@192.168.0.122:~/
sudo scp -r /usr/local/jdk1.8.0_291 winfred@192.168.0.123:~/

#在server182和server183上运行
sudo mv /home/winfred/hadoop-3.2.2/ /usr/local/
sudo mv /home/winfred/jdk1.8.0_291/ /usr/local/
source .bashrc

启动

hdfs namenode -format
start-all.sh 

验证：
http://192.168.0.121:18088/cluster/nodes
http://192.168.0.121:9870/dfshealth.html#tab-overview

参考