CDH5.16.1集群企业真正离线部署(三台阿里云主机按量付费)
一.准备工作
1.集群规划
| 节点 | MySQL部署组件 | Parcel文件离线源 | CM服务进程 | 大数据组件 |
| hadoop001 | MySQL | Parcel | Activity Monitor | NN RM DN NM |
| hadoop002 | Alert Publisher Event Server | DN NM | ||
| hadoop003 | Alert Publisher Event Server | DN NM |
2.下载源
CM：
http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz
Parcel：
http://archive.cloudera.com/cdh5/parcels/5.16.1/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
http://archive.cloudera.com/cdh5/parcels/5.16.1/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1
http://archive.cloudera.com/cdh5/parcels/5.16.1/manifest.json
JDK：
https://www.oracle.com/technetwork/java/javase/downloads/java-archive-javase8-2177648.html
MySQL
https://dev.mysql.com/downloads/mysql/5.7.html#downloads
MySQL jdbc jar
http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.47/mysql-connector-java-5.1.47.jar
下载完成后要重命名去掉版本号，
mv mysql-connector-java-5.1.47.jar mysql-connector-java.jar
二.集群节点初始化
1.阿里云上海海区购买3台，按量量付费虚拟机
CentOS7.2操作系统，2核8G最低配置
2.当前笔记本或台式机配置hosts文件
MAC: /etc/hosts
Window: C:\windows\system32\drivers\etc\hosts
172.17.79.118 hadoop001
172.17.79.117 hadoop002
172.17.79.116 hadoop0033.设置所有节点的hosts文件
vim /etc/hosts
172.17.79.118 hadoop001
172.17.79.117 hadoop002
172.17.79.116 hadoop0034.关闭所有节点的防火墙及清空规则
centos关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
iptables -F
iptables -L
一般阿里云的防火墙都是关闭的，但是阿里云服务器上有一个Web的防火墙，要打开7180的端口
5.关闭所有节点的selinux
[root@hadoop001 ~]# vim /etc/selinux/config
将SELINUX=enforcing改为SELINUX=disabled
设置后需要重启才能生效
[root@hadoop001 ~]# reboot
6.查看设置所有节点的时区一致及时钟同步
6.1 时区
[root@hadoop001 ~]# date
Sat Oct 5 19:22:21 CST 2019
[root@hadoop001 ~]# timedatectl
Local time: Sat 2019-10-05 19:21:55 CST
Universal time: Sat 2019-10-05 11:21:55 UTC
RTC time: Sat 2019-10-05 19:21:54
Time zone: Asia/Shanghai (CST, +0800)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: yes
DST active: n/a
Warning: The system is configured to read the RTC time in the local time zone.
This mode can not be fully supported. It will create various problems
with time zone changes and daylight saving time adjustments. The RTC
time is never updated, it relies on external facilities to maintain it.
If at all possible, use RTC in UTC by calling
'timedatectl set-local-rtc 0'.
[root@hadoop001 ~]#
如果时区不一致，使用如下命令
[root@hadoop001 ~]# timedatectl set-timezone Asia/Shanghai6.2 时间
为所有节点安装ntp
[root@hadoop001 ~]# yum install -y ntp选取hadoop001为ntp的主节点
[root@hadoop001 ~]# vi /etc/ntp.conf
server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org
#当外部时间不不可用时，可使⽤用本地硬件时间
server 127.127.1.0 iburst local clock
#允许哪些网段的机器（如果网络时间不可用，使用本网段的时间）来同步时间（查看自己机器的网段）
restrict 172.17.79.0 mask 255.255.255.0 nomodify notrap
开启ntpd及查看状态
[root@hadoop001 ~]# systemctl start ntpd
[root@hadoop001 ~]# systemctl status ntpd
● ntpd.service - Network Time Service
Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2019-10-05 19:27:05 CST; 21min ago
Main PID: 18524 (ntpd)
CGroup: /system.slice/ntpd.service
└─18524 /usr/sbin/ntpd -u ntp:ntp -g
Oct 05 19:27:05 hadoop001 systemd[1]: Starting Network Time Service...
Oct 05 19:27:05 hadoop001 ntpd[18524]: proto: precision = 0.088 usec
Oct 05 19:27:05 hadoop001 ntpd[18524]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Oct 05 19:27:05 hadoop001 systemd[1]: Started Network Time Service.
[root@hadoop001 ~]# 验证时间的配置
[root@hadoop001 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
LOCAL(0) .LOCL. 10 l 1381 64 0 0.000 0.000 0.000
-120.25.115.20 10.137.53.7 2 u - 64 377 47.131 15.104 7.207
10.143.33.49 .INIT. 16 u - 1024 0 0.000 0.000 0.000
+100.100.3.1 10.137.55.181 2 u 21 64 377 34.082 24.574 9.506
*100.100.3.2 10.137.55.181 2 u 30 64 377 25.756 15.962 6.527
+100.100.3.3 10.137.55.181 2 u 26 64 377 30.820 22.105 6.735
+203.107.6.88 100.107.25.114 2 u 20 64 377 12.715 16.232 6.725
10.143.33.50 .INIT. 16 u - 1024 0 0.000 0.000 0.000
10.143.33.51 .INIT. 16 u - 1024 0 0.000 0.000 0.000
10.143.0.44 .INIT. 16 u - 1024 0 0.000 0.000 0.000
10.143.0.45 .INIT. 16 u - 1024 0 0.000 0.000 0.000
10.143.0.46 .INIT. 16 u - 1024 0 0.000 0.000 0.000
+100.100.5.1 10.137.55.181 2 u 25 64 377 26.518 23.554 9.289
-100.100.5.2 10.137.55.181 2 u 12 64 377 26.562 10.974 9.288
+100.100.5.3 10.137.55.181 2 u 38 64 377 25.519 24.802 9.669其他从节点停止并禁用ntpd服务
[root@hadoop002 ~]# vim /etc/ntp.conf
[root@hadoop002 ~]# clear
[root@hadoop002 ~]# systemctl stop ntpd
[root@hadoop002 ~]# systemctl disable ntpd
Removed symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.
[root@hadoop002 ~]# which ntpdate
/usr/sbin/ntpdate
[root@hadoop002 ~]# /usr/sbin/ntpdate hadoop001
5 Oct 19:54:22 ntpdate[18602]: adjust time server 172.17.79.118 offset -0.000208 sec每天凌晨同步hadoop001节点时间
[root@hadoop002 ~]# crontab -e
00 00 * * * /usr/sbin/ntpdate hadoop001
hadoop003机器做同样的配置操作
7.部署集群的JDK
mkdir /usr/java
tar -xzvf jdk-8u45-linux-x64.tar.gz -C /usr/java/
#切记必须修正所属⽤用户及⽤用户组
chown -R root:root /usr/java/jdk1.8.0_45
vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=$JAVA_HOME/bin:$PATH
source /etc/profile
which java8.hadoop001节点离线部署MySQL5.7
https://blog.csdn.net/muyingmiao/article/details/101685786
9.创建CDH的元数据库和用户、amon服务的数据库及用户
mysql> create database cmf DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)
mysql> create database amon DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)
mysql> grant all on cmf.* TO 'cmf'@'%' IDENTIFIED BY 'root';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'root';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
10.hadoop001节点部署mysql jdbc jar
[root@hadoop001 ~]# mkdir -p /usr/share/java/
[root@hadoop001 ~]# cd cdh5.16.1/
[root@hadoop001 cdh5.16.1]# ll
total 3605080
-rw-r--r-- 1 root root 2127506677 Oct 5 20:54 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
-rw-r--r-- 1 root root 41 Oct 5 19:51 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1
-rw-r--r-- 1 root root 841524318 Oct 5 20:27 cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz
-rw-r--r-- 1 root root 173271626 Oct 5 19:40 jdk-8u45-linux-x64.gz
-rw-r--r-- 1 root root 66538 Oct 5 19:32 manifest.json
-rw-r--r-- 1 root root 548193637 Oct 5 19:48 mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz
-rw-r--r-- 1 root root 1007502 Oct 5 19:32 mysql-connector-java-5.1.47.jar
[root@hadoop001 cdh5.16.1]# cp mysql-connector-java-5.1.47.jar /usr/share/java/
[root@hadoop001 cdh5.16.1]# cd /usr/share/java/
[root@hadoop001 java]# ll
total 984
-rw-r--r-- 1 root root 1007502 Oct 5 22:22 mysql-connector-java-5.1.47.jar
[root@hadoop001 java]# mv mysql-connector-java-5.1.47.jar mysql-connector-java.jar
三.CDH部署
1.1 将cdh安装包分发到其他的从节点机器
[root@hadoop001 cdh5.16.1]# scp cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz hadoop002:/root/cdh5.16.1/
root@hadoop003's password:
cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz 100% 803MB 50.2MB/s 00:16
[root@hadoop001 cdh5.16.1]#
[root@hadoop001 cdh5.16.1]# scp cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz hadoop003:/root/cdh5.16.1/
root@hadoop003's password:
cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz 100% 803MB 50.2MB/s 00:16
[root@hadoop001 cdh5.16.1]#
1.2 所有节点创建目录及解压
mkdir /opt/cloudera-manager
tar -zxvf cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz -C /opt/cloudera-manager/
1.3 所有节点修改agent的配置，指向server的节点hadoop001[root@hadoop001 cloudera-scm-agent]# cd /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/
[root@hadoop001 cloudera-scm-agent]# vim config.ini
[root@hadoop001 cloudera-scm-agent]#
server_host=hadoop0011.4 主节点【hadoop001】修改server的配置
vi /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-server/db.properties
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=hadoop001
com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.password=Ruozedata123456!
com.cloudera.cmf.db.setupType=EXTERNAL
1.5.所有节点创建用户
useradd --system --home=/opt/cloudera-manager/cm-5.16.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
1.6.目录修改用户及用户组
chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager
2.hadoop001节点部署离线parcel源
[离线部署 cm
解压
agent配置 hadoop001-hadoop003
server配置 hadoop001
创建cloudera-scm
文件夹用户用户组修改
parcel文件离线源 hadoop001
]
2.1 部署离线parcel源，将CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel，CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1， manifest.json，上传到hadoop001的cdh5.16.1目录下
[root@hadoop001 ~]# mkdir -p /opt/cloudera/parcel-repo
[root@hadoop001 ~]# cd cdh5.16.1/
[root@hadoop001 cdh5.16.1]# ll
total 3605080
-rw-r--r-- 1 root root 2127506677 Oct 5 20:54 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
-rw-r--r-- 1 root root 41 Oct 5 19:51 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1
-rw-r--r-- 1 root root 841524318 Oct 5 20:27 cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz
-rw-r--r-- 1 root root 173271626 Oct 5 19:40 jdk-8u45-linux-x64.gz
-rw-r--r-- 1 root root 66538 Oct 5 19:32 manifest.json
-rw-r--r-- 1 root root 548193637 Oct 5 19:48 mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz
-rw-r--r-- 1 root root 1007502 Oct 5 19:32 mysql-connector-java-5.1.47.jar
[root@hadoop001 cdh5.16.1]# mv CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel /opt/cloudera/parcel-repo
[root@hadoop001 cdh5.16.1]# mv CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha
[root@hadoop001 cdh5.16.1]# mv manifest.json /opt/cloudera/parcel-repo
[root@hadoop001 ~]# cd /opt/cloudera/parcel-repo
## 校验O包裹文件
[root@hadoop001 parcel-repo]# ll
total 2077720
-rw-r--r-- 1 root root 2127506677 Oct 22 19:15 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
-rw-r--r-- 1 cloudera-scm cloudera-scm 41 Oct 22 20:04 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha
-rw-r--r-- 1 cloudera-scm cloudera-scm 66538 Oct 22 20:04 manifest.json
[root@hadoop001 parcel-repo]# /usr/bin/sha1sum CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
703728dfa7690861ecd3a9bcd412b04ac8de7148 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
2.2 目录修改用户及用户组
[root@hadoop001 cdh5.16.1]# chown -R cloudera-scm:cloudera-scm /opt/cloudera/
3.所有节点创建软件安装目录、用户及用户组权限（所有节点创建大数据软件安装目录，用户和用户组权限）
[root@hadoop00* parcel-repo]# mkdir -p /opt/cloudera/parcels
[root@hadoop00* parcel-repo]# chown -R cloudera-scm:cloudera-scm /opt/cloudera/
4.hadoop001节点启动Server
4.1.启动server
/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server start
[root@hadoop001 cloudera]# cd /opt/cloudera-manager/cm-5.16.1/etc/init.d
[root@hadoop001 init.d]# ll
total 32
-rwxr-xr-x 1 cloudera-scm cloudera-scm 8871 Nov 21 2018 cloudera-scm-agent
-rwxr-xr-x 1 cloudera-scm cloudera-scm 8417 Nov 21 2018 cloudera-scm-server
-rwxr-xr-x 1 cloudera-scm cloudera-scm 4444 Nov 21 2018 cloudera-scm-server-db
[root@hadoop001 init.d]# ./cloudera-scm-server start
Starting cloudera-scm-server: [ OK ]打开另一个terminal，查看server的启动日志
等待一分钟，看到7180的端口
[root@hadoop001 cloudera-scm-server]# cd /opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-server
[root@hadoop001 cloudera-scm-server]# ll
total 44
-rw-r----- 1 root root 33569 Oct 23 17:31 cloudera-scm-server.log
-rw-r--r-- 1 root root 7296 Oct 23 17:31 cloudera-scm-server.out
-rw-r----- 1 root root 0 Oct 23 17:31 cmf-server-perf.log
[root@hadoop001 cloudera-scm-server]# tail -f cloudera-scm-server.log
........
........
cloudera-scm-server]
2019-10-23 17:43:09,884 INFO WebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.4
2019-10-23 17:43:09,885 INFO WebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7180
2019-10-23 17:43:09,885 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
2019-10-23 17:43:10,168 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 10% complete
2019-10-23 17:43:10,995 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 20% complete
2019-10-23 17:43:11,777 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 30% complete
2019-10-23 17:43:12,548 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 40% complete
2019-10-23 17:43:13,314 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 50% complete
2019-10-23 17:43:14,070 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 60% complete
2019-10-23 17:43:14,846 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 70% complete
2019-10-23 17:43:15,628 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 80% complete
2019-10-23 17:43:16,408 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: ... 90% complete
2019-10-23 17:43:17,201 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Hashed 1 file(s) (2127506677 bytes) in 4058 pieces (4058 expected) in 9791.8ms.
2019-10-23 17:43:17,227 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Single-file torrent information:
2019-10-23 17:43:17,227 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Torrent name: CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
2019-10-23 17:43:17,227 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Announced at: Seems to be trackerless
2019-10-23 17:43:17,227 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Created on..: Wed Oct 23 17:43:07 CST 2019
2019-10-23 17:43:17,227 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Created by..: cm-server
2019-10-23 17:43:17,227 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Pieces......: 4058 piece(s) (524288 byte(s)/piece)
2019-10-23 17:43:17,228 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Total size..: 2,127,506,677 byte(s)然后所有的节点启动agent：
/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent start
4.2.阿里里云web界面，设置该hadoop001节点防火墙放开7180端口
4.3.等待1min，打开 http://hadoop001:7180 账号密码:admin/admin
4.4.假如打不不开，去看server的log，根据错误仔细排查错误，日志目录如下
[root@hadoop001 cloudera-scm-server]# pwd
/opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-server
5.所有节点启动Agent（包括hadoop001主节点）
/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent start
6.接下来，全部Web界⾯面操作
http://ip:7180/
账号密码:admin/admin
7 欢迎使用Cloudera Manager--最终用户许可条款与条件。勾选
8.欢迎使用Cloudera Manager--您想要部署哪个版本？选择Cloudera Express免费版本
9.感谢您选择 Cloudera Manager 和 CDH。
10 为 CDH 群集安装指定主机。
11 群集安装
选择存储库
12.集群安装--正在安装选定Parcel假如
本地parcel离线源配置正确，则"下载"阶段瞬间完成，其余阶段视节点数与内部网络情况决定。
如果页面加载很慢，去/opt/cloudera/parcel-repo查看，如下
[root@hadoop001 parcel-repo]# ll
total 2077800
-rw-r--r-- 1 cloudera-scm cloudera-scm 2127506677 Nov 2 22:16 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
-rw-r--r-- 1 cloudera-scm cloudera-scm 41 Nov 2 21:24 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha
-rw-r----- 1 root root 181326 Nov 2 22:56 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.part
-rw-r----- 1 root root 81326 Nov 2 22:56 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.torrent
-rw-r--r-- 1 cloudera-scm cloudera-scm 66538 Nov 2 21:16 manifest.json
[root@hadoop001 parcel-repo]#
那么就要卸载CDH，重新安装（不用卸载mysql）https://blog.csdn.net/muyingmiao/article/details/102741735
13.检查主机正确性
处理大页面的问题：
Transparent Huge Page Compaction is enabled and can cause significant performance problems. Run "echo never > /sys/kernel/mm/transparent_hugepage/defrag" and "echo never > /sys/kernel/mm/transparent_hugepage/enabled" to disable this, and then add the same command to an init script such as /etc/rc.local so it will be set on system reboot.
三个节点一起做：
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled然后在页面点【run again】，黄色警告消失。
14.自定义服务，选择部署Zookeeper、HDFS、Yarn服务
15 自定义⻆角色分配
16 数据库设置
17 审改设置，默认即可
18 首次运行
19.恭喜您!
20主页图
错误排查
1.A package was not selected.
原因:
测试连接时，等待时间过长，我就点了返回键重新加载，然后出现packet找不到的异常。
解决:
返回到选择大数据组件的页面后，重新进行操作，就可以测试成功了。
2.无法找到主机的 NTP 服务，或该服务未响应时钟偏差请求。
通过在服务器上使用date命令查看时间是一致的，所以点击上图的Suppress...
再次查看主页：