社区微信群开通啦,扫一扫抢先加入社区官方微信群
社区微信群
下载相关安装包
Cloudera Manager 下载地址:
jdk 下载地址:
CDH 安装包下载地址:https://archive.cloudera.com/cdh6/6.1.0/parcels/,需要下载以下文件:
CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel
manifest.json
安装三台 visualbox 虚拟机
参考:https://blog.csdn.net/chirs_chen/article/details/84978941
虚拟机管理软件:Oracle VM VisualBox
虚拟机系统:CentOS-7-x86_64-Minimal-1810.iso
IP地址 | 主机名 |
---|---|
192.168.56.2 | cdh-2-master |
192.168.56.2 | cdh-3-slave |
192.168.56.2 | cdh-4-slave |
系统安装成功后,先修改网卡 /etc/sysconfig/network-scripts/ifcfg-enp0s3 的 ONBOOT 值为yes,重启 network 服务
systemctl restart network
修改主机名及映射主机名(所有节点/重启)
修改主机名:hostnamectl set-hostname
设置映射主机名,将以下内容添加到 /etc/hosts 文件:
192.168.56.2 cdh-2-master
192.168.56.3 cdh-3-slave
192.168.56.4 cdh-4-slave
关闭防火墙和 selinux(所有节点)
查看防火墙:systemctl status firewalld.service
执行关闭命令:systemctl stop firewalld.service
执行开机禁用防火墙命令:systemctl disable firewalld.service
关闭selinux:vi /etc/sysconfig/selinux 将 SELINUX=enforcing 改为 SELINUX=disabled
时钟同步(所有节点)
安装 NTP
yum -y install ntp
编辑 /etc/ntp.conf 文件添加 NTP 服务器
server 0.cn.pool.ntp.org
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org
server 3.cn.pool.ntp.org
启动 NTP
systemctl enable ntpd
systemctl start ntpd
手工同步网络时间
ntpdate -u 0.cn.pool.ntp.org
同步系统时钟
hwclock --systohc
配置 ssh 免密登录(所有节点)
被登录方 操作
生成 rsa 公私钥对
ssh-keygen -t rsa -P ""
拷贝公钥到本地 authorized_keys
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
修改 authorized_keys 权限
chmod 600 .ssh/authorized_keys
验证本地免密登录
ssh localhost
登录方 操作
获取 slave2 的公钥
ssh {user}@{host} cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
禁用透明大页面压缩(所有节点)
打开编辑 /etc/rc.local 文件:
vi /etc/rc.local
添加以下内容:
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
优化交换分区(所有节点)
RHEL6
编辑 /etc/sysctl.conf 并添加 vm.swappiness=0
vi /etc/sysctl.conf
RHEL7
tuned 服务会动态调整系统参数,查找 tuned 中配置,直接将配置修改为 vm.swappiness=0
cd /usr/lib/tuned/
grep "vm.swappiness" * -R
安装jdk(所有节点)
yum -y install oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
/etc/profile 末尾添加以下内容
#set jdk environment
export JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
安装 mysql(master)
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
rpm -ivh mysql-community-release-el7-5.noarch.rpm
yum update
yum install mysql-server
systemctl start mysqld
配置 mysql(master)
将旧的 InnoDB 日志文件 /var/lib/mysql/ib_logfile0 和 /var/lib/mysql/ib_logfile1 从 /var/lib/mysql/ 中移除
rm -rf /var/lib/mysql/ib_logfile0 /var/lib/mysql/ib_logfile1
更新 /etc/my.cnf 配置文件,要满足以下要求:
To prevent deadlocks, set the isolation level to READ-COMMITTED.
Configure the InnoDB engine. Cloudera Manager will not start if its tables are configured with the MyISAM engine. (Typically, tables revert to MyISAM if the InnoDB engine is misconfigured.) To check which engine your tables are using, run the following command from the MySQL shell:
mysql> show table status;
The default settings in the MySQL installations in most distributions use conservative buffer sizes and memory usage. Cloudera Management Service roles need high write throughput because they might insert many records in the database. Cloudera recommends that you set the innodb_flush_method property to O_DIRECT.
Set the max_connections property according to the size of your cluster:
Binary logging is not a requirement for Cloudera Manager installations. Binary logging provides benefits such as MySQL replication or point-in-time incremental recovery after database restore. Examples of this configuration follow. For more information, see The Binary Log.
推荐配置如下:
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
启动 mysql(master)
启动服务并设置开机启动:
systemctl enable mysqld
systemctl start mysqld
设置 root 密码和其他安全相关设置:
/usr/bin/mysql_secure_installation
[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] Y
New password:
Re-enter new password:
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!
安装 mysql JDBC 驱动(master)
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar zxvf mysql-connector-java-5.1.46.tar.gz
mkdir -p /usr/share/java/
cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
创建 mysql 数据库(master)
root 用户登录
mysql -u root -p
为下表中每个服务创建 utf8 字符集的数据库:
CREATE DATABASE <database> DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON <database>.* TO '<user>'@'%' IDENTIFIED BY '<password>';
Service | database | user |
---|---|---|
Cloudera Manager Server | scm | scm |
Activity Monitor | amon | amon |
Reports Manager | rman | rman |
Hue | hue | hue |
Hive Metastore Server | metastore | hive |
Sentry Server | sentry | sentry |
Cloudera Navigator Audit Server | nav | nav |
Cloudera Navigator Metadata Server | navms | navms |
Oozie | oozie | oozie |
代码如下:
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';
flush privileges;
安装 cloudera manager 包
master 节点:
yum -y install cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm
yum -y install cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm
yum -y install cloudera-manager-server-6.1.0-769885.el7.x86_64.rpm
slave 节点:
yum -y install cloudera-manager-daemons-6.1.0-769885.el7.x86_64.rpm
yum -y install cloudera-manager-agent-6.1.0-769885.el7.x86_64.rpm
修改 agent 的服务器地址(所有节点)
打开并编辑 /etc/cloudera-scm-agent/config.ini 文件:
/etc/cloudera-scm-agent/config.ini
将 server_host=localhost 改为 master 节点主机名 cdh-2-master
启动 agent 服务(所有节点)
systemctl start cloudera-scm-agent
准备 Cloudera Manager 数据库(master)
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm scm
若已存在,则执行:
rm /etc/cloudera-scm-server/db.mgmt.properties
准备 parcel 包(master)
将已下载的 CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel 和 manifest.json 拷贝至 /opt/cloudera/parcel-repo/ 目录,并且生成 parcel 包的 sha 值:
sha1sum /opt/cloudera/parcel-repo/CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel
将 sha 值填入到 CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha 文件中
vi /opt/cloudera/parcel-repo/CDH-6.1.0-1.cdh6.1.0.p0.770702-el7.parcel.sha
启动 Cloudera Manager Server(master)
启动:
systemctl start cloudera-scm-server
查看启动日志:
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
看到下面的信息,说明启动完成
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
启用 Auto-TLS 以自动创建证书(master)
JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera /opt/cloudera/cm-agent/bin/certmanager --location /opt/cloudera/CMCA setup --configure-services
登录 web 页面(忽略许可授权,选择免费版)
账号/密码:admin / admin
添加集群
由于环境不满足(cpu、内存),在添加集群时网络经常断开,无法继续配置,内存也满了,有几个注意的要点:
- 配置存储库时,不要选择自定义,使用默认的 https://archive.cloudera.com/cm6/
- 安装jdk时,不要勾选,直接下一步,因为已经提前安装好了
- Install Agents时,由于已经提前安装了agent,所以这一步等进度条走完
挺遗憾的,没有完全部署完,只能等以后有环境了在补充。
如果觉得我的文章对您有用,请随意打赏。你的支持将鼓励我继续创作!