如何使用 ceph-deploy 部署一个指定版本号的 ceph 集群

前言

在 上文 中我们使用 ceph-deploy 搭建 ceph 集群。而当前版本 ceph-deploy 不支持用户指定小版本号，使用--release只支持安装最新稳定版。当然我们可以手动搭建，但是操作下来耗时且容易出错，毕竟人生苦短（我太懒了）。

版本信息

1. 系统版本

   [root@admin-node my-cluster]# cat /etc/system-release
   CentOS Linux release 7.6.1810 (Core) 
   [root@admin-node my-cluster]# uname -a
   Linux admin-node 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

2. ceph 版本

   [root@admin-node my-cluster]# ceph -v
   ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable)

   在本地配置离线存储库

安装网络服务

1. 安装 nginx

   yum install epel-release -y
   yum install nginx -y

2. 启动 nginx 并设置开机自启

   systemctl start nginx
   systemctl enable nginx
   systemctl status nginx

3. 添加防火墙规则

   firewall-cmd --zone=public --permanent --add-service=http
   firewall-cmd --zone=public --permanent --add-service=https
   firewall-cmd --reload

   创建本地镜像源

4. 安装必需工具

   yum install createrepo yum-utils wget -y

5. 创建镜像源文件夹

   mkdir -p /var/www/html/repos/{SRPMS,x86_64,noarch}

6. 下载相应软件包
   可以手动在以下 3 个链接中下载各自架构下的相应软件包

   https://download.ceph.com/rpm-nautilus/el7/SRPMS/
   https://download.ceph.com/rpm-nautilus/el7/noarch/
   https://download.ceph.com/rpm-nautilus/el7/x86_64/

   另外，我们也可以从阿里云的镜像源链接http://mirrors.aliyun.com/ceph/下载（国内用户你懂的）

   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-base-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-common-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-debuginfo-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-fuse-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-mds-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-mgr-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-mon-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-osd-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-radosgw-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-resource-agents-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-selinux-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ceph-test-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/cephfs-java-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/libcephfs-devel-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/libcephfs2-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/libcephfs_jni-devel-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/libcephfs_jni1-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/librados-devel-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/librados2-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/libradosstriper-devel-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/libradosstriper1-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/librbd-devel-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/librbd1-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/librgw-devel-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/librgw2-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python-ceph-compat-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python-cephfs-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python-rados-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python-rbd-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python-rgw-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python34-ceph-argparse-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python34-cephfs-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python34-rados-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python34-rbd-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python34-rgw-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/rados-objclass-devel-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/rbd-fuse-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/rbd-mirror-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/rbd-nbd-14.2.5-0.el7.x86_64.rpm
   wget http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/python-ceph-argparse-14.2.5-0.el7.x86_64.rpm
   wget https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch/ceph-deploy-2.0.0-0.noarch.rpm

7. 执行镜像源创建

   createrepo -v /var/www/html/repos/SRPMS/
   createrepo -v /var/www/html/repos/x86_64/
   createrepo -v /var/www/html/repos/noarch/

8. 修改 Nginx 配置

   vi /etc/nginx/conf.d/repos.conf

   server {
           listen   80;
           server_name  172.18.1.101;	# 修改此处为本机地址
           root   /var/www/html/repos;
           location / {
                   index  index.php index.html index.htm;
                   autoindex on;	#enable listing of directory index
           }
   }

9. 重新启动 nginx，然后再次访问本地源测试

   systemctl restart nginx

   

安装 Ceph

1. 声明本地仓库

   vi/etc/yum.repos.d/ceph.repo

   [ceph]
   name=Ceph packages for $basearch
   baseurl=http://172.18.1.101/x86_64/		# 注意此处ip为镜像源的ip
   enabled=1
   priority=2
   gpgcheck=0
   gpgkey=https://download.ceph.com/keys/release.asc
   
   [ceph-noarch]
   name=Ceph noarch packages
   baseurl=http://172.18.1.101/noarch
   enabled=1
   priority=2
   gpgcheck=0
   gpgkey=https://download.ceph.com/keys/release.asc
   
   [ceph-source]
   name=Ceph source packages
   baseurl=http://172.18.1.101/SRPMS
   enabled=0
   priority=2
   gpgcheck=0
   gpgkey=https://download.ceph.com/keys/release.asc
   EOF

2. 然后像往常一样进行 ceph 部署

   yum -y install ceph-deploy

3. 转到使用 ceph-deploy 安装的步骤，我们将使用 –repo-url 替换命令

   ceph-deploy install --repo-url http://172.18.1.101/ ceph1 ceph2 ceph3

4. 安装完成后，再次检查ceph version版本

   [node2][DEBUG ] 
   [node2][DEBUG ] Complete!
   [node2][INFO  ] Running command: sudo ceph --version
   [node2][DEBUG ] ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable)
   [root@admin-node x86_64]# ceph -v
   ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable)

   本地镜像安装

5. 下载源文件到本地

   mkdir ceph-mirror
   cd ceph-mirror
   
   yum install --downloadonly --downloaddir=. ceph
   yum install --downloadonly --downloaddir=. yum-plugin-priorities

6. 对文件进行删选
   仅保留我们需要的版本的 rpm 文件，其余的删除（否则同步会很耗时）

7. 使用--local-mirror指定使用本地镜像安装

   ceph-deploy install --local-mirror {/opt/ceph-mirror/rpm-nautilus/el7/} node1 nodex

   后面一次跟的指令为本地 mirror 所在路径、节点 hostname（多个或一个）

问题定位

同步时同步文件报内存不足

[cdnode1][INFO  ] syncing file: aarch64/ceph-debuginfo-14.2.2-0.el7.aarch64.rpm
[ceph_deploy][ERROR ] Traceback (most recent call last):
[ceph_deploy][ERROR ]   File "/usr/lib/python2.7/site-packages/ceph_deploy/util/decorators.py", line 69, in newfunc

# 太长省略
[ceph_deploy][ERROR ]     return _Serializer().save(obj)
[ceph_deploy][ERROR ]   File "/usr/lib/python2.7/site-packages/ceph_deploy/lib/vendor/remoto/lib/vendor/execnet/gateway_base.py", line 1312, in save
[ceph_deploy][ERROR ]     return type(streamlist[0])().join(streamlist)
[ceph_deploy][ERROR ] MemoryError
[ceph_deploy][ERROR ] 
Error in sys.exitfunc:

分析了一下，这个文件大概 2G 大小，如果直接 rsync 的话会导致内存爆炸，所以我的处理办法是删除ceph-debuginfo这个文件，后续有问题再装。

总结

ceph-deploy 可以通过三种指定方式部署 ceph,分别为ceph-deploy install --release（用于指定大版本号如nautilus）、ceph-deploy install --testing(用于最新开发版本)和ceph-deploy install --dev（用于指定分支或者 tag）,起初我以为--dev可以满足我指定版本号的需求，后来一顿操作发现报错:

[ceph_deploy.hosts.centos.install][DEBUG ] fetching repo information from: https://shaman.ceph.com/api/repos/ceph/14.5.2/latest/centos/7/repo/?arch=x86_64
[ceph_deploy.util.net][ERROR ] repository might not be available yet
[ceph_deploy][ERROR ] RuntimeError: HTTP Error 504: Gateway Timeout, failed to fetch https://shaman.ceph.com/api/repos/ceph/14.5.2/latest/centos/7/repo/?arch=x86_64

访问https://shaman.ceph.com/api/repos/ceph发现可用库里根本没有对应版本。通过阅读install — ceph-deploy 2.0.2 documentation我们知道可以指定 --repo-url搭配--gpg-url使用指定源链接的方式指定本地源安装。此外，使用--local-mirrors指定本地源的方式也可以参考。参见install — ceph-deploy 2.0.2 documentation。

本文通过搭建本地镜像源然后指定的方式间接完成了 ceph 集群的安装。

参考链接

• install — ceph-deploy 2.0.2 documentation
• ceph-deploy-specific-version.md at thaonguyenvan/notes-storage