kubernetes ceph is locked by other nodes 错误解决方案

1.问题描述

pod的状态一直处于ContainerCreating状态

[root@master ~]# kubectl get pods -n wangweidd | grep -v Running
NAME         READY     STATUS              RESTARTS   AGE
enncloud-1   0/1       ContainerCreating   0          14d

[root@master ~]# kubectl describe pod enncloud-1 -n wangweidd
...
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----            -------------   --------    ------      -------
  14d       1m      10013   kubelet, test           Warning     FailedMount MountVolume.SetUp failed for volume "wangweidd.datadir-enncloud-2" : rbd: image wangweidd.CID-516874818ed4.datadir-enncloud-2 is locked by other nodes
  14d       54s     8993    kubelet, test           Warning     FailedMount Unable to mount volumes for pod "enncloud-1_wangweidd(69b763cd-f9a3-11e7-9d36-5254b24cbf5e)": timeout expired waiting for volumes to attach/mount for pod "wangweidd"/"enncloud-1". list of unattached/unmounted volumes=[datadir]
  14d       54s     8993    kubelet, test           Warning     FailedSync  Error syncing pod

有错误的信息

rbd: image wangweidd.CID-516874818ed4.datadir-enncloud-2 is locked by other nodes

2.解决问题

1. 登录任何一个ceph 节点上 
  执行：
  rbd -p test-pool status  enngastest.CID-516874818ed4.datadir-mi-es-pro-1   
  Watchers:
           watcher=10.39.1.95:0/3703494529 client.104824 cookie=2
 查看的信息如下: 
 挂载在10.39.1.95 机器上 

2. rbd  showmapped  查询这台机器上挂载的rbd 对应的设备文件
    rbd showmapped 
    2017-07-13 14:10:15.704627 7f824e7657c0 -1 asok(0x3791960) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to /var/run/ceph/rbd-clients/ceph-client.admin.8255.58268576.asok: (2) No such file or directory
   id pool      image                                           snap device    
     0  tenx-pool lixueyang.CID-516874818ed4.datadir-iauc-crm-0   -    /dev/rbd0 
     1  tenx-pool enngastest.CID-516874818ed4.datadir-mi-es-pro-1 -   /dev/rbd1 
     2  tenx-pool ceres-prod.CID-516874818ed4.ceres-etcd-0        -      /dev/rbd2 
查出
rbd lock list tenx-pool/kubernetes-dynamic-pvc-d1043bbb-d5e4-11e7-a259-5254b24cbf5e Locker ID Address
client.811885 kubelet_lock_magic_paas-slave-20-48 10.39.20.48:0/2805197474

解锁 rbd unmap /dev/rbd2

ceph rbd 扩容

rbd showmapped   
df -h   查看/dev/rbdx   的大小 
扩容  
rbd  resize --size    1024   foo   
rbd resize --size  1024  liuguanghuif.CID-516874818ed4.datadir-lk-test-1  -p tenx-pool 
df -Th   查看大小是否变化
如果文件系统还未增大
blockdev --getsize64 /dev/rbd4 
resize2fs /dev/rbd4 

查看大小ok！ 

注意:对于xfs 要在resize 之后执行   xfs_growfs /mnt