Migrating VMs with attached RBDs

From the title, this is obviously a very common scenario that you may want to do. One thing that we rarely think about though is “backends” for the attached volumes when we create volumes.

When you create a volume, the volume is created on a cinder backend and kept attached to this backend until it’s deleted , or migrated to another backend. The backends are defined in cinder configuration and are provided by your host(s) running the cinder-volume service. To find your backends, run the following command

cinder get-pools

When you attach the volume to a VM,  the volume keeps its backend. It relies on this backend to do any operations to that volume. This includes migrating the VM from a host to another.

You may run into a scenario where you get this error when trying to migrate a VM with attached RBD

 CinderConnectionFailed: Connection to cinder host failed: Unable to establish connection to

But when you go and check, Cinder is working correctly. You are able to create new volumes and attach them to instances. But a particular VM is unable to migrate. You may find also you’r unable to snapshot the volume attached to the VM. The thing to check for here is the RBD backend of the volume

You can find this using

cinder show VOLUME_ID

this will show you alot of details on the volume including the following attribute

| os-vol-host-attr:host | HOSTNAME@ceph#RBD |

HOSTNAME will likely be “one” of your controllers. You will need to go and check that cinder-volume service is running correctly on that controller. If it’s down, you can’t operate that volume for anything (snapshots, attach/detach and migrate)

If you’ve lost your controller forever, or you were testing a new backend that no longer exists, then you might want to migrate the volume from the dead backend. This is detailed in the following manual

https://docs.openstack.org/cinder/pike/admin/blockstorage-volume-migration.html

Happy VM migrations !

cinder-manage: Did you know about it ?

A tool that’s less known-about for cinder is cinder-manage. You might have run into it during upgrades. The most common use case is

cinder-manage db sync

This is normally executed during upgrades to bring the database to the latest version, or to create the schema for a new installation. But there’s actually additional usages for it. Few of them are

cinder-manage service list

the output will look like that

Binary Host Zone Status State Updated At RPC Version Object Version Cluster 
cinder-scheduler controller-server nova enabled :-) 2017-10-15 19:45:37 4.5 4.5 
cinder-volume controller-server@ceph nova enabled :-) 2017-10-15 19:45:31 4.6 4.6

The output can be used to diagnose issues when cinder-scheduler reports that the volume backend is down although cinder-volume is up. The output of the above command is the only reliable source to show how cinder-scheduler, cinder-volume and cinder-backup status is.

If you have multiple backends for cinder, or use multiple cinder-scheduler/cinder-volume on multiple controller nodes. The output will look like this

Binary Host Zone Status State Updated At RPC Version Object Version Cluster 
cinder-scheduler controller-server1 nova enabled :-) 2017-10-15 19:45:37 4.5 4.5 
cinder-volume controller-server1@ceph nova enabled :-) 2017-10-15 19:45:31 4.6 4.6
cinder-volume controller-server1@ceph2 nova enabled :-) 2017-10-15 19:45:31 4.6 4.6
cinder-scheduler controller-server2 nova enabled :-) 2017-10-15 19:45:37 4.5 4.65
cinder-volume controller-server2@ceph nova enabled XX 2017-10-15 19:45:37 4.6 4.6

As you can see above, there are multiple backends for cinder-volume on controller-server1. One of them is ceph and the other is ceph2 and both are enabled and up. It’s easy to spot that cinder-volume on the controller-server2 is showing as down, so you should expect the ceph backend to not be available. If you check the cinder-volume service using systemctl status, the service itself might be running. If that is the case you need to look deeper to why the ceph backend for cinder-volume is down

If you decide to remove a certain cinder-volume/cinder-scheduler/cinder-backup service from your deployment, you can do that by stopping the service on the controller host, and then removing it using

cinder-manage service remove cinder-scheduler controller-server2
cinder-manage service remove cinder-volume controller-server2

If this small use case got you excited, check out the following uses as well

cinder-manage logs errors
cinder-manage logs syslog
cinder-manage volume delete --> Important in the case of stuck volumes
cinder-manage host list
cinder-manage config list --> you can use it to verify what the running configuration for cinder is

The manual for cinder-manage is at

https://docs.openstack.org/cinder/latest/man/cinder-manage.html

Have fun !