When VM resizing or cold migrating between hypervisors, it’s highly probable that you will come across an SSH error indicating “Permission denied” as below in /var/log/nova/nova-compute.log of host that is running the instance for cold migration.
ERROR oslo_messaging.rpc.server Traceback (most recent call last): ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 191, in decorated_function ERROR oslo_messaging.rpc.server "Error: %s", e, instance=instance) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 161, in decorated_function ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1372, in decorated_function ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 219, in decorated_function ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 207, in decorated_function ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4894, in resize_instance ERROR oslo_messaging.rpc.server self._revert_allocation(context, instance, migration) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4891, in resize_instance ERROR oslo_messaging.rpc.server instance_type, clean_shutdown, request_spec) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4950, in _resize_instance ERROR oslo_messaging.rpc.server request_spec) ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__ ERROR oslo_messaging.rpc.server self.gen.throw(type, value, traceback) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 9032, in _error_out_instance_on_exception ERROR oslo_messaging.rpc.server raise error.inner_exception ERROR oslo_messaging.rpc.server ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command. ERROR oslo_messaging.rpc.server Command: ssh -o BatchMode=yes 10.180.32.21 mkdir -p /var/lib/nova/instances/d98b74ca-01e5-4365-8cc3-939745da9380 ERROR oslo_messaging.rpc.server Exit code: 255 ERROR oslo_messaging.rpc.server Stdout: u'' ERROR oslo_messaging.rpc.server Stderr: u'Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r\n'
How VM resizing and cold migration works?
When you attempt to resize a VM, the VM needs to be migrated to other host. This requires an SSH connection to the destination host for the migration process. However, a standard OpenStack installation is typically not configured to allow this for security reasons.
By default OpenStack is not allowed to resize VM’s to the same host.
You can still follow to resize the VM to the same host without migration process. However, for resource optimization, high availability and adaptable resource scaling it is advised to configure VM resizing and cold migrations. Resizing on OpenStack(KVM based) is implemented by transferring the images between compute nodes over ssh. For KVM you need hostnames to resolve properly and passwordless the ssh access between your compute hosts. Direct access from one compute host to another is needed to copy the VM file from one host to another.
Resizing disks downward is not supported.
How to configure VM resizing and cold migration between hypervisors?
During OpenStack installation, the nova user is created without shell access. To facilitate VM resizing between nodes and enable cold migration, that includes node evacuation, it’s necessary to modify the nova user’s shell and establish passwordless access for nova across compute nodes.
Configure Controller Node
Step 1: Enable shell access for nova user.
# usermod --shell /bin/bash nova
Step 2: Add the controller and compute node entries to /etc/hosts file
192.168.10.2 controller.techglimpse.com controller 192.168.10.3 compute.techglimpse.com compute
Step 3: Switch to nova user. The $HOME for nova user is /var/lib/nova
# su - nova
Step 4: Create a .ssh folder uder /var/lib/nova
$ mkdir /var/lib/nova/.ssh
Step 5: Generate a key pair for nova user without any passphrase.
$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/var/lib/nova/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /var/lib/nova/.ssh/id_rsa. Your public key has been saved in /var/lib/nova/.ssh/id_rsa.pub. The key fingerprint is: SHA256:hWRepoCG0zxif6+slY++l/OoAwSAETa9TwiuJfgf75Y nova@controller.techglimpse.com The key's randomart image is: +---[RSA 2048]----+ |=*.+ .. o o | |o.B.* = = | |o..*o. + . | |o..oo.. . | |.+ .o. .S | |. . o. .. | | . =oo . | | ..Eo+. | | .=*=oo. | +----[SHA256]-----+
Step 6: Copy the public keys to authorized_keys file.
$ cat id_rsa.pub >> authorized_keys
$ cat authorized_keys from="192.168.10.*" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsWf0NJFCe1ZUWuqNtO4/Fe8NaB44UORP+sEdQ+bEWVoivhA8oRarndANcNv1H8ghpRRnt+hOwByLTLVHTVZUjfXhfdbQkryGrLdDHKhxGB4IodByfIHY8SJbKLfrEZKanvx6/DJ//vBFL9PFR+3e97HE6klZgXsTZwBGRvupAQ4lDy7yMLbuP7IrJh0HJglEpMpzmjaQvQXNSe8f43ksFIT70xlrfib0hf9mvSTdBCeqdWE0pL8t7zoeCJSD/78fIV38HOX07rm5fFl1wydCCgCcWGyciIKVGp61RO4+20Iz1QzMrqNslGtHoUAHlg+ufWIoNCZpum1s2mjhFv0/3 nova@controller.techglimpse.com
Add the “from=” clause at the beginning of authorized_keys file to restrict the IP addresses from which nova users can access.
Step 7: Generate known_hosts file to include all the compute hosts which the nova user may access
$ ssh-keyscan compute >> known_hosts # compute:22 SSH-2.0-OpenSSH_7.4 # compute:22 SSH-2.0-OpenSSH_7.4 # compute:22 SSH-2.0-OpenSSH_7.4
Step 8: Copy the newly created .ssh folder from controller node to all the compute nodes nova home directory (/var/lib/nova)
$ scp -r .ssh/ root@compute:/var/lib/nova/
Configure Compute Nodes
Step 1: Login to compute node and change the .ssh folder ownership to nova
# chown nova:nova .ssh/
Step 2: Update the shell access to nova user
# usermod --shell /bin/bash nova
Restart Nova Service on Controller and Compute
Restart the Nova service both on controller and compute nodes as below
On controller node
# systemctl restart openstack-nova-compute openstack-nova-api
On Compute node
# systemctl restart openstack-nova-compute
Test the VM resize or Cold Migration
Now you can perform the testing of a VM resize or cold migration using OpenStack command line or through Horizon Dashboard.
