I am attempting to attach a volume to an image. My attempts consistently fail.
I am using the following command to create the attachment:
openstack server add volume test_instance testvol
where test_instance is my instance and testvol is the 20G volume that I created with Cinder.
I get no error messages when I attempt this attachment, but when I check my volume I do not see the attachment:
cinder list
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
| 3dce2112-eddc-4d39-b47d-8717b16dd5e9 | available | testvol | 20 | __DEFAULT__ | false | |
| a9aa8845-dd3a-4f47-8b5e-445c6170abbe | available | volume1 | 20 | iscsi | false | |
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
I see no attachment. The above is what I get whether I attempt to attach testvol or volume1.
Looking at the nova-compute log, I am getting the following messages:
2020-09-06 12:41:07.172 2734 INFO nova.compute.manager [req-f84fbcef-b5fb-4f43-87ed-25d213b371c2 ce97118ec1f348a09d196c6da826749f 65d7e58a440440c3894c0fb7bf958052 - default default] [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] Attaching volume 3dce2112-eddc-4d39-b47d-8717b16dd5e9 to /dev/vda
2020-09-06 12:41:11.114 2734 INFO os_brick.initiator.connectors.iscsi [req-f84fbcef-b5fb-4f43-87ed-25d213b371c2 ce97118ec1f348a09d196c6da826749f 65d7e58a440440c3894c0fb7bf958052 - default default] Trying to connect to iSCSI portal 192.168.1.105:3260
2020-09-06 12:41:11.141 2734 WARNING os_brick.initiator.connectors.iscsi [req-f84fbcef-b5fb-4f43-87ed-25d213b371c2 ce97118ec1f348a09d196c6da826749f 65d7e58a440440c3894c0fb7bf958052 - default default] Couldn't find iSCSI nodes because iscsiadm err: iscsiadm: No records found: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2020-09-06 12:41:11.150 2734 WARNING os_brick.initiator.connectors.iscsi [req-f84fbcef-b5fb-4f43-87ed-25d213b371c2 ce97118ec1f348a09d196c6da826749f 65d7e58a440440c3894c0fb7bf958052 - default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [req-f84fbcef-b5fb-4f43-87ed-25d213b371c2 ce97118ec1f348a09d196c6da826749f 65d7e58a440440c3894c0fb7bf958052 - default default] [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] Driver failed to attach volume 3dce2112-eddc-4d39-b47d-8717b16dd5e9 at /dev/vda: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: iscsiadm -m node -T iqn.2010-10.org.openstack:volume-3dce2112-eddc-4d39-b47d-8717b16dd5e9 -p 192.168.1.105:3260 --interface default --op new
Exit code: 6
Stdout: ''
Stderr: 'iscsiadm: Could not open /var/lib/iscsi/nodes/iqn.2010-10.org.openstack:volume-3dce2112-eddc-4d39-b47d-8717b16dd5e9/192.168.1.105,3260: Permission deniedd\niscsiadm: Error while adding record: encountered iSCSI database failure\n'
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] Traceback (most recent call last):
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 598, in _volume_attach
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] device_type=self['device_type'], encryption=encryption)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1827, in attach_volume
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] encryption=encryption)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1607, in _connect_volume
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] vol_driver.connect_volume(connection_info, instance)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/volume/iscsi.py", line 64, in connect_volume
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] device_info = self.connector.connect_volume(connection_info['data'])
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] return f(*args, **kwargs)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] return f(*args, **kwargs)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 519, in connect_volume
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] self._cleanup_connection(connection_properties, force=True)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] self.force_reraise()
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] six.reraise(self.type_, self.value, self.tb)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] raise value
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 513, in connect_volume
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] return self._connect_single_volume(connection_properties)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/os_brick/utils.py", line 61, in _wrapper
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] return r.call(f, *args, **kwargs)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] return attempt.get(self._wrap_exception)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] six.reraise(self.value[0], self.value[1], self.value[2])
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] raise value
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 569, in _connect_single_volume
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] self._connect_vol(self.device_scan_attempts, props, data)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 626, in _connect_vol
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] session, manual_scan = self._connect_to_iscsi_portal(props)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 1040, in _connect_to_iscsi_portal
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] '--op', 'new'))
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/os_brick/initiator/connectors/iscsi.py", line 993, in _run_iscsiadm
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] delay_on_retry=delay_on_retry)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/os_brick/executor.py", line 52, in _execute
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] result = self.__execute(*args, **kwargs)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/os_brick/privileged/rootwrap.py", line 169, in execute
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] return execute_root(*cmd, **kwargs)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 247, in _wrap
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] return self.channel.remote_call(name, args, kwargs)
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 204, in remote_call
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] raise exc_type(*result[2])
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] Command: iscsiadm -m node -T iqn.2010-10.org.openstack:volume-3dce2112-eddc-4d39-b47d-8717b16dd5e9 -p 192.168.1.105:3260 --interface default --op new
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] Exit code: 6
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] Stdout: ''
2020-09-06 12:41:11.150 2734 ERROR nova.virt.block_device [instance: efdf9e5a-6a8c-4ef1-8bfd-7d072c750b6b] Stderr: 'iscsiadm: Could not open /var/lib/iscsi/nodes/iqn.2010-10.org.openstack:volume-3dce2112-eddc-4d39-b47d-8717b16dd5e9/192.168.1.105,3260: Permission deniedd\niscsiadm: Error while adding record: encountered iSCSI database failure\n'
2020
Does anyone know what is causing this failure? How can I get a volume attached to an instance?