)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"ac66ffc892191bbcf796618e0e38b0d642bf62c2","unresolved":true,"context_lines":[{"line_number":28,"context_line":"instead."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Change-Id: I07e404c59d035ed25ae5903a9fdd5a5f509b9c1a"},{"line_number":31,"context_line":"Related-Bug: #2141355"},{"line_number":32,"context_line":"Assisted-By: Cursor claude-4.6-opus"},{"line_number":33,"context_line":"Signed-off-by: Sylvain Bauza \u003csbauza@redhat.com\u003e"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"5843fcb1_e15e0b29","line":31,"updated":"2026-09-17 09:13:29.000000000","message":"OK so we are not resolving this problem as this problem talks about the case when the RPC timeouts first, no BDM, conductor moves on, compute creates the BDM. This case is still open.\n\nWe resolving a different (but related) problem where the BDM exist when the conductor gets the timeout, but it is picked up by a concurrent request. Could you point at / explain how can be concurrent request picking up this BDM that was created due to the first request?\n\nAlso how can be a concurrent request enter the picture at all for this instance. Are we not setting some task_state on the instance when the first attach is received that would prevent the second attach to be accepted while the first attach is waiting for the RPC call to finish / time out?","commit_id":"0a4edb3eab27b6f920276af3242fc38ee912028f"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"af18eb11fb495779d70f7b71acf65c6bc0fec299","unresolved":false,"context_lines":[{"line_number":28,"context_line":"instead."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Change-Id: I07e404c59d035ed25ae5903a9fdd5a5f509b9c1a"},{"line_number":31,"context_line":"Related-Bug: #2141355"},{"line_number":32,"context_line":"Assisted-By: Cursor claude-4.6-opus"},{"line_number":33,"context_line":"Signed-off-by: Sylvain Bauza \u003csbauza@redhat.com\u003e"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"cbd2fa9b_e25161af","line":31,"in_reply_to":"5843fcb1_e15e0b29","updated":"2026-09-17 09:26:13.000000000","message":"aand I realized that this is basically the same question Dan asked above. Closing this thread and moving there.","commit_id":"0a4edb3eab27b6f920276af3242fc38ee912028f"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"8b58cc364909d72a12803ec86a2e92a3d9841b71","unresolved":true,"context_lines":[{"line_number":26,"context_line":"BDM. If request 2 has already reserved the volume in Cinder"},{"line_number":27,"context_line":"(attachment_id is set) and the attach_volume cast was sent, this"},{"line_number":28,"context_line":"leaves a three-way inconsistency: Cinder shows the volume in-use,"},{"line_number":29,"context_line":"libvirt has the block device, but Nova has no BDM."},{"line_number":30,"context_line":""},{"line_number":31,"context_line":"Before destroying the BDM, check whether attachment_id is set. If"},{"line_number":32,"context_line":"it is, the BDM has been reserved in Cinder by a different request"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"3fce8637_7b416a87","line":29,"updated":"2026-09-21 11:46:02.000000000","message":"Do I understand the sequence of events\n1. attach_volume nr1 reaches the conductor and sends the  reserve_block_device_name RPC call down to the compute. Conductor waits for the RPC response\n2. attach_volume nr2 reached the conductor and sends the reserve_block_device_name RPC call down to the compute. Conductor waits for the RPC response.\n3. compute handles nr2 reserve_block_device_name and sends the RPC response.\n4. conductor receives the nr2 reserve_block_device_name and casts attach_volume to the compute.\n5. nr1 reserve_block_device_name times out. nr1 lost somewhere in rabbit never received. The clean up logic deletes the BDM nr2 was created.","commit_id":"cabe711e58bcb9e314098361f61dfb0215436e6c"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"8b58cc364909d72a12803ec86a2e92a3d9841b71","unresolved":true,"context_lines":[{"line_number":30,"context_line":""},{"line_number":31,"context_line":"Before destroying the BDM, check whether attachment_id is set. If"},{"line_number":32,"context_line":"it is, the BDM has been reserved in Cinder by a different request"},{"line_number":33,"context_line":"and must not be deleted."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"This does not fully close the race: if request 2 has created the"},{"line_number":36,"context_line":"BDM but has not yet called Cinder (attachment_id is still None),"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"2616f551_41503035","line":33,"updated":"2026-09-21 11:46:02.000000000","message":"If I understand the scenario above then attachment_id means at least on parallel request reached the attach_volume RPC and such RPC is handled by the compute already creating the attachment. So yes the timeout for a previous RPC reserve_block_device_name should not clean that BDM up.","commit_id":"cabe711e58bcb9e314098361f61dfb0215436e6c"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"8b58cc364909d72a12803ec86a2e92a3d9841b71","unresolved":true,"context_lines":[{"line_number":44,"context_line":""},{"line_number":45,"context_line":"The full fix for preventing concurrent requests would require"},{"line_number":46,"context_line":"setting a task_state on the instance during attach, which has"},{"line_number":47,"context_line":"upgrade implications and is left for a follow-up."},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"Change-Id: I07e404c59d035ed25ae5903a9fdd5a5f509b9c1a"},{"line_number":50,"context_line":"Closes-Bug: #2167569"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"9d412a34_0f8cb93e","line":47,"updated":"2026-09-21 11:46:02.000000000","message":"I thought more about this and I think we have an opportunity here. As far as I see neither the reserve_block_device_name nor the attach_volume RPC changes or saves the instance ever. So start using a task_state can be built in a backward compatible way. (Please check my logic)\n\n1. conductor rejects any attach_volume call if the instance.task_state is not empty\n2. conductor rejects any attach_volume call if the BDM already exists\n3. conductor set instance.task_state to BLOCK_DEVICE_MAPPING and saves the instance\n3. conductor calls reserve_block_device_name and wait for the response\n4. compute creates the BDM\n5a. if reserve_block_device_name succeeds conductor resets the task_state to None and saves the instance.\n5b. if reserve_block_device_name times out the BDM can be deleted if exists and the instance.task_state needs to be reset to None. (and hopefully we record a failed instance action too)\n6a. conductor casts attach_volume to the compute\n\nThis prevents the two overlapping reserve_block_device_name RPCs and therefore any reordering issues between them. This change does not effect upgrade as it only touches conductors that are expected to be upgraded at once. (correct me if I\u0027m wrong here)\n\nAlso it used the BDM exists as a signal too. So even if the second attach_volume arrives to the conductor *after* the first reserve_block_device_name returned and the task_state is reset to None, the fact that the BDM exists is enough to prevent the second request to overlap with the first request doing things on the compute side in the attach_volume RPC.\n\nWhat do you think?","commit_id":"cabe711e58bcb9e314098361f61dfb0215436e6c"}],"/PATCHSET_LEVEL":[{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"8b58cc364909d72a12803ec86a2e92a3d9841b71","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"96b01c4a_16cb6bb7","updated":"2026-09-21 11:46:02.000000000","message":"ttac","commit_id":"cabe711e58bcb9e314098361f61dfb0215436e6c"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"f967a54fda470c6f31db96878b3dd58d87378e1b","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"bf601b8d_c8fb5f15","in_reply_to":"96b01c4a_16cb6bb7","updated":"2026-09-21 11:46:18.000000000","message":"I don\u0027t know what this is. ignore me here :)","commit_id":"cabe711e58bcb9e314098361f61dfb0215436e6c"}],"nova/conductor/manager.py":[{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"71cfccf072b482c8989179220c1ad81d1078cb4e","unresolved":true,"context_lines":[{"line_number":2283,"context_line":"            # we never received the answer. In this case it is safe to delete"},{"line_number":2284,"context_line":"            # the attachment as nobody will ever pick it up again."},{"line_number":2285,"context_line":"            # NOTE(sbauza): We need to verify the BDM wasn\u0027t already"},{"line_number":2286,"context_line":"            # progressed by a concurrent request (e.g. a K8s retry) that"},{"line_number":2287,"context_line":"            # successfully reserved the volume in Cinder. If attachment_id"},{"line_number":2288,"context_line":"            # is set, the BDM belongs to that other request and must not be"},{"line_number":2289,"context_line":"            # deleted — doing so would leave Cinder and libvirt out of sync"}],"source_content_type":"text/x-python","patch_set":1,"id":"15492e5b_535ecfd5","line":2286,"updated":"2026-09-15 14:11:20.000000000","message":"So the theory is that we make this call in request 1, which gets stalled in some way, then request 2 comes in and it goes through, to compute, returns successfully, and makes it through L2316 and reserves the volume, all before we get the timeout. Is that right?\n\nI guess I\u0027m not sure that this really closes the window as request 2 could be stalled waiting for a slow cinder below while we return and still go about deleting the BDM right?\n\nJust to be clear, both requests can\u0027t have both made any progress on the compute node since the first thing they do is lock on the instance uuid and then they confirm (under lock) that the instance has no matching BDM and then create it (again under lock). So the only way this can happen is if the first request doesn\u0027t progress at all and the second sails through, right?\n\nIt seems like this must be a very specific situation and that we\u0027re still not closing the race. Without a task_state or volume state lock, I don\u0027t think we\u0027re actually fixing anything here. What am I missing?","commit_id":"0a4edb3eab27b6f920276af3242fc38ee912028f"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"75aede873897f0b1474dfb84a6ee09b6c6364b1f","unresolved":true,"context_lines":[{"line_number":2283,"context_line":"            # we never received the answer. In this case it is safe to delete"},{"line_number":2284,"context_line":"            # the attachment as nobody will ever pick it up again."},{"line_number":2285,"context_line":"            # NOTE(sbauza): We need to verify the BDM wasn\u0027t already"},{"line_number":2286,"context_line":"            # progressed by a concurrent request (e.g. a K8s retry) that"},{"line_number":2287,"context_line":"            # successfully reserved the volume in Cinder. If attachment_id"},{"line_number":2288,"context_line":"            # is set, the BDM belongs to that other request and must not be"},{"line_number":2289,"context_line":"            # deleted — doing so would leave Cinder and libvirt out of sync"}],"source_content_type":"text/x-python","patch_set":1,"id":"b171b8d7_397d291a","line":2286,"in_reply_to":"15492e5b_535ecfd5","updated":"2026-09-15 15:00:52.000000000","message":"Right, the window isn\u0027t fully closed but the severity is different depending on whether attachment_id is set or not.\n\nWhen it is set (request 2 runs correctly up to the end): attach_volume cast was already sent, Cinder says \"in-use\", libvirt has the device. Deleting the BDM creates a full inconsistency where you can\u0027t detach (404) and can\u0027t re-attach (volume in-use). That\u0027s the stuck state from the Launchpad bug report.\n\nWhen it\u0027s not set (request 2 stalles as request 1): Cinder has at most a reserved attachment, no attach_volume cast was sent, no libvirt device. Request 2\u0027s volume_bdm.save() would hit BDMNotFound (soft-deleted row filtered by model_query) and fail. The dangling reservation is recoverable then from the Cinder side.\n\nSo this separates the severe case from the residual one. The residual window is also very narrow in practice (sub-second between _create_volume_bdm() returning and the Cinder call completing, vs 60-120s timeout as I saw it on some downstream issue too).\n\nI agree the task_state approach would fully close it but it\u0027s harder to backport. I see this as a first step, that\u0027s it but of course I\u0027d want the long-term effort would be accepted as a spec later.","commit_id":"0a4edb3eab27b6f920276af3242fc38ee912028f"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"6632b43e8c602a700754406cb5bfa45818a47eeb","unresolved":false,"context_lines":[{"line_number":2283,"context_line":"            # we never received the answer. In this case it is safe to delete"},{"line_number":2284,"context_line":"            # the attachment as nobody will ever pick it up again."},{"line_number":2285,"context_line":"            # NOTE(sbauza): We need to verify the BDM wasn\u0027t already"},{"line_number":2286,"context_line":"            # progressed by a concurrent request (e.g. a K8s retry) that"},{"line_number":2287,"context_line":"            # successfully reserved the volume in Cinder. If attachment_id"},{"line_number":2288,"context_line":"            # is set, the BDM belongs to that other request and must not be"},{"line_number":2289,"context_line":"            # deleted — doing so would leave Cinder and libvirt out of sync"}],"source_content_type":"text/x-python","patch_set":1,"id":"21a8c8a4_7223fc1f","line":2286,"in_reply_to":"1c67a6f6_87f908bf","updated":"2026-09-17 13:53:35.000000000","message":"Done","commit_id":"0a4edb3eab27b6f920276af3242fc38ee912028f"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"8f8988999352b74719a54f145b0d1257b4420440","unresolved":true,"context_lines":[{"line_number":2283,"context_line":"            # we never received the answer. In this case it is safe to delete"},{"line_number":2284,"context_line":"            # the attachment as nobody will ever pick it up again."},{"line_number":2285,"context_line":"            # NOTE(sbauza): We need to verify the BDM wasn\u0027t already"},{"line_number":2286,"context_line":"            # progressed by a concurrent request (e.g. a K8s retry) that"},{"line_number":2287,"context_line":"            # successfully reserved the volume in Cinder. If attachment_id"},{"line_number":2288,"context_line":"            # is set, the BDM belongs to that other request and must not be"},{"line_number":2289,"context_line":"            # deleted — doing so would leave Cinder and libvirt out of sync"}],"source_content_type":"text/x-python","patch_set":1,"id":"c93a3da8_740fcc31","line":2286,"in_reply_to":"b171b8d7_397d291a","updated":"2026-09-15 17:04:27.000000000","message":"Ah, okay, can we augment the comment here a little with those details? It seems like this is intending to close the window, but it doesn\u0027t. If it just causes a better failure scenario that\u0027s quite a bit more complex and will be hard to remember later.\n\nBut, your explanation makes sense. Indeed, BDM.save() has a specific callout for an update failing and turning that into BDMNotFound which lines up with what you said. I was thinking BDMs might be one of the ones we always use read_deleted\u003dTrue with because we probably update those during/after instance deletion, but I guess not.\n\nIt would also be nice to have a reproducer for the full two-request race/win scenarios but that\u0027s probably too much to ask I suppose.","commit_id":"0a4edb3eab27b6f920276af3242fc38ee912028f"},{"author":{"_account_id":37309,"name":"Bryan Fraschetti","display_name":"Bryan Fraschetti","email":"bryan.fraschetti@canonical.com","username":"bryanfraschetti"},"change_message_id":"ccf38da266cbdff36bf56ebe34ae31527cf59569","unresolved":true,"context_lines":[{"line_number":2283,"context_line":"            # we never received the answer. In this case it is safe to delete"},{"line_number":2284,"context_line":"            # the attachment as nobody will ever pick it up again."},{"line_number":2285,"context_line":"            # NOTE(sbauza): We need to verify the BDM wasn\u0027t already"},{"line_number":2286,"context_line":"            # progressed by a concurrent request (e.g. a K8s retry) that"},{"line_number":2287,"context_line":"            # successfully reserved the volume in Cinder. If attachment_id"},{"line_number":2288,"context_line":"            # is set, the BDM belongs to that other request and must not be"},{"line_number":2289,"context_line":"            # deleted — doing so would leave Cinder and libvirt out of sync"}],"source_content_type":"text/x-python","patch_set":1,"id":"f2e1c89a_90c0d57d","line":2286,"in_reply_to":"c93a3da8_740fcc31","updated":"2026-09-15 20:44:11.000000000","message":"Hello, thanks for taking a look at this!\nI have tested the patch and I think it resolves a different one than what I reported in the Launchpad bug. In #2141355 the issue is that nova sees the volume as in-use while cinder sees the volume as available. From your description, this patch seems to solve the opposite case (cinder sees in-use while nova sees available). I\u0027ve tested this patch in a Devstack environment and my reproducer still generates a dangling partially completed attachment in Nova.\nThis patch is probably still useful, but I\u0027m not sure it resolves the bug","commit_id":"0a4edb3eab27b6f920276af3242fc38ee912028f"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"af18eb11fb495779d70f7b71acf65c6bc0fec299","unresolved":true,"context_lines":[{"line_number":2283,"context_line":"            # we never received the answer. In this case it is safe to delete"},{"line_number":2284,"context_line":"            # the attachment as nobody will ever pick it up again."},{"line_number":2285,"context_line":"            # NOTE(sbauza): We need to verify the BDM wasn\u0027t already"},{"line_number":2286,"context_line":"            # progressed by a concurrent request (e.g. a K8s retry) that"},{"line_number":2287,"context_line":"            # successfully reserved the volume in Cinder. If attachment_id"},{"line_number":2288,"context_line":"            # is set, the BDM belongs to that other request and must not be"},{"line_number":2289,"context_line":"            # deleted — doing so would leave Cinder and libvirt out of sync"}],"source_content_type":"text/x-python","patch_set":1,"id":"1c67a6f6_87f908bf","line":2286,"in_reply_to":"f2e1c89a_90c0d57d","updated":"2026-09-17 09:26:13.000000000","message":"I\u0027m still puzzled that how can be a concurrent request picking up this BDM that was created due to the first request?\n\nAlso how can be a concurrent request enter the picture at all for this instance? Are we not setting some task_state on the instance when the first attach is received that would prevent the second attach to be accepted while the first attach is waiting for the RPC call to finish / time out?\n\n// later\nYeah I don\u0027t see any task_state being set at https://github.com/openstack/nova/blob/master/nova/compute/api.py#L5178\n\nI feel like task_state would be a better approach removing a lot of potential overlaps. But that probably an upgrade impact if the compute side expects no task_state on the instance when the RPC is received.\n\nMaybe we could modify the receiving side of the RPC to set a task state that might work to make this backward compatible. (We need to trace the task_state data flow through attach to ensure it)\n\nCan we have all this at least described in the commit message or in a separate bug report that describes the exact case we are solving here with all reasoning why we do a two step approach with one backportable band aid and one proper fix with an upgrade impact patch.","commit_id":"0a4edb3eab27b6f920276af3242fc38ee912028f"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"8f8988999352b74719a54f145b0d1257b4420440","unresolved":true,"context_lines":[{"line_number":2311,"context_line":"                            volume[\u0027id\u0027])"},{"line_number":2312,"context_line":"                except exception.VolumeBDMNotFound:"},{"line_number":2313,"context_line":"                    LOG.debug(\"BDM not found, ignoring removal. \""},{"line_number":2314,"context_line":"                              f\"Error attaching volume {volume[\u0027id\u0027]}\")"},{"line_number":2315,"context_line":"        try:"},{"line_number":2316,"context_line":"            compute_utils.check_attach_and_reserve_volume("},{"line_number":2317,"context_line":"                context, self.volume_api, volume, instance, volume_bdm,"}],"source_content_type":"text/x-python","patch_set":1,"id":"fc19110d_43befcc8","line":2314,"updated":"2026-09-15 17:04:27.000000000","message":"Coverage says this is not ever hit. That\u0027s pre-existing but feels relevant based on the above justification.","commit_id":"0a4edb3eab27b6f920276af3242fc38ee912028f"}]}