)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"6ca72b5f19f10ce1fbaedf37393fbf7d561a1d1c","unresolved":true,"context_lines":[{"line_number":10,"context_line":"compute and compute is stalled (e.g. the instance lock is held by"},{"line_number":11,"context_line":"another operation), the conductor\u0027s call_monitor_timeout fires after"},{"line_number":12,"context_line":"rpc_response_timeout seconds because no heartbeat was received. The"},{"line_number":13,"context_line":"conductor raises MessagingTimeout and moves on."},{"line_number":14,"context_line":""},{"line_number":15,"context_line":"However, when compute eventually dequeues and processes the stale"},{"line_number":16,"context_line":"message, it still creates a BDM in the database. Since the conductor"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"3cb2a22a_9b18de87","line":13,"updated":"2026-09-17 11:29:10.000000000","message":"simply saying the RPC call times out as the compute is holding a lock. \n\nBtw what operation holds the lock on this instance? Are we talking about the second parallel attach here? So the first holding the instance lock, we allow the second to enter as we did not set a task_state during the first and then this second is being held up at the compute side due to the first?\n\nIs it so?","commit_id":"491c4d1e86ec9dbbe097503276006d8b3c5a05b4"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"d3c0d7bee57726ed11a2327371fa6bd5842de4f0","unresolved":true,"context_lines":[{"line_number":10,"context_line":"compute and compute is stalled (e.g. the instance lock is held by"},{"line_number":11,"context_line":"another operation), the conductor\u0027s call_monitor_timeout fires after"},{"line_number":12,"context_line":"rpc_response_timeout seconds because no heartbeat was received. The"},{"line_number":13,"context_line":"conductor raises MessagingTimeout and moves on."},{"line_number":14,"context_line":""},{"line_number":15,"context_line":"However, when compute eventually dequeues and processes the stale"},{"line_number":16,"context_line":"message, it still creates a BDM in the database. Since the conductor"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"cd7af436_d33ec846","line":13,"in_reply_to":"3cb2a22a_9b18de87","updated":"2026-09-17 15:46:32.000000000","message":"You\u0027re right, the commit message is misleading about the lock. The actual scenario is simpler: all compute RPC worker threads are busy handling other operations, so the reserve_block_device_name message sits in the RabbitMQ queue undelivered. No thread picks it up, no heartbeats, conductor times out after rpc_response_timeout.\n\nThe instance lock is irrelevant here. I\u0027ll fix the commit message in the next PS.","commit_id":"491c4d1e86ec9dbbe097503276006d8b3c5a05b4"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"6ca72b5f19f10ce1fbaedf37393fbf7d561a1d1c","unresolved":true,"context_lines":[{"line_number":12,"context_line":"rpc_response_timeout seconds because no heartbeat was received. The"},{"line_number":13,"context_line":"conductor raises MessagingTimeout and moves on."},{"line_number":14,"context_line":""},{"line_number":15,"context_line":"However, when compute eventually dequeues and processes the stale"},{"line_number":16,"context_line":"message, it still creates a BDM in the database. Since the conductor"},{"line_number":17,"context_line":"already gave up, attach_volume will never be called for this BDM,"},{"line_number":18,"context_line":"leaving an orphan record that blocks future attachment attempts for"},{"line_number":19,"context_line":"the same volume on the same instance (the duplicate check in"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"8c8d79f8_d0ddce6a","line":16,"range":{"start_line":15,"start_character":0,"end_line":16,"end_character":48},"updated":"2026-09-17 11:29:10.000000000","message":"could it be that the first action a (see my previous comment) created the BDM and the second action reuses it, or this is always be a new BDM created at the second action?","commit_id":"491c4d1e86ec9dbbe097503276006d8b3c5a05b4"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"d3c0d7bee57726ed11a2327371fa6bd5842de4f0","unresolved":true,"context_lines":[{"line_number":12,"context_line":"rpc_response_timeout seconds because no heartbeat was received. The"},{"line_number":13,"context_line":"conductor raises MessagingTimeout and moves on."},{"line_number":14,"context_line":""},{"line_number":15,"context_line":"However, when compute eventually dequeues and processes the stale"},{"line_number":16,"context_line":"message, it still creates a BDM in the database. Since the conductor"},{"line_number":17,"context_line":"already gave up, attach_volume will never be called for this BDM,"},{"line_number":18,"context_line":"leaving an orphan record that blocks future attachment attempts for"},{"line_number":19,"context_line":"the same volume on the same instance (the duplicate check in"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"4e636699_0b704b18","line":16,"range":{"start_line":15,"start_character":0,"end_line":16,"end_character":48},"in_reply_to":"8c8d79f8_d0ddce6a","updated":"2026-09-17 15:46:32.000000000","message":"Always a new BDM. do_reserve() runs under @utils.synchronized(instance.uuid) and checks for duplicates — if a BDM for the same volume_id already exists, it raises InvalidVolume(\"volume already attached\"). So the second request can\u0027t reuse the first\u0027s BDM; it would fail.\n\nBut this patch isn\u0027t about two concurrent requests, it\u0027s about a single request where the conductor gives up (timeout) and compute later creates the BDM from the stale message.","commit_id":"491c4d1e86ec9dbbe097503276006d8b3c5a05b4"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"6ca72b5f19f10ce1fbaedf37393fbf7d561a1d1c","unresolved":true,"context_lines":[{"line_number":23,"context_line":"BDM. The context.timestamp reflects the original API request time,"},{"line_number":24,"context_line":"and the conductor sends the RPC almost immediately after, so elapsed"},{"line_number":25,"context_line":"time is a reliable proxy for how long the message has been waiting."},{"line_number":26,"context_line":"If the request is older than rpc_response_timeout, the conductor has"},{"line_number":27,"context_line":"almost certainly timed out (that value is used as the"},{"line_number":28,"context_line":"call_monitor_timeout for this specific RPC call) and any BDM created"},{"line_number":29,"context_line":"would be orphaned."}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"9734c805_d5fc24eb","line":26,"range":{"start_line":26,"start_character":29,"end_line":26,"end_character":49},"updated":"2026-09-17 11:29:10.000000000","message":"Can we assume that this is set symmetrically across the different services involved? I.e. both the sender and the receiver agrees when the message is stale?","commit_id":"491c4d1e86ec9dbbe097503276006d8b3c5a05b4"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"d3c0d7bee57726ed11a2327371fa6bd5842de4f0","unresolved":true,"context_lines":[{"line_number":23,"context_line":"BDM. The context.timestamp reflects the original API request time,"},{"line_number":24,"context_line":"and the conductor sends the RPC almost immediately after, so elapsed"},{"line_number":25,"context_line":"time is a reliable proxy for how long the message has been waiting."},{"line_number":26,"context_line":"If the request is older than rpc_response_timeout, the conductor has"},{"line_number":27,"context_line":"almost certainly timed out (that value is used as the"},{"line_number":28,"context_line":"call_monitor_timeout for this specific RPC call) and any BDM created"},{"line_number":29,"context_line":"would be orphaned."}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"29e68e54_a4c2a83e","line":26,"range":{"start_line":26,"start_character":29,"end_line":26,"end_character":49},"in_reply_to":"9734c805_d5fc24eb","updated":"2026-09-17 15:46:32.000000000","message":"Yes, both sides read from the same oslo.messaging config. The sender sets call_monitor_timeout\u003dCONF.rpc_response_timeout and the receiver reads the same CONF.rpc_response_timeout. It\u0027s a config-level assumption (same deployment), not a protocol guarantee, but that\u0027s how all existing call_monitor_timeout usages work in Nova.","commit_id":"491c4d1e86ec9dbbe097503276006d8b3c5a05b4"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"6ca72b5f19f10ce1fbaedf37393fbf7d561a1d1c","unresolved":true,"context_lines":[{"line_number":32,"context_line":"once compute starts processing and sends heartbeats, the conductor"},{"line_number":33,"context_line":"switches to the long_rpc_timeout deadline, but the LP #2141355"},{"line_number":34,"context_line":"scenario is specifically about compute not processing at all within"},{"line_number":35,"context_line":"the heartbeat window."},{"line_number":36,"context_line":""},{"line_number":37,"context_line":"Orphan BDMs that slip through the narrow remaining window"},{"line_number":38,"context_line":"(rpc_response_timeout \u003c elapsed \u003c long_rpc_timeout, where compute"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"9e87dc27_46c351a1","line":35,"updated":"2026-09-17 11:29:10.000000000","message":"wait, what triggers the switch from normal to long? If it is the consumption of the message from the message bus then there is a contradiction. Above it is stated that the request is held up by the instance lock, to reach that point the compute needed to consume the message from the bus, started processing it, reched the lock and stopped the thread handling the message.","commit_id":"491c4d1e86ec9dbbe097503276006d8b3c5a05b4"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"d3c0d7bee57726ed11a2327371fa6bd5842de4f0","unresolved":true,"context_lines":[{"line_number":32,"context_line":"once compute starts processing and sends heartbeats, the conductor"},{"line_number":33,"context_line":"switches to the long_rpc_timeout deadline, but the LP #2141355"},{"line_number":34,"context_line":"scenario is specifically about compute not processing at all within"},{"line_number":35,"context_line":"the heartbeat window."},{"line_number":36,"context_line":""},{"line_number":37,"context_line":"Orphan BDMs that slip through the narrow remaining window"},{"line_number":38,"context_line":"(rpc_response_timeout \u003c elapsed \u003c long_rpc_timeout, where compute"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"77f229d4_83c8bf4a","line":35,"in_reply_to":"9e87dc27_46c351a1","updated":"2026-09-17 15:46:32.000000000","message":"Good catch, you\u0027re right, there is a contradiction. The switch from call_monitor_timeout to long_rpc_timeout happens when the oslo.messaging dispatcher starts the watchdog thread, which is when a worker thread picks up the message and calls dispatch(). From that point, heartbeats flow every rpc_response_timeout/2 seconds.\n\nSo if compute is stuck on the instance lock, it means dispatch() was already called, heartbeats are flowing, and the conductor is using long_rpc_timeout (1800s). The lock can\u0027t cause a rpc_response_timeout timeout.\n\nThe only scenario that causes the rpc_response_timeout timeout is the message stuck in the queue with no worker thread available. I\u0027ll rewrite the commit message to fix this.","commit_id":"491c4d1e86ec9dbbe097503276006d8b3c5a05b4"}],"/PATCHSET_LEVEL":[{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"6ca72b5f19f10ce1fbaedf37393fbf7d561a1d1c","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"201f1e42_eb7ab3e4","updated":"2026-09-17 11:29:10.000000000","message":"sorry many more questions","commit_id":"491c4d1e86ec9dbbe097503276006d8b3c5a05b4"}],"nova/compute/manager.py":[{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"6ca72b5f19f10ce1fbaedf37393fbf7d561a1d1c","unresolved":true,"context_lines":[{"line_number":7843,"context_line":""},{"line_number":7844,"context_line":"        @utils.synchronized(instance.uuid)"},{"line_number":7845,"context_line":"        def do_reserve():"},{"line_number":7846,"context_line":"            bdms \u003d ("},{"line_number":7847,"context_line":"                objects.BlockDeviceMappingList.get_by_instance_uuid("},{"line_number":7848,"context_line":"                    context, instance.uuid))"},{"line_number":7849,"context_line":""}],"source_content_type":"text/x-python","patch_set":1,"id":"d372568a_13f4889b","line":7846,"updated":"2026-09-17 11:29:10.000000000","message":"could it be that the request was not stale at the first check above, then waited on the instance lock top of this helper and when enters here it is already stale?\n\nThe commit message goes back an forth about it is a lock that casing the staleness or it is the message not picked up from the bus causing the staleness.","commit_id":"491c4d1e86ec9dbbe097503276006d8b3c5a05b4"},{"author":{"_account_id":37309,"name":"Bryan Fraschetti","display_name":"Bryan Fraschetti","email":"bryan.fraschetti@canonical.com","username":"bryanfraschetti"},"change_message_id":"669a816a6527bf89ccf4f0f389ce92f12e8bbda8","unresolved":true,"context_lines":[{"line_number":7843,"context_line":""},{"line_number":7844,"context_line":"        @utils.synchronized(instance.uuid)"},{"line_number":7845,"context_line":"        def do_reserve():"},{"line_number":7846,"context_line":"            bdms \u003d ("},{"line_number":7847,"context_line":"                objects.BlockDeviceMappingList.get_by_instance_uuid("},{"line_number":7848,"context_line":"                    context, instance.uuid))"},{"line_number":7849,"context_line":""}],"source_content_type":"text/x-python","patch_set":1,"id":"09d3953c_0f6396db","line":7846,"in_reply_to":"8145e16d_753afaa5","updated":"2026-09-18 14:11:59.000000000","message":"Hello, thanks again for looking into this! I performed some testing and found that while this approach should work, it strangely didn\u0027t. It did work with some slight modification, though. I added some more log statements and saw that at the start of the reserve_block_device_name function the elapsed time was smaller than the configured timeout and so the bdm creation was not rejected.\nI moved the exact same logic down to line 7836 - just before the new_bdm.create() call in the nested do_reserve function and it resolved the bug. I haven\u0027t done any profiling to understand why there\u0027s a large time disparity between the two points in the code. However, given that the problem is that the bdm is being created after the timeout, it does make sense to check the elapsed time to see if the request has expired at the instant just before we actually create the bdm and commit the record to the db. At that moment we can decide whether we should proceed with the creation or not.","commit_id":"491c4d1e86ec9dbbe097503276006d8b3c5a05b4"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"d3c0d7bee57726ed11a2327371fa6bd5842de4f0","unresolved":true,"context_lines":[{"line_number":7843,"context_line":""},{"line_number":7844,"context_line":"        @utils.synchronized(instance.uuid)"},{"line_number":7845,"context_line":"        def do_reserve():"},{"line_number":7846,"context_line":"            bdms \u003d ("},{"line_number":7847,"context_line":"                objects.BlockDeviceMappingList.get_by_instance_uuid("},{"line_number":7848,"context_line":"                    context, instance.uuid))"},{"line_number":7849,"context_line":""}],"source_content_type":"text/x-python","patch_set":1,"id":"8145e16d_753afaa5","line":7846,"in_reply_to":"d372568a_13f4889b","updated":"2026-09-17 15:46:32.000000000","message":"No, once our code runs, it means dispatch() was called, which means the watchdog is already sending heartbeats. The conductor is using long_rpc_timeout at that point. Even if the request passes our check and then waits on the instance lock, the conductor is still waiting (heartbeats are flowing), so the BDM won\u0027t be orphaned.\n\nThe timestamp check catches the case where the message sat in the queue for longer than rpc_response_timeout before any worker thread picked it up.","commit_id":"491c4d1e86ec9dbbe097503276006d8b3c5a05b4"}]}