)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"19ee16e051761e88eb2cad6698442d217cbb9b0f","unresolved":false,"context_lines":[{"line_number":1,"context_line":"Parent:     3d733f65 (agent: separate driver loading from driver initialisation)"},{"line_number":2,"context_line":"Author:     Chandan Kumar (raukadah) \u003cchkumar@redhat.com\u003e"},{"line_number":3,"context_line":"AuthorDate: 2026-08-09 09:10:33 +0530"},{"line_number":4,"context_line":"Commit:     Chandan Kumar (raukadah) \u003cchkumar@redhat.com\u003e"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":19,"id":"3b311541_6561ffbc","line":1,"updated":"2026-08-24 15:49:38.000000000","message":"The message states: \u0027RPCService.stop() calls it unconditionally, removing the hasattr special case.\u0027 The parent commit\u0027s RPCService.stop() (3d733f6) contains no hasattr check; it only wrapped rpcserver.stop()/wait() in try/except. This change purely adds the manager.shutdown() call, so the \u0027removing the hasattr special case\u0027 clause describes a change that did not happen.\n\n**Severity**: SUGGESTION | **Confidence**: 0.9\n\n**Impact**: The permanent change history asserts a refactor that did not occur, which can mislead future readers auditing the shutdown path or backporting.\n\n**Recommendation**:\nDrop the \u0027removing the hasattr special case\u0027 clause (or reword to \u0027calls manager.shutdown() unconditionally\u0027) before merge.","commit_id":"517162ec859ab340100c216a8dd215f260b950f3"}],"/PATCHSET_LEVEL":[{"author":{"_account_id":12393,"name":"chandan kumar","display_name":"Chandan Kumar","email":"chkumar@redhat.com","username":"chkumar246"},"change_message_id":"f3b9270167e59d3ad9a742aac86b9be4fd1085b7","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":16,"id":"345e80e0_a65f3c76","updated":"2026-08-18 07:31:49.000000000","message":"recheck","commit_id":"d307590ec19d88b771e45c1a7c5efc534de39d7d"}],"cyborg/agent/manager.py":[{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"54419de28287772f4a636ecc5b91bc22ea14a8ea","unresolved":false,"context_lines":[{"line_number":99,"context_line":"        self.agent_api \u003d AgentAPI()"},{"line_number":100,"context_line":"        self.image_api \u003d ImageAPI()"},{"line_number":101,"context_line":"        self._rt \u003d ResourceTracker(self.resource_provider_name, self.cond_api)"},{"line_number":102,"context_line":"        self._cleanup_executor \u003d futurist.ThreadPoolExecutor(max_workers\u003d4)"},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"    def init_host(self):"},{"line_number":105,"context_line":"        \"\"\"Hook called by RPCService.start() after the RPC server is up.\"\"\""}],"source_content_type":"text/x-python","patch_set":2,"id":"e308a34d_760330a8","line":102,"updated":"2026-08-06 11:07:47.000000000","message":"A futurist.ThreadPoolExecutor is instantiated in AgentManager.__init__ but there is no corresponding shutdown() call in any cleanup or stop method. Pending futures and worker threads may leak when the agent service shuts down.\n\n**Severity**: WARNING | **Confidence**: 0.8\n\n**Impact**: On agent shutdown or restart, in-flight cleanup operations are abandoned, leaving devices in \u0027cleaning\u0027 state permanently. Worker threads are not joined, which can cause process hang or resource leaks.\n\n**Suggestion**:\nAdd a cleanup/stop method to AgentManager (or wire into the existing RPCService stop hook) that calls self._cleanup_executor.shutdown(wait\u003dTrue). Alternatively, register an atexit handler. Consider setting a reasonable shutdown grace period.","commit_id":"716be9b89024fcf2ac4c98aebea7559469c9550a"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"0dcddf9511a9ed6a50e39d10397dbe705f581d7e","unresolved":false,"context_lines":[{"line_number":122,"context_line":"        )"},{"line_number":123,"context_line":"        try:"},{"line_number":124,"context_line":"            future.result(timeout\u003dCONF.nvme.cleanup_timeout)"},{"line_number":125,"context_line":"        except ("},{"line_number":126,"context_line":"            concurrent.futures.TimeoutError,"},{"line_number":127,"context_line":"            processutils.ProcessExecutionError,"},{"line_number":128,"context_line":"            OSError,"}],"source_content_type":"text/x-python","patch_set":3,"id":"989f6dd7_98e4fdea","line":125,"updated":"2026-08-06 15:15:54.000000000","message":"The except clause in AgentManager.cleanup_device only catches concurrent.futures.TimeoutError, processutils.ProcessExecutionError, OSError, and exception.CyborgException. If the driver cleanup() raises any other exception type (RuntimeError, ValueError, etc.), it propagates unhandled out of the R...\n\n**Severity**: CRITICAL | **Confidence**: 1.0\n\n**Risk**: In production, any driver cleanup failure that raises a non-CyborgException will leave the device stuck in \u0027cleaning\u0027 state, making it permanently unusable. The RPC layer may also return an unhandled error to oslo.messaging. The unit test test_cleanup_device_failure_sets_error will fail when run,...\n\n**Priority**: Immediate\n**Why This Matters**: In production, any driver cleanup failure that raises a non-CyborgException will leave the device stuck in \u0027cleaning\u0027 state, making it permanently unusable. The RPC layer may also return an unhandled error to oslo.messaging. The unit test test_cleanup_device_failure_sets_error will fail when run,...\n\n**Recommendation**:\nEither broaden the except clause to catch Exception (and log appropriately) or change the test to raise a CyborgException subclass. The safer fix is to add a broad except Exception as a final handler: except Exception: that logs and sets error state, ensuring no exception type can leave a device stuck in cleaning.","commit_id":"acbb8023b1e43a11246679a491ff35019b3fb28b"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"2eb9128c159269fdf92ae15c395e52a79d5d97cc","unresolved":false,"context_lines":[{"line_number":122,"context_line":"        )"},{"line_number":123,"context_line":"        try:"},{"line_number":124,"context_line":"            future.result(timeout\u003dCONF.nvme.cleanup_timeout)"},{"line_number":125,"context_line":"        except ("},{"line_number":126,"context_line":"            concurrent.futures.TimeoutError,"},{"line_number":127,"context_line":"            processutils.ProcessExecutionError,"},{"line_number":128,"context_line":"            OSError,"}],"source_content_type":"text/x-python","patch_set":4,"id":"dfcc6dd2_31b2a66e","line":125,"updated":"2026-08-06 16:59:51.000000000","message":"The except tuple in cleanup_device only catches TimeoutError, ProcessExecutionError, OSError, and CyborgException. Any other exception (e.g., RuntimeError) from driver cleanup() propagates unhandled, leaving the device stuck in CLEANING state. The test test_cleanup_device_failure_sets_error uses...\n\n**Severity**: HIGH | **Confidence**: 0.9\n\n**Risk**: Any driver cleanup() exception not in the caught tuple leaves the device in \u0027cleaning\u0027 state indefinitely, making it permanently unavailable. The RPC handler crashes with an unhandled exception. The unit test itself will fail at runtime.\n\n**Priority**: Before merge\n**Why This Matters**: Any driver cleanup() exception not in the caught tuple leaves the device in \u0027cleaning\u0027 state indefinitely, making it permanently unavailable. The RPC handler crashes with an unhandled exception. The unit test itself will fail at runtime.\n\n**Recommendation**:\nBroaden the except clause to catch Exception, since the intent is to handle any cleanup failure and transition to error state. Update the test accordingly or ensure it uses an exception type within the caught tuple.","commit_id":"43d703e3d65f5e3f8af2e18e015a1fcfea00670b"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"cb464259b9eddb061f777958498390bbd4f31419","unresolved":false,"context_lines":[{"line_number":115,"context_line":"        device.device_state \u003d constants.DEVICE_STATE_CLEANING"},{"line_number":116,"context_line":"        device.save(context)"},{"line_number":117,"context_line":""},{"line_number":118,"context_line":"        self._cleanup_executor.submit(self._run_cleanup_async, context, device)"},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"    def _run_cleanup_async(self, context, device):"},{"line_number":121,"context_line":"        try:"}],"source_content_type":"text/x-python","patch_set":9,"id":"f0848dad_e09f64dd","line":118,"updated":"2026-08-11 15:42:04.000000000","message":"The [nvme] cleanup_timeout option (default 900s) caps NVMe sanitization duration. _run_cleanup_async submits driver cleanup() to a futurist ThreadPoolExecutor but applies no timeout. If cleanup() blocks, the worker thread is consumed indefinitely. Tests reference mock_conf.nvme.cleanup_timeout\u003d30...\n\n**Severity**: WARNING | **Confidence**: 0.8\n\n**Impact**: Currently NVMe cleanup() is a no-op so no immediate hang. When implemented with real nvme-cli sanitize, a hung operation permanently consumes one of 4 worker threads. After 4 hangs the pool is exhausted and no cleanups can proceed. Configured timeout provides no protection.\n\n**Suggestion**:\nWrap the driver cleanup() call with a timeout using futurist or concurrent.futures applying CONF.nvme.cleanup_timeout. On timeout, transition the device to ERROR.","commit_id":"8dae04dacf3982bae7b61c8fc01621a1349a032e"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"cb464259b9eddb061f777958498390bbd4f31419","unresolved":false,"context_lines":[{"line_number":119,"context_line":""},{"line_number":120,"context_line":"    def _run_cleanup_async(self, context, device):"},{"line_number":121,"context_line":"        try:"},{"line_number":122,"context_line":"            drv \u003d self._rt.get_driver_for_type(device.type)"},{"line_number":123,"context_line":"            drv.cleanup(device)"},{"line_number":124,"context_line":"        except Exception:"},{"line_number":125,"context_line":"            LOG.exception(\u0027Cleanup failed for %s\u0027, device.uuid)"}],"source_content_type":"text/x-python","patch_set":9,"id":"38de254a_b383471c","line":122,"updated":"2026-08-11 15:42:04.000000000","message":"_run_cleanup_async calls self._rt.get_driver_for_type(device.type), but ResourceTracker defines no such method. Its interface is only _initialize_drivers, init_drivers, discover_all, check_duplicate_devices, update_usage. Drivers are stored in acc_drivers (keyed by stevedore name, not type). The...\n\n**Severity**: CRITICAL | **Confidence**: 1.0\n\n**Risk**: Every cleanup_device RPC will fail at AttributeError, transition device to ERROR, keep placement reserved. The entire cleanup lifecycle is non-functional. Devices needing sanitization will never be cleaned, stuck in ERROR with reserved placement inventory.\n\n**Priority**: Immediate\n**Why This Matters**: Every cleanup_device RPC will fail at AttributeError, transition device to ERROR, keep placement reserved. The entire cleanup lifecycle is non-functional. Devices needing sanitization will never be cleaned, stuck in ERROR with reserved placement inventory.\n\n**Recommendation**:\nAdd a get_driver_for_type method to ResourceTracker mapping device.type to the correct driver from self.acc_drivers, or restructure _run_cleanup_async to look up the driver directly. Ensure the lookup maps device type (e.g. NVME) to the corresponding stevedore driver entry.","commit_id":"8dae04dacf3982bae7b61c8fc01621a1349a032e"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"9bbd1f6d7ae2fbcf7b1589c4f026fab23db81500","unresolved":false,"context_lines":[{"line_number":117,"context_line":"        device.device_state \u003d constants.DEVICE_STATE_CLEANING"},{"line_number":118,"context_line":"        device.save(context)"},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"        self._cleanup_executor.submit(self._run_cleanup_async, context, device)"},{"line_number":121,"context_line":""},{"line_number":122,"context_line":"    def _run_cleanup_async(self, context, device):"},{"line_number":123,"context_line":"        timeout \u003d CONF.nvme.cleanup_timeout"}],"source_content_type":"text/x-python","patch_set":11,"id":"71d7cc9d_0e002973","line":120,"updated":"2026-08-13 05:07:27.000000000","message":"The agent\u0027s cleanup_device RPC handler calls self._cleanup_executor.submit() without a try/except. If the executor is shut down (e.g., during del_host shutdown), submit() raises RuntimeError(\u0027cannot schedule new futures after shutdown\u0027). This unhandled exception will propagate through the RPC layer.\n\n**Severity**: WARNING | **Confidence**: 0.8\n\n**Impact**: During service shutdown, if a cleanup_device RPC arrives after del_host has been called, the agent will log an unhandled exception traceback. The device will remain in CLEANING state with no recovery path.\n\n**Suggestion**:\nWrap the submit() call in a try/except RuntimeError, log a warning, and set the device to ERROR state so it can be manually recovered.","commit_id":"6dbda6c47e78c6194618ba19818e87cfca15e8eb"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"9bbd1f6d7ae2fbcf7b1589c4f026fab23db81500","unresolved":false,"context_lines":[{"line_number":124,"context_line":"        cleanup_executor \u003d futures.ThreadPoolExecutor(max_workers\u003d1)"},{"line_number":125,"context_line":"        cleanup_future \u003d None"},{"line_number":126,"context_line":"        try:"},{"line_number":127,"context_line":"            drv \u003d self._rt.get_driver_for_type(device.type)"},{"line_number":128,"context_line":"            cleanup_future \u003d cleanup_executor.submit(drv.cleanup, device)"},{"line_number":129,"context_line":"            cleanup_future.result(timeout\u003dtimeout)"},{"line_number":130,"context_line":"        except futures.TimeoutError:"}],"source_content_type":"text/x-python","patch_set":11,"id":"f2fb4ac6_b42fc0d7","line":127,"updated":"2026-08-13 05:07:27.000000000","message":"The _run_cleanup_async method calls self._rt.get_driver_for_type(device.type), but ResourceTracker has no such method. It only has an acc_drivers dict with stevedore driver names as keys. This will raise AttributeError at runtime whenever a device cleanup is attempted.\n\n**Severity**: CRITICAL | **Confidence**: 1.0\n\n**Risk**: Every device cleanup attempt will raise AttributeError, which is caught by the broad except clause and sets the device to ERROR state. This means the cleanup feature introduced by this patch cannot work at all in production — devices will always go to ERROR instead of being cleaned and returned t...\n\n**Priority**: Immediate\n**Why This Matters**: Every device cleanup attempt will raise AttributeError, which is caught by the broad except clause and sets the device to ERROR state. This means the cleanup feature introduced by this patch cannot work at all in production — devices will always go to ERROR instead of being cleaned and returned t...\n\n**Recommendation**:\nAdd a get_driver_for_type method to ResourceTracker that maps device type strings to loaded driver instances, or look up the driver directly from self._rt.acc_drivers using an appropriate key lookup mechanism.","commit_id":"6dbda6c47e78c6194618ba19818e87cfca15e8eb"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"9bbd1f6d7ae2fbcf7b1589c4f026fab23db81500","unresolved":false,"context_lines":[{"line_number":151,"context_line":"        device.save(context)"},{"line_number":152,"context_line":"        self._set_nvme_placement_reserved(context, device, False)"},{"line_number":153,"context_line":""},{"line_number":154,"context_line":"    def _set_nvme_placement_reserved(self, context, device, reserved):"},{"line_number":155,"context_line":"        try:"},{"line_number":156,"context_line":"            deployables \u003d objects.Deployable.get_list_by_device_id("},{"line_number":157,"context_line":"                context, device.id"}],"source_content_type":"text/x-python","patch_set":11,"id":"da669cd2_63b2506d","line":154,"updated":"2026-08-13 05:07:27.000000000","message":"Nearly identical logic for updating Placement inventory reserved counts exists in three places: AgentManager._set_nvme_placement_reserved (agent/manager.py:154-173), ConductorManager._set_device_placement_reserved (conductor/manager.py:97-108), and ext_arq._set_deployable_placement_reserved (obje...\n\n**Severity**: SUGGESTION | **Confidence**: 0.8\n\n**Benefit**: Future changes to the reservation logic (e.g., error handling, batch updates, different RC lookup) must be applied in three places. Divergence between copies is likely, leading to inconsistent placement state.\n\n**Recommendation**:\nExtract a shared utility function (e.g., in cyborg/common/placement_client.py or a new helper module) that all three call sites use. This reduces duplication and ensures consistent behavior.","commit_id":"6dbda6c47e78c6194618ba19818e87cfca15e8eb"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"0dfd5c0a14bbe6c05733e2ab43304cbe8d01b1b7","unresolved":false,"context_lines":[{"line_number":124,"context_line":"        cleanup_executor \u003d futures.ThreadPoolExecutor(max_workers\u003d1)"},{"line_number":125,"context_line":"        cleanup_future \u003d None"},{"line_number":126,"context_line":"        try:"},{"line_number":127,"context_line":"            drv \u003d self._rt.get_driver_for_type(device.type)"},{"line_number":128,"context_line":"            cleanup_future \u003d cleanup_executor.submit(drv.cleanup, device)"},{"line_number":129,"context_line":"            cleanup_future.result(timeout\u003dtimeout)"},{"line_number":130,"context_line":"        except futures.TimeoutError:"}],"source_content_type":"text/x-python","patch_set":13,"id":"44006fae_3a00f78a","line":127,"updated":"2026-08-14 15:51:04.000000000","message":"The new async cleanup path calls self._rt.get_driver_for_type(device.type), but the ResourceTracker class defines no such method, so every cleanup attempt raises AttributeError and is treated as a cleanup failure.\n\n**Severity**: CRITICAL | **Confidence**: 0.9\n\n**Risk**: Every real NVMe cleanup fails immediately; devices go to ERROR with inventory permanently reserved and no device is ever sanitized. Unit tests pass only because ResourceTracker is fully mocked.\n\n**Priority**: Immediate\n**Why This Matters**: Every real NVMe cleanup fails immediately; devices go to ERROR with inventory permanently reserved and no device is ever sanitized. Unit tests pass only because ResourceTracker is fully mocked.\n\n**Recommendation**:\nAdd a get_driver_for_type method to ResourceTracker (dispatching on device.type via acc_drivers or the stevedore registry) or use an existing driver lookup; add at least one test that exercises the path with a real or minimally stubbed ResourceTracker.","commit_id":"8e487f0b123aea37d93e8d88b56e12eb6919b8c9"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"2079ec583b30fa46222f73b179f5e3657db6fc16","unresolved":false,"context_lines":[{"line_number":124,"context_line":"        cleanup_executor \u003d futures.ThreadPoolExecutor(max_workers\u003d1)"},{"line_number":125,"context_line":"        cleanup_future \u003d None"},{"line_number":126,"context_line":"        try:"},{"line_number":127,"context_line":"            drv \u003d self._rt.get_driver_for_type(device.type)"},{"line_number":128,"context_line":"            cleanup_future \u003d cleanup_executor.submit(drv.cleanup, device)"},{"line_number":129,"context_line":"            cleanup_future.result(timeout\u003dtimeout)"},{"line_number":130,"context_line":"        except futures.TimeoutError:"}],"source_content_type":"text/x-python","patch_set":14,"id":"2d75a48a_510c0a96","line":127,"updated":"2026-08-15 16:42:48.000000000","message":"_run_cleanup_async calls self._rt.get_driver_for_type(device.type), but ResourceTracker defines no such method anywhere in the codebase; it only stores drivers in self.acc_drivers. Every real agent-side cleanup will raise AttributeError.\n\n**Severity**: CRITICAL | **Confidence**: 1.0\n\n**Risk**: The intended ERROR path is still reached (the generic except sets device_state\u003derror and keeps inventory reserved), so devices do not become AVAILABLE incorrectly, but cleanup never runs and every NVMe device goes to ERROR after first unbind.\n\n**Priority**: Immediate\n**Why This Matters**: The intended ERROR path is still reached (the generic except sets device_state\u003derror and keeps inventory reserved), so devices do not become AVAILABLE incorrectly, but cleanup never runs and every NVMe device goes to ERROR after first unbind.\n\n**Recommendation**:\nAdd a get_driver_for_type() method to ResourceTracker (e.g. iterate self.acc_drivers matching the driver\u0027s device type) or look up the driver by CONF-based enabled driver names. Add a test against a real ResourceTracker or assert the method exists on the class.","commit_id":"dca8f97b449ebcd2ab171eedf43c7b87f8d23614"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"8c527eadc9b5f28d1a902bc572ebcc964e07309a","unresolved":false,"context_lines":[{"line_number":124,"context_line":"        cleanup_executor \u003d futures.ThreadPoolExecutor(max_workers\u003d1)"},{"line_number":125,"context_line":"        cleanup_future \u003d None"},{"line_number":126,"context_line":"        try:"},{"line_number":127,"context_line":"            drv \u003d self._rt.get_driver_for_type(device.type)"},{"line_number":128,"context_line":"            cleanup_future \u003d cleanup_executor.submit(drv.cleanup, device)"},{"line_number":129,"context_line":"            cleanup_future.result(timeout\u003dtimeout)"},{"line_number":130,"context_line":"        except futures.TimeoutError:"}],"source_content_type":"text/x-python","patch_set":15,"id":"3a3ff7bc_d46b0486","line":127,"updated":"2026-08-17 08:15:38.000000000","message":"_run_cleanup_async resolves the driver via self._rt.get_driver_for_type(device.type), but ResourceTracker defines no such method; it only exposes the acc_drivers dict keyed by enabled driver name. The AttributeError is swallowed by the broad except, so every real NVMe cleanup is recorded as a fai...\n\n**Severity**: CRITICAL | **Confidence**: 1.0\n\n**Risk**: In production every NVMe device cleanup immediately fails: the driver cleanup never runs, devices are permanently stuck in ERROR with Placement inventory reserved, exhausting accelerator capacity that can never be reclaimed automatically.\n\n**Priority**: Immediate\n**Why This Matters**: In production every NVMe device cleanup immediately fails: the driver cleanup never runs, devices are permanently stuck in ERROR with Placement inventory reserved, exhausting accelerator capacity that can never be reclaimed automatically.\n\n**Recommendation**:\nImplement get_driver_for_type on ResourceTracker (mapping device.type to the loaded acc_drivers entry), or look the driver up directly from ResourceTracker.acc_drivers. Add a unit test that runs cleanup against a real (non-mocked) ResourceTracker driver mapping.","commit_id":"8ded769b2d17779ce2471aa5989d34a23f3fc886"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"8c527eadc9b5f28d1a902bc572ebcc964e07309a","unresolved":false,"context_lines":[{"line_number":128,"context_line":"            cleanup_future \u003d cleanup_executor.submit(drv.cleanup, device)"},{"line_number":129,"context_line":"            cleanup_future.result(timeout\u003dtimeout)"},{"line_number":130,"context_line":"        except futures.TimeoutError:"},{"line_number":131,"context_line":"            LOG.warning("},{"line_number":132,"context_line":"                \u0027Cleanup timed out for %s after %ss\u0027,"},{"line_number":133,"context_line":"                device.uuid,"},{"line_number":134,"context_line":"                timeout,"}],"source_content_type":"text/x-python","patch_set":15,"id":"3b5349c6_d26e624b","line":131,"updated":"2026-08-17 08:15:38.000000000","message":"On timeout, cleanup_future.cancel() only works if the future has not started; a running drv.cleanup cannot be cancelled, and shutdown(wait\u003dFalse, cancel_futures\u003dTrue) abandons the worker thread. The sanitizer keeps executing on a device that was already marked ERROR and left reserved.\n\n**Severity**: WARNING | **Confidence**: 0.8\n\n**Impact**: After a timeout the underlying nvme-cli sanitize may still be running against a device reported as failed; the orphan thread leaks per timed-out cleanup, and on agent shutdown abandoned workers can delay process exit or continue touching the device after ERROR was recorded.\n\n**Suggestion**:\nHave the driver cleanup enforce the timeout cooperatively (e.g., nvme-cli subprocess timeout inside NVMeDriver.cleanup), or track orphaned futures and reconcile device state when they eventually complete instead of abandoning them; reuse a shared bounded executor rather than creating a new one-worker pool per call.","commit_id":"8ded769b2d17779ce2471aa5989d34a23f3fc886"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"d93ab15f098a45b257ce4b51e08687fc58aec90e","unresolved":false,"context_lines":[{"line_number":124,"context_line":"        cleanup_executor \u003d futures.ThreadPoolExecutor(max_workers\u003d1)"},{"line_number":125,"context_line":"        cleanup_future \u003d None"},{"line_number":126,"context_line":"        try:"},{"line_number":127,"context_line":"            drv \u003d self._rt.get_driver_for_type(device.type)"},{"line_number":128,"context_line":"            cleanup_future \u003d cleanup_executor.submit(drv.cleanup, device)"},{"line_number":129,"context_line":"            cleanup_future.result(timeout\u003dtimeout)"},{"line_number":130,"context_line":"        except futures.TimeoutError:"}],"source_content_type":"text/x-python","patch_set":16,"id":"9191cf8d_e7db4669","line":127,"updated":"2026-08-18 06:23:02.000000000","message":"AgentManager._run_cleanup_async resolves the driver via self._rt.get_driver_for_type(device.type), but ResourceTracker (cyborg/agent/resource_tracker.py) defines no such method (only __init__, _initialize_drivers, init_drivers, discover_all, check_duplicate_devices, update_usage). Every real cleanup attempt raises AttributeError, which the broad \u0027except Exception\u0027 converts into DEVICE_STATE_ERROR plus a reserved-inventory hold.\n\n**Severity**: HIGH | **Confidence**: 0.95\n\n**Impact**: Every device cleanup on a real agent fails immediately: the device is marked ERROR and its Placement inventory stays fully reserved, permanently removing the device\u0027s capacity until an operator intervenes. The unit tests mask this because ResourceTracker is fully mocked.\n\n**Priority**: Before merge\n**Recommendation**:\nResolve the driver from the agent\u0027s loaded drivers, e.g. add a get_driver_for_type(type) helper to ResourceTracker (matching device.type against self.acc_drivers values), and add a test that does not mock ResourceTracker for this path.","commit_id":"d307590ec19d88b771e45c1a7c5efc534de39d7d"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"0eb1d94aa8aa5967d7daac4bea70519ca8f05ee5","unresolved":false,"context_lines":[{"line_number":124,"context_line":"        cleanup_executor \u003d futures.ThreadPoolExecutor(max_workers\u003d1)"},{"line_number":125,"context_line":"        cleanup_future \u003d None"},{"line_number":126,"context_line":"        try:"},{"line_number":127,"context_line":"            drv \u003d self._rt.get_driver_for_type(device.type)"},{"line_number":128,"context_line":"            cleanup_future \u003d cleanup_executor.submit(drv.cleanup, device)"},{"line_number":129,"context_line":"            cleanup_future.result(timeout\u003dtimeout)"},{"line_number":130,"context_line":"        except futures.TimeoutError:"}],"source_content_type":"text/x-python","patch_set":16,"id":"745eede5_5edd731b","line":127,"updated":"2026-08-18 08:48:35.000000000","message":"AgentManager._run_cleanup_async resolves the driver via self._rt.get_driver_for_type(device.type), but ResourceTracker (cyborg/agent/resource_tracker.py) defines only __init__, _initialize_drivers, init_drivers, discover_all, check_duplicate_devices and update_usage. There is no get_driver_for_type anywhere in the codebase except this call site. At runtime the lookup raises AttributeError, which the broad \u0027except Exception\u0027 handler catches; the device is then saved with DEVICE_STATE_ERROR and its Placement inventory is kept fully reserved.\n\n**Severity**: CRITICAL | **Confidence**: 0.97\n\n**Impact**: Every ARQ unbind of a cleaning-capable device ends in ERROR with inventory permanently reserved: the device is unusable for all future allocations until an operator manually edits device_state and Placement. The headline feature of the patch never works in production.\n\n**Priority**: Immediate\n**Recommendation**:\nImplement get_driver_for_type(device_type) on ResourceTracker (map DEVICE_NVME/DEVICE_FPGA/DEVICE_GPU to the loaded driver in self.acc_drivers, e.g. via the type-to-driver mapping already used by discover), and add a unit test that exercises the real ResourceTracker (not a full mock) or asserts the method exists so the regression is caught before merge.","commit_id":"d307590ec19d88b771e45c1a7c5efc534de39d7d"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"d93ab15f098a45b257ce4b51e08687fc58aec90e","unresolved":false,"context_lines":[{"line_number":127,"context_line":"            drv \u003d self._rt.get_driver_for_type(device.type)"},{"line_number":128,"context_line":"            cleanup_future \u003d cleanup_executor.submit(drv.cleanup, device)"},{"line_number":129,"context_line":"            cleanup_future.result(timeout\u003dtimeout)"},{"line_number":130,"context_line":"        except futures.TimeoutError:"},{"line_number":131,"context_line":"            LOG.warning("},{"line_number":132,"context_line":"                \u0027Cleanup timed out for %s after %ss\u0027,"},{"line_number":133,"context_line":"                device.uuid,"}],"source_content_type":"text/x-python","patch_set":16,"id":"ebb52ae2_10f6c0ad","line":130,"updated":"2026-08-18 06:23:02.000000000","message":"On timeout, _run_cleanup_async calls cleanup_future.cancel() and cleanup_executor.shutdown(wait\u003dFalse, cancel_futures\u003dTrue). concurrent.futures.Future.cancel() cannot cancel a task that is already running, so a hung drv.cleanup keeps executing against the device after it has been marked ERROR, and a brand-new ThreadPoolExecutor and thread are created on every cleanup attempt.\n\n**Severity**: WARNING | **Confidence**: 0.8\n\n**Impact**: Each timed-out cleanup leaks one thread; a stale NVMe sanitize keeps writing to a device that was flagged ERROR and may later be returned to service by an operator, and repeated unbind failures accumulate unbounded threads on the agent.\n\n**Suggestion**:\nMake cancellation cooperative: enforce the timeout inside NVMeDriver.cleanup (e.g. subprocess timeout on the nvme-cli invocation) instead of wrapping it in a disposable executor, and reuse the existing bounded _cleanup_executor.","commit_id":"d307590ec19d88b771e45c1a7c5efc534de39d7d"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caa791a2e4397cbc6e51b71e728feaad4ea4779a","unresolved":true,"context_lines":[{"line_number":100,"context_line":"        self.agent_api \u003d AgentAPI()"},{"line_number":101,"context_line":"        self.image_api \u003d ImageAPI()"},{"line_number":102,"context_line":"        self._rt \u003d ResourceTracker(self.resource_provider_name, self.cond_api)"},{"line_number":103,"context_line":"        self._cleanup_executor \u003d futurist.ThreadPoolExecutor(max_workers\u003d4)"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"    def del_host(self):"},{"line_number":106,"context_line":"        self._cleanup_executor.shutdown(wait\u003dTrue)"}],"source_content_type":"text/x-python","patch_set":18,"id":"ceaee0bc_256d3fec","line":103,"range":{"start_line":103,"start_character":73,"end_line":103,"end_character":74},"updated":"2026-08-21 17:51:34.000000000","message":"this should not be hardcoded if we are intoducing a new executor we shoudl have a config option for it","commit_id":"72156e2a410ebbe9a66f888978f7b4a40664eb49"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caa791a2e4397cbc6e51b71e728feaad4ea4779a","unresolved":true,"context_lines":[{"line_number":102,"context_line":"        self._rt \u003d ResourceTracker(self.resource_provider_name, self.cond_api)"},{"line_number":103,"context_line":"        self._cleanup_executor \u003d futurist.ThreadPoolExecutor(max_workers\u003d4)"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"    def del_host(self):"},{"line_number":106,"context_line":"        self._cleanup_executor.shutdown(wait\u003dTrue)"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"    def init_host(self):"}],"source_content_type":"text/x-python","patch_set":18,"id":"6d2120b3_2a391c89","line":105,"range":{"start_line":105,"start_character":8,"end_line":105,"end_character":16},"updated":"2026-08-21 17:51:34.000000000","message":"del host is not the recipcal of init_host\n\ni woudl call this `shutdown`.","commit_id":"72156e2a410ebbe9a66f888978f7b4a40664eb49"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caa791a2e4397cbc6e51b71e728feaad4ea4779a","unresolved":true,"context_lines":[{"line_number":145,"context_line":""},{"line_number":146,"context_line":"    def _set_nvme_placement_reserved(self, context, device, reserved):"},{"line_number":147,"context_line":"        try:"},{"line_number":148,"context_line":"            deployables \u003d objects.Deployable.get_list_by_device_id("},{"line_number":149,"context_line":"                context, device.id"},{"line_number":150,"context_line":"            )"},{"line_number":151,"context_line":"            for dep in deployables:"},{"line_number":152,"context_line":"                filters \u003d {\u0027deployable_id\u0027: dep.id, \u0027key\u0027: \u0027rc\u0027}"},{"line_number":153,"context_line":"                attrs \u003d objects.Attribute.get_by_filter(context, filters)"},{"line_number":154,"context_line":"                if not attrs:"},{"line_number":155,"context_line":"                    continue"},{"line_number":156,"context_line":"                rc \u003d attrs[0].value"},{"line_number":157,"context_line":"                reserved_count \u003d dep.num_accelerators if reserved else 0"},{"line_number":158,"context_line":"                self.placement_client.update_rp_inventory_reserved("},{"line_number":159,"context_line":"                    dep.rp_uuid, rc, dep.num_accelerators, reserved_count"},{"line_number":160,"context_line":"                )"},{"line_number":161,"context_line":"        except Exception:"},{"line_number":162,"context_line":"            LOG.exception("},{"line_number":163,"context_line":"                \u0027Failed to update Placement reserved for NVMe device %s\u0027,"}],"source_content_type":"text/x-python","patch_set":18,"id":"78913694_4cc856de","line":160,"range":{"start_line":148,"start_character":10,"end_line":160,"end_character":17},"updated":"2026-08-21 17:51:34.000000000","message":"why are you hitting the db for this?\n\nwe do not allow specifying the resouce class to be used via the config\n\nhttps://review.opendev.org/c/openstack/cyborg/+/999936/15/cyborg/conf/nvme.py\n\nso we alwasy know the resouce classs will be `CUSTOM_NVME_\u003cVENDOR_ID\u003e_\u003cPRODUCT_ID\u003e`\n\nand we know the resouce provider will be `\u003chostname\u003e_\u003cpci_address\u003e`\n\nif device is a driver_device object we have the contolpalth ide \n\nhttps://github.com/openstack/cyborg/blob/master/cyborg/objects/driver_objects/driver_device.py#L46-L48 \nwhich old the pci adress\n\nso we can determin both the resouce class and resouce provider form teh device object directly without the db lookup or looping.\n\nwe also have the list of deployabels in that but we also know that we have a 1 to 1 mapping currently so we know the deployable ocunt.\n\n\n\nif we are usign the generic device object\n\nhttps://github.com/openstack/cyborg/blob/master/cyborg/objects/device.py\n\nwe still have the resvendor ids and product id i belive althoguh it smore indirect\n\n\nwe also really shoudl ounly have on device or atribute object\n\n\nwe need to clean this up next cycles as we shoudl not have sperte driver versions\n\n\nwe also need to keep the two in sync today so i need to go back and look at your peralier patches","commit_id":"72156e2a410ebbe9a66f888978f7b4a40664eb49"}],"cyborg/agent/rpcapi.py":[{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"2eb9128c159269fdf92ae15c395e52a79d5d97cc","unresolved":false,"context_lines":[{"line_number":107,"context_line":"            ah_uuid\u003dah_uuid,"},{"line_number":108,"context_line":"        )"},{"line_number":109,"context_line":""},{"line_number":110,"context_line":"    def cleanup_device(self, context, hostname, device):"},{"line_number":111,"context_line":"        version \u003d \u00271.1\u0027"},{"line_number":112,"context_line":"        if not self.client.can_send_version(version):"},{"line_number":113,"context_line":"            raise exception.AgentRPCVersionNotSupported(version\u003dversion)"}],"source_content_type":"text/x-python","patch_set":4,"id":"380b501f_e0097403","line":110,"updated":"2026-08-06 16:59:51.000000000","message":"AgentAPI.__init__ hardcodes version_cap\u003dself.RPC_API_VERSION (\u00271.1\u0027), so can_send_version(\u00271.1\u0027) always returns True. AgentRPCVersionNotSupported is never raised, making the conductor\u0027s mixed-version fallback (conductor/manager.py lines 108-116) dead code. During rolling upgrade, a 1.1 conductor...\n\n**Severity**: WARNING | **Confidence**: 0.8\n\n**Impact**: During rolling upgrade with mixed 1.0/1.1 agents, cleanup RPCs to older agents silently fail, leaving devices in pending_cleaning state indefinitely. The graceful fallback described in commit message and release note never activates.\n\n**Suggestion**:\nExpose a config option (e.g., [agent] rpc_version_cap) so operators can pin the version during upgrades. Alternatively, document that all agents must be upgraded before conductors and remove the dead fallback code.","commit_id":"43d703e3d65f5e3f8af2e18e015a1fcfea00670b"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"2079ec583b30fa46222f73b179f5e3657db6fc16","unresolved":false,"context_lines":[{"line_number":106,"context_line":"            ah_uuid\u003dah_uuid,"},{"line_number":107,"context_line":"        )"},{"line_number":108,"context_line":""},{"line_number":109,"context_line":"    def cleanup_device(self, context, hostname, device):"},{"line_number":110,"context_line":"        cctxt \u003d self.client.prepare(server\u003dhostname, version\u003d\u00271.1\u0027)"},{"line_number":111,"context_line":"        cctxt.cast(context, \u0027cleanup_device\u0027, device\u003ddevice)"}],"source_content_type":"text/x-python","patch_set":14,"id":"0a53521d_6ae08c76","line":109,"updated":"2026-08-15 16:42:48.000000000","message":"Existing versioned methods in AgentAPI guard the version bump with if not self.client.can_send_version(version): raise ...; the new cleanup_device unconditionally prepares version\u003d\u00271.1\u0027.\n\n**Severity**: HIGH | **Confidence**: 0.9\n\n**Risk**: Against a 1.0-capped agent, the cast raises instead of failing gracefully, losing the standard error semantics for this method.\n\n**Priority**: Before merge\n**Why This Matters**: Against a 1.0-capped agent, the cast raises instead of failing gracefully, losing the standard error semantics for this method.\n\n**Recommendation**:\nFollow the existing pattern: check can_send_version(\u00271.1\u0027) and raise a clear exception so the conductor can handle the downgrade path, or document why the cast is intentionally unconditional.","commit_id":"dca8f97b449ebcd2ab171eedf43c7b87f8d23614"}],"cyborg/common/exception.py":[{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"ef1d573440100e4dd8f26d79074217f826724757","unresolved":false,"context_lines":[{"line_number":449,"context_line":"    _msg_fmt \u003d _(\"Invalid configuration: %(reason)s.\")"},{"line_number":450,"context_line":""},{"line_number":451,"context_line":""},{"line_number":452,"context_line":"class AgentRPCVersionNotSupported(CyborgException):"},{"line_number":453,"context_line":"    _msg_fmt \u003d _(\"Agent does not support RPC version %(version)s.\")"},{"line_number":454,"context_line":""},{"line_number":455,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"4de951d7_20add827","line":452,"updated":"2026-08-07 07:12:11.000000000","message":"The new AgentRPCVersionNotSupported exception class in cyborg/common/exception.py is never imported, raised, or caught anywhere in the codebase. It appears to have been intended for the version-checking fallback logic that the release note describes but which was not implemented.\n\n**Severity**: SUGGESTION | **Confidence**: 0.9\n\n**Benefit**: Unused code adds maintenance burden and may confuse future developers who encounter it. Its presence suggests incomplete implementation of the version-checking logic that CF-001 identifies as missing.\n\n**Recommendation**:\nRemove the unused exception class, or implement the version-checking fallback logic that uses it.","commit_id":"12730005df148167fd60ba7c3e7ea79c90af9cc3"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"c0e6e52fa1ca41eb07f7c713ad72405f1ebd6683","unresolved":false,"context_lines":[{"line_number":449,"context_line":"    _msg_fmt \u003d _(\"Invalid configuration: %(reason)s.\")"},{"line_number":450,"context_line":""},{"line_number":451,"context_line":""},{"line_number":452,"context_line":"class AgentRPCVersionNotSupported(CyborgException):"},{"line_number":453,"context_line":"    _msg_fmt \u003d _(\"Agent does not support RPC version %(version)s.\")"},{"line_number":454,"context_line":""},{"line_number":455,"context_line":""}],"source_content_type":"text/x-python","patch_set":6,"id":"ed6c5d18_d78ca15d","line":452,"updated":"2026-08-07 16:36:29.000000000","message":"The AgentRPCVersionNotSupported exception class is added to exception.py but is never raised, caught, or imported anywhere in the codebase. It appears to be a leftover from an earlier design iteration where dispatch_cleanup would raise it instead of logging a warning.\n\n**Severity**: SUGGESTION | **Confidence**: 0.9\n\n**Benefit**: Minimal — dead code that adds maintenance noise and may confuse future developers into thinking the exception is handled somewhere.\n\n**Recommendation**:\nRemove the AgentRPCVersionNotSupported class, or use it in dispatch_cleanup where the version-check branch currently only logs a warning.","commit_id":"1d00970d959fbaa2025c54b5cbc0ce74c79eeecb"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"c1f42870dcaba2416d6ac882c6d8fbe6106e4736","unresolved":false,"context_lines":[{"line_number":445,"context_line":"    _msg_fmt \u003d _(\"Device %(device)s not found.\")"},{"line_number":446,"context_line":""},{"line_number":447,"context_line":""},{"line_number":448,"context_line":"class DeviceCleanupFailed(CyborgException):"},{"line_number":449,"context_line":"    _msg_fmt \u003d _(\"Device cleanup failed: %(reason)s.\")"},{"line_number":450,"context_line":""},{"line_number":451,"context_line":""}],"source_content_type":"text/x-python","patch_set":7,"id":"d99888d4_f3990126","line":448,"updated":"2026-08-09 05:23:10.000000000","message":"Two exception classes are added to exception.py but are never imported or raised anywhere in the codebase. DeviceCleanupFailed could be raised in the cleanup_device failure path; AgentRPCVersionNotSupported could be raised when can_send_version returns False. Neither is used.\n\n**Severity**: SUGGESTION | **Confidence**: 0.9\n\n**Benefit**: Dead code that adds maintenance burden. A reviewer will ask why these exceptions exist if they are never used. They either should be wired into the failure paths or removed until needed.\n\n**Recommendation**:\nEither raise DeviceCleanupFailed in the cleanup_device except block and AgentRPCVersionNotSupported in the dispatch_cleanup version-check branch, or remove the unused classes and add them when the consuming code is ready.","commit_id":"9a79e50a2d9a06e8b237fdb6214a65245cf1a1f2"}],"cyborg/common/placement_client.py":[{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"c1f42870dcaba2416d6ac882c6d8fbe6106e4736","unresolved":false,"context_lines":[{"line_number":431,"context_line":"        update_inventory \u003d {resource: {\"total\": total, \"reserved\": reserved}}"},{"line_number":432,"context_line":"        self.update_inventory(rp_uuid, update_inventory, version\u003d\u00271.26\u0027)"},{"line_number":433,"context_line":""},{"line_number":434,"context_line":"    def set_rp_inventory_reserved(self, rp_uuid, rc, total, reserved):"},{"line_number":435,"context_line":"        \"\"\"Update RP inventory reserved count."},{"line_number":436,"context_line":""},{"line_number":437,"context_line":"        :param rp_uuid: UUID of the resource provider."}],"source_content_type":"text/x-python","patch_set":7,"id":"1a713880_f00c9aeb","line":434,"updated":"2026-08-09 05:23:10.000000000","message":"update_rp_inventory_reserved takes reserved as an integer count, while the newly added set_rp_inventory_reserved takes reserved as a boolean. Both methods are public on the same class and share the word \u0027reserved\u0027 in their signature, making it easy to pass the wrong type.\n\n**Severity**: WARNING | **Confidence**: 0.8\n\n**Impact**: A developer calling update_rp_inventory_reserved(rp, rc, total, True) would set reserved to 1 instead of total, silently under-reserving. Conversely, calling set_rp_inventory_reserved with an integer would be truthy and reserve all capacity regardless of the intended count.\n\n**Suggestion**:\nRename the boolean parameter in set_rp_inventory_reserved to \u0027reserve_all\u0027 or \u0027reserve\u0027 to distinguish it from the integer \u0027reserved\u0027 in update_rp_inventory_reserved. Or inline the three-line wrapper to avoid the overloaded name entirely.","commit_id":"9a79e50a2d9a06e8b237fdb6214a65245cf1a1f2"}],"cyborg/common/service.py":[{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"caa791a2e4397cbc6e51b71e728feaad4ea4779a","unresolved":true,"context_lines":[{"line_number":72,"context_line":"    def stop(self, graceful\u003dTrue):"},{"line_number":73,"context_line":"        if hasattr(self.manager, \u0027del_host\u0027):"},{"line_number":74,"context_line":"            try:"},{"line_number":75,"context_line":"                self.manager.del_host()"},{"line_number":76,"context_line":"            except Exception as e:"},{"line_number":77,"context_line":"                LOG.exception(\u0027Error during manager cleanup: %s\u0027, e)"},{"line_number":78,"context_line":""}],"source_content_type":"text/x-python","patch_set":18,"id":"ee09d6b7_58d573f7","line":75,"range":{"start_line":75,"start_character":15,"end_line":75,"end_character":39},"updated":"2026-08-21 17:51:34.000000000","message":"we shoudl defien `showdown()` for all our manger object even if its a noop so that we do not have to special case like this we could also call this stop() if that is not already used to keep the names aligned.","commit_id":"72156e2a410ebbe9a66f888978f7b4a40664eb49"}],"cyborg/conductor/manager.py":[{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"0dcddf9511a9ed6a50e39d10397dbe705f581d7e","unresolved":false,"context_lines":[{"line_number":102,"context_line":"        device.device_state \u003d constants.DEVICE_STATE_PENDING_CLEANING"},{"line_number":103,"context_line":"        device.save(context)"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"        agent_api \u003d AgentAPI()"},{"line_number":106,"context_line":"        try:"},{"line_number":107,"context_line":"            agent_api.cleanup_device(context, device.hostname, device)"},{"line_number":108,"context_line":"        except exception.AgentRPCVersionNotSupported:"}],"source_content_type":"text/x-python","patch_set":3,"id":"cc034256_7cb915b8","line":105,"updated":"2026-08-06 15:15:54.000000000","message":"ConductorManager.dispatch_cleanup creates a new AgentAPI() instance on every invocation. AgentAPI.__init__ creates a new RPC client with serializer and target on each call. This is unnecessary overhead since the conductor manager is a long-lived singleton.\n\n**Severity**: SUGGESTION | **Confidence**: 0.8\n\n**Benefit**: Repeated RPC client initialization on each unbind operation. Minor performance overhead and inconsistent with the codebase pattern of caching RPC clients.\n\n**Recommendation**:\nStore self.agent_api \u003d AgentAPI() in ConductorManager.__init__ and reference it in dispatch_cleanup. This matches the pattern used by AgentManager.","commit_id":"acbb8023b1e43a11246679a491ff35019b3fb28b"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"c0e6e52fa1ca41eb07f7c713ad72405f1ebd6683","unresolved":false,"context_lines":[{"line_number":104,"context_line":"        device.save(context)"},{"line_number":105,"context_line":""},{"line_number":106,"context_line":"        agent_api \u003d AgentAPI()"},{"line_number":107,"context_line":"        if not agent_api.client.can_send_version(\u00271.1\u0027):"},{"line_number":108,"context_line":"            LOG.warning("},{"line_number":109,"context_line":"                \u0027Agent for device %s does not support cleanup RPC 1.1 \u0027"},{"line_number":110,"context_line":"                \u0027(cluster pinned below 1.1); setting device to error.\u0027,"}],"source_content_type":"text/x-python","patch_set":6,"id":"e5efedb1_60ddc366","line":107,"updated":"2026-08-07 16:36:29.000000000","message":"The conductor\u0027s dispatch_cleanup uses `agent_api.client.can_send_version(\u00271.1\u0027)` as a rolling-upgrade safety guard, but AgentAPI hardcodes `version_cap\u003dself.RPC_API_VERSION` (currently \u00271.1\u0027). Because can_send_version only checks the local version_cap (not the remote agent\u0027s actual version), it a...\n\n**Severity**: HIGH | **Confidence**: 0.8\n\n**Risk**: During a rolling upgrade with a 1.1 conductor and a 1.0 agent, the conductor sets the device to pending_cleaning and casts cleanup_device to the 1.0 agent. The agent cannot handle the 1.1 method and silently drops the cast. The device remains stuck in pending_cleaning indefinitely, requiring manu...\n\n**Priority**: Before merge\n**Why This Matters**: During a rolling upgrade with a 1.1 conductor and a 1.0 agent, the conductor sets the device to pending_cleaning and casts cleanup_device to the 1.0 agent. The agent cannot handle the 1.1 method and silently drops the cast. The device remains stuck in pending_cleaning indefinitely, requiring manu...\n\n**Recommendation**:\nEither (a) remove the can_send_version guard and correct the release note to remove the false rolling-upgrade fallback claim, or (b) implement a CONF-based version_cap for the agent (e.g., CONF.upgrade_levels.agent) so the guard works as documented during rolling upgrades.","commit_id":"1d00970d959fbaa2025c54b5cbc0ce74c79eeecb"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"fde840b94ca66d4570251de6d9463bd790eb0b4f","unresolved":false,"context_lines":[{"line_number":104,"context_line":"        device.save(context)"},{"line_number":105,"context_line":""},{"line_number":106,"context_line":"        agent_api \u003d AgentAPI()"},{"line_number":107,"context_line":"        if not agent_api.client.can_send_version(\u00271.1\u0027):"},{"line_number":108,"context_line":"            LOG.warning("},{"line_number":109,"context_line":"                \u0027Agent for device %s does not support cleanup RPC 1.1 \u0027"},{"line_number":110,"context_line":"                \u0027(cluster pinned below 1.1); setting device to error.\u0027,"}],"source_content_type":"text/x-python","patch_set":8,"id":"5940915b_38eddd55","line":107,"updated":"2026-08-11 09:45:30.000000000","message":"dispatch_cleanup() uses can_send_version(\u00271.1\u0027) to detect old agents, but AgentAPI hardcodes version_cap\u003dself.RPC_API_VERSION (always latest, currently 1.1). Since the cap equals the checked version, can_send_version always returns True. During rolling upgrades a 1.1 cast to a 1.0 agent fails at...\n\n**Severity**: HIGH | **Confidence**: 0.8\n\n**Risk**: During rolling upgrades with old agents, cleanup_device RPCs are sent to 1.0 agents that cannot handle them. The device is already saved as pending_cleaning before the cast, so it remains stuck indefinitely. An operator must manually reset the device state.\n\n**Priority**: Before merge\n**Why This Matters**: During rolling upgrades with old agents, cleanup_device RPCs are sent to 1.0 agents that cannot handle them. The device is already saved as pending_cleaning before the cast, so it remains stuck indefinitely. An operator must manually reset the device state.\n\n**Recommendation**:\nEither (a) introduce an upgrade_levels config option that allows pinning the AgentAPI version_cap during rolling upgrades, or (b) wrap the cleanup_device cast in try/except for oslo_messaging errors and transition the device to error on failure, or (c) remove the guard and rely on cast failure handling.","commit_id":"8826c1db883c57114595c17381159ed5ab942236"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"cb464259b9eddb061f777958498390bbd4f31419","unresolved":false,"context_lines":[{"line_number":103,"context_line":"        device.device_state \u003d constants.DEVICE_STATE_PENDING_CLEANING"},{"line_number":104,"context_line":"        device.save(context)"},{"line_number":105,"context_line":""},{"line_number":106,"context_line":"        agent_api \u003d AgentAPI()"},{"line_number":107,"context_line":"        if not agent_api.client.can_send_version(\u00271.1\u0027):"},{"line_number":108,"context_line":"            LOG.warning("},{"line_number":109,"context_line":"                \u0027Agent for device %s does not support cleanup RPC 1.1 \u0027"}],"source_content_type":"text/x-python","patch_set":9,"id":"89f721ec_dc1f131e","line":106,"updated":"2026-08-11 15:42:04.000000000","message":"dispatch_cleanup creates AgentAPI() and checks can_send_version(\u00271.1\u0027). AgentAPI hardcodes version_cap\u003dRPC_API_VERSION (\u00271.1\u0027) and Cyborg has no upgrade_levels config. Since cap equals the checked version, can_send_version always returns True. Commit message and release notes claim rolling-upgrad...\n\n**Severity**: HIGH | **Confidence**: 0.9\n\n**Risk**: During rolling upgrade with 1.0 agents, conductor sends cleanup_device (1.1 cast) to old agents. Cast is fire-and-forget so conductor learns nothing. Devices set to PENDING_CLEANING never get cleaned, stuck indefinitely with reserved placement inventory. Release notes upgrade-safety claim is false.\n\n**Priority**: Before merge\n**Why This Matters**: During rolling upgrade with 1.0 agents, conductor sends cleanup_device (1.1 cast) to old agents. Cast is fire-and-forget so conductor learns nothing. Devices set to PENDING_CLEANING never get cleaned, stuck indefinitely with reserved placement inventory. Release notes upgrade-safety claim is false.\n\n**Recommendation**:\nAdd configurable upgrade_levels option feeding version_cap for agent RPC API (like Nova), or remove can_send_version check and document cleanup requires all agents at 1.1+. If keeping the check, the AgentAPI version probe needs externally controllable version_cap.","commit_id":"8dae04dacf3982bae7b61c8fc01621a1349a032e"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"38007cfaa82d2a9787db7ecc68db6604371cd0d6","unresolved":false,"context_lines":[{"line_number":118,"context_line":"        device.save(context)"},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"        agent_api \u003d AgentAPI()"},{"line_number":121,"context_line":"        if not agent_api.client.can_send_version(\u00271.1\u0027):"},{"line_number":122,"context_line":"            LOG.warning("},{"line_number":123,"context_line":"                \u0027Agent for device %s does not support cleanup RPC 1.1 \u0027"},{"line_number":124,"context_line":"                \u0027(cluster pinned below 1.1); setting device to error.\u0027,"}],"source_content_type":"text/x-python","patch_set":10,"id":"711c99d8_83bb148d","line":121,"updated":"2026-08-12 09:19:29.000000000","message":"In ConductorManager.dispatch_cleanup(), agent_api.client.can_send_version(\u00271.1\u0027) checks whether the target agent supports cleanup RPC. However, AgentAPI sets version_cap\u003dself.RPC_API_VERSION which is now 1.1. oslo_messaging can_send_version() checks whether the requested version is \u003c\u003d version_cap...\n\n**Severity**: HIGH | **Confidence**: 0.8\n\n**Risk**: During a rolling upgrade where a 1.1 conductor sends cleanup_device to a 1.0 agent, the RPC call will be dispatched but the 1.0 agent has no cleanup_device endpoint. oslo_messaging will raise a MessageDeliveryFailure or the agent will drop the message. The device will be stuck in PENDING_CLEANING...\n\n**Priority**: Before merge\n**Why This Matters**: During a rolling upgrade where a 1.1 conductor sends cleanup_device to a 1.0 agent, the RPC call will be dispatched but the 1.0 agent has no cleanup_device endpoint. oslo_messaging will raise a MessageDeliveryFailure or the agent will drop the message. The device will be stuck in PENDING_CLEANING...\n\n**Recommendation**:\nTo properly support rolling upgrades, add a configurable version cap (e.g., CONF.upgrade_levels.agent) that operators can pin during upgrades. Alternatively, if rolling upgrade support is not yet needed, remove the can_send_version guard and document that all agents must be upgraded before conductors. Do not claim rolling upgrade support that does not work.","commit_id":"b96255e6998ff1e279540dd1d7f0fc125b9b9d78"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"9bbd1f6d7ae2fbcf7b1589c4f026fab23db81500","unresolved":false,"context_lines":[{"line_number":118,"context_line":"        device.save(context)"},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"        agent_api \u003d AgentAPI()"},{"line_number":121,"context_line":"        if not agent_api.client.can_send_version(\u00271.1\u0027):"},{"line_number":122,"context_line":"            LOG.warning("},{"line_number":123,"context_line":"                \u0027Agent for device %s does not support cleanup RPC 1.1 \u0027"},{"line_number":124,"context_line":"                \u0027(cluster pinned below 1.1); setting device to error.\u0027,"}],"source_content_type":"text/x-python","patch_set":11,"id":"7ebb8aa0_20d6b780","line":121,"updated":"2026-08-13 05:07:27.000000000","message":"The conductor\u0027s dispatch_cleanup method uses agent_api.client.can_send_version(\u00271.1\u0027) to guard against rolling upgrades with old agents. However, AgentAPI.__init__ sets version_cap\u003dself.RPC_API_VERSION which is now \u00271.1\u0027. oslo.messaging\u0027s can_send_version returns True whenever the requested versi...\n\n**Severity**: HIGH | **Confidence**: 0.9\n\n**Risk**: During a rolling upgrade where conductors are upgraded to 1.1 but some agents remain at 1.0, the conductor will attempt to cast cleanup_device to old agents. The old agents cannot handle the 1.1 RPC and will either error or silently drop the message, leaving devices in PENDING_CLEANING state inde...\n\n**Priority**: Before merge\n**Why This Matters**: During a rolling upgrade where conductors are upgraded to 1.1 but some agents remain at 1.0, the conductor will attempt to cast cleanup_device to old agents. The old agents cannot handle the 1.1 RPC and will either error or silently drop the message, leaving devices in PENDING_CLEANING state inde...\n\n**Recommendation**:\nEither add a CONF option (e.g., CONF.agent.agent_rpc_version_cap) that operators can set to \u00271.0\u0027 during upgrades, or remove the can_send_version guard and document that this feature requires all agents to be upgraded to 1.1 first.","commit_id":"6dbda6c47e78c6194618ba19818e87cfca15e8eb"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"c90b9496748c89a06f57986fe181f894b08c14d0","unresolved":false,"context_lines":[{"line_number":94,"context_line":"    def periodic_tasks(self, context, raise_on_error\u003dFalse):"},{"line_number":95,"context_line":"        pass"},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"    def _set_device_placement_reserved(self, context, device, reserved):"},{"line_number":98,"context_line":"        deployables \u003d Deployable.get_list_by_device_id(context, device.id)"},{"line_number":99,"context_line":"        for dep in deployables:"},{"line_number":100,"context_line":"            filters \u003d {\u0027deployable_id\u0027: dep.id, \u0027key\u0027: \u0027rc\u0027}"}],"source_content_type":"text/x-python","patch_set":12,"id":"a33384ca_b3d4b6a5","line":97,"updated":"2026-08-13 14:11:02.000000000","message":"_set_deployable_placement_reserved (ext_arq.py:43), _set_nvme_placement_reserved (agent/manager.py:154) and _set_device_placement_reserved (conductor/manager.py:97) reimplement the same rc-attribute lookup plus update_rp_inventory_reserved call.\n\n**Severity**: WARNING | **Confidence**: 0.9\n\n**Impact**: Fixing the reserved computation (per-ARQ counting required by CF-001/CF-002) must be repeated in three modules that have already diverged in error handling, inviting inconsistent Placement state between bind and cleanup paths.\n\n**Suggestion**:\nExtract a single helper, e.g. PlacementClient.set_deployable_reserved(context, deployable, reserved_count), and call it from bind, dispatch_cleanup and the agent cleanup path.","commit_id":"9a26c304b37f3aa961e47af455e2a323fdc6b3c8"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"2079ec583b30fa46222f73b179f5e3657db6fc16","unresolved":false,"context_lines":[{"line_number":118,"context_line":"        device.save(context)"},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"        agent_api \u003d AgentAPI()"},{"line_number":121,"context_line":"        if not agent_api.client.can_send_version(\u00271.1\u0027):"},{"line_number":122,"context_line":"            LOG.warning("},{"line_number":123,"context_line":"                \u0027Agent for device %s does not support cleanup RPC 1.1 \u0027"},{"line_number":124,"context_line":"                \u0027(cluster pinned below 1.1); setting device to error.\u0027,"}],"source_content_type":"text/x-python","patch_set":14,"id":"2e97c856_a1026083","line":121,"updated":"2026-08-15 16:42:48.000000000","message":"The conductor builds AgentAPI with version_cap\u003dAgentAPI.RPC_API_VERSION (1.1), then calls client.can_send_version(\u00271.1\u0027), which only compares against the local version cap and not remote server versions.\n\n**Severity**: HIGH | **Confidence**: 0.9\n\n**Risk**: During a rolling upgrade, dispatching cleanup_device to a 1.0 agent raises an UnsupportedVersionError from oslo.messaging instead of the graceful skip-to-error path; the device stays in PENDING_CLEANING and its Placement reservation is never released.\n\n**Priority**: Before merge\n**Why This Matters**: During a rolling upgrade, dispatching cleanup_device to a 1.0 agent raises an UnsupportedVersionError from oslo.messaging instead of the graceful skip-to-error path; the device stays in PENDING_CLEANING and its Placement reservation is never released.\n\n**Recommendation**:\nDetermine the remote agent\u0027s version via a versioned call/heartbeat mechanism, or catch oslo_messaging.UnsupportedVersionError (and MessagingTimeout) around cleanup_device and then set the device to ERROR with inventory reserved.","commit_id":"dca8f97b449ebcd2ab171eedf43c7b87f8d23614"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"0eb1d94aa8aa5967d7daac4bea70519ca8f05ee5","unresolved":false,"context_lines":[{"line_number":94,"context_line":"    def periodic_tasks(self, context, raise_on_error\u003dFalse):"},{"line_number":95,"context_line":"        pass"},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"    def _set_device_placement_reserved(self, context, device, reserved):"},{"line_number":98,"context_line":"        deployables \u003d Deployable.get_list_by_device_id(context, device.id)"},{"line_number":99,"context_line":"        for dep in deployables:"},{"line_number":100,"context_line":"            filters \u003d {\u0027deployable_id\u0027: dep.id, \u0027key\u0027: \u0027rc\u0027}"}],"source_content_type":"text/x-python","patch_set":16,"id":"e3533f95_79acbc44","line":97,"updated":"2026-08-18 08:48:35.000000000","message":"The same invariant (reserved \u003d num_accelerators when reserved else 0, keyed off the deployable\u0027s \u0027rc\u0027 attribute) is independently implemented in ext_arq._set_deployable_placement_reserved, ConductorManager._set_device_placement_reserved and AgentManager._set_nvme_placement_reserved. The ext_arq and agent copies catch exceptions; the conductor copy does not, so in the non-cleaning fast path of dispatch_cleanup a Placement failure raises out of the RPC handler after the device was already saved as AVAILABLE, leaving the device AVAILABLE in Cyborg while its inventory stays fully reserved in Placement.\n\n**Severity**: WARNING | **Confidence**: 0.8\n\n**Impact**: Divergent edits to one of the three copies will desynchronize bind-time, conductor-time and agent-time reservation behavior; concretely, a transient Placement outage during the unbind fast path leaves an over-reserved provider and a device Cyborg reports AVAILABLE, starving future allocations of that RC.\n\n**Suggestion**:\nExtract a single shared helper (e.g. on PlacementClient or a common module) taking (context, deployable/device, reserved) and a consistent error policy, and call it from all three sites; at minimum wrap the conductor fast path so a Placement failure does not silently strand a reservation.","commit_id":"d307590ec19d88b771e45c1a7c5efc534de39d7d"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"0eb1d94aa8aa5967d7daac4bea70519ca8f05ee5","unresolved":false,"context_lines":[{"line_number":114,"context_line":"            self._set_device_placement_reserved(context, device, False)"},{"line_number":115,"context_line":"            return"},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"        device.device_state \u003d constants.DEVICE_STATE_PENDING_CLEANING"},{"line_number":118,"context_line":"        device.save(context)"},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"        agent_api \u003d AgentAPI()"}],"source_content_type":"text/x-python","patch_set":16,"id":"f4ca87ea_f4660270","line":117,"updated":"2026-08-18 08:48:35.000000000","message":"dispatch_cleanup saves DEVICE_STATE_PENDING_CLEANING and then fires a one-way cast to the agent; if the cast is lost, rejected, or the agent is down or restarts before finishing, the device stays PENDING_CLEANING (or CLEANING) forever with its inventory fully reserved. _dispatch_device_cleanup in ext_arq swallows every exception with a log line, the agent\u0027s timeout only bounds the driver call after the RPC already arrived, and neither init_host nor any periodic task reconciles devices stuck in intermediate cleaning states.\n\n**Severity**: HIGH | **Confidence**: 0.8\n\n**Impact**: A single lost cast or an agent restart at the wrong moment permanently removes a cleaning-capable device from service (state stuck mid-lifecycle, Placement inventory reserved, so it is neither schedulable nor self-healing) until manual DB surgery. The commit\u0027s own goal that a device \u0027only returns to AVAILABLE after cleanup succeeds\u0027 has no failure-path recovery for the dispatch itself.\n\n**Priority**: Before merge\n**Recommendation**:\nAdd a reconciliation step: on agent init_host, re-drive or fail devices found in CLEANING on that host; in the conductor, either use a call with timeout/retry for dispatch or run a periodic task that moves PENDING_CLEANING/CLEANING devices older than a threshold into ERROR (with inventory kept reserved) so operators get a recoverable signal instead of silent stuck state.","commit_id":"d307590ec19d88b771e45c1a7c5efc534de39d7d"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"aedb3bbafb604b0165582bfc3fcba59e84fee246","unresolved":false,"context_lines":[{"line_number":84,"context_line":"    def periodic_tasks(self, context, raise_on_error\u003dFalse):"},{"line_number":85,"context_line":"        pass"},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"    def dispatch_cleanup(self, context, device):"},{"line_number":88,"context_line":"        if not device.supports_cleaning:"},{"line_number":89,"context_line":"            device.device_state \u003d constants.DEVICE_STATE_AVAILABLE"},{"line_number":90,"context_line":"            device.save(context)"}],"source_content_type":"text/x-python","patch_set":17,"id":"ba440397_2538d12f","line":87,"updated":"2026-08-21 11:59:20.000000000","message":"ConductorAPI.dispatch_cleanup prepares the cast with version\u003d\u00271.1\u0027 (cyborg/conductor/rpcapi.py:140) and the client class documents 1.1, but the server-side ConductorManager was not bumped: cyborg/conductor/manager.py:44 still has RPC_API_VERSION \u003d \u00271.0\u0027 and target \u003d messaging.Target(version\u003d\u00271.0\u0027). oslo_messaging\u0027s dispatcher only routes a message to endpoints whose target version is \u003e\u003d the message version, so a 1.1 cast to a 1.0 conductor endpoint fails dispatch (NoSuchMethodError/MessageDeliveryFailure). For a cast the failure is not propagated to the caller, so dispatch_cleanup silently never executes.\n\n**Severity**: HIGH | **Confidence**: 0.85\n\n**Impact**: In real deployments every unbind with a deployable casts a 1.1 message the conductor rejects: cleanup never dispatched, device_state never set to pending_cleaning, driver cleanup and Placement reserved updates never happen. NVMe devices are never cleaned on unbind, which is exactly the behavior the commit message says it is fixing; the failure is silent because casts are fire-and-forget.\n\n**Priority**: Before merge\n**Recommendation**:\nBump ConductorManager.RPC_API_VERSION to \u00271.1\u0027 (mirroring the AgentManager change) and add a test that exercises the cast against a real oslo_messaging server endpoint or at least asserts the manager target version matches the rpcapi version, so the client/server version skew the mocked unit tests hide cannot recur.","commit_id":"2e8ca0aa1d70e16320e82680d6a524ec5c6e5c17"}],"cyborg/objects/ext_arq.py":[{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"38007cfaa82d2a9787db7ecc68db6604371cd0d6","unresolved":false,"context_lines":[{"line_number":365,"context_line":"        self.deployable_id \u003d None"},{"line_number":366,"context_line":"        self.save(context)"},{"line_number":367,"context_line":""},{"line_number":368,"context_line":"    def _dispatch_device_cleanup(self, context):"},{"line_number":369,"context_line":"        try:"},{"line_number":370,"context_line":"            dep \u003d objects.Deployable.get_by_id(context, self.deployable_id)"},{"line_number":371,"context_line":"            device \u003d objects.Device.get_by_device_id(context, dep.device_id)"}],"source_content_type":"text/x-python","patch_set":10,"id":"fcbe9e77_b3fae9f9","line":368,"updated":"2026-08-12 09:19:29.000000000","message":"_dispatch_device_cleanup() catches all exceptions and only logs them. When the conductor RPC dispatch fails, the unbind proceeds as if it succeeded, marking the ARQ as unbound and clearing deployable_id. The device remains in ALLOCATED state with Placement inventory still reserved, but the system...\n\n**Severity**: HIGH | **Confidence**: 0.8\n\n**Risk**: If the conductor is unreachable or dispatch_cleanup fails (DB error, network issue, conductor crash), the NVMe device is never cleaned but the ARQ is marked unbound. The device remains in ALLOCATED state with inventory reserved, so it will not be reallocated immediately. However, there is no retr...\n\n**Priority**: Before merge\n**Why This Matters**: If the conductor is unreachable or dispatch_cleanup fails (DB error, network issue, conductor crash), the NVMe device is never cleaned but the ARQ is marked unbound. The device remains in ALLOCATED state with inventory reserved, so it will not be reallocated immediately. However, there is no retr...\n\n**Recommendation**:\nAt minimum, set the device to ERROR state when cleanup dispatch fails so it cannot be silently reused. Consider adding a periodic reconciliation task or operator-visible alarm. Alternatively, propagate the failure and leave the ARQ in an error state rather than completing the unbind cleanly.","commit_id":"b96255e6998ff1e279540dd1d7f0fc125b9b9d78"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"c90b9496748c89a06f57986fe181f894b08c14d0","unresolved":false,"context_lines":[{"line_number":293,"context_line":"            self.arq.uuid,"},{"line_number":294,"context_line":"        )"},{"line_number":295,"context_line":""},{"line_number":296,"context_line":"    def bind(self, context, deployable):"},{"line_number":297,"context_line":"        self._allocate_attach_handle(context, deployable)"},{"line_number":298,"context_line":""},{"line_number":299,"context_line":"        device \u003d objects.Device.get_by_device_id(context, deployable.device_id)"}],"source_content_type":"text/x-python","patch_set":12,"id":"8014b894_7813446a","line":296,"updated":"2026-08-13 14:11:02.000000000","message":"ExtARQ.bind now marks the device ALLOCATED and calls _set_deployable_placement_reserved(..., True) for any device whose device_state is not None, and that helper sets Placement reserved \u003d deployable.num_accelerators.\n\n**Severity**: HIGH | **Confidence**: 0.8\n\n**Risk**: After the backfill, the first ARQ bind on any multi-unit FPGA/GPU deployable fully reserves its Placement inventory, so Nova can no longer schedule the remaining units even though attach handles are free; device is also marked ALLOCATED though only one unit is used.\n\n**Priority**: Before merge\n**Why This Matters**: After the backfill, the first ARQ bind on any multi-unit FPGA/GPU deployable fully reserves its Placement inventory, so Nova can no longer schedule the remaining units even though attach handles are free; device is also marked ALLOCATED though only one unit is used.\n\n**Recommendation**:\nReserve one unit per bound ARQ (reserved \u003d number of in-use attach handles, or increment/decrement a per-deployable counter) instead of num_accelerators, and add a unit test binding two ARQs to one multi-unit deployable.","commit_id":"9a26c304b37f3aa961e47af455e2a323fdc6b3c8"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"0dfd5c0a14bbe6c05733e2ab43304cbe8d01b1b7","unresolved":false,"context_lines":[{"line_number":297,"context_line":"        self._allocate_attach_handle(context, deployable)"},{"line_number":298,"context_line":""},{"line_number":299,"context_line":"        device \u003d objects.Device.get_by_device_id(context, deployable.device_id)"},{"line_number":300,"context_line":"        if device.device_state is not None:"},{"line_number":301,"context_line":"            device.device_state \u003d constants.DEVICE_STATE_ALLOCATED"},{"line_number":302,"context_line":"            device.save(context)"},{"line_number":303,"context_line":"            _set_deployable_placement_reserved(context, deployable, True)"}],"source_content_type":"text/x-python","patch_set":13,"id":"2d89f2c3_8e7862ab","line":300,"updated":"2026-08-14 15:51:04.000000000","message":"The new code in ExtARQ.bind() sets device_state to ALLOCATED for any device whose state is not None, including devices in PENDING_CLEANING, CLEANING, or ERROR, so there is no enforcement that only AVAILABLE devices can be bound.\n\n**Severity**: HIGH | **Confidence**: 0.8\n\n**Risk**: A device in ERROR (failed sanitize) or still CLEANING can be bound to a new ARQ, so a partially sanitized NVMe device may be reassigned to another tenant, directly contradicting the security goal of the change.\n\n**Priority**: Before merge\n**Why This Matters**: A device in ERROR (failed sanitize) or still CLEANING can be bound to a new ARQ, so a partially sanitized NVMe device may be reassigned to another tenant, directly contradicting the security goal of the change.\n\n**Recommendation**:\nOnly transition to ALLOCATED when the current device_state is AVAILABLE (raise/409 otherwise), keeping the documented behavior that unusable devices cannot be reallocated.","commit_id":"8e487f0b123aea37d93e8d88b56e12eb6919b8c9"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"0dfd5c0a14bbe6c05733e2ab43304cbe8d01b1b7","unresolved":false,"context_lines":[{"line_number":359,"context_line":"            self._deallocate_attach_handle(context, ah_id, hostname)"},{"line_number":360,"context_line":""},{"line_number":361,"context_line":"        if self.deployable_id:"},{"line_number":362,"context_line":"            self._dispatch_device_cleanup(context)"},{"line_number":363,"context_line":""},{"line_number":364,"context_line":"        self.attach_handle_id \u003d None"},{"line_number":365,"context_line":"        self.deployable_id \u003d None"}],"source_content_type":"text/x-python","patch_set":13,"id":"3aa97109_5b2024c3","line":362,"updated":"2026-08-14 15:51:04.000000000","message":"unbind() dispatches cleanup for every ARQ that has a deployable_id, and each successful cleanup immediately marks the device AVAILABLE and releases the entire Placement reservation, without checking whether other ARQs still hold the device.\n\n**Severity**: HIGH | **Confidence**: 0.8\n\n**Risk**: Sanitization can start on a device still attached to another instance, and Placement capacity is returned to the pool while a consumer still exists, enabling over-allocation or corrupting a live tenant\u0027s device.\n\n**Priority**: Before merge\n**Why This Matters**: Sanitization can start on a device still attached to another instance, and Placement capacity is returned to the pool while a consumer still exists, enabling over-allocation or corrupting a live tenant\u0027s device.\n\n**Recommendation**:\nBefore dispatching, query in-use attach handles / active ARQs for the deployable; only dispatch cleanup and release the reservation on the last unbind (equivalent to reference counting).","commit_id":"8e487f0b123aea37d93e8d88b56e12eb6919b8c9"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"2079ec583b30fa46222f73b179f5e3657db6fc16","unresolved":false,"context_lines":[{"line_number":297,"context_line":"        self._allocate_attach_handle(context, deployable)"},{"line_number":298,"context_line":""},{"line_number":299,"context_line":"        device \u003d objects.Device.get_by_device_id(context, deployable.device_id)"},{"line_number":300,"context_line":"        if device.device_state is not None:"},{"line_number":301,"context_line":"            device.device_state \u003d constants.DEVICE_STATE_ALLOCATED"},{"line_number":302,"context_line":"            device.save(context)"},{"line_number":303,"context_line":"            _set_deployable_placement_reserved(context, deployable, True)"}],"source_content_type":"text/x-python","patch_set":14,"id":"0740ac1c_28094df2","line":300,"updated":"2026-08-15 16:42:48.000000000","message":"ExtARQ.bind unconditionally sets device_state\u003dALLOCATED and reserves dep.num_accelerators in Placement for every device whose device_state is not None, not only devices that support cleaning.\n\n**Severity**: HIGH | **Confidence**: 0.8\n\n**Risk**: Any multi-function device whose deployable exposes num_accelerators \u003e 1 is fully reserved after the first ARQ bind, so concurrent ARQs for the same deployable fail allocation even though attach handles remain free. The fake-driver split in this same commit is a workaround for exactly this defect...\n\n**Priority**: Before merge\n**Why This Matters**: Any multi-function device whose deployable exposes num_accelerators \u003e 1 is fully reserved after the first ARQ bind, so concurrent ARQs for the same deployable fail allocation even though attach handles remain free. The fake-driver split in this same commit is a workaround for exactly this defect...\n\n**Recommendation**:\nGate the bind reservation on device.supports_cleaning (matching dispatch_cleanup), and reserve only the number of units actually bound rather than the deployable total.","commit_id":"dca8f97b449ebcd2ab171eedf43c7b87f8d23614"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"2079ec583b30fa46222f73b179f5e3657db6fc16","unresolved":false,"context_lines":[{"line_number":297,"context_line":"        self._allocate_attach_handle(context, deployable)"},{"line_number":298,"context_line":""},{"line_number":299,"context_line":"        device \u003d objects.Device.get_by_device_id(context, deployable.device_id)"},{"line_number":300,"context_line":"        if device.device_state is not None:"},{"line_number":301,"context_line":"            device.device_state \u003d constants.DEVICE_STATE_ALLOCATED"},{"line_number":302,"context_line":"            device.save(context)"},{"line_number":303,"context_line":"            _set_deployable_placement_reserved(context, deployable, True)"}],"source_content_type":"text/x-python","patch_set":14,"id":"2cdfac02_a0d0dbd3","line":300,"updated":"2026-08-15 16:42:48.000000000","message":"The only state check in bind() is \u0027if device.device_state is not None\u0027; there is no guard that the state is AVAILABLE, so a device left in ERROR or mid-CLEANING can be rebound to another tenant.\n\n**Severity**: HIGH | **Confidence**: 0.8\n\n**Risk**: Defeats the stated safety property: a partially cleaned or error device whose Placement reservation failed to apply can be re-bound, exposing data from the previous tenant.\n\n**Priority**: Before merge\n**Why This Matters**: Defeats the stated safety property: a partially cleaned or error device whose Placement reservation failed to apply can be re-bound, exposing data from the previous tenant.\n\n**Recommendation**:\nRefuse the bind (or skip allocation) when device_state is in ERROR/PENDING_CLEANING/CLEANING, mirroring conductor _NVME_CLEANUP_LOCKED_STATES, and only allocate from AVAILABLE.","commit_id":"dca8f97b449ebcd2ab171eedf43c7b87f8d23614"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"8c527eadc9b5f28d1a902bc572ebcc964e07309a","unresolved":false,"context_lines":[{"line_number":40,"context_line":"LOG \u003d logging.getLogger(__name__)"},{"line_number":41,"context_line":""},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"def _set_deployable_placement_reserved(context, deployable, reserved):"},{"line_number":44,"context_line":"    filters \u003d {\u0027deployable_id\u0027: deployable.id, \u0027key\u0027: \u0027rc\u0027}"},{"line_number":45,"context_line":"    attrs \u003d objects.Attribute.get_by_filter(context, filters)"},{"line_number":46,"context_line":"    if not attrs:"}],"source_content_type":"text/x-python","patch_set":15,"id":"a5ea3f76_626549b8","line":43,"updated":"2026-08-17 08:15:38.000000000","message":"Three near-identical implementations of the Attribute(rc) lookup + update_rp_inventory_reserved update were added: ext_arq._set_deployable_placement_reserved, conductor _set_device_placement_reserved, and agent _set_nvme_placement_reserved. The ext_arq copy also constructs a fresh PlacementClient...\n\n**Severity**: WARNING | **Confidence**: 0.8\n\n**Impact**: Future changes to reserved-inventory semantics must be replicated in three places; divergence would silently produce inconsistent Placement state between bind, unbind, and cleanup paths.\n\n**Suggestion**:\nConsolidate into a single helper (e.g., a PlacementClient method such as set_deployable_inventory_reserved(deployable, reserved)) reused by all three call sites, reusing one PlacementClient instance.","commit_id":"8ded769b2d17779ce2471aa5989d34a23f3fc886"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"8c527eadc9b5f28d1a902bc572ebcc964e07309a","unresolved":false,"context_lines":[{"line_number":365,"context_line":"        self.deployable_id \u003d None"},{"line_number":366,"context_line":"        self.save(context)"},{"line_number":367,"context_line":""},{"line_number":368,"context_line":"    def _dispatch_device_cleanup(self, context):"},{"line_number":369,"context_line":"        try:"},{"line_number":370,"context_line":"            dep \u003d objects.Deployable.get_by_id(context, self.deployable_id)"},{"line_number":371,"context_line":"            device \u003d objects.Device.get_by_device_id(context, dep.device_id)"}],"source_content_type":"text/x-python","patch_set":15,"id":"e240dfde_d69f2ad0","line":368,"updated":"2026-08-17 08:15:38.000000000","message":"_dispatch_device_cleanup catches all exceptions and only logs. If Deployable.get_by_id, Device.get_by_device_id, or the conductor RPC fails, unbind completes and frees the ARQ, but the device stays ALLOCATED with Placement inventory reserved and is never marked ERROR, contradicting the stated fai...\n\n**Severity**: WARNING | **Confidence**: 0.8\n\n**Impact**: A transient DB or RPC failure during unbind leaves a device permanently ALLOCATED/reserved with no error signal beyond a log line; because the ARQ is freed, nothing will ever retry the dispatch and capacity is leaked.\n\n**Suggestion**:\nOn dispatch failure, set the device to ERROR (or at least re-raise to a caller that can mark the ARQ unbind incomplete) and/or enqueue a retry; ensure the reserved inventory state is auditable so operators can detect the leak.","commit_id":"8ded769b2d17779ce2471aa5989d34a23f3fc886"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"d93ab15f098a45b257ce4b51e08687fc58aec90e","unresolved":false,"context_lines":[{"line_number":296,"context_line":"    def bind(self, context, deployable):"},{"line_number":297,"context_line":"        self._allocate_attach_handle(context, deployable)"},{"line_number":298,"context_line":""},{"line_number":299,"context_line":"        device \u003d objects.Device.get_by_device_id(context, deployable.device_id)"},{"line_number":300,"context_line":"        if device.device_state is not None:"},{"line_number":301,"context_line":"            device.device_state \u003d constants.DEVICE_STATE_ALLOCATED"},{"line_number":302,"context_line":"            device.save(context)"}],"source_content_type":"text/x-python","patch_set":16,"id":"191d9304_39f5f9f9","line":299,"updated":"2026-08-18 06:23:02.000000000","message":"ExtARQ.bind() sets device_state\u003dALLOCATED and calls _set_deployable_placement_reserved(context, deployable, True) whenever device_state is not None, with no supports_cleaning check. The helper sets reserved \u003d deployable.num_accelerators (the full total). device_state is backfilled to AVAILABLE for every device (data_migrations.backfill_device_state; conductor/manager.py:290-291), so this applies to GPUs and FPGAs, not just NVMe.\n\n**Severity**: HIGH | **Confidence**: 0.75\n\n**Impact**: After the first ARQ bind on a vGPU/FPGA deployable with N\u003e1 units, reserved\u003dN and total\u003dN leave zero remaining Placement capacity, so subsequent server allocations against that resource provider fail (or Placement rejects the update with 409 because capacity minus reserved is below existing usage). That failure is swallowed by the try/except in _set_deployable_placement_reserved, so even the NVMe safety guarantee can silently not take effect.\n\n**Priority**: Before merge\n**Recommendation**:\nGate the bind-time reservation on device.supports_cleaning (matching the stated intent), and derive reserved from in-use attach handles rather than the full total; at minimum, surface or retry Placement update failures instead of only logging.","commit_id":"d307590ec19d88b771e45c1a7c5efc534de39d7d"},{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"d93ab15f098a45b257ce4b51e08687fc58aec90e","unresolved":false,"context_lines":[{"line_number":365,"context_line":"        self.deployable_id \u003d None"},{"line_number":366,"context_line":"        self.save(context)"},{"line_number":367,"context_line":""},{"line_number":368,"context_line":"    def _dispatch_device_cleanup(self, context):"},{"line_number":369,"context_line":"        try:"},{"line_number":370,"context_line":"            dep \u003d objects.Deployable.get_by_id(context, self.deployable_id)"},{"line_number":371,"context_line":"            device \u003d objects.Device.get_by_device_id(context, dep.device_id)"}],"source_content_type":"text/x-python","patch_set":16,"id":"cdde8070_22078494","line":368,"updated":"2026-08-18 06:23:02.000000000","message":"ExtARQ._dispatch_device_cleanup swallows every exception with LOG.exception, then unbind proceeds and clears deployable_id. Both dispatch_cleanup and cleanup_device are fire-and-forget casts with no result path, so if the cast cannot be delivered (RPC error, conductor not yet at 1.1, agent restart before processing), the device stays ALLOCATED or PENDING_CLEANING/CLEANING with its inventory reserved and nothing ever moves it to a terminal state.\n\n**Severity**: WARNING | **Confidence**: 0.8\n\n**Impact**: A single lost RPC message permanently strands device capacity (reserved inventory, non-terminal device_state) until manual DB/Placement surgery; the safety property advertised in the commit message does not hold on this path.\n\n**Suggestion**:\nOn dispatch failure set device_state\u003dERROR while keeping the reservation instead of swallowing the exception, and add a periodic reconciliation task that moves devices stuck in PENDING_CLEANING/CLEANING beyond cleanup_timeout to ERROR.","commit_id":"d307590ec19d88b771e45c1a7c5efc534de39d7d"}],"cyborg/tests/unit/objects/test_extarq.py":[{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"c90b9496748c89a06f57986fe181f894b08c14d0","unresolved":false,"context_lines":[{"line_number":158,"context_line":"        mock_get_bind_st,"},{"line_number":159,"context_line":"        mock_save,"},{"line_number":160,"context_line":"    ):"},{"line_number":161,"context_line":"        mock_get_dev.return_value \u003d mock.Mock(device_state\u003dNone)"},{"line_number":162,"context_line":"        good_states \u003d constants.ARQ_STATES_TRANSFORM_MATRIX["},{"line_number":163,"context_line":"            constants.ARQ_BIND_STARTED"},{"line_number":164,"context_line":"        ]"}],"source_content_type":"text/x-python","patch_set":12,"id":"293d173c_c9d92252","line":161,"updated":"2026-08-13 14:11:02.000000000","message":"The only test touching the new bind() logic mocks Device.get_by_device_id to return device_state\u003dNone, which skips the added reservation and state-transition branch entirely.\n\n**Severity**: WARNING | **Confidence**: 0.8\n\n**Impact**: The core data-integrity behavior of this change (reserving inventory on bind) ships untested; regressions such as CF-001\u0027s full-inventory reservation would not be caught by the suite.\n\n**Suggestion**:\nAdd a unit test that binds an ARQ against a deployable whose device has non-None device_state and asserts device_state becomes \u0027allocated\u0027 and update_rp_inventory_reserved is called with the expected reserved count.","commit_id":"9a26c304b37f3aa961e47af455e2a323fdc6b3c8"}],"releasenotes/notes/add-cleanup-device-rpc-b2c3d4e5f6a7.yaml":[{"author":{"_account_id":28006,"name":"teim-ci","display_name":"teim-ci","email":"ci@seanmooney.info","username":"ci-sean-mooney","status":"this is a third-party ci account run by sean-k-mooney on irc\nhosted at zuul.teim.app"},"tag":"autogenerated:zuul:automatic-ci","change_message_id":"ef1d573440100e4dd8f26d79074217f826724757","unresolved":false,"context_lines":[{"line_number":7,"context_line":"    the agent hosting the device."},{"line_number":8,"context_line":"upgrade:"},{"line_number":9,"context_line":"  - |"},{"line_number":10,"context_line":"    Agent and conductor RPC API versions are bumped from 1.0 to 1.1."},{"line_number":11,"context_line":"    During a rolling upgrade, conductors at 1.1 will automatically"},{"line_number":12,"context_line":"    fall back to skipping cleanup dispatch when talking to 1.0 agents."},{"line_number":13,"context_line":"  - |"}],"source_content_type":"text/x-yaml","patch_set":5,"id":"187a9215_8e7b5b29","line":10,"updated":"2026-08-07 07:12:11.000000000","message":"The upgrade release note states conductors at 1.1 will automatically fall back to skipping cleanup dispatch when talking to 1.0 agents. No such fallback logic exists in the code. The conductor\u0027s dispatch_cleanup casts to the agent with version\u003d\u00271.1\u0027 without any version check or try/except. During...\n\n**Severity**: HIGH | **Confidence**: 0.9\n\n**Risk**: During a rolling upgrade, NVMe devices that are unbound while the agent is still at 1.0 will be set to pending_cleaning and never recover. The operator will see devices stuck in an intermediate state with no automatic remediation, directly contradicting what the release note promises.\n\n**Priority**: Before merge\n**Why This Matters**: During a rolling upgrade, NVMe devices that are unbound while the agent is still at 1.0 will be set to pending_cleaning and never recover. The operator will see devices stuck in an intermediate state with no automatic remediation, directly contradicting what the release note promises.\n\n**Recommendation**:\nEither implement version-aware fallback in dispatch_cleanup (check the agent RPC version before casting and skip cleanup if \u003c 1.1, possibly using the unused AgentRPCVersionNotSupported exception), or correct the release note to accurately state that both conductor and agent must be upgraded before ARQ unbind is triggered for NVMe devices.","commit_id":"12730005df148167fd60ba7c3e7ea79c90af9cc3"}]}