)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"d17319ac9e1679c6cfc04754f71346e559c29118","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"84739636_293d42b3","updated":"2025-05-05 10:05:15.000000000","message":"I\u0027m +1 on having a document describing a contract between Nova\u0027s OTU feature and an external cleaner script. But it should be focused on what OTU provides to the external script and what Nova expects from the external script, and should not be prescribing what the external script do. \n\nI\u0027m -1 on going into (technical, legal, risk) details of the a hypothetical production ready cleaner script. We don\u0027t provide one. We should just draw a support boundary around Nova and Nova\u0027s OTU feature.\n\nI\u0027m -1 on stating we don\u0027t support statefull PCI passthrough as it is not well defined and in my eyes we do support it for a long time with some handwaving around NIC firmware and GPU PF reset. \n\nI\u0027m -1 on listing not supported lifecycle operations in a way that is OTU specific as the listed limitations are not at all OTU specific.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"e3eac5a7_1d600986","updated":"2025-04-30 14:30:49.000000000","message":"To me, this adds too much detail about a procedure we\u0027re intentionally not supporting. I think we should stick to documenting what *we* expect, which along the lines of:\n\n- You do whatever you want to the device (might not even be cleaning)\n- You return it to the state it was before nova passed it through\n- You unreserve it\n\nThat\u0027s really all we need, IMHO.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"}],"doc/source/admin/pci-passthrough.rst":[{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":745,"context_line":"Cleaning Requirements"},{"line_number":746,"context_line":"~~~~~~~~~~~~~~~~~~~~~"},{"line_number":747,"context_line":""},{"line_number":748,"context_line":"The introduction of One Time Use device does not change the upstream"},{"line_number":749,"context_line":"support for stateful devices. Specifically upstream Nova does not"},{"line_number":750,"context_line":"support PCI Passthrough of stateful pci device or sriov device such"},{"line_number":751,"context_line":"as NVME SSDs. The One Time Use device feature instead provides a way"}],"source_content_type":"text/x-rst","patch_set":1,"id":"c89ad6f3_1d87a0a1","line":748,"range":{"start_line":748,"start_character":33,"end_line":748,"end_character":39},"updated":"2025-04-30 14:30:49.000000000","message":"devices","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":746,"context_line":"~~~~~~~~~~~~~~~~~~~~~"},{"line_number":747,"context_line":""},{"line_number":748,"context_line":"The introduction of One Time Use device does not change the upstream"},{"line_number":749,"context_line":"support for stateful devices. Specifically upstream Nova does not"},{"line_number":750,"context_line":"support PCI Passthrough of stateful pci device or sriov device such"},{"line_number":751,"context_line":"as NVME SSDs. The One Time Use device feature instead provides a way"},{"line_number":752,"context_line":"for a cloud administrator to develop and external system to clean One"},{"line_number":753,"context_line":"Time use devices such that they can be reused for future workloads."}],"source_content_type":"text/x-rst","patch_set":1,"id":"7115c9fe_4566604e","line":750,"range":{"start_line":749,"start_character":57,"end_line":750,"end_character":23},"updated":"2025-04-30 14:30:49.000000000","message":"I don\u0027t think this is a reasonable thing to say. People may read this and think that it doesn\u0027t work, when of course it does. Not everyone is using Nova in a public-facing, multi-tenant, untrusted environment. I assume your point is that Nova doesn\u0027t provide _security guarantees_ or _information isolation_ between users of stateful devices. Something like that would make more sense to me to say here.\n\nI also don\u0027t really like the term \"stateful device\" in this context. The fact that it has state doesn\u0027t really matter so much as the fact that it contains sensitive information.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":747,"context_line":""},{"line_number":748,"context_line":"The introduction of One Time Use device does not change the upstream"},{"line_number":749,"context_line":"support for stateful devices. Specifically upstream Nova does not"},{"line_number":750,"context_line":"support PCI Passthrough of stateful pci device or sriov device such"},{"line_number":751,"context_line":"as NVME SSDs. The One Time Use device feature instead provides a way"},{"line_number":752,"context_line":"for a cloud administrator to develop and external system to clean One"},{"line_number":753,"context_line":"Time use devices such that they can be reused for future workloads."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9423db69_9cbf54b9","line":750,"range":{"start_line":750,"start_character":0,"end_line":750,"end_character":67},"updated":"2025-04-30 14:30:49.000000000","message":"PCI","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":747,"context_line":""},{"line_number":748,"context_line":"The introduction of One Time Use device does not change the upstream"},{"line_number":749,"context_line":"support for stateful devices. Specifically upstream Nova does not"},{"line_number":750,"context_line":"support PCI Passthrough of stateful pci device or sriov device such"},{"line_number":751,"context_line":"as NVME SSDs. The One Time Use device feature instead provides a way"},{"line_number":752,"context_line":"for a cloud administrator to develop and external system to clean One"},{"line_number":753,"context_line":"Time use devices such that they can be reused for future workloads."}],"source_content_type":"text/x-rst","patch_set":1,"id":"a02b82a0_040d5e08","line":750,"range":{"start_line":750,"start_character":50,"end_line":750,"end_character":55},"updated":"2025-04-30 14:30:49.000000000","message":"SRIOV","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"d74732f985acc960d38f256467dada31db063ce1","unresolved":true,"context_lines":[{"line_number":746,"context_line":"~~~~~~~~~~~~~~~~~~~~~"},{"line_number":747,"context_line":""},{"line_number":748,"context_line":"The introduction of One Time Use device does not change the upstream"},{"line_number":749,"context_line":"support for stateful devices. Specifically upstream Nova does not"},{"line_number":750,"context_line":"support PCI Passthrough of stateful pci device or sriov device such"},{"line_number":751,"context_line":"as NVME SSDs. The One Time Use device feature instead provides a way"},{"line_number":752,"context_line":"for a cloud administrator to develop and external system to clean One"},{"line_number":753,"context_line":"Time use devices such that they can be reused for future workloads."}],"source_content_type":"text/x-rst","patch_set":1,"id":"71cfc13a_39d16938","line":750,"range":{"start_line":749,"start_character":57,"end_line":750,"end_character":23},"in_reply_to":"1bb625f5_1671aab8","updated":"2025-05-09 13:09:24.000000000","message":"So it\u0027s not really retroactive\n\nGPU PF has its state reset by qemu and the kernel driver when they are detached from the guest and reattached.\n\nIt\u0027s true that the firmware is not restored to the original version, but i did not consider that to device state.\n\nbefore OTU device was added and even with it now we have never supported a device with intentional provides persistent storage like block storage devices rather then indrectly providign it by havign a writable nrom for the firmware.\n\nif im beign really hosnt im very conlficted because i drectly objected to including the NVME usecase in the spec and explicitly ask that we do not mention it in the docs. i wanted it to be removed entirly form the spec.\n\ni accpated the spec because sylvain said the usecasue are exmples and do not affect what is supproted and dan acknowlaged the comment i.e. meanign the have seen and agree yet im gettign push back on bringin or docs into alignment with that.\n\nim am ok with refarming the docs to express the pre/post condition of cleaning but not how its done and focusing on defining where nova support begins\ni.e. the reservation and release of the device via the allcaitons.\n\nbut i feel like we weren\u0027t agreed on the desgin at all if other thing this is enough to actually fully support devices with persistent storage upstream.\n\nFor Intel Persistent Memory (which also sues namspaces) we fully implemented the device cleaning in nova and cold/live migration supprot.\n\nwe coudl have done that for nvme device too and i know how to make that work today with libvit for nvme device automatically, i.e. supprot the live migration of a vm with it. \n\nhosetly that is the only way i think we shoudl say we supprot nvme passthough in nova.\n\n\nto me OTU device is really only suitbale for the frimeware reset usecase.\n\nwe can provide a much more cloud native, operator and user friendly way of doing nvme pashtough supprot in nova. i think we should do that in the future and it shoudl not be built on any of the OTU supprot.\n\nwe can do that addtivly next cycel or when we have time.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"31b3184c7b662a8c684867cf06821e2c895bc134","unresolved":true,"context_lines":[{"line_number":746,"context_line":"~~~~~~~~~~~~~~~~~~~~~"},{"line_number":747,"context_line":""},{"line_number":748,"context_line":"The introduction of One Time Use device does not change the upstream"},{"line_number":749,"context_line":"support for stateful devices. Specifically upstream Nova does not"},{"line_number":750,"context_line":"support PCI Passthrough of stateful pci device or sriov device such"},{"line_number":751,"context_line":"as NVME SSDs. The One Time Use device feature instead provides a way"},{"line_number":752,"context_line":"for a cloud administrator to develop and external system to clean One"},{"line_number":753,"context_line":"Time use devices such that they can be reused for future workloads."}],"source_content_type":"text/x-rst","patch_set":1,"id":"f9184794_24152309","line":750,"range":{"start_line":749,"start_character":57,"end_line":750,"end_character":23},"in_reply_to":"7115c9fe_4566604e","updated":"2025-04-30 15:43:04.000000000","message":"Well, I can specifically say NVMe device if you prefer.\n\nIt\u0027s partly motivated by the security concern, but also related to this comment from the spec\n\nhttps://review.opendev.org/c/openstack/nova-specs/+/943486/comment/dca0b2a2_965fe7a9/\n\nwhere we agree that OTU device feature would not make NVME device pass-through or other stateful devices supported by Nova.\n\nMy spec approval was based on that understanding since you acknowledged the comment.\n\nSo part of my reason for stating this here is to codify it in the docs for the feature.\n\nSimply refusing to relocate an NVME device after it\u0027s used does not\nmeet the self-service requirement that if I delete a server in Nova\nthat all data associated with the instance is removed. that is a requirement for a single tenant cloud IMO so this is not even about multi tenatncy it about ensure \ndelete actully means delete.\n\ni don\u0027t consider firmware ot be the persistent state when I say stateful device but i acknowledge that it technically is. I was totally fine with OTU devices for that use case.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"d17319ac9e1679c6cfc04754f71346e559c29118","unresolved":true,"context_lines":[{"line_number":746,"context_line":"~~~~~~~~~~~~~~~~~~~~~"},{"line_number":747,"context_line":""},{"line_number":748,"context_line":"The introduction of One Time Use device does not change the upstream"},{"line_number":749,"context_line":"support for stateful devices. Specifically upstream Nova does not"},{"line_number":750,"context_line":"support PCI Passthrough of stateful pci device or sriov device such"},{"line_number":751,"context_line":"as NVME SSDs. The One Time Use device feature instead provides a way"},{"line_number":752,"context_line":"for a cloud administrator to develop and external system to clean One"},{"line_number":753,"context_line":"Time use devices such that they can be reused for future workloads."}],"source_content_type":"text/x-rst","patch_set":1,"id":"1bb625f5_1671aab8","line":750,"range":{"start_line":749,"start_character":57,"end_line":750,"end_character":23},"in_reply_to":"f9184794_24152309","updated":"2025-05-05 10:05:15.000000000","message":"Most of the PCI PF devices we pass through are statefull in some way. Nova before OTU ignored the problem of leaking that state between users. Saying that statefull devices are unsupported basically means direct-physical ports and any GPU PF passthrough got unsupported retroactively regardless of the fact that we advertised those features for a long time and deployers are using them.\n\nWhat happening here in my eyes is that Nova stops ignoring the problem of statefull PCI devices and starts offering a limited solution for the problem in the form of OTU (supported) + external cleaner (not supported, but example provided). If OTU is used for rewriting a NIC firmware or resetting the GPU PF between users that should be in the same support envelope as wiping NVMe disks with OTU.\n\nIn my eyes OTU is fully supported upstream as we added all the necessary bits (code, test coverage, doc). If the deployer decides to use OTU + an external cleaner script to wipe NVMes then the act of wiping is not supported by nova, but creating the OTU reservation is.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":749,"context_line":"support for stateful devices. Specifically upstream Nova does not"},{"line_number":750,"context_line":"support PCI Passthrough of stateful pci device or sriov device such"},{"line_number":751,"context_line":"as NVME SSDs. The One Time Use device feature instead provides a way"},{"line_number":752,"context_line":"for a cloud administrator to develop and external system to clean One"},{"line_number":753,"context_line":"Time use devices such that they can be reused for future workloads."},{"line_number":754,"context_line":""},{"line_number":755,"context_line":"This cleaning can take many form such as asserting the version of the"},{"line_number":756,"context_line":"version of the firmware, erasing device state, filing a ticket in an"}],"source_content_type":"text/x-rst","patch_set":1,"id":"e5fac845_66d4d001","line":753,"range":{"start_line":752,"start_character":66,"end_line":753,"end_character":16},"updated":"2025-04-30 14:30:49.000000000","message":"them","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":752,"context_line":"for a cloud administrator to develop and external system to clean One"},{"line_number":753,"context_line":"Time use devices such that they can be reused for future workloads."},{"line_number":754,"context_line":""},{"line_number":755,"context_line":"This cleaning can take many form such as asserting the version of the"},{"line_number":756,"context_line":"version of the firmware, erasing device state, filing a ticket in an"},{"line_number":757,"context_line":"internal issue tracker for manual replacement of the device."},{"line_number":758,"context_line":"Regardless of what form the cleaning takes there are a common set of"}],"source_content_type":"text/x-rst","patch_set":1,"id":"150faf98_dcc84918","line":755,"range":{"start_line":755,"start_character":14,"end_line":755,"end_character":32},"updated":"2025-04-30 14:30:49.000000000","message":"\"may take many forms\"","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":753,"context_line":"Time use devices such that they can be reused for future workloads."},{"line_number":754,"context_line":""},{"line_number":755,"context_line":"This cleaning can take many form such as asserting the version of the"},{"line_number":756,"context_line":"version of the firmware, erasing device state, filing a ticket in an"},{"line_number":757,"context_line":"internal issue tracker for manual replacement of the device."},{"line_number":758,"context_line":"Regardless of what form the cleaning takes there are a common set of"},{"line_number":759,"context_line":"pre and post conditions that are required to be fulfilled."}],"source_content_type":"text/x-rst","patch_set":1,"id":"834d1463_6718bcb0","line":756,"range":{"start_line":756,"start_character":45,"end_line":756,"end_character":53},"updated":"2025-04-30 14:30:49.000000000","message":"\", or filing\"","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":753,"context_line":"Time use devices such that they can be reused for future workloads."},{"line_number":754,"context_line":""},{"line_number":755,"context_line":"This cleaning can take many form such as asserting the version of the"},{"line_number":756,"context_line":"version of the firmware, erasing device state, filing a ticket in an"},{"line_number":757,"context_line":"internal issue tracker for manual replacement of the device."},{"line_number":758,"context_line":"Regardless of what form the cleaning takes there are a common set of"},{"line_number":759,"context_line":"pre and post conditions that are required to be fulfilled."}],"source_content_type":"text/x-rst","patch_set":1,"id":"297499ab_b9faf005","line":756,"range":{"start_line":756,"start_character":0,"end_line":756,"end_character":10},"updated":"2025-04-30 14:30:49.000000000","message":"this is duplicated","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":767,"context_line":"  the ``HW_PCI_ONE_TIME_USE`` trait."},{"line_number":768,"context_line":"* The placement inventory must have ``reserved``\u003d\u003d``total``"},{"line_number":769,"context_line":"* The placement inventory must not have any allocations"},{"line_number":770,"context_line":"* The device must not be in use on compute node by any instance."},{"line_number":771,"context_line":""},{"line_number":772,"context_line":".. note::"},{"line_number":773,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"e5f548a0_0fac3b7a","line":770,"updated":"2025-04-30 14:30:49.000000000","message":"The last one isn\u0027t really a requirement you can/should assert, I think. If the first three are true, it\u0027s not in use by Nova, so it\u0027s really out of our hands at that point. Something on the host that grabs the device as it comes back to the host could make it \"in use\" by this definition. While, yes, in most cases that will have to be addressed, it really has nothing to do with being eligible for cleaning, it\u0027s more the cleaning process itself. I think you should just remove this.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"31b3184c7b662a8c684867cf06821e2c895bc134","unresolved":true,"context_lines":[{"line_number":767,"context_line":"  the ``HW_PCI_ONE_TIME_USE`` trait."},{"line_number":768,"context_line":"* The placement inventory must have ``reserved``\u003d\u003d``total``"},{"line_number":769,"context_line":"* The placement inventory must not have any allocations"},{"line_number":770,"context_line":"* The device must not be in use on compute node by any instance."},{"line_number":771,"context_line":""},{"line_number":772,"context_line":".. note::"},{"line_number":773,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"d2c733da_8e6a8f15","line":770,"in_reply_to":"e5f548a0_0fac3b7a","updated":"2025-04-30 15:43:04.000000000","message":"i put this here becuase i know that operators (or supprot engieners helpign them) have in the past deleted all the allcotion in placement as part fo a proceudre and then healed them after the fact.\n\nif you dont check that the device is not in use when the allocation isremoved and base the cleanign on just the allocation it could cause data loss or crash the vm.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":777,"context_line":"   to guard against accidental deletion of allocations in placement."},{"line_number":778,"context_line":"   In the case of the libvirt virt driver this can be done by ensuring"},{"line_number":779,"context_line":"   that no libvirt domain or qemu process exist that reference the device"},{"line_number":780,"context_line":"   to be cleaned."},{"line_number":781,"context_line":""},{"line_number":782,"context_line":"Post-Conditions"},{"line_number":783,"context_line":"^^^^^^^^^^^^^^^"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7ce1f5fa_b45e3b51","line":780,"updated":"2025-04-30 14:30:49.000000000","message":"I guess this is related to the last bullet above, but it really doesn\u0027t make sense to me. If the rest of the preconditions are met, why would being in-use by any libvirt domain have any impact on placement allocations?\n\nFurther, I don\u0027t really agree that we should do these checks and/or wait until they\u0027re true. If Nova has dropped the allocation, it thinks the device is not used, at that point, the data should be destroyed, full stop.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"31b3184c7b662a8c684867cf06821e2c895bc134","unresolved":true,"context_lines":[{"line_number":777,"context_line":"   to guard against accidental deletion of allocations in placement."},{"line_number":778,"context_line":"   In the case of the libvirt virt driver this can be done by ensuring"},{"line_number":779,"context_line":"   that no libvirt domain or qemu process exist that reference the device"},{"line_number":780,"context_line":"   to be cleaned."},{"line_number":781,"context_line":""},{"line_number":782,"context_line":"Post-Conditions"},{"line_number":783,"context_line":"^^^^^^^^^^^^^^^"}],"source_content_type":"text/x-rst","patch_set":1,"id":"a15781d7_2f5ee3c0","line":780,"in_reply_to":"7ce1f5fa_b45e3b51","updated":"2025-04-30 15:43:04.000000000","message":"Well i know that people do incorrect things liek docuemnting that we shoudl fix no valid host by deleteing \"stale allocations\" by which they mean allocations related to unconfirmed resizes.\n\ni was trying to be robust to those kinds of incorrect procedures.\nPerhaps that is too much of a guardrail for something we dont support.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":786,"context_line":"a cleaning system will not updated the inventory reserved value to 0."},{"line_number":787,"context_line":""},{"line_number":788,"context_line":"* The pci device must have the same pci address and capabilities"},{"line_number":789,"context_line":"  as it had before it was allocated to an instance."},{"line_number":790,"context_line":"* The pci device must be bound to a valid driver such that it"},{"line_number":791,"context_line":"  can be allocated to a new workload. i.e. ``vfio-pci``"},{"line_number":792,"context_line":"* The device must not be in use by any host level service"}],"source_content_type":"text/x-rst","patch_set":1,"id":"73710a30_b8a93646","line":789,"updated":"2025-04-30 14:30:49.000000000","message":"I\u0027m not sure what you mean here specifically, so please elaborate (for me). Are you trying to say something like \"don\u0027t turn a GPU into a NIC at the same address\" or \"don\u0027t reprogram the device to have a different vendor/model\"?","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"31b3184c7b662a8c684867cf06821e2c895bc134","unresolved":true,"context_lines":[{"line_number":786,"context_line":"a cleaning system will not updated the inventory reserved value to 0."},{"line_number":787,"context_line":""},{"line_number":788,"context_line":"* The pci device must have the same pci address and capabilities"},{"line_number":789,"context_line":"  as it had before it was allocated to an instance."},{"line_number":790,"context_line":"* The pci device must be bound to a valid driver such that it"},{"line_number":791,"context_line":"  can be allocated to a new workload. i.e. ``vfio-pci``"},{"line_number":792,"context_line":"* The device must not be in use by any host level service"}],"source_content_type":"text/x-rst","patch_set":1,"id":"b75d7ad9_4b564747","line":789,"in_reply_to":"73710a30_b8a93646","updated":"2025-04-30 15:43:04.000000000","message":"Reflashing the firmware can change the address and it can change if some device state (set via uefi) can change the pci adress.\n\nWhat im saying is that whatever cleaning you do should not change the actual pci address of the device. if the pci address changes, it may not match the match the pci dev_spec anymore and it will also result in neding to delte and recreate the pci device resouce provider.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":788,"context_line":"* The pci device must have the same pci address and capabilities"},{"line_number":789,"context_line":"  as it had before it was allocated to an instance."},{"line_number":790,"context_line":"* The pci device must be bound to a valid driver such that it"},{"line_number":791,"context_line":"  can be allocated to a new workload. i.e. ``vfio-pci``"},{"line_number":792,"context_line":"* The device must not be in use by any host level service"},{"line_number":793,"context_line":"  i.e. for graphics card there frame buffer should not be in use"},{"line_number":794,"context_line":"  by mesa or any other host level service that could cause allocation"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3567c6ca_a4495440","line":791,"updated":"2025-04-30 14:30:49.000000000","message":"This is not a requirement for passing through (for all devices), and definitely can\u0027t be a requirement for cleaning something that requires type-specific access. So I\u0027m really not sure what you\u0027re getting at here. Are you just trying to say \"has to be returned to a state that PCI-passthrough can pass it through, just like it was at boot\"?","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"31b3184c7b662a8c684867cf06821e2c895bc134","unresolved":true,"context_lines":[{"line_number":788,"context_line":"* The pci device must have the same pci address and capabilities"},{"line_number":789,"context_line":"  as it had before it was allocated to an instance."},{"line_number":790,"context_line":"* The pci device must be bound to a valid driver such that it"},{"line_number":791,"context_line":"  can be allocated to a new workload. i.e. ``vfio-pci``"},{"line_number":792,"context_line":"* The device must not be in use by any host level service"},{"line_number":793,"context_line":"  i.e. for graphics card there frame buffer should not be in use"},{"line_number":794,"context_line":"  by mesa or any other host level service that could cause allocation"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba69d8b2_d20b09d7","line":791,"in_reply_to":"3567c6ca_a4495440","updated":"2025-04-30 15:43:04.000000000","message":"```\nAre you just trying to say \"has to be returned to a state that PCI-passthrough can pass it through, just like it was at boot\"?```\n\nyes while your cleaning you can bind it to anything you want but when its done it shoul be back to the same state as boot.\n\nNormally, we recommend prebinding all devices that will be passed to a guest to vfio-pci so that they are never bound to a normal kernel driver.\n\nThere are a number of reasons for it, one being that having VF bound to the kernel driver can cause higher IRQ load on the host.\n\nlibvirt can rebind the device between the default driver and one that can be presented to qemu, but that\u0027s not always advisable or reliable.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":792,"context_line":"* The device must not be in use by any host level service"},{"line_number":793,"context_line":"  i.e. for graphics card there frame buffer should not be in use"},{"line_number":794,"context_line":"  by mesa or any other host level service that could cause allocation"},{"line_number":795,"context_line":"  to a nova instance to fail."},{"line_number":796,"context_line":"* Any device state that could leak information between tenants"},{"line_number":797,"context_line":"  has been reset."},{"line_number":798,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"1c556cb7_4197125c","line":795,"updated":"2025-04-30 14:30:49.000000000","message":"Again, this is the same as the requirement before and without OTU and cleaning, so I\u0027m not sure this is necessary to say.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"31b3184c7b662a8c684867cf06821e2c895bc134","unresolved":true,"context_lines":[{"line_number":792,"context_line":"* The device must not be in use by any host level service"},{"line_number":793,"context_line":"  i.e. for graphics card there frame buffer should not be in use"},{"line_number":794,"context_line":"  by mesa or any other host level service that could cause allocation"},{"line_number":795,"context_line":"  to a nova instance to fail."},{"line_number":796,"context_line":"* Any device state that could leak information between tenants"},{"line_number":797,"context_line":"  has been reset."},{"line_number":798,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"6196cfe0_59a5d335","line":795,"in_reply_to":"1c556cb7_4197125c","updated":"2025-04-30 15:43:04.000000000","message":"fair, this applies to any past-through device.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"f76818882df93fed69c885dc2cdb52dc0a0a4605","unresolved":true,"context_lines":[{"line_number":795,"context_line":"  to a nova instance to fail."},{"line_number":796,"context_line":"* Any device state that could leak information between tenants"},{"line_number":797,"context_line":"  has been reset."},{"line_number":798,"context_line":""},{"line_number":799,"context_line":"Life cycle limitations"},{"line_number":800,"context_line":"~~~~~~~~~~~~~~~~~~~~~~"},{"line_number":801,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"60f1a923_2b35be1e","line":798,"updated":"2025-04-29 22:47:19.000000000","message":"i also wonder if we should have a warning here or a risk section.\n\nSpecifically, there are 2 obvious risks to not implementing cleaning properly.\n\nThe operator who chooses to automate cleaning is committed to ensuring the pre and post-conditions hold before marking the device as available.\n\nFor some regulatory domains, there may be a legal requirement around\nthe timescale in which the cleaning must be done or an auditability requirement.\n\nObviously there is nothing that Nova can really do in that regard, but there is anonter consideration.\n\nAs a normal tenant, I cannot see if a flavor uses a OTU device.\nin a cloud environment, it\u0027s expected to be abel to endlessly create and delete short lived vms as needed within the bounds of quota.\n\nOTU devices are diffent then every other resource in nova bar ironic nodes in that\nWhen the instance is delete, the device is not immediately available again synchronously.\n\nTo me, this potentially is a limited denial of service vector, since if the cleaning is not done promptly and quickly it may lead to no valid hosts.\n\nA normal use can infer that it\u0027s a OTU device from the guest if they know that Nova does not support NVME device pass-through in general, although if the flavor extra specs are hidden by policy, they can\u0027t know if it cyborg or Nova providing the device.\n\nThat is fine, but im just wondering how we should signal this consideration to the operator, and do we want to provide guidance on how long cleaning should take.\n\ni think the later will depend on the device an how you choose to clean it.\nlike if your going to DD a disk then that purportional to the size of the disk adn the same is likely ture of FPGA ectra. \n\nThe former risk i believe, should be highlighted but in general, i think it\u0027s minor i dont fully know how to phrase the No valid host problem while cleaning is pending but its somewhat out of nova scope.  \n\nwhile the placement triat combined with looking at reserved\u003dtotal allow you a view into the current state, they kind of need ot monitor the cleanign themselves out of band.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"d17319ac9e1679c6cfc04754f71346e559c29118","unresolved":true,"context_lines":[{"line_number":795,"context_line":"  to a nova instance to fail."},{"line_number":796,"context_line":"* Any device state that could leak information between tenants"},{"line_number":797,"context_line":"  has been reset."},{"line_number":798,"context_line":""},{"line_number":799,"context_line":"Life cycle limitations"},{"line_number":800,"context_line":"~~~~~~~~~~~~~~~~~~~~~~"},{"line_number":801,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"59092255_89104cb9","line":798,"in_reply_to":"5f082904_1a483903","updated":"2025-05-05 10:05:15.000000000","message":"\u003e Specifically, there are 2 obvious risks to not implementing cleaning properly.\n\nI think instead we should simply state that upstream nova does not provide support for any external cleaning script (including the example one in tree), the vendor or the deployer bares all support responsibility for that.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":795,"context_line":"  to a nova instance to fail."},{"line_number":796,"context_line":"* Any device state that could leak information between tenants"},{"line_number":797,"context_line":"  has been reset."},{"line_number":798,"context_line":""},{"line_number":799,"context_line":"Life cycle limitations"},{"line_number":800,"context_line":"~~~~~~~~~~~~~~~~~~~~~~"},{"line_number":801,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"5f082904_1a483903","line":798,"in_reply_to":"60f1a923_2b35be1e","updated":"2025-04-30 14:30:49.000000000","message":"To me, you\u0027re getting way too far down the path of prescribing how this can and should be done. I\u0027m not sure I can reconcile this level of prescription with your warnings on L867 below, but I think I\u0027m much more comfortable leaving the legal, regulatory, and discovery concerns to the operators and their lawyers.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"d17319ac9e1679c6cfc04754f71346e559c29118","unresolved":true,"context_lines":[{"line_number":796,"context_line":"* Any device state that could leak information between tenants"},{"line_number":797,"context_line":"  has been reset."},{"line_number":798,"context_line":""},{"line_number":799,"context_line":"Life cycle limitations"},{"line_number":800,"context_line":"~~~~~~~~~~~~~~~~~~~~~~"},{"line_number":801,"context_line":""},{"line_number":802,"context_line":"As the one time use device feature in nova does not extend nova\u0027s"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fd36c9d2_341bdfcf","line":799,"updated":"2025-05-05 10:05:15.000000000","message":"The below list is not OTU device specific, it is totally generic for any PCI passthrough cases. So I would not put it under OTU.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"d74732f985acc960d38f256467dada31db063ce1","unresolved":true,"context_lines":[{"line_number":796,"context_line":"* Any device state that could leak information between tenants"},{"line_number":797,"context_line":"  has been reset."},{"line_number":798,"context_line":""},{"line_number":799,"context_line":"Life cycle limitations"},{"line_number":800,"context_line":"~~~~~~~~~~~~~~~~~~~~~~"},{"line_number":801,"context_line":""},{"line_number":802,"context_line":"As the one time use device feature in nova does not extend nova\u0027s"}],"source_content_type":"text/x-rst","patch_set":1,"id":"1dbd5abd_13601e64","line":799,"in_reply_to":"fd36c9d2_341bdfcf","updated":"2025-05-09 13:09:24.000000000","message":"i dissagree but in general i see how it can be generalised.\n\nfor normal pci pass-through of a stateless device, there is no end user impact to cold migrating for example i.e. 1 QAT crypto accelerator with a given vendor id and product id should be identical to another from the guests perspective.\n\nThe same is true of sriov VF and vdpa to different degrees.\n\nits the inclusion of nvme devices, which were never allowed to be used in the pci devstacp before that changes this somewhat.\n\nfor the new type of VGPUs we actully have the ablity to transfer the state fo the devices vram between host allowing live mgiration.\n\nits also possible to do that with nvme device in qemu just as qemu coudl do it with intel nvdim namespace i.e. PMEM.\n\nOTU device does nto do that transfer, there is more work required to make that function in both the live and cold migration case. we did that work for PMEM device.\n\ni can un-nest this and refine it,  but not all of it is generic.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":800,"context_line":"~~~~~~~~~~~~~~~~~~~~~~"},{"line_number":801,"context_line":""},{"line_number":802,"context_line":"As the one time use device feature in nova does not extend nova\u0027s"},{"line_number":803,"context_line":"pci passthrough feature to support stateful devices there are a number"},{"line_number":804,"context_line":"of life cycle implication to consider if admins choose to use it"},{"line_number":805,"context_line":"with stateful devices such as NVME devices."},{"line_number":806,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"01cb9a3d_c3c5c882","line":803,"range":{"start_line":803,"start_character":3,"end_line":803,"end_character":4},"updated":"2025-04-30 14:30:49.000000000","message":"PCI","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":801,"context_line":""},{"line_number":802,"context_line":"As the one time use device feature in nova does not extend nova\u0027s"},{"line_number":803,"context_line":"pci passthrough feature to support stateful devices there are a number"},{"line_number":804,"context_line":"of life cycle implication to consider if admins choose to use it"},{"line_number":805,"context_line":"with stateful devices such as NVME devices."},{"line_number":806,"context_line":""},{"line_number":807,"context_line":"Cold Migration and Resize"}],"source_content_type":"text/x-rst","patch_set":1,"id":"558b80d0_e1241afd","line":804,"range":{"start_line":804,"start_character":14,"end_line":804,"end_character":25},"updated":"2025-04-30 14:30:49.000000000","message":"implications","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":807,"context_line":"Cold Migration and Resize"},{"line_number":808,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^"},{"line_number":809,"context_line":""},{"line_number":810,"context_line":"As nova is not aware of any state that may exist on OTU pci devices"},{"line_number":811,"context_line":"No attempt is made to copy any state."},{"line_number":812,"context_line":""},{"line_number":813,"context_line":"Shelve, Snapshot and Rebuild"}],"source_content_type":"text/x-rst","patch_set":1,"id":"8e6a0339_ee10e5cd","line":810,"range":{"start_line":810,"start_character":56,"end_line":810,"end_character":59},"updated":"2025-04-30 14:30:49.000000000","message":"PCI","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":813,"context_line":"Shelve, Snapshot and Rebuild"},{"line_number":814,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^"},{"line_number":815,"context_line":""},{"line_number":816,"context_line":"snapshots in nova are only of the root disk."},{"line_number":817,"context_line":"That means that if a vm with OTU devices is shelved"},{"line_number":818,"context_line":"no data state form OTU devices will be preserved in the"},{"line_number":819,"context_line":"shelve snapshot. Conversely, as a snapshot only references"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bf499cf1_412e6f1c","line":816,"range":{"start_line":816,"start_character":0,"end_line":816,"end_character":9},"updated":"2025-04-30 14:30:49.000000000","message":"Snapshots","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":814,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^"},{"line_number":815,"context_line":""},{"line_number":816,"context_line":"snapshots in nova are only of the root disk."},{"line_number":817,"context_line":"That means that if a vm with OTU devices is shelved"},{"line_number":818,"context_line":"no data state form OTU devices will be preserved in the"},{"line_number":819,"context_line":"shelve snapshot. Conversely, as a snapshot only references"},{"line_number":820,"context_line":"the root disk, rebuilding an instance form a snapshot"}],"source_content_type":"text/x-rst","patch_set":1,"id":"8e80d903_04be615e","line":817,"range":{"start_line":817,"start_character":19,"end_line":817,"end_character":23},"updated":"2025-04-30 14:30:49.000000000","message":"\"an instance\"","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":820,"context_line":"the root disk, rebuilding an instance form a snapshot"},{"line_number":821,"context_line":"will not erase persistent state on the OTU device."},{"line_number":822,"context_line":"any non-persistent state such as data in the devices volatile"},{"line_number":823,"context_line":"memory may still be lost as the vm is recreated."},{"line_number":824,"context_line":""},{"line_number":825,"context_line":"Live Migration"},{"line_number":826,"context_line":"^^^^^^^^^^^^^^"}],"source_content_type":"text/x-rst","patch_set":1,"id":"a766556f_ce27eb41","line":823,"updated":"2025-04-30 14:30:49.000000000","message":"I think this is a very verbose way of saying what should be obvious, given there are tons of other things we don\u0027t persist during these operations. I think you\u0027re focused on one very specific use case here (_almost_ as if to try to discourage its implementation) so much that it seems awkward. For the case of something like an NVMe, I suppose it\u0027s possible that an operator thought that passing through a device magically made us preserve its state, but it\u0027s pretty unlikely. However, if you\u0027re using OTU to update firmware on GPUs or other accelerators, this paragraph doesn\u0027t really make sense.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":845,"context_line":"As with all local state any state associated with a OTU device"},{"line_number":846,"context_line":"will not be transferred to the new host when evacuating an instance."},{"line_number":847,"context_line":""},{"line_number":848,"context_line":"Special care must be taking in cleaning utilities to handled cleanup"},{"line_number":849,"context_line":"of OTU devices that were associated with an instance that has been evacuated."},{"line_number":850,"context_line":""},{"line_number":851,"context_line":"* Firstly nova only allow evacuation if the admin forces down a host"}],"source_content_type":"text/x-rst","patch_set":1,"id":"e2225422_202e0c51","line":848,"range":{"start_line":848,"start_character":53,"end_line":848,"end_character":60},"updated":"2025-04-30 14:30:49.000000000","message":"handle","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":845,"context_line":"As with all local state any state associated with a OTU device"},{"line_number":846,"context_line":"will not be transferred to the new host when evacuating an instance."},{"line_number":847,"context_line":""},{"line_number":848,"context_line":"Special care must be taking in cleaning utilities to handled cleanup"},{"line_number":849,"context_line":"of OTU devices that were associated with an instance that has been evacuated."},{"line_number":850,"context_line":""},{"line_number":851,"context_line":"* Firstly nova only allow evacuation if the admin forces down a host"}],"source_content_type":"text/x-rst","patch_set":1,"id":"342d1b80_ee693ca4","line":848,"range":{"start_line":848,"start_character":21,"end_line":848,"end_character":27},"updated":"2025-04-30 14:30:49.000000000","message":"taken","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":848,"context_line":"Special care must be taking in cleaning utilities to handled cleanup"},{"line_number":849,"context_line":"of OTU devices that were associated with an instance that has been evacuated."},{"line_number":850,"context_line":""},{"line_number":851,"context_line":"* Firstly nova only allow evacuation if the admin forces down a host"},{"line_number":852,"context_line":"  or it has missed multiple heartbeats. While this protection exits"},{"line_number":853,"context_line":"  to help prevent data corruption it is still the admins responsibility"},{"line_number":854,"context_line":"  to ensure the instance is not running on the original host when evacuation"}],"source_content_type":"text/x-rst","patch_set":1,"id":"1aabc550_75b6aada","line":851,"range":{"start_line":851,"start_character":20,"end_line":851,"end_character":25},"updated":"2025-04-30 14:30:49.000000000","message":"allows","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":849,"context_line":"of OTU devices that were associated with an instance that has been evacuated."},{"line_number":850,"context_line":""},{"line_number":851,"context_line":"* Firstly nova only allow evacuation if the admin forces down a host"},{"line_number":852,"context_line":"  or it has missed multiple heartbeats. While this protection exits"},{"line_number":853,"context_line":"  to help prevent data corruption it is still the admins responsibility"},{"line_number":854,"context_line":"  to ensure the instance is not running on the original host when evacuation"},{"line_number":855,"context_line":"  is performed."}],"source_content_type":"text/x-rst","patch_set":1,"id":"d31dd176_7eac2792","line":852,"range":{"start_line":852,"start_character":62,"end_line":852,"end_character":67},"updated":"2025-04-30 14:30:49.000000000","message":"exists","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":850,"context_line":""},{"line_number":851,"context_line":"* Firstly nova only allow evacuation if the admin forces down a host"},{"line_number":852,"context_line":"  or it has missed multiple heartbeats. While this protection exits"},{"line_number":853,"context_line":"  to help prevent data corruption it is still the admins responsibility"},{"line_number":854,"context_line":"  to ensure the instance is not running on the original host when evacuation"},{"line_number":855,"context_line":"  is performed."},{"line_number":856,"context_line":"* Second as evacuate is an emergency operation not all action are expected"}],"source_content_type":"text/x-rst","patch_set":1,"id":"81c8b627_749e7abc","line":853,"range":{"start_line":853,"start_character":50,"end_line":853,"end_character":56},"updated":"2025-04-30 14:30:49.000000000","message":"admin\u0027s","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":853,"context_line":"  to help prevent data corruption it is still the admins responsibility"},{"line_number":854,"context_line":"  to ensure the instance is not running on the original host when evacuation"},{"line_number":855,"context_line":"  is performed."},{"line_number":856,"context_line":"* Second as evacuate is an emergency operation not all action are expected"},{"line_number":857,"context_line":"  to have been performed on the failed compute host until the error is"},{"line_number":858,"context_line":"  corrected. For example the allocation associated with the instance"},{"line_number":859,"context_line":"  may not be cleaned up until the original compute host is restored"}],"source_content_type":"text/x-rst","patch_set":1,"id":"f1b8d24b_c522fca3","line":856,"range":{"start_line":856,"start_character":55,"end_line":856,"end_character":61},"updated":"2025-04-30 14:30:49.000000000","message":"actions","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"f76818882df93fed69c885dc2cdb52dc0a0a4605","unresolved":true,"context_lines":[{"line_number":848,"context_line":"Special care must be taking in cleaning utilities to handled cleanup"},{"line_number":849,"context_line":"of OTU devices that were associated with an instance that has been evacuated."},{"line_number":850,"context_line":""},{"line_number":851,"context_line":"* Firstly nova only allow evacuation if the admin forces down a host"},{"line_number":852,"context_line":"  or it has missed multiple heartbeats. While this protection exits"},{"line_number":853,"context_line":"  to help prevent data corruption it is still the admins responsibility"},{"line_number":854,"context_line":"  to ensure the instance is not running on the original host when evacuation"},{"line_number":855,"context_line":"  is performed."},{"line_number":856,"context_line":"* Second as evacuate is an emergency operation not all action are expected"},{"line_number":857,"context_line":"  to have been performed on the failed compute host until the error is"},{"line_number":858,"context_line":"  corrected. For example the allocation associated with the instance"},{"line_number":859,"context_line":"  may not be cleaned up until the original compute host is restored"},{"line_number":860,"context_line":"  to operation. If the instance is deleted while the compute host"},{"line_number":861,"context_line":"  is offline or fenced the cleaning utility may be unable"},{"line_number":862,"context_line":"  to clean the device at that time and therefore must be able to do"},{"line_number":863,"context_line":"  so at a later date. As such any cleaning utility should be robust"},{"line_number":864,"context_line":"  to intermittent failures and should be able to reconcile the host state"},{"line_number":865,"context_line":"  on startup."},{"line_number":866,"context_line":""},{"line_number":867,"context_line":"Example code"},{"line_number":868,"context_line":"~~~~~~~~~~~~"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5f51b225_afda70a4","line":865,"range":{"start_line":851,"start_character":1,"end_line":865,"end_character":13},"updated":"2025-04-29 22:47:19.000000000","message":"this kind of feel like requirement for the cleaning script\n\nperhaps i shoudl ass a Fault tolerance section there and move this to it.\n\nbasically i wanted to express that any production ready clean script need to be tolerant to missing notification and should be able to resolve that over time.\n\nthere are a number of ways to do that such as peroicly checkign for inventoires that shoudl be cleaned but havnt\u0027t been or as i suggest here doing it on start up of the cleanign agent.\n\nthe ohter consideration is that when you are clean a host that has had some isntance evacuated, the cleaning aganet could start before the nova-comptue agent and nova-comptue many not have cleaned up the domain yet.\n\nor the evcaution may have failed and the instance may be restered following a hard-reboot i.e the instance is still on the host.\n\nso the cleaning agent may need to pool libvirt to wait for nova ot delete the domain if it shoudl be deleted before it cleans the device.\n\nwhat it shoudl avoid at all cost is cleanign the device just because someone acindetelly deleted the allcoation as part of fixing the host.\n\nif the instance is still on this host in nova db the compute agent will not delete the domain or clean up the instance directory.\n\nthe cleaning script shoudl really be robost and not clean to early but i feel like fully explaining that in the docs is over kill.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":848,"context_line":"Special care must be taking in cleaning utilities to handled cleanup"},{"line_number":849,"context_line":"of OTU devices that were associated with an instance that has been evacuated."},{"line_number":850,"context_line":""},{"line_number":851,"context_line":"* Firstly nova only allow evacuation if the admin forces down a host"},{"line_number":852,"context_line":"  or it has missed multiple heartbeats. While this protection exits"},{"line_number":853,"context_line":"  to help prevent data corruption it is still the admins responsibility"},{"line_number":854,"context_line":"  to ensure the instance is not running on the original host when evacuation"},{"line_number":855,"context_line":"  is performed."},{"line_number":856,"context_line":"* Second as evacuate is an emergency operation not all action are expected"},{"line_number":857,"context_line":"  to have been performed on the failed compute host until the error is"},{"line_number":858,"context_line":"  corrected. For example the allocation associated with the instance"},{"line_number":859,"context_line":"  may not be cleaned up until the original compute host is restored"},{"line_number":860,"context_line":"  to operation. If the instance is deleted while the compute host"},{"line_number":861,"context_line":"  is offline or fenced the cleaning utility may be unable"},{"line_number":862,"context_line":"  to clean the device at that time and therefore must be able to do"},{"line_number":863,"context_line":"  so at a later date. As such any cleaning utility should be robust"},{"line_number":864,"context_line":"  to intermittent failures and should be able to reconcile the host state"},{"line_number":865,"context_line":"  on startup."},{"line_number":866,"context_line":""},{"line_number":867,"context_line":"Example code"},{"line_number":868,"context_line":"~~~~~~~~~~~~"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9dae8ffb_6fd415b8","line":865,"range":{"start_line":851,"start_character":1,"end_line":865,"end_character":13},"in_reply_to":"5f51b225_afda70a4","updated":"2025-04-30 14:30:49.000000000","message":"Personally, I think explaining all of this is too much detail and confuses the point of \"we don\u0027t support what you do with this\" by specifying \"and here are all the teeny tiny details you should consider\". Something blanket like \"after cleaning the device must be in the same state it was at time zero, ready to be passed through to another instance.\" I think that\u0027s obvious, but if something like that needs to be said, I think that\u0027s enough.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"a4e4ddabc5750cb6b6db51855c0ff401b8743953","unresolved":true,"context_lines":[{"line_number":874,"context_line":"The contrib example demonstrate how to use notifications to"},{"line_number":875,"context_line":"implement the minimum viable cleaning approach and is provided"},{"line_number":876,"context_line":"as is with no implication of support or suitability for purpose"},{"line_number":877,"context_line":"beyond basic testing of the OTU device cleaning workflow."}],"source_content_type":"text/x-rst","patch_set":1,"id":"cab0b3d1_02fee1fb","line":877,"updated":"2025-04-30 14:30:49.000000000","message":"It\u0027s in the contrib/ directory so I don\u0027t think you need to say that it\u0027s not supported for production use. I also don\u0027t think you should document what it does and doesn\u0027t do, because if it (or this doc) evolves, this will either be out of date or we\u0027ll have to update these docs to reflect that. Updating a script in contrib with docs in the main tree feels like it gives far too much \"supported glow\" to something that would otherwise be just an example.\n\nPerhaps I should just trim down the example to be a single-shot thing you can run from cron every 5 minutes, which will make it not look like a serious implementation but still be plenty safe and effective and then we could just dispense with this whole section?","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"31b3184c7b662a8c684867cf06821e2c895bc134","unresolved":true,"context_lines":[{"line_number":874,"context_line":"The contrib example demonstrate how to use notifications to"},{"line_number":875,"context_line":"implement the minimum viable cleaning approach and is provided"},{"line_number":876,"context_line":"as is with no implication of support or suitability for purpose"},{"line_number":877,"context_line":"beyond basic testing of the OTU device cleaning workflow."}],"source_content_type":"text/x-rst","patch_set":1,"id":"f8bc6ab8_704cd668","line":877,"in_reply_to":"cab0b3d1_02fee1fb","updated":"2025-04-30 15:43:04.000000000","message":"we can, how woudl you feel about having a readme.rst in the contrib dir that just say this is not intended for production use.","commit_id":"dcf94fa0d6b89b6ecf217e57437f9d2896ab91c0"}]}