)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":35263,"name":"Matt Crees","email":"mattc@stackhpc.com","username":"mattcrees"},"change_message_id":"2ed47d6ab19425d64112f1fdfcf11ebaebc704bc","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"265fd6b7_2557625a","updated":"2026-02-05 10:45:15.000000000","message":"I\u0027ve dropped the Quobyte fixes from the old patch, as I do not have anywhere to test these are correct.","commit_id":"4468376a8e21fb976e4499924aa21531b2d31845"},{"author":{"_account_id":15197,"name":"Pierre Riteau","email":"pierre@stackhpc.com","username":"priteau","status":"StackHPC"},"change_message_id":"d4a1aaeacb6ca334100cc6f5d5f1486ee890c64d","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"1c0e36a5_ae8122c8","updated":"2026-02-09 14:16:02.000000000","message":"If this can be deployed without impact on running instances, it would be nice to have a `fixes` release note.\n\nIf this cannot be deployed without impact on running instances, we must have an `upgrade` release note.","commit_id":"4468376a8e21fb976e4499924aa21531b2d31845"},{"author":{"_account_id":35263,"name":"Matt Crees","email":"mattc@stackhpc.com","username":"mattcrees"},"change_message_id":"cb8cc1c4a2433a3dd6e1a2d3eb02de0316508f31","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"c8716b1e_036dd353","in_reply_to":"1c0e36a5_ae8122c8","updated":"2026-02-10 13:58:42.000000000","message":"From my testing, this doesn\u0027t affect instances when being deployed. I\u0027ve added the reno","commit_id":"4468376a8e21fb976e4499924aa21531b2d31845"},{"author":{"_account_id":37157,"name":"Robbert Nijgh","display_name":"Robbert Nijgh","email":"devops@rmnijgh.nl","username":"rmnijgh"},"change_message_id":"7e333e02b653d780d13d9a4152c363d075d597e1","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":2,"id":"542f805d_3af0a278","updated":"2026-05-12 14:46:42.000000000","message":"I\u0027ve been using this patch for a while now to solve the mounting problem. However we\u0027re now unable to hard reboot boot-from-volume instances based on nfs volumes.\n\nNova-compute logs say the following after failing to hard reboot, with the error:\n```\n2026-05-12 16:24:07.137 7 ERROR nova.compute.manager [None req-9c592103-ce4c-45f9-b88b-754c39768ba0 ca5a98567fa64021a37883b38e1a4a90 30f77f4ca5504c81aaf294abf87a7b51 - - ac23823215e24239b1f693ed5a5ee2e6 ac23823215e24239b1f693ed5a5ee2e6] [instance: aa2a7be1-d015-4c74-aadd-dc7ed67a89b7] Cannot reboot instance: [Errno 16] Device or resource busy: OSError: [Errno 16] Device or resource busy\n```\n\nMounts in nova-compute container before \u0027Hard reboot\u0027:\n```\n(nova-compute)[root@c4bf1 /]# df -h\nFilesystem                                  Size  Used Avail Use% Mounted on\n10.154.75.77:/nfs_cinder012  3.0T  4.4G  3.0T   1% /var/lib/nova-mnt/68e9dc6dfb003d400795130e225adf40\n```\n\nMounts in nova-compute container after \u0027Hard reboot\u0027 and reset-state:\n```\n(nova-compute)[root@c4bf1 /]# df -h\nFilesystem                                  Size  Used Avail Use% Mounted on\n10.154.75.77:/nfs_cinder012  3.0T  4.4G  3.0T   1% /var/lib/nova/mnt/68e9dc6dfb003d400795130e225adf40\n```\n\nAs you can see it now changes back to nova/mnt after the error, where nova-mnt is to be expected.\n\n-----\nI\u0027ve tried two things to fix this:\n\n1 - Adding path /var/lib/nova-mnt to permissions of the config.json:\n```\n    \"permissions\": [\n        {\n            \"path\": \"/var/lib/nova-mnt\",\n            \"owner\": \"nova:nova\",\n            \"recurse\": true\n        }\n```\nThis unfortunately doesn\u0027t help.\n\n2 - Also changing the mount path of the host to nova-mnt (so both host and container using nova-mnt) like so:\n```\n  - \"{% if enable_shared_var_lib_nova_mnt | bool %}/var/lib/nova-mnt:/var/lib/nova-mnt:shared{% endif %}\"\n```\nThis resolved the problem, and should also still cover the mounting issue from the bug report (This change does require adding the path nova-mnt to permissions like in number 1.).\n\n-----\n\nI\u0027m afraid a change like this would have impact on upgrades for the operators using NFS for instances. Perhaps someone else can reproduce this bug and figure out a better solution to this problem.","commit_id":"004fdd42a83961e0364cc9906412a64f06cec02a"},{"author":{"_account_id":35263,"name":"Matt Crees","email":"mattc@stackhpc.com","username":"mattcrees"},"change_message_id":"2dc589629e45b06fdf342da53b7fbc830ed78b6e","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"d20a4389_fda611eb","updated":"2026-02-17 17:29:11.000000000","message":"recheck","commit_id":"004fdd42a83961e0364cc9906412a64f06cec02a"},{"author":{"_account_id":35263,"name":"Matt Crees","email":"mattc@stackhpc.com","username":"mattcrees"},"change_message_id":"7b2f0d01237d7ec89fdee049036e0efc0b2c88db","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"4bd1fbba_2e232fc1","updated":"2026-02-18 13:26:22.000000000","message":"recheck: POST_FAILURE","commit_id":"004fdd42a83961e0364cc9906412a64f06cec02a"},{"author":{"_account_id":22629,"name":"Michal Nasiadka","email":"mnasiadka@gmail.com","username":"mnasiadka"},"change_message_id":"fd84c1aa287f47393f0afc0585fd5fefe8f32389","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"f294b34c_2777b204","in_reply_to":"3f35d8a0_7e71d100","updated":"2026-07-27 06:39:59.000000000","message":"Acknowledged","commit_id":"004fdd42a83961e0364cc9906412a64f06cec02a"},{"author":{"_account_id":37157,"name":"Robbert Nijgh","display_name":"Robbert Nijgh","email":"devops@rmnijgh.nl","username":"rmnijgh"},"change_message_id":"e7b613b2e5bef6b0f1a9178f51a5d3ab9bc56194","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":2,"id":"3f35d8a0_7e71d100","in_reply_to":"542f805d_3af0a278","updated":"2026-05-12 15:01:16.000000000","message":"This also affects instances with a mounted nfs cinder volume, these also fail with hard reboot. I\u0027ve discovered that after restarting nova_compute the mounts changes aswell:\n\nBefore restarting nova-compute:\n```\n(nova-compute)[root@node5 /]# df -h\nFilesystem                                  Size  Used Avail Use% Mounted on\n10.200.75.87:/nfs_cinder022  3.0T   40M  3.0T   1% /var/lib/nova-mnt/b56b573f24e8fb73c438b1905aea2432\n```\nAfter restarting nova-compute:\n```\n(nova-compute)[root@node5 /]# df -h\nFilesystem                                  Size  Used Avail Use% Mounted on\n10.200.75.87:/nfs_cinder022  3.0T   40M  3.0T   1% /var/lib/nova/mnt/b56b573f24e8fb73c438b1905aea2432\n```","commit_id":"004fdd42a83961e0364cc9906412a64f06cec02a"},{"author":{"_account_id":33263,"name":"Lukasz Chrustek","email":"skidoo@tlen.pl","username":"lukasz"},"change_message_id":"f35255b683649f6ce0227c70b607b98df92beeb1","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"33516d54_43415fa4","updated":"2026-07-15 12:39:11.000000000","message":"Uploaded a new patchset making the mount path symmetric (/var/lib/nova-mnt on both host and container side) and adding a permissions entry for the new path, as suggested by Robbert.\n\nWe validated this variant on a 2025.1 deployment with the NetApp NFS cinder backend: BFV instance boot, hard reboot (no EBUSY), volume attach/detach, start of a pre-existing BFV instance with stale connection_info, live migration of 10 BFV instances, and docker restart nova_compute with zero mount table growth (previously doubling per restart, leading to systemd overload as in bug 2072485). Existing instances keep the old path until live migrated or hard rebooted — covered in the upgrade release note.","commit_id":"b382413d2a6133f4a930f9427fb1f1b021344eda"},{"author":{"_account_id":37203,"name":"Bertrand Lanson","display_name":"Bertrand Lanson","email":"bertrand.lanson@infomaniak.com","username":"lanson","status":"Infomaniak Network SA"},"change_message_id":"e7d5b76a5cc86bad110f1f8b588a39cce1233394","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"d785de57_7be68f14","updated":"2026-07-17 17:44:59.000000000","message":"We do not set the `quobyte_mount_point_base` as per https://docs.openstack.org/nova/latest/configuration/sample-config.html, which will also default to /var/lib/nova/mnt. we should probbly set it in libvirt.conf template if quobyte driver is enabled (`enable_cinder_backend_quobyte | bool`)","commit_id":"b382413d2a6133f4a930f9427fb1f1b021344eda"},{"author":{"_account_id":34034,"name":"Franciszek Przewoźny","display_name":"Franciszek Przewozny","email":"przewozny.franciszek@gmail.com","username":"fprzewozn","status":"OpenStack Engineer / SRE @ Opera Software"},"change_message_id":"7735b5fc7c99167d9e6386f660a713a186e33ec9","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"d86f7c28_e760217d","in_reply_to":"d785de57_7be68f14","updated":"2026-07-21 05:30:35.000000000","message":"Added.","commit_id":"b382413d2a6133f4a930f9427fb1f1b021344eda"},{"author":{"_account_id":34034,"name":"Franciszek Przewoźny","display_name":"Franciszek Przewozny","email":"przewozny.franciszek@gmail.com","username":"fprzewozn","status":"OpenStack Engineer / SRE @ Opera Software"},"change_message_id":"2bc0f7ba256eab84a734a3bb88abfacac804863d","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"2c401e42_54f21f45","updated":"2026-07-21 08:08:26.000000000","message":"recheck","commit_id":"22c8507a6305140c8dfce3c0db127dd1aa01d70e"},{"author":{"_account_id":34034,"name":"Franciszek Przewoźny","display_name":"Franciszek Przewozny","email":"przewozny.franciszek@gmail.com","username":"fprzewozn","status":"OpenStack Engineer / SRE @ Opera Software"},"change_message_id":"5c68ef36266df49eab51d818579e37d28bbca90b","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"8f54d04d_a2288395","updated":"2026-07-22 04:57:08.000000000","message":"recheck - failure during vm scheduling","commit_id":"22c8507a6305140c8dfce3c0db127dd1aa01d70e"},{"author":{"_account_id":34034,"name":"Franciszek Przewoźny","display_name":"Franciszek Przewozny","email":"przewozny.franciszek@gmail.com","username":"fprzewozn","status":"OpenStack Engineer / SRE @ Opera Software"},"change_message_id":"c839af4c7e0676aa3dabc61d11aa55ab6c7ded60","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"8f8d5287_263e2c0e","updated":"2026-07-27 13:21:31.000000000","message":"recheck","commit_id":"eccf82ebe2327c687a2a48e78e7cd7b660aa59a7"},{"author":{"_account_id":34034,"name":"Franciszek Przewoźny","display_name":"Franciszek Przewozny","email":"przewozny.franciszek@gmail.com","username":"fprzewozn","status":"OpenStack Engineer / SRE @ Opera Software"},"change_message_id":"0cefd5133b6830a5717294ae3b8956757b5a81d2","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"dc44f394_f7baa3b9","updated":"2026-07-27 06:27:37.000000000","message":"recheck: unrelated errors","commit_id":"eccf82ebe2327c687a2a48e78e7cd7b660aa59a7"}],"releasenotes/notes/fix-mount-leak-with-nfs-f6410ac7b2c3c4eb.yaml":[{"author":{"_account_id":22629,"name":"Michal Nasiadka","email":"mnasiadka@gmail.com","username":"mnasiadka"},"change_message_id":"5a5cb91b42175b40ba4013dc8bdffecd44f5a14c","unresolved":true,"context_lines":[{"line_number":8,"context_line":"  - |"},{"line_number":9,"context_line":"    The NFS/Quobyte mount point base for nova-compute moved from"},{"line_number":10,"context_line":"    ``/var/lib/nova/mnt`` to ``/var/lib/nova-mnt`` (both on the host and in"},{"line_number":11,"context_line":"    the containers). Running instances keep using the old path until they are"},{"line_number":12,"context_line":"    live migrated or hard rebooted; leftover mounts under the old path can"},{"line_number":13,"context_line":"    then be unmounted."}],"source_content_type":"text/x-yaml","patch_set":4,"id":"2ca4cf97_7360c31e","line":13,"range":{"start_line":11,"start_character":21,"end_line":13,"end_character":22},"updated":"2026-07-22 07:55:08.000000000","message":"I think we need to be more specific that operators need to have a second filesystem under /var/lib/nova-mnt for the migration purposes?\nOr evacuate the host before running deploy with that fix?\nIf yes - should we have a precheck?","commit_id":"22c8507a6305140c8dfce3c0db127dd1aa01d70e"},{"author":{"_account_id":33263,"name":"Lukasz Chrustek","email":"skidoo@tlen.pl","username":"lukasz"},"change_message_id":"0e70c50d0393a0ca4e9ac15e083e6decfeec8f4e","unresolved":false,"context_lines":[{"line_number":8,"context_line":"  - |"},{"line_number":9,"context_line":"    The NFS/Quobyte mount point base for nova-compute moved from"},{"line_number":10,"context_line":"    ``/var/lib/nova/mnt`` to ``/var/lib/nova-mnt`` (both on the host and in"},{"line_number":11,"context_line":"    the containers). Running instances keep using the old path until they are"},{"line_number":12,"context_line":"    live migrated or hard rebooted; leftover mounts under the old path can"},{"line_number":13,"context_line":"    then be unmounted."}],"source_content_type":"text/x-yaml","patch_set":4,"id":"d3ebee23_69748712","line":13,"range":{"start_line":11,"start_character":21,"end_line":13,"end_character":22},"in_reply_to":"2ca4cf97_7360c31e","updated":"2026-07-27 06:18:04.000000000","message":"No second filesystem is needed: /var/lib/nova-mnt is just a directory on the host root filesystem, exactly like /var/lib/nova/mnt was — Docker creates it as the bind mount source on container start. Operators who deliberately placed a dedicated filesystem at /var/lib/nova/mnt can mount the same filesystem at the new path before reconfiguring.\n\nHost evacuation is not required either. We rolled this out on a multi-node 2025.1 PoC deployment (NetApp NFS cinder backend) with running NFS boot-from-volume instances: qemu keeps the volume files open through the old (still mounted) paths, survives the container recreation, and keeps running. New attachments and instance starts use the new base immediately. Instances holding the old path switch on live migration or hard reboot — we verified BFV boot, hard reboot (no EBUSY), volume attach/detach, live migration of 10 BFV instances and nova_compute container restarts with zero mount table growth. Afterwards the leftover mounts can be unmounted (attempts on mounts still in use fail with EBUSY, so cleanup is safe at any time).\n\nExtended the upgrade release note to spell this out. A precheck doesn\u0027t seem necessary given the non-disruptive upgrade path, but happy to add one if you prefer.","commit_id":"22c8507a6305140c8dfce3c0db127dd1aa01d70e"}]}