)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":31652,"name":"Nobuhiro MIKI","email":"nmiki@lycorp.co.jp","username":"nmiki"},"change_message_id":"417c4c2648b5f68b3a2b7b5949529adff19c40fa","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"24014a86_1f1ba82e","updated":"2023-01-12 05:26:12.000000000","message":"Hi sean,\n\nThank you for your review. I am new to touching the objects involved in live migration and their tests. I feel my understanding of them is poor. Any comments would be appreciated.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"c63ed7c4c9fee0acf602cba7243b5f5a658c2bf2","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"c5c80cdd_cce1a811","updated":"2023-10-30 11:02:00.000000000","message":"Nobuhrio, please do let me know if were going to keep working on this, and we can catch up to if I can help in any way.\n\nUnder the assumption you no longer have time to work on this spec, I am hoping to help pick this spec up, and ideally get it implemented and merged this cycle.\n\nI have some small parts of this prototyped. If there are is other work out there I would love to know more, as it could save us all a bunch of time!","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"29c3d18f3198d808d08ae8ecd61b5c7b161c3a78","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"44a03fb5_74e676bc","updated":"2023-01-12 13:58:00.000000000","message":"by the way techniialy the spec freeze was planned for last week.\n\nwe have extended the deadline to today but i dont think this will be ready to merged today.\n\nwe will create the specs repo for the B/2023.2 cycle in about a week so once that is created you can update this to the new location and  proceed with proposing it for the b cycle. you coudl even start on a draft implementaion but i dont think this is somethign that can be completed this cycle.\n\ni just want to set expecation since this cycel is shorter then ususal and we are ~4 week till feature freeze.\n","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"b2af94641b1ea9ac911730941440233b5ff782de","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"98441878_53c528a2","updated":"2023-11-07 13:36:47.000000000","message":"putting a procedural -1 because if someone wants to work on it for Caracal, I don\u0027t want to block here","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"ac52b7a0d2d30bfe5e413a28fe093b9c347932a8","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"61f26523_1002408f","in_reply_to":"03b87796_dd3bbe1e","updated":"2023-11-16 12:47:38.000000000","message":"I believe at the PTG we agreed live-migration isn\u0027t an issue yet, because its currently either not support, or involves a live hot plug then live unplug.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"073ef682c0771c9a5d87e93e4e3b3d0523b69fd5","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"96bff842_49381706","in_reply_to":"0c67a751_59adb83a","updated":"2023-11-07 13:00:12.000000000","message":"Ah, I have just re-read through Sean\u0027s comments. Live-migration being a potential sticking point. At the moment its a hot unplug and then re-plug. So I think its not currently an issue, although there are some features planned that make this harder.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":31652,"name":"Nobuhiro MIKI","email":"nmiki@lycorp.co.jp","username":"nmiki"},"change_message_id":"6dc120e267ab2b0cfc684095ca3b4a6c081268e8","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"5a8a8e60_edf4c70a","in_reply_to":"157cdb9f_a185f19c","updated":"2023-11-21 03:34:10.000000000","message":"Thanks for letting me know that live migration need not be taken care of by this spec. The updated patch looks good for me.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":31652,"name":"Nobuhiro MIKI","email":"nmiki@lycorp.co.jp","username":"nmiki"},"change_message_id":"9aeda2f11ceeb21c0842a81a7f4e8d259109b486","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"fe95e63f_89e8d280","in_reply_to":"44a03fb5_74e676bc","updated":"2023-01-13 07:55:36.000000000","message":"This spec is still under discussion and will be moved to the new\nlocation for the next cycle.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"65233bb816c001598c5d3be960387ba3cfd091b4","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"157cdb9f_a185f19c","in_reply_to":"61f26523_1002408f","updated":"2023-11-16 13:37:07.000000000","message":"yes for neutron sriov ports we do hotplug live migration\n\nfor pci devices requested via the pci_alias or with a cyborg device-profile we block the live migration request at the api.\n\nso we dont have to consier live migration in the context of this spec\n\ngeneric pci deivce live migration woudl be its own spec and not currently planned for this cycle. but it may hapen in the next one depending on how that functionaly matures","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"c06d0e09b2ba26cdee66b07d52c13d8534e1af67","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"0c67a751_59adb83a","in_reply_to":"723b32d7_7c699971","updated":"2023-11-07 12:54:41.000000000","message":"Have you uploaded the existing libvirt XML generation code you have to Gerrit? I would be happy to review that. If you can upload that to gerrit please, I can take a look at testing that for my target use cases.\n\nFor my use case, the existing PCI NUMA affinity is sufficient. Are there specific issues you have with the existing system?\n\nUpgrade is an interesting question... making it opt in via flavour extra specs might be one way forward, although I was hoping to avoid that.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":31652,"name":"Nobuhiro MIKI","email":"nmiki@lycorp.co.jp","username":"nmiki"},"change_message_id":"c7e365e794575299f343d98e3f2bcfd4a7d38f80","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"03b87796_dd3bbe1e","in_reply_to":"96bff842_49381706","updated":"2023-11-08 01:40:50.000000000","message":"I haven\u0027t uploaded the libvirt XML generation code to Gerrit yet. It was written a long time ago. But I will prepare the upload.\n\n\u003e For my use case, the existing PCI NUMA affinity is sufficient. Are there specific issues you have with the existing system?\n\nIt\u0027s not a problem with my existing system. But if there are use cases where it is an issue, it might be a good thing to discuss.\n\nIf this spec is rebased for 2024.1, I would like to restart our discussion on how to address live migration and other planned features I am not aware of, based on the spec rebased for 2024.1.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":31652,"name":"Nobuhiro MIKI","email":"nmiki@lycorp.co.jp","username":"nmiki"},"change_message_id":"c802caccfb571bc862efe9384e8d4114fb0dbed5","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"723b32d7_7c699971","in_reply_to":"c5c80cdd_cce1a811","updated":"2023-11-01 03:53:50.000000000","message":"Hi John, thanks for your comment.\n\nAs you may already know, this spec still had the following challenges. I do not have a proposal to solve them.\n\n* Scheduling that cares about host numa topology and virtual numa topology of PCI devices\n* Upgrading from the old release\n\nSo I would be very grateful if you could work on this spec.\nBased on your prototype, it might be a good idea to send the spec again with another change.\nI only have the prototype of the Libvirt XML generation, but I can cooperate with you in development and review.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"c4594fd82948fcd6ba374f0919d06a91884e8409","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"b1c2fd64_f4f6600c","updated":"2023-12-06 13:16:00.000000000","message":"It is a well written spec. My only concern is breaking an existing guest with a guest reboot by \"moving\" the PCI devs around from the guest perspective. But I\u0027m not sure how big this problem is","commit_id":"8a8db66caf4552aa5cbfc948e9962b1969de3951"}],"specs/2023.1/approved/libvirt-pxb-support.rst":[{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"b2af94641b1ea9ac911730941440233b5ff782de","unresolved":true,"context_lines":[{"line_number":5,"context_line":" http://creativecommons.org/licenses/by/3.0/legalcode"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":8,"context_line":"Add PXB support for Libvirt"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/libvirt-pxb-support"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5c1374bc_13dae6ac","line":8,"updated":"2023-11-07 13:36:47.000000000","message":"FWIW, we agreed on a direction during the PTG so if you want, please rebase this spec to 2024.1 so we can start to look at it.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"2214836592bbd16cd6b8002d2d30c987af7abf96","unresolved":true,"context_lines":[{"line_number":22,"context_line":"sysfs returns -1 [1]_ to indicate unknown::"},{"line_number":23,"context_line":""},{"line_number":24,"context_line":"  root@guest:~# cat /sys/class/net/enp2s0/device/numa_node"},{"line_number":25,"context_line":"  -1"},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"Use Cases"},{"line_number":28,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"20c01b57_02833b0d","line":25,"updated":"2023-01-09 21:12:56.000000000","message":"yes this is a well know limitaion","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"2214836592bbd16cd6b8002d2d30c987af7abf96","unresolved":true,"context_lines":[{"line_number":142,"context_line":"#. The index 0 is reserved for Root Bus"},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"Since Nova already has a parser for Libvirt Guest Config, it can satisfy these"},{"line_number":145,"context_line":"by obtaining the largest index in use."},{"line_number":146,"context_line":""},{"line_number":147,"context_line":"Alternatives"},{"line_number":148,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ce95eae4_7856ddd1","line":145,"updated":"2023-01-09 21:12:56.000000000","message":"we have discsused doing this in the past but this spec glosses over the work requried to supprot this proeprly.\n\nhttps://bugzilla.redhat.com/show_bug.cgi?id\u003d1762119\nhttps://bugzilla.redhat.com/show_bug.cgi?id\u003d1626130\n\nits been 3 years or more since i looked at this in detail but to day nova does not generate teh pci adresses of any of the devices.\n\nwe delegate the virtual pci device adress asignment to libvirt.\n\nif we were to make this change we would need to ensure that vms booted on a non upgraded host could be live migrated to the upgraded host with this feature.\n\nthat vm would have to continue to be able to attach/detach interface/volumes without rebooting \n\nthere is also an upgrade question with regards to is it ever ok to take an existing vm and move ti to the new pci toplogy as that may break the guest workload.\n\ngoing to the pxb approch will change the pci adresses of device in the guest which may break applciation in the guest that use dpdk or otherwise had a depency on the device adress.\n\non one hand that is ok because nova does not provide a stabel pci adress guarentee but we also do not break it intentionally.\n\nso we need to consider if its ok for a vm to hard reboot and change form the old to new pci toplogy.\n\n\ni personally do not like the idea of having two was of doing this so i would want this new toplogy  to apploy to all numa vms and ideally all vms but again we need to consider the upgrade impact.\n\ni stongly suspect there might be some impact on live migration in general too in terms of what data we need to pass int eh migrate_data object to generate the correct destination xml.\n\nfor example we support transparent live migration swith macvtap sriov vf attachted to guest numa 0\n\nwehn we live migrate form a host with the sriov nic on hsot numa 0 to a host were the pci device is on host numa 1 we would not want to change the virtual guest numa toplogy in this case.\n\n\nso we would need to enaure that gust  numa 0 is now mapped to host numa 1\n\nfor a singel numa node guest that is simple but for a multi numa ndoe guest that is more complex.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"cf1363931a72046661c395cdaa817ea7959a7dcb","unresolved":true,"context_lines":[{"line_number":142,"context_line":"#. The index 0 is reserved for Root Bus"},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"Since Nova already has a parser for Libvirt Guest Config, it can satisfy these"},{"line_number":145,"context_line":"by obtaining the largest index in use."},{"line_number":146,"context_line":""},{"line_number":147,"context_line":"Alternatives"},{"line_number":148,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"c10ca147_8e31af6e","line":145,"in_reply_to":"9cea4ab7_5651a995","updated":"2023-01-12 13:55:29.000000000","message":"yes you are correct about the allowed and denied case however that behavior will need to be enforced in the numa toplogy filter before a host is selected.\n\nim not entirly sure fi we need to include the guest numa node info in the migate_data object or if relaying on the source vm xml is sufficent since the intent is to not change the guest toplogy.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":31652,"name":"Nobuhiro MIKI","email":"nmiki@lycorp.co.jp","username":"nmiki"},"change_message_id":"9aeda2f11ceeb21c0842a81a7f4e8d259109b486","unresolved":true,"context_lines":[{"line_number":142,"context_line":"#. The index 0 is reserved for Root Bus"},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"Since Nova already has a parser for Libvirt Guest Config, it can satisfy these"},{"line_number":145,"context_line":"by obtaining the largest index in use."},{"line_number":146,"context_line":""},{"line_number":147,"context_line":"Alternatives"},{"line_number":148,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"185a30ce_bfb5f2be","line":145,"in_reply_to":"c10ca147_8e31af6e","updated":"2023-01-13 07:55:36.000000000","message":"Well, yeah, I think Nova needs to select hosts considering Virtual NUMA Topology by using the numa topology filter.\n\nI see, I can restore the virtual numa topology without migrate_data by referencing the source VM XML.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":31652,"name":"Nobuhiro MIKI","email":"nmiki@lycorp.co.jp","username":"nmiki"},"change_message_id":"417c4c2648b5f68b3a2b7b5949529adff19c40fa","unresolved":true,"context_lines":[{"line_number":142,"context_line":"#. The index 0 is reserved for Root Bus"},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"Since Nova already has a parser for Libvirt Guest Config, it can satisfy these"},{"line_number":145,"context_line":"by obtaining the largest index in use."},{"line_number":146,"context_line":""},{"line_number":147,"context_line":"Alternatives"},{"line_number":148,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9cea4ab7_5651a995","line":145,"in_reply_to":"ce95eae4_7856ddd1","updated":"2023-01-12 05:26:12.000000000","message":"Thanks for sharing the bugzilla URLs.\nI understand that there are requests for enhancements (RFE), but it\u0027s not been implemented yet. As you said, to do this, we need to generate a Virtual PCIe Address and index in Nova instead of Libvirt. As for attaching/detaching interfaces/volumes, I think generating virtual PCIe addresses and indexes is closed in a host. So the uniqueness of these values can be achieved relatively easily.\n\n\u003e so we need to consider if its ok for a vm to hard reboot and change form the old to new pci toplogy.\n\nYes. I think so too.\nPCIe devices on existing numa VMs are not linked with any PXB, so I think these devices can be dealt with as non-numa PCIe devices even if the device is linked with a host numa. And PCIe devices on new numa VMs are linked with a PXB. This approach may not need to renumber the virtual PCIe addresses.\n \n\u003e for example we support transparent live migration swith macvtap sriov vf attachted to guest numa 0\n\u003e \n\u003e wehn we live migrate form a host with the sriov nic on hsot numa 0 to a host were the pci device is on host numa 1 we would not want to change the virtual guest numa toplogy in this case.\n\u003e \n\u003e \n\u003e so we would need to enaure that gust  numa 0 is now mapped to host numa 1\n\u003e \n\u003e for a singel numa node guest that is simple but for a multi numa ndoe guest that is more complex.\n\nIf a guest has multi numa nodes, the PCIe devices on the same virtual guest numa must be linked with the same host numa on both source and destination host like below:\n\nAllowed cases:\n- device1 (guest numa0, host numa0) and device2 (guest numa0, host numa0) can be mapped to device1 (guest numa0, host numa 1) and device2(guest numa0, host numa1).\n\nDenied cases:\n- device1 (guest numa0, host numa0) and device2 (guest numa0, host numa0) can **NOT** be mapped to device1 (guest numa0, host numa 1) and device2(guest numa0, host numa0). This is because these two devices are linked with the same PXB on the source host, but these devices are linked with different PXBs on the destination host.\n\nIn light of this, I feel that Nova need to have a virtual PCIe address and\na virtual guest numa ID for each PCIe device and include it in migrate_data.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"2214836592bbd16cd6b8002d2d30c987af7abf96","unresolved":true,"context_lines":[{"line_number":152,"context_line":"Data model impact"},{"line_number":153,"context_line":"-----------------"},{"line_number":154,"context_line":""},{"line_number":155,"context_line":"None."},{"line_number":156,"context_line":""},{"line_number":157,"context_line":"REST API impact"},{"line_number":158,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"1fa78f3a_d5d9910d","line":155,"range":{"start_line":155,"start_character":0,"end_line":155,"end_character":4},"updated":"2023-01-09 21:12:56.000000000","message":"there may be datamodle impact if we need to pass addtional information for live migrate.\n\n\nwe may not need to do that but its not entrily cleare if we will have enough info in the exsitng source vm xml and live migrate_data object to ensure the gust numa toplogy is not altereed and the host numa affintiy is correct.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":31652,"name":"Nobuhiro MIKI","email":"nmiki@lycorp.co.jp","username":"nmiki"},"change_message_id":"417c4c2648b5f68b3a2b7b5949529adff19c40fa","unresolved":true,"context_lines":[{"line_number":152,"context_line":"Data model impact"},{"line_number":153,"context_line":"-----------------"},{"line_number":154,"context_line":""},{"line_number":155,"context_line":"None."},{"line_number":156,"context_line":""},{"line_number":157,"context_line":"REST API impact"},{"line_number":158,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"e35ec3fe_cd00342d","line":155,"range":{"start_line":155,"start_character":0,"end_line":155,"end_character":4},"in_reply_to":"1fa78f3a_d5d9910d","updated":"2023-01-12 05:26:12.000000000","message":"It may be necessary to include the virtual guest numa ID and virtual PCIe address in the VIFMigrateData and check if that PCI topology can be restored on the destination side.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"cf1363931a72046661c395cdaa817ea7959a7dcb","unresolved":true,"context_lines":[{"line_number":152,"context_line":"Data model impact"},{"line_number":153,"context_line":"-----------------"},{"line_number":154,"context_line":""},{"line_number":155,"context_line":"None."},{"line_number":156,"context_line":""},{"line_number":157,"context_line":"REST API impact"},{"line_number":158,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"e4736110_80bc77fd","line":155,"range":{"start_line":155,"start_character":0,"end_line":155,"end_character":4},"in_reply_to":"e35ec3fe_cd00342d","updated":"2023-01-12 13:55:29.000000000","message":"perhaps but as i note above we may be able to just look at the source xml to determin the virtual numa node affinity of any device.\n\ni think the schduling change is likely to be the larger problem.\n\nbasically right now if you ask for a 2 numa node guest and a pci device we just enforce that the pci device on a destination host alines with one of the 2 virtual numa nodes. that would have to be changed so that the affinity in the guest does not change during a live migration. the other thing is the way we currently map guest numa ndoes to host numa nodes always takes the first numa node the meets the requirements.\n\nin other words we first try to find a host numa node for guest numa node 0 then guest numa nodw 1 ectra.\n\nthat process will need to now ensure we preserve the virtual guest numa node affintiy and preserve teh toplogy instead of assuming its ok for the pci device to misaling on live migration.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":31652,"name":"Nobuhiro MIKI","email":"nmiki@lycorp.co.jp","username":"nmiki"},"change_message_id":"9aeda2f11ceeb21c0842a81a7f4e8d259109b486","unresolved":true,"context_lines":[{"line_number":152,"context_line":"Data model impact"},{"line_number":153,"context_line":"-----------------"},{"line_number":154,"context_line":""},{"line_number":155,"context_line":"None."},{"line_number":156,"context_line":""},{"line_number":157,"context_line":"REST API impact"},{"line_number":158,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"997b64b1_b864890e","line":155,"range":{"start_line":155,"start_character":0,"end_line":155,"end_character":4},"in_reply_to":"e4736110_80bc77fd","updated":"2023-01-13 07:55:36.000000000","message":"Simply checking for the presence of a PCIe device on either virtual numa\nnode 0,1 appears insufficient. Instead, it may be necessary to test in\nthe numa topology filter to see if the virtual numa topology can be\nconstructed for at least one of the following combinations:\n\n- guest numa 0 with PCI device is mapped to host numa 0. and guest numa 1 is mapped to host numa 1.\n- guest numa 0 with PCI device is mapped to host numa 1. and guest numa 1 is mapped to host numa 0.\n\nThanks for the advice.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"2214836592bbd16cd6b8002d2d30c987af7abf96","unresolved":true,"context_lines":[{"line_number":177,"context_line":"Performance Impact"},{"line_number":178,"context_line":"------------------"},{"line_number":179,"context_line":""},{"line_number":180,"context_line":"None."},{"line_number":181,"context_line":""},{"line_number":182,"context_line":"Other deployer impact"},{"line_number":183,"context_line":"---------------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"921f3efc_4293ff4c","line":180,"range":{"start_line":180,"start_character":0,"end_line":180,"end_character":4},"updated":"2023-01-09 21:12:56.000000000","message":"in princiapl this shoudl imporve the performance of multi numa node guest\n\ntoday since the numa info is not avaiable in teh guest one of two behaivor are assumed by applciation in the guest.\n\nsome treat -1 as meaning there is equal disstance form any cpu to the device\nother applcaitons treat -1 as the same as numa node 0\n\n\nin both cases this can lead to cross nuam traffic where interups are pinned to the wrong virtual guest numa node causeing interpers to always cross numa.\n\n\n\nif this was implemnted then in principal the performance for sriov uscase could impove significantly.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":31652,"name":"Nobuhiro MIKI","email":"nmiki@lycorp.co.jp","username":"nmiki"},"change_message_id":"417c4c2648b5f68b3a2b7b5949529adff19c40fa","unresolved":true,"context_lines":[{"line_number":177,"context_line":"Performance Impact"},{"line_number":178,"context_line":"------------------"},{"line_number":179,"context_line":""},{"line_number":180,"context_line":"None."},{"line_number":181,"context_line":""},{"line_number":182,"context_line":"Other deployer impact"},{"line_number":183,"context_line":"---------------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"215e999e_4532af8a","line":180,"range":{"start_line":180,"start_character":0,"end_line":180,"end_character":4},"in_reply_to":"921f3efc_4293ff4c","updated":"2023-01-12 05:26:12.000000000","message":"It\u0027s exactly the motivation of this spec.\nI\u0027ll add here.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"2214836592bbd16cd6b8002d2d30c987af7abf96","unresolved":true,"context_lines":[{"line_number":192,"context_line":"Upgrade impact"},{"line_number":193,"context_line":"--------------"},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"None."},{"line_number":196,"context_line":""},{"line_number":197,"context_line":"Implementation"},{"line_number":198,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"20315732_398f8801","line":195,"range":{"start_line":195,"start_character":0,"end_line":195,"end_character":4},"updated":"2023-01-09 21:12:56.000000000","message":"there are deffintly upgaded consideration here","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":31652,"name":"Nobuhiro MIKI","email":"nmiki@lycorp.co.jp","username":"nmiki"},"change_message_id":"417c4c2648b5f68b3a2b7b5949529adff19c40fa","unresolved":true,"context_lines":[{"line_number":192,"context_line":"Upgrade impact"},{"line_number":193,"context_line":"--------------"},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"None."},{"line_number":196,"context_line":""},{"line_number":197,"context_line":"Implementation"},{"line_number":198,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"4c93bd78_8eb4f11d","line":195,"range":{"start_line":195,"start_character":0,"end_line":195,"end_character":4},"in_reply_to":"20315732_398f8801","updated":"2023-01-12 05:26:12.000000000","message":"Sorry, I haven\u0027t considered this yet. I would first like to consider\nwhat data structure changes are necessary.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"2214836592bbd16cd6b8002d2d30c987af7abf96","unresolved":true,"context_lines":[{"line_number":234,"context_line":"Testing"},{"line_number":235,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":236,"context_line":""},{"line_number":237,"context_line":"Add the following unit tests:"},{"line_number":238,"context_line":""},{"line_number":239,"context_line":"* Check that PXB is attached for each NUMA node"},{"line_number":240,"context_line":"* Check that all indexes are unique"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7b9168ea_b17ec4fa","line":237,"updated":"2023-01-09 21:12:56.000000000","message":"unit tests are not sufficent\n\nyou would need comprehensicve funtional test coverage for move operations and we would ideaaly need to ensure that we have grenade coverage for upgrades or simulate this isn functional tests too.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"cf1363931a72046661c395cdaa817ea7959a7dcb","unresolved":true,"context_lines":[{"line_number":234,"context_line":"Testing"},{"line_number":235,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":236,"context_line":""},{"line_number":237,"context_line":"Add the following unit tests:"},{"line_number":238,"context_line":""},{"line_number":239,"context_line":"* Check that PXB is attached for each NUMA node"},{"line_number":240,"context_line":"* Check that all indexes are unique"}],"source_content_type":"text/x-rst","patch_set":1,"id":"42a4dbc6_518c7681","line":237,"in_reply_to":"08750cdc_0a56bee6","updated":"2023-01-12 13:55:29.000000000","message":"grenade is how we test upgrades today however we cant actully test pci passhtouh in the firt party ci.\n\nintel and melonox used to provide some level of pci/sriov testign but i dont think either tested live migration in there ci.\n\nthey used to have some limited cold migration testing.\n\nfunctional test are likely the best we can do in the first party ci.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":31652,"name":"Nobuhiro MIKI","email":"nmiki@lycorp.co.jp","username":"nmiki"},"change_message_id":"9aeda2f11ceeb21c0842a81a7f4e8d259109b486","unresolved":true,"context_lines":[{"line_number":234,"context_line":"Testing"},{"line_number":235,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":236,"context_line":""},{"line_number":237,"context_line":"Add the following unit tests:"},{"line_number":238,"context_line":""},{"line_number":239,"context_line":"* Check that PXB is attached for each NUMA node"},{"line_number":240,"context_line":"* Check that all indexes are unique"}],"source_content_type":"text/x-rst","patch_set":1,"id":"affc431e_5a23d737","line":237,"in_reply_to":"42a4dbc6_518c7681","updated":"2023-01-13 07:55:36.000000000","message":"Sure. I\u0027ll add functional tests.","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"},{"author":{"_account_id":31652,"name":"Nobuhiro MIKI","email":"nmiki@lycorp.co.jp","username":"nmiki"},"change_message_id":"417c4c2648b5f68b3a2b7b5949529adff19c40fa","unresolved":true,"context_lines":[{"line_number":234,"context_line":"Testing"},{"line_number":235,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":236,"context_line":""},{"line_number":237,"context_line":"Add the following unit tests:"},{"line_number":238,"context_line":""},{"line_number":239,"context_line":"* Check that PXB is attached for each NUMA node"},{"line_number":240,"context_line":"* Check that all indexes are unique"}],"source_content_type":"text/x-rst","patch_set":1,"id":"08750cdc_0a56bee6","line":237,"in_reply_to":"7b9168ea_b17ec4fa","updated":"2023-01-12 05:26:12.000000000","message":"OK. I\u0027ll consider adding functional tests for live migrations.\nI\u0027m not familiar with Grenade [1], but I feel it\u0027s useful for testing upgrades.\n\n[1] https://opendev.org/openstack/grenade","commit_id":"a688ae9394af495d22aee91fe59ac10a9d565f18"}],"specs/2024.1/approved/libvirt-pxb-support.rst":[{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"b773ec37a74773fdc77e098f16ea16e024d46226","unresolved":true,"context_lines":[{"line_number":38,"context_line":""},{"line_number":39,"context_line":"For VMs that do not have any NUMA pinned CPUs or memory, there will be"},{"line_number":40,"context_line":"no change to how PCI devices attached. They will remain as attached"},{"line_number":41,"context_line":"directly to the root PCI device, and appear within NUMA node -1."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"For VMs with CPUs pinned to specific NUMA nodes, we need to ensure PCI"},{"line_number":44,"context_line":"devices are attached to a guest NUMA node that matches the CPU and"}],"source_content_type":"text/x-rst","patch_set":4,"id":"f72e3f12_febd9f20","line":41,"updated":"2023-12-06 13:25:21.000000000","message":"ok in that specific case we could report it as numa node 0 but leaving it as -1 is also valid so im ok with this","commit_id":"8a8db66caf4552aa5cbfc948e9962b1969de3951"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"b773ec37a74773fdc77e098f16ea16e024d46226","unresolved":true,"context_lines":[{"line_number":42,"context_line":""},{"line_number":43,"context_line":"For VMs with CPUs pinned to specific NUMA nodes, we need to ensure PCI"},{"line_number":44,"context_line":"devices are attached to a guest NUMA node that matches the CPU and"},{"line_number":45,"context_line":"memory that is phyiscally on the same NUMA node as the PCI devices."},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"When a PCI device is attached to a VM that is not assigned any CPU or"},{"line_number":48,"context_line":"RAM from the same physical NUMA node, we fallback to attaching devices"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5b43de41_c5b306b9","line":45,"updated":"2023-12-06 13:25:21.000000000","message":"this is already done today unless you disable it by setting the prefer numa affintiy policy so what is actully changing is in this case we will attach the device to the numa specific pbx.\n\nor if legacy is used and the pci device on the host does not have a numa affinity reported the we also dont enforce affinity.\n\ni guess i would be ok with attching","commit_id":"8a8db66caf4552aa5cbfc948e9962b1969de3951"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"b773ec37a74773fdc77e098f16ea16e024d46226","unresolved":true,"context_lines":[{"line_number":47,"context_line":"When a PCI device is attached to a VM that is not assigned any CPU or"},{"line_number":48,"context_line":"RAM from the same physical NUMA node, we fallback to attaching devices"},{"line_number":49,"context_line":"to the root PCI device. We are also looking only at strict NUMA affinity"},{"line_number":50,"context_line":"here, not socket level affinity."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"For servers with a single SR-IOV capable NIC, you might start to get"},{"line_number":53,"context_line":"different guest topologies, depending on which NUMA node your pinned"}],"source_content_type":"text/x-rst","patch_set":4,"id":"539bc808_c4fb6492","line":50,"updated":"2023-12-06 13:25:21.000000000","message":"i would prefer if you did not talke abotu cpu or ram affinity like this\nits confusint the intent of the spec.\n\nwe should be phasing this in terms of if the guest has an implict numa toploty (hw:cpu_policy\u003ddedicated or hw:mem_page_size) or explict numa toplogy (hw:numa_nodes) and if the pci device resised on the same host numa node as a guest. numa node.\n\n\n\nbasically the proposal is.\n1.) if a vm has an implict or explcit numa toplogy\n2.) and the vm requests pci passthough\n3.) and the allocated pci device has a numa affinity reported.\n\nthen  where the gust has a numa node mapped to the  host numa node\nwhere the device residdes we will map it to a pxb local to the guest virutal numa node.\n\nif the pci device does not corralte with a guest numa node then it will be attach to the pci root and reported as numa node -1 in the guest as result.","commit_id":"8a8db66caf4552aa5cbfc948e9962b1969de3951"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"b773ec37a74773fdc77e098f16ea16e024d46226","unresolved":true,"context_lines":[{"line_number":54,"context_line":"VM happens to get scheduled on, as such, this must be opt in behavour."},{"line_number":55,"context_line":"We will use strict NUMA affinity on the PCI devices as the way to"},{"line_number":56,"context_line":"opt into correctly associating PCI devices with appropriate guest"},{"line_number":57,"context_line":"NUMA nodes."},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"Libvirt xml changes"},{"line_number":60,"context_line":"-------------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"89dce716_b16c9efd","line":57,"updated":"2023-12-06 13:25:21.000000000","message":"im not sure about this.\n\nwhen you enable strict mode it means we cant use any pci devices that dont report a numa affinity in /sys\n\ni wonder if it would be better to provide an opt in mechanium with a new flavor extra spec or image proeprty instead.\n\nim also wondering if it should be opt in or opt out.","commit_id":"8a8db66caf4552aa5cbfc948e9962b1969de3951"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"b773ec37a74773fdc77e098f16ea16e024d46226","unresolved":true,"context_lines":[{"line_number":150,"context_line":".. note:: PCI-NUMA affinity policies in scheduling has already been"},{"line_number":151,"context_line":"          implemented in Nova [6]_. This proposal does not affect scheduling"},{"line_number":152,"context_line":"          but is intended to give additional information to guests."},{"line_number":153,"context_line":""},{"line_number":154,"context_line":"Index Numbering"},{"line_number":155,"context_line":"---------------"},{"line_number":156,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"b078f447_fe8a94d3","line":153,"updated":"2023-12-06 13:25:21.000000000","message":"+1","commit_id":"8a8db66caf4552aa5cbfc948e9962b1969de3951"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"c4594fd82948fcd6ba374f0919d06a91884e8409","unresolved":true,"context_lines":[{"line_number":222,"context_line":"Upgrade impact"},{"line_number":223,"context_line":"--------------"},{"line_number":224,"context_line":""},{"line_number":225,"context_line":"When a user does a hard reboot of their VM, the guest"},{"line_number":226,"context_line":"NUMA topology will be updated to the new approach."},{"line_number":227,"context_line":""},{"line_number":228,"context_line":"Implementation"},{"line_number":229,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9c64bfc6_b01859c3","line":226,"range":{"start_line":225,"start_character":0,"end_line":226,"end_character":50},"updated":"2023-12-06 13:16:00.000000000","message":"Will this mean that from the guest perspective some PCIe devices will appear with different PCI addresses? Could this break naive application in the guest?","commit_id":"8a8db66caf4552aa5cbfc948e9962b1969de3951"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"b773ec37a74773fdc77e098f16ea16e024d46226","unresolved":true,"context_lines":[{"line_number":223,"context_line":"--------------"},{"line_number":224,"context_line":""},{"line_number":225,"context_line":"When a user does a hard reboot of their VM, the guest"},{"line_number":226,"context_line":"NUMA topology will be updated to the new approach."},{"line_number":227,"context_line":""},{"line_number":228,"context_line":"Implementation"},{"line_number":229,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1f97a87b_5897e897","line":226,"updated":"2023-12-06 13:25:21.000000000","message":"this would most likely change the pci adress and also impact the info present in the metadata api for device role tagging.\n\ni think the main reaosn you wnated ot have an opt into this new behaivor above was to mitigate that upgrade imact since chanigng the numa layout and pci adress of devie for exisitng vms my break the applciation that resied internally to it.\n\ni.e. dpdk applcation that rely on the adress or runting things like openshift/k8s ontop of nova vms where speicifc pci devices are exposed contienrs.\n\n\nto mitigate this properly i think we will need a new extra spec and image property to enable this behvior.\n\nhw:pci_numa_toplogy\u003dTrue|False\n\nby defualt in this cycle we sould assume False if not set in the flavor or image and recored that valie as an image proepty in the instance_system_metadata table.\n\nthat will mean that new vms or vms resized to a new flavor can opt into this feature for this release. in the E release we can then change the default to True\n\nthat will mean we will get the impvoed performacne out of the box for all new instance and we wont break existing instance since they will have had there toplogy requirements already recored in the db.\n\nwe have done this before for the instnace machine_type.","commit_id":"8a8db66caf4552aa5cbfc948e9962b1969de3951"}]}