)]}'
{"doc/source/admin/virtual-gpu.rst":[{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"0649d0b80c292b73d7d60c38ffdd701fb6fec215","unresolved":false,"context_lines":[{"line_number":211,"context_line":"* Live migrating an instance that has vGPUs doesn\u0027t yet work. If a user attempts"},{"line_number":212,"context_line":"  to do live migration, it will result XenAPI driver raising an exception which"},{"line_number":213,"context_line":"  will cause nova setting this instance to be in ERROR status. You can use the"},{"line_number":214,"context_line":"  command of ``nova reset-state --active \u003cVM\u003e`` to set it back to ACTIVE."},{"line_number":215,"context_line":""},{"line_number":216,"context_line":".. Links"},{"line_number":217,"context_line":".. _Intel GVT-g: https://01.org/igvt-g"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_d22d29a9","line":214,"updated":"2018-02-05 10:24:35.000000000","message":"I wonder if resizing and cold-migrating are also impacted with XenServer since we don\u0027t pass allocations down to the driver.\nI guess that\u0027s then XenServer driver which keeps the allocated GPUs and pass them back when the resize is done ?\n\nAlso, I wonder if rescued instances could have the same race condition than in libvirt. Could you please either confirm or infirm those points ?","commit_id":"10be340ce75b80f2eb0b72289531423f1dc2bd4f"},{"author":{"_account_id":17920,"name":"Jianghua Wang","email":"jianghua.wang@citrix.com","username":"wjhfresh"},"change_message_id":"fd3147c656fe1ac607991852c48864f9cf454fef","unresolved":false,"context_lines":[{"line_number":211,"context_line":"* Live migrating an instance that has vGPUs doesn\u0027t yet work. If a user attempts"},{"line_number":212,"context_line":"  to do live migration, it will result XenAPI driver raising an exception which"},{"line_number":213,"context_line":"  will cause nova setting this instance to be in ERROR status. You can use the"},{"line_number":214,"context_line":"  command of ``nova reset-state --active \u003cVM\u003e`` to set it back to ACTIVE."},{"line_number":215,"context_line":""},{"line_number":216,"context_line":".. Links"},{"line_number":217,"context_line":".. _Intel GVT-g: https://01.org/igvt-g"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_4074c735","line":214,"in_reply_to":"3fa0c359_d22d29a9","updated":"2018-02-06 07:10:23.000000000","message":"Sylvain, thanks. I confirmed XenServer also has the same issue for cold-migrating/resizing. We ran tests for this with some automation tests but unfortunately the issue is hidden as the operations finished successfully but the test code has problem to check the vGPU instance after the operations.\n\nFor rescue, the rescued VM doesn\u0027t have vGPU. But after *unrescue* the VM will have the vGPU attached again once the VM boots up on the original image. Placement also will hold the allocation during the whole rescue/unrescue process. So I think it\u0027s not a problem. There is no need to apply the workaround recommended for libvirt.","commit_id":"10be340ce75b80f2eb0b72289531423f1dc2bd4f"},{"author":{"_account_id":6735,"name":"Bob Ball","email":"bob.ball@citrix.com","username":"bob-ball"},"change_message_id":"243c2cc1e14e64cd1937a8d47fbedb43a53f976a","unresolved":false,"context_lines":[{"line_number":201,"context_line":""},{"line_number":202,"context_line":"For XenServer:"},{"line_number":203,"context_line":""},{"line_number":204,"context_line":"* Suspending an instance that has vGPUs doesn\u0027t yet work. XenServer\u0027s toolstack"},{"line_number":205,"context_line":"  will reject the suspending request if the instance has vGPUs. So the instance"},{"line_number":206,"context_line":"  will keep in the status of ``running`` in XenServer. But it will result"},{"line_number":207,"context_line":"  XenAPI raising an exception that will cause nova setting this instance to be"},{"line_number":208,"context_line":"  in ERROR status. You can use the command of ``nova reset-state --active \u003cVM\u003e``"},{"line_number":209,"context_line":"  to set it back to ACTIVE."},{"line_number":210,"context_line":""},{"line_number":211,"context_line":"* Resizing an instance with a new flavor that has vGPU resources doesn\u0027t"},{"line_number":212,"context_line":"  allocate those vGPUs to the instance (the instance is created without"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3fa0c359_b1b71b87","line":209,"range":{"start_line":204,"start_character":2,"end_line":209,"end_character":27},"updated":"2018-02-06 10:01:26.000000000","message":"Suspend and live migration with vGPUs attached depends on support from the underlying XenServer version.  Please see XenServer release notes for up to date information on when a hypervisor supporting live migration and suspend/resume with vGPUs is available.\nIf a suspend or live migrate operation is attempted with a XenServer version that does not support that operation, an internal exception that will cause nova setting this instance to be in ERROR status. You can use the command of ``nova reset-state --active \u003cVM\u003e`` to set it back to ACTIVE.","commit_id":"d7e5dceb5573f6ff1543c25627022104e0b2512b"},{"author":{"_account_id":17920,"name":"Jianghua Wang","email":"jianghua.wang@citrix.com","username":"wjhfresh"},"change_message_id":"204fe5ae847ef40e3e641fb9077aff3347e1572a","unresolved":false,"context_lines":[{"line_number":201,"context_line":""},{"line_number":202,"context_line":"For XenServer:"},{"line_number":203,"context_line":""},{"line_number":204,"context_line":"* Suspending an instance that has vGPUs doesn\u0027t yet work. XenServer\u0027s toolstack"},{"line_number":205,"context_line":"  will reject the suspending request if the instance has vGPUs. So the instance"},{"line_number":206,"context_line":"  will keep in the status of ``running`` in XenServer. But it will result"},{"line_number":207,"context_line":"  XenAPI raising an exception that will cause nova setting this instance to be"},{"line_number":208,"context_line":"  in ERROR status. You can use the command of ``nova reset-state --active \u003cVM\u003e``"},{"line_number":209,"context_line":"  to set it back to ACTIVE."},{"line_number":210,"context_line":""},{"line_number":211,"context_line":"* Resizing an instance with a new flavor that has vGPU resources doesn\u0027t"},{"line_number":212,"context_line":"  allocate those vGPUs to the instance (the instance is created without"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3fa0c359_72ee7223","line":209,"range":{"start_line":204,"start_character":2,"end_line":209,"end_character":27},"in_reply_to":"3fa0c359_b1b71b87","updated":"2018-02-06 11:12:16.000000000","message":"Understood actually suspend/live migration are already supported in some XenServer releases. Thanks. Will update it accordingly.","commit_id":"d7e5dceb5573f6ff1543c25627022104e0b2512b"},{"author":{"_account_id":6735,"name":"Bob Ball","email":"bob.ball@citrix.com","username":"bob-ball"},"change_message_id":"243c2cc1e14e64cd1937a8d47fbedb43a53f976a","unresolved":false,"context_lines":[{"line_number":217,"context_line":"  resize. If you want to migrate an instance, make sure to rebuild it after the"},{"line_number":218,"context_line":"  migration."},{"line_number":219,"context_line":""},{"line_number":220,"context_line":"* Live migrating an instance that has vGPUs doesn\u0027t yet work. If a user"},{"line_number":221,"context_line":"  attempts to do live migration, it will result XenAPI driver raising an"},{"line_number":222,"context_line":"  exception which will cause nova setting this instance to be in ERROR status."},{"line_number":223,"context_line":"  You can use the command of ``nova reset-state --active \u003cVM\u003e`` to set it back"},{"line_number":224,"context_line":"  to ACTIVE."},{"line_number":225,"context_line":""},{"line_number":226,"context_line":"* Rescue an instance: The instance being rescued doesn\u0027t have vGPUs. But"},{"line_number":227,"context_line":"  XenServer will keep the vGPU info in the VM record. After ``unrescue``,"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3fa0c359_d14ae77e","line":224,"range":{"start_line":220,"start_character":2,"end_line":224,"end_character":12},"updated":"2018-02-06 10:01:26.000000000","message":"Delete","commit_id":"d7e5dceb5573f6ff1543c25627022104e0b2512b"},{"author":{"_account_id":17920,"name":"Jianghua Wang","email":"jianghua.wang@citrix.com","username":"wjhfresh"},"change_message_id":"204fe5ae847ef40e3e641fb9077aff3347e1572a","unresolved":false,"context_lines":[{"line_number":217,"context_line":"  resize. If you want to migrate an instance, make sure to rebuild it after the"},{"line_number":218,"context_line":"  migration."},{"line_number":219,"context_line":""},{"line_number":220,"context_line":"* Live migrating an instance that has vGPUs doesn\u0027t yet work. If a user"},{"line_number":221,"context_line":"  attempts to do live migration, it will result XenAPI driver raising an"},{"line_number":222,"context_line":"  exception which will cause nova setting this instance to be in ERROR status."},{"line_number":223,"context_line":"  You can use the command of ``nova reset-state --active \u003cVM\u003e`` to set it back"},{"line_number":224,"context_line":"  to ACTIVE."},{"line_number":225,"context_line":""},{"line_number":226,"context_line":"* Rescue an instance: The instance being rescued doesn\u0027t have vGPUs. But"},{"line_number":227,"context_line":"  XenServer will keep the vGPU info in the VM record. After ``unrescue``,"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3fa0c359_12f36e0c","line":224,"range":{"start_line":220,"start_character":2,"end_line":224,"end_character":12},"in_reply_to":"3fa0c359_d14ae77e","updated":"2018-02-06 11:12:16.000000000","message":"Agreed as it will be covered in 1st item.","commit_id":"d7e5dceb5573f6ff1543c25627022104e0b2512b"},{"author":{"_account_id":6735,"name":"Bob Ball","email":"bob.ball@citrix.com","username":"bob-ball"},"change_message_id":"243c2cc1e14e64cd1937a8d47fbedb43a53f976a","unresolved":false,"context_lines":[{"line_number":223,"context_line":"  You can use the command of ``nova reset-state --active \u003cVM\u003e`` to set it back"},{"line_number":224,"context_line":"  to ACTIVE."},{"line_number":225,"context_line":""},{"line_number":226,"context_line":"* Rescue an instance: The instance being rescued doesn\u0027t have vGPUs. But"},{"line_number":227,"context_line":"  XenServer will keep the vGPU info in the VM record. After ``unrescue``,"},{"line_number":228,"context_line":"  the instance will boot from the normal boot disk again. Then a vGPU will"},{"line_number":229,"context_line":"  be attached back to the instance. So it should work after ``unrescue``."},{"line_number":230,"context_line":""},{"line_number":231,"context_line":".. Links"},{"line_number":232,"context_line":".. _Intel GVT-g: https://01.org/igvt-g"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3fa0c359_5156d72b","line":229,"range":{"start_line":226,"start_character":2,"end_line":229,"end_character":73},"updated":"2018-02-06 10:01:26.000000000","message":"Copy/paste the libvirt advice?","commit_id":"d7e5dceb5573f6ff1543c25627022104e0b2512b"},{"author":{"_account_id":17920,"name":"Jianghua Wang","email":"jianghua.wang@citrix.com","username":"wjhfresh"},"change_message_id":"204fe5ae847ef40e3e641fb9077aff3347e1572a","unresolved":false,"context_lines":[{"line_number":223,"context_line":"  You can use the command of ``nova reset-state --active \u003cVM\u003e`` to set it back"},{"line_number":224,"context_line":"  to ACTIVE."},{"line_number":225,"context_line":""},{"line_number":226,"context_line":"* Rescue an instance: The instance being rescued doesn\u0027t have vGPUs. But"},{"line_number":227,"context_line":"  XenServer will keep the vGPU info in the VM record. After ``unrescue``,"},{"line_number":228,"context_line":"  the instance will boot from the normal boot disk again. Then a vGPU will"},{"line_number":229,"context_line":"  be attached back to the instance. So it should work after ``unrescue``."},{"line_number":230,"context_line":""},{"line_number":231,"context_line":".. Links"},{"line_number":232,"context_line":".. _Intel GVT-g: https://01.org/igvt-g"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3fa0c359_92e69e44","line":229,"range":{"start_line":226,"start_character":2,"end_line":229,"end_character":73},"in_reply_to":"3fa0c359_5156d72b","updated":"2018-02-06 11:12:16.000000000","message":"I believe it should work after unrescue at least for XenServer. So I\u0027d keep this paragraph.","commit_id":"d7e5dceb5573f6ff1543c25627022104e0b2512b"},{"author":{"_account_id":6735,"name":"Bob Ball","email":"bob.ball@citrix.com","username":"bob-ball"},"change_message_id":"7168e2f0e5d3b46ebb5857ae4548f7aaa95b6056","unresolved":false,"context_lines":[{"line_number":219,"context_line":"  resize. If you want to migrate an instance, make sure to rebuild it after the"},{"line_number":220,"context_line":"  migration."},{"line_number":221,"context_line":""},{"line_number":222,"context_line":"* Rescue an instance: The instance being rescued doesn\u0027t have vGPUs. But"},{"line_number":223,"context_line":"  XenServer will keep the vGPU info in the VM record. After ``unrescue``,"},{"line_number":224,"context_line":"  the instance will boot from the normal boot disk again. Then a vGPU will"},{"line_number":225,"context_line":"  be attached back to the instance. So it should work after ``unrescue``."}],"source_content_type":"text/x-rst","patch_set":3,"id":"3fa0c359_38bd2b65","line":222,"range":{"start_line":222,"start_character":2,"end_line":222,"end_character":67},"updated":"2018-02-06 14:02:24.000000000","message":"Rescue images do not use vGPUs","commit_id":"150d4ffa9475019e7ff7ec4335b504f13d8b7f45"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"9ca9846739691cc20f01faedbbdfdf19d1d8c6c7","unresolved":false,"context_lines":[{"line_number":222,"context_line":"* Rescue images do not use vGPUs. But XenServer will keep the vGPU info in"},{"line_number":223,"context_line":"  the VM record. After ``unrescue``, the instance will boot from the normal"},{"line_number":224,"context_line":"  boot disk again. Then a vGPU will be attached back to the instance. So it"},{"line_number":225,"context_line":"  should work after ``unrescue``."},{"line_number":226,"context_line":""},{"line_number":227,"context_line":".. Links"},{"line_number":228,"context_line":".. _Intel GVT-g: https://01.org/igvt-g"}],"source_content_type":"text/x-rst","patch_set":4,"id":"3fa0c359_103fbf25","line":225,"updated":"2018-02-07 17:21:27.000000000","message":"That\u0027s a normal behaviour then, you don\u0027t need to mention it.\n\nThe problem I mentioned for libvirt is that we could potentially allocate the existing mdevs to another guest while the instance is rescued (since the mdevs are not in the current guest XML), which could be an issue when unrescuing because we reuse the pre-rescue XML (so two instances could share the same mediated devices \u003d\u003e KABOOM).","commit_id":"bc1dcfcb46ee3be08ff69b05564ce16d92fd77af"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"82ea2dfa1619f8629fd6dbc4a9aba4650da97c85","unresolved":false,"context_lines":[{"line_number":116,"context_line":"- For XenServer, virtual GPU types are created by XenServer at startup"},{"line_number":117,"context_line":"  depending on the available hardware and config files present in dom0."},{"line_number":118,"context_line":"  You can run the command of ``xe vgpu-type-list`` from dom0 to get the"},{"line_number":119,"context_line":"  avaiable vGPU types. The value for the field of ``model-name ( RO):``"},{"line_number":120,"context_line":"  is the vGPU type\u0027s name which can be used to set the nova configure -"},{"line_number":121,"context_line":"  ``[devices]/enabled_vgpu_types``. See the following example:"},{"line_number":122,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"3fa0c359_570c593a","line":119,"range":{"start_line":119,"start_character":2,"end_line":119,"end_character":10},"updated":"2018-02-07 19:47:33.000000000","message":"available","commit_id":"05f0b24d762bdb0777a1d9704f20f589230deb93"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"82ea2dfa1619f8629fd6dbc4a9aba4650da97c85","unresolved":false,"context_lines":[{"line_number":117,"context_line":"  depending on the available hardware and config files present in dom0."},{"line_number":118,"context_line":"  You can run the command of ``xe vgpu-type-list`` from dom0 to get the"},{"line_number":119,"context_line":"  avaiable vGPU types. The value for the field of ``model-name ( RO):``"},{"line_number":120,"context_line":"  is the vGPU type\u0027s name which can be used to set the nova configure -"},{"line_number":121,"context_line":"  ``[devices]/enabled_vgpu_types``. See the following example:"},{"line_number":122,"context_line":""},{"line_number":123,"context_line":"  .. code-block:: console"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3fa0c359_b71cbd6a","line":120,"range":{"start_line":120,"start_character":60,"end_line":120,"end_character":69},"updated":"2018-02-07 19:47:33.000000000","message":"config option","commit_id":"05f0b24d762bdb0777a1d9704f20f589230deb93"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"82ea2dfa1619f8629fd6dbc4a9aba4650da97c85","unresolved":false,"context_lines":[{"line_number":206,"context_line":"  date information on when a hypervisor supporting live migration and"},{"line_number":207,"context_line":"  suspend/resume with vGPUs is available. If a suspend or live migrate operation"},{"line_number":208,"context_line":"  is attempted with a XenServer version that does not support that operation, an"},{"line_number":209,"context_line":"  internal exception that will cause nova setting this instance to be in ERROR"},{"line_number":210,"context_line":"  status. You can use the command of ``nova reset-state --active \u003cVM\u003e`` to set"},{"line_number":211,"context_line":"  it back to ACTIVE."},{"line_number":212,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"3fa0c359_1737b1e5","line":209,"range":{"start_line":209,"start_character":11,"end_line":209,"end_character":20},"updated":"2018-02-07 19:47:33.000000000","message":"exception will occur?","commit_id":"05f0b24d762bdb0777a1d9704f20f589230deb93"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"82ea2dfa1619f8629fd6dbc4a9aba4650da97c85","unresolved":false,"context_lines":[{"line_number":207,"context_line":"  suspend/resume with vGPUs is available. If a suspend or live migrate operation"},{"line_number":208,"context_line":"  is attempted with a XenServer version that does not support that operation, an"},{"line_number":209,"context_line":"  internal exception that will cause nova setting this instance to be in ERROR"},{"line_number":210,"context_line":"  status. You can use the command of ``nova reset-state --active \u003cVM\u003e`` to set"},{"line_number":211,"context_line":"  it back to ACTIVE."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"* Resizing an instance with a new flavor that has vGPU resources doesn\u0027t"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3fa0c359_d745c944","line":210,"range":{"start_line":210,"start_character":39,"end_line":210,"end_character":69},"updated":"2018-02-07 19:47:33.000000000","message":"Might be good to try and direct people to using OSC commands:\n\nopenstack server set --state active \u003cserver\u003e","commit_id":"05f0b24d762bdb0777a1d9704f20f589230deb93"}]}