)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"732814f7ade84b6b088e72f69f451a1d2f4a806b","unresolved":false,"context_lines":[{"line_number":9,"context_line":"This commit deals with the following 2 scenarios that leave an instance"},{"line_number":10,"context_line":"with incorrect port binding."},{"line_number":11,"context_line":""},{"line_number":12,"context_line":"If an evacuation of an instance fails part way through, after the point"},{"line_number":13,"context_line":"where we reassign the port binding to the new host but before we change"},{"line_number":14,"context_line":"the instance host, we end up with the ports assigned to the wrong host."},{"line_number":15,"context_line":"This change adds a check to determine if there\u0027s any port binding host"},{"line_number":16,"context_line":"mismatches and if so trigger setup of instance network."},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"During recovery of failed hosts, neutron could get overwhelmed and lose"},{"line_number":19,"context_line":"messages, for example when active controller was powered-off in the"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":7,"id":"3f79a3b5_cc0c1a3b","line":16,"range":{"start_line":12,"start_character":0,"end_line":16,"end_character":55},"updated":"2018-10-11 22:10:15.000000000","message":"So what\u0027s the scenario here?\n\n1. evacuate from failed host\n2. bind ports to dest host\n3. something fails before we update instance.host to dest\n4. now the user reboots/rebuilds the server on the source host? but if they are evacuating from the source host, the source host is down so how can they rebuild/reboot the server there?","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"ab15631a00cea5ae482476195f627e38491bd86f","unresolved":false,"context_lines":[{"line_number":9,"context_line":"This commit deals with the following 2 scenarios that leave an instance"},{"line_number":10,"context_line":"with incorrect port binding."},{"line_number":11,"context_line":""},{"line_number":12,"context_line":"If an evacuation of an instance fails part way through, after the point"},{"line_number":13,"context_line":"where we reassign the port binding to the new host but before we change"},{"line_number":14,"context_line":"the instance host, we end up with the ports assigned to the wrong host."},{"line_number":15,"context_line":"This change adds a check to determine if there\u0027s any port binding host"},{"line_number":16,"context_line":"mismatches and if so trigger setup of instance network."},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"During recovery of failed hosts, neutron could get overwhelmed and lose"},{"line_number":19,"context_line":"messages, for example when active controller was powered-off in the"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":7,"id":"3f79a3b5_f1613fd7","line":16,"range":{"start_line":12,"start_character":0,"end_line":16,"end_character":55},"in_reply_to":"3f79a3b5_cc0c1a3b","updated":"2018-10-30 17:58:39.000000000","message":"The scenario here is a multi-node reboot. Source host and dest were both rebooted in a short duration. Step 4 happened when source host came back from reboot.","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1093535d0fc29353aa7f7278749f53fa5d584674","unresolved":false,"context_lines":[{"line_number":9,"context_line":"This commit deals with the following 2 scenarios that leave an instance"},{"line_number":10,"context_line":"with incorrect port binding."},{"line_number":11,"context_line":""},{"line_number":12,"context_line":"If an evacuation of an instance fails part way through, after the point"},{"line_number":13,"context_line":"where we reassign the port binding to the new host but before we change"},{"line_number":14,"context_line":"the instance host, we end up with the ports assigned to the wrong host."},{"line_number":15,"context_line":"This change adds a check to determine if there\u0027s any port binding host"},{"line_number":16,"context_line":"mismatches and if so trigger setup of instance network."},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"During recovery of failed hosts, neutron could get overwhelmed and lose"},{"line_number":19,"context_line":"messages, for example when active controller was powered-off in the"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":7,"id":"3f79a3b5_6c706642","line":16,"range":{"start_line":12,"start_character":0,"end_line":16,"end_character":55},"in_reply_to":"3f79a3b5_cc0c1a3b","updated":"2018-10-11 22:34:02.000000000","message":"if the image does not change i think rebuild skips the schduler correct because it cannot have changed the resources it needs. so in that case it would not hit the comptue node filter and would not chenck that the node status.\n\nso provided the souce host is actully still alive it would be able to start the vm. that said depending on where it failed in the clean up it might not spawn correctly?\n\nif it dose spawn then there woulds be a possiblity of data currption if ti was on a shared file system or was a bfv instance as you would have two vms acessing the same block device","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"732814f7ade84b6b088e72f69f451a1d2f4a806b","unresolved":false,"context_lines":[{"line_number":23,"context_line":"instance unrecoverable."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"This commit updates port binding in neutron if the host has changed"},{"line_number":26,"context_line":"during a failed evacuation or if vif_type has become \u0027binding_failed\u0027 or"},{"line_number":27,"context_line":"\u0027unbound\u0027 due to previous lost neutron messages."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Closes-Bug: #1659062"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":7,"id":"3f79a3b5_ec9d56ba","line":26,"range":{"start_line":26,"start_character":0,"end_line":26,"end_character":26},"updated":"2018-10-11 22:10:15.000000000","message":"Alternatively we could rollback port bindings if evacuate fails after the point of updating port bindings to the dest host...that would be less invasive than always checking for corrupted port bindings on every rebuild, right?","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1093535d0fc29353aa7f7278749f53fa5d584674","unresolved":false,"context_lines":[{"line_number":23,"context_line":"instance unrecoverable."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"This commit updates port binding in neutron if the host has changed"},{"line_number":26,"context_line":"during a failed evacuation or if vif_type has become \u0027binding_failed\u0027 or"},{"line_number":27,"context_line":"\u0027unbound\u0027 due to previous lost neutron messages."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Closes-Bug: #1659062"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":7,"id":"3f79a3b5_6c4146a7","line":26,"range":{"start_line":26,"start_character":0,"end_line":26,"end_character":26},"in_reply_to":"3f79a3b5_ec9d56ba","updated":"2018-10-11 22:34:02.000000000","message":"we may be able to just use the multiple port binding code also and only activate the bindings right before we start the vm and not delete the souce binding until we confim.\n\nthat way if we need to rule back we jsut activate the binding on the original host and delete the new ones.","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"732814f7ade84b6b088e72f69f451a1d2f4a806b","unresolved":false,"context_lines":[{"line_number":26,"context_line":"during a failed evacuation or if vif_type has become \u0027binding_failed\u0027 or"},{"line_number":27,"context_line":"\u0027unbound\u0027 due to previous lost neutron messages."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Closes-Bug: #1659062"},{"line_number":30,"context_line":""},{"line_number":31,"context_line":"Change-Id: I75fd15ac2a29e420c09499f2c41d11259ca811ae"},{"line_number":32,"context_line":"Signed-off-by: Jack Ding \u003cjack.ding@windriver.com\u003e"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":7,"id":"3f79a3b5_ec68f6bc","line":29,"updated":"2018-10-11 22:10:15.000000000","message":"Bug 1784579 also seems related, see change Ia963a093a1b26d90b4de2e8fc623031cf175aece.","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"6994175cf3cda15dccc8aa8f3e3e4b649e4662f2","unresolved":false,"context_lines":[{"line_number":7,"context_line":"Correct instance port binding for rebuilds"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"The following 2 scenarios could result in an instance with incorrect"},{"line_number":10,"context_line":"port binding and cause subsequent rebuids to fail."},{"line_number":11,"context_line":""},{"line_number":12,"context_line":"If an evacuation of an instance fails part way through, after the point"},{"line_number":13,"context_line":"where we reassign the port binding to the new host but before we change"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":18,"id":"9fdfeff1_a01111af","line":10,"range":{"start_line":10,"start_character":34,"end_line":10,"end_character":41},"updated":"2019-02-05 15:34:56.000000000","message":"/s/rebuids/rebuilds","commit_id":"ca93e059be7397ac2ba9175238e88a20d6ce59b7"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"0ffb20eae50eae907e08291a557f0bca215e36ea","unresolved":false,"context_lines":[{"line_number":7,"context_line":"Correct instance port binding for rebuilds"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"The following 2 scenarios could result in an instance with incorrect"},{"line_number":10,"context_line":"port binding and cause subsequent rebuids to fail."},{"line_number":11,"context_line":""},{"line_number":12,"context_line":"If an evacuation of an instance fails part way through, after the point"},{"line_number":13,"context_line":"where we reassign the port binding to the new host but before we change"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":18,"id":"9fdfeff1_d9ca0c19","line":10,"range":{"start_line":10,"start_character":34,"end_line":10,"end_character":41},"in_reply_to":"9fdfeff1_a01111af","updated":"2019-02-05 17:29:08.000000000","message":"Done","commit_id":"ca93e059be7397ac2ba9175238e88a20d6ce59b7"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"c04044328d384e3ef881467ab10f01b035532fbe","unresolved":false,"context_lines":[{"line_number":9,"context_line":"The following 2 scenarios could result in an instance with incorrect"},{"line_number":10,"context_line":"port binding and cause subsequent rebuilds to fail."},{"line_number":11,"context_line":""},{"line_number":12,"context_line":"If an evacuation of an instance fails part way through, after the point"},{"line_number":13,"context_line":"where we reassign the port binding to the new host but before we change"},{"line_number":14,"context_line":"the instance host, we end up with the ports assigned to the wrong host."},{"line_number":15,"context_line":"This change adds a check to determine if there\u0027s any port binding host"},{"line_number":16,"context_line":"mismatches and if so trigger setup of instance network."},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"During recovery of failed hosts, neutron could get overwhelmed and lose"},{"line_number":19,"context_line":"messages, for example when active controller was powered-off in the"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":19,"id":"9fdfeff1_06eb39b0","line":16,"range":{"start_line":12,"start_character":0,"end_line":16,"end_character":55},"updated":"2019-02-06 21:48:33.000000000","message":"Look familiar?\n\nhttps://review.openstack.org/#/c/635080/1/nova/compute/manager.py@5029\n\nI just wrote that yesterday and noticed it after seeing the same problem in the normal finish_resize code:\n\nhttps://github.com/openstack/nova/blob/b2299908d3bad434dd25d6c6bb0a9e6b4ab9eba2/nova/compute/manager.py#L4501\n\nAlthough for resize I guess that\u0027s less of an issue because before we get that far we update the instance.host to the dest compute:\n\nhttps://github.com/openstack/nova/blob/b2299908d3bad434dd25d6c6bb0a9e6b4ab9eba2/nova/compute/manager.py#L4390","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"8e755fd94a417a393758586772c271ffe60c7375","unresolved":false,"context_lines":[{"line_number":9,"context_line":"The following 2 scenarios could result in an instance with incorrect"},{"line_number":10,"context_line":"port binding and cause subsequent rebuilds to fail."},{"line_number":11,"context_line":""},{"line_number":12,"context_line":"If an evacuation of an instance fails part way through, after the point"},{"line_number":13,"context_line":"where we reassign the port binding to the new host but before we change"},{"line_number":14,"context_line":"the instance host, we end up with the ports assigned to the wrong host."},{"line_number":15,"context_line":"This change adds a check to determine if there\u0027s any port binding host"},{"line_number":16,"context_line":"mismatches and if so trigger setup of instance network."},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"During recovery of failed hosts, neutron could get overwhelmed and lose"},{"line_number":19,"context_line":"messages, for example when active controller was powered-off in the"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":19,"id":"9fdfeff1_5aab4e74","line":16,"range":{"start_line":12,"start_character":0,"end_line":16,"end_character":55},"in_reply_to":"9fdfeff1_06eb39b0","updated":"2019-02-07 23:22:53.000000000","message":"Glad to see all things fall into place :)","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"c04044328d384e3ef881467ab10f01b035532fbe","unresolved":false,"context_lines":[{"line_number":22,"context_line":"type\" exception during instance hard_reboot or rebuild, leaving the"},{"line_number":23,"context_line":"instance unrecoverable."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"This commit changes _heal_instance_info_cache periodic task to update"},{"line_number":26,"context_line":"port binding if evacuation fails due to above errors so that the"},{"line_number":27,"context_line":"instance can be recovered later."},{"line_number":28,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":19,"id":"9fdfeff1_864409b9","line":25,"updated":"2019-02-06 21:48:33.000000000","message":"In the case of evacuate, we would assume that the instance.host is still pointing at the source compute and if we\u0027re evacuating, that source compute service should be down, so _heal_instance_info_cache wouldn\u0027t be running. Now if you restart that source compute service and the instance is still there, then the periodic could \u0027heal\u0027 it. That\u0027s also how we can get here on restart of the source compute:\n\nhttps://github.com/openstack/nova/blob/b2299908d3bad434dd25d6c6bb0a9e6b4ab9eba2/nova/compute/manager.py#L963\n\nSpeaking of which, you could now update the comment in that code where it says \"We could add a check to _heal_instance_info_cache to handle this, but probably only if\nthe instance task_state is None.\" since that\u0027s exactly what this change is doing right?","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"8e755fd94a417a393758586772c271ffe60c7375","unresolved":false,"context_lines":[{"line_number":22,"context_line":"type\" exception during instance hard_reboot or rebuild, leaving the"},{"line_number":23,"context_line":"instance unrecoverable."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"This commit changes _heal_instance_info_cache periodic task to update"},{"line_number":26,"context_line":"port binding if evacuation fails due to above errors so that the"},{"line_number":27,"context_line":"instance can be recovered later."},{"line_number":28,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":19,"id":"9fdfeff1_da33be61","line":25,"in_reply_to":"9fdfeff1_864409b9","updated":"2019-02-07 23:22:53.000000000","message":"Right. Updated the comments.","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"c04044328d384e3ef881467ab10f01b035532fbe","unresolved":false,"context_lines":[{"line_number":26,"context_line":"port binding if evacuation fails due to above errors so that the"},{"line_number":27,"context_line":"instance can be recovered later."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Closes-Bug: #1659062"},{"line_number":30,"context_line":""},{"line_number":31,"context_line":"Co-Authored-By: Gerry Kopec \u003cgerry.kopec@windriver.com\u003e"},{"line_number":32,"context_line":"Co-Authored-By: Jim Gauld \u003cJames.Gauld@windriver.com\u003e"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":19,"id":"9fdfeff1_2678bdf6","line":29,"updated":"2019-02-06 21:48:33.000000000","message":"You could also add \"Related-Bug: #1784579\" here.","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"8e755fd94a417a393758586772c271ffe60c7375","unresolved":false,"context_lines":[{"line_number":26,"context_line":"port binding if evacuation fails due to above errors so that the"},{"line_number":27,"context_line":"instance can be recovered later."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Closes-Bug: #1659062"},{"line_number":30,"context_line":""},{"line_number":31,"context_line":"Co-Authored-By: Gerry Kopec \u003cgerry.kopec@windriver.com\u003e"},{"line_number":32,"context_line":"Co-Authored-By: Jim Gauld \u003cJames.Gauld@windriver.com\u003e"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":19,"id":"9fdfeff1_5a47cebd","line":29,"in_reply_to":"9fdfeff1_2678bdf6","updated":"2019-02-07 23:22:53.000000000","message":"Done","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"e657fe7fcf4f32df5cde8b1d6a48852a463331db","unresolved":false,"context_lines":[{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Closes-Bug: #1659062"},{"line_number":30,"context_line":"Related-Bug: #1784579"},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"Co-Authored-By: Gerry Kopec \u003cgerry.kopec@windriver.com\u003e"},{"line_number":33,"context_line":"Co-Authored-By: Jim Gauld \u003cJames.Gauld@windriver.com\u003e"},{"line_number":34,"context_line":"Change-Id: I75fd15ac2a29e420c09499f2c41d11259ca811ae"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":22,"id":"9fdfeff1_201674a9","line":31,"updated":"2019-03-04 22:26:46.000000000","message":"so there is a related neutron bug that is also being fixed rithg now  Closes-Bug: #1815345\nthey are changing how binding are deleted when we revert a migration.\nwe need to be careful that there is not an unintened interaction between these two pathces. https://review.openstack.org/#/c/634276/","commit_id":"542635034882e1b6897e1935f09d6feb6e77d1ce"}],"nova/compute/manager.py":[{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"732814f7ade84b6b088e72f69f451a1d2f4a806b","unresolved":false,"context_lines":[{"line_number":960,"context_line":"            self.driver.plug_vifs(instance, net_info)"},{"line_number":961,"context_line":"        except NotImplementedError as e:"},{"line_number":962,"context_line":"            LOG.debug(e, instance\u003dinstance)"},{"line_number":963,"context_line":"        except exception.VirtualInterfacePlugException:"},{"line_number":964,"context_line":"            # NOTE(mriedem): If we get here, it could be because the vif_type"},{"line_number":965,"context_line":"            # in the cache is \"binding_failed\". The only way to fix that is to"},{"line_number":966,"context_line":"            # try and bind the ports again, which would be expensive here on"},{"line_number":967,"context_line":"            # host startup. We could add a check to _heal_instance_info_cache"},{"line_number":968,"context_line":"            # to handle this, but probably only if the instance task_state is"},{"line_number":969,"context_line":"            # None."},{"line_number":970,"context_line":"            LOG.exception(\u0027Virtual interface plugging failed for instance. \u0027"},{"line_number":971,"context_line":"                          \u0027The port binding:host_id may need to be manually \u0027"},{"line_number":972,"context_line":"                          \u0027updated.\u0027, instance\u003dinstance)"},{"line_number":973,"context_line":"            self._set_instance_obj_error_state(context, instance)"},{"line_number":974,"context_line":"            return"},{"line_number":975,"context_line":""},{"line_number":976,"context_line":"        if instance.task_state \u003d\u003d task_states.RESIZE_MIGRATING:"},{"line_number":977,"context_line":"            # We crashed during resize/migration, so roll back for safety"}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_71411da7","line":974,"range":{"start_line":963,"start_character":7,"end_line":974,"end_character":18},"updated":"2018-10-11 22:10:15.000000000","message":"Sort of reminds me of this case...","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"732814f7ade84b6b088e72f69f451a1d2f4a806b","unresolved":false,"context_lines":[{"line_number":3172,"context_line":"        instance.save(expected_task_state\u003d[task_states.REBUILDING])"},{"line_number":3173,"context_line":""},{"line_number":3174,"context_line":"        if evacuate or self._require_nw_info_update(context, instance):"},{"line_number":3175,"context_line":"            self.network_api.setup_networks_on_host("},{"line_number":3176,"context_line":"                    context, instance, self.host)"},{"line_number":3177,"context_line":"            # For nova-network this is needed to move floating IPs"},{"line_number":3178,"context_line":"            # For neutron this updates the host in the port binding"}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_914f198b","line":3175,"range":{"start_line":3175,"start_character":29,"end_line":3175,"end_character":51},"updated":"2018-10-11 22:10:15.000000000","message":"OK this is a noop if rebuilding (not evacuating) because the instance.host should match the self.host.","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"0291b4171a92505954b0332946548c8d971f264c","unresolved":false,"context_lines":[{"line_number":3172,"context_line":"        instance.save(expected_task_state\u003d[task_states.REBUILDING])"},{"line_number":3173,"context_line":""},{"line_number":3174,"context_line":"        if evacuate or self._require_nw_info_update(context, instance):"},{"line_number":3175,"context_line":"            self.network_api.setup_networks_on_host("},{"line_number":3176,"context_line":"                    context, instance, self.host)"},{"line_number":3177,"context_line":"            # For nova-network this is needed to move floating IPs"},{"line_number":3178,"context_line":"            # For neutron this updates the host in the port binding"}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_8f428a93","line":3175,"range":{"start_line":3175,"start_character":29,"end_line":3175,"end_character":51},"in_reply_to":"3f79a3b5_29398bcf","updated":"2018-12-19 23:45:02.000000000","message":"\u003e Previous failed evacuate could leave the instance stuck with an\n \u003e incorrect host_id(dest), so that the subsequent rebuilding could\n \u003e have host mismatch.\n\nHost mismatch where? You mean the guest is running on the source host but the database says the instance is on the dest host? If that\u0027s the case, conductor will RPC cast to the dest host (if instance.host says it\u0027s there) and try to rebuild there, and then couldn\u0027t we end up with the same guest running on multiple hosts? Which is...bad.","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"921ad07a1815d7a2b9a2dfae5cad10c197b33217","unresolved":false,"context_lines":[{"line_number":3172,"context_line":"        instance.save(expected_task_state\u003d[task_states.REBUILDING])"},{"line_number":3173,"context_line":""},{"line_number":3174,"context_line":"        if evacuate or self._require_nw_info_update(context, instance):"},{"line_number":3175,"context_line":"            self.network_api.setup_networks_on_host("},{"line_number":3176,"context_line":"                    context, instance, self.host)"},{"line_number":3177,"context_line":"            # For nova-network this is needed to move floating IPs"},{"line_number":3178,"context_line":"            # For neutron this updates the host in the port binding"}],"source_content_type":"text/x-python","patch_set":7,"id":"bfdaf3ff_16b5dcee","line":3175,"range":{"start_line":3175,"start_character":29,"end_line":3175,"end_character":51},"in_reply_to":"3f79a3b5_8f428a93","updated":"2019-01-10 20:20:22.000000000","message":"\u003e \u003e Previous failed evacuate could leave the instance stuck with an\n \u003e \u003e incorrect host_id(dest), so that the subsequent rebuilding could\n \u003e \u003e have host mismatch.\n \u003e \n \u003e Host mismatch where? You mean the guest is running on the source\n \u003e host but the database says the instance is on the dest host? If\n \u003e that\u0027s the case, conductor will RPC cast to the dest host (if\n \u003e instance.host says it\u0027s there) and try to rebuild there, and then\n \u003e couldn\u0027t we end up with the same guest running on multiple hosts?\n \u003e Which is...bad.\n\nI meant port[\u0027binding:host_id\u0027], not instance.host_id. So it is a mismatch between port[\u0027binding:host_id\u0027] and compute host_id (which should be the same as instance.host_id).","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"3c8cdc1f4a6d71fc97053346d96a3c227762f6c6","unresolved":false,"context_lines":[{"line_number":3172,"context_line":"        instance.save(expected_task_state\u003d[task_states.REBUILDING])"},{"line_number":3173,"context_line":""},{"line_number":3174,"context_line":"        if evacuate or self._require_nw_info_update(context, instance):"},{"line_number":3175,"context_line":"            self.network_api.setup_networks_on_host("},{"line_number":3176,"context_line":"                    context, instance, self.host)"},{"line_number":3177,"context_line":"            # For nova-network this is needed to move floating IPs"},{"line_number":3178,"context_line":"            # For neutron this updates the host in the port binding"}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_c02d4b91","line":3175,"range":{"start_line":3175,"start_character":29,"end_line":3175,"end_character":51},"in_reply_to":"3f79a3b5_914f198b","updated":"2018-10-30 22:58:33.000000000","message":"Done","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"59a3c030bdce54bc410197932fcc4d6e17593778","unresolved":false,"context_lines":[{"line_number":3172,"context_line":"        instance.save(expected_task_state\u003d[task_states.REBUILDING])"},{"line_number":3173,"context_line":""},{"line_number":3174,"context_line":"        if evacuate or self._require_nw_info_update(context, instance):"},{"line_number":3175,"context_line":"            self.network_api.setup_networks_on_host("},{"line_number":3176,"context_line":"                    context, instance, self.host)"},{"line_number":3177,"context_line":"            # For nova-network this is needed to move floating IPs"},{"line_number":3178,"context_line":"            # For neutron this updates the host in the port binding"}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_29398bcf","line":3175,"range":{"start_line":3175,"start_character":29,"end_line":3175,"end_character":51},"in_reply_to":"3f79a3b5_914f198b","updated":"2018-10-30 23:37:26.000000000","message":"Previous failed evacuate could leave the instance stuck with an incorrect host_id(dest), so that the subsequent rebuilding could have host mismatch.","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"732814f7ade84b6b088e72f69f451a1d2f4a806b","unresolved":false,"context_lines":[{"line_number":3306,"context_line":"            if p.get(\u0027binding:host_id\u0027) !\u003d self.host:"},{"line_number":3307,"context_line":"                return True"},{"line_number":3308,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":3309,"context_line":"            if (vif_type \u003d\u003d \u0027unbound\u0027 or"},{"line_number":3310,"context_line":"                    vif_type \u003d\u003d network_model.VIF_TYPE_BINDING_FAILED):"},{"line_number":3311,"context_line":"                return True"},{"line_number":3312,"context_line":"        return False"}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_71ecbdbb","line":3309,"range":{"start_line":3309,"start_character":28,"end_line":3309,"end_character":37},"updated":"2018-10-11 22:10:15.000000000","message":"OK I guess this is a valid vif type:\n\nhttp://git.openstack.org/cgit/openstack/neutron-lib/tree/neutron_lib/api/definitions/portbindings.py#n85\n\nWe should have a constant for this in models like for binding failed.","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"3c8cdc1f4a6d71fc97053346d96a3c227762f6c6","unresolved":false,"context_lines":[{"line_number":3306,"context_line":"            if p.get(\u0027binding:host_id\u0027) !\u003d self.host:"},{"line_number":3307,"context_line":"                return True"},{"line_number":3308,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":3309,"context_line":"            if (vif_type \u003d\u003d \u0027unbound\u0027 or"},{"line_number":3310,"context_line":"                    vif_type \u003d\u003d network_model.VIF_TYPE_BINDING_FAILED):"},{"line_number":3311,"context_line":"                return True"},{"line_number":3312,"context_line":"        return False"}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_8020b39f","line":3309,"range":{"start_line":3309,"start_character":28,"end_line":3309,"end_character":37},"in_reply_to":"3f79a3b5_71ecbdbb","updated":"2018-10-30 22:58:33.000000000","message":"Done","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"732814f7ade84b6b088e72f69f451a1d2f4a806b","unresolved":false,"context_lines":[{"line_number":3334,"context_line":"        block_device_info \u003d self._get_instance_block_device_info("},{"line_number":3335,"context_line":"            context, instance, bdms\u003dbdms)"},{"line_number":3336,"context_line":""},{"line_number":3337,"context_line":"        network_info \u003d self.network_api.get_instance_nw_info(context, instance)"},{"line_number":3338,"context_line":"        # If after a failed rebuild an instance is rebooted it\u0027s possible"},{"line_number":3339,"context_line":"        # that during this moment the neutron server is down. The neutron ports"},{"line_number":3340,"context_line":"        # are then not updated and has stale data from the evacuated node."}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_f1df6d08","line":3337,"range":{"start_line":3337,"start_character":23,"end_line":3337,"end_character":79},"updated":"2018-10-11 22:10:15.000000000","message":"Doesn\u0027t this do a refresh of the network info cache from neutron?","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"cedfdfbb6fa81cf0ab91b86ba7aed737f9eb374b","unresolved":false,"context_lines":[{"line_number":3334,"context_line":"        block_device_info \u003d self._get_instance_block_device_info("},{"line_number":3335,"context_line":"            context, instance, bdms\u003dbdms)"},{"line_number":3336,"context_line":""},{"line_number":3337,"context_line":"        network_info \u003d self.network_api.get_instance_nw_info(context, instance)"},{"line_number":3338,"context_line":"        # If after a failed rebuild an instance is rebooted it\u0027s possible"},{"line_number":3339,"context_line":"        # that during this moment the neutron server is down. The neutron ports"},{"line_number":3340,"context_line":"        # are then not updated and has stale data from the evacuated node."}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_8f9e0af9","line":3337,"range":{"start_line":3337,"start_character":23,"end_line":3337,"end_character":79},"in_reply_to":"3f79a3b5_3a9feb51","updated":"2018-12-19 23:47:59.000000000","message":"\u003e If this is already pulling the current state of the ports and putting them on the instance.info_cache, then _require_nw_info_update shouldn\u0027t need to get the ports from neutron again, right?\n\nThis doesn\u0027t get the current state from neutron, see:\n\nhttps://review.openstack.org/#/c/591607/","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"c049bbb9e99f25f5b2e65657dd6f83c931165b65","unresolved":false,"context_lines":[{"line_number":3334,"context_line":"        block_device_info \u003d self._get_instance_block_device_info("},{"line_number":3335,"context_line":"            context, instance, bdms\u003dbdms)"},{"line_number":3336,"context_line":""},{"line_number":3337,"context_line":"        network_info \u003d self.network_api.get_instance_nw_info(context, instance)"},{"line_number":3338,"context_line":"        # If after a failed rebuild an instance is rebooted it\u0027s possible"},{"line_number":3339,"context_line":"        # that during this moment the neutron server is down. The neutron ports"},{"line_number":3340,"context_line":"        # are then not updated and has stale data from the evacuated node."}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_7adfa30a","line":3337,"range":{"start_line":3337,"start_character":23,"end_line":3337,"end_character":79},"in_reply_to":"3f79a3b5_3a9feb51","updated":"2018-10-12 15:55:47.000000000","message":"That might work here, but maybe doesn\u0027t work in the evacuate case above?","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"3c8cdc1f4a6d71fc97053346d96a3c227762f6c6","unresolved":false,"context_lines":[{"line_number":3334,"context_line":"        block_device_info \u003d self._get_instance_block_device_info("},{"line_number":3335,"context_line":"            context, instance, bdms\u003dbdms)"},{"line_number":3336,"context_line":""},{"line_number":3337,"context_line":"        network_info \u003d self.network_api.get_instance_nw_info(context, instance)"},{"line_number":3338,"context_line":"        # If after a failed rebuild an instance is rebooted it\u0027s possible"},{"line_number":3339,"context_line":"        # that during this moment the neutron server is down. The neutron ports"},{"line_number":3340,"context_line":"        # are then not updated and has stale data from the evacuated node."}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_669d70b9","line":3337,"range":{"start_line":3337,"start_character":23,"end_line":3337,"end_character":79},"in_reply_to":"3f79a3b5_3a9feb51","updated":"2018-10-30 22:58:33.000000000","message":"network_info has vif_type, but does not have host_id","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"72c844bc3d27cd9f30c5999d65c742ca5ecfb72a","unresolved":false,"context_lines":[{"line_number":3334,"context_line":"        block_device_info \u003d self._get_instance_block_device_info("},{"line_number":3335,"context_line":"            context, instance, bdms\u003dbdms)"},{"line_number":3336,"context_line":""},{"line_number":3337,"context_line":"        network_info \u003d self.network_api.get_instance_nw_info(context, instance)"},{"line_number":3338,"context_line":"        # If after a failed rebuild an instance is rebooted it\u0027s possible"},{"line_number":3339,"context_line":"        # that during this moment the neutron server is down. The neutron ports"},{"line_number":3340,"context_line":"        # are then not updated and has stale data from the evacuated node."}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_2f3ed6e1","line":3337,"range":{"start_line":3337,"start_character":23,"end_line":3337,"end_character":79},"in_reply_to":"3f79a3b5_669d70b9","updated":"2018-12-19 23:51:36.000000000","message":"\u003e network_info has vif_type, but does not have host_id\n\nBut couldn\u0027t we *only* do the list_ports call if we have determined from the cache that the vif_type is unbound or binding_failed? Meaning under normal operations where everything is fine, we don\u0027t make any API calls to neutron. We would only automatically try to heal if and only if we have detected a bad state in the system.\n\nAlternatively, would it be useful to start storing the binding:host_id in the network info cache? It\u0027s just a big dict blob, we can store more stuff in there if we need to.","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"b51506eb4b0429693f4132e3a30bae0813037a59","unresolved":false,"context_lines":[{"line_number":3334,"context_line":"        block_device_info \u003d self._get_instance_block_device_info("},{"line_number":3335,"context_line":"            context, instance, bdms\u003dbdms)"},{"line_number":3336,"context_line":""},{"line_number":3337,"context_line":"        network_info \u003d self.network_api.get_instance_nw_info(context, instance)"},{"line_number":3338,"context_line":"        # If after a failed rebuild an instance is rebooted it\u0027s possible"},{"line_number":3339,"context_line":"        # that during this moment the neutron server is down. The neutron ports"},{"line_number":3340,"context_line":"        # are then not updated and has stale data from the evacuated node."}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_3a9feb51","line":3337,"range":{"start_line":3337,"start_character":23,"end_line":3337,"end_character":79},"in_reply_to":"3f79a3b5_f1df6d08","updated":"2018-10-12 15:54:02.000000000","message":"If this is already pulling the current state of the ports and putting them on the instance.info_cache, then _require_nw_info_update shouldn\u0027t need to get the ports from neutron again, right? If we can avoid the extraneous call to list_ports in _require_nw_info_update then I\u0027d be much more OK with this change since it would just be checking the cache we already have and *only* correcting port bindings if the cache shows we\u0027ve got busted ports.","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"732814f7ade84b6b088e72f69f451a1d2f4a806b","unresolved":false,"context_lines":[{"line_number":3335,"context_line":"            context, instance, bdms\u003dbdms)"},{"line_number":3336,"context_line":""},{"line_number":3337,"context_line":"        network_info \u003d self.network_api.get_instance_nw_info(context, instance)"},{"line_number":3338,"context_line":"        # If after a failed rebuild an instance is rebooted it\u0027s possible"},{"line_number":3339,"context_line":"        # that during this moment the neutron server is down. The neutron ports"},{"line_number":3340,"context_line":"        # are then not updated and has stale data from the evacuated node."},{"line_number":3341,"context_line":"        # Trigger setup of instance network in this case to update network"}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_d1c5913e","line":3338,"range":{"start_line":3338,"start_character":28,"end_line":3338,"end_character":35},"updated":"2018-10-11 22:10:15.000000000","message":"evacuate or rebuild? Because you say evacuate later in this comment.","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"59a3c030bdce54bc410197932fcc4d6e17593778","unresolved":false,"context_lines":[{"line_number":3335,"context_line":"            context, instance, bdms\u003dbdms)"},{"line_number":3336,"context_line":""},{"line_number":3337,"context_line":"        network_info \u003d self.network_api.get_instance_nw_info(context, instance)"},{"line_number":3338,"context_line":"        # If after a failed rebuild an instance is rebooted it\u0027s possible"},{"line_number":3339,"context_line":"        # that during this moment the neutron server is down. The neutron ports"},{"line_number":3340,"context_line":"        # are then not updated and has stale data from the evacuated node."},{"line_number":3341,"context_line":"        # Trigger setup of instance network in this case to update network"}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_a9935bb0","line":3338,"range":{"start_line":3338,"start_character":28,"end_line":3338,"end_character":35},"in_reply_to":"3f79a3b5_d1c5913e","updated":"2018-10-30 23:37:26.000000000","message":"should be evacuate.","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"732814f7ade84b6b088e72f69f451a1d2f4a806b","unresolved":false,"context_lines":[{"line_number":3336,"context_line":""},{"line_number":3337,"context_line":"        network_info \u003d self.network_api.get_instance_nw_info(context, instance)"},{"line_number":3338,"context_line":"        # If after a failed rebuild an instance is rebooted it\u0027s possible"},{"line_number":3339,"context_line":"        # that during this moment the neutron server is down. The neutron ports"},{"line_number":3340,"context_line":"        # are then not updated and has stale data from the evacuated node."},{"line_number":3341,"context_line":"        # Trigger setup of instance network in this case to update network"},{"line_number":3342,"context_line":"        # info."}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_71befdaf","line":3339,"range":{"start_line":3339,"start_character":15,"end_line":3339,"end_character":60},"updated":"2018-10-11 22:10:15.000000000","message":"Ew, couldn\u0027t this be a whack-a-mole case that\u0027s true during pretty much any operation? What makes reboot special?\n\nWe also have the heal instance info cache periodic task which would do a full refresh from neutron with this change:\n\nhttps://review.openstack.org/#/c/591607/\n\nBut that change was a bit controversial at the Stein PTG.","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"732814f7ade84b6b088e72f69f451a1d2f4a806b","unresolved":false,"context_lines":[{"line_number":3342,"context_line":"        # info."},{"line_number":3343,"context_line":"        if self._require_nw_info_update(context, instance):"},{"line_number":3344,"context_line":"            LOG.info(\"Updating ports in neutron\", instance\u003dinstance)"},{"line_number":3345,"context_line":"            self.network_api.setup_instance_network_on_host(context, instance,"},{"line_number":3346,"context_line":"                                                            self.host)"},{"line_number":3347,"context_line":"            network_info \u003d self.network_api.get_instance_nw_info(context,"},{"line_number":3348,"context_line":"                                                                 instance)"}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_917639d5","line":3345,"range":{"start_line":3345,"start_character":29,"end_line":3345,"end_character":59},"updated":"2018-10-11 22:10:15.000000000","message":"OK so this just updates the port binding to this host which should correct the vif_type and binding profile stuff, etc.","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"732814f7ade84b6b088e72f69f451a1d2f4a806b","unresolved":false,"context_lines":[{"line_number":3749,"context_line":"        admin_password \u003d (rescue_password if rescue_password else"},{"line_number":3750,"context_line":"                      utils.generate_password())"},{"line_number":3751,"context_line":""},{"line_number":3752,"context_line":"        network_info \u003d self.network_api.get_instance_nw_info(context, instance)"},{"line_number":3753,"context_line":""},{"line_number":3754,"context_line":"        rescue_image_meta \u003d self._get_rescue_image(context, instance,"},{"line_number":3755,"context_line":"                                                   rescue_image_ref)"}],"source_content_type":"text/x-python","patch_set":7,"id":"3f79a3b5_2c810e4e","line":3752,"range":{"start_line":3752,"start_character":8,"end_line":3752,"end_character":79},"updated":"2018-10-11 22:10:15.000000000","message":"You can also rescue an instance in ERROR state, which presumably could suffer from the same issue, right?","commit_id":"e844f5b87efe9dd70a93e2695c1773ce05bdb4f5"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"0291b4171a92505954b0332946548c8d971f264c","unresolved":false,"context_lines":[{"line_number":3344,"context_line":"            context, instance, bdms\u003dbdms)"},{"line_number":3345,"context_line":""},{"line_number":3346,"context_line":"        network_info \u003d self.network_api.get_instance_nw_info(context, instance)"},{"line_number":3347,"context_line":"        # If after a failed evacuate an instance is rebooted it\u0027s possible"},{"line_number":3348,"context_line":"        # that during this moment the neutron server is down. The neutron ports"},{"line_number":3349,"context_line":"        # are then not updated and has stale data from the evacuated node."},{"line_number":3350,"context_line":"        # Trigger setup of instance network in this case to update network"},{"line_number":3351,"context_line":"        # info."}],"source_content_type":"text/x-python","patch_set":11,"id":"3f79a3b5_2fe2569c","line":3348,"range":{"start_line":3347,"start_character":8,"end_line":3348,"end_character":61},"updated":"2018-12-19 23:45:02.000000000","message":"Sounds like bugs in the starlingx/titanium maintenance/orchestration code. I mean, how often is this expected to happen in real life where some automated system isn\u0027t clobbering everything? I\u0027m hesitant to add random \"oh crap did we break something? let\u0027s always check\" stuff. I would much rather try to deal with the failures at the point of failure if possible.\n\nBecause as noted in PS7 rescue would have the same issue, but you guys don\u0027t care about rescue so that\u0027s why you\u0027re not adding the spot check there. I just really don\u0027t like this spot check stuff in random places in the code based on how your automation tooling works.","commit_id":"108e338d5d76630fca81833bd501087f84fbc4c6"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"a25ca3a5998c06c48b875ba1d195b9837221a0bb","unresolved":false,"context_lines":[{"line_number":3103,"context_line":"                # later."},{"line_number":3104,"context_line":"                if isinstance(e, exception.VirtualInterfaceCreateException):"},{"line_number":3105,"context_line":"                    LOG.info(\"Updating ports in neutron\", instance\u003dinstance)"},{"line_number":3106,"context_line":"                    self.network_api.setup_instance_network_on_host("},{"line_number":3107,"context_line":"                        context, instance, self.host)"},{"line_number":3108,"context_line":"                self._set_migration_status(migration, \u0027failed\u0027)"},{"line_number":3109,"context_line":"                if evacuate or scheduled_node is not None:"},{"line_number":3110,"context_line":"                    rt.delete_allocation_for_evacuated_instance("}],"source_content_type":"text/x-python","patch_set":18,"id":"9fdfeff1_3df1d202","line":3107,"range":{"start_line":3106,"start_character":0,"end_line":3107,"end_character":53},"updated":"2019-02-05 00:28:03.000000000","message":"Operator can recover the instance by:\n    nova reset-state --active \u003cserver\u003e\n    nova rebuild \u003cserver\u003e \u003cimage\u003e\n\nCould retry self._do_rebuild_instance_with_claim after updating ports but that would require quite complex change: put self._do_rebuild_instance_with_claim in a loop.","commit_id":"ca93e059be7397ac2ba9175238e88a20d6ce59b7"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"c04044328d384e3ef881467ab10f01b035532fbe","unresolved":false,"context_lines":[{"line_number":6975,"context_line":"            action\u003dfields.NotificationAction.LIVE_MIGRATION_ROLLBACK_DEST,"},{"line_number":6976,"context_line":"            phase\u003dfields.NotificationPhase.END)"},{"line_number":6977,"context_line":""},{"line_number":6978,"context_line":"    def _require_nw_info_update(self, context, instance):"},{"line_number":6979,"context_line":"        \"\"\"Detect whether there is a mismatch in binding:host_id, or"},{"line_number":6980,"context_line":"        binding_failed or unbound binding:vif_type for any of the instances"},{"line_number":6981,"context_line":"        ports."}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_c65bf151","line":6978,"updated":"2019-02-06 21:48:33.000000000","message":"There is no test coverage for this.","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"8e755fd94a417a393758586772c271ffe60c7375","unresolved":false,"context_lines":[{"line_number":6975,"context_line":"            action\u003dfields.NotificationAction.LIVE_MIGRATION_ROLLBACK_DEST,"},{"line_number":6976,"context_line":"            phase\u003dfields.NotificationPhase.END)"},{"line_number":6977,"context_line":""},{"line_number":6978,"context_line":"    def _require_nw_info_update(self, context, instance):"},{"line_number":6979,"context_line":"        \"\"\"Detect whether there is a mismatch in binding:host_id, or"},{"line_number":6980,"context_line":"        binding_failed or unbound binding:vif_type for any of the instances"},{"line_number":6981,"context_line":"        ports."}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_da640c4c","line":6978,"in_reply_to":"9fdfeff1_c65bf151","updated":"2019-02-07 23:22:53.000000000","message":"Tests added","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"c04044328d384e3ef881467ab10f01b035532fbe","unresolved":false,"context_lines":[{"line_number":6983,"context_line":"        if not utils.is_neutron():"},{"line_number":6984,"context_line":"            return False"},{"line_number":6985,"context_line":""},{"line_number":6986,"context_line":"        search_opts \u003d {\u0027device_id\u0027: instance.uuid}"},{"line_number":6987,"context_line":"        ports \u003d self.network_api.list_ports(context, **search_opts)"},{"line_number":6988,"context_line":"        for p in ports[\u0027ports\u0027]:"},{"line_number":6989,"context_line":"            if p.get(\u0027binding:host_id\u0027) !\u003d self.host:"}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_8672c910","line":6986,"updated":"2019-02-06 21:48:33.000000000","message":"nit: we could further filter the response by adding:\n\n\u0027fields\u0027: [\u0027binding:host_id\u0027, \u0027binding:vif_type\u0027]\n\nto search_opts but I don\u0027t really know that it makes anything faster on the neutron side.","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"8e755fd94a417a393758586772c271ffe60c7375","unresolved":false,"context_lines":[{"line_number":6983,"context_line":"        if not utils.is_neutron():"},{"line_number":6984,"context_line":"            return False"},{"line_number":6985,"context_line":""},{"line_number":6986,"context_line":"        search_opts \u003d {\u0027device_id\u0027: instance.uuid}"},{"line_number":6987,"context_line":"        ports \u003d self.network_api.list_ports(context, **search_opts)"},{"line_number":6988,"context_line":"        for p in ports[\u0027ports\u0027]:"},{"line_number":6989,"context_line":"            if p.get(\u0027binding:host_id\u0027) !\u003d self.host:"}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_a9a1de97","line":6986,"in_reply_to":"9fdfeff1_8672c910","updated":"2019-02-07 23:22:53.000000000","message":"Done","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"c04044328d384e3ef881467ab10f01b035532fbe","unresolved":false,"context_lines":[{"line_number":7075,"context_line":"                # Fix potential mismatch in port binding if evacuation failed"},{"line_number":7076,"context_line":"                # after reassigning the port binding to the dest host but"},{"line_number":7077,"context_line":"                # before the instance host is changed."},{"line_number":7078,"context_line":"                if self._require_nw_info_update(context, instance):"},{"line_number":7079,"context_line":"                    LOG.info(\"Updating ports in neutron\", instance\u003dinstance)"},{"line_number":7080,"context_line":"                    self.network_api.setup_instance_network_on_host("},{"line_number":7081,"context_line":"                        context, instance, self.host)"}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_66580556","line":7078,"updated":"2019-02-06 21:48:33.000000000","message":"There is no test coverage that we call this during the periodic task.","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"8e755fd94a417a393758586772c271ffe60c7375","unresolved":false,"context_lines":[{"line_number":7075,"context_line":"                # Fix potential mismatch in port binding if evacuation failed"},{"line_number":7076,"context_line":"                # after reassigning the port binding to the dest host but"},{"line_number":7077,"context_line":"                # before the instance host is changed."},{"line_number":7078,"context_line":"                if self._require_nw_info_update(context, instance):"},{"line_number":7079,"context_line":"                    LOG.info(\"Updating ports in neutron\", instance\u003dinstance)"},{"line_number":7080,"context_line":"                    self.network_api.setup_instance_network_on_host("},{"line_number":7081,"context_line":"                        context, instance, self.host)"}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_7a5ba086","line":7078,"in_reply_to":"9fdfeff1_66580556","updated":"2019-02-07 23:22:53.000000000","message":"test coverage added","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"c04044328d384e3ef881467ab10f01b035532fbe","unresolved":false,"context_lines":[{"line_number":7077,"context_line":"                # before the instance host is changed."},{"line_number":7078,"context_line":"                if self._require_nw_info_update(context, instance):"},{"line_number":7079,"context_line":"                    LOG.info(\"Updating ports in neutron\", instance\u003dinstance)"},{"line_number":7080,"context_line":"                    self.network_api.setup_instance_network_on_host("},{"line_number":7081,"context_line":"                        context, instance, self.host)"},{"line_number":7082,"context_line":"                # Call to network API to get instance info.. this will"},{"line_number":7083,"context_line":"                # force an update to the instance\u0027s info_cache"}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_c6d691c4","line":7080,"updated":"2019-02-06 21:48:33.000000000","message":"There are some generic checks above for BUILDING and DELETING but I would think here we want to avoid messing with port bindings if the task_state is not None, in other words only mess with the instance port bindings if it\u0027s in steady state, similar to how _sync_power_states works.","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"8e755fd94a417a393758586772c271ffe60c7375","unresolved":false,"context_lines":[{"line_number":7077,"context_line":"                # before the instance host is changed."},{"line_number":7078,"context_line":"                if self._require_nw_info_update(context, instance):"},{"line_number":7079,"context_line":"                    LOG.info(\"Updating ports in neutron\", instance\u003dinstance)"},{"line_number":7080,"context_line":"                    self.network_api.setup_instance_network_on_host("},{"line_number":7081,"context_line":"                        context, instance, self.host)"},{"line_number":7082,"context_line":"                # Call to network API to get instance info.. this will"},{"line_number":7083,"context_line":"                # force an update to the instance\u0027s info_cache"}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_0987cad9","line":7080,"in_reply_to":"9fdfeff1_c6d691c4","updated":"2019-02-07 23:22:53.000000000","message":"Done","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"a534aa74b83393a3597a3c5ecb32000cfb675016","unresolved":false,"context_lines":[{"line_number":7020,"context_line":"                       \u0027fields\u0027: [\u0027binding:host_id\u0027, \u0027binding:vif_type\u0027]}"},{"line_number":7021,"context_line":"        ports \u003d self.network_api.list_ports(context, **search_opts)"},{"line_number":7022,"context_line":"        for p in ports[\u0027ports\u0027]:"},{"line_number":7023,"context_line":"            if p.get(\u0027binding:host_id\u0027) !\u003d self.host:"},{"line_number":7024,"context_line":"                return True"},{"line_number":7025,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":7026,"context_line":"            if (vif_type \u003d\u003d network_model.VIF_TYPE_UNBOUND or"},{"line_number":7027,"context_line":"                    vif_type \u003d\u003d network_model.VIF_TYPE_BINDING_FAILED):"}],"source_content_type":"text/x-python","patch_set":21,"id":"9fdfeff1_af5fba4b","line":7024,"range":{"start_line":7023,"start_character":0,"end_line":7024,"end_character":27},"updated":"2019-03-04 16:38:20.000000000","message":"if the binding host is the new target host, and this code runs on the original source host after the original source host is \"fixed\", this will return True which will cause line 7117 to execute which will attempt to set up the instance network to *the original source host* regardless of whether the vif_type is UNBOUND or BINDING_FAILED. In other words, if the evacuation succeeded but the source host doesn\u0027t realize it, this would set the instance\u0027s ports back to the original source host...\n\nIs that what is really wanted here or should this also check to see if the vif type is UNBOUND/BINDING_FAILED?","commit_id":"052f60896dc9c2ecb6908154ea5739172cd690ad"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"dc47b35abf73fe08cb9ea6ccf43a2cb19d72f357","unresolved":false,"context_lines":[{"line_number":7020,"context_line":"                       \u0027fields\u0027: [\u0027binding:host_id\u0027, \u0027binding:vif_type\u0027]}"},{"line_number":7021,"context_line":"        ports \u003d self.network_api.list_ports(context, **search_opts)"},{"line_number":7022,"context_line":"        for p in ports[\u0027ports\u0027]:"},{"line_number":7023,"context_line":"            if p.get(\u0027binding:host_id\u0027) !\u003d self.host:"},{"line_number":7024,"context_line":"                return True"},{"line_number":7025,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":7026,"context_line":"            if (vif_type \u003d\u003d network_model.VIF_TYPE_UNBOUND or"},{"line_number":7027,"context_line":"                    vif_type \u003d\u003d network_model.VIF_TYPE_BINDING_FAILED):"}],"source_content_type":"text/x-python","patch_set":21,"id":"9fdfeff1_dfb98779","line":7024,"range":{"start_line":7023,"start_character":0,"end_line":7024,"end_character":27},"in_reply_to":"9fdfeff1_1f9dcfba","updated":"2019-03-04 20:02:46.000000000","message":"OK, yeah, I guess... I wonder what would happen on the destination host if there was a disagreement about which thing owned the port (i.e. will there be some sort of race between the source and dest host thinking their instance info cache entry is the right one...)","commit_id":"052f60896dc9c2ecb6908154ea5739172cd690ad"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"4860721d330fa5345db50119a02c93f4a80c2c66","unresolved":false,"context_lines":[{"line_number":7020,"context_line":"                       \u0027fields\u0027: [\u0027binding:host_id\u0027, \u0027binding:vif_type\u0027]}"},{"line_number":7021,"context_line":"        ports \u003d self.network_api.list_ports(context, **search_opts)"},{"line_number":7022,"context_line":"        for p in ports[\u0027ports\u0027]:"},{"line_number":7023,"context_line":"            if p.get(\u0027binding:host_id\u0027) !\u003d self.host:"},{"line_number":7024,"context_line":"                return True"},{"line_number":7025,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":7026,"context_line":"            if (vif_type \u003d\u003d network_model.VIF_TYPE_UNBOUND or"},{"line_number":7027,"context_line":"                    vif_type \u003d\u003d network_model.VIF_TYPE_BINDING_FAILED):"}],"source_content_type":"text/x-python","patch_set":21,"id":"9fdfeff1_1f9dcfba","line":7024,"range":{"start_line":7023,"start_character":0,"end_line":7024,"end_character":27},"in_reply_to":"9fdfeff1_af5fba4b","updated":"2019-03-04 19:27:54.000000000","message":"This should only get called on instances that are on this host per L7057, so if the instance is tracked on this host but the binding:host_id is for another host, then the port binding host is likely the wrong one.","commit_id":"052f60896dc9c2ecb6908154ea5739172cd690ad"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"e657fe7fcf4f32df5cde8b1d6a48852a463331db","unresolved":false,"context_lines":[{"line_number":7126,"context_line":"                       \u0027fields\u0027: [\u0027binding:host_id\u0027, \u0027binding:vif_type\u0027]}"},{"line_number":7127,"context_line":"        ports \u003d self.network_api.list_ports(context, **search_opts)"},{"line_number":7128,"context_line":"        for p in ports[\u0027ports\u0027]:"},{"line_number":7129,"context_line":"            if p.get(\u0027binding:host_id\u0027) !\u003d self.host:"},{"line_number":7130,"context_line":"                return True"},{"line_number":7131,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":7132,"context_line":"            if (vif_type \u003d\u003d network_model.VIF_TYPE_UNBOUND or"}],"source_content_type":"text/x-python","patch_set":22,"id":"9fdfeff1_0011f0af","line":7129,"updated":"2019-03-04 22:26:46.000000000","message":"there are case today where we dont update the instance host correctly today\nsuch as when there is an uncaugt exception reasidn in post_live_migration at dest. in which case the vm is running on dest but we have not updated that in the db correctly. so im not sure  this is correct in all cases.\nthis could mismatch because self.host is wrong or becuase binding:host_id is wrong and we cant be sure which.","commit_id":"542635034882e1b6897e1935f09d6feb6e77d1ce"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"f1472dde31c223642b078835616d9dea16b66f64","unresolved":false,"context_lines":[{"line_number":7126,"context_line":"                       \u0027fields\u0027: [\u0027binding:host_id\u0027, \u0027binding:vif_type\u0027]}"},{"line_number":7127,"context_line":"        ports \u003d self.network_api.list_ports(context, **search_opts)"},{"line_number":7128,"context_line":"        for p in ports[\u0027ports\u0027]:"},{"line_number":7129,"context_line":"            if p.get(\u0027binding:host_id\u0027) !\u003d self.host:"},{"line_number":7130,"context_line":"                return True"},{"line_number":7131,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":7132,"context_line":"            if (vif_type \u003d\u003d network_model.VIF_TYPE_UNBOUND or"}],"source_content_type":"text/x-python","patch_set":22,"id":"9fdfeff1_60debcdb","line":7129,"in_reply_to":"9fdfeff1_0011f0af","updated":"2019-03-04 22:36:13.000000000","message":"that\u0027s almost exactly what I described in the last revision comment:\n\nhttps://review.openstack.org/#/c/603844/21/nova/compute/manager.py\n\nMatt said it wasn\u0027t something we needed to worry about.","commit_id":"542635034882e1b6897e1935f09d6feb6e77d1ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5666489f80c34fb1a9d7634df175d180aa34a8cb","unresolved":false,"context_lines":[{"line_number":7126,"context_line":"                       \u0027fields\u0027: [\u0027binding:host_id\u0027, \u0027binding:vif_type\u0027]}"},{"line_number":7127,"context_line":"        ports \u003d self.network_api.list_ports(context, **search_opts)"},{"line_number":7128,"context_line":"        for p in ports[\u0027ports\u0027]:"},{"line_number":7129,"context_line":"            if p.get(\u0027binding:host_id\u0027) !\u003d self.host:"},{"line_number":7130,"context_line":"                return True"},{"line_number":7131,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":7132,"context_line":"            if (vif_type \u003d\u003d network_model.VIF_TYPE_UNBOUND or"}],"source_content_type":"text/x-python","patch_set":22,"id":"9fdfeff1_66c00ae2","line":7129,"in_reply_to":"9fdfeff1_1456112c","updated":"2019-03-05 11:35:26.000000000","message":"yes however if we fail after the port bindings are create on the dest but before they are activated on the dest when \nnova delete the destination port binding as part of rollback it does not clean up the neuton port bindigns properly in the neuton database. this is tracked by https://bugs.launchpad.net/neutron/+bug/1815345\nand adressed by\nhttps://review.openstack.org/#/c/634276/\n\n\nas a result trying to evacuate or migratie the instance again to the same host failed due to a conflict in the neutron database and port bidning would fail.\n\nim still a little un sure about this line but ill change my -1 to a +1 and look at modifying the post live migration code to always set the instance host correctly. we should fix that anyway. this should be fine 95% of the time and when its not fine the fact the instance.host is wrong in the nova db means the vm in question has bigger issues then potentially losing network connectivity such as it can rebooted safely or otherwise be managed properly by nova.","commit_id":"542635034882e1b6897e1935f09d6feb6e77d1ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"2d66ccb669ae5d9bcf0f0827c5529967ed47e7e1","unresolved":false,"context_lines":[{"line_number":7126,"context_line":"                       \u0027fields\u0027: [\u0027binding:host_id\u0027, \u0027binding:vif_type\u0027]}"},{"line_number":7127,"context_line":"        ports \u003d self.network_api.list_ports(context, **search_opts)"},{"line_number":7128,"context_line":"        for p in ports[\u0027ports\u0027]:"},{"line_number":7129,"context_line":"            if p.get(\u0027binding:host_id\u0027) !\u003d self.host:"},{"line_number":7130,"context_line":"                return True"},{"line_number":7131,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":7132,"context_line":"            if (vif_type \u003d\u003d network_model.VIF_TYPE_UNBOUND or"}],"source_content_type":"text/x-python","patch_set":22,"id":"9fdfeff1_d41089ae","line":7129,"in_reply_to":"9fdfeff1_54f6d992","updated":"2019-03-04 23:27:45.000000000","message":"by the way im not worried about the evacuate case.\nim worried about what happens if in a normal migration we get a timeout or other error from cinder and we end up messing up the network connectivity on the instance that was actully migrated.\n\nthat said that instace while working is kind of broke as nova things it on a different host but that is a seperate bug.","commit_id":"542635034882e1b6897e1935f09d6feb6e77d1ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"4a0f1a5bc7bcde606d62f1e1b3e5cdcd07874608","unresolved":false,"context_lines":[{"line_number":7126,"context_line":"                       \u0027fields\u0027: [\u0027binding:host_id\u0027, \u0027binding:vif_type\u0027]}"},{"line_number":7127,"context_line":"        ports \u003d self.network_api.list_ports(context, **search_opts)"},{"line_number":7128,"context_line":"        for p in ports[\u0027ports\u0027]:"},{"line_number":7129,"context_line":"            if p.get(\u0027binding:host_id\u0027) !\u003d self.host:"},{"line_number":7130,"context_line":"                return True"},{"line_number":7131,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":7132,"context_line":"            if (vif_type \u003d\u003d network_model.VIF_TYPE_UNBOUND or"}],"source_content_type":"text/x-python","patch_set":22,"id":"9fdfeff1_54f6d992","line":7129,"in_reply_to":"9fdfeff1_60debcdb","updated":"2019-03-04 23:23:49.000000000","message":"right but we dont update teh instace.host until here in post_live_migration_at_dest.\nhttps://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6911-L6916\n\nif we raise an exception form calling self._get_instance_block_device_info\nhttps://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6888-L6889\n\nwe will never update the host\n\nthat call hits the cinder api and can definetly raise exceptions from here https://github.com/openstack/nova/blob/master/nova/volume/cinder.py#L781-L804\n\nwe have had downstream customer issue related to this in the past so i do not like to assume that the host id in the nova db is actully set to the host the vm is running on if there is a failure in post_live_migarte before we updated it.","commit_id":"542635034882e1b6897e1935f09d6feb6e77d1ce"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"492bab7430828c7979ebdbb0d0826e7ef8fcc678","unresolved":false,"context_lines":[{"line_number":7126,"context_line":"                       \u0027fields\u0027: [\u0027binding:host_id\u0027, \u0027binding:vif_type\u0027]}"},{"line_number":7127,"context_line":"        ports \u003d self.network_api.list_ports(context, **search_opts)"},{"line_number":7128,"context_line":"        for p in ports[\u0027ports\u0027]:"},{"line_number":7129,"context_line":"            if p.get(\u0027binding:host_id\u0027) !\u003d self.host:"},{"line_number":7130,"context_line":"                return True"},{"line_number":7131,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":7132,"context_line":"            if (vif_type \u003d\u003d network_model.VIF_TYPE_UNBOUND or"}],"source_content_type":"text/x-python","patch_set":22,"id":"9fdfeff1_3281d84e","line":7129,"in_reply_to":"9fdfeff1_66c00ae2","updated":"2019-03-05 14:19:26.000000000","message":"I\u0027m reminded of https://review.openstack.org/#/c/594139/ as well.","commit_id":"542635034882e1b6897e1935f09d6feb6e77d1ce"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"7d8a75d7f9aaf666331651ffbc4d6d254fc192fd","unresolved":false,"context_lines":[{"line_number":7126,"context_line":"                       \u0027fields\u0027: [\u0027binding:host_id\u0027, \u0027binding:vif_type\u0027]}"},{"line_number":7127,"context_line":"        ports \u003d self.network_api.list_ports(context, **search_opts)"},{"line_number":7128,"context_line":"        for p in ports[\u0027ports\u0027]:"},{"line_number":7129,"context_line":"            if p.get(\u0027binding:host_id\u0027) !\u003d self.host:"},{"line_number":7130,"context_line":"                return True"},{"line_number":7131,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":7132,"context_line":"            if (vif_type \u003d\u003d network_model.VIF_TYPE_UNBOUND or"}],"source_content_type":"text/x-python","patch_set":22,"id":"9fdfeff1_1456112c","line":7129,"in_reply_to":"9fdfeff1_d41089ae","updated":"2019-03-04 23:48:12.000000000","message":"For the live migration thing, if we fail during live migration after port bindings were changed to the dest host but before we changed the instance.host \u003d dest, then doesn\u0027t _rollback_live_migration update the port binding to point back at the source host?","commit_id":"542635034882e1b6897e1935f09d6feb6e77d1ce"}],"nova/network/neutronv2/api.py":[{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"c04044328d384e3ef881467ab10f01b035532fbe","unresolved":false,"context_lines":[{"line_number":3196,"context_line":"            # the binding is clearly wrong due to previous lost messages."},{"line_number":3197,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":3198,"context_line":"            if ((p.get(BINDING_HOST_ID) !\u003d host) or"},{"line_number":3199,"context_line":"                    (vif_type \u003d\u003d network_model.VIF_TYPE_UNBOUND) or"},{"line_number":3200,"context_line":"                    (vif_type \u003d\u003d network_model.VIF_TYPE_BINDING_FAILED)):"},{"line_number":3201,"context_line":"                updates[BINDING_HOST_ID] \u003d host"},{"line_number":3202,"context_line":"                # If the host changed, the AZ could have also changed so we"},{"line_number":3203,"context_line":"                # need to update the device_owner."}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_c68451b4","line":3200,"range":{"start_line":3199,"start_character":20,"end_line":3200,"end_character":71},"updated":"2019-02-06 21:48:33.000000000","message":"nit: define a constant or variable outside the for loop for these bad vif states and then just do an \u0027in\u0027 check:\n\nif p.get(BINDING_HOST_ID) !\u003d host or vif_type in FAILED_VIF_TYPES:\n   ...","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"8e755fd94a417a393758586772c271ffe60c7375","unresolved":false,"context_lines":[{"line_number":3196,"context_line":"            # the binding is clearly wrong due to previous lost messages."},{"line_number":3197,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":3198,"context_line":"            if ((p.get(BINDING_HOST_ID) !\u003d host) or"},{"line_number":3199,"context_line":"                    (vif_type \u003d\u003d network_model.VIF_TYPE_UNBOUND) or"},{"line_number":3200,"context_line":"                    (vif_type \u003d\u003d network_model.VIF_TYPE_BINDING_FAILED)):"},{"line_number":3201,"context_line":"                updates[BINDING_HOST_ID] \u003d host"},{"line_number":3202,"context_line":"                # If the host changed, the AZ could have also changed so we"},{"line_number":3203,"context_line":"                # need to update the device_owner."}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_89e09a98","line":3200,"range":{"start_line":3199,"start_character":20,"end_line":3200,"end_character":71},"in_reply_to":"9fdfeff1_c68451b4","updated":"2019-02-07 23:22:53.000000000","message":"Done","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"e657fe7fcf4f32df5cde8b1d6a48852a463331db","unresolved":false,"context_lines":[{"line_number":3280,"context_line":"            binding_profile \u003d _get_binding_profile(p)"},{"line_number":3281,"context_line":""},{"line_number":3282,"context_line":"            # We need to update the port binding if the host has changed or if"},{"line_number":3283,"context_line":"            # the binding is clearly wrong due to previous lost messages."},{"line_number":3284,"context_line":"            vif_type \u003d p.get(\u0027binding:vif_type\u0027)"},{"line_number":3285,"context_line":"            if p.get(BINDING_HOST_ID) !\u003d host or vif_type in FAILED_VIF_TYPES:"},{"line_number":3286,"context_line":"                # TODO(gibi): To support ports with resource request during"}],"source_content_type":"text/x-python","patch_set":22,"id":"9fdfeff1_c00a0878","line":3283,"updated":"2019-03-04 22:26:46.000000000","message":"how do you know its \"clearly wrong\" there are valid cases where binding could fail. in which case we might not want to just blindly retry binding over and over.","commit_id":"542635034882e1b6897e1935f09d6feb6e77d1ce"}],"nova/tests/unit/compute/test_compute.py":[{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"8f3e7b3849ce86f2286c88f43a59dd1b38987e4b","unresolved":false,"context_lines":[{"line_number":7156,"context_line":"                       \u0027fields\u0027: [\u0027binding:host_id\u0027, \u0027binding:vif_type\u0027]}"},{"line_number":7157,"context_line":""},{"line_number":7158,"context_line":"        val \u003d self.compute._require_nw_info_update(ctxt, instance)"},{"line_number":7159,"context_line":"        self.assertFalse(val)"},{"line_number":7160,"context_line":"        mock_list_ports.assert_called_once_with(ctxt, **search_opts)"},{"line_number":7161,"context_line":""},{"line_number":7162,"context_line":"    @mock.patch(\u0027nova.network.neutronv2.api.API.list_ports\u0027)"}],"source_content_type":"text/x-python","patch_set":20,"id":"9fdfeff1_42e912b5","line":7159,"updated":"2019-02-08 01:57:16.000000000","message":"nit: a comment about why the result is false would be good, or a docstring on the test (just explain that the vif_type is not unbound or binding_failed).","commit_id":"415d7448e88cafdd103ac46f8c0e44a090be31c6"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"871de2ffdfacc030581648a751aadbd0478fc656","unresolved":false,"context_lines":[{"line_number":7156,"context_line":"                       \u0027fields\u0027: [\u0027binding:host_id\u0027, \u0027binding:vif_type\u0027]}"},{"line_number":7157,"context_line":""},{"line_number":7158,"context_line":"        val \u003d self.compute._require_nw_info_update(ctxt, instance)"},{"line_number":7159,"context_line":"        self.assertFalse(val)"},{"line_number":7160,"context_line":"        mock_list_ports.assert_called_once_with(ctxt, **search_opts)"},{"line_number":7161,"context_line":""},{"line_number":7162,"context_line":"    @mock.patch(\u0027nova.network.neutronv2.api.API.list_ports\u0027)"}],"source_content_type":"text/x-python","patch_set":20,"id":"9fdfeff1_0dffbb67","line":7159,"in_reply_to":"9fdfeff1_42e912b5","updated":"2019-02-08 04:00:04.000000000","message":"Done","commit_id":"415d7448e88cafdd103ac46f8c0e44a090be31c6"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"8f3e7b3849ce86f2286c88f43a59dd1b38987e4b","unresolved":false,"context_lines":[{"line_number":7189,"context_line":"                            network_model.VIF_TYPE_BINDING_FAILED)"},{"line_number":7190,"context_line":""},{"line_number":7191,"context_line":"        for vif_type in FAILED_VIF_TYPES:"},{"line_number":7192,"context_line":"            mock_list_ports.return_value \u003d {\u0027ports\u0027: ["},{"line_number":7193,"context_line":"                {\u0027binding:host_id\u0027: self.compute.host,"},{"line_number":7194,"context_line":"                 \u0027binding:vif_type\u0027: vif_type}"},{"line_number":7195,"context_line":"            ]}"}],"source_content_type":"text/x-python","patch_set":20,"id":"9fdfeff1_02cb0a4a","line":7192,"range":{"start_line":7192,"start_character":12,"end_line":7192,"end_character":40},"updated":"2019-02-08 01:57:16.000000000","message":"Don\u0027t you have to reset this mock between calls? Maybe not.","commit_id":"415d7448e88cafdd103ac46f8c0e44a090be31c6"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"871de2ffdfacc030581648a751aadbd0478fc656","unresolved":false,"context_lines":[{"line_number":7189,"context_line":"                            network_model.VIF_TYPE_BINDING_FAILED)"},{"line_number":7190,"context_line":""},{"line_number":7191,"context_line":"        for vif_type in FAILED_VIF_TYPES:"},{"line_number":7192,"context_line":"            mock_list_ports.return_value \u003d {\u0027ports\u0027: ["},{"line_number":7193,"context_line":"                {\u0027binding:host_id\u0027: self.compute.host,"},{"line_number":7194,"context_line":"                 \u0027binding:vif_type\u0027: vif_type}"},{"line_number":7195,"context_line":"            ]}"}],"source_content_type":"text/x-python","patch_set":20,"id":"9fdfeff1_6d2d27fc","line":7192,"range":{"start_line":7192,"start_character":12,"end_line":7192,"end_character":40},"in_reply_to":"9fdfeff1_02cb0a4a","updated":"2019-02-08 04:00:04.000000000","message":"added reset to correct call history.","commit_id":"415d7448e88cafdd103ac46f8c0e44a090be31c6"}],"nova/tests/unit/network/test_neutronv2.py":[{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"c04044328d384e3ef881467ab10f01b035532fbe","unresolved":false,"context_lines":[{"line_number":4332,"context_line":"                         neutronapi.BINDING_PROFILE: binding_profile,"},{"line_number":4333,"context_line":"                         neutronapi.BINDING_HOST_ID: instance.host},"},{"line_number":4334,"context_line":"                        {\u0027id\u0027: \u0027fake-port-2\u0027,"},{"line_number":4335,"context_line":"                         \u0027binding:vif_type\u0027: \u0027unbound\u0027,"},{"line_number":4336,"context_line":"                         neutronapi.BINDING_PROFILE: binding_profile,"},{"line_number":4337,"context_line":"                         neutronapi.BINDING_HOST_ID: instance.host}"},{"line_number":4338,"context_line":"        ]}"}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_c6ad3132","line":4335,"range":{"start_line":4335,"start_character":45,"end_line":4335,"end_character":54},"updated":"2019-02-06 21:48:33.000000000","message":"use your VIF_TYPE_UNBOUND constant here","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"8e755fd94a417a393758586772c271ffe60c7375","unresolved":false,"context_lines":[{"line_number":4332,"context_line":"                         neutronapi.BINDING_PROFILE: binding_profile,"},{"line_number":4333,"context_line":"                         neutronapi.BINDING_HOST_ID: instance.host},"},{"line_number":4334,"context_line":"                        {\u0027id\u0027: \u0027fake-port-2\u0027,"},{"line_number":4335,"context_line":"                         \u0027binding:vif_type\u0027: \u0027unbound\u0027,"},{"line_number":4336,"context_line":"                         neutronapi.BINDING_PROFILE: binding_profile,"},{"line_number":4337,"context_line":"                         neutronapi.BINDING_HOST_ID: instance.host}"},{"line_number":4338,"context_line":"        ]}"}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_e96b66ee","line":4335,"range":{"start_line":4335,"start_character":45,"end_line":4335,"end_character":54},"in_reply_to":"9fdfeff1_c6ad3132","updated":"2019-02-07 23:22:53.000000000","message":"Done","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"c04044328d384e3ef881467ab10f01b035532fbe","unresolved":false,"context_lines":[{"line_number":4354,"context_line":"                                     }})"},{"line_number":4355,"context_line":""},{"line_number":4356,"context_line":"    @mock.patch.object(neutronapi, \u0027get_client\u0027, return_value\u003dmock.Mock())"},{"line_number":4357,"context_line":"    def test_update_port_bindings_for_instance_with_same_host_failed_vif_type("},{"line_number":4358,"context_line":"        self, get_client_mock):"},{"line_number":4359,"context_line":"        instance \u003d fake_instance.fake_instance_obj(self.context)"},{"line_number":4360,"context_line":"        self.api._has_port_binding_extension \u003d mock.Mock(return_value\u003dTrue)"}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_e644f576","line":4357,"updated":"2019-02-06 21:48:33.000000000","message":"This test and the one above are 99% the same except for the vif_type used for fake-port-2, just write a single test that uses ddt (or a simple for loop) for the different value.","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"8e755fd94a417a393758586772c271ffe60c7375","unresolved":false,"context_lines":[{"line_number":4354,"context_line":"                                     }})"},{"line_number":4355,"context_line":""},{"line_number":4356,"context_line":"    @mock.patch.object(neutronapi, \u0027get_client\u0027, return_value\u003dmock.Mock())"},{"line_number":4357,"context_line":"    def test_update_port_bindings_for_instance_with_same_host_failed_vif_type("},{"line_number":4358,"context_line":"        self, get_client_mock):"},{"line_number":4359,"context_line":"        instance \u003d fake_instance.fake_instance_obj(self.context)"},{"line_number":4360,"context_line":"        self.api._has_port_binding_extension \u003d mock.Mock(return_value\u003dTrue)"}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_ba791a30","line":4357,"in_reply_to":"9fdfeff1_e644f576","updated":"2019-02-07 23:22:53.000000000","message":"Done. Combined 2 TCs in a for loop.","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"c04044328d384e3ef881467ab10f01b035532fbe","unresolved":false,"context_lines":[{"line_number":4398,"context_line":"                           neutronapi.MIGRATING_ATTR: \u0027my-dest-host\u0027}"},{"line_number":4399,"context_line":"        fake_ports \u003d {\u0027ports\u0027: ["},{"line_number":4400,"context_line":"                        {\u0027id\u0027: \u0027fake-port-1\u0027,"},{"line_number":4401,"context_line":"                         \u0027binding:vif_type\u0027: \u0027unbound\u0027,"},{"line_number":4402,"context_line":"                         neutronapi.BINDING_PROFILE: binding_profile,"},{"line_number":4403,"context_line":"                         neutronapi.BINDING_HOST_ID: instance.host},"},{"line_number":4404,"context_line":"        ]}"}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_e66b95ee","line":4401,"range":{"start_line":4401,"start_character":45,"end_line":4401,"end_character":54},"updated":"2019-02-06 21:48:33.000000000","message":"use VIF_TYPE_UNBOUND","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"8e755fd94a417a393758586772c271ffe60c7375","unresolved":false,"context_lines":[{"line_number":4398,"context_line":"                           neutronapi.MIGRATING_ATTR: \u0027my-dest-host\u0027}"},{"line_number":4399,"context_line":"        fake_ports \u003d {\u0027ports\u0027: ["},{"line_number":4400,"context_line":"                        {\u0027id\u0027: \u0027fake-port-1\u0027,"},{"line_number":4401,"context_line":"                         \u0027binding:vif_type\u0027: \u0027unbound\u0027,"},{"line_number":4402,"context_line":"                         neutronapi.BINDING_PROFILE: binding_profile,"},{"line_number":4403,"context_line":"                         neutronapi.BINDING_HOST_ID: instance.host},"},{"line_number":4404,"context_line":"        ]}"}],"source_content_type":"text/x-python","patch_set":19,"id":"9fdfeff1_1a75c698","line":4401,"range":{"start_line":4401,"start_character":45,"end_line":4401,"end_character":54},"in_reply_to":"9fdfeff1_e66b95ee","updated":"2019-02-07 23:22:53.000000000","message":"Done","commit_id":"841b5f78ebe126d9d5b545258f8feb45422e273b"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"8f3e7b3849ce86f2286c88f43a59dd1b38987e4b","unresolved":false,"context_lines":[{"line_number":4349,"context_line":"                                                       instance.host)"},{"line_number":4350,"context_line":"            # Assert that update_port was called on the port with a"},{"line_number":4351,"context_line":"            # failed vif_type and MIGRATING_ATTR is removed"},{"line_number":4352,"context_line":"            update_port_mock.assert_called_once_with("},{"line_number":4353,"context_line":"                \u0027fake-port-2\u0027,"},{"line_number":4354,"context_line":"                {\u0027port\u0027: {neutronapi.BINDING_HOST_ID: instance.host,"},{"line_number":4355,"context_line":"                          neutronapi.BINDING_PROFILE: {"}],"source_content_type":"text/x-python","patch_set":20,"id":"9fdfeff1_c2ed02b8","line":4352,"range":{"start_line":4352,"start_character":29,"end_line":4352,"end_character":52},"updated":"2019-02-08 01:57:16.000000000","message":"I would expect this to fail on the 2nd iteration of the loop because you are asserting the mock is called once but it\u0027s actually called twice if you don\u0027t reset the mock between iterations.","commit_id":"415d7448e88cafdd103ac46f8c0e44a090be31c6"},{"author":{"_account_id":28433,"name":"Jack Ding","email":"jackding@gmail.com","username":"jackding"},"change_message_id":"871de2ffdfacc030581648a751aadbd0478fc656","unresolved":false,"context_lines":[{"line_number":4349,"context_line":"                                                       instance.host)"},{"line_number":4350,"context_line":"            # Assert that update_port was called on the port with a"},{"line_number":4351,"context_line":"            # failed vif_type and MIGRATING_ATTR is removed"},{"line_number":4352,"context_line":"            update_port_mock.assert_called_once_with("},{"line_number":4353,"context_line":"                \u0027fake-port-2\u0027,"},{"line_number":4354,"context_line":"                {\u0027port\u0027: {neutronapi.BINDING_HOST_ID: instance.host,"},{"line_number":4355,"context_line":"                          neutronapi.BINDING_PROFILE: {"}],"source_content_type":"text/x-python","patch_set":20,"id":"9fdfeff1_0d825bc9","line":4352,"range":{"start_line":4352,"start_character":29,"end_line":4352,"end_character":52},"in_reply_to":"9fdfeff1_c2ed02b8","updated":"2019-02-08 04:00:04.000000000","message":"Since update_port_mock is created inside the loop so no reset is needed. However I have moved the creation of the mocks out of the loop and added reset inside loop to make it more efficient.","commit_id":"415d7448e88cafdd103ac46f8c0e44a090be31c6"}]}