)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"d38e93a23a078015240a92b24f9044b4bf10d742","unresolved":false,"context_lines":[{"line_number":4,"context_line":"Commit:     Nikolay Fedotov \u003cnfedotov@cisco.com\u003e"},{"line_number":5,"context_line":"CommitDate: 2019-04-23 16:02:27 +0300"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Fix disappearing ironic hypervisors after rebuilding hashring"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"Missing record in host_mappings table causes nova to print"},{"line_number":10,"context_line":"\"Unable to find service\" DEBUG message (see below) and"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"ffb9cba7_f9daea48","line":7,"updated":"2019-04-29 19:09:38.000000000","message":"I think what you mean here is not that rebuilding the hash ring causes a problem, but adding a compute service that isn\u0027t mapped, will (after a hashring rebalance), cause the ironic nodes to be assigned to an un-mapped service.\n\nAs Matt said, the solution is to use the \"by service\" discovery method at the point of adding the new service.","commit_id":"0a7416bbaf2a6d014273c4aa7298c364e5062506"},{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"ddfef9714369547472e3cdf9cbb4311bb2ab9add","unresolved":false,"context_lines":[{"line_number":4,"context_line":"Commit:     Nikolay Fedotov \u003cnfedotov@cisco.com\u003e"},{"line_number":5,"context_line":"CommitDate: 2019-04-23 16:02:27 +0300"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Fix disappearing ironic hypervisors after rebuilding hashring"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"Missing record in host_mappings table causes nova to print"},{"line_number":10,"context_line":"\"Unable to find service\" DEBUG message (see below) and"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"ffb9cba7_c7960af1","line":7,"in_reply_to":"ffb9cba7_0596684d","updated":"2019-04-30 23:33:26.000000000","message":"compute_nodes !\u003d services where binary\u003dnova-compute. For ironic, a nova-compute service can host N number of ironic nodes which are modeled as compute_nodes in the cell DB. So ironic deployments are 1:M for service:compute_node but something like libvirt is 1:1.\n\nAs for the difference, you\u0027d use the --by-service option in the discover_hosts CLI:\n\nhttps://docs.openstack.org/nova/latest/cli/nova-manage.html#nova-cells-v2","commit_id":"0a7416bbaf2a6d014273c4aa7298c364e5062506"},{"author":{"_account_id":8767,"name":"Nikolay Fedotov","email":"nfedotov@cisco.com","username":"nfedotov"},"change_message_id":"e30e266bcf1fcd9e0d812d66b21836e1cd66abbc","unresolved":false,"context_lines":[{"line_number":4,"context_line":"Commit:     Nikolay Fedotov \u003cnfedotov@cisco.com\u003e"},{"line_number":5,"context_line":"CommitDate: 2019-04-23 16:02:27 +0300"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Fix disappearing ironic hypervisors after rebuilding hashring"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"Missing record in host_mappings table causes nova to print"},{"line_number":10,"context_line":"\"Unable to find service\" DEBUG message (see below) and"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"dfbec78f_bc6ccb36","line":7,"in_reply_to":"ffb9cba7_c7960af1","updated":"2019-05-06 17:24:28.000000000","message":"Thanks!\n--by-service flag did the trick! There is no need to \"cn.mapped \u003d 0\" because all mapping is already created. \nBut changes in host_mapping.py  are still valid and they fixing useless looping over  compute records having mapped\u003d0. Agree? Do you mind if I revert resource_tracker.py but leave host_mapping.py as is?","commit_id":"0a7416bbaf2a6d014273c4aa7298c364e5062506"},{"author":{"_account_id":8767,"name":"Nikolay Fedotov","email":"nfedotov@cisco.com","username":"nfedotov"},"change_message_id":"045e8c2ab12525071f258b38b9ad46c6403511f1","unresolved":false,"context_lines":[{"line_number":4,"context_line":"Commit:     Nikolay Fedotov \u003cnfedotov@cisco.com\u003e"},{"line_number":5,"context_line":"CommitDate: 2019-04-23 16:02:27 +0300"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Fix disappearing ironic hypervisors after rebuilding hashring"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"Missing record in host_mappings table causes nova to print"},{"line_number":10,"context_line":"\"Unable to find service\" DEBUG message (see below) and"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"ffb9cba7_0596684d","line":7,"in_reply_to":"ffb9cba7_f9daea48","updated":"2019-04-30 07:17:38.000000000","message":"Yes. I think so. Is the \"compute service\" same as \"compute node\" or record in ComputeNode table?\nMatt suggested to use _check_and_create_service_host_mappings. I do not see to much difference between that one and _check_and_create_node_host_mappings except two lines \"compute.mapped \u003d 1\n        compute.save()\"\n\nCould you explain where is \"point of adding the new service\" ?","commit_id":"0a7416bbaf2a6d014273c4aa7298c364e5062506"}],"nova/compute/resource_tracker.py":[{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"536f29a690d751a0bad74402d149406255745fdc","unresolved":false,"context_lines":[{"line_number":540,"context_line":"            LOG.info(\"ComputeNode %(name)s moving from %(old)s to %(new)s\","},{"line_number":541,"context_line":"                     {\"name\": nodename, \"old\": cn.host, \"new\": self.host})"},{"line_number":542,"context_line":"            cn.host \u003d self.host"},{"line_number":543,"context_line":"            cn.mapped \u003d 0"},{"line_number":544,"context_line":"            self.compute_nodes[nodename] \u003d cn"},{"line_number":545,"context_line":"            self._copy_resources(cn, resources)"},{"line_number":546,"context_line":"            self._setup_pci_tracker(context, cn, resources)"}],"source_content_type":"text/x-python","patch_set":1,"id":"ffb9cba7_3897a093","line":543,"updated":"2019-04-22 17:54:42.000000000","message":"There should definitely be a comment here about why this is needed.","commit_id":"19096c0c3b01b249c5d43112ae36d3eb653a0ffd"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"d38e93a23a078015240a92b24f9044b4bf10d742","unresolved":false,"context_lines":[{"line_number":541,"context_line":"                     {\"name\": nodename, \"old\": cn.host, \"new\": self.host})"},{"line_number":542,"context_line":"            cn.host \u003d self.host"},{"line_number":543,"context_line":"            # reset mapped because self.host may not have mapping yet."},{"line_number":544,"context_line":"            cn.mapped \u003d 0"},{"line_number":545,"context_line":"            self.compute_nodes[nodename] \u003d cn"},{"line_number":546,"context_line":"            self._copy_resources(cn, resources)"},{"line_number":547,"context_line":"            self._setup_pci_tracker(context, cn, resources)"}],"source_content_type":"text/x-python","patch_set":2,"id":"ffb9cba7_d972867b","line":544,"updated":"2019-04-29 19:09:38.000000000","message":"This is not okay. The compute node can\u0027t know if it\u0027s mapped or not, as that is state in the api database (which it has no ability to read).","commit_id":"0a7416bbaf2a6d014273c4aa7298c364e5062506"},{"author":{"_account_id":8767,"name":"Nikolay Fedotov","email":"nfedotov@cisco.com","username":"nfedotov"},"change_message_id":"045e8c2ab12525071f258b38b9ad46c6403511f1","unresolved":false,"context_lines":[{"line_number":541,"context_line":"                     {\"name\": nodename, \"old\": cn.host, \"new\": self.host})"},{"line_number":542,"context_line":"            cn.host \u003d self.host"},{"line_number":543,"context_line":"            # reset mapped because self.host may not have mapping yet."},{"line_number":544,"context_line":"            cn.mapped \u003d 0"},{"line_number":545,"context_line":"            self.compute_nodes[nodename] \u003d cn"},{"line_number":546,"context_line":"            self._copy_resources(cn, resources)"},{"line_number":547,"context_line":"            self._setup_pci_tracker(context, cn, resources)"}],"source_content_type":"text/x-python","patch_set":2,"id":"ffb9cba7_a5ef5c1c","line":544,"in_reply_to":"ffb9cba7_d972867b","updated":"2019-04-30 07:17:38.000000000","message":"Yes. For same reason cn.host value cannot be changed to self.host at line #542. Who knows if there is a host mapping for self.host? cn.mapped remains equal to 1.\nSetting cn.mapped to 0 give a chance to stumble upon existing HostMapping record for the self.host. It is handled properly in _check_and_create_node_host_mappings.","commit_id":"0a7416bbaf2a6d014273c4aa7298c364e5062506"}],"nova/objects/host_mapping.py":[{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"536f29a690d751a0bad74402d149406255745fdc","unresolved":false,"context_lines":[{"line_number":192,"context_line":"                cell_mapping\u003dcm)"},{"line_number":193,"context_line":"            host_mapping.create()"},{"line_number":194,"context_line":"            host_mappings.append(host_mapping)"},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"        compute.mapped \u003d 1"},{"line_number":197,"context_line":"        compute.save()"},{"line_number":198,"context_line":"    return host_mappings"}],"source_content_type":"text/x-python","patch_set":1,"id":"ffb9cba7_58d93481","line":195,"updated":"2019-04-22 17:54:42.000000000","message":"Need a comment here as well - I\u0027m assuming you\u0027re ignoring the HostMappingNotFound because a compute node could have a host mapping but for the wrong service? Shouldn\u0027t you be using _check_and_create_service_host_mappings for discovering ironic computes anyway?","commit_id":"19096c0c3b01b249c5d43112ae36d3eb653a0ffd"},{"author":{"_account_id":8767,"name":"Nikolay Fedotov","email":"nfedotov@cisco.com","username":"nfedotov"},"change_message_id":"2fc43cc64753e708d821c1d6f30d672ae0c660e9","unresolved":false,"context_lines":[{"line_number":192,"context_line":"                cell_mapping\u003dcm)"},{"line_number":193,"context_line":"            host_mapping.create()"},{"line_number":194,"context_line":"            host_mappings.append(host_mapping)"},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"        compute.mapped \u003d 1"},{"line_number":197,"context_line":"        compute.save()"},{"line_number":198,"context_line":"    return host_mappings"}],"source_content_type":"text/x-python","patch_set":1,"id":"ffb9cba7_7898b869","line":195,"in_reply_to":"ffb9cba7_58d93481","updated":"2019-04-22 18:06:47.000000000","message":"Ignoring HostMappingNotFound because host mapping exists but mapped\u003d0 . compute_nodes is a list of not (mapped \u003d 0) computes. So that, I suppose, mapped chould be set to 1 even if exception has not been raised.  Otherwise we can see 1+ \"not mapped\" computes (actually baremetal nodes) because the have same host value.\n\nMariaDB [nova]\u003e select uuid, host, mapped from nova.compute_nodes;\n+--------------------------------------+-------------+--------+\n| uuid                                 | host        | mapped |\n+--------------------------------------+-------------+--------+\n| 33f451dd-f609-4927-b59d-92d2a135d6a5 | ironic.aio1 |      1 |\n| eb8ecd77-d7d0-41ad-90a7-2747f2b3ac70 | ironic.aio1 |      0 |\n| 63b9fbc7-e5f5-4695-bdb0-6a7b90bcc1b9 | ironic.aio1 |      0 |\n\nBy the way \"not mapped\" computes is going to be mapped once there host value changed.  Which make impossible to the bug with \u003cbaremetal nodes amount\u003e greater than or equal to \u003cironic-conductor services amount\u003e\n\nI do not know about _check_and_create_service_host_mappings . Need to take a look. It night here. Will reply later.","commit_id":"19096c0c3b01b249c5d43112ae36d3eb653a0ffd"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"d38e93a23a078015240a92b24f9044b4bf10d742","unresolved":false,"context_lines":[{"line_number":194,"context_line":"            host_mappings.append(host_mapping)"},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"        compute.mapped \u003d 1"},{"line_number":197,"context_line":"        compute.save()"},{"line_number":198,"context_line":"    return host_mappings"},{"line_number":199,"context_line":""},{"line_number":200,"context_line":""}],"source_content_type":"text/x-python","patch_set":2,"id":"ffb9cba7_19125e0a","line":197,"updated":"2019-04-29 19:09:38.000000000","message":"This is going to cause us to save each node each time through the loop, regardless of whether it was already mapped or not before. This is just forced cleanup of the compute node changing its mapped state to be inconsistent with the actual state of the mappings managed here.","commit_id":"0a7416bbaf2a6d014273c4aa7298c364e5062506"},{"author":{"_account_id":8767,"name":"Nikolay Fedotov","email":"nfedotov@cisco.com","username":"nfedotov"},"change_message_id":"045e8c2ab12525071f258b38b9ad46c6403511f1","unresolved":false,"context_lines":[{"line_number":194,"context_line":"            host_mappings.append(host_mapping)"},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"        compute.mapped \u003d 1"},{"line_number":197,"context_line":"        compute.save()"},{"line_number":198,"context_line":"    return host_mappings"},{"line_number":199,"context_line":""},{"line_number":200,"context_line":""}],"source_content_type":"text/x-python","patch_set":2,"id":"ffb9cba7_0542e83e","line":197,"in_reply_to":"ffb9cba7_19125e0a","updated":"2019-04-30 07:17:38.000000000","message":"No.\n\nSee nova/objects/host_mapping.py:228\ncompute_nodes \u003d objects.ComputeNodeList.get_all_by_not_mapped(ctxt, mapped_less_than\u003d1)\nadded_hm \u003d _check_and_create_node_host_mappings(ctxt, cm, compute_nodes, status_fn)\n\ncompute_nodes is a list of not mapped (mapped \u003c 1) computes. \nLet say: \nN - number of baremetal node, compute nodes. ComputeNode table has N records\nn - number of ironic-conductors. Or number of controller nodes if ironic-conductors are running on them. Usually n\u003d3. \u0027host\u0027 field of ComputeNode record could be set to any of n values. \nN is way bigger than n\n\nN/n - ComputeNode records have same \u0027host\u0027 value because on 1st iteration the code on left side catches HostMappingNotFound exception \u003d\u003e create host mapping, compute.mapped \u003d 1 for the 1st record. On 2..(N/n) iterations HostMappingNotFound is not raised thus (N/n)-1 record still have mapped\u003d0.\n\nEvery 120 seconds (CONF.scheduler.discover_hosts_in_cells_interval) the code on left side selects (N/n)-1 records (mapped\u003d0) and at line #185 it gets same HostMapping record (N/n)-1 times. \n\nCode above sets mapped\u003d1 for the rest (N/n)-1 records and stopes looping over them every 120 seconds (CONF.scheduler.discover_hosts_in_cells_interval). All N/n computes have identical \u0027host\u0027 and there is HostMapping for that host. Correct? \n\nNo one have seen looping over (N/n-1)*n because \u0027status_fn\u0027 is not set in openstack/nova / nova/scheduler/manager.py:70. Otherwise there were (N/n-1)*n messages \u0027Checking host mapping for compute host...\u0027 in log.","commit_id":"0a7416bbaf2a6d014273c4aa7298c364e5062506"}]}