)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"325e7232046b58da55928ff02810f02158d13c87","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"4dbff7de_7f8cb0d6","updated":"2022-03-10 15:29:09.000000000","message":"\u003e Patch Set 1:\n\u003e \n\u003e This basically changes \"no valid host was found\" from always fatal to never fatal.  Is that really the case?\n\nThe no valid host errors are returned by Nova when its placement service is unable to find a valid location to place a VM. In a typical deployment of openstack this means that either Nova is truly full up or that there is an accounting error of some kind that requires cloud operator intervention. In my opinion it is desireable in both cases to have Nodepool return an error so that we can try to schedule the node in another provider or if there are no other providers bubble up the NODE_FAILURE so that people can look into why the cloud isn\u0027t operating as expected.","commit_id":"dcbba26c01eaa2637e30b7b5be1e39395c457be4"},{"author":{"_account_id":33934,"name":"Joshua Watt","email":"JPEWhacker@gmail.com","username":"jpew"},"change_message_id":"1fb23bfbd886acd6190643646671108ece8083cd","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"01d265ef_cd78b0c6","updated":"2022-03-10 15:33:47.000000000","message":"\u003e Patch Set 1:\n\u003e \n\u003e This basically changes \"no valid host was found\" from always fatal to never fatal.  Is that really the case?\n\nUnfortunately, I inherited this change, so I\u0027m not exactly sure. It seems we were occasionally getting that error when spinning up nodes; speculation was that it happened when the quota was changed after the node started spinning up, but this was never confirmed.\n\nI\u0027d be willing to try without that particular part of the change and try to sort out whats going on if we can figure out an upstream way to keep retrying when there is no quota.","commit_id":"dcbba26c01eaa2637e30b7b5be1e39395c457be4"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"48f054a8c981759ffaddcdb3911975e4b424bdf4","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"3a237e72_8101deae","in_reply_to":"01d265ef_cd78b0c6","updated":"2022-03-10 15:58:28.000000000","message":"\u003e I\u0027d be willing to try without that particular part of the change and try to sort out whats going on if we can figure out an upstream way to keep retrying when there is no quota.\n\nThis seems much safer. Currently nodepool will pause when it hits no quota and wait for quota to free up. But if we retry during that pause we shouldn\u0027t change the functional behavior of nodepool very much. We\u0027ll just do some extra API calls but that shouldn\u0027t be a big deal.\n\nOld Behavior:\n\n  Boot node\n  Get quota error\n  Pause\n  Delete used node freeing quota\n  Unpause\n  Boot node\n\nNew Behavior:\n\n  Boot node\n  While quote error\n    Sleep\n  Boot node","commit_id":"dcbba26c01eaa2637e30b7b5be1e39395c457be4"},{"author":{"_account_id":33934,"name":"Joshua Watt","email":"JPEWhacker@gmail.com","username":"jpew"},"change_message_id":"3bc91be3683b335eab26f5f8af804e085116a426","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"40f3f878_355dfa01","updated":"2022-03-10 17:26:54.000000000","message":"Is this what you had in mind?","commit_id":"92aca68152e1dc226e4cd69e32543201aab685fb"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"9d77e22dbd2619ba85a80b1e3372bc6345f36098","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"aa95572e_03fb6aab","updated":"2022-03-29 22:46:44.000000000","message":"This feels a bit clunky particularly since I\u0027m not sure this is a standard feature of openstack. However, nodepool has long worked around provider specific behaviors to the benefit of users so this seems to fall in line with that history.","commit_id":"0db1eb2382078fe567f3e3b8c8a053a79925359f"}],"doc/source/openstack.rst":[{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"9bcf5eb68fe1d0f3f552d5caba0f1ddd4c61b591","unresolved":false,"context_lines":[{"line_number":459,"context_line":"        circumstances where the provider is incorrectly calculating"},{"line_number":460,"context_line":"        quota."},{"line_number":461,"context_line":""},{"line_number":462,"context_line":"     .. attr:: reactive-quota"},{"line_number":463,"context_line":"        :type: bool"},{"line_number":464,"context_line":"        :default: False"},{"line_number":465,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"d9fb3a4f_11067a50","line":462,"updated":"2022-04-26 22:15:03.000000000","message":"The name of this option and the description seems overly prescriptive.  Really what this option does is cause quota errors to not be treated as fatal for the provider (and so node requests will be continually retried).  Could we change the name and description to match that, rather than the specific scenario that you want to use it for?  I think it would be okay to also describe your use case in the docs if you want.","commit_id":"0db1eb2382078fe567f3e3b8c8a053a79925359f"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"9bcf5eb68fe1d0f3f552d5caba0f1ddd4c61b591","unresolved":false,"context_lines":[{"line_number":470,"context_line":"        remains and the quota can be increased."},{"line_number":471,"context_line":""},{"line_number":472,"context_line":"        Note that this means a node allocation that fails due to a quota"},{"line_number":473,"context_line":"        problem will not fall thought to another provider."},{"line_number":474,"context_line":""},{"line_number":475,"context_line":"     .. attr:: availability-zones"},{"line_number":476,"context_line":"        :type: list"}],"source_content_type":"text/x-rst","patch_set":5,"id":"c7c6be41_76a71cf5","line":473,"updated":"2022-04-26 22:15:03.000000000","message":"typo: fall \"through\"","commit_id":"0db1eb2382078fe567f3e3b8c8a053a79925359f"}],"nodepool/driver/openstack/handler.py":[{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"282f09f8e7218cd7689b98e4a99c58f238e289ce","unresolved":true,"context_lines":[{"line_number":299,"context_line":"                    # out immediately with a specific exception."},{"line_number":300,"context_line":"                    self.log.info(\"Quota exceeded, invalidating quota cache\")"},{"line_number":301,"context_line":"                    self.handler.manager.invalidateQuotaCache()"},{"line_number":302,"context_line":"                    time.sleep(10)"},{"line_number":303,"context_line":"                else:"},{"line_number":304,"context_line":"                    attempts +\u003d 1"},{"line_number":305,"context_line":""}],"source_content_type":"text/x-python","patch_set":2,"id":"8e6308b9_307f2432","line":302,"updated":"2022-03-14 16:12:52.000000000","message":"I think this long sleep is why the testing fails. We probably check states before this sleep is completed.\n\nAs far as implementing the sleep I wonder if we need to do a hybrid between the current pause and the sleeping with retries to ensure we don\u0027t end up with a provider that grabs all the node requests but they fail for a long time due to quota errors. Something like having the first node to hit a quota error continue to retry but pause all other requests? That should provide the request pressure you want against the API while also avoiding having a single provider take all requests then fail to service them due to quotas.","commit_id":"92aca68152e1dc226e4cd69e32543201aab685fb"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"fdc7c0beeaf6311e2381f6a2f890ca291b6c8e75","unresolved":true,"context_lines":[{"line_number":299,"context_line":"                    # out immediately with a specific exception."},{"line_number":300,"context_line":"                    self.log.info(\"Quota exceeded, invalidating quota cache\")"},{"line_number":301,"context_line":"                    self.handler.manager.invalidateQuotaCache()"},{"line_number":302,"context_line":"                    time.sleep(10)"},{"line_number":303,"context_line":"                else:"},{"line_number":304,"context_line":"                    attempts +\u003d 1"},{"line_number":305,"context_line":""}],"source_content_type":"text/x-python","patch_set":2,"id":"af888a63_92659560","line":302,"in_reply_to":"73a226b5_2520071a","updated":"2022-03-23 20:07:10.000000000","message":"We also need to take into consideration that we raise here for a reason which is to release the node request and give a different provider on a different cloud the chance to fulfill the node request. The reason for this is that otherwise we\u0027d block the request until it can be satisfied while having free resources on a different cloud. This can be especially an issue on small multi-cloud systems.","commit_id":"92aca68152e1dc226e4cd69e32543201aab685fb"},{"author":{"_account_id":33934,"name":"Joshua Watt","email":"JPEWhacker@gmail.com","username":"jpew"},"change_message_id":"b782f2f40c23ec99225cd3f742b970d3a4cd01e9","unresolved":true,"context_lines":[{"line_number":299,"context_line":"                    # out immediately with a specific exception."},{"line_number":300,"context_line":"                    self.log.info(\"Quota exceeded, invalidating quota cache\")"},{"line_number":301,"context_line":"                    self.handler.manager.invalidateQuotaCache()"},{"line_number":302,"context_line":"                    time.sleep(10)"},{"line_number":303,"context_line":"                else:"},{"line_number":304,"context_line":"                    attempts +\u003d 1"},{"line_number":305,"context_line":""}],"source_content_type":"text/x-python","patch_set":2,"id":"73a226b5_2520071a","line":302,"in_reply_to":"8e6308b9_307f2432","updated":"2022-03-22 15:29:23.000000000","message":"I did some digging, and I don\u0027t think it would take all the requests. My understanding from reading the code is that only the requests that race in `_waitForNodeSet` checking the quota would enter this loop. The requests after that will automatically pause the handler (especially since we invalidate the quota cache). I don\u0027t think any \"pausing\" is necessary here since the code in `_waitForNodeSet` handles that already; the only thing we could *maybe* do here is try and detect if multiple threads are in this loop because of quota reasons and kick all but one out.... but this seems complex and error prone so I\u0027d rather not unless it\u0027s a huge problem to have a few nodes in the loop until quota is satisfied (realizing that it won\u0027t consume all nodes because of the reasons stated above).\n\n\nIf my understanding is incorrect, please let me know :)","commit_id":"92aca68152e1dc226e4cd69e32543201aab685fb"},{"author":{"_account_id":33934,"name":"Joshua Watt","email":"JPEWhacker@gmail.com","username":"jpew"},"change_message_id":"d02e07c8474365c81d87667692080d0b9f3515e3","unresolved":true,"context_lines":[{"line_number":299,"context_line":"                    # out immediately with a specific exception."},{"line_number":300,"context_line":"                    self.log.info(\"Quota exceeded, invalidating quota cache\")"},{"line_number":301,"context_line":"                    self.handler.manager.invalidateQuotaCache()"},{"line_number":302,"context_line":"                    time.sleep(10)"},{"line_number":303,"context_line":"                else:"},{"line_number":304,"context_line":"                    attempts +\u003d 1"},{"line_number":305,"context_line":""}],"source_content_type":"text/x-python","patch_set":2,"id":"cb2a62a3_70872e48","line":302,"in_reply_to":"af888a63_92659560","updated":"2022-03-24 13:18:33.000000000","message":"Right, I\u0027m not sure how to reconcile those two; our setup needs the quota pressure to get the quote increased, but another setup might want to fall through to another provider... I\u0027m not sure how to saitisfy both constraints since they seem contradictory; I hate to add a configuration option to control the behavior, but it\u0027s all I can really think of right know.","commit_id":"92aca68152e1dc226e4cd69e32543201aab685fb"},{"author":{"_account_id":33934,"name":"Joshua Watt","email":"JPEWhacker@gmail.com","username":"jpew"},"change_message_id":"ffaf4a488b791ffff69e6b5076e410bd3f39c6d9","unresolved":false,"context_lines":[{"line_number":299,"context_line":"                    # out immediately with a specific exception."},{"line_number":300,"context_line":"                    self.log.info(\"Quota exceeded, invalidating quota cache\")"},{"line_number":301,"context_line":"                    self.handler.manager.invalidateQuotaCache()"},{"line_number":302,"context_line":"                    time.sleep(10)"},{"line_number":303,"context_line":"                else:"},{"line_number":304,"context_line":"                    attempts +\u003d 1"},{"line_number":305,"context_line":""}],"source_content_type":"text/x-python","patch_set":2,"id":"bd2e70a2_66313ae1","line":302,"in_reply_to":"cb2a62a3_70872e48","updated":"2022-04-04 13:15:27.000000000","message":"Done","commit_id":"92aca68152e1dc226e4cd69e32543201aab685fb"}]}