)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"2c583a5d1778be0ca11b45dd5fa4f253fc9189d1","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"6144b7ea_06e835d8","updated":"2021-10-15 17:06:16.000000000","message":"Awesome work! I have long been puzzled by the \"oslo_messaging.exceptions.MessagingTimeout: No reply on topic conductor\" and \"oslo_db.exception.DBNonExistentTable\" non-deterministic gate failures, it is so nice to finally know what caused it!\n\nI saw the earlier discussion on IRC [1] about the possibility of tracking greenlets per service and then killing them at the end of each test. I agree that would be a much better solution and also agree this patch is a fine fix for now while the other approach is investigated. +2 on that basis.\n\nIf you choose to go ahead and try the green pool approach instead of this, I am happy to review that instead as well.\n\n[1] https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2021-10-15.log.html#t2021-10-15T13:15:23","commit_id":"61fc81a6761d34afdfc4a6d1c4c953802fd8a179"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"4eee40d08a57279696feff610017db5c2aa0c5f5","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"9039122d_fd3a0788","updated":"2021-10-25 17:40:32.000000000","message":"I abandoned the alternative (the-kill-all-eventlet one) as I was not able to make that working even after couple of days of active troubleshooting. So let\u0027s merge this that seems to be safe and at least prevents the currently known interference","commit_id":"61fc81a6761d34afdfc4a6d1c4c953802fd8a179"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"b10b7b79e87346dc76b48ca52b328966662b0f62","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"66460a9b_fd846515","updated":"2021-10-21 14:22:13.000000000","message":"I will push an alternative review that kills leaked greenlets at the end of the each testcase. Lets compare these options...","commit_id":"61fc81a6761d34afdfc4a6d1c4c953802fd8a179"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"7c624deab76ba8bfa95aad45f6acdc5afe57c780","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"4951599a_1a939064","updated":"2021-10-15 09:19:14.000000000","message":"It works from gate perspective too. Looking at the logs and searching for the RuntimeError the NotificationFixture now raises in the late eventlet it can be seen that this is a common problem that late eventlets are acting during other test executions. But the current fix seems to be enough to prevent failures.\n\nhttps://zuul.opendev.org/t/openstack/build/7e4a6a497cf14ce8a2a76e71762f13f0/log/job-output.txt#5140\nhttps://zuul.opendev.org/t/openstack/build/09b7af7c6cf442d08bf5620657cf678c/log/job-output.txt#4375\nhttps://zuul.opendev.org/t/openstack/build/830694cf6a5146b293863e1873243f63/log/job-output.txt#4435\n","commit_id":"61fc81a6761d34afdfc4a6d1c4c953802fd8a179"},{"author":{"_account_id":10135,"name":"Lee Yarwood","display_name":"Lee Yarwood","email":"lyarwood@redhat.com","username":"lyarwood"},"change_message_id":"fef207544f0aaf54c4e2843de4846ec18abebaff","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"29021c1b_8160fd40","updated":"2021-11-03 15:08:46.000000000","message":"LGTM, thanks for the detailed commit message, this makes sense and should avoid the issue.","commit_id":"61fc81a6761d34afdfc4a6d1c4c953802fd8a179"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"98472e6869204e1ecd2765d59bc87f28802f8fda","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"8ed6dfbf_a75d9c82","in_reply_to":"66460a9b_fd846515","updated":"2021-10-21 15:05:21.000000000","message":"here is the alternative https://review.opendev.org/c/openstack/nova/+/815017","commit_id":"61fc81a6761d34afdfc4a6d1c4c953802fd8a179"}],"nova/tests/fixtures/notifications.py":[{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"c75789c0caa65bed374b6138e72b5b4f0278947a","unresolved":true,"context_lines":[{"line_number":164,"context_line":"        # NOTE(gibi): this is here to prevent late notifications from already"},{"line_number":165,"context_line":"        # finished test cases to break the currently running test case. See"},{"line_number":166,"context_line":"        # more in https://bugs.launchpad.net/nova/+bug/1946339"},{"line_number":167,"context_line":"        if sender_test_case_id !\u003d self.test_case_id:"},{"line_number":168,"context_line":"            raise RuntimeError("},{"line_number":169,"context_line":"                \u0027FakeVersionedNotifier received %s notification emitted by %s \u0027"},{"line_number":170,"context_line":"                \u0027test case which is different from the currently running test \u0027"}],"source_content_type":"text/x-python","patch_set":1,"id":"b30d1010_992c5923","line":167,"range":{"start_line":167,"start_character":7,"end_line":167,"end_character":52},"updated":"2021-10-20 12:06:49.000000000","message":"ok so this really is the important part.\nthis will only raise the excption if the currnt fake notifcaiton fixture that is in use\nis from a different test tehn the calling eventlet was spawned from and the excpetion will propagate up the call stack of the leaked greenthread killing it without impacting the running test which make this self contained.\n\ne.g a  leaking test wont break other unrelated test anymore.","commit_id":"61fc81a6761d34afdfc4a6d1c4c953802fd8a179"}]}