)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":24872,"name":"YumengBao","email":"yumeng_bao@yahoo.com","username":"Yumeng_Bao"},"change_message_id":"c2ab4f41c6185d51da26aa083e32a60241dc4e7c","unresolved":false,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Solve py37 timeout"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"The clue shows that we should suspect this method:"},{"line_number":10,"context_line":"test_apply_patch_fpga_arq_monitor_job"},{"line_number":11,"context_line":""},{"line_number":12,"context_line":"Let\u0027s comment this method first, and leave the code link[1]."}],"source_content_type":"text/x-gerrit-commit-message","patch_set":4,"id":"3fa7e38b_58a418a4","line":9,"range":{"start_line":9,"start_character":3,"end_line":9,"end_character":9},"updated":"2020-02-13 13:30:21.000000000","message":"You\u0027ve dug deep in this problem. Can you add more info about the problem and the clue in the commit message?\nWhere do you find the timeout problem? How you draw this conclusion? List the main source patches and also the key clue that comes to the conclusion.","commit_id":"db0efe85f57543a5a974c85b9a696b3a3227934d"},{"author":{"_account_id":14131,"name":"shaohef","email":"shaohe.feng@intel.com","username":"shaohefeng"},"change_message_id":"0fb979a54c41cfe6b32c9d77d94ea4d51c3b0246","unresolved":false,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Solve py37 timeout"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"The clue shows that we should suspect this method:"},{"line_number":10,"context_line":"test_apply_patch_fpga_arq_monitor_job"},{"line_number":11,"context_line":""},{"line_number":12,"context_line":"Let\u0027s comment this method first, and leave the code link[1]."}],"source_content_type":"text/x-gerrit-commit-message","patch_set":4,"id":"3fa7e38b_ce53626d","line":9,"range":{"start_line":9,"start_character":3,"end_line":9,"end_character":9},"in_reply_to":"3fa7e38b_58a418a4","updated":"2020-02-13 15:01:18.000000000","message":"\u003e You\u0027ve dug deep in this problem. Can you add more info about the\n \u003e problem and the clue in the commit message?\n \u003e Where do you find the timeout problem? How you draw this\n \u003e conclusion? List the main source patches and also the key clue that\n \u003e comes to the conclusion.\n\nIMHO no need to add the info to this message.\nA Standard engineering debug method can debug it. \n\nI did not find every commit has add develop method to commit message.\n\nLet me explain:\nActually, we have already have timeout when py37 enable. \nPlease see my summary comments. \nFor it is random, so we did not pay attention to on it.\nJust recheck it every time, we thought this is the fault of CI Gate. \n\nBut a later a patch add a basepython in tox, then all timeout disappears. \nAnd then an new patch ignore basepython commit, timeout come out again. \n\nDuring these period， We make many change commit, so that maybe inadvertently eliminate some timeout UT.\nBut this one still remain at last. \n\nmost of us did not use Py37 or Py38. If you use Py37/Py38, you can catch this issue. \n\nSo suggest all developers:\n1. Move to Py38, it backwards compatible with Py36/Py37.\n2. Pay attention to timeout, look into it before recheck.\n\nAll these no need in the commit message.","commit_id":"db0efe85f57543a5a974c85b9a696b3a3227934d"},{"author":{"_account_id":21672,"name":"Sundar Nadathur","email":"sundar.nadathur@intel.com","username":"nsundar"},"change_message_id":"d3abe66d9afcab7334cc6d475ffbbb08dae358fb","unresolved":false,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Solve py37 timeout"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"The clue shows that we should suspect this method:"},{"line_number":10,"context_line":"test_apply_patch_fpga_arq_monitor_job"},{"line_number":11,"context_line":""},{"line_number":12,"context_line":"Let\u0027s comment this method first, and leave the code link[1]."}],"source_content_type":"text/x-gerrit-commit-message","patch_set":4,"id":"3fa7e38b_e3dbd01c","line":9,"range":{"start_line":9,"start_character":3,"end_line":9,"end_character":9},"in_reply_to":"3fa7e38b_ce53626d","updated":"2020-02-13 19:11:12.000000000","message":"Yumeng, to add some context:\n\nhttps://zuul.opendev.org/t/openstack/builds?job_name\u003dopenstack-tox-py37\u0026result\u003dTIMED_OUT\n\nshows that a lot of Cyborg patches timed out on py37. Sme neutron patches used to time out but that\u0027s because they were exceeding the job\u0027s time limit and they (will) fix it by increasing the timeout. But Cyborg py37 usually takes just a few minutes, whereas the time out is 40 minutes: so increasing the timeout is not the right solution for us. \n\nSome IRC logs for reference:\n[1] First discussion with infra team:\n\n http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2020-01-22.log.html#t2020-01-22T15:22:32\n\n[2] Last discussion with infra team so far:\n\n http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2020-02-12.log.html#t2020-02-12T16:46:18","commit_id":"db0efe85f57543a5a974c85b9a696b3a3227934d"},{"author":{"_account_id":14131,"name":"shaohef","email":"shaohe.feng@intel.com","username":"shaohefeng"},"change_message_id":"7c978d95d9c13282604637005e61305fc6e891ee","unresolved":false,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Solve py37 timeout"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"Py37 job always reports timeout error recently."},{"line_number":10,"context_line":"Please see [1] [2] [3]."},{"line_number":11,"context_line":"At first it was suspected that the error was reported"},{"line_number":12,"context_line":"because of the patch [4]."},{"line_number":13,"context_line":"Therefore, Feng Shaohe\u0027s patch [5] revoked the merge,"},{"line_number":14,"context_line":"and at this time, disappeared at py37 timeout."},{"line_number":15,"context_line":""},{"line_number":16,"context_line":"But in fact, this problem is just hidden."},{"line_number":17,"context_line":"After removing this setting, the job of py37"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":5,"id":"3fa7e38b_b2bc326b","line":14,"range":{"start_line":9,"start_character":0,"end_line":14,"end_character":46},"updated":"2020-02-14 16:39:52.000000000","message":"The whole history can be 3 stage:\n1. After we enable py37 there are some random timeout as follow:\nhttps://review.opendev.org/#/c/679406/  On Sep 19 11:30 AM\nhttps://review.opendev.org/#/c/688239/  On Oct 12 3:31 PM\nhttps://review.opendev.org/#/c/688231/  On Oct 28 11:38 PM Oct 29 9:11 AM Oct 29 12:05 PM\nhttps://review.opendev.org/#/c/685542/  On Nov 14 12:31 AM Nov 14 7:42 AM\nhttps://review.opendev.org/#/c/691872/  On Oct 29 11:55 PM\nhttps://review.opendev.org/#/c/690509/  On Nov 15 4:33 PM\n\n2. After this https://review.opendev.org/#/c/688593/ timeout Disappeared for a while. This patch is merged on Nov 19 5:05 PM.\nThis patch set a wrong python path evn.\n\n3. After another patch https://review.opendev.org/#/c/696397， timeout comes out again. \nThe intention of this patch is to suppress confusing pep8 message, but it also fix python path evn unintentionally.","commit_id":"bfaae7ada37113f438b75c6672e537775d4f2d87"},{"author":{"_account_id":14131,"name":"shaohef","email":"shaohe.feng@intel.com","username":"shaohefeng"},"change_message_id":"fa93900ff1b0f284dba4899bcd772abb44a5bdfe","unresolved":false,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Solve py37 timeout"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"Py37 job always reports timeout error recently."},{"line_number":10,"context_line":"Please see [1] [2] [3]."},{"line_number":11,"context_line":"At first it was suspected that the error was reported"},{"line_number":12,"context_line":"because of the patch [4]."},{"line_number":13,"context_line":"Therefore, Feng Shaohe\u0027s patch [5] revoked the merge,"},{"line_number":14,"context_line":"and at this time, disappeared at py37 timeout."},{"line_number":15,"context_line":""},{"line_number":16,"context_line":"But in fact, this problem is just hidden."},{"line_number":17,"context_line":"After removing this setting, the job of py37"},{"line_number":18,"context_line":"is actually running on the environment of python 3.6"},{"line_number":19,"context_line":"(community CI default version is 3.6), please see [6]"},{"line_number":20,"context_line":"for detailed reasons."},{"line_number":21,"context_line":""},{"line_number":22,"context_line":"Therefore, this patch exposes the hidden py37 timeout problem,"},{"line_number":23,"context_line":"and at the same time, found method test_apply_patch_fpga_arq_monitor_job"},{"line_number":24,"context_line":", think it is the reason of the timeout. The reason I can find"},{"line_number":25,"context_line":"this method is based on the the troubleshooting of tox -epy37 log."},{"line_number":26,"context_line":"After commenting out this method, I found that tox -epy37 can run"},{"line_number":27,"context_line":"normally and there is no longer a timeout problem."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"If you want to test, please ensure that you have a local"},{"line_number":30,"context_line":"python3.7 environment, not 3.6, and execute rm .tox / -rf."},{"line_number":31,"context_line":"Then execute tox -epy37."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"Therefore, the best way is to comment out this method and"},{"line_number":34,"context_line":"restore py37 job at the same time."},{"line_number":35,"context_line":"If a friend discovers further reasons, this method can be"},{"line_number":36,"context_line":"restored, please refer to [7]."},{"line_number":37,"context_line":""},{"line_number":38,"context_line":"What went wrong in this method? I personally suspect that it"},{"line_number":39,"context_line":"is caused by deadlock or asynchronous waiting, just speculation."},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"[1]. https://review.opendev.org/#/c/702578/"},{"line_number":42,"context_line":"[2]. https://review.opendev.org/#/c/703049/"},{"line_number":43,"context_line":"[3]. https://review.opendev.org/#/c/703253/"},{"line_number":44,"context_line":"[4]. https://review.opendev.org/#/c/696397/"},{"line_number":45,"context_line":"[5]. https://review.opendev.org/#/c/706911/"},{"line_number":46,"context_line":"[6]. http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2020-02-12.log.html#t2020-02-12T16:46:18"},{"line_number":47,"context_line":"[7]. https://github.com/openstack/cyborg/tree/deed9c822e3dc54a522ede0d8fafda5890075803"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"Change-Id: I09db889fe665c6246ec9503af92c909e7d0da24f"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":5,"id":"3fa7e38b_8009a7db","line":47,"range":{"start_line":9,"start_character":0,"end_line":47,"end_character":86},"updated":"2020-02-14 11:02:56.000000000","message":"IMHO, need to describe these, no helpful for others to get \nexperience if you wonder how to debug the issue.\n\nI really use bisection to narrow down the scope issue.\nIf you know well about threadpool lib, just ctr+c signal can more easily to find the issue directly by the call trace stack. \n\nI found the reason:\nThe thread pool lib always loop to check if there are new jobs.\nThe action of testunit framework in py37 is different with py36. If there are thread pool run, the testunit will never return. \n\nyou can a simple test as follow in this file:\n\ndef job1(t1\u003d0):\n    print(time.time())\n    print(\"sleep %s second\" % t1)\n    time.sleep(t1)\n    print(time.time())\n    return \"Hello, world\"\n\nclass TestExtARQObject(base.DbTestCase):\n    def test_foo(self):\n        print(\"Test Foo\")\n\n    def test_bar(self):\n        print(\"Test bar\")\n\n    def test_apply_patch_fpga_arq_monitor_job(self):\n        works \u003d utils.ThreadWorks()\n        job \u003d works.spawn(job1, 1)\n        return job","commit_id":"bfaae7ada37113f438b75c6672e537775d4f2d87"}],"cyborg/tests/unit/objects/test_extarq.py":[{"author":{"_account_id":26458,"name":"Brin Zhang","email":"zhangbailin@inspur.com","username":"zhangbailin"},"change_message_id":"524e86cf109f73879fb981c85e3197fb21913305","unresolved":false,"context_lines":[{"line_number":245,"context_line":"    @mock.patch(\u0027cyborg.objects.ExtARQ.update_check_state\u0027)"},{"line_number":246,"context_line":"    @mock.patch(\u0027cyborg.objects.deployable.Deployable.get_by_device_rp_uuid\u0027)"},{"line_number":247,"context_line":"    @mock.patch(\u0027cyborg.common.utils.ThreadWorks.spawn_master\u0027)"},{"line_number":248,"context_line":"    def test_apply_patch_fpga_arq_monitor_job("},{"line_number":249,"context_line":"        self, mock_master, mock_get_dep, mock_check_state, mock_list,"},{"line_number":250,"context_line":"        mock_get, mock_attach_handle, mock_notify_bind, mock_conn):"},{"line_number":251,"context_line":""}],"source_content_type":"text/x-python","patch_set":4,"id":"3fa7e38b_9579f930","side":"PARENT","line":248,"updated":"2020-02-13 12:01:46.000000000","message":"In this test, which code impact the py37 failed? Anyone can give guess?","commit_id":"deed9c822e3dc54a522ede0d8fafda5890075803"},{"author":{"_account_id":14131,"name":"shaohef","email":"shaohe.feng@intel.com","username":"shaohefeng"},"change_message_id":"40d7d6f82ef523cb42935f027e00e176469c0a38","unresolved":false,"context_lines":[{"line_number":245,"context_line":"    @mock.patch(\u0027cyborg.objects.ExtARQ.update_check_state\u0027)"},{"line_number":246,"context_line":"    @mock.patch(\u0027cyborg.objects.deployable.Deployable.get_by_device_rp_uuid\u0027)"},{"line_number":247,"context_line":"    @mock.patch(\u0027cyborg.common.utils.ThreadWorks.spawn_master\u0027)"},{"line_number":248,"context_line":"    def test_apply_patch_fpga_arq_monitor_job("},{"line_number":249,"context_line":"        self, mock_master, mock_get_dep, mock_check_state, mock_list,"},{"line_number":250,"context_line":"        mock_get, mock_attach_handle, mock_notify_bind, mock_conn):"},{"line_number":251,"context_line":""}],"source_content_type":"text/x-python","patch_set":4,"id":"3fa7e38b_234c8895","side":"PARENT","line":248,"in_reply_to":"3fa7e38b_63c7e03e","updated":"2020-02-13 19:30:40.000000000","message":"\u003e Not exactly at present.\n \u003e Guess extarq_get call query.with_for_update in \"cyborg/db/sqlalchemy/api.py\"\n \u003e \n \u003e but not know what\u0027s the difference between py3.6 and py3.7\n\nnot the issue of with_for_update","commit_id":"deed9c822e3dc54a522ede0d8fafda5890075803"},{"author":{"_account_id":14131,"name":"shaohef","email":"shaohe.feng@intel.com","username":"shaohefeng"},"change_message_id":"c7bcc62aac7e853946c94a5bfba77f6b01de47bd","unresolved":false,"context_lines":[{"line_number":245,"context_line":"    @mock.patch(\u0027cyborg.objects.ExtARQ.update_check_state\u0027)"},{"line_number":246,"context_line":"    @mock.patch(\u0027cyborg.objects.deployable.Deployable.get_by_device_rp_uuid\u0027)"},{"line_number":247,"context_line":"    @mock.patch(\u0027cyborg.common.utils.ThreadWorks.spawn_master\u0027)"},{"line_number":248,"context_line":"    def test_apply_patch_fpga_arq_monitor_job("},{"line_number":249,"context_line":"        self, mock_master, mock_get_dep, mock_check_state, mock_list,"},{"line_number":250,"context_line":"        mock_get, mock_attach_handle, mock_notify_bind, mock_conn):"},{"line_number":251,"context_line":""}],"source_content_type":"text/x-python","patch_set":4,"id":"3fa7e38b_a693a64c","side":"PARENT","line":248,"in_reply_to":"3fa7e38b_9579f930","updated":"2020-02-13 20:03:28.000000000","message":"\u003e In this test, which code impact the py37 failed? Anyone can give\n \u003e guess?\n\nThis case can run successfully independent.\nIt can also can run successfully with each other.\nBut it can not run  successfully with any 2 others.\n\neven it can not works well the simple 2 UTs.\n+    def test_foo(self):\n+        print(\"test\")\n\n+    def test_bar(self):\n+        print(\"test\")\n\nThe unittest framework can test it well, and no error for it.  But it can not get conclusion result for more than 2 cases(include this one), unittest will hang on return(result).","commit_id":"deed9c822e3dc54a522ede0d8fafda5890075803"},{"author":{"_account_id":14131,"name":"shaohef","email":"shaohe.feng@intel.com","username":"shaohefeng"},"change_message_id":"425f430431d226940d2052157a5ea606c2a670b6","unresolved":false,"context_lines":[{"line_number":245,"context_line":"    @mock.patch(\u0027cyborg.objects.ExtARQ.update_check_state\u0027)"},{"line_number":246,"context_line":"    @mock.patch(\u0027cyborg.objects.deployable.Deployable.get_by_device_rp_uuid\u0027)"},{"line_number":247,"context_line":"    @mock.patch(\u0027cyborg.common.utils.ThreadWorks.spawn_master\u0027)"},{"line_number":248,"context_line":"    def test_apply_patch_fpga_arq_monitor_job("},{"line_number":249,"context_line":"        self, mock_master, mock_get_dep, mock_check_state, mock_list,"},{"line_number":250,"context_line":"        mock_get, mock_attach_handle, mock_notify_bind, mock_conn):"},{"line_number":251,"context_line":""}],"source_content_type":"text/x-python","patch_set":4,"id":"3fa7e38b_63c7e03e","side":"PARENT","line":248,"in_reply_to":"3fa7e38b_9579f930","updated":"2020-02-13 19:27:31.000000000","message":"\u003e In this test, which code impact the py37 failed? Anyone can give\n \u003e guess?\nNot exactly at present. \nGuess extarq_get call query.with_for_update in \"cyborg/db/sqlalchemy/api.py\"\n\nbut not know what\u0027s the difference between py3.6 and py3.7","commit_id":"deed9c822e3dc54a522ede0d8fafda5890075803"}],"tox.ini":[{"author":{"_account_id":14131,"name":"shaohef","email":"shaohe.feng@intel.com","username":"shaohefeng"},"change_message_id":"e82191ccbdcd0e40c5e9d69d1ee13abb6a1537cb","unresolved":false,"context_lines":[{"line_number":5,"context_line":"# Automatic envs (pyXX) will only use the python version appropriate to that"},{"line_number":6,"context_line":"# env and ignore basepython inherited from [testenv] if we set"},{"line_number":7,"context_line":"# ignore_basepython_conflict."},{"line_number":8,"context_line":"ignore_basepython_conflict \u003d True"},{"line_number":9,"context_line":""},{"line_number":10,"context_line":"[testenv]"},{"line_number":11,"context_line":"usedevelop \u003d True"}],"source_content_type":"text/x-properties","patch_set":4,"id":"3fa7e38b_f4c2b601","line":8,"range":{"start_line":8,"start_character":29,"end_line":8,"end_character":33},"updated":"2020-02-13 05:22:53.000000000","message":"it is revert here.","commit_id":"db0efe85f57543a5a974c85b9a696b3a3227934d"}]}