)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"bcb4039f5ca4e1a45ab4dffcd0c1f71955d16b95","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"33165441_f824d392","updated":"2024-07-03 22:07:18.000000000","message":"the storage jobs were also timigng out before we merged this and this has resulted in a significat reduciton in job failures in nova since it merged.\n\n\nwe had almost 4 weeks where we were unabel to merge any changes prior to enbaleing these optimisations\n\ninstead of revertign this can you instead try disabling\nENABLE_ZSWAP and ENABLE_SYSCTL_MEM_TUNING and ENABLE_SYSCTL_NET_TUNING\nin tempest-integrated-storage-import to determin which of the optimisation is impacting your job.\n\nif we were to regert thi im prtty sure it will entirely block the nova gate again.\n\nwe tempoerly bumpted the timeouts diarly to day so it may not fully block it but i do expect it to make the jobs slower.","commit_id":"ec80cedd8856c3069390a437277fd294e66b5281"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"9931deaf216dd1e98bdd651bd3e4b94adc584eb9","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":2,"id":"d262bae3_7383dd20","in_reply_to":"33165441_f824d392","updated":"2024-07-03 23:07:32.000000000","message":"for context this is the the runtime of the concurrent part of tempst-integrated-comptue prior to the optimistiatongs\n\nhttps://fileshare.seanmooney.info/average_tempest_runtime_for_tempest-integrated-compute_by_provider.png\n\nand this is the same caluated over the succesful josb in the last 2 weeks\n\nhttps://fileshare.seanmooney.info/average_tempest_runtime_for_tempest-integrated-compute_by_provider_2024-07-03.png\n\nthe average runtime for each ci provder was\n\nprovider     old      new       % speedup (1-(new/old))*100\novh-BHS1   3799       3783     0.421163464\novh-GRA1   3847       3753     2.443462438\nrax-DFW    3769       3694     1.98991775\nrax-IAD    3825       3556     7.032679739\nrax-ORD    4020       3515     12.562189055\n\nthat a 1-12% performance increase depeding on the provider\n\nif we look at the variance\nprovider   old      new      % reduction (1-(new/old))*100\novh-BHS1   482      351     27.178423237\novh-GRA1   462      403     12.770562771\nrax-DFW    321      346     −7.788161994\nrax-IAD    384      353     8.072916667\nrax-ORD    360      534     −48.333333333\n\nthis is more of a mixed bag but ovh-BHS1 and\novh-GRA1 appare to be more consitent where as has more job to job variance.\n\nthat is likely an artifact of the age of some of rax spaces hardware.\n\nwe see similar uplift in performance in the new defaults job\n\nbefore optimistaion\nhttps://fileshare.seanmooney.info/average_tempest_runtime_for_tempest-integrated-compute-enforce-scope-new-defaults_by_provider.png\nafter\nhttps://fileshare.seanmooney.info/average_tempest_runtime_for_tempest-integrated-compute-enforce-scope-new-defaults_by_provider_2024-07-03.png\n\n\nwe also went form a 10% timeout rate of tempest-integrated-compute to ~5% in that time too\n\nthe data for this came form \n\nhttps://opensearch.logs.openstack.org/_dashboards/app/data-explorer/discover?security_tenant\u003dglobal#?_a\u003d(discover:(columns:!(build_status,build_name,hosts_region,build_ref),isDirty:!t,sort:!()),metadata:(indexPattern:\u002794869730-aea8-11ec-9e6a-83741af3fdcd\u0027,view:discover))\u0026_g\u003d(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-2w,to:now))\u0026_q\u003d(filters:!((\u0027$state\u0027:(store:appState),meta:(alias:!n,disabled:!f,index:\u002794869730-aea8-11ec-9e6a-83741af3fdcd\u0027,key:build_name,negate:!f,params:(query:tempest-integrated-compute),type:phrase),query:(match_phrase:(build_name:tempest-integrated-compute))),(\u0027$state\u0027:(store:appState),meta:(alias:!n,disabled:!f,index:\u002794869730-aea8-11ec-9e6a-83741af3fdcd\u0027,key:filename,negate:!f,params:(query:job-output.txt),type:phrase),query:(match_phrase:(filename:job-output.txt))),(\u0027$state\u0027:(store:appState),meta:(alias:!n,disabled:!f,index:\u002794869730-aea8-11ec-9e6a-83741af3fdcd\u0027,key:build_status,negate:!f,params:(query:SUCCESS),type:phrase),query:(match_phrase:(build_status:SUCCESS)))),query:(language:kuery,query:\u0027message:%22Ran:%201076%20tests%20in%22%20OR%20message:%22Ran:%201075%20tests%20in%22\u0027))\n\nand i generated the images with this terrible python script\nhttps://fileshare.seanmooney.info/test-stats.py\nusign the json data form the opensarch respocne in the inspect tab.","commit_id":"ec80cedd8856c3069390a437277fd294e66b5281"},{"author":{"_account_id":27615,"name":"Rajat Dhasmana","email":"rajatdhasmana@gmail.com","username":"whoami-rajat"},"change_message_id":"ef8205bd86d6afe76887d4a1daceb9de00e1db39","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":2,"id":"eb25ec88_1c6be09f","in_reply_to":"d262bae3_7383dd20","updated":"2024-07-04 07:02:29.000000000","message":"Thanks for all the details. I used this patch for my testing[1] as this was the most recent devstack patch that merged and the timeouts started becoming more frequent but looks like this is unrelated.\nas you pointed out, the timeouts were happening before as well and I\u0027m suspecting the reduced concurrency (3) to be the issue there.\nI will abandon this patch as i didn\u0027t yield the desired result.\n\n[1] https://review.opendev.org/c/openstack/glance/+/922909","commit_id":"ec80cedd8856c3069390a437277fd294e66b5281"}]}