)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"31e7197c21670156a52cea9228bd1f6bc7696ca0","unresolved":true,"context_lines":[{"line_number":15,"context_line":"   to increase the performace fo swap usage."},{"line_number":16,"context_line":"- sysctl memory and disk tunnings are enabled"},{"line_number":17,"context_line":"- the qemu tb_cache_size is reduced"},{"line_number":18,"context_line":"- mysql memory redcution is enabled"},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"The intent of the optimisation is to allow concurrancy"},{"line_number":21,"context_line":"to be increased to 6 to reduce the over all job time."}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"f5a84385_0b3864e0","line":18,"updated":"2024-06-18 00:45:13.000000000","message":"i also enabled the osc deamon mode thing which seams to be working well in the nova ci so its proably woth enabeling that by default at teh devstack level if that has not already been done","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":8556,"name":"Ghanshyam Maan","display_name":"Ghanshyam Maan","email":"gmaan.os14@gmail.com","username":"ghanshyam"},"change_message_id":"1ba88999885b4693b13e50c32130bacaf1402c30","unresolved":true,"context_lines":[{"line_number":17,"context_line":"- the qemu tb_cache_size is reduced"},{"line_number":18,"context_line":"- mysql memory redcution is enabled"},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"The intent of the optimisation is to allow concurrancy"},{"line_number":21,"context_line":"to be increased to 6 to reduce the over all job time."},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"Change-Id: Ib9c36bd678e8d67aee8ca140d2b0104ed504ec57"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"26355703_73f0ca1f","line":21,"range":{"start_line":20,"start_character":37,"end_line":21,"end_character":53},"updated":"2024-06-21 01:00:45.000000000","message":"++, I am big fan of doing that. It did not work well when we increased the default tempest run concurrency to 6 and it was reduced to 4 in many jobs.","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"}],"/PATCHSET_LEVEL":[{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"ea633fdc951683a88629418c6f4e5cc769474e5e","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"55f99b66_2754860a","updated":"2024-06-20 16:36:05.000000000","message":"Nice investigation and analysis in here. I think the changes look sensible and I\u0027m interested to see how this will do with the increased concurrency. If we\u0027re able to run reliably with concurrency of 6, I think that\u0027s a big win to not be limited to 4 due to resource constraints.","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":8556,"name":"Ghanshyam Maan","display_name":"Ghanshyam Maan","email":"gmaan.os14@gmail.com","username":"ghanshyam"},"change_message_id":"1ba88999885b4693b13e50c32130bacaf1402c30","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"7acb8f40_52122008","updated":"2024-06-21 01:00:45.000000000","message":"lgtm, thanks Sean for working on this. job running fine in https://review.opendev.org/c/openstack/tempest/+/922172 but we will get more data when we will have move job run and on slow node.","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"c43cae70dff86fba3c1974df80c15c5694d6e990","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"5d35318f_32256ac3","updated":"2024-06-19 10:43:51.000000000","message":"massive caveat on all the stats im creating and including in comments that i only got a B in that module in university and its been 14 years since i took that module so they are  proably wrong but i think this support the ideal that the optimisation are actully improving over all performance.\n\ni also surmise that if we are avgerging an execution time that is 92.5% of the timeout with a job varancie around in the 15 to 25% range that support extending the job by the 30 mins i have done which is about the average variance to ensure stablity\n\nwe could maybe set the timeout at 2  hours 15 mins and split the diffentce but i think the extra buffer is not terrible.\n\nif peopel agree i would like to merge this and the patch below it sooner rather then later","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"efd631a9ab968ff297faf9009f0615c39025e5ed","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"0d2a4997_5af01a93","updated":"2024-06-18 10:59:52.000000000","message":"over all i think this is promising.","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":8556,"name":"Ghanshyam Maan","display_name":"Ghanshyam Maan","email":"gmaan.os14@gmail.com","username":"ghanshyam"},"change_message_id":"1ba88999885b4693b13e50c32130bacaf1402c30","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"c9243a95_8964ff5e","in_reply_to":"41571010_1edbc48e","updated":"2024-06-21 01:00:45.000000000","message":"++ on the plan. I am ok to try these in tempest-full-py3 job in parallel and later one if all goes well we can change the default configs.","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1533f9378dab3b814fe12e8977856bd7b93ae725","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"41571010_1edbc48e","in_reply_to":"55f99b66_2754860a","updated":"2024-06-20 20:15:12.000000000","message":"im hoping that if we can run with tese optiomisation for a month that will give us enough data to determin if we should change the default in devstack or in the devstack-tempest job to apply the same change to all or most devstack jobs.\n\ni have a similar set of optimisation for all of the nova jobs but they are currently blocked behind the timeouts which is why i started lookign into this in the first place. i.e speading up the nova gate.","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"ea633fdc951683a88629418c6f4e5cc769474e5e","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"0f07801e_d0bb27b6","in_reply_to":"5d35318f_32256ac3","updated":"2024-06-20 16:36:05.000000000","message":"\u003e massive caveat on all the stats im creating and including in comments that i only got a B in that module in university and its been 14 years since i took that module \n\n😂","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":22873,"name":"Martin Kopec","email":"mkopec@redhat.com","username":"mkopec"},"change_message_id":"5e9f1eb7c0325e66e7ab7af40bcee71c2298b9aa","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"440a8ca6_4cce7fd2","updated":"2024-06-23 22:30:28.000000000","message":"this is excellent, thank you Sean","commit_id":"79d378dc212b1e0234daef58057fb45bad4173ea"}],"zuul.d/integrated-gate.yaml":[{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"601bd13991fc5db71f63d5d3c97d450c40a6145f","unresolved":true,"context_lines":[{"line_number":128,"context_line":""},{"line_number":129,"context_line":"- job:"},{"line_number":130,"context_line":"    name: tempest-integrated-compute"},{"line_number":131,"context_line":"    parent: devstack-tempest"},{"line_number":132,"context_line":"    description: |"},{"line_number":133,"context_line":"      This job runs integration tests for compute. This is"},{"line_number":134,"context_line":"      subset of \u0027tempest-full-py3\u0027 job and run Nova, Neutron, Cinder (except backup tests)"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"6f5fc68a_3c9e3b15","line":131,"updated":"2024-06-18 00:41:59.000000000","message":"assuming this has the desired effect we may want to apply this to the parent job but perhaps we should start with the compute variant to see if there is a noticeable difference.","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"601bd13991fc5db71f63d5d3c97d450c40a6145f","unresolved":true,"context_lines":[{"line_number":150,"context_line":"      # NOTE(sean-k-mooney): we should evaluate if enabling zswap"},{"line_number":151,"context_line":"      # and or increase swap would allow use to avoid OOM but increase"},{"line_number":152,"context_line":"      # concurrency to decrease overall job execution time."},{"line_number":153,"context_line":"      tempest_concurrency: 6"},{"line_number":154,"context_line":"      tox_envlist: integrated-compute"},{"line_number":155,"context_line":"      tempest_exclude_regex: \"\""},{"line_number":156,"context_line":"      configure_swap_size: 8192"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"2c010832_0d0b0176","line":153,"updated":"2024-06-18 00:41:59.000000000","message":"this might be overly optimistic\ni woudl generally be happy with leaveign the concurrency at 4 if its stable but faster due to the overall turnings below.","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"efd631a9ab968ff297faf9009f0615c39025e5ed","unresolved":true,"context_lines":[{"line_number":150,"context_line":"      # NOTE(sean-k-mooney): we should evaluate if enabling zswap"},{"line_number":151,"context_line":"      # and or increase swap would allow use to avoid OOM but increase"},{"line_number":152,"context_line":"      # concurrency to decrease overall job execution time."},{"line_number":153,"context_line":"      tempest_concurrency: 6"},{"line_number":154,"context_line":"      tox_envlist: integrated-compute"},{"line_number":155,"context_line":"      tempest_exclude_regex: \"\""},{"line_number":156,"context_line":"      configure_swap_size: 8192"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"e4ae8077_49a3e999","line":153,"in_reply_to":"2c010832_0d0b0176","updated":"2024-06-18 10:59:52.000000000","message":"over all this seamed to have almost no effect on execution time although with the other memeory treaks below we go no where near an OOM event form what i can tell\nso this seams to be safe with the other turnings.","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"c43cae70dff86fba3c1974df80c15c5694d6e990","unresolved":true,"context_lines":[{"line_number":150,"context_line":"      # NOTE(sean-k-mooney): we should evaluate if enabling zswap"},{"line_number":151,"context_line":"      # and or increase swap would allow use to avoid OOM but increase"},{"line_number":152,"context_line":"      # concurrency to decrease overall job execution time."},{"line_number":153,"context_line":"      tempest_concurrency: 6"},{"line_number":154,"context_line":"      tox_envlist: integrated-compute"},{"line_number":155,"context_line":"      tempest_exclude_regex: \"\""},{"line_number":156,"context_line":"      configure_swap_size: 8192"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"ecc2cb45_c75f6116","line":153,"in_reply_to":"49f976bb_0bd74c0a","updated":"2024-06-19 10:43:51.000000000","message":"ah excelent \n\nhttps://fileshare.seanmooney.info/average_tempest_serial_runtime_for_tempest-integrated-compute_by_provider.png\n\nthis is what i was hoping to see.\n\nthe rax-* providers show an even even more pronoucned slowness to execute the serial tests while still maintianing the tigher variance.\n\nthat implies that the overall cpu/disk performance of the rax noeds is lower for single-threaded tasks  but athat is less pronounced when we use \n\ntakeing the worst case  after filtering form the 3 grpahs we get\n\ndevstack 1010\nserial 1291\nconcurrent 4380\nresutls in 6681  or 1.85 hours for 95% of jobs to complete those 3 phases.\n\nthat doesn not take into account the time for all pre and post playbooks to run which is addaitonal overhead\n\nif i instead look at the optimistic view assumeing the minum within the 2 sigma rage for each value we get \ndevstack 761\nserial 827\nconcurrent 3381\nresulting in 4969 or 1.38 hours\n\nthat corralated to our intuition that in general the job varinace is about 30 mins in aggreate for succeful runs.\n\n\ni proably shoudl have comptued this with a 3 sigma tresshold instead of 2 sigma as a 5% failure rate in ci  for this job when we run 3 variance of it that have to pas in check and gate is  likely to cause a failure due to timeouts\n\n\nrefarming the results form the first and second exectuion of this path with that context\n\nthe first DNM run was on rax-ord \n\nRan: 1075 tests in 3825.3881 sec.\nRan: 28 tests in 1120.4675 sec.\n\n\nthe second run was on rax-iad\nRan: 1075 tests in 3072.9066 sec.\nRan: 28 tests in 1087.0896 sec.\n\nbreaking this down per execution startign with concurrnt\n\nthe rax-ord run took 3825 second  where the min expercte is 3659 and average is 4020 so that about a 4.8% speedup vs the average but its less then the 9% varicne we expect in this provider. its hevillay sewed towards faster over all however.\n\nthe rax-iad node is similar\n3072 vs expected min of 3441 this beats the 1 sigma varinace for a speedup of ~19% when the job varince of 10% woudl acouunt for.\n\nthe rax-iad node seam to have a marge better single traded performace then rax-ord\nat least that is what the serial tempest and devstack install timies imply\n\ni woudl infer that as a result the cpu overhaed of zswap is much less noticable there and it beifits more form the optimisation in this patch.","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"3ca988206bfb59dca5d0786653cd147f0ebdbf15","unresolved":true,"context_lines":[{"line_number":150,"context_line":"      # NOTE(sean-k-mooney): we should evaluate if enabling zswap"},{"line_number":151,"context_line":"      # and or increase swap would allow use to avoid OOM but increase"},{"line_number":152,"context_line":"      # concurrency to decrease overall job execution time."},{"line_number":153,"context_line":"      tempest_concurrency: 6"},{"line_number":154,"context_line":"      tox_envlist: integrated-compute"},{"line_number":155,"context_line":"      tempest_exclude_regex: \"\""},{"line_number":156,"context_line":"      configure_swap_size: 8192"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"49f976bb_0bd74c0a","line":153,"in_reply_to":"92318466_94984c09","updated":"2024-06-19 10:04:17.000000000","message":"https://fileshare.seanmooney.info/average_tempest_runtime_for_tempest-integrated-compute_by_provider.png\n\nlookig a the parallel test execution only we see that  the average comes out to arouhnd 3852 seconds when outlieers are removed.\n\noutliers are removed by applying a z-score treshold of 2 sigma giving a 95% confindance level.\n\nthe gaph also shows the the one sigma max/min time by add/subtracting the computed variance to the average \n\nif we average the the computed mean + variance denoted as max on the graph\nthat \n\nas expected rax-ord is statistically slower on average. although the rax providers also have a lower job to job variance which is interesting compared to ovh.\n\n\nill compute the same for the serial tests.","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"b8c066bb69f04ad92c769dd11939a856d61c9715","unresolved":true,"context_lines":[{"line_number":150,"context_line":"      # NOTE(sean-k-mooney): we should evaluate if enabling zswap"},{"line_number":151,"context_line":"      # and or increase swap would allow use to avoid OOM but increase"},{"line_number":152,"context_line":"      # concurrency to decrease overall job execution time."},{"line_number":153,"context_line":"      tempest_concurrency: 6"},{"line_number":154,"context_line":"      tox_envlist: integrated-compute"},{"line_number":155,"context_line":"      tempest_exclude_regex: \"\""},{"line_number":156,"context_line":"      configure_swap_size: 8192"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"92318466_94984c09","line":153,"in_reply_to":"e4ae8077_49a3e999","updated":"2024-06-18 19:22:11.000000000","message":"the second run compled tempet in \nRan: 1075 tests in 3072.9066 sec.\nRan: 28 tests in 1087.0896 sec.\n\nthe first DNM run was\n\nRan: 1075 tests in 3825.3881 sec.\nRan: 28 tests in 1120.4675 sec.\n\nthe over all job variance on sussful runs is ~ 30 mins so untill i actully \ncompute data i can really infer if this is faster in a statiticlly relevent way but my intuition is sayign https://zuul.opendev.org/t/openstack/build/3c26d402823b4dcb936512b53ee69a88 is not noticeably slower.\n\nhttps://tinyurl.com/5377euzu\nshoudl provdied the data i need but i wont get a chance to review that for a whiel","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"efd631a9ab968ff297faf9009f0615c39025e5ed","unresolved":true,"context_lines":[{"line_number":153,"context_line":"      tempest_concurrency: 6"},{"line_number":154,"context_line":"      tox_envlist: integrated-compute"},{"line_number":155,"context_line":"      tempest_exclude_regex: \"\""},{"line_number":156,"context_line":"      configure_swap_size: 8192"},{"line_number":157,"context_line":"      devstack_localrc:"},{"line_number":158,"context_line":"        USE_PYTHON3: true"},{"line_number":159,"context_line":"        FORCE_CONFIG_DRIVE: true"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"6caaf690_68a3ed48","line":156,"range":{"start_line":156,"start_character":6,"end_line":156,"end_character":31},"updated":"2024-06-18 10:59:52.000000000","message":"we did not seam to actully need this change although i woudl suggest keeping it in any case.\n\n\nthe maxium swap usage in the new run was 2095616KB  just under 2GB\n\nthe older resoult used slightly less swap at 1754112KB or 1.7GB\n\nhavign the extra swap ising a bad thign and i woudl generally like to standardise on 8G i think although im ok with reverting this","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"efd631a9ab968ff297faf9009f0615c39025e5ed","unresolved":true,"context_lines":[{"line_number":158,"context_line":"        USE_PYTHON3: true"},{"line_number":159,"context_line":"        FORCE_CONFIG_DRIVE: true"},{"line_number":160,"context_line":"        ENABLE_VOLUME_MULTIATTACH: true"},{"line_number":161,"context_line":"        ENABLE_SYSCTL_MEM_TUNING: true"},{"line_number":162,"context_line":"        ENABLE_SYSCTL_NET_TUNING: true"},{"line_number":163,"context_line":"        ENABLE_ZSWAP: true"},{"line_number":164,"context_line":"        MYSQL_REDUCE_MEMORY: true"},{"line_number":165,"context_line":"      devstack_local_conf:"},{"line_number":166,"context_line":"        post-config:"},{"line_number":167,"context_line":"          $NOVA_CPU_CONF:"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"9125b8d2_c6d3f975","line":164,"range":{"start_line":161,"start_character":0,"end_line":164,"end_character":33},"updated":"2024-06-18 10:59:52.000000000","message":"over all the memory usage in the new run is lower\n\nmem low point for the first run was 240572 KB\nand on the run with these tweaks was  816636KB\n\nold: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_be5/854823/42/check/tempest-integrated-compute/be535f3/controller/logs/screen-memory_tracker.txt \n```\nJun 18 09:23:06.585593 np0037749383 memory_tracker.sh[117919]: MemAvailable:     239568 kB\nJun 18 09:23:06.585593 np0037749383 memory_tracker.sh[117919]: Buffers:           34508 kB\nJun 18 09:23:06.585593 np0037749383 memory_tracker.sh[117919]: Cached:           295304 kB\nJun 18 09:23:06.585593 np0037749383 memory_tracker.sh[117919]: SwapCached:        63136 kB\n```\n\nnew: https://1e487f940fe7ac2fa65f-5682448930a1accc6cfb803760f4c177.ssl.cf5.rackcdn.com/922172/4/check/tempest-integrated-compute/859ed15/controller/logs/screen-memory_tracker.txt\n```\njun 18 02:11:31.897229 np0037745973 memory_tracker.sh[121032]: MemAvailable:     816384 kB\nJun 18 02:11:31.897229 np0037745973 memory_tracker.sh[121032]: Buffers:          252692 kB\nJun 18 02:11:31.897229 np0037745973 memory_tracker.sh[121032]: Cached:           473076 kB\nJun 18 02:11:31.897229 np0037745973 memory_tracker.sh[121032]: SwapCached:       169580 kB\n```\n\nsince we have more available memory we see that the new node is free to use it more activly for buffers and caching in general partly due to the more active use fo the swap cache.\n\nthe logic over all is tha tby leveraging zswap we are mitigating the slowness of swap and disk io by compuressign the data so we get a speedup aproxiamte to the compuression ratio.\n\ni have not tweaked the algortims used for the optimial values and stuck with the defaults in this run.","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":8556,"name":"Ghanshyam Maan","display_name":"Ghanshyam Maan","email":"gmaan.os14@gmail.com","username":"ghanshyam"},"change_message_id":"1ba88999885b4693b13e50c32130bacaf1402c30","unresolved":true,"context_lines":[{"line_number":170,"context_line":"              # libvirt\u003e\u003d8.0.0"},{"line_number":171,"context_line":"              tb_cache_size: 128"},{"line_number":172,"context_line":"      devstack_services:"},{"line_number":173,"context_line":"        openstack-cli-server: true"},{"line_number":174,"context_line":"        s-account: false"},{"line_number":175,"context_line":"        s-container: false"},{"line_number":176,"context_line":"        s-object: false"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"b58f3749_bc70632c","line":173,"range":{"start_line":173,"start_character":0,"end_line":173,"end_character":34},"updated":"2024-06-21 01:00:45.000000000","message":"irrespective of anything, we should enable it in devstack by default.","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"efd631a9ab968ff297faf9009f0615c39025e5ed","unresolved":true,"context_lines":[{"line_number":170,"context_line":"              # libvirt\u003e\u003d8.0.0"},{"line_number":171,"context_line":"              tb_cache_size: 128"},{"line_number":172,"context_line":"      devstack_services:"},{"line_number":173,"context_line":"        openstack-cli-server: true"},{"line_number":174,"context_line":"        s-account: false"},{"line_number":175,"context_line":"        s-container: false"},{"line_number":176,"context_line":"        s-object: false"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"9320b5db_521e025e","line":173,"range":{"start_line":173,"start_character":0,"end_line":173,"end_character":2},"updated":"2024-06-18 10:59:52.000000000","message":"ok with a sample size of 1 this defincatlly reduced the time executing osc\n\n```\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\nDevStack Component Timing\n (times are in seconds)  \n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\nwait_for_service      20\nasync_wait            64\nosc                  230\napt-get              143\ntest_with_retry        6\ndbsync                 4\npip_install          180\napt-get-update         1\nrun_process           23\n-------------------------\nUnaccounted time      98\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\nTotal runtime        769\n```\nvs\n```\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\nDevStack Component Timing\n (times are in seconds)  \n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\nwait_for_service      21\nasync_wait            49\nosc                  104\napt-get              163\ntest_with_retry        7\ndbsync                 5\npip_install          203\napt-get-update         2\nrun_process           28\n-------------------------\nUnaccounted time     190\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\nTotal runtime        772\n```\nthe pip and apt tiem is higher implying thte storage or network perfomance is slower on the new run\n\nboth were on differnt rax providers\n\nthe unmodifed resutl was from rax-dfw vs rax-ord for the modifed result\n\nbased on the data collection i did both are on the faster side for these providers \nby more then the statistical variance so these late night runs obvioulsy were not experince noisy neighber effect.\n\nhttps://fileshare.seanmooney.info/average_runtime_for_tempest-integrated-compute_by_provider.png\n\nin any case cutting the osc time by more than half I think is definitely worth it.","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"b8c066bb69f04ad92c769dd11939a856d61c9715","unresolved":true,"context_lines":[{"line_number":170,"context_line":"              # libvirt\u003e\u003d8.0.0"},{"line_number":171,"context_line":"              tb_cache_size: 128"},{"line_number":172,"context_line":"      devstack_services:"},{"line_number":173,"context_line":"        openstack-cli-server: true"},{"line_number":174,"context_line":"        s-account: false"},{"line_number":175,"context_line":"        s-container: false"},{"line_number":176,"context_line":"        s-object: false"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"98730582_5e3185a1","line":173,"range":{"start_line":173,"start_character":0,"end_line":173,"end_character":2},"in_reply_to":"9320b5db_521e025e","updated":"2024-06-18 19:22:11.000000000","message":"second run of the dnm patch was faster again. this time on rax-iad\n\n```\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\nDevStack Component Timing\n (times are in seconds)  \n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\nwait_for_service      20\nasync_wait            47 \nosc                   97\napt-get              152\ntest_with_retry        7\ndbsync                 5\npip_install          192\napt-get-update         1\nrun_process           27\n-------------------------\nUnaccounted time     176\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\nTotal runtime        724\n```\nthis is almost entirly due to openstack-cli-server: true","commit_id":"b45ec676be56e930550fa3097e56afd0c7e4827b"},{"author":{"_account_id":13252,"name":"Dr. Jens Harbott","display_name":"Jens Harbott (frickler)","email":"frickler@offenerstapel.de","username":"jrosenboom"},"change_message_id":"845e2e00acb09d6cb5569f815f9ee95e1d232e4d","unresolved":true,"context_lines":[{"line_number":149,"context_line":"      # oom issue, setting the concurrency to 4 in this job."},{"line_number":150,"context_line":"      # NOTE(sean-k-mooney): we should evaluate if enabling zswap"},{"line_number":151,"context_line":"      # and or increase swap would allow use to avoid OOM but increase"},{"line_number":152,"context_line":"      # concurrency to decrease overall job execution time."},{"line_number":153,"context_line":"      tempest_concurrency: 6"},{"line_number":154,"context_line":"      tox_envlist: integrated-compute"},{"line_number":155,"context_line":"      tempest_exclude_regex: \"\""}],"source_content_type":"text/x-yaml","patch_set":2,"id":"d8893fad_94645b29","line":152,"updated":"2024-06-24 14:47:54.000000000","message":"can you amend these comments to match what is now being configured?","commit_id":"79d378dc212b1e0234daef58057fb45bad4173ea"}]}