)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"6a01888d372de25d19c8d5206d7ac0c97472d2dd","unresolved":true,"context_lines":[{"line_number":9,"context_line":"We make a lot of ShardRange objects, pretty much all the time.  Try to"},{"line_number":10,"context_line":"make it a little better:"},{"line_number":11,"context_line":""},{"line_number":12,"context_line":"  * Add __slots__ to improve memory consumption and attribute lookups."},{"line_number":13,"context_line":"  * Avoid the overhead of catch_warnings() by not tripping the warning."},{"line_number":14,"context_line":""},{"line_number":15,"context_line":"Change-Id: Ib1c698be9e9f579649bc81acf3562c92feb6c8d3"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"1b37123b_cc9a1dd6","line":12,"updated":"2022-08-25 19:55:35.000000000","message":"Helps a little, but not *that* much: using the same shard range set as before, `[ShardRange.from_dict(x) for x in json.loads(d)]` took 116-118ms on the parent patch, compared to 112-113ms with just adding __slots__ -- an improvement, but low single digit percentage.","commit_id":"38866a396310a3cebcdc92b8e82b77aed5098be1"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"6a01888d372de25d19c8d5206d7ac0c97472d2dd","unresolved":true,"context_lines":[{"line_number":10,"context_line":"make it a little better:"},{"line_number":11,"context_line":""},{"line_number":12,"context_line":"  * Add __slots__ to improve memory consumption and attribute lookups."},{"line_number":13,"context_line":"  * Avoid the overhead of catch_warnings() by not tripping the warning."},{"line_number":14,"context_line":""},{"line_number":15,"context_line":"Change-Id: Ib1c698be9e9f579649bc81acf3562c92feb6c8d3"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"f6ce2e24_8b046d3f","line":13,"updated":"2022-08-25 19:55:35.000000000","message":"...whereas just doing this brings it down to 77-79ms.\n\n(All these numbers are on py39.)","commit_id":"38866a396310a3cebcdc92b8e82b77aed5098be1"}],"/PATCHSET_LEVEL":[{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"6a01888d372de25d19c8d5206d7ac0c97472d2dd","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"405504ce_d53d5acf","updated":"2022-08-25 19:55:35.000000000","message":"FWIW, it\u0027s removing the catch_warnings context that\u0027s the big win.","commit_id":"38866a396310a3cebcdc92b8e82b77aed5098be1"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"0c2fd512b15ac3c4c7d635a946ba5b86b2d92d06","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"915daa6e_51516f2e","updated":"2022-08-24 20:23:30.000000000","message":"Few data points using a large (~5900 shard) DB -- first, py2 before the patch:\n\n ++ python -m timeit -s \u0027import json; d\u003dopen(\"nvetl-data-prod.shards.json\").read()\u0027 \u0027json.loads(d)\u0027\n 10 loops, best of 3: 44 msec per loop\n ++ python -m timeit -s \u0027import json; from swift.common.utils import ShardRange; d\u003dopen(\"nvetl-data-prod.shards.json\").read()\u0027 \u0027[ShardRange.from_dict(x) for x in json.loads(d)]\u0027\n 10 loops, best of 3: 217 msec per loop\n\nAnd after:\n\n ++ python -m timeit -s \u0027import json; d\u003dopen(\"nvetl-data-prod.shards.json\").read()\u0027 \u0027json.loads(d)\u0027\n 10 loops, best of 3: 43.5 msec per loop\n ++ python -m timeit -s \u0027import json; from swift.common.utils import ShardRange; d\u003dopen(\"nvetl-data-prod.shards.json\").read()\u0027 \u0027[ShardRange.from_dict(x) for x in json.loads(d)]\u0027\n 10 loops, best of 3: 167 msec per loop\n\nSo like a 25% speed up -- not bad! Let\u0027s switch to py37, though. Before:\n\n ++ python -m timeit -s \u0027import json; d\u003dopen(\"nvetl-data-prod.shards.json\").read()\u0027 \u0027json.loads(d)\u0027\n 20 loops, best of 5: 18.8 msec per loop\n ++ python -m timeit -s \u0027import json; from swift.common.utils import ShardRange; d\u003dopen(\"nvetl-data-prod.shards.json\").read()\u0027 \u0027[ShardRange.from_dict(x) for x in json.loads(d)]\u0027\n 2 loops, best of 5: 147 msec per loop\n\nAfter:\n\n ++ python -m timeit -s \u0027import json; d\u003dopen(\"nvetl-data-prod.shards.json\").read()\u0027 \u0027json.loads(d)\u0027\n 20 loops, best of 5: 18.2 msec per loop\n ++ python -m timeit -s \u0027import json; from swift.common.utils import ShardRange; d\u003dopen(\"nvetl-data-prod.shards.json\").read()\u0027 \u0027[ShardRange.from_dict(x) for x in json.loads(d)]\u0027\n 2 loops, best of 5: 99.9 msec per loop\n\nWith py39, it gets *even better*. Before:\n\n ++ python -m timeit -s \u0027import json; d\u003dopen(\"nvetl-data-prod.shards.json\").read()\u0027 \u0027json.loads(d)\u0027\n 20 loops, best of 5: 16.7 msec per loop\n ++ python -m timeit -s \u0027import json; from swift.common.utils import ShardRange; d\u003dopen(\"nvetl-data-prod.shards.json\").read()\u0027 \u0027[ShardRange.from_dict(x) for x in json.loads(d)]\u0027\n 2 loops, best of 5: 114 msec per loop\n\nAfter:\n\n ++ python -m timeit -s \u0027import json; d\u003dopen(\"nvetl-data-prod.shards.json\").read()\u0027 \u0027json.loads(d)\u0027\n 20 loops, best of 5: 16.5 msec per loop\n ++ python -m timeit -s \u0027import json; from swift.common.utils import ShardRange; d\u003dopen(\"nvetl-data-prod.shards.json\").read()\u0027 \u0027[ShardRange.from_dict(x) for x in json.loads(d)]\u0027\n 5 loops, best of 5: 78.6 msec per loop\n\nSo, this patch brings a 25-45% improvement within any given python version, and upgrading python can *separately* bring like a 50% improvement!","commit_id":"38866a396310a3cebcdc92b8e82b77aed5098be1"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"01a9210fce3bd2dd7890ccbbd8d4857f94e0e121","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"feb253d4_2af70a0c","updated":"2022-08-25 05:06:47.000000000","message":"This is pretty cool. Thanks for the timings. Needed to go lookup this whole _slots_ thing to get an idea. Any spead up is good.\nhttps://docs.python.org/3/reference/datamodel.html#object.__slots__ is pretty useful for those who want to read up.","commit_id":"38866a396310a3cebcdc92b8e82b77aed5098be1"}]}