)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"c6056df0_7a6751d2","updated":"2024-12-06 23:34:59.000000000","message":"this works!\n\nThere\u0027s some significant opportunities to improve the probetest, but MOSTLY in ways that just borrow from existing patterns that I didn\u0027t know about and now have decided I don\u0027t like - so those probably shouldn\u0027t be a blocker.  The `sleep(20)` might be.\n\nI think we need some unittests to merge this.  The probetest assertions themselves strike me as fairly weak by design (basically a \"functional sanity check\") - which is why I think more targeted unittests on the aggregation business logic that shows which keys get rolled up and how - and explore robustness to missing keys or null values are necessary before we merge e.g. a TypeError would blow up the whole process; it\u0027s an open question (possibly worthy of debate) if aggregation *fails* (for whatever reason, maybe a bug or permission/IOError) should the process should die - or \"just\" emit a warning and try again next cycle?\n\nMaybe the \"removed devices eventually reap from aggregation drops\" needs to be addressed before merge as well; maybe those requirements can be expressed if we have some existing aggregation unittests to extend.\n\nMost significant to me in playing with these recon drops I found myself wanting to get a LOT more information out of them - so it\u0027d be REALLY good to fix the `run_forever` bug - because there\u0027s more we can do with this infra to follow-up!","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":15,"id":"fdf09cb5_66536f19","updated":"2024-12-13 00:46:25.000000000","message":"\u003d requested changes \u003d\n\nI don\u0027t think the aggregation logic is correct for removing disks, counting unique a/c across all devices (probably not possible), or estimating memory usage.\n\n\u003d nits \u003d\n\nProbe test is looking better; I think we can do better still.\n\nThe run_once/forever are starting to look better; I think we can get DRY\u0027er still.\n\nTest clean-up is looking good; i don\u0027t remember if there was any blockers.\n\n\u003d conclusion \u003d\n\nNew tests are really helping get into the nuts of the business logic - I think there\u0027s some bugs; we need better tests.  I added some follow-ups.  Good luck!","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"4c8727ccc5e99d740c4c2cb1b1bece906be3488c","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":15,"id":"9a97534c_baa6d7f5","updated":"2024-12-12 22:08:22.000000000","message":"Sorry haven\u0027t really been following the work here. Question/query inline.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"2d52cdf17f58515b3c5eb7854adb59508a2401e7","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":23,"id":"a23b7903_4d539ff4","updated":"2024-12-17 20:09:15.000000000","message":"this is looking great!   add some more tests for the aggreation logic when missing keys or devices and such.","commit_id":"409d545e661d4ed44a7f95fae9f40f76defd2f4b"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"43ef5511af7279dd651dd9d59c86c37db79c1ef2","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":27,"id":"6ef0045d_be1ba32e","updated":"2024-12-18 17:03:22.000000000","message":"WOMM\n\n```\nvagrant@saio:~$ cat /var/cache/swift/node1/object.recon | jq .\n{\n  \"object_updater_per_device\": {\n    \"sdb1\": {\n      \"failures_account_container_count\": 9,\n      \"failures_oldest_timestamp\": 1734540235.14818,\n      \"failures_oldest_timestamp_account_containers\": {\n        \"oldest_count\": 5,\n        \"oldest_entries\": [\n          {\n            \"account\": \"AUTH_user-682400b7-7690-4098-a406-cf9667bb729b-000\",\n            \"container\": \"cont-94e0afe4-2c7c-474a-8a4a-f9ade523efa5-001\",\n            \"timestamp\": 1734540235.14818\n          },\n          {\n            \"account\": \"AUTH_user-682400b7-7690-4098-a406-cf9667bb729b-000\",\n            \"container\": \"cont-bb4eb1f0-89cd-4f45-a953-5b27bec335b6-003\",\n            \"timestamp\": 1734540235.36236\n          },\n          {\n            \"account\": \"AUTH_user-2580e3c8-ea2e-4c50-8070-639beae555e5-001\",\n            \"container\": \"cont-3e7e2ee4-f876-4763-8298-67dec840620e-000\",\n            \"timestamp\": 1734540235.46688\n          },\n          {\n            \"account\": \"AUTH_user-2580e3c8-ea2e-4c50-8070-639beae555e5-001\",\n            \"container\": \"cont-6c9669ec-3dc2-45a5-8793-f47def8b8a24-001\",\n            \"timestamp\": 1734540235.52366\n          },\n          {\n            \"account\": \"AUTH_user-2580e3c8-ea2e-4c50-8070-639beae555e5-001\",\n            \"container\": \"cont-61ec48f9-ad54-457d-ad72-546f56bd6945-002\",\n            \"timestamp\": 1734540235.62496\n          }\n        ]\n      },\n      \"failures_oldest_timestamp_age\": 6.924507141113281,\n      \"tracker_memory_usage\": 4619\n    }\n  },\n  \"object_updater_stats\": {\n    \"failures_account_container_count\": 9,\n    \"failures_oldest_timestamp\": 1734540235.14818,\n    \"failures_oldest_timestamp_account_containers\": {\n      \"oldest_count\": 5,\n      \"oldest_entries\": [\n        {\n          \"account\": \"AUTH_user-682400b7-7690-4098-a406-cf9667bb729b-000\",\n          \"container\": \"cont-94e0afe4-2c7c-474a-8a4a-f9ade523efa5-001\",\n          \"timestamp\": 1734540235.14818\n        },\n        {\n          \"account\": \"AUTH_user-682400b7-7690-4098-a406-cf9667bb729b-000\",\n          \"container\": \"cont-bb4eb1f0-89cd-4f45-a953-5b27bec335b6-003\",\n          \"timestamp\": 1734540235.36236\n        },\n        {\n          \"account\": \"AUTH_user-2580e3c8-ea2e-4c50-8070-639beae555e5-001\",\n          \"container\": \"cont-3e7e2ee4-f876-4763-8298-67dec840620e-000\",\n          \"timestamp\": 1734540235.46688\n        },\n        {\n          \"account\": \"AUTH_user-2580e3c8-ea2e-4c50-8070-639beae555e5-001\",\n          \"container\": \"cont-6c9669ec-3dc2-45a5-8793-f47def8b8a24-001\",\n          \"timestamp\": 1734540235.52366\n        },\n        {\n          \"account\": \"AUTH_user-2580e3c8-ea2e-4c50-8070-639beae555e5-001\",\n          \"container\": \"cont-61ec48f9-ad54-457d-ad72-546f56bd6945-002\",\n          \"timestamp\": 1734540235.62496\n        }\n      ]\n    },\n    \"tracker_memory_usage\": 2677.5\n  },\n  \"object_updater_sweep\": 1.0880858898162842\n}\n```","commit_id":"f2d7fd79d531471924b125977c9367e455deae00"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"cf99ebe849bf597ff2dc84eeb6930033a44e136d","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":33,"id":"2266aac9_6d4e8ef7","updated":"2024-12-19 06:06:02.000000000","message":"Those list-filter-lambda-list-comprehension-or-defaults are ugly as sin, but I don\u0027t really have an improvement to offer.\n\nI wonder if the aggregated memory usage could be improved, though -- or maybe I\u0027m letting the perfect be the enemy of the good?","commit_id":"3a6c587671fac06d0672dda23f7b81a5c2518c17"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"d917173da1b2456362b7443b0fbde4866853a0ca","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":33,"id":"08df9715_c1a95e02","updated":"2024-12-19 03:16:48.000000000","message":"recheck\n\nLooks like an unrelated failure (503 Service Unavailable for Container PUT failed)","commit_id":"3a6c587671fac06d0672dda23f7b81a5c2518c17"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"57478226d0455af03700110d9a41dead04cf3928","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":34,"id":"06b8a26d_732563af","updated":"2024-12-19 17:17:36.000000000","message":"I don\u0027t think the spelling of the call to min/max is worth holding up the fix to master for `run_forever` oldest async stats.","commit_id":"af57922cd8b30cdb333d536f0506c673b591c069"}],"swift/obj/updater.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":558,"context_line":"                self.oldest_async_pendings.get_memory_usage(),"},{"line_number":559,"context_line":"        }"},{"line_number":560,"context_line":"        if elapsed is not None:"},{"line_number":561,"context_line":"            stats[\u0027elapsed\u0027] \u003d elapsed"},{"line_number":562,"context_line":"        return stats"},{"line_number":563,"context_line":""},{"line_number":564,"context_line":"    def dump_recon(self, elapsed):"}],"source_content_type":"text/x-python","patch_set":4,"id":"fa011523_e91fad27","line":561,"updated":"2024-12-06 23:34:59.000000000","message":"why exclude this key?\n\nWhat should the \"empty device\" report look like?\n\n```\n    \"sdb5\": {\n      \"elapsed\": 0.002420186996459961,\n      \"failures_account_container_count\": 0,\n      \"failures_oldest_timestamp\": null,\n      \"failures_oldest_timestamp_account_containers\": {\n        \"oldest_count\": 0,\n        \"oldest_entries\": []\n      },\n      \"failures_oldest_timestamp_age\": null,\n      \"tracker_memory_usage\": 487\n    }\n```\n\nCan we differentiate it from the \"all successful\" report?\n\n```\n  \"object_updater_per_device\": {\n    \"sdb1\": {\n      \"elapsed\": 0.0016231536865234375,\n      \"failures_account_container_count\": 0,\n      \"failures_oldest_timestamp\": null,\n      \"failures_oldest_timestamp_account_containers\": {\n        \"oldest_count\": 0,\n        \"oldest_entries\": []\n      },\n      \"failures_oldest_timestamp_age\": null,\n      \"tracker_memory_usage\": 487\n    },\n    \"sdb5\": {\n      \"elapsed\": 0.001383066177368164,\n      \"failures_account_container_count\": 0,\n      \"failures_oldest_timestamp\": null,\n      \"failures_oldest_timestamp_account_containers\": {\n        \"oldest_count\": 0,\n        \"oldest_entries\": []\n      },\n      \"failures_oldest_timestamp_age\": null,\n      \"tracker_memory_usage\": 487\n    }\n  },\n  \"object_updater_stats\": {\n    \"failures_account_container_count\": 0,\n    \"failures_oldest_timestamp\": null,\n    \"failures_oldest_timestamp_account_containers\": {\n      \"oldest_count\": 0,\n      \"oldest_entries\": []\n    },\n    \"failures_oldest_timestamp_age\": null,\n    \"tracker_memory_usage\": 974\n  },\n```","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"580c03e576fd53b7a2e76772abf50e03124d7271","unresolved":false,"context_lines":[{"line_number":558,"context_line":"                self.oldest_async_pendings.get_memory_usage(),"},{"line_number":559,"context_line":"        }"},{"line_number":560,"context_line":"        if elapsed is not None:"},{"line_number":561,"context_line":"            stats[\u0027elapsed\u0027] \u003d elapsed"},{"line_number":562,"context_line":"        return stats"},{"line_number":563,"context_line":""},{"line_number":564,"context_line":"    def dump_recon(self, elapsed):"}],"source_content_type":"text/x-python","patch_set":4,"id":"972e2e90_455e7d29","line":561,"in_reply_to":"fa011523_e91fad27","updated":"2024-12-12 19:53:26.000000000","message":"At the moment, I don\u0027t believe there\u0027s any differentiation","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":561,"context_line":"            stats[\u0027elapsed\u0027] \u003d elapsed"},{"line_number":562,"context_line":"        return stats"},{"line_number":563,"context_line":""},{"line_number":564,"context_line":"    def dump_recon(self, elapsed):"},{"line_number":565,"context_line":"        \"\"\"Gathers stats and dumps recon cache.\"\"\""},{"line_number":566,"context_line":"        object_updater_stats \u003d self._gather_recon_stats()"},{"line_number":567,"context_line":"        dump_recon_cache("}],"source_content_type":"text/x-python","patch_set":4,"id":"ffa9f989_11267919","line":564,"updated":"2024-12-06 23:34:59.000000000","message":"oh wow, so this is still used by once mode to pull stats from the main process oldest_async_pendings tracker (which do get updated when there\u0027s no forking)\n\nMaybe part of THIS fix would be DRY out `run_once` to use the same fork loop as `run_forever` so we can\u0027t accidently forget when testing with `once` it\u0027s a totally different code path for recon aggregation than `run_forever`!","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"580c03e576fd53b7a2e76772abf50e03124d7271","unresolved":false,"context_lines":[{"line_number":561,"context_line":"            stats[\u0027elapsed\u0027] \u003d elapsed"},{"line_number":562,"context_line":"        return stats"},{"line_number":563,"context_line":""},{"line_number":564,"context_line":"    def dump_recon(self, elapsed):"},{"line_number":565,"context_line":"        \"\"\"Gathers stats and dumps recon cache.\"\"\""},{"line_number":566,"context_line":"        object_updater_stats \u003d self._gather_recon_stats()"},{"line_number":567,"context_line":"        dump_recon_cache("}],"source_content_type":"text/x-python","patch_set":4,"id":"e7799a15_dbf878ac","line":564,"in_reply_to":"ffa9f989_11267919","updated":"2024-12-12 19:53:26.000000000","message":"Right, moved the forking behaviour to `run_once` as well.","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":580,"context_line":"            {\u0027object_updater_per_device\u0027: {device: disk_recon_stats}},"},{"line_number":581,"context_line":"            self.rcache,"},{"line_number":582,"context_line":"            self.logger,"},{"line_number":583,"context_line":"        )"},{"line_number":584,"context_line":""},{"line_number":585,"context_line":"    def aggregate_and_dump_recon(self, elapsed):"},{"line_number":586,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":4,"id":"34186d8c_13d10ed5","line":583,"updated":"2024-12-06 23:34:59.000000000","message":"this seems to be working about like we\u0027d hope/want:\n\n```\nvagrant@saio:~$ cat /var/cache/swift/node1/object.recon | jq .\n{\n  \"object_updater_per_device\": {\n    \"sdb1\": {\n      \"elapsed\": 0.43767619132995605,\n      \"failures_account_container_count\": 9,\n      \"failures_oldest_timestamp\": 1733524795.79359,\n      \"failures_oldest_timestamp_account_containers\": {\n        \"oldest_count\": 5,\n        \"oldest_entries\": [\n          {\n            \"account\": \"AUTH_user000\",\n            \"container\": \"cont001\",\n            \"timestamp\": \"1733524795.79359\"\n          },\n          {\n            \"account\": \"AUTH_user001\",\n            \"container\": \"cont000\",\n            \"timestamp\": \"1733524796.20382\"\n          },\n          {\n            \"account\": \"AUTH_user001\",\n            \"container\": \"cont001\",\n            \"timestamp\": \"1733524796.22699\"\n          },\n          {\n            \"account\": \"AUTH_user001\",\n            \"container\": \"cont002\",\n            \"timestamp\": \"1733524796.32789\"\n          },\n          {\n            \"account\": \"AUTH_user001\",\n            \"container\": \"cont003\",\n            \"timestamp\": \"1733524796.47432\"\n          }\n        ]\n      },\n      \"failures_oldest_timestamp_age\": 14.380641222000122,\n      \"tracker_memory_usage\": 4249\n    },\n    \"sdb5\": {\n      \"elapsed\": 0.006646394729614258,\n      \"failures_account_container_count\": 0,\n      \"failures_oldest_timestamp\": null,\n      \"failures_oldest_timestamp_account_containers\": {\n        \"oldest_count\": 0,\n        \"oldest_entries\": []\n      },\n      \"failures_oldest_timestamp_age\": null,\n      \"tracker_memory_usage\": 487\n    }\n  },\n  \"object_updater_stats\": {\n    \"failures_account_container_count\": 9,\n    \"failures_oldest_timestamp\": 1733524795.79359,\n    \"failures_oldest_timestamp_account_containers\": {\n      \"oldest_count\": 5,\n      \"oldest_entries\": [\n        {\n          \"account\": \"AUTH_user000\",\n          \"container\": \"cont001\",\n          \"timestamp\": \"1733524795.79359\"\n        },\n        {\n          \"account\": \"AUTH_user001\",\n          \"container\": \"cont000\",\n          \"timestamp\": \"1733524796.20382\"\n        },\n        {\n          \"account\": \"AUTH_user001\",\n          \"container\": \"cont001\",\n          \"timestamp\": \"1733524796.22699\"\n        },\n        {\n          \"account\": \"AUTH_user001\",\n          \"container\": \"cont002\",\n          \"timestamp\": \"1733524796.32789\"\n        },\n        {\n          \"account\": \"AUTH_user001\",\n          \"container\": \"cont003\",\n          \"timestamp\": \"1733524796.47432\"\n        }\n      ]\n    },\n    \"tracker_memory_usage\": 4736\n  },\n  \"object_updater_sweep\": 0.9413132667541504\n}\n```","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"580c03e576fd53b7a2e76772abf50e03124d7271","unresolved":false,"context_lines":[{"line_number":580,"context_line":"            {\u0027object_updater_per_device\u0027: {device: disk_recon_stats}},"},{"line_number":581,"context_line":"            self.rcache,"},{"line_number":582,"context_line":"            self.logger,"},{"line_number":583,"context_line":"        )"},{"line_number":584,"context_line":""},{"line_number":585,"context_line":"    def aggregate_and_dump_recon(self, elapsed):"},{"line_number":586,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":4,"id":"bb6487ca_486fe625","line":583,"in_reply_to":"34186d8c_13d10ed5","updated":"2024-12-12 19:53:26.000000000","message":"Acknowledged","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":582,"context_line":"            self.logger,"},{"line_number":583,"context_line":"        )"},{"line_number":584,"context_line":""},{"line_number":585,"context_line":"    def aggregate_and_dump_recon(self, elapsed):"},{"line_number":586,"context_line":"        \"\"\""},{"line_number":587,"context_line":"        Aggregate recon stats across devices and dump the result to the recon"},{"line_number":588,"context_line":"        cache."}],"source_content_type":"text/x-python","patch_set":4,"id":"de9fcbdd_8f605c70","line":585,"updated":"2024-12-06 23:34:59.000000000","message":"can we unittest `dump_device_recon` and `aggregate_and_dump_recon` to establish their expected cooperation contract?","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"580c03e576fd53b7a2e76772abf50e03124d7271","unresolved":false,"context_lines":[{"line_number":582,"context_line":"            self.logger,"},{"line_number":583,"context_line":"        )"},{"line_number":584,"context_line":""},{"line_number":585,"context_line":"    def aggregate_and_dump_recon(self, elapsed):"},{"line_number":586,"context_line":"        \"\"\""},{"line_number":587,"context_line":"        Aggregate recon stats across devices and dump the result to the recon"},{"line_number":588,"context_line":"        cache."}],"source_content_type":"text/x-python","patch_set":4,"id":"9c9f8b24_681f04d0","line":585,"in_reply_to":"de9fcbdd_8f605c70","updated":"2024-12-12 19:53:26.000000000","message":"Done","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":621,"context_line":"            \u0027failures_oldest_timestamp_account_containers\u0027: {"},{"line_number":622,"context_line":"                \u0027oldest_count\u0027: aggregated_oldest_count,"},{"line_number":623,"context_line":"                \u0027oldest_entries\u0027: aggregated_oldest_entries,"},{"line_number":624,"context_line":"            },"},{"line_number":625,"context_line":"        }"},{"line_number":626,"context_line":"        dump_recon_cache("},{"line_number":627,"context_line":"            {"}],"source_content_type":"text/x-python","patch_set":4,"id":"dc5ffacd_faeb8511","line":624,"updated":"2024-12-06 23:34:59.000000000","message":"is it interesting/annoying/disappointing that we don\u0027t get ANY of the existing per-device log stats aggregated in recon?\n\n```\nDec  6 22:40:18 saio object-updater-6010: Object update sweep of sdb1 completed: 0.43s, 0 successes, 22 failures, 0 quarantines, 0 unlinks, 0 outdated_unlinks, 0 errors, 0 redirects, 0 skips, 0 deferrals, \n0 drains\n```\n\nOn my nodes they *all* reported 22-24 failures on their replica policy device; which maybe makes sense in that probe test because we create 5 async in 12 containers with half of the containers shut down?\n\n    (12ac * 5async) * 3replica / 2failures / 4nodes ~\u003d 22.5","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"580c03e576fd53b7a2e76772abf50e03124d7271","unresolved":false,"context_lines":[{"line_number":621,"context_line":"            \u0027failures_oldest_timestamp_account_containers\u0027: {"},{"line_number":622,"context_line":"                \u0027oldest_count\u0027: aggregated_oldest_count,"},{"line_number":623,"context_line":"                \u0027oldest_entries\u0027: aggregated_oldest_entries,"},{"line_number":624,"context_line":"            },"},{"line_number":625,"context_line":"        }"},{"line_number":626,"context_line":"        dump_recon_cache("},{"line_number":627,"context_line":"            {"}],"source_content_type":"text/x-python","patch_set":4,"id":"5f3ce5d4_831ab365","line":624,"in_reply_to":"dc5ffacd_faeb8511","updated":"2024-12-12 19:53:26.000000000","message":"Acknowledged","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":625,"context_line":"        }"},{"line_number":626,"context_line":"        dump_recon_cache("},{"line_number":627,"context_line":"            {"},{"line_number":628,"context_line":"                \u0027object_updater_sweep\u0027: elapsed,"},{"line_number":629,"context_line":"                \u0027object_updater_stats\u0027: aggregated_stats,"},{"line_number":630,"context_line":"            },"},{"line_number":631,"context_line":"            self.rcache,"}],"source_content_type":"text/x-python","patch_set":4,"id":"b0bcf03b_911832e6","line":628,"updated":"2024-12-06 23:34:59.000000000","message":"I found myself *really* missing an `object_updater_last` stat like other processes dump that says when the previous cycle finished (i.e. the time when the recon was generated)\n\n```\n  \"replication_last\": 1733526547.050478,\n  \"replication_stats\": {\n    \"attempted\": 43,\n    \"failure\": 0,\n    \"hashmatch\": 86,\n    \"remove\": 0,\n    \"rsync\": 0,\n    \"success\": 86,\n    \"suffix_count\": 44,\n    \"suffix_hash\": 12,\n    \"suffix_sync\": 0\n  },\n  \"replication_time\": 0.022743554910024007\n```","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"580c03e576fd53b7a2e76772abf50e03124d7271","unresolved":false,"context_lines":[{"line_number":625,"context_line":"        }"},{"line_number":626,"context_line":"        dump_recon_cache("},{"line_number":627,"context_line":"            {"},{"line_number":628,"context_line":"                \u0027object_updater_sweep\u0027: elapsed,"},{"line_number":629,"context_line":"                \u0027object_updater_stats\u0027: aggregated_stats,"},{"line_number":630,"context_line":"            },"},{"line_number":631,"context_line":"            self.rcache,"}],"source_content_type":"text/x-python","patch_set":4,"id":"3a64c705_2b123dcd","line":628,"in_reply_to":"b0bcf03b_911832e6","updated":"2024-12-12 19:53:26.000000000","message":"added here: https://review.opendev.org/c/openstack/swift/+/937632","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":626,"context_line":"        dump_recon_cache("},{"line_number":627,"context_line":"            {"},{"line_number":628,"context_line":"                \u0027object_updater_sweep\u0027: elapsed,"},{"line_number":629,"context_line":"                \u0027object_updater_stats\u0027: aggregated_stats,"},{"line_number":630,"context_line":"            },"},{"line_number":631,"context_line":"            self.rcache,"},{"line_number":632,"context_line":"            self.logger,"}],"source_content_type":"text/x-python","patch_set":4,"id":"4a7a3cec_8ce464b4","line":629,"updated":"2024-12-06 23:34:59.000000000","message":"I feel like this is also where we\u0027d want to clear out stats from removed devices?  Sort of like the replicator?\n\n```\n        # Clear out entries for old local devices that we no longer have\n        devices_to_remove \u003d set(per_disk_stats) - set(self.all_local_devices)\n        if devices_to_remove:\n            recon_update[\u0027object_replication_per_disk\u0027] \u003d {\n                dtr: {} for dtr in devices_to_remove}\n```","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"580c03e576fd53b7a2e76772abf50e03124d7271","unresolved":false,"context_lines":[{"line_number":626,"context_line":"        dump_recon_cache("},{"line_number":627,"context_line":"            {"},{"line_number":628,"context_line":"                \u0027object_updater_sweep\u0027: elapsed,"},{"line_number":629,"context_line":"                \u0027object_updater_stats\u0027: aggregated_stats,"},{"line_number":630,"context_line":"            },"},{"line_number":631,"context_line":"            self.rcache,"},{"line_number":632,"context_line":"            self.logger,"}],"source_content_type":"text/x-python","patch_set":4,"id":"81939a64_3579e3a5","line":629,"in_reply_to":"4a7a3cec_8ce464b4","updated":"2024-12-12 19:53:26.000000000","message":"Right, done.","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":630,"context_line":"            },"},{"line_number":631,"context_line":"            self.rcache,"},{"line_number":632,"context_line":"            self.logger,"},{"line_number":633,"context_line":"        )"},{"line_number":634,"context_line":""},{"line_number":635,"context_line":"    def _load_update(self, device, update_path):"},{"line_number":636,"context_line":"        try:"}],"source_content_type":"text/x-python","patch_set":4,"id":"dac98919_0ee8a6fe","line":633,"updated":"2024-12-06 23:34:59.000000000","message":"this last bit feels like it could get out of sync with the code in `dump_recon_cache` leading to further inconsistencies with run_once and run_forever stats or making it more likely we write this bug again.\n\nI would really like the outcome of this fix to be \"it\u0027s not really possible anymore to have stats that work in run_once but break in run_forever ever again\"","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"580c03e576fd53b7a2e76772abf50e03124d7271","unresolved":false,"context_lines":[{"line_number":630,"context_line":"            },"},{"line_number":631,"context_line":"            self.rcache,"},{"line_number":632,"context_line":"            self.logger,"},{"line_number":633,"context_line":"        )"},{"line_number":634,"context_line":""},{"line_number":635,"context_line":"    def _load_update(self, device, update_path):"},{"line_number":636,"context_line":"        try:"}],"source_content_type":"text/x-python","patch_set":4,"id":"cb8029cc_cc0bc96a","line":633,"in_reply_to":"dac98919_0ee8a6fe","updated":"2024-12-12 19:53:26.000000000","message":"Merged the logic for `run_once` and `run_forever`","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":499,"context_line":"                    # so a simple warning is sufficient."},{"line_number":500,"context_line":"                    self.logger.warning(\u0027Skipping: %s\u0027, err)"},{"line_number":501,"context_line":"                    continue"},{"line_number":502,"context_line":"                while len(pids) \u003e\u003d self.updater_workers:"},{"line_number":503,"context_line":"                    pids.remove(os.wait()[0])"},{"line_number":504,"context_line":"                pid \u003d os.fork()"},{"line_number":505,"context_line":"                if pid:"}],"source_content_type":"text/x-python","patch_set":15,"id":"2f0bf760_48763d56","side":"PARENT","line":502,"updated":"2024-12-13 00:46:25.000000000","message":"I guess previously `updater_workers \u003d -500` wasn\u0027t an \"error\" - but the result was always \"you get one process\"","commit_id":"fe7928ea8aa45b15fde3154cd8ccdfabc43053a0"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":499,"context_line":"                    # so a simple warning is sufficient."},{"line_number":500,"context_line":"                    self.logger.warning(\u0027Skipping: %s\u0027, err)"},{"line_number":501,"context_line":"                    continue"},{"line_number":502,"context_line":"                while len(pids) \u003e\u003d self.updater_workers:"},{"line_number":503,"context_line":"                    pids.remove(os.wait()[0])"},{"line_number":504,"context_line":"                pid \u003d os.fork()"},{"line_number":505,"context_line":"                if pid:"}],"source_content_type":"text/x-python","patch_set":15,"id":"453a2a18_18388ffd","side":"PARENT","line":502,"in_reply_to":"2f0bf760_48763d56","updated":"2024-12-17 16:25:55.000000000","message":"This is true.","commit_id":"fe7928ea8aa45b15fde3154cd8ccdfabc43053a0"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":427,"context_line":"        self.interval \u003d float(conf.get(\u0027interval\u0027, 300))"},{"line_number":428,"context_line":"        self.container_ring \u003d None"},{"line_number":429,"context_line":"        self.concurrency \u003d int(conf.get(\u0027concurrency\u0027, 8))"},{"line_number":430,"context_line":"        self.updater_workers \u003d int(conf.get(\u0027updater_workers\u0027, 1))"},{"line_number":431,"context_line":"        if \u0027slowdown\u0027 in conf:"},{"line_number":432,"context_line":"            self.logger.warning("},{"line_number":433,"context_line":"                \u0027The slowdown option is deprecated in favor of \u0027"}],"source_content_type":"text/x-python","patch_set":15,"id":"d1b51607_d3e8fc0e","line":430,"updated":"2024-12-13 00:46:25.000000000","message":"this should probably use `utils.conf.non_negtive_int`\n\nI think we can fix this in this change; but we\u0027d probably have to write more tests and include an \"UpgradeImpact\" in the commit message.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"11c8c3da7063aac03db0234be28fdbd62efa4cb6","unresolved":false,"context_lines":[{"line_number":427,"context_line":"        self.interval \u003d float(conf.get(\u0027interval\u0027, 300))"},{"line_number":428,"context_line":"        self.container_ring \u003d None"},{"line_number":429,"context_line":"        self.concurrency \u003d int(conf.get(\u0027concurrency\u0027, 8))"},{"line_number":430,"context_line":"        self.updater_workers \u003d int(conf.get(\u0027updater_workers\u0027, 1))"},{"line_number":431,"context_line":"        if \u0027slowdown\u0027 in conf:"},{"line_number":432,"context_line":"            self.logger.warning("},{"line_number":433,"context_line":"                \u0027The slowdown option is deprecated in favor of \u0027"}],"source_content_type":"text/x-python","patch_set":15,"id":"3826e288_57491ac9","line":430,"in_reply_to":"d1b51607_d3e8fc0e","updated":"2024-12-13 20:29:29.000000000","message":"I thought it made more sense to have this as a separate patch:\nhttps://review.opendev.org/c/openstack/swift/+/937738","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":488,"context_line":"            self.begin \u003d time.time()"},{"line_number":489,"context_line":"            self._process_devices()"},{"line_number":490,"context_line":"            elapsed \u003d time.time() - self.begin"},{"line_number":491,"context_line":"            self.logger.info(\u0027Object update sweep completed: %.02fs\u0027,"},{"line_number":492,"context_line":"                             elapsed)"},{"line_number":493,"context_line":"            self.aggregate_and_dump_recon(elapsed)"},{"line_number":494,"context_line":"            if elapsed \u003c self.interval:"}],"source_content_type":"text/x-python","patch_set":15,"id":"1a417280_aa2c479c","line":491,"updated":"2024-12-13 00:46:25.000000000","message":"compare this to the log msg of the \"Object update sweep of device completed\" and it might be helpful when grepping logs if this explicitly said \"of all devices\" or something to the effect?","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":488,"context_line":"            self.begin \u003d time.time()"},{"line_number":489,"context_line":"            self._process_devices()"},{"line_number":490,"context_line":"            elapsed \u003d time.time() - self.begin"},{"line_number":491,"context_line":"            self.logger.info(\u0027Object update sweep completed: %.02fs\u0027,"},{"line_number":492,"context_line":"                             elapsed)"},{"line_number":493,"context_line":"            self.aggregate_and_dump_recon(elapsed)"},{"line_number":494,"context_line":"            if elapsed \u003c self.interval:"}],"source_content_type":"text/x-python","patch_set":15,"id":"dc7670e5_3f985b16","line":491,"in_reply_to":"1a417280_aa2c479c","updated":"2024-12-17 16:25:55.000000000","message":"Done","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":490,"context_line":"            elapsed \u003d time.time() - self.begin"},{"line_number":491,"context_line":"            self.logger.info(\u0027Object update sweep completed: %.02fs\u0027,"},{"line_number":492,"context_line":"                             elapsed)"},{"line_number":493,"context_line":"            self.aggregate_and_dump_recon(elapsed)"},{"line_number":494,"context_line":"            if elapsed \u003c self.interval:"},{"line_number":495,"context_line":"                time.sleep(self.interval - elapsed)"},{"line_number":496,"context_line":""}],"source_content_type":"text/x-python","patch_set":15,"id":"7ffe4681_6c2a026a","line":493,"updated":"2024-12-13 00:46:25.000000000","message":"the body of this loop smells suspiciously similar to the body of `run_once`\n\nI think it would be great if `run_forever` just called `run_once` in a loop; with special handling for the startup/inter-cycle delay.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":490,"context_line":"            elapsed \u003d time.time() - self.begin"},{"line_number":491,"context_line":"            self.logger.info(\u0027Object update sweep completed: %.02fs\u0027,"},{"line_number":492,"context_line":"                             elapsed)"},{"line_number":493,"context_line":"            self.aggregate_and_dump_recon(elapsed)"},{"line_number":494,"context_line":"            if elapsed \u003c self.interval:"},{"line_number":495,"context_line":"                time.sleep(self.interval - elapsed)"},{"line_number":496,"context_line":""}],"source_content_type":"text/x-python","patch_set":15,"id":"af6f6e72_fa523975","line":493,"in_reply_to":"7ffe4681_6c2a026a","updated":"2024-12-17 16:25:55.000000000","message":"Done.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":496,"context_line":""},{"line_number":497,"context_line":"    def run_once(self, *args, **kwargs):"},{"line_number":498,"context_line":"        \"\"\"Run the updater once.\"\"\""},{"line_number":499,"context_line":"        self.logger.info(\u0027Begin object update single threaded sweep\u0027)"},{"line_number":500,"context_line":"        self.begin \u003d time.time()"},{"line_number":501,"context_line":"        self.stats.reset()"},{"line_number":502,"context_line":"        self.oldest_async_pendings.reset()"}],"source_content_type":"text/x-python","patch_set":15,"id":"74839265_2292d091","line":499,"updated":"2024-12-13 00:46:25.000000000","message":"\"single threaded\" probably isn\u0027t accurate anymore.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":496,"context_line":""},{"line_number":497,"context_line":"    def run_once(self, *args, **kwargs):"},{"line_number":498,"context_line":"        \"\"\"Run the updater once.\"\"\""},{"line_number":499,"context_line":"        self.logger.info(\u0027Begin object update single threaded sweep\u0027)"},{"line_number":500,"context_line":"        self.begin \u003d time.time()"},{"line_number":501,"context_line":"        self.stats.reset()"},{"line_number":502,"context_line":"        self.oldest_async_pendings.reset()"}],"source_content_type":"text/x-python","patch_set":15,"id":"4618b326_000e2e96","line":499,"in_reply_to":"74839265_2292d091","updated":"2024-12-17 16:25:55.000000000","message":"Changed","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":499,"context_line":"        self.logger.info(\u0027Begin object update single threaded sweep\u0027)"},{"line_number":500,"context_line":"        self.begin \u003d time.time()"},{"line_number":501,"context_line":"        self.stats.reset()"},{"line_number":502,"context_line":"        self.oldest_async_pendings.reset()"},{"line_number":503,"context_line":"        self._process_devices()"},{"line_number":504,"context_line":"        elapsed \u003d time.time() - self.begin"},{"line_number":505,"context_line":"        self.logger.info("}],"source_content_type":"text/x-python","patch_set":15,"id":"4b874cf9_6857ab06","line":502,"updated":"2024-12-13 00:46:25.000000000","message":"is it bad that run_forever doesn\u0027t appear to reset these between cycles?","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":499,"context_line":"        self.logger.info(\u0027Begin object update single threaded sweep\u0027)"},{"line_number":500,"context_line":"        self.begin \u003d time.time()"},{"line_number":501,"context_line":"        self.stats.reset()"},{"line_number":502,"context_line":"        self.oldest_async_pendings.reset()"},{"line_number":503,"context_line":"        self._process_devices()"},{"line_number":504,"context_line":"        elapsed \u003d time.time() - self.begin"},{"line_number":505,"context_line":"        self.logger.info("}],"source_content_type":"text/x-python","patch_set":15,"id":"86ce9de2_b0694e7d","line":502,"in_reply_to":"4b874cf9_6857ab06","updated":"2024-12-17 16:25:55.000000000","message":"Well, I think it kind of does. Just not directly.  In the child processes `_process_device_in_child` calls self.oldest_async_pendings.reset().","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":505,"context_line":"        self.logger.info("},{"line_number":506,"context_line":"            (\u0027Object update single-threaded sweep completed: \u0027"},{"line_number":507,"context_line":"             \u0027%(elapsed).02fs, %(stats)s\u0027),"},{"line_number":508,"context_line":"            {\u0027elapsed\u0027: elapsed, \u0027stats\u0027: self.stats})"},{"line_number":509,"context_line":"        self.aggregate_and_dump_recon(elapsed)"},{"line_number":510,"context_line":""},{"line_number":511,"context_line":"    def _process_devices(self):"}],"source_content_type":"text/x-python","patch_set":15,"id":"1ed642e1_fd3b909c","line":508,"updated":"2024-12-13 00:46:25.000000000","message":"maybe it\u0027s just not helpful for `run_once` to continue to emit it\u0027s `self.stats` since all the interesting numbers were accumulated in the children\u0027s stats and logged at the end of their sweep(s).","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":505,"context_line":"        self.logger.info("},{"line_number":506,"context_line":"            (\u0027Object update single-threaded sweep completed: \u0027"},{"line_number":507,"context_line":"             \u0027%(elapsed).02fs, %(stats)s\u0027),"},{"line_number":508,"context_line":"            {\u0027elapsed\u0027: elapsed, \u0027stats\u0027: self.stats})"},{"line_number":509,"context_line":"        self.aggregate_and_dump_recon(elapsed)"},{"line_number":510,"context_line":""},{"line_number":511,"context_line":"    def _process_devices(self):"}],"source_content_type":"text/x-python","patch_set":15,"id":"e0be42c2_3db04044","line":508,"in_reply_to":"1ed642e1_fd3b909c","updated":"2024-12-17 16:25:55.000000000","message":"Removed self.stats from log message","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":522,"context_line":"                # We don\u0027t count this as an error. The occasional"},{"line_number":523,"context_line":"                # unmounted drive is part of normal cluster operations,"},{"line_number":524,"context_line":"                # so a simple warning is sufficient."},{"line_number":525,"context_line":"                self.logger.warning(\u0027Skipping: %s\u0027, err)"},{"line_number":526,"context_line":"                continue"},{"line_number":527,"context_line":"            while len(pids) \u003e\u003d self.updater_workers:"},{"line_number":528,"context_line":"                pids.remove(os.wait()[0])"}],"source_content_type":"text/x-python","patch_set":15,"id":"74d48965_a35b0007","line":525,"updated":"2024-12-13 00:46:25.000000000","message":"FWIW `check_drive` includes the path and reason:\n\n```\nraise ValueError(\u0027%s is not mounted\u0027 % path)\n```","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":522,"context_line":"                # We don\u0027t count this as an error. The occasional"},{"line_number":523,"context_line":"                # unmounted drive is part of normal cluster operations,"},{"line_number":524,"context_line":"                # so a simple warning is sufficient."},{"line_number":525,"context_line":"                self.logger.warning(\u0027Skipping: %s\u0027, err)"},{"line_number":526,"context_line":"                continue"},{"line_number":527,"context_line":"            while len(pids) \u003e\u003d self.updater_workers:"},{"line_number":528,"context_line":"                pids.remove(os.wait()[0])"}],"source_content_type":"text/x-python","patch_set":15,"id":"7ff44f75_fa6eccae","line":525,"in_reply_to":"74d48965_a35b0007","updated":"2024-12-17 16:25:55.000000000","message":"Right. This `self.logger.warning(\u0027Skipping: %s\u0027, err)` was just taken from the previous incarnation of `run_forever`. If you think it should change it please let me know.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":534,"context_line":"                sys.exit()"},{"line_number":535,"context_line":""},{"line_number":536,"context_line":"        while pids:"},{"line_number":537,"context_line":"            pids.remove(os.wait()[0])"},{"line_number":538,"context_line":""},{"line_number":539,"context_line":"    def _process_device_in_child(self, dev_path, device):"},{"line_number":540,"context_line":"        \"\"\"Process a single device in a forked child process.\"\"\""}],"source_content_type":"text/x-python","patch_set":15,"id":"21158a07_b554fca6","line":537,"updated":"2024-12-13 00:46:25.000000000","message":"it would help my puny brain if this part of the diff appeared above `run_forever` closer to from where it was extracted.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":534,"context_line":"                sys.exit()"},{"line_number":535,"context_line":""},{"line_number":536,"context_line":"        while pids:"},{"line_number":537,"context_line":"            pids.remove(os.wait()[0])"},{"line_number":538,"context_line":""},{"line_number":539,"context_line":"    def _process_device_in_child(self, dev_path, device):"},{"line_number":540,"context_line":"        \"\"\"Process a single device in a forked child process.\"\"\""}],"source_content_type":"text/x-python","patch_set":15,"id":"debb5d6b_218cc19b","line":537,"in_reply_to":"21158a07_b554fca6","updated":"2024-12-17 16:25:55.000000000","message":"Moved _process_device_in_child and _process_devices up.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":542,"context_line":"        os.environ.pop(\u0027NOTIFY_SOCKET\u0027, None)"},{"line_number":543,"context_line":"        eventlet_monkey_patch()"},{"line_number":544,"context_line":"        self.stats.reset()"},{"line_number":545,"context_line":"        self.oldest_async_pendings.reset()"},{"line_number":546,"context_line":"        forkbegin \u003d time.time()"},{"line_number":547,"context_line":"        self.object_sweep(dev_path)"},{"line_number":548,"context_line":"        elapsed \u003d time.time() - forkbegin"}],"source_content_type":"text/x-python","patch_set":15,"id":"50a50d68_7cca5736","line":545,"updated":"2024-12-13 00:46:25.000000000","message":"maybe it\u0027s just not helpful anymore for `run_once` to reset these since it always happens in the child?  Maybe the only \"stats\" that come out of the parent come from aggregation (which would make sense)","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":542,"context_line":"        os.environ.pop(\u0027NOTIFY_SOCKET\u0027, None)"},{"line_number":543,"context_line":"        eventlet_monkey_patch()"},{"line_number":544,"context_line":"        self.stats.reset()"},{"line_number":545,"context_line":"        self.oldest_async_pendings.reset()"},{"line_number":546,"context_line":"        forkbegin \u003d time.time()"},{"line_number":547,"context_line":"        self.object_sweep(dev_path)"},{"line_number":548,"context_line":"        elapsed \u003d time.time() - forkbegin"}],"source_content_type":"text/x-python","patch_set":15,"id":"51cbeddc_50fb083c","line":545,"in_reply_to":"50a50d68_7cca5736","updated":"2024-12-17 16:25:55.000000000","message":"Right. Removed.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":554,"context_line":"        self.dump_device_recon(device)"},{"line_number":555,"context_line":""},{"line_number":556,"context_line":"    def _gather_recon_stats(self):"},{"line_number":557,"context_line":"        \"\"\"Gather stats common to both device and overall recon dumps.\"\"\""},{"line_number":558,"context_line":"        stats \u003d {"},{"line_number":559,"context_line":"            \u0027failures_oldest_timestamp\u0027:"},{"line_number":560,"context_line":"                self.oldest_async_pendings.get_oldest_timestamp(),"}],"source_content_type":"text/x-python","patch_set":15,"id":"82fc1250_d38c2ced","line":557,"updated":"2024-12-13 00:46:25.000000000","message":"Is this docstring correct?  It looks like it\u0027s only ever called from `dump_device_recon`?","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":554,"context_line":"        self.dump_device_recon(device)"},{"line_number":555,"context_line":""},{"line_number":556,"context_line":"    def _gather_recon_stats(self):"},{"line_number":557,"context_line":"        \"\"\"Gather stats common to both device and overall recon dumps.\"\"\""},{"line_number":558,"context_line":"        stats \u003d {"},{"line_number":559,"context_line":"            \u0027failures_oldest_timestamp\u0027:"},{"line_number":560,"context_line":"                self.oldest_async_pendings.get_oldest_timestamp(),"}],"source_content_type":"text/x-python","patch_set":15,"id":"46c06fa4_61d6f123","line":557,"in_reply_to":"82fc1250_d38c2ced","updated":"2024-12-17 16:25:55.000000000","message":"It is not. Updated.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":587,"context_line":"        recon_cache \u003d load_recon_cache(self.rcache)"},{"line_number":588,"context_line":"        device_stats \u003d recon_cache.get(\u0027object_updater_per_device\u0027, {})"},{"line_number":589,"context_line":""},{"line_number":590,"context_line":"        devices_to_remove \u003d set(device_stats) - set(self._all_local_devices)"},{"line_number":591,"context_line":"        if devices_to_remove:"},{"line_number":592,"context_line":"            for dtr in devices_to_remove:"},{"line_number":593,"context_line":"                device_stats[dtr] \u003d {}"}],"source_content_type":"text/x-python","patch_set":15,"id":"6d0cafb8_87893eb0","line":590,"updated":"2024-12-13 00:46:25.000000000","message":"it\u0027s probably six-one-way-half-a-dozen the other; but from a testing perspective and maybe easy-of-maintenace my \"gut\" says the list of devices should be a local to `run_once`\n\n```\ndef run_once()\n    start \u003d time()\n    log.begin()\n    devices \u003d self._list_devices()\n    self._process_devices(devices)\n    elapsed \u003d time() - start\n    log.finished(N devices)\n    self._aggregate_devices(devices, elapsed)\n```    \n\nAs it is I think if you call `ou._aggreate_devices` w/o setting `ou._all_local_devices \u003d [something valid]` you get ... probably I guess an `AttributeError`?\n\nMaking it an input to the method makes it a smaller/more-isolated \"unit\"","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":587,"context_line":"        recon_cache \u003d load_recon_cache(self.rcache)"},{"line_number":588,"context_line":"        device_stats \u003d recon_cache.get(\u0027object_updater_per_device\u0027, {})"},{"line_number":589,"context_line":""},{"line_number":590,"context_line":"        devices_to_remove \u003d set(device_stats) - set(self._all_local_devices)"},{"line_number":591,"context_line":"        if devices_to_remove:"},{"line_number":592,"context_line":"            for dtr in devices_to_remove:"},{"line_number":593,"context_line":"                device_stats[dtr] \u003d {}"}],"source_content_type":"text/x-python","patch_set":15,"id":"5b9d5ea1_ac6a392e","line":590,"in_reply_to":"6d0cafb8_87893eb0","updated":"2024-12-17 16:25:55.000000000","message":"Done.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"4c8727ccc5e99d740c4c2cb1b1bece906be3488c","unresolved":true,"context_lines":[{"line_number":600,"context_line":"            aggregated_oldest_entries.extend(container_data.get("},{"line_number":601,"context_line":"                \u0027oldest_entries\u0027, []))"},{"line_number":602,"context_line":""},{"line_number":603,"context_line":"        aggregated_oldest_entries \u003d aggregated_oldest_entries[:self.dump_count]"},{"line_number":604,"context_line":"        aggregated_oldest_count \u003d len(aggregated_oldest_entries)"},{"line_number":605,"context_line":""},{"line_number":606,"context_line":"        aggregated_stats \u003d {"}],"source_content_type":"text/x-python","patch_set":15,"id":"3ca1528c_6e727e61","line":603,"updated":"2024-12-12 22:08:22.000000000","message":"Would we need to sort the aggregated_oldest_entries before we take the first self.dump_count of  them? We seem to just extendings from the devices.. I\u0027d assume we\u0027re want to make sure we\u0027re only dumping x of the oldest we have?\n\nOr am I miss understanding what\u0027s happening here?","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":600,"context_line":"            aggregated_oldest_entries.extend(container_data.get("},{"line_number":601,"context_line":"                \u0027oldest_entries\u0027, []))"},{"line_number":602,"context_line":""},{"line_number":603,"context_line":"        aggregated_oldest_entries \u003d aggregated_oldest_entries[:self.dump_count]"},{"line_number":604,"context_line":"        aggregated_oldest_count \u003d len(aggregated_oldest_entries)"},{"line_number":605,"context_line":""},{"line_number":606,"context_line":"        aggregated_stats \u003d {"}],"source_content_type":"text/x-python","patch_set":15,"id":"e1f9d107_2d13bcc1","line":603,"updated":"2024-12-13 00:46:25.000000000","message":"i don\u0027t understand don\u0027t we have to resort this list after we aggregate all the devices entries?  Is that something the parent process OldestAsyncTracker can help with?","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":600,"context_line":"            aggregated_oldest_entries.extend(container_data.get("},{"line_number":601,"context_line":"                \u0027oldest_entries\u0027, []))"},{"line_number":602,"context_line":""},{"line_number":603,"context_line":"        aggregated_oldest_entries \u003d aggregated_oldest_entries[:self.dump_count]"},{"line_number":604,"context_line":"        aggregated_oldest_count \u003d len(aggregated_oldest_entries)"},{"line_number":605,"context_line":""},{"line_number":606,"context_line":"        aggregated_stats \u003d {"}],"source_content_type":"text/x-python","patch_set":15,"id":"d32ed984_0179df01","line":603,"in_reply_to":"3ca1528c_6e727e61","updated":"2024-12-17 16:25:55.000000000","message":"Yes, we would. thank you.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":600,"context_line":"            aggregated_oldest_entries.extend(container_data.get("},{"line_number":601,"context_line":"                \u0027oldest_entries\u0027, []))"},{"line_number":602,"context_line":""},{"line_number":603,"context_line":"        aggregated_oldest_entries \u003d aggregated_oldest_entries[:self.dump_count]"},{"line_number":604,"context_line":"        aggregated_oldest_count \u003d len(aggregated_oldest_entries)"},{"line_number":605,"context_line":""},{"line_number":606,"context_line":"        aggregated_stats \u003d {"}],"source_content_type":"text/x-python","patch_set":15,"id":"adc8b448_ff629233","line":603,"in_reply_to":"e1f9d107_2d13bcc1","updated":"2024-12-17 16:25:55.000000000","message":"Yes, we should re sort the list.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":601,"context_line":"                \u0027oldest_entries\u0027, []))"},{"line_number":602,"context_line":""},{"line_number":603,"context_line":"        aggregated_oldest_entries \u003d aggregated_oldest_entries[:self.dump_count]"},{"line_number":604,"context_line":"        aggregated_oldest_count \u003d len(aggregated_oldest_entries)"},{"line_number":605,"context_line":""},{"line_number":606,"context_line":"        aggregated_stats \u003d {"},{"line_number":607,"context_line":"            \u0027failures_account_container_count\u0027: sum("}],"source_content_type":"text/x-python","patch_set":15,"id":"65f454fd_d5d168fe","line":604,"updated":"2024-12-13 00:46:25.000000000","message":"the idea here is that the `aggregated_oldest_failures` might be *smaller* than the `dump_count`","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":601,"context_line":"                \u0027oldest_entries\u0027, []))"},{"line_number":602,"context_line":""},{"line_number":603,"context_line":"        aggregated_oldest_entries \u003d aggregated_oldest_entries[:self.dump_count]"},{"line_number":604,"context_line":"        aggregated_oldest_count \u003d len(aggregated_oldest_entries)"},{"line_number":605,"context_line":""},{"line_number":606,"context_line":"        aggregated_stats \u003d {"},{"line_number":607,"context_line":"            \u0027failures_account_container_count\u0027: sum("}],"source_content_type":"text/x-python","patch_set":15,"id":"1db35d66_adf923b3","line":604,"in_reply_to":"65f454fd_d5d168fe","updated":"2024-12-17 16:25:55.000000000","message":"Acknowledged","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":607,"context_line":"            \u0027failures_account_container_count\u0027: sum("},{"line_number":608,"context_line":"                stats.get(\u0027failures_account_container_count\u0027, 0)"},{"line_number":609,"context_line":"                for stats in device_stats.values()"},{"line_number":610,"context_line":"            ),"},{"line_number":611,"context_line":"            \u0027tracker_memory_usage\u0027: ("},{"line_number":612,"context_line":"                sum("},{"line_number":613,"context_line":"                    stats.get(\u0027tracker_memory_usage\u0027, 0)"}],"source_content_type":"text/x-python","patch_set":15,"id":"c5b56bd8_3b9cd506","line":610,"updated":"2024-12-13 00:46:25.000000000","message":"i don\u0027t get it; there\u0027s possibly going to be some dupes across devices - maybe we just shouldn\u0027t aggregate this number?","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":607,"context_line":"            \u0027failures_account_container_count\u0027: sum("},{"line_number":608,"context_line":"                stats.get(\u0027failures_account_container_count\u0027, 0)"},{"line_number":609,"context_line":"                for stats in device_stats.values()"},{"line_number":610,"context_line":"            ),"},{"line_number":611,"context_line":"            \u0027tracker_memory_usage\u0027: ("},{"line_number":612,"context_line":"                sum("},{"line_number":613,"context_line":"                    stats.get(\u0027tracker_memory_usage\u0027, 0)"}],"source_content_type":"text/x-python","patch_set":15,"id":"b745a737_a0785d69","line":610,"in_reply_to":"c5b56bd8_3b9cd506","updated":"2024-12-17 16:25:55.000000000","message":"Right, I think maybe a max might have some merit? The sum doesn\u0027t make much sense","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":612,"context_line":"                sum("},{"line_number":613,"context_line":"                    stats.get(\u0027tracker_memory_usage\u0027, 0)"},{"line_number":614,"context_line":"                    for stats in device_stats.values()"},{"line_number":615,"context_line":"                ) / self.updater_workers if self.updater_workers else 0"},{"line_number":616,"context_line":"            ),"},{"line_number":617,"context_line":"            \u0027failures_oldest_timestamp\u0027: min("},{"line_number":618,"context_line":"                ("}],"source_content_type":"text/x-python","patch_set":15,"id":"d8dfbf65_f92c93a6","line":615,"updated":"2024-12-13 00:46:25.000000000","message":"`else / 0` - that raises a `ZeroDivisionError` right!?\n\noh... no I guess the `/` binds tighter than the else, so `if self.updater_workers \u003d 0` (what does that even mean?) we just report `tracker_memory_usage \u003d 0` is that correct?\n\nProbably the best thing to do would be:\n\n```\nsum(per_device_memory) / max(self.updater_workers, 1)\n```","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":612,"context_line":"                sum("},{"line_number":613,"context_line":"                    stats.get(\u0027tracker_memory_usage\u0027, 0)"},{"line_number":614,"context_line":"                    for stats in device_stats.values()"},{"line_number":615,"context_line":"                ) / self.updater_workers if self.updater_workers else 0"},{"line_number":616,"context_line":"            ),"},{"line_number":617,"context_line":"            \u0027failures_oldest_timestamp\u0027: min("},{"line_number":618,"context_line":"                ("}],"source_content_type":"text/x-python","patch_set":15,"id":"41ea5845_783ea61d","line":615,"in_reply_to":"d8dfbf65_f92c93a6","updated":"2024-12-17 16:25:55.000000000","message":"Yes, it won\u0027t raise a `ZeroDivisionError`. Changed to:\n\n```\n            \u0027tracker_memory_usage\u0027: sum(\n                stats.get(\u0027tracker_memory_usage\u0027, 0)\n                for stats in device_stats.values()\n            ) / max(self.updater_workers, 1),\n```","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":618,"context_line":"                ("},{"line_number":619,"context_line":"                    stats.get(\u0027failures_oldest_timestamp\u0027)"},{"line_number":620,"context_line":"                    for stats in device_stats.values()"},{"line_number":621,"context_line":"                    if stats.get(\u0027failures_oldest_timestamp\u0027) is not None"},{"line_number":622,"context_line":"                ),"},{"line_number":623,"context_line":"                default\u003dNone,"},{"line_number":624,"context_line":"            ),"}],"source_content_type":"text/x-python","patch_set":15,"id":"27729cb3_52e12ada","line":621,"updated":"2024-12-13 00:46:25.000000000","message":"this seems like a weird spelling;  I guess it\u0027s trying to do three things at once\n\n1) the `failures_oldest_timestamp` may not exist\n2) the `failures_oldest_timestamp` may be None/null\n3) find the oldest (of the existing \u0026 non-none) `failures_oldest_timestamp`\n\nCan we eliminate one of those possibilities?  Like make it so the key always exists; but sometimes is None.  Or if the key exists it WILL be non-None?\n\nLike this spelling seems reasonable:\n\n```\n\u003e\u003e\u003e min((d[\u0027foo\u0027] for d in [{}, {\u0027bar\u0027: 3}] if \u0027foo\u0027 in d), default\u003dNone)\n\u003e\u003e\u003e min((d[\u0027foo\u0027] for d in [{}, {\u0027bar\u0027: 3}, {\u0027foo\u0027: 1}, {\u0027foo\u0027: 1000}] if \u0027foo\u0027 in d), default\u003dNone)\n1\n```\n\nbut blows up if `{\u0027foo\u0027: None}`\n\nIf you *really* wanted to simplify collection the BEST data-exchange formats have the property of \"the key will ALWAYS exist and ALWAYS be the same type\"\n\n```\n\u003e\u003e\u003e import json\n\u003e\u003e\u003e json.dumps({\u0027oldest_timestamp\u0027: float(\u0027inf\u0027)})\n\u0027{\"oldest_timestamp\": Infinity}\u0027\n```\n\nShould allow for a simple:\n\n```\noldest_failure \u003d min(stats, key\u003doperator.itemgetter(\u0027oldest_timestamp\u0027))\n```\n\n^ which is why it\u0027s nice if data-exchange-formats/apis have the property \"the key will always exist and always be the same type\" (easier to write less bugs in consumers).  You always hear people complaining about bad APIs with inconsistent/malformed response structures.\n\nAlthough *apparently* JSON5 is not widely supported for cross-machine data-exchange:\n\nhttps://evanhahn.com/pythons-nonstandard-json-encoding/\n\nI would expect the golang node exporter to choke on a device with no failures reporting the \"oldest_timestamp\" as `Infinity`:\n\nhttps://github.com/golang/go/issues/59627\n\nSo maybe better:\n\n```\n\u003e\u003e\u003e datetime.datetime.max.timestamp()\n253402322400.0\n```\n\nBut that could make graphs look weird if there\u0027s ever not any failures... it\u0027s tricky!\n\nHow about:\n\n```\n\u003e\u003e\u003e stats \u003d [{}, {\u0027oldest_timestamp\u0027: None}, {\u0027oldest_timestamp\u0027: 1}, {\u0027oldest_timestmap\u0027: 100}]\n\u003e\u003e\u003e min(filter(lambda x: x is not None, (s.get(\u0027oldest_timestamp\u0027) for s in stats)), default\u003dNone)\n1\n\u003e\u003e\u003e stats \u003d []\n\u003e\u003e\u003e min(filter(lambda x: x is not None, (s.get(\u0027oldest_timestamp\u0027) for s in stats)), default\u003dNone)\n\u003e\u003e\u003e\n```","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"f0d1c72b24874e9e254e83c8923d7615870e2225","unresolved":false,"context_lines":[{"line_number":618,"context_line":"                ("},{"line_number":619,"context_line":"                    stats.get(\u0027failures_oldest_timestamp\u0027)"},{"line_number":620,"context_line":"                    for stats in device_stats.values()"},{"line_number":621,"context_line":"                    if stats.get(\u0027failures_oldest_timestamp\u0027) is not None"},{"line_number":622,"context_line":"                ),"},{"line_number":623,"context_line":"                default\u003dNone,"},{"line_number":624,"context_line":"            ),"}],"source_content_type":"text/x-python","patch_set":15,"id":"377ae7d5_3e2e5c4e","line":621,"in_reply_to":"27729cb3_52e12ada","updated":"2024-12-17 20:10:28.000000000","message":"Acknowledged","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"43ef5511af7279dd651dd9d59c86c37db79c1ef2","unresolved":true,"context_lines":[{"line_number":529,"context_line":"        time.sleep(random() * self.interval)"},{"line_number":530,"context_line":"        while True:"},{"line_number":531,"context_line":"            self.run_once(*args, **kwargs)"},{"line_number":532,"context_line":"            elapsed \u003d time.time() - self.begin"},{"line_number":533,"context_line":"            if elapsed \u003c self.interval:"},{"line_number":534,"context_line":"                time.sleep(self.interval - elapsed)"},{"line_number":535,"context_line":""}],"source_content_type":"text/x-python","patch_set":26,"id":"b23c199b_7c22cf2d","line":532,"updated":"2024-12-18 17:03:22.000000000","message":"you could return this from `run_once` and use it in run_forever ...","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"26728f09eafbb5ba626e079eee682cf220f1a367","unresolved":false,"context_lines":[{"line_number":529,"context_line":"        time.sleep(random() * self.interval)"},{"line_number":530,"context_line":"        while True:"},{"line_number":531,"context_line":"            self.run_once(*args, **kwargs)"},{"line_number":532,"context_line":"            elapsed \u003d time.time() - self.begin"},{"line_number":533,"context_line":"            if elapsed \u003c self.interval:"},{"line_number":534,"context_line":"                time.sleep(self.interval - elapsed)"},{"line_number":535,"context_line":""}],"source_content_type":"text/x-python","patch_set":26,"id":"2f3d1d9e_58cbffb1","line":532,"in_reply_to":"b23c199b_7c22cf2d","updated":"2024-12-18 17:51:35.000000000","message":"Done.","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"43ef5511af7279dd651dd9d59c86c37db79c1ef2","unresolved":true,"context_lines":[{"line_number":580,"context_line":"        recon_cache \u003d load_recon_cache(self.rcache)"},{"line_number":581,"context_line":"        device_stats \u003d recon_cache.get(\u0027object_updater_per_device\u0027, {})"},{"line_number":582,"context_line":"        if not isinstance(device_stats, dict):"},{"line_number":583,"context_line":"            raise TypeError(\u0027object_updater_per_device must be a dict\u0027)"},{"line_number":584,"context_line":"        device_stats \u003d {k: (v if v is not None else {})"},{"line_number":585,"context_line":"                        for k, v in device_stats.items()}"},{"line_number":586,"context_line":""}],"source_content_type":"text/x-python","patch_set":26,"id":"1b5bb15d_5cb92db4","line":583,"updated":"2024-12-18 17:03:22.000000000","message":"ok, so missing `object_updater_per_device` key is ok (maybe all unmounted?); but an unexpected type will raise a type error.","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"26728f09eafbb5ba626e079eee682cf220f1a367","unresolved":false,"context_lines":[{"line_number":580,"context_line":"        recon_cache \u003d load_recon_cache(self.rcache)"},{"line_number":581,"context_line":"        device_stats \u003d recon_cache.get(\u0027object_updater_per_device\u0027, {})"},{"line_number":582,"context_line":"        if not isinstance(device_stats, dict):"},{"line_number":583,"context_line":"            raise TypeError(\u0027object_updater_per_device must be a dict\u0027)"},{"line_number":584,"context_line":"        device_stats \u003d {k: (v if v is not None else {})"},{"line_number":585,"context_line":"                        for k, v in device_stats.items()}"},{"line_number":586,"context_line":""}],"source_content_type":"text/x-python","patch_set":26,"id":"76dd7ecb_b644e89d","line":583,"in_reply_to":"1b5bb15d_5cb92db4","updated":"2024-12-18 17:51:35.000000000","message":"Acknowledged","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"43ef5511af7279dd651dd9d59c86c37db79c1ef2","unresolved":true,"context_lines":[{"line_number":582,"context_line":"        if not isinstance(device_stats, dict):"},{"line_number":583,"context_line":"            raise TypeError(\u0027object_updater_per_device must be a dict\u0027)"},{"line_number":584,"context_line":"        device_stats \u003d {k: (v if v is not None else {})"},{"line_number":585,"context_line":"                        for k, v in device_stats.items()}"},{"line_number":586,"context_line":""},{"line_number":587,"context_line":"        devices_to_remove \u003d set(device_stats) - set(devices)"},{"line_number":588,"context_line":"        update_device_stats \u003d {d: {} for d in devices_to_remove}"}],"source_content_type":"text/x-python","patch_set":26,"id":"2aafb26b_0d35f72d","line":585,"updated":"2024-12-18 17:03:22.000000000","message":"... rather than an AttributeError","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"26728f09eafbb5ba626e079eee682cf220f1a367","unresolved":false,"context_lines":[{"line_number":582,"context_line":"        if not isinstance(device_stats, dict):"},{"line_number":583,"context_line":"            raise TypeError(\u0027object_updater_per_device must be a dict\u0027)"},{"line_number":584,"context_line":"        device_stats \u003d {k: (v if v is not None else {})"},{"line_number":585,"context_line":"                        for k, v in device_stats.items()}"},{"line_number":586,"context_line":""},{"line_number":587,"context_line":"        devices_to_remove \u003d set(device_stats) - set(devices)"},{"line_number":588,"context_line":"        update_device_stats \u003d {d: {} for d in devices_to_remove}"}],"source_content_type":"text/x-python","patch_set":26,"id":"d4c885bd_441acd58","line":585,"in_reply_to":"2aafb26b_0d35f72d","updated":"2024-12-18 17:51:35.000000000","message":"Acknowledged","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"43ef5511af7279dd651dd9d59c86c37db79c1ef2","unresolved":true,"context_lines":[{"line_number":594,"context_line":"                \u0027failures_oldest_timestamp_account_containers\u0027, {})"},{"line_number":595,"context_line":"            aggregated_oldest_entries.extend(container_data.get("},{"line_number":596,"context_line":"                \u0027oldest_entries\u0027, []))"},{"line_number":597,"context_line":"        aggregated_oldest_entries.sort(key\u003dlambda x: x[\u0027timestamp\u0027])"},{"line_number":598,"context_line":"        aggregated_oldest_entries \u003d aggregated_oldest_entries[:self.dump_count]"},{"line_number":599,"context_line":"        aggregated_oldest_count \u003d len(aggregated_oldest_entries)"},{"line_number":600,"context_line":""}],"source_content_type":"text/x-python","patch_set":26,"id":"cf548d57_b71b473e","line":597,"updated":"2024-12-18 17:03:22.000000000","message":"ok so elements in `oldest_entries` must be dicts and must have the key `timestamp`","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"26728f09eafbb5ba626e079eee682cf220f1a367","unresolved":false,"context_lines":[{"line_number":594,"context_line":"                \u0027failures_oldest_timestamp_account_containers\u0027, {})"},{"line_number":595,"context_line":"            aggregated_oldest_entries.extend(container_data.get("},{"line_number":596,"context_line":"                \u0027oldest_entries\u0027, []))"},{"line_number":597,"context_line":"        aggregated_oldest_entries.sort(key\u003dlambda x: x[\u0027timestamp\u0027])"},{"line_number":598,"context_line":"        aggregated_oldest_entries \u003d aggregated_oldest_entries[:self.dump_count]"},{"line_number":599,"context_line":"        aggregated_oldest_count \u003d len(aggregated_oldest_entries)"},{"line_number":600,"context_line":""}],"source_content_type":"text/x-python","patch_set":26,"id":"e88fa2ca_0f81f5da","line":597,"in_reply_to":"cf548d57_b71b473e","updated":"2024-12-18 17:51:35.000000000","message":"Acknowledged","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"43ef5511af7279dd651dd9d59c86c37db79c1ef2","unresolved":true,"context_lines":[{"line_number":613,"context_line":"                )"},{"line_number":614,"context_line":"                / len(device_stats)"},{"line_number":615,"context_line":"            )"},{"line_number":616,"context_line":"            * max(self.updater_workers, 1)"},{"line_number":617,"context_line":"            if device_stats"},{"line_number":618,"context_line":"            else 0,"},{"line_number":619,"context_line":"            \u0027failures_oldest_timestamp\u0027: min("}],"source_content_type":"text/x-python","patch_set":26,"id":"700ac1c3_4d08ca1e","line":616,"updated":"2024-12-18 17:03:22.000000000","message":"so `(avg-per-worker-memory * #-of-concurrent-workers) if device_stats else 0`","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"26728f09eafbb5ba626e079eee682cf220f1a367","unresolved":false,"context_lines":[{"line_number":613,"context_line":"                )"},{"line_number":614,"context_line":"                / len(device_stats)"},{"line_number":615,"context_line":"            )"},{"line_number":616,"context_line":"            * max(self.updater_workers, 1)"},{"line_number":617,"context_line":"            if device_stats"},{"line_number":618,"context_line":"            else 0,"},{"line_number":619,"context_line":"            \u0027failures_oldest_timestamp\u0027: min("}],"source_content_type":"text/x-python","patch_set":26,"id":"200fe947_c1edfef6","line":616,"in_reply_to":"700ac1c3_4d08ca1e","updated":"2024-12-18 17:51:35.000000000","message":"Acknowledged","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"43ef5511af7279dd651dd9d59c86c37db79c1ef2","unresolved":true,"context_lines":[{"line_number":617,"context_line":"            if device_stats"},{"line_number":618,"context_line":"            else 0,"},{"line_number":619,"context_line":"            \u0027failures_oldest_timestamp\u0027: min("},{"line_number":620,"context_line":"                filter("},{"line_number":621,"context_line":"                    lambda x: x is not None,"},{"line_number":622,"context_line":"                    [stats.get(\u0027failures_oldest_timestamp\u0027)"},{"line_number":623,"context_line":"                     for stats in device_stats.values()],"}],"source_content_type":"text/x-python","patch_set":26,"id":"76fef917_64e0f799","line":620,"updated":"2024-12-18 17:03:22.000000000","message":"so if there\u0027s no failures this filter is empty and we get the default None","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"26728f09eafbb5ba626e079eee682cf220f1a367","unresolved":false,"context_lines":[{"line_number":617,"context_line":"            if device_stats"},{"line_number":618,"context_line":"            else 0,"},{"line_number":619,"context_line":"            \u0027failures_oldest_timestamp\u0027: min("},{"line_number":620,"context_line":"                filter("},{"line_number":621,"context_line":"                    lambda x: x is not None,"},{"line_number":622,"context_line":"                    [stats.get(\u0027failures_oldest_timestamp\u0027)"},{"line_number":623,"context_line":"                     for stats in device_stats.values()],"}],"source_content_type":"text/x-python","patch_set":26,"id":"3690a967_a50cb597","line":620,"in_reply_to":"76fef917_64e0f799","updated":"2024-12-18 17:51:35.000000000","message":"Acknowledged","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"43ef5511af7279dd651dd9d59c86c37db79c1ef2","unresolved":true,"context_lines":[{"line_number":633,"context_line":"            {"},{"line_number":634,"context_line":"                \u0027object_updater_sweep\u0027: elapsed,"},{"line_number":635,"context_line":"                \u0027object_updater_stats\u0027: aggregated_stats,"},{"line_number":636,"context_line":"                \u0027object_updater_per_device\u0027: update_device_stats,"},{"line_number":637,"context_line":"            },"},{"line_number":638,"context_line":"            self.rcache,"},{"line_number":639,"context_line":"            self.logger,"}],"source_content_type":"text/x-python","patch_set":26,"id":"b462cb39_ba2a9c10","line":636,"updated":"2024-12-18 17:03:22.000000000","message":"nice, and these updates remove stale devices!","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"26728f09eafbb5ba626e079eee682cf220f1a367","unresolved":false,"context_lines":[{"line_number":633,"context_line":"            {"},{"line_number":634,"context_line":"                \u0027object_updater_sweep\u0027: elapsed,"},{"line_number":635,"context_line":"                \u0027object_updater_stats\u0027: aggregated_stats,"},{"line_number":636,"context_line":"                \u0027object_updater_per_device\u0027: update_device_stats,"},{"line_number":637,"context_line":"            },"},{"line_number":638,"context_line":"            self.rcache,"},{"line_number":639,"context_line":"            self.logger,"}],"source_content_type":"text/x-python","patch_set":26,"id":"3bc71e94_d4dc05e7","line":636,"in_reply_to":"b462cb39_ba2a9c10","updated":"2024-12-18 17:51:35.000000000","message":"Acknowledged","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"43ef5511af7279dd651dd9d59c86c37db79c1ef2","unresolved":true,"context_lines":[{"line_number":539,"context_line":"        self.begin \u003d time.time()"},{"line_number":540,"context_line":"        devices \u003d self._listdir(self.devices)"},{"line_number":541,"context_line":"        self._process_devices(devices)"},{"line_number":542,"context_line":"        elapsed \u003d time.time() - self.begin"},{"line_number":543,"context_line":"        self.logger.info("},{"line_number":544,"context_line":"            (\u0027Object update sweep of all devices completed: \u0027"},{"line_number":545,"context_line":"             \u0027%(elapsed).02fs\u0027),"}],"source_content_type":"text/x-python","patch_set":27,"id":"caaee1c5_1e9aa97f","line":542,"updated":"2024-12-18 17:03:22.000000000","message":"... instead of re-calculating.","commit_id":"f2d7fd79d531471924b125977c9367e455deae00"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"26728f09eafbb5ba626e079eee682cf220f1a367","unresolved":false,"context_lines":[{"line_number":539,"context_line":"        self.begin \u003d time.time()"},{"line_number":540,"context_line":"        devices \u003d self._listdir(self.devices)"},{"line_number":541,"context_line":"        self._process_devices(devices)"},{"line_number":542,"context_line":"        elapsed \u003d time.time() - self.begin"},{"line_number":543,"context_line":"        self.logger.info("},{"line_number":544,"context_line":"            (\u0027Object update sweep of all devices completed: \u0027"},{"line_number":545,"context_line":"             \u0027%(elapsed).02fs\u0027),"}],"source_content_type":"text/x-python","patch_set":27,"id":"168b2cdb_a11e963d","line":542,"in_reply_to":"caaee1c5_1e9aa97f","updated":"2024-12-18 17:51:35.000000000","message":"Acknowledged","commit_id":"f2d7fd79d531471924b125977c9367e455deae00"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"cf99ebe849bf597ff2dc84eeb6930033a44e136d","unresolved":true,"context_lines":[{"line_number":603,"context_line":"                list(filter("},{"line_number":604,"context_line":"                    lambda x: x is not None,"},{"line_number":605,"context_line":"                    ("},{"line_number":606,"context_line":"                        stats.get(\u0027failures_account_container_count\u0027, 0)"},{"line_number":607,"context_line":"                        for stats in device_stats.values()"},{"line_number":608,"context_line":"                    )"},{"line_number":609,"context_line":"                )) or [0],"}],"source_content_type":"text/x-python","patch_set":33,"id":"2a9dfa4b_56081ab0","line":606,"range":{"start_line":606,"start_character":30,"end_line":606,"end_character":72},"updated":"2024-12-19 06:06:02.000000000","message":"We\u0027re passing a default -- when would it be explicitly `None`? Can we get rid of the `filter`/`lambda`?","commit_id":"3a6c587671fac06d0672dda23f7b81a5c2518c17"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"515b5169bc80ea4e28487ee6ed19649b3f845c8b","unresolved":false,"context_lines":[{"line_number":603,"context_line":"                list(filter("},{"line_number":604,"context_line":"                    lambda x: x is not None,"},{"line_number":605,"context_line":"                    ("},{"line_number":606,"context_line":"                        stats.get(\u0027failures_account_container_count\u0027, 0)"},{"line_number":607,"context_line":"                        for stats in device_stats.values()"},{"line_number":608,"context_line":"                    )"},{"line_number":609,"context_line":"                )) or [0],"}],"source_content_type":"text/x-python","patch_set":33,"id":"897ed12a_e8b2c2e2","line":606,"range":{"start_line":606,"start_character":30,"end_line":606,"end_character":72},"in_reply_to":"2a9dfa4b_56081ab0","updated":"2024-12-19 10:03:19.000000000","message":"Yes we can.","commit_id":"3a6c587671fac06d0672dda23f7b81a5c2518c17"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"57478226d0455af03700110d9a41dead04cf3928","unresolved":false,"context_lines":[{"line_number":603,"context_line":"                list(filter("},{"line_number":604,"context_line":"                    lambda x: x is not None,"},{"line_number":605,"context_line":"                    ("},{"line_number":606,"context_line":"                        stats.get(\u0027failures_account_container_count\u0027, 0)"},{"line_number":607,"context_line":"                        for stats in device_stats.values()"},{"line_number":608,"context_line":"                    )"},{"line_number":609,"context_line":"                )) or [0],"}],"source_content_type":"text/x-python","patch_set":33,"id":"bc1b0ff0_607e233e","line":606,"range":{"start_line":606,"start_character":30,"end_line":606,"end_character":72},"in_reply_to":"897ed12a_e8b2c2e2","updated":"2024-12-19 17:17:36.000000000","message":"\u003e Can we get rid of the filter/lambda?\n\ntests\u0027s suggest we can not:\n\n```\nswift/test/unit/obj/test_updater.py::TestObjectUpdater::test_aggregate_and_dump_recon_partial_device_updates - TypeError: \u0027\u003c\u0027 not supported between instances of \u0027NoneType\u0027 and \u0027NoneType\u0027\n```\n\nI prefer using `default\u003dNone` with `min` over `list() or [0]`\n\n```\ndiff --git a/swift/obj/updater.py b/swift/obj/updater.py\nindex 6ff5f9e89..9ae5432f6 100644\n--- a/swift/obj/updater.py\n+++ b/swift/obj/updater.py\n@@ -617,10 +617,10 @@ class ObjectUpdater(Daemon):\n             if device_stats\n             else 0,\n             \u0027failures_oldest_timestamp\u0027: min(\n-                list(filter(lambda x: x is not None,\n-                            [stats.get(\u0027failures_oldest_timestamp\u0027)\n-                             for stats in device_stats.values()]\n-                            )) or [None],\n+                filter(lambda x: x is not None, [\n+                    stats.get(\u0027failures_oldest_timestamp\u0027)\n+                    for stats in device_stats.values()]),\n+                 default\u003dNone\n             ),\n             \u0027failures_oldest_timestamp_age\u0027: max(\n                 list(filter(lambda x: x is not None,\n```","commit_id":"3a6c587671fac06d0672dda23f7b81a5c2518c17"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"cf99ebe849bf597ff2dc84eeb6930033a44e136d","unresolved":true,"context_lines":[{"line_number":609,"context_line":"                )) or [0],"},{"line_number":610,"context_line":"            ),"},{"line_number":611,"context_line":"            \u0027tracker_memory_usage\u0027: ("},{"line_number":612,"context_line":"                float(sum("},{"line_number":613,"context_line":"                    stats.get(\u0027tracker_memory_usage\u0027, 0)"},{"line_number":614,"context_line":"                    for stats in device_stats.values()"},{"line_number":615,"context_line":"                ))"},{"line_number":616,"context_line":"                / float(len(device_stats))"},{"line_number":617,"context_line":"            )"},{"line_number":618,"context_line":"            * max(self.updater_workers, 1)"},{"line_number":619,"context_line":"            if device_stats"},{"line_number":620,"context_line":"            else 0,"},{"line_number":621,"context_line":"            \u0027failures_oldest_timestamp\u0027: min("}],"source_content_type":"text/x-python","patch_set":33,"id":"6354338e_4db31043","line":618,"range":{"start_line":612,"start_character":16,"end_line":618,"end_character":42},"updated":"2024-12-19 06:06:02.000000000","message":"I wonder if this would be better as something like\n```\nmemory_usage_values \u003d sorted(\n    (stats.get(\u0027tracker_memory_usage\u0027, 0)\n     for stats in device_stats.values()),\n    reversed\u003dTrue)\naggregated_stats[\u0027tracker_memory_usage\u0027] \u003d sum(\n    memory_usage_values[:max(self.updater_workers, 1)])\n```\n\nI.e., capture the worst-case memory footprint, rather than average memory footprint.","commit_id":"3a6c587671fac06d0672dda23f7b81a5c2518c17"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"57478226d0455af03700110d9a41dead04cf3928","unresolved":false,"context_lines":[{"line_number":609,"context_line":"                )) or [0],"},{"line_number":610,"context_line":"            ),"},{"line_number":611,"context_line":"            \u0027tracker_memory_usage\u0027: ("},{"line_number":612,"context_line":"                float(sum("},{"line_number":613,"context_line":"                    stats.get(\u0027tracker_memory_usage\u0027, 0)"},{"line_number":614,"context_line":"                    for stats in device_stats.values()"},{"line_number":615,"context_line":"                ))"},{"line_number":616,"context_line":"                / float(len(device_stats))"},{"line_number":617,"context_line":"            )"},{"line_number":618,"context_line":"            * max(self.updater_workers, 1)"},{"line_number":619,"context_line":"            if device_stats"},{"line_number":620,"context_line":"            else 0,"},{"line_number":621,"context_line":"            \u0027failures_oldest_timestamp\u0027: min("}],"source_content_type":"text/x-python","patch_set":33,"id":"ba6aed9c_c930256e","line":618,"range":{"start_line":612,"start_character":16,"end_line":618,"end_character":42},"in_reply_to":"6354338e_4db31043","updated":"2024-12-19 17:17:36.000000000","message":"well, but we don\u0027t know if the two worst case ran at the same time... it\u0027s just an estimate regardless.  I\u0027m not sure it\u0027s worth a respin until we get some feedback from prod; but since master stats in `run_forever` is *broken* I think it\u0027s better if we merge this fix.\n\nHopefully we can get it settled down before the next upstream release.","commit_id":"3a6c587671fac06d0672dda23f7b81a5c2518c17"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"cf99ebe849bf597ff2dc84eeb6930033a44e136d","unresolved":true,"context_lines":[{"line_number":618,"context_line":"            * max(self.updater_workers, 1)"},{"line_number":619,"context_line":"            if device_stats"},{"line_number":620,"context_line":"            else 0,"},{"line_number":621,"context_line":"            \u0027failures_oldest_timestamp\u0027: min("},{"line_number":622,"context_line":"                list(filter(lambda x: x is not None,"},{"line_number":623,"context_line":"                            [stats.get(\u0027failures_oldest_timestamp\u0027)"},{"line_number":624,"context_line":"                             for stats in device_stats.values()]"}],"source_content_type":"text/x-python","patch_set":33,"id":"8e7d6d83_f418019f","line":621,"updated":"2024-12-19 06:06:02.000000000","message":"Should we also have an aggregated `failures_oldest_timestamp_age`?","commit_id":"3a6c587671fac06d0672dda23f7b81a5c2518c17"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"515b5169bc80ea4e28487ee6ed19649b3f845c8b","unresolved":false,"context_lines":[{"line_number":618,"context_line":"            * max(self.updater_workers, 1)"},{"line_number":619,"context_line":"            if device_stats"},{"line_number":620,"context_line":"            else 0,"},{"line_number":621,"context_line":"            \u0027failures_oldest_timestamp\u0027: min("},{"line_number":622,"context_line":"                list(filter(lambda x: x is not None,"},{"line_number":623,"context_line":"                            [stats.get(\u0027failures_oldest_timestamp\u0027)"},{"line_number":624,"context_line":"                             for stats in device_stats.values()]"}],"source_content_type":"text/x-python","patch_set":33,"id":"dbda1060_93588f7a","line":621,"in_reply_to":"8e7d6d83_f418019f","updated":"2024-12-19 10:03:19.000000000","message":"Good question. I think so. Added","commit_id":"3a6c587671fac06d0672dda23f7b81a5c2518c17"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"57478226d0455af03700110d9a41dead04cf3928","unresolved":false,"context_lines":[{"line_number":618,"context_line":"            * max(self.updater_workers, 1)"},{"line_number":619,"context_line":"            if device_stats"},{"line_number":620,"context_line":"            else 0,"},{"line_number":621,"context_line":"            \u0027failures_oldest_timestamp\u0027: min("},{"line_number":622,"context_line":"                list(filter(lambda x: x is not None,"},{"line_number":623,"context_line":"                            [stats.get(\u0027failures_oldest_timestamp\u0027)"},{"line_number":624,"context_line":"                             for stats in device_stats.values()]"}],"source_content_type":"text/x-python","patch_set":33,"id":"085c8a91_8f962c20","line":621,"in_reply_to":"dbda1060_93588f7a","updated":"2024-12-19 17:17:36.000000000","message":"```\n             \u0027failures_oldest_timestamp_age\u0027: max(\n-                list(filter(lambda x: x is not None,\n-                            [stats.get(\u0027failures_oldest_timestamp_age\u0027)\n-                             for stats in device_stats.values()]\n-                            )) or [None],\n+                filter(lambda x: x is not None,\n+                       [stats.get(\u0027failures_oldest_timestamp_age\u0027)\n+                        for stats in device_stats.values()]),\n+                default\u003dNone\n```","commit_id":"3a6c587671fac06d0672dda23f7b81a5c2518c17"}],"test/probe/test_object_async_update.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":432,"context_line":"        self.conf_dest \u003d os.path.join("},{"line_number":433,"context_line":"            \u0027/tmp/\u0027,"},{"line_number":434,"context_line":"            datetime.now().strftime(\u0027swift-%Y-%m-%d_%H-%M-%S-%f\u0027)"},{"line_number":435,"context_line":"        )"},{"line_number":436,"context_line":"        os.mkdir(self.conf_dest)"},{"line_number":437,"context_line":"        object_server_dir \u003d os.path.join(self.conf_dest, \u0027object-server\u0027)"},{"line_number":438,"context_line":"        os.mkdir(object_server_dir)"}],"source_content_type":"text/x-python","patch_set":4,"id":"2a57f2f2_adb834e0","line":435,"updated":"2024-12-06 23:34:59.000000000","message":"probably better with stdlib directory name:\n\nhttps://docs.python.org/3/library/tempfile.html#tempfile.mkdtemp\n\nagain this pattern is copied from `test.probe.test_dark_data`","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"580c03e576fd53b7a2e76772abf50e03124d7271","unresolved":false,"context_lines":[{"line_number":432,"context_line":"        self.conf_dest \u003d os.path.join("},{"line_number":433,"context_line":"            \u0027/tmp/\u0027,"},{"line_number":434,"context_line":"            datetime.now().strftime(\u0027swift-%Y-%m-%d_%H-%M-%S-%f\u0027)"},{"line_number":435,"context_line":"        )"},{"line_number":436,"context_line":"        os.mkdir(self.conf_dest)"},{"line_number":437,"context_line":"        object_server_dir \u003d os.path.join(self.conf_dest, \u0027object-server\u0027)"},{"line_number":438,"context_line":"        os.mkdir(object_server_dir)"}],"source_content_type":"text/x-python","patch_set":4,"id":"a993c06e_c9293dbb","line":435,"in_reply_to":"2a57f2f2_adb834e0","updated":"2024-12-12 19:53:26.000000000","message":"Acknowledged","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":440,"context_line":"            config \u003d readconf(conf_file)"},{"line_number":441,"context_line":"            if CONF_SECTION not in config:"},{"line_number":442,"context_line":"                continue  # Ensure the object-updater is set up to run"},{"line_number":443,"context_line":"            config[CONF_SECTION].update({\u0027interval\u0027: \u00271\u0027})"},{"line_number":444,"context_line":""},{"line_number":445,"context_line":"            parser \u003d ConfigParser()"},{"line_number":446,"context_line":"            parser.add_section(CONF_SECTION)"}],"source_content_type":"text/x-python","patch_set":4,"id":"cdc4f47a_8d0d16bd","line":443,"updated":"2024-12-06 23:34:59.000000000","message":"is this forced update the main (only?) reason we\u0027re rewriting configs?\n\nI can understand why the default of 300s makes it annoying to use `run_forever` in a test.\n\nIf we could rely on a consistent base config with conf.d we could \"just\" add the override in a new file that could be cleaned up in tearDown.","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"580c03e576fd53b7a2e76772abf50e03124d7271","unresolved":false,"context_lines":[{"line_number":440,"context_line":"            config \u003d readconf(conf_file)"},{"line_number":441,"context_line":"            if CONF_SECTION not in config:"},{"line_number":442,"context_line":"                continue  # Ensure the object-updater is set up to run"},{"line_number":443,"context_line":"            config[CONF_SECTION].update({\u0027interval\u0027: \u00271\u0027})"},{"line_number":444,"context_line":""},{"line_number":445,"context_line":"            parser \u003d ConfigParser()"},{"line_number":446,"context_line":"            parser.add_section(CONF_SECTION)"}],"source_content_type":"text/x-python","patch_set":4,"id":"13481df9_1e04a560","line":443,"in_reply_to":"cdc4f47a_8d0d16bd","updated":"2024-12-12 19:53:26.000000000","message":"Yes, that is the only reason. Like you mentioned this borrows the logic from test.probe.test_dark_data","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":449,"context_line":""},{"line_number":450,"context_line":"            file_name \u003d os.path.basename(conf_file)"},{"line_number":451,"context_line":"            if file_name.endswith(\u0027.d\u0027):"},{"line_number":452,"context_line":"                # Work around conf.d setups (like you might see with VSAIO)"},{"line_number":453,"context_line":"                file_name \u003d file_name[:-2]"},{"line_number":454,"context_line":"            with open(os.path.join(object_server_dir, file_name), \u0027w\u0027) as fp:"},{"line_number":455,"context_line":"                parser.write(fp)"}],"source_content_type":"text/x-python","patch_set":4,"id":"8b59e92e_0f7c5364","line":452,"updated":"2024-12-06 23:34:59.000000000","message":"gross\n\n`test.probe.test_dark_data` does this too...\n\nprobably better when we discover `conf.d` we take advantage and \"just\" drop an anemic `deleteme_test_file_override.conf` and only \"work around gate NOT having conf.d using the monkey patch SWIFT_DIR hack\"","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"580c03e576fd53b7a2e76772abf50e03124d7271","unresolved":false,"context_lines":[{"line_number":449,"context_line":""},{"line_number":450,"context_line":"            file_name \u003d os.path.basename(conf_file)"},{"line_number":451,"context_line":"            if file_name.endswith(\u0027.d\u0027):"},{"line_number":452,"context_line":"                # Work around conf.d setups (like you might see with VSAIO)"},{"line_number":453,"context_line":"                file_name \u003d file_name[:-2]"},{"line_number":454,"context_line":"            with open(os.path.join(object_server_dir, file_name), \u0027w\u0027) as fp:"},{"line_number":455,"context_line":"                parser.write(fp)"}],"source_content_type":"text/x-python","patch_set":4,"id":"bcf14bf5_20f6e72b","line":452,"in_reply_to":"8b59e92e_0f7c5364","updated":"2024-12-12 19:53:26.000000000","message":"Acknowledged","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":482,"context_line":"        for acct, cont in ac_pairs:"},{"line_number":483,"context_line":"            for o in range(num_objs_per_ac):"},{"line_number":484,"context_line":"                obj \u003d \u0027obj%03d\u0027 % o"},{"line_number":485,"context_line":"                self.int_client.upload_object(BytesIO(b\u0027\u0027), acct, cont, obj)"},{"line_number":486,"context_line":""},{"line_number":487,"context_line":"        all_asyncs \u003d self.gather_async_pendings()"},{"line_number":488,"context_line":"        # Between 1-2 asyncs per object"}],"source_content_type":"text/x-python","patch_set":4,"id":"9fad0b0a_e0ce326c","line":485,"updated":"2024-12-06 23:34:59.000000000","message":"it\u0027s interesting; I noticed that NONE of the AUTH_user002 asyncs got reported in my \"top5\" - I think because they were created more recently than the AUTH_user000 and AUTH_user001 asyncs?\n\nIf we could make something along those lines reliable enough - maybe that\u0027s a useful behavior we could assert on!?  i.e. \"the oldest async tracker *actually* reports *the oldest*!!!\"","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"11c8c3da7063aac03db0234be28fdbd62efa4cb6","unresolved":false,"context_lines":[{"line_number":482,"context_line":"        for acct, cont in ac_pairs:"},{"line_number":483,"context_line":"            for o in range(num_objs_per_ac):"},{"line_number":484,"context_line":"                obj \u003d \u0027obj%03d\u0027 % o"},{"line_number":485,"context_line":"                self.int_client.upload_object(BytesIO(b\u0027\u0027), acct, cont, obj)"},{"line_number":486,"context_line":""},{"line_number":487,"context_line":"        all_asyncs \u003d self.gather_async_pendings()"},{"line_number":488,"context_line":"        # Between 1-2 asyncs per object"}],"source_content_type":"text/x-python","patch_set":4,"id":"787ddf32_f4a9ace2","line":485,"in_reply_to":"5b0dee75_d8b90319","updated":"2024-12-13 20:29:29.000000000","message":"Yes, thank you!","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":482,"context_line":"        for acct, cont in ac_pairs:"},{"line_number":483,"context_line":"            for o in range(num_objs_per_ac):"},{"line_number":484,"context_line":"                obj \u003d \u0027obj%03d\u0027 % o"},{"line_number":485,"context_line":"                self.int_client.upload_object(BytesIO(b\u0027\u0027), acct, cont, obj)"},{"line_number":486,"context_line":""},{"line_number":487,"context_line":"        all_asyncs \u003d self.gather_async_pendings()"},{"line_number":488,"context_line":"        # Between 1-2 asyncs per object"}],"source_content_type":"text/x-python","patch_set":4,"id":"5b0dee75_d8b90319","line":485,"in_reply_to":"9fad0b0a_e0ce326c","updated":"2024-12-13 00:46:25.000000000","message":"937653: sq? maybe better assertions/comments | https://review.opendev.org/c/openstack/swift/+/937653","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":492,"context_line":""},{"line_number":493,"context_line":"        # Run the updater and check stats"},{"line_number":494,"context_line":"        old_swift_dir \u003d manager.SWIFT_DIR"},{"line_number":495,"context_line":"        manager.SWIFT_DIR \u003d self.conf_dest"},{"line_number":496,"context_line":"        try:"},{"line_number":497,"context_line":"            self.object_updater.start()"},{"line_number":498,"context_line":"            updater_status \u003d self.object_updater.status()"}],"source_content_type":"text/x-python","patch_set":4,"id":"5430ff30_0a45df3d","line":495,"updated":"2024-12-06 23:34:59.000000000","message":"yikes, maybe consider mock.patch_object instead of try/finally to make it more obvious kind of mutation is generally a bad idea.\n\nI discovered this idea was borrowed from `test.probe.test_dark_data`\n\nThe manager class already parameterized `run_dir` as a first class overridable object attribute; probably we should allow the parameterization of the Manager class with a `self.conf_dir` that merely DEFAULTS to the SWIFT_DIR constant so you can easily:\n\n    self.custom_object_updater \u003d Manager([\u0027object-updater\u0027], conf_dir\u003d\u0027/my/root\u0027)\n    \nAs much as I would prefer it - the `Manager` refactor shouldn\u0027t happen as a drive-by in this change.  Begrudgingly the most practical path forward is probably to borrow the `test.probe.test_dark_data` pattern verbatim as this patch does and write a bug for the Manager saying \"it\u0027s making life hard for people that just want to start a process from an alternative config\" (which should be easy/supported).  If we have the bug we could link to these probetests as an example and clean them up when we introduce the `conf_dir` kwarg support.","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"580c03e576fd53b7a2e76772abf50e03124d7271","unresolved":false,"context_lines":[{"line_number":492,"context_line":""},{"line_number":493,"context_line":"        # Run the updater and check stats"},{"line_number":494,"context_line":"        old_swift_dir \u003d manager.SWIFT_DIR"},{"line_number":495,"context_line":"        manager.SWIFT_DIR \u003d self.conf_dest"},{"line_number":496,"context_line":"        try:"},{"line_number":497,"context_line":"            self.object_updater.start()"},{"line_number":498,"context_line":"            updater_status \u003d self.object_updater.status()"}],"source_content_type":"text/x-python","patch_set":4,"id":"68c86e5e_02523596","line":495,"in_reply_to":"5430ff30_0a45df3d","updated":"2024-12-12 19:53:26.000000000","message":"Made a bug and started working on this patch","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":494,"context_line":"        old_swift_dir \u003d manager.SWIFT_DIR"},{"line_number":495,"context_line":"        manager.SWIFT_DIR \u003d self.conf_dest"},{"line_number":496,"context_line":"        try:"},{"line_number":497,"context_line":"            self.object_updater.start()"},{"line_number":498,"context_line":"            updater_status \u003d self.object_updater.status()"},{"line_number":499,"context_line":"            self.assertEqual(updater_status, 0,"},{"line_number":500,"context_line":"                             \"Object updater failed to start\")"}],"source_content_type":"text/x-python","patch_set":4,"id":"b5d179b1_126e208c","line":497,"updated":"2024-12-06 23:34:59.000000000","message":"how are these things even finding container.ring.gz?\n\n```\nvagrant@saio:~$ find /tmp/swift-2024-12-06_22-13-11-664844\n/tmp/swift-2024-12-06_22-13-11-664844\n/tmp/swift-2024-12-06_22-13-11-664844/object-server\n/tmp/swift-2024-12-06_22-13-11-664844/object-server/4.conf\n/tmp/swift-2024-12-06_22-13-11-664844/object-server/1.conf\n/tmp/swift-2024-12-06_22-13-11-664844/object-server/2.conf\n/tmp/swift-2024-12-06_22-13-11-664844/object-server/3.conf\n```\n\noic, this only effects the call to the Manager (duh?) when building the config path to pass to the process when invoking subprocess.Popen - NOT e.g. `swift.common.utils.SWIFT_CONF_FILE` or even `self.swift_dir \u003d conf.get(\u0027swift_dir\u0027, \u0027/etc/swift\u0027)` when the updater actually starts!","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"580c03e576fd53b7a2e76772abf50e03124d7271","unresolved":false,"context_lines":[{"line_number":494,"context_line":"        old_swift_dir \u003d manager.SWIFT_DIR"},{"line_number":495,"context_line":"        manager.SWIFT_DIR \u003d self.conf_dest"},{"line_number":496,"context_line":"        try:"},{"line_number":497,"context_line":"            self.object_updater.start()"},{"line_number":498,"context_line":"            updater_status \u003d self.object_updater.status()"},{"line_number":499,"context_line":"            self.assertEqual(updater_status, 0,"},{"line_number":500,"context_line":"                             \"Object updater failed to start\")"}],"source_content_type":"text/x-python","patch_set":4,"id":"321de4a0_eccf20fe","line":497,"in_reply_to":"b5d179b1_126e208c","updated":"2024-12-12 19:53:26.000000000","message":"Acknowledged","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":500,"context_line":"                             \"Object updater failed to start\")"},{"line_number":501,"context_line":""},{"line_number":502,"context_line":"            # Wait for updates to process"},{"line_number":503,"context_line":"            time.sleep(20)"},{"line_number":504,"context_line":"        finally:"},{"line_number":505,"context_line":"            manager.SWIFT_DIR \u003d old_swift_dir"},{"line_number":506,"context_line":"        recons \u003d []"}],"source_content_type":"text/x-python","patch_set":4,"id":"98a8f7c0_7023b841","line":503,"updated":"2024-12-06 23:34:59.000000000","message":"this is a huge smell, we can\u0027t be sure that 20s is enough or too much\n\nWe should start the updaters and then poll recon up-to-a-timeout until we get the cycle stats we expect and break as soon as we do and shut them down.  20s might be a reasonable timeout for polling; but if we invest in the code to make sure we stop as soon they\u0027re done we could even start with a 360s timeout and only expect the test to \"hang\" if there\u0027s an actual problem:\n\n    self.fail(\"didn\u0027t get all device recon even after 6m!?\")","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"580c03e576fd53b7a2e76772abf50e03124d7271","unresolved":false,"context_lines":[{"line_number":500,"context_line":"                             \"Object updater failed to start\")"},{"line_number":501,"context_line":""},{"line_number":502,"context_line":"            # Wait for updates to process"},{"line_number":503,"context_line":"            time.sleep(20)"},{"line_number":504,"context_line":"        finally:"},{"line_number":505,"context_line":"            manager.SWIFT_DIR \u003d old_swift_dir"},{"line_number":506,"context_line":"        recons \u003d []"}],"source_content_type":"text/x-python","patch_set":4,"id":"20cb074b_8f12f582","line":503,"in_reply_to":"98a8f7c0_7023b841","updated":"2024-12-12 19:53:26.000000000","message":"Added polling","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":536,"context_line":"                account \u003d entry[\u0027account\u0027]"},{"line_number":537,"context_line":"                container \u003d entry[\u0027container\u0027]"},{"line_number":538,"context_line":"                timestamp \u003d entry[\u0027timestamp\u0027]"},{"line_number":539,"context_line":"                self.assertIsNotNone(timestamp)"},{"line_number":540,"context_line":"                ac_set.add((account, container))"},{"line_number":541,"context_line":""},{"line_number":542,"context_line":"        for ac in ac_set:"}],"source_content_type":"text/x-python","patch_set":4,"id":"fb02725e_f4c7e1fe","line":539,"updated":"2024-12-06 23:34:59.000000000","message":"these \"assertions\" about the format of the \"oldest_entries\" elements are probably the most useful sanity check that we\u0027re getting something useful out the aggregated recon drops.\n\nThe fact that each of these replicated nodes only finds asyncs on a single device means we\u0027re not getting a lot of assertion about the actual aggregation logic - this is maybe where using a EC policy might help; but for \"functionally verify we\u0027re at least getting *some kind of* data\" - this might be good enough; i.e.\n\nit fails as expected with the change reverted:\n\n```\n            oldest_count \u003d updater_stats[\n                \u0027failures_oldest_timestamp_account_containers\u0027\n            ][\u0027oldest_count\u0027]\n\u003e           self.assertEqual(oldest_count, 5)\nE           AssertionError: 0 !\u003d 5\n\nvagrant@saio:~$ cat /var/cache/swift/node1/object.recon | jq .\n{\n  \"object_updater_stats\": {\n    \"failures_account_container_count\": 0,\n    \"failures_oldest_timestamp\": null,\n    \"failures_oldest_timestamp_account_containers\": {\n      \"oldest_count\": 0,\n      \"oldest_entries\": []\n    },\n    \"failures_oldest_timestamp_age\": null,\n    \"tracker_memory_usage\": 487\n  },\n  \"object_updater_sweep\": 0.9876916408538818\n}\n```","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"11c8c3da7063aac03db0234be28fdbd62efa4cb6","unresolved":false,"context_lines":[{"line_number":536,"context_line":"                account \u003d entry[\u0027account\u0027]"},{"line_number":537,"context_line":"                container \u003d entry[\u0027container\u0027]"},{"line_number":538,"context_line":"                timestamp \u003d entry[\u0027timestamp\u0027]"},{"line_number":539,"context_line":"                self.assertIsNotNone(timestamp)"},{"line_number":540,"context_line":"                ac_set.add((account, container))"},{"line_number":541,"context_line":""},{"line_number":542,"context_line":"        for ac in ac_set:"}],"source_content_type":"text/x-python","patch_set":4,"id":"05ce1f4c_2e94d552","line":539,"in_reply_to":"fb02725e_f4c7e1fe","updated":"2024-12-13 20:29:29.000000000","message":"Acknowledged","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":false,"context_lines":[{"line_number":536,"context_line":"                account \u003d entry[\u0027account\u0027]"},{"line_number":537,"context_line":"                container \u003d entry[\u0027container\u0027]"},{"line_number":538,"context_line":"                timestamp \u003d entry[\u0027timestamp\u0027]"},{"line_number":539,"context_line":"                self.assertIsNotNone(timestamp)"},{"line_number":540,"context_line":"                ac_set.add((account, container))"},{"line_number":541,"context_line":""},{"line_number":542,"context_line":"        for ac in ac_set:"}],"source_content_type":"text/x-python","patch_set":4,"id":"8b97572b_cfb29773","line":539,"in_reply_to":"fb02725e_f4c7e1fe","updated":"2024-12-13 00:46:25.000000000","message":"Acknowledged","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8ec7e18b0bbebc1a2d3ca2b15afdae2e48d9d30d","unresolved":true,"context_lines":[{"line_number":543,"context_line":"            self.assertIn(ac, set(ac_pairs))"},{"line_number":544,"context_line":""},{"line_number":545,"context_line":"        for found_count in found_counts:"},{"line_number":546,"context_line":"            self.assertLessEqual(found_count, len(ac_pairs))"},{"line_number":547,"context_line":""},{"line_number":548,"context_line":""},{"line_number":549,"context_line":"if __name__ \u003d\u003d \u0027__main__\u0027:"}],"source_content_type":"text/x-python","patch_set":4,"id":"f5947e16_847bfbd4","line":546,"updated":"2024-12-06 23:34:59.000000000","message":"I don\u0027t really know what this assertion is buying us just asserting the *upper* bound:\n\n```\n(Pdb) !found_counts\n[9, 12, 11, 11, 9, 12, 11, 11]\n(Pdb) l\n542             for ac in ac_set:\n543                 self.assertIn(ac, set(ac_pairs))\n544  \n545             import pdb\n546             pdb.set_trace()\n547  -\u003e         for found_count in found_counts:\n548                 self.assertLessEqual(found_count, len(ac_pairs))\n549  \n550  \n551     if __name__ \u003d\u003d \u0027__main__\u0027:\n552         main()\n(Pdb) ac_pairs\n[(\u0027AUTH_user000\u0027, \u0027cont000\u0027), (\u0027AUTH_user000\u0027, \u0027cont001\u0027), (\u0027AUTH_user000\u0027, \u0027cont002\u0027), (\u0027AUTH_user000\u0027, \u0027cont003\u0027), (\u0027AUTH_user001\u0027, \u0027cont000\u0027), (\u0027AUTH_user001\u0027, \u0027cont001\u0027), (\u0027AUTH_user001\u0027, \u0027cont002\u0027), (\u0027AUTH_user001\u0027, \u0027cont003\u0027), (\u0027AUTH_user002\u0027, \u0027cont000\u0027), (\u0027AUTH_user002\u0027, \u0027cont001\u0027), (\u0027AUTH_user002\u0027, \u0027cont002\u0027), (\u0027AUTH_user002\u0027, \u0027cont003\u0027)]\n(Pdb) len(ac_pairs)\n12\n```\n\nLike obviously we didn\u0027t find MORE a/c than we created - we just resetswift.  But can we place a lower bound - or it is totally subject to the random placement of the failed container updates such that some nodes might not have *any* async?\n\nThe duplication of recon data per device (8), even tho the reported stats are aggregated per node (4) is really annoying:\n\nhttps://bugs.launchpad.net/swift/+bug/1612168","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":false,"context_lines":[{"line_number":543,"context_line":"            self.assertIn(ac, set(ac_pairs))"},{"line_number":544,"context_line":""},{"line_number":545,"context_line":"        for found_count in found_counts:"},{"line_number":546,"context_line":"            self.assertLessEqual(found_count, len(ac_pairs))"},{"line_number":547,"context_line":""},{"line_number":548,"context_line":""},{"line_number":549,"context_line":"if __name__ \u003d\u003d \u0027__main__\u0027:"}],"source_content_type":"text/x-python","patch_set":4,"id":"674ab872_338fac6e","line":546,"in_reply_to":"f5947e16_847bfbd4","updated":"2024-12-13 00:46:25.000000000","message":"Acknowledged","commit_id":"ef0b4218297cac5a4a3893a615f437664efafc69"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":496,"context_line":"        try:"},{"line_number":497,"context_line":"            self.object_updater.start()"},{"line_number":498,"context_line":"        finally:"},{"line_number":499,"context_line":"            manager.SWIFT_DIR \u003d old_swift_dir"},{"line_number":500,"context_line":"        updater_status \u003d self.object_updater.status()"},{"line_number":501,"context_line":"        self.assertEqual("},{"line_number":502,"context_line":"            updater_status, 0, \"Object updater failed to start\")"}],"source_content_type":"text/x-python","patch_set":15,"id":"d0746e62_39613888","line":499,"updated":"2024-12-13 00:46:25.000000000","message":"these six lines probably could be:\n\n```\nwith mock.patch.object(manager, \u0027SWIFT_DIR\u0027, self.conf_dest):\n    self.object_updater.start()\n```","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"11c8c3da7063aac03db0234be28fdbd62efa4cb6","unresolved":false,"context_lines":[{"line_number":496,"context_line":"        try:"},{"line_number":497,"context_line":"            self.object_updater.start()"},{"line_number":498,"context_line":"        finally:"},{"line_number":499,"context_line":"            manager.SWIFT_DIR \u003d old_swift_dir"},{"line_number":500,"context_line":"        updater_status \u003d self.object_updater.status()"},{"line_number":501,"context_line":"        self.assertEqual("},{"line_number":502,"context_line":"            updater_status, 0, \"Object updater failed to start\")"}],"source_content_type":"text/x-python","patch_set":15,"id":"d4f0d8ea_92797f6d","line":499,"in_reply_to":"d0746e62_39613888","updated":"2024-12-13 20:29:29.000000000","message":"Done","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":501,"context_line":"        self.assertEqual("},{"line_number":502,"context_line":"            updater_status, 0, \"Object updater failed to start\")"},{"line_number":503,"context_line":""},{"line_number":504,"context_line":"        dev_to_node_dict \u003d {}"},{"line_number":505,"context_line":"        for onode in self.object_ring.devs:"},{"line_number":506,"context_line":"            if any(self.is_local_to(node, onode)"},{"line_number":507,"context_line":"                    for node in dev_to_node_dict.values()):"}],"source_content_type":"text/x-python","patch_set":15,"id":"a29b47fa_ef3ffb48","line":504,"updated":"2024-12-13 00:46:25.000000000","message":"probably worth a comment:\n\n```\n# pick out a device dict from each node to pull recon from\n```","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"11c8c3da7063aac03db0234be28fdbd62efa4cb6","unresolved":false,"context_lines":[{"line_number":501,"context_line":"        self.assertEqual("},{"line_number":502,"context_line":"            updater_status, 0, \"Object updater failed to start\")"},{"line_number":503,"context_line":""},{"line_number":504,"context_line":"        dev_to_node_dict \u003d {}"},{"line_number":505,"context_line":"        for onode in self.object_ring.devs:"},{"line_number":506,"context_line":"            if any(self.is_local_to(node, onode)"},{"line_number":507,"context_line":"                    for node in dev_to_node_dict.values()):"}],"source_content_type":"text/x-python","patch_set":15,"id":"c8ea1da3_61954b86","line":504,"in_reply_to":"a29b47fa_ef3ffb48","updated":"2024-12-13 20:29:29.000000000","message":"Done","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"43ef5511af7279dd651dd9d59c86c37db79c1ef2","unresolved":true,"context_lines":[{"line_number":572,"context_line":"            self.assertIn(ac, ac_set)"},{"line_number":573,"context_line":"        # Where as the more recent failures are NOT!"},{"line_number":574,"context_line":"        for ac in ac_pairs[-3:]:"},{"line_number":575,"context_line":"            self.assertNotIn(ac, ac_set)"},{"line_number":576,"context_line":""},{"line_number":577,"context_line":""},{"line_number":578,"context_line":"if __name__ \u003d\u003d \u0027__main__\u0027:"}],"source_content_type":"text/x-python","patch_set":26,"id":"296b91cc_731a422f","line":575,"updated":"2024-12-18 17:03:22.000000000","message":"I\u0027m guessing some of these assertions would also work in the `test_run_once` case\n\n938007: tests: dry out updater stats probe | https://review.opendev.org/c/openstack/swift/+/938007","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"26728f09eafbb5ba626e079eee682cf220f1a367","unresolved":false,"context_lines":[{"line_number":572,"context_line":"            self.assertIn(ac, ac_set)"},{"line_number":573,"context_line":"        # Where as the more recent failures are NOT!"},{"line_number":574,"context_line":"        for ac in ac_pairs[-3:]:"},{"line_number":575,"context_line":"            self.assertNotIn(ac, ac_set)"},{"line_number":576,"context_line":""},{"line_number":577,"context_line":""},{"line_number":578,"context_line":"if __name__ \u003d\u003d \u0027__main__\u0027:"}],"source_content_type":"text/x-python","patch_set":26,"id":"d0008c2e_1403204d","line":575,"in_reply_to":"296b91cc_731a422f","updated":"2024-12-18 17:51:35.000000000","message":"Thank you 🙏","commit_id":"5cceff6de0b60b0e5468fdc26ce026f15980fc29"}],"test/unit/obj/test_server.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"2d52cdf17f58515b3c5eb7854adb59508a2401e7","unresolved":false,"context_lines":[{"line_number":1207,"context_line":"                mock_ring \u003d mock.MagicMock()"},{"line_number":1208,"context_line":"                mock_ring.get_nodes.return_value \u003d (99, [node])"},{"line_number":1209,"context_line":"                object_updater.container_ring \u003d mock_ring"},{"line_number":1210,"context_line":"                object_updater._process_device_in_child(self.sda1, \u0027sda1\u0027)"},{"line_number":1211,"context_line":""},{"line_number":1212,"context_line":"        self.assertEqual(1, len(conn.requests))"},{"line_number":1213,"context_line":"        self.assertEqual(\u0027/cdevice/99/.sharded_a/c_shard_1/o\u0027,"}],"source_content_type":"text/x-python","patch_set":20,"id":"b4779b85_1917f6a1","line":1210,"updated":"2024-12-17 20:09:15.000000000","message":"super surprised to find calls to the object-updater in test-server","commit_id":"216b2bd30ba71cbb05d883d2a92e0f5e788c75de"}],"test/unit/obj/test_updater.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":462,"context_line":"        odd_dir \u003d os.path.join(async_dir, \u0027not really supposed \u0027"},{"line_number":463,"context_line":"                               \u0027to be here\u0027)"},{"line_number":464,"context_line":"        os.mkdir(odd_dir)"},{"line_number":465,"context_line":"        ou.run_once()"},{"line_number":466,"context_line":"        self.assertTrue(os.path.exists(async_dir))"},{"line_number":467,"context_line":"        self.assertTrue(os.path.exists(odd_dir))  # skipped - not mounted!"},{"line_number":468,"context_line":"        self.assertEqual(["}],"source_content_type":"text/x-python","patch_set":15,"id":"63736f5a_06917a0b","side":"PARENT","line":465,"updated":"2024-12-13 00:46:25.000000000","message":"so it\u0027s now kind of a smell to me when a test calls `run_once` w/o mocking out `fork`\n\n... but it turns out this test never actually spawns and children because the mock_check_drive always raises ValueError to simulate \"unmounted\"\n\n937660: sq? some more unittests | https://review.opendev.org/c/openstack/swift/+/937660","commit_id":"fe7928ea8aa45b15fde3154cd8ccdfabc43053a0"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"11c8c3da7063aac03db0234be28fdbd62efa4cb6","unresolved":false,"context_lines":[{"line_number":462,"context_line":"        odd_dir \u003d os.path.join(async_dir, \u0027not really supposed \u0027"},{"line_number":463,"context_line":"                               \u0027to be here\u0027)"},{"line_number":464,"context_line":"        os.mkdir(odd_dir)"},{"line_number":465,"context_line":"        ou.run_once()"},{"line_number":466,"context_line":"        self.assertTrue(os.path.exists(async_dir))"},{"line_number":467,"context_line":"        self.assertTrue(os.path.exists(odd_dir))  # skipped - not mounted!"},{"line_number":468,"context_line":"        self.assertEqual(["}],"source_content_type":"text/x-python","patch_set":15,"id":"acfa5a3b_99727c8e","side":"PARENT","line":465,"in_reply_to":"63736f5a_06917a0b","updated":"2024-12-13 20:29:29.000000000","message":"Squashed","commit_id":"fe7928ea8aa45b15fde3154cd8ccdfabc43053a0"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":473,"context_line":"    @mock.patch(\u0027swift.obj.updater.dump_recon_cache\u0027)"},{"line_number":474,"context_line":"    @mock.patch.object(object_updater, \u0027check_drive\u0027)"},{"line_number":475,"context_line":"    def test_run_once(self, mock_check_drive, mock_dump_recon):"},{"line_number":476,"context_line":"        mock_check_drive.side_effect \u003d lambda r, d, mc: os.path.join(r, d)"},{"line_number":477,"context_line":"        ou \u003d object_updater.ObjectUpdater({"},{"line_number":478,"context_line":"            \u0027devices\u0027: self.devices_dir,"},{"line_number":479,"context_line":"            \u0027mount_check\u0027: \u0027false\u0027,"}],"source_content_type":"text/x-python","patch_set":15,"id":"c0734482_256ec06f","side":"PARENT","line":476,"updated":"2024-12-13 00:46:25.000000000","message":"right, so \"root, device, mount_check\" args return `os.path.join(root, device)`","commit_id":"fe7928ea8aa45b15fde3154cd8ccdfabc43053a0"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"11c8c3da7063aac03db0234be28fdbd62efa4cb6","unresolved":false,"context_lines":[{"line_number":473,"context_line":"    @mock.patch(\u0027swift.obj.updater.dump_recon_cache\u0027)"},{"line_number":474,"context_line":"    @mock.patch.object(object_updater, \u0027check_drive\u0027)"},{"line_number":475,"context_line":"    def test_run_once(self, mock_check_drive, mock_dump_recon):"},{"line_number":476,"context_line":"        mock_check_drive.side_effect \u003d lambda r, d, mc: os.path.join(r, d)"},{"line_number":477,"context_line":"        ou \u003d object_updater.ObjectUpdater({"},{"line_number":478,"context_line":"            \u0027devices\u0027: self.devices_dir,"},{"line_number":479,"context_line":"            \u0027mount_check\u0027: \u0027false\u0027,"}],"source_content_type":"text/x-python","patch_set":15,"id":"3d8ab9d1_ab0a34c5","side":"PARENT","line":476,"in_reply_to":"c0734482_256ec06f","updated":"2024-12-13 20:29:29.000000000","message":"Acknowledged","commit_id":"fe7928ea8aa45b15fde3154cd8ccdfabc43053a0"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":492,"context_line":"            mock.call(self.devices_dir, \u0027sda1\u0027, False),"},{"line_number":493,"context_line":"            mock.call(self.devices_dir, \u0027sda1\u0027, False),"},{"line_number":494,"context_line":"        ], mock_check_drive.mock_calls)"},{"line_number":495,"context_line":"        mock_check_drive.reset_mock()"},{"line_number":496,"context_line":"        self.assertEqual([], ou.logger.get_lines_for_level(\u0027error\u0027))"},{"line_number":497,"context_line":""},{"line_number":498,"context_line":"        ou \u003d object_updater.ObjectUpdater({"}],"source_content_type":"text/x-python","patch_set":15,"id":"a7a57dae_b161ac2d","side":"PARENT","line":495,"updated":"2024-12-13 00:46:25.000000000","message":"I think in this test specifically it would be nice to somehow see that the value we pass into `_process_device_in_child` is same as what `ou.run_once` would have done by calling `mount_check`\n\nProbably by calling the unmocked mount_check right before `_process_device_in_child`\n\n937659: sq? maybe don\u0027t use self.dev_path as alias for self.sda1 | https://review.opendev.org/c/openstack/swift/+/937659","commit_id":"fe7928ea8aa45b15fde3154cd8ccdfabc43053a0"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":492,"context_line":"            mock.call(self.devices_dir, \u0027sda1\u0027, False),"},{"line_number":493,"context_line":"            mock.call(self.devices_dir, \u0027sda1\u0027, False),"},{"line_number":494,"context_line":"        ], mock_check_drive.mock_calls)"},{"line_number":495,"context_line":"        mock_check_drive.reset_mock()"},{"line_number":496,"context_line":"        self.assertEqual([], ou.logger.get_lines_for_level(\u0027error\u0027))"},{"line_number":497,"context_line":""},{"line_number":498,"context_line":"        ou \u003d object_updater.ObjectUpdater({"}],"source_content_type":"text/x-python","patch_set":15,"id":"f968cec8_5f40e5dd","side":"PARENT","line":495,"in_reply_to":"a7a57dae_b161ac2d","updated":"2024-12-17 16:25:55.000000000","message":"Squashed and removed self.dev_path alias","commit_id":"fe7928ea8aa45b15fde3154cd8ccdfabc43053a0"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":497,"context_line":""},{"line_number":498,"context_line":"        ou \u003d object_updater.ObjectUpdater({"},{"line_number":499,"context_line":"            \u0027devices\u0027: self.devices_dir,"},{"line_number":500,"context_line":"            \u0027mount_check\u0027: \u0027TrUe\u0027,"},{"line_number":501,"context_line":"            \u0027swift_dir\u0027: self.testdir,"},{"line_number":502,"context_line":"            \u0027interval\u0027: \u00271\u0027,"},{"line_number":503,"context_line":"            \u0027concurrency\u0027: \u00271\u0027,"}],"source_content_type":"text/x-python","patch_set":15,"id":"77bd41ba_e1f27241","side":"PARENT","line":500,"updated":"2024-12-13 00:46:25.000000000","message":"I think this typo was actually trying to say something about the behavior of the `mount_check` config option handling.\n\nSince we can\u0027t *actually* call the *real* `check_drive` function with `mount_check\u003dTrue` on a path that\u0027s in a mounted device if we wanted to validate that \"run_once will call mount_check based on it\u0027s config\" we HAVE to use a mocked `check_drive`.\n\n\nHowever, since this test is now just calling `_process_device_in_child` we\u0027re not actually calling `check_drive` anymore so it\u0027s almost a distraction to have it in the config.","commit_id":"fe7928ea8aa45b15fde3154cd8ccdfabc43053a0"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":497,"context_line":""},{"line_number":498,"context_line":"        ou \u003d object_updater.ObjectUpdater({"},{"line_number":499,"context_line":"            \u0027devices\u0027: self.devices_dir,"},{"line_number":500,"context_line":"            \u0027mount_check\u0027: \u0027TrUe\u0027,"},{"line_number":501,"context_line":"            \u0027swift_dir\u0027: self.testdir,"},{"line_number":502,"context_line":"            \u0027interval\u0027: \u00271\u0027,"},{"line_number":503,"context_line":"            \u0027concurrency\u0027: \u00271\u0027,"}],"source_content_type":"text/x-python","patch_set":15,"id":"be80ea15_1aba296e","side":"PARENT","line":500,"in_reply_to":"77bd41ba_e1f27241","updated":"2024-12-17 16:25:55.000000000","message":"Removed","commit_id":"fe7928ea8aa45b15fde3154cd8ccdfabc43053a0"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":696,"context_line":""},{"line_number":697,"context_line":"    @mock.patch(\u0027swift.obj.updater.dump_recon_cache\u0027)"},{"line_number":698,"context_line":"    @mock.patch.object(object_updater, \u0027check_drive\u0027)"},{"line_number":699,"context_line":"    def test_run_once_recon_dump(self, mock_check_drive, mock_dump_recon):"},{"line_number":700,"context_line":"        self.maxDiff \u003d None"},{"line_number":701,"context_line":""},{"line_number":702,"context_line":"        def assert_and_reset_recon_dump(exp):"}],"source_content_type":"text/x-python","patch_set":15,"id":"a559c49d_2d2e9fa2","side":"PARENT","line":699,"updated":"2024-12-13 00:46:25.000000000","message":"this is not that good, a test called \"test_recon_dump\" probably shouldn\u0027t mock_dump_recon\n\nIf I wanted to test recon dumps I\u0027d probably just pass in a `recon_cache_path` that\u0027s equal to some subdir of `self.testdir` and just let it write files in there and then I\u0027d read them and make sure they\u0027re correct.","commit_id":"fe7928ea8aa45b15fde3154cd8ccdfabc43053a0"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":696,"context_line":""},{"line_number":697,"context_line":"    @mock.patch(\u0027swift.obj.updater.dump_recon_cache\u0027)"},{"line_number":698,"context_line":"    @mock.patch.object(object_updater, \u0027check_drive\u0027)"},{"line_number":699,"context_line":"    def test_run_once_recon_dump(self, mock_check_drive, mock_dump_recon):"},{"line_number":700,"context_line":"        self.maxDiff \u003d None"},{"line_number":701,"context_line":""},{"line_number":702,"context_line":"        def assert_and_reset_recon_dump(exp):"}],"source_content_type":"text/x-python","patch_set":15,"id":"3a209bdf_82ea33d2","side":"PARENT","line":699,"in_reply_to":"a559c49d_2d2e9fa2","updated":"2024-12-17 16:25:55.000000000","message":"Done in the `test_dump_device_recon` in your squash. Thank you.","commit_id":"fe7928ea8aa45b15fde3154cd8ccdfabc43053a0"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":104,"context_line":"        os.mkdir(self.devices_dir)"},{"line_number":105,"context_line":"        self.sda1 \u003d os.path.join(self.devices_dir, \u0027sda1\u0027)"},{"line_number":106,"context_line":"        os.mkdir(self.sda1)"},{"line_number":107,"context_line":"        self.dev_path \u003d check_drive(self.devices_dir, \u0027sda1\u0027, False)"},{"line_number":108,"context_line":"        for policy in POLICIES:"},{"line_number":109,"context_line":"            os.mkdir(os.path.join(self.sda1, get_tmp_dir(policy)))"},{"line_number":110,"context_line":"        self.logger \u003d debug_logger()"}],"source_content_type":"text/x-python","patch_set":15,"id":"9ca2e56e_93baeb23","line":107,"updated":"2024-12-13 00:46:25.000000000","message":"it\u0027s maybe a little confusing that `dev_path` here in this global setup is *actually* *specifically* *just* the `sda1_dev_path`\n\nit\u0027s also worth noting it\u0027s exactly the same as the value we already had as `self.sda1`\n\n937659: sq? maybe don\u0027t use self.dev_path as alias for self.sda1 | https://review.opendev.org/c/openstack/swift/+/937659","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"11c8c3da7063aac03db0234be28fdbd62efa4cb6","unresolved":false,"context_lines":[{"line_number":104,"context_line":"        os.mkdir(self.devices_dir)"},{"line_number":105,"context_line":"        self.sda1 \u003d os.path.join(self.devices_dir, \u0027sda1\u0027)"},{"line_number":106,"context_line":"        os.mkdir(self.sda1)"},{"line_number":107,"context_line":"        self.dev_path \u003d check_drive(self.devices_dir, \u0027sda1\u0027, False)"},{"line_number":108,"context_line":"        for policy in POLICIES:"},{"line_number":109,"context_line":"            os.mkdir(os.path.join(self.sda1, get_tmp_dir(policy)))"},{"line_number":110,"context_line":"        self.logger \u003d debug_logger()"}],"source_content_type":"text/x-python","patch_set":15,"id":"964509e4_1ac1fac7","line":107,"in_reply_to":"9ca2e56e_93baeb23","updated":"2024-12-13 20:29:29.000000000","message":"Squashed 🙏","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":496,"context_line":"        with self.assertRaises(SystemExit):"},{"line_number":497,"context_line":"            ou.run_once()"},{"line_number":498,"context_line":"        self.assertEqual(["},{"line_number":499,"context_line":"            mock.call(self.dev_path, \u0027sda1\u0027),"},{"line_number":500,"context_line":"        ], mock_process.mock_calls)"},{"line_number":501,"context_line":""},{"line_number":502,"context_line":"    @mock.patch.object(object_updater, \u0027check_drive\u0027)"}],"source_content_type":"text/x-python","patch_set":15,"id":"93ab0753_9267f157","line":499,"updated":"2024-12-13 00:46:25.000000000","message":"it\u0027d be a weird and confusing test to write; but with a `update_workers \u003d 2` config and `mock_os.fork_side_effect \u003d [1, 0]` you could show that the *second* child gets the second device.\n\n937660: sq? some more unittests | https://review.opendev.org/c/openstack/swift/+/937660","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":true,"context_lines":[{"line_number":496,"context_line":"        with self.assertRaises(SystemExit):"},{"line_number":497,"context_line":"            ou.run_once()"},{"line_number":498,"context_line":"        self.assertEqual(["},{"line_number":499,"context_line":"            mock.call(self.dev_path, \u0027sda1\u0027),"},{"line_number":500,"context_line":"        ], mock_process.mock_calls)"},{"line_number":501,"context_line":""},{"line_number":502,"context_line":"    @mock.patch.object(object_updater, \u0027check_drive\u0027)"}],"source_content_type":"text/x-python","patch_set":15,"id":"c41e6962_da65fa41","line":499,"in_reply_to":"93ab0753_9267f157","updated":"2024-12-17 16:25:55.000000000","message":"Right, not sure if my new `test_run_once_child_with_two_workers` test does what you expect?","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"f0d1c72b24874e9e254e83c8923d7615870e2225","unresolved":false,"context_lines":[{"line_number":496,"context_line":"        with self.assertRaises(SystemExit):"},{"line_number":497,"context_line":"            ou.run_once()"},{"line_number":498,"context_line":"        self.assertEqual(["},{"line_number":499,"context_line":"            mock.call(self.dev_path, \u0027sda1\u0027),"},{"line_number":500,"context_line":"        ], mock_process.mock_calls)"},{"line_number":501,"context_line":""},{"line_number":502,"context_line":"    @mock.patch.object(object_updater, \u0027check_drive\u0027)"}],"source_content_type":"text/x-python","patch_set":15,"id":"53507996_7d9d4b9d","line":499,"in_reply_to":"c41e6962_da65fa41","updated":"2024-12-17 20:10:28.000000000","message":"Fixed.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"2d52cdf17f58515b3c5eb7854adb59508a2401e7","unresolved":false,"context_lines":[{"line_number":496,"context_line":"        with self.assertRaises(SystemExit):"},{"line_number":497,"context_line":"            ou.run_once()"},{"line_number":498,"context_line":"        self.assertEqual(["},{"line_number":499,"context_line":"            mock.call(self.dev_path, \u0027sda1\u0027),"},{"line_number":500,"context_line":"        ], mock_process.mock_calls)"},{"line_number":501,"context_line":""},{"line_number":502,"context_line":"    @mock.patch.object(object_updater, \u0027check_drive\u0027)"}],"source_content_type":"text/x-python","patch_set":15,"id":"9080d2b8_9229224b","line":499,"in_reply_to":"c41e6962_da65fa41","updated":"2024-12-17 20:09:15.000000000","message":"yeah `test_run_once_child_with_more_workers` seems legit.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":544,"context_line":"        mock_os.listdir \u003d os.listdir"},{"line_number":545,"context_line":"        ou \u003d object_updater.ObjectUpdater({"},{"line_number":546,"context_line":"            \u0027devices\u0027: self.devices_dir,"},{"line_number":547,"context_line":"            \u0027mount_check\u0027: \u0027false\u0027,"},{"line_number":548,"context_line":"            \u0027swift_dir\u0027: self.testdir,"},{"line_number":549,"context_line":"            \u0027updater_workers\u0027: \u00274\u0027,"},{"line_number":550,"context_line":"            \u0027interval\u0027: \u00271\u0027,"}],"source_content_type":"text/x-python","patch_set":15,"id":"b81e4aaf_2a0d5f93","line":547,"updated":"2024-12-13 00:46:25.000000000","message":"it might be nice to test some \"truty\" value for `mount_check: 1` to get some coverage on calling `check_drive(r, d, True)` since we have it mocked anyway.\n\n937660: sq? some more unittests | https://review.opendev.org/c/openstack/swift/+/937660","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"11c8c3da7063aac03db0234be28fdbd62efa4cb6","unresolved":false,"context_lines":[{"line_number":544,"context_line":"        mock_os.listdir \u003d os.listdir"},{"line_number":545,"context_line":"        ou \u003d object_updater.ObjectUpdater({"},{"line_number":546,"context_line":"            \u0027devices\u0027: self.devices_dir,"},{"line_number":547,"context_line":"            \u0027mount_check\u0027: \u0027false\u0027,"},{"line_number":548,"context_line":"            \u0027swift_dir\u0027: self.testdir,"},{"line_number":549,"context_line":"            \u0027updater_workers\u0027: \u00274\u0027,"},{"line_number":550,"context_line":"            \u0027interval\u0027: \u00271\u0027,"}],"source_content_type":"text/x-python","patch_set":15,"id":"c153426a_085088a6","line":547,"in_reply_to":"b81e4aaf_2a0d5f93","updated":"2024-12-13 20:29:29.000000000","message":"Squashed","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":566,"context_line":"            mock.call(self.devices_dir, \u0027sda1\u0027, False),"},{"line_number":567,"context_line":"            mock.call(self.devices_dir, \u0027sda0\u0027, False),"},{"line_number":568,"context_line":"            mock.call(self.devices_dir, \u0027sda2\u0027, False),"},{"line_number":569,"context_line":"            mock.call(self.devices_dir, \u0027sda3\u0027, False),"},{"line_number":570,"context_line":"        ], mock_check_drive.mock_calls)"},{"line_number":571,"context_line":"        self.assertEqual(["},{"line_number":572,"context_line":"            mock.call.fork(),"}],"source_content_type":"text/x-python","patch_set":15,"id":"41cf2285_0f553057","line":569,"updated":"2024-12-13 00:46:25.000000000","message":"I wonder if it\u0027s really reasonable to assume that `os.listdir` will consistently return the devices in this order or if we should just force it since we\u0027ve got it mocked anyway.\n\n```\ndef fake_list_dir(path):\n    if path \u003d\u003d self.devicies_dir:\n        return [\u0027sda1\u0027, \u0027sda0\u0027, \u0027sda2\u0027, \u0027sda3\u0027]\n    else:\n        return os.listdir(path)\n\nmock_os.side_effect \u003d fake_list_dir\n```","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"11c8c3da7063aac03db0234be28fdbd62efa4cb6","unresolved":false,"context_lines":[{"line_number":566,"context_line":"            mock.call(self.devices_dir, \u0027sda1\u0027, False),"},{"line_number":567,"context_line":"            mock.call(self.devices_dir, \u0027sda0\u0027, False),"},{"line_number":568,"context_line":"            mock.call(self.devices_dir, \u0027sda2\u0027, False),"},{"line_number":569,"context_line":"            mock.call(self.devices_dir, \u0027sda3\u0027, False),"},{"line_number":570,"context_line":"        ], mock_check_drive.mock_calls)"},{"line_number":571,"context_line":"        self.assertEqual(["},{"line_number":572,"context_line":"            mock.call.fork(),"}],"source_content_type":"text/x-python","patch_set":15,"id":"a97edf99_b5d3f9d6","line":569,"in_reply_to":"41cf2285_0f553057","updated":"2024-12-13 20:29:29.000000000","message":"Done.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":793,"context_line":"            ]))"},{"line_number":794,"context_line":""},{"line_number":795,"context_line":"    @mock.patch(\u0027swift.obj.updater.dump_recon_cache\u0027)"},{"line_number":796,"context_line":"    @mock.patch.object(object_updater, \u0027check_drive\u0027)"},{"line_number":797,"context_line":"    def test_run_once_recon_dump(self, mock_check_drive, mock_dump_recon):"},{"line_number":798,"context_line":"        self.maxDiff \u003d None"},{"line_number":799,"context_line":""}],"source_content_type":"text/x-python","patch_set":15,"id":"447ba2d0_b67138a6","line":796,"updated":"2024-12-13 00:46:25.000000000","message":"this isn\u0027t called; test still passes if it\u0027s not mocked.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":793,"context_line":"            ]))"},{"line_number":794,"context_line":""},{"line_number":795,"context_line":"    @mock.patch(\u0027swift.obj.updater.dump_recon_cache\u0027)"},{"line_number":796,"context_line":"    @mock.patch.object(object_updater, \u0027check_drive\u0027)"},{"line_number":797,"context_line":"    def test_run_once_recon_dump(self, mock_check_drive, mock_dump_recon):"},{"line_number":798,"context_line":"        self.maxDiff \u003d None"},{"line_number":799,"context_line":""}],"source_content_type":"text/x-python","patch_set":15,"id":"17380340_57abf361","line":796,"in_reply_to":"447ba2d0_b67138a6","updated":"2024-12-17 16:25:55.000000000","message":"Removed.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":797,"context_line":"    def test_run_once_recon_dump(self, mock_check_drive, mock_dump_recon):"},{"line_number":798,"context_line":"        self.maxDiff \u003d None"},{"line_number":799,"context_line":""},{"line_number":800,"context_line":"        def assert_and_reset_recon_dump(exp):"},{"line_number":801,"context_line":"            recon_dumps \u003d ["},{"line_number":802,"context_line":"                call[0][0] for call in mock_dump_recon.call_args_list]"},{"line_number":803,"context_line":"            for dump in recon_dumps:"}],"source_content_type":"text/x-python","patch_set":15,"id":"32e8e1c7_20b1f7b9","line":800,"updated":"2024-12-13 00:46:25.000000000","message":"I think this test is now \"per_device\" - probably good to include that idea in the name.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":797,"context_line":"    def test_run_once_recon_dump(self, mock_check_drive, mock_dump_recon):"},{"line_number":798,"context_line":"        self.maxDiff \u003d None"},{"line_number":799,"context_line":""},{"line_number":800,"context_line":"        def assert_and_reset_recon_dump(exp):"},{"line_number":801,"context_line":"            recon_dumps \u003d ["},{"line_number":802,"context_line":"                call[0][0] for call in mock_dump_recon.call_args_list]"},{"line_number":803,"context_line":"            for dump in recon_dumps:"}],"source_content_type":"text/x-python","patch_set":15,"id":"f55195d8_875bc453","line":800,"in_reply_to":"32e8e1c7_20b1f7b9","updated":"2024-12-17 16:25:55.000000000","message":"Added.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":806,"context_line":"                    if stats.get(\u0027failures_oldest_timestamp\u0027):"},{"line_number":807,"context_line":"                        stats[\u0027failures_oldest_timestamp\u0027] \u003d \u0027{:.5f}\u0027.format("},{"line_number":808,"context_line":"                            float(stats[\u0027failures_oldest_timestamp\u0027])"},{"line_number":809,"context_line":"                        )"},{"line_number":810,"context_line":"                    for entry in stats.get("},{"line_number":811,"context_line":"                        \u0027failures_oldest_timestamp_account_containers\u0027, {}"},{"line_number":812,"context_line":"                    ).get(\u0027oldest_entries\u0027, []):"}],"source_content_type":"text/x-python","patch_set":15,"id":"8782f1ef_ceda509b","line":809,"updated":"2024-12-13 00:46:25.000000000","message":"i don\u0027t understand?  was `stats[\u0027failures_oldest_timestmap\u0027]` somehow not already a float?  Is it *supposed* to be a string?","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":806,"context_line":"                    if stats.get(\u0027failures_oldest_timestamp\u0027):"},{"line_number":807,"context_line":"                        stats[\u0027failures_oldest_timestamp\u0027] \u003d \u0027{:.5f}\u0027.format("},{"line_number":808,"context_line":"                            float(stats[\u0027failures_oldest_timestamp\u0027])"},{"line_number":809,"context_line":"                        )"},{"line_number":810,"context_line":"                    for entry in stats.get("},{"line_number":811,"context_line":"                        \u0027failures_oldest_timestamp_account_containers\u0027, {}"},{"line_number":812,"context_line":"                    ).get(\u0027oldest_entries\u0027, []):"}],"source_content_type":"text/x-python","patch_set":15,"id":"baa58a39_669451cd","line":809,"in_reply_to":"8782f1ef_ceda509b","updated":"2024-12-17 16:25:55.000000000","message":"Removed.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":811,"context_line":"                        \u0027failures_oldest_timestamp_account_containers\u0027, {}"},{"line_number":812,"context_line":"                    ).get(\u0027oldest_entries\u0027, []):"},{"line_number":813,"context_line":"                        entry[\u0027timestamp\u0027] \u003d \u0027{:.5f}\u0027.format("},{"line_number":814,"context_line":"                            float(entry[\u0027timestamp\u0027]))"},{"line_number":815,"context_line":"            self.assertEqual([exp], recon_dumps)"},{"line_number":816,"context_line":"            mock_dump_recon.reset_mock()"},{"line_number":817,"context_line":""}],"source_content_type":"text/x-python","patch_set":15,"id":"332de806_36238eb8","line":814,"updated":"2024-12-13 00:46:25.000000000","message":"is this... \"assert and reset\" *modifying* the value we assert matches the exp we pass in?","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":811,"context_line":"                        \u0027failures_oldest_timestamp_account_containers\u0027, {}"},{"line_number":812,"context_line":"                    ).get(\u0027oldest_entries\u0027, []):"},{"line_number":813,"context_line":"                        entry[\u0027timestamp\u0027] \u003d \u0027{:.5f}\u0027.format("},{"line_number":814,"context_line":"                            float(entry[\u0027timestamp\u0027]))"},{"line_number":815,"context_line":"            self.assertEqual([exp], recon_dumps)"},{"line_number":816,"context_line":"            mock_dump_recon.reset_mock()"},{"line_number":817,"context_line":""}],"source_content_type":"text/x-python","patch_set":15,"id":"13b4aa6b_a204fff9","line":814,"in_reply_to":"332de806_36238eb8","updated":"2024-12-17 16:25:55.000000000","message":"Removed","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":822,"context_line":"        ou \u003d object_updater.ObjectUpdater("},{"line_number":823,"context_line":"            {"},{"line_number":824,"context_line":"                \u0027devices\u0027: self.devices_dir,"},{"line_number":825,"context_line":"                \u0027mount_check\u0027: \u0027true\u0027,"},{"line_number":826,"context_line":"                \u0027swift_dir\u0027: self.testdir,"},{"line_number":827,"context_line":"                \u0027interval\u0027: \u00271\u0027,"},{"line_number":828,"context_line":"                \u0027concurrency\u0027: \u00271\u0027,"}],"source_content_type":"text/x-python","patch_set":15,"id":"95e195db_0bbec4e2","line":825,"updated":"2024-12-13 00:46:25.000000000","message":"since we don\u0027t call `run_once`/`mount_check` anymore in this test it\u0027s probably not really relevant to continue to configure this option.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":822,"context_line":"        ou \u003d object_updater.ObjectUpdater("},{"line_number":823,"context_line":"            {"},{"line_number":824,"context_line":"                \u0027devices\u0027: self.devices_dir,"},{"line_number":825,"context_line":"                \u0027mount_check\u0027: \u0027true\u0027,"},{"line_number":826,"context_line":"                \u0027swift_dir\u0027: self.testdir,"},{"line_number":827,"context_line":"                \u0027interval\u0027: \u00271\u0027,"},{"line_number":828,"context_line":"                \u0027concurrency\u0027: \u00271\u0027,"}],"source_content_type":"text/x-python","patch_set":15,"id":"b2614c97_0a5c95f7","line":825,"in_reply_to":"95e195db_0bbec4e2","updated":"2024-12-17 16:25:55.000000000","message":"Removed","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":829,"context_line":"                \u0027node_timeout\u0027: \u002715\u0027,"},{"line_number":830,"context_line":"            },"},{"line_number":831,"context_line":"            logger\u003dself.logger,"},{"line_number":832,"context_line":"        )"},{"line_number":833,"context_line":"        with mock.patch.object("},{"line_number":834,"context_line":"            ou, \u0027object_update\u0027, return_value\u003d(False, \u0027node-id\u0027, None)"},{"line_number":835,"context_line":"        ):"}],"source_content_type":"text/x-python","patch_set":15,"id":"6f4c54f3_28b04fc5","line":832,"updated":"2024-12-13 00:46:25.000000000","message":"I don\u0027t think anything actually changed here.\n\nWas there something wrong with the original formatting?\n\nTry to keep your diffs tight please.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":829,"context_line":"                \u0027node_timeout\u0027: \u002715\u0027,"},{"line_number":830,"context_line":"            },"},{"line_number":831,"context_line":"            logger\u003dself.logger,"},{"line_number":832,"context_line":"        )"},{"line_number":833,"context_line":"        with mock.patch.object("},{"line_number":834,"context_line":"            ou, \u0027object_update\u0027, return_value\u003d(False, \u0027node-id\u0027, None)"},{"line_number":835,"context_line":"        ):"}],"source_content_type":"text/x-python","patch_set":15,"id":"3b08c23c_78f9da08","line":832,"in_reply_to":"6f4c54f3_28b04fc5","updated":"2024-12-17 16:25:55.000000000","message":"Reverted","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":877,"context_line":"            \u0027object_updater_per_device\u0027: {"},{"line_number":878,"context_line":"                \u0027sda1\u0027: {"},{"line_number":879,"context_line":"                    \u0027failures_account_container_count\u0027: 1,"},{"line_number":880,"context_line":"                    \u0027failures_oldest_timestamp\u0027: \u0027{:.5f}\u0027.format(float(ts)),"},{"line_number":881,"context_line":"                    \u0027failures_oldest_timestamp_account_containers\u0027: {"},{"line_number":882,"context_line":"                        \u0027oldest_count\u0027: 1,"},{"line_number":883,"context_line":"                        \u0027oldest_entries\u0027: ["}],"source_content_type":"text/x-python","patch_set":15,"id":"1de91c0e_2cba33de","line":880,"updated":"2024-12-13 00:46:25.000000000","message":"wait, is it supposed to be a float or a string?","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":877,"context_line":"            \u0027object_updater_per_device\u0027: {"},{"line_number":878,"context_line":"                \u0027sda1\u0027: {"},{"line_number":879,"context_line":"                    \u0027failures_account_container_count\u0027: 1,"},{"line_number":880,"context_line":"                    \u0027failures_oldest_timestamp\u0027: \u0027{:.5f}\u0027.format(float(ts)),"},{"line_number":881,"context_line":"                    \u0027failures_oldest_timestamp_account_containers\u0027: {"},{"line_number":882,"context_line":"                        \u0027oldest_count\u0027: 1,"},{"line_number":883,"context_line":"                        \u0027oldest_entries\u0027: ["}],"source_content_type":"text/x-python","patch_set":15,"id":"c6aa65f6_125a2a7f","line":880,"in_reply_to":"1de91c0e_2cba33de","updated":"2024-12-17 16:25:55.000000000","message":"Float, updated.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":896,"context_line":"        assert_and_reset_recon_dump(exp_recon_dump)"},{"line_number":897,"context_line":""},{"line_number":898,"context_line":"    @mock.patch(\u0027swift.obj.updater.dump_recon_cache\u0027)"},{"line_number":899,"context_line":"    def test_dump_device_recon(self, mock_dump_recon):"},{"line_number":900,"context_line":"        ou \u003d object_updater.ObjectUpdater({"},{"line_number":901,"context_line":"            \u0027devices\u0027: self.devices_dir,"},{"line_number":902,"context_line":"            \u0027swift_dir\u0027: self.testdir,"}],"source_content_type":"text/x-python","patch_set":15,"id":"5f4f76c5_847db235","line":899,"updated":"2024-12-13 00:46:25.000000000","message":"I *really* like the idea of writing a *new* test","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":896,"context_line":"        assert_and_reset_recon_dump(exp_recon_dump)"},{"line_number":897,"context_line":""},{"line_number":898,"context_line":"    @mock.patch(\u0027swift.obj.updater.dump_recon_cache\u0027)"},{"line_number":899,"context_line":"    def test_dump_device_recon(self, mock_dump_recon):"},{"line_number":900,"context_line":"        ou \u003d object_updater.ObjectUpdater({"},{"line_number":901,"context_line":"            \u0027devices\u0027: self.devices_dir,"},{"line_number":902,"context_line":"            \u0027swift_dir\u0027: self.testdir,"}],"source_content_type":"text/x-python","patch_set":15,"id":"eea5506f_f084c095","line":899,"in_reply_to":"5f4f76c5_847db235","updated":"2024-12-17 16:25:55.000000000","message":"Done","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":916,"context_line":"            },"},{"line_number":917,"context_line":"            \u0027tracker_memory_usage\u0027: 1024,"},{"line_number":918,"context_line":"        }"},{"line_number":919,"context_line":"        with mock.patch.object(ou, \u0027_gather_recon_stats\u0027,"},{"line_number":920,"context_line":"                               return_value\u003dmock_stats):"},{"line_number":921,"context_line":"            ou.dump_device_recon(\u0027sda1\u0027)"},{"line_number":922,"context_line":"        mock_dump_recon.assert_called_once_with("}],"source_content_type":"text/x-python","patch_set":15,"id":"760c499f_ee43cc1e","line":919,"updated":"2024-12-13 00:46:25.000000000","message":"where\u0027s the test that says _gather_recon_stats return a format like this?","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"f0d1c72b24874e9e254e83c8923d7615870e2225","unresolved":false,"context_lines":[{"line_number":916,"context_line":"            },"},{"line_number":917,"context_line":"            \u0027tracker_memory_usage\u0027: 1024,"},{"line_number":918,"context_line":"        }"},{"line_number":919,"context_line":"        with mock.patch.object(ou, \u0027_gather_recon_stats\u0027,"},{"line_number":920,"context_line":"                               return_value\u003dmock_stats):"},{"line_number":921,"context_line":"            ou.dump_device_recon(\u0027sda1\u0027)"},{"line_number":922,"context_line":"        mock_dump_recon.assert_called_once_with("}],"source_content_type":"text/x-python","patch_set":15,"id":"acf94b2f_625c59b2","line":919,"in_reply_to":"760c499f_ee43cc1e","updated":"2024-12-17 20:10:28.000000000","message":"Done","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"2d52cdf17f58515b3c5eb7854adb59508a2401e7","unresolved":false,"context_lines":[{"line_number":916,"context_line":"            },"},{"line_number":917,"context_line":"            \u0027tracker_memory_usage\u0027: 1024,"},{"line_number":918,"context_line":"        }"},{"line_number":919,"context_line":"        with mock.patch.object(ou, \u0027_gather_recon_stats\u0027,"},{"line_number":920,"context_line":"                               return_value\u003dmock_stats):"},{"line_number":921,"context_line":"            ou.dump_device_recon(\u0027sda1\u0027)"},{"line_number":922,"context_line":"        mock_dump_recon.assert_called_once_with("}],"source_content_type":"text/x-python","patch_set":15,"id":"582cd3d9_b76589d1","line":919,"in_reply_to":"760c499f_ee43cc1e","updated":"2024-12-17 20:09:15.000000000","message":"this test doesn\u0027t mock anymore","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":923,"context_line":"            {\u0027object_updater_per_device\u0027: {\u0027sda1\u0027: mock_stats}},"},{"line_number":924,"context_line":"            ou.rcache,"},{"line_number":925,"context_line":"            ou.logger,"},{"line_number":926,"context_line":"        )"},{"line_number":927,"context_line":""},{"line_number":928,"context_line":"    @mock.patch(\u0027swift.obj.updater.load_recon_cache\u0027)"},{"line_number":929,"context_line":"    @mock.patch(\u0027swift.obj.updater.dump_recon_cache\u0027)"}],"source_content_type":"text/x-python","patch_set":15,"id":"82abfc80_a0302d5b","line":926,"updated":"2024-12-13 00:46:25.000000000","message":"this is a *really* thin test, it\u0027s basically just saying \"dump_device_recon calls dump_recon with whatever it gets from _gather_recon_stats\"\n\nand we\u0027re mocking on both sides.  Let\u0027s think about what might be better:\n\n937663: sq? more real recon dump testing | https://review.opendev.org/c/openstack/swift/+/937663","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":923,"context_line":"            {\u0027object_updater_per_device\u0027: {\u0027sda1\u0027: mock_stats}},"},{"line_number":924,"context_line":"            ou.rcache,"},{"line_number":925,"context_line":"            ou.logger,"},{"line_number":926,"context_line":"        )"},{"line_number":927,"context_line":""},{"line_number":928,"context_line":"    @mock.patch(\u0027swift.obj.updater.load_recon_cache\u0027)"},{"line_number":929,"context_line":"    @mock.patch(\u0027swift.obj.updater.dump_recon_cache\u0027)"}],"source_content_type":"text/x-python","patch_set":15,"id":"96b7e7e4_1de2884a","line":926,"in_reply_to":"82abfc80_a0302d5b","updated":"2024-12-17 16:25:55.000000000","message":"Squashed","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":928,"context_line":"    @mock.patch(\u0027swift.obj.updater.load_recon_cache\u0027)"},{"line_number":929,"context_line":"    @mock.patch(\u0027swift.obj.updater.dump_recon_cache\u0027)"},{"line_number":930,"context_line":"    def test_aggregate_and_dump_recon(self, mock_dump_recon, mock_load_recon):"},{"line_number":931,"context_line":"        mock_load_recon.return_value \u003d {"},{"line_number":932,"context_line":"            \u0027object_updater_per_device\u0027: {"},{"line_number":933,"context_line":"                \u0027sda1\u0027: {"},{"line_number":934,"context_line":"                    \u0027failures_account_container_count\u0027: 1,"}],"source_content_type":"text/x-python","patch_set":15,"id":"1315a4b3_2dd06e27","line":931,"updated":"2024-12-13 00:46:25.000000000","message":"or just write it do disk using `os.dump_device_recon()`","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":928,"context_line":"    @mock.patch(\u0027swift.obj.updater.load_recon_cache\u0027)"},{"line_number":929,"context_line":"    @mock.patch(\u0027swift.obj.updater.dump_recon_cache\u0027)"},{"line_number":930,"context_line":"    def test_aggregate_and_dump_recon(self, mock_dump_recon, mock_load_recon):"},{"line_number":931,"context_line":"        mock_load_recon.return_value \u003d {"},{"line_number":932,"context_line":"            \u0027object_updater_per_device\u0027: {"},{"line_number":933,"context_line":"                \u0027sda1\u0027: {"},{"line_number":934,"context_line":"                    \u0027failures_account_container_count\u0027: 1,"}],"source_content_type":"text/x-python","patch_set":15,"id":"f8408339_9f7e292b","line":931,"in_reply_to":"1315a4b3_2dd06e27","updated":"2024-12-17 16:25:55.000000000","message":"Acknowledged","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":964,"context_line":"            \u0027updater_workers\u0027: 2,"},{"line_number":965,"context_line":"            \u0027async_tracker_dump_count\u0027: 2,"},{"line_number":966,"context_line":"        })"},{"line_number":967,"context_line":"        ou._all_local_devices \u003d [\u0027sda1\u0027, \u0027sda2\u0027]"},{"line_number":968,"context_line":"        ou.aggregate_and_dump_recon(30)"},{"line_number":969,"context_line":""},{"line_number":970,"context_line":"        expected_aggregated_stats \u003d {"}],"source_content_type":"text/x-python","patch_set":15,"id":"1304ac67_65d11ab9","line":967,"updated":"2024-12-13 00:46:25.000000000","message":"haha!  I knew it would AttributeError ;)\n\nit\u0027s not *always* a bad thing if a unittest has to manipulate private state but it\u0027s also reasonable that if `aggregate_and_dump_recon` can\u0027t blow-up with-out *someone* doing this that it\u0027s *kind of* an \"input\" to this method.\n\nFunctional purists would be all about explicitly passing the state as argument, I just think it would make it a little easier to test.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":964,"context_line":"            \u0027updater_workers\u0027: 2,"},{"line_number":965,"context_line":"            \u0027async_tracker_dump_count\u0027: 2,"},{"line_number":966,"context_line":"        })"},{"line_number":967,"context_line":"        ou._all_local_devices \u003d [\u0027sda1\u0027, \u0027sda2\u0027]"},{"line_number":968,"context_line":"        ou.aggregate_and_dump_recon(30)"},{"line_number":969,"context_line":""},{"line_number":970,"context_line":"        expected_aggregated_stats \u003d {"}],"source_content_type":"text/x-python","patch_set":15,"id":"a1e10c13_9b2d4ca3","line":967,"in_reply_to":"1304ac67_65d11ab9","updated":"2024-12-17 16:25:55.000000000","message":"Right, updated","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":969,"context_line":""},{"line_number":970,"context_line":"        expected_aggregated_stats \u003d {"},{"line_number":971,"context_line":"            \u0027failures_account_container_count\u0027: 3,"},{"line_number":972,"context_line":"            \u0027tracker_memory_usage\u0027: 384,  # (512 + 256) / 2"},{"line_number":973,"context_line":"            \u0027failures_oldest_timestamp\u0027: 123.45678,"},{"line_number":974,"context_line":"            \u0027failures_oldest_timestamp_account_containers\u0027: {"},{"line_number":975,"context_line":"                \u0027oldest_count\u0027: 2,"}],"source_content_type":"text/x-python","patch_set":15,"id":"c53a8a32_c40324cb","line":972,"updated":"2024-12-13 00:46:25.000000000","message":"but since we had update_workers: 2 - weren\u0027t BOTH of their memory in usage at the same time?  This seems like a significant under-reporting.\n\nAlso consider/test what does the \"aggregated average peak memory\" stat it look like if use this logic when we have updater_worker\u003d2 and 16 disks?","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":true,"context_lines":[{"line_number":969,"context_line":""},{"line_number":970,"context_line":"        expected_aggregated_stats \u003d {"},{"line_number":971,"context_line":"            \u0027failures_account_container_count\u0027: 3,"},{"line_number":972,"context_line":"            \u0027tracker_memory_usage\u0027: 384,  # (512 + 256) / 2"},{"line_number":973,"context_line":"            \u0027failures_oldest_timestamp\u0027: 123.45678,"},{"line_number":974,"context_line":"            \u0027failures_oldest_timestamp_account_containers\u0027: {"},{"line_number":975,"context_line":"                \u0027oldest_count\u0027: 2,"}],"source_content_type":"text/x-python","patch_set":15,"id":"ce07206b_874f51b2","line":972,"in_reply_to":"c53a8a32_c40324cb","updated":"2024-12-17 16:25:55.000000000","message":"This is my new logic. Please let me know if it remains incorrect:\n```\n            \u0027tracker_memory_usage\u0027: (\n                sum(\n                    stats.get(\u0027tracker_memory_usage\u0027, 0)\n                    for stats in device_stats.values()\n                ) / len(device_stats)\n            ) * self.updater_workers if device_stats else 0,\n```","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"f0d1c72b24874e9e254e83c8923d7615870e2225","unresolved":false,"context_lines":[{"line_number":969,"context_line":""},{"line_number":970,"context_line":"        expected_aggregated_stats \u003d {"},{"line_number":971,"context_line":"            \u0027failures_account_container_count\u0027: 3,"},{"line_number":972,"context_line":"            \u0027tracker_memory_usage\u0027: 384,  # (512 + 256) / 2"},{"line_number":973,"context_line":"            \u0027failures_oldest_timestamp\u0027: 123.45678,"},{"line_number":974,"context_line":"            \u0027failures_oldest_timestamp_account_containers\u0027: {"},{"line_number":975,"context_line":"                \u0027oldest_count\u0027: 2,"}],"source_content_type":"text/x-python","patch_set":15,"id":"46b9ccd4_c31ddf91","line":972,"in_reply_to":"ce07206b_874f51b2","updated":"2024-12-17 20:10:28.000000000","message":"Acknowledged","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"2d52cdf17f58515b3c5eb7854adb59508a2401e7","unresolved":false,"context_lines":[{"line_number":969,"context_line":""},{"line_number":970,"context_line":"        expected_aggregated_stats \u003d {"},{"line_number":971,"context_line":"            \u0027failures_account_container_count\u0027: 3,"},{"line_number":972,"context_line":"            \u0027tracker_memory_usage\u0027: 384,  # (512 + 256) / 2"},{"line_number":973,"context_line":"            \u0027failures_oldest_timestamp\u0027: 123.45678,"},{"line_number":974,"context_line":"            \u0027failures_oldest_timestamp_account_containers\u0027: {"},{"line_number":975,"context_line":"                \u0027oldest_count\u0027: 2,"}],"source_content_type":"text/x-python","patch_set":15,"id":"a2049d6e_a2c81e92","line":972,"in_reply_to":"ce07206b_874f51b2","updated":"2024-12-17 20:09:15.000000000","message":"Done","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"79b59e25f8a0fefb0287908d15ddf296e85573e2","unresolved":true,"context_lines":[{"line_number":987,"context_line":"            ou.rcache,"},{"line_number":988,"context_line":"            ou.logger,"},{"line_number":989,"context_line":"        )"},{"line_number":990,"context_line":""},{"line_number":991,"context_line":"    def test_obj_put_legacy_updates(self):"},{"line_number":992,"context_line":"        ts \u003d (normalize_timestamp(t) for t in"},{"line_number":993,"context_line":"              itertools.count(int(time())))"}],"source_content_type":"text/x-python","patch_set":15,"id":"efd28b76_5d97e84f","line":990,"updated":"2024-12-13 00:46:25.000000000","message":"so the test I\u0027d really like to see is what happens if some of the devices write empty stats.","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"2c0c5cfafa7325a054ba2e771bcbd9db7384dd67","unresolved":false,"context_lines":[{"line_number":987,"context_line":"            ou.rcache,"},{"line_number":988,"context_line":"            ou.logger,"},{"line_number":989,"context_line":"        )"},{"line_number":990,"context_line":""},{"line_number":991,"context_line":"    def test_obj_put_legacy_updates(self):"},{"line_number":992,"context_line":"        ts \u003d (normalize_timestamp(t) for t in"},{"line_number":993,"context_line":"              itertools.count(int(time())))"}],"source_content_type":"text/x-python","patch_set":15,"id":"b729694f_527160d7","line":990,"in_reply_to":"efd28b76_5d97e84f","updated":"2024-12-17 16:25:55.000000000","message":"Done in `test_aggregate_and_dump_recon`","commit_id":"9f423d1fbd5503caed504715a09800963f9fc611"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"5d789c4be89658438a874fd76209707660f488cf","unresolved":true,"context_lines":[{"line_number":1391,"context_line":"            755.5,"},{"line_number":1392,"context_line":"            found_data[\u0027object_updater_stats\u0027][\u0027tracker_memory_usage\u0027],"},{"line_number":1393,"context_line":"            delta\u003d100"},{"line_number":1394,"context_line":"        )"},{"line_number":1395,"context_line":"    def test_obj_put_legacy_updates(self):"},{"line_number":1396,"context_line":"        ts \u003d (normalize_timestamp(t) for t in"},{"line_number":1397,"context_line":"              itertools.count(int(time())))"}],"source_content_type":"text/x-python","patch_set":28,"id":"1e7fcc0a_337bcda9","line":1394,"updated":"2024-12-18 17:54:57.000000000","message":"needs a blank line at here","commit_id":"d438fff5bb364f2bfe5ed36d80847a3163c2db89"},{"author":{"_account_id":37271,"name":"Chinemerem Chigbo","display_name":"Chinemerem","email":"cchigbo@nvidia.com","username":"chinemerem"},"change_message_id":"19f315968a766fdcec01309ec34142ff7ea3b8a1","unresolved":false,"context_lines":[{"line_number":1391,"context_line":"            755.5,"},{"line_number":1392,"context_line":"            found_data[\u0027object_updater_stats\u0027][\u0027tracker_memory_usage\u0027],"},{"line_number":1393,"context_line":"            delta\u003d100"},{"line_number":1394,"context_line":"        )"},{"line_number":1395,"context_line":"    def test_obj_put_legacy_updates(self):"},{"line_number":1396,"context_line":"        ts \u003d (normalize_timestamp(t) for t in"},{"line_number":1397,"context_line":"              itertools.count(int(time())))"}],"source_content_type":"text/x-python","patch_set":28,"id":"aea49ed6_42e50d29","line":1394,"in_reply_to":"1e7fcc0a_337bcda9","updated":"2024-12-18 17:58:13.000000000","message":"Done","commit_id":"d438fff5bb364f2bfe5ed36d80847a3163c2db89"}]}