)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":22629,"name":"Michal Nasiadka","email":"mnasiadka@gmail.com","username":"mnasiadka"},"change_message_id":"954c0542064b1042f11b74752e6a5530b4e89b24","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":5,"id":"61ad26eb_f2e371d3","updated":"2026-04-09 17:50:27.000000000","message":"From my perspective for long term retention we should invenstigate one of these three:\n- Grafana Mimir\n- Thanos\n- GreptimeDB\n\nGreptimeDB is one of the newest options, in theory giving us unified storage for logs, events and metrics and official benchmarks show 5x reduction in resource consumption compared to Mimir (and a single binary).\n\nBut implementing what we have in the meantime and exploring which one of these we\u0027d want to look into.","commit_id":"04ca8b234a90cd9d45ad77cb6bc7c21fdfef7d2e"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"5a355987a87ae25d6a58ddbe0e0ab1cfb6b51ca4","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"a31e7dfd_63128789","updated":"2026-03-18 21:30:36.000000000","message":"This looks great. I have a few notes inline but the one about storage retention is probably the only one that I\u0027m really hoping for a response on.\n\nThen we will also need the infra-prod-service-prometheus job to deploy this service to the production server. This is the grafana example: https://opendev.org/opendev/system-config/src/branch/master/zuul.d/infra-prod.yaml#L540-L556 and we run that here: https://opendev.org/opendev/system-config/src/branch/master/zuul.d/project.yaml#L487-L491 and here: https://opendev.org/opendev/system-config/src/branch/master/zuul.d/project.yaml#L676\n\nWe will need that job and the pipelien config for it so that we can deploy a real server. If you\u0027d prefer to add that in a change that adds the node to the inventory I think that is fine too. But it should also be ok to stage it all.\n\nAs a side note I think grafana is a good comparison as we\u0027re using their upstream container image as well so the structure is similar.","commit_id":"04ca8b234a90cd9d45ad77cb6bc7c21fdfef7d2e"},{"author":{"_account_id":37598,"name":"Ivan Anfimov","display_name":"Ivan Anfimov","email":"lazekteam@gmail.com","username":"anfimovir"},"change_message_id":"a657e5b2da2b0baa9349e6ab1807d00394e58671","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":5,"id":"8a5cad25_331fbbf0","in_reply_to":"61ad26eb_f2e371d3","updated":"2026-04-11 21:31:37.000000000","message":"@mnasiadka@gmail.com What do your think about VictoriaMetrics or ClickHouse for this?","commit_id":"04ca8b234a90cd9d45ad77cb6bc7c21fdfef7d2e"},{"author":{"_account_id":22629,"name":"Michal Nasiadka","email":"mnasiadka@gmail.com","username":"mnasiadka"},"change_message_id":"16e79bd8ab2343f39864afe3a6185240ca757bf0","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"b68c43ad_7d1b4d88","in_reply_to":"8a5cad25_331fbbf0","updated":"2026-05-13 13:23:36.000000000","message":"I think we want to start with something minimal and simplistic, but that\u0027s for recommendations @lazekteam@gmail.com","commit_id":"04ca8b234a90cd9d45ad77cb6bc7c21fdfef7d2e"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"7c8cc470c22d681125a73e9d3fc9ff0910c77f03","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":10,"id":"b4392483_514a905a","updated":"2026-06-08 22:06:54.000000000","message":"The only update since my last review was rebasing for the change to bridge node definitions in system-config-run.yaml","commit_id":"3a123c39e8fb444990d25166460566e9251cbb0b"}],"playbooks/roles/prometheus/tasks/main.yaml":[{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"5a355987a87ae25d6a58ddbe0e0ab1cfb6b51ca4","unresolved":true,"context_lines":[{"line_number":72,"context_line":"    cmd: docker image prune -f"},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"- name: Wait for prometheus to be up and running"},{"line_number":75,"context_line":"  wait_for:"},{"line_number":76,"context_line":"    timeout: 60"},{"line_number":77,"context_line":"    port: 9090"},{"line_number":78,"context_line":"    host: 127.0.0.1"}],"source_content_type":"text/x-yaml","patch_set":5,"id":"a9aec65a_f607d8bb","line":75,"updated":"2026-03-18 21:30:36.000000000","message":"I\u0027m not sure we do this for any of our other services. Is this something we think is helpful so that we get a clear error if the service doesn\u0027t start in the expected amount of time? Mostly just pondering out loud here if this is necessary and what the value is.","commit_id":"04ca8b234a90cd9d45ad77cb6bc7c21fdfef7d2e"},{"author":{"_account_id":22629,"name":"Michal Nasiadka","email":"mnasiadka@gmail.com","username":"mnasiadka"},"change_message_id":"75f9a530eeca8439afac3315ca7a88360f0438f1","unresolved":false,"context_lines":[{"line_number":72,"context_line":"    cmd: docker image prune -f"},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"- name: Wait for prometheus to be up and running"},{"line_number":75,"context_line":"  wait_for:"},{"line_number":76,"context_line":"    timeout: 60"},{"line_number":77,"context_line":"    port: 9090"},{"line_number":78,"context_line":"    host: 127.0.0.1"}],"source_content_type":"text/x-yaml","patch_set":5,"id":"7473560f_5bf3e8ae","line":75,"in_reply_to":"a9aec65a_f607d8bb","updated":"2026-05-13 13:23:50.000000000","message":"Done","commit_id":"04ca8b234a90cd9d45ad77cb6bc7c21fdfef7d2e"}],"playbooks/roles/prometheus/templates/prometheus.yml.j2":[{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"5a355987a87ae25d6a58ddbe0e0ab1cfb6b51ca4","unresolved":true,"context_lines":[{"line_number":1,"context_line":"global:"},{"line_number":2,"context_line":"  scrape_interval: 15s"},{"line_number":3,"context_line":"  evaluation_interval: 15s"},{"line_number":4,"context_line":""},{"line_number":5,"context_line":"scrape_configs:"},{"line_number":6,"context_line":"  - job_name: \u0027prometheus\u0027"}],"source_content_type":"text/x-jinja2","patch_set":5,"id":"d8ef8f64_e491af45","line":3,"updated":"2026-03-18 21:30:36.000000000","message":"Looks like we can set `storage.tsdb.retention.size` which i think can go in here? to limit the disk consumption of the tsdb storage. The docs recommend setting that to 80-85% of your disk storage space. Without knowing how much disk to expect to use would 1TB total be reasonable with an 850GB limit in this config?\n\nI\u0027m looking at https://prometheus.io/docs/prometheus/latest/storage/#right-sizing-retention-size fwiw.\n\nThe main thing I\u0027m thinking about is that we will have a limit that we can adjust over time with lvm, but having something configured to start gives us a guide for how to deploy the server as well.","commit_id":"04ca8b234a90cd9d45ad77cb6bc7c21fdfef7d2e"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"ca00855ea27d511651fcab83b99b4e97f7414a8b","unresolved":true,"context_lines":[{"line_number":1,"context_line":"global:"},{"line_number":2,"context_line":"  scrape_interval: 15s"},{"line_number":3,"context_line":"  evaluation_interval: 15s"},{"line_number":4,"context_line":""},{"line_number":5,"context_line":"scrape_configs:"},{"line_number":6,"context_line":"  - job_name: \u0027prometheus\u0027"}],"source_content_type":"text/x-jinja2","patch_set":5,"id":"83194134_46b2d99d","line":3,"in_reply_to":"286e969f_2bb2082f","updated":"2026-03-20 20:05:56.000000000","message":"We currently have ~86 nodes. This number goes up and down as we add and remove hosts to replace them. If we target say 100 nodes then that is ~400GB per ~30 days. In that case a 1TB volume with 80% utilization should give us ~60 days of retention? If I\u0027ve done that math correctly then I think starting with a 1TB volume is a good starting point.","commit_id":"04ca8b234a90cd9d45ad77cb6bc7c21fdfef7d2e"},{"author":{"_account_id":22629,"name":"Michal Nasiadka","email":"mnasiadka@gmail.com","username":"mnasiadka"},"change_message_id":"f6bc5b511cf1aa2d3b62ac28de2c56b963a4a7fc","unresolved":false,"context_lines":[{"line_number":1,"context_line":"global:"},{"line_number":2,"context_line":"  scrape_interval: 15s"},{"line_number":3,"context_line":"  evaluation_interval: 15s"},{"line_number":4,"context_line":""},{"line_number":5,"context_line":"scrape_configs:"},{"line_number":6,"context_line":"  - job_name: \u0027prometheus\u0027"}],"source_content_type":"text/x-jinja2","patch_set":5,"id":"f0e09c55_3aa3c97b","line":3,"in_reply_to":"83194134_46b2d99d","updated":"2026-05-26 17:16:39.000000000","message":"With GrepTimeDB approach and storing that in Swift/S3 - we don\u0027t need to size the Cinder volume - resolving.","commit_id":"04ca8b234a90cd9d45ad77cb6bc7c21fdfef7d2e"},{"author":{"_account_id":22629,"name":"Michal Nasiadka","email":"mnasiadka@gmail.com","username":"mnasiadka"},"change_message_id":"3df443c379d037acdd5b210f4751b084e6302339","unresolved":true,"context_lines":[{"line_number":1,"context_line":"global:"},{"line_number":2,"context_line":"  scrape_interval: 15s"},{"line_number":3,"context_line":"  evaluation_interval: 15s"},{"line_number":4,"context_line":""},{"line_number":5,"context_line":"scrape_configs:"},{"line_number":6,"context_line":"  - job_name: \u0027prometheus\u0027"}],"source_content_type":"text/x-jinja2","patch_set":5,"id":"286e969f_2bb2082f","line":3,"in_reply_to":"d8ef8f64_e491af45","updated":"2026-03-20 19:29:24.000000000","message":"Node exporter metrics are usually around 1k of metrics per host, iirc it’s around 3-4GB per host for 30 days worth of metrics/indexes/etc. Prometheus recommends to use something else than TSDB for historical metrics (beyond 90 days IIRC) - but we can investigate when we need that?","commit_id":"04ca8b234a90cd9d45ad77cb6bc7c21fdfef7d2e"}],"playbooks/test-prometheus.yaml":[{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"5a355987a87ae25d6a58ddbe0e0ab1cfb6b51ca4","unresolved":true,"context_lines":[{"line_number":3,"context_line":""},{"line_number":4,"context_line":"    - name: Run selenium container"},{"line_number":5,"context_line":"      include_role:"},{"line_number":6,"context_line":"        name: run-selenium"}],"source_content_type":"text/x-yaml","patch_set":5,"id":"502ce51c_c6f2f3fc","line":6,"updated":"2026-03-18 21:30:36.000000000","message":"Currently none of the testinfra tests are taking screenshots so I don\u0027t think this is strictly necessary. However, having it will make adding screenshots later easier.","commit_id":"04ca8b234a90cd9d45ad77cb6bc7c21fdfef7d2e"},{"author":{"_account_id":22629,"name":"Michal Nasiadka","email":"mnasiadka@gmail.com","username":"mnasiadka"},"change_message_id":"16e79bd8ab2343f39864afe3a6185240ca757bf0","unresolved":false,"context_lines":[{"line_number":3,"context_line":""},{"line_number":4,"context_line":"    - name: Run selenium container"},{"line_number":5,"context_line":"      include_role:"},{"line_number":6,"context_line":"        name: run-selenium"}],"source_content_type":"text/x-yaml","patch_set":5,"id":"af6bca1c_021e806a","line":6,"in_reply_to":"502ce51c_c6f2f3fc","updated":"2026-05-13 13:23:36.000000000","message":"Done","commit_id":"04ca8b234a90cd9d45ad77cb6bc7c21fdfef7d2e"}]}