)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":17669,"name":"Doug Szumski","email":"doug@stackhpc.com","username":"DougSzumski"},"change_message_id":"0abdabc1f069ceffdfc10e16abcc203ea1648f1c","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"f3800572_a09f974a","updated":"2025-01-29 11:33:13.000000000","message":"To get a significant benefit out of this, each *service role* must be responsible for registering itself with Prometheus by writing out a configuration file and enabling its own exporter, metric endpoint etc. \n\nEg. If you already have Prometheus enabled, and you want to enable OpenSearch, you should be able to run `kolla deploy -t opensearch`, and it should deploy OpenSearch, the OpenSearch exporter and register the exporter with Prometheus. In some other change, the service role could also be responsible for loading a Grafana dashboard, alerting rules etc. The service role should also be responsible for cleaning up if it is disabled. The objective is to avoid having to run the Prometheus role, when you have enabled / disabled some other service.","commit_id":"6e0f6972cb4d60f46a6723103f13da0a1d19e79b"},{"author":{"_account_id":36702,"name":"Roman Krcek","display_name":"Roman Krček","email":"roman.krcek@tietoevry.com","username":"r-krcek"},"change_message_id":"f61caddf383fd0c2b006dc9903e1780c88fceb9b","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":5,"id":"918e68f3_9d61651e","in_reply_to":"f3800572_a09f974a","updated":"2025-02-25 09:27:21.000000000","message":"Hi, I am here to offer my two cents. I agree that this would be a much sleeker approach, but wouldn\u0027t the implementation of it be quite hard? You have one service on one host modifying files of another service on another host. And what happens when the user runs deploy with --limit? Also what would happen first time deploying OS? You would have circular dependency between haproxy and prometheus.","commit_id":"6e0f6972cb4d60f46a6723103f13da0a1d19e79b"}],"ansible/roles/prometheus/defaults/main.yml":[{"author":{"_account_id":22629,"name":"Michal Nasiadka","email":"mnasiadka@gmail.com","username":"mnasiadka"},"change_message_id":"8f3b81dd600ab91804ec5b5e04ba2d016e0e94c2","unresolved":true,"context_lines":[{"line_number":392,"context_line":""},{"line_number":393,"context_line":"prometheus_server_default_volumes:"},{"line_number":394,"context_line":"  - \"{{ node_config_directory }}/prometheus-server/:{{ container_config_directory }}/:ro\""},{"line_number":395,"context_line":"  - \"{{ node_config_directory }}/prometheus-server/file_sd_configs:/etc/prometheus/file_sd_configs/:ro\""},{"line_number":396,"context_line":"  - \"/etc/localtime:/etc/localtime:ro\""},{"line_number":397,"context_line":"  - \"{{ \u0027/etc/timezone:/etc/timezone:ro\u0027 if ansible_facts.os_family \u003d\u003d \u0027Debian\u0027 else \u0027\u0027 }}\""},{"line_number":398,"context_line":"  - \"prometheus_v2:/var/lib/prometheus\""}],"source_content_type":"text/x-yaml","patch_set":1,"id":"422b82f2_8c46c81f","line":395,"updated":"2024-12-17 17:32:56.000000000","message":"Any reason why? binding anything to final target is not really what we do - especially for config files","commit_id":"b1e848e8e0b465bad903e84db9ff9f4fe693d616"},{"author":{"_account_id":35264,"name":"Alex Welsh","email":"alex@stackhpc.com","username":"alex-welsh"},"change_message_id":"595f0cedce8567b666a814336d82e0060dbc0f26","unresolved":true,"context_lines":[{"line_number":392,"context_line":""},{"line_number":393,"context_line":"prometheus_server_default_volumes:"},{"line_number":394,"context_line":"  - \"{{ node_config_directory }}/prometheus-server/:{{ container_config_directory }}/:ro\""},{"line_number":395,"context_line":"  - \"{{ node_config_directory }}/prometheus-server/file_sd_configs:/etc/prometheus/file_sd_configs/:ro\""},{"line_number":396,"context_line":"  - \"/etc/localtime:/etc/localtime:ro\""},{"line_number":397,"context_line":"  - \"{{ \u0027/etc/timezone:/etc/timezone:ro\u0027 if ansible_facts.os_family \u003d\u003d \u0027Debian\u0027 else \u0027\u0027 }}\""},{"line_number":398,"context_line":"  - \"prometheus_v2:/var/lib/prometheus\""}],"source_content_type":"text/x-yaml","patch_set":1,"id":"db7ba2ab_a8e318de","line":395,"in_reply_to":"422b82f2_8c46c81f","updated":"2024-12-18 08:51:06.000000000","message":"To be able to update the targets without restarting the container, I needed a directory that would be watched continually for changes. I agree, it\u0027s a new pattern, but I\u0027m not sure there\u0027s any other way of doing it.\n\nThat\u0027s why I created the file_sd_configs_staging directory, so that you can run a genconfig without the changes applying instantly","commit_id":"b1e848e8e0b465bad903e84db9ff9f4fe693d616"},{"author":{"_account_id":22629,"name":"Michal Nasiadka","email":"mnasiadka@gmail.com","username":"mnasiadka"},"change_message_id":"9db74241afdf32b15df37fcecba8ec63b3fa82e1","unresolved":false,"context_lines":[{"line_number":392,"context_line":""},{"line_number":393,"context_line":"prometheus_server_default_volumes:"},{"line_number":394,"context_line":"  - \"{{ node_config_directory }}/prometheus-server/:{{ container_config_directory }}/:ro\""},{"line_number":395,"context_line":"  - \"{{ node_config_directory }}/prometheus-server/file_sd_configs:/etc/prometheus/file_sd_configs/:ro\""},{"line_number":396,"context_line":"  - \"/etc/localtime:/etc/localtime:ro\""},{"line_number":397,"context_line":"  - \"{{ \u0027/etc/timezone:/etc/timezone:ro\u0027 if ansible_facts.os_family \u003d\u003d \u0027Debian\u0027 else \u0027\u0027 }}\""},{"line_number":398,"context_line":"  - \"prometheus_v2:/var/lib/prometheus\""}],"source_content_type":"text/x-yaml","patch_set":1,"id":"f9308c49_069e09e3","line":395,"in_reply_to":"db7ba2ab_a8e318de","updated":"2024-12-18 09:00:42.000000000","message":"Acknowledged","commit_id":"b1e848e8e0b465bad903e84db9ff9f4fe693d616"}],"ansible/roles/prometheus/tasks/config.yml":[{"author":{"_account_id":22629,"name":"Michal Nasiadka","email":"mnasiadka@gmail.com","username":"mnasiadka"},"change_message_id":"9db74241afdf32b15df37fcecba8ec63b3fa82e1","unresolved":true,"context_lines":[{"line_number":23,"context_line":"        - file_sd_configs"},{"line_number":24,"context_line":"        - file_sd_configs_staging"},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"    - name: Template file SD configuration files"},{"line_number":27,"context_line":"      include_tasks: file-sd-config.yml"},{"line_number":28,"context_line":"      loop_control:"},{"line_number":29,"context_line":"        loop_var: scrape_job"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"648f1b09_290e8787","line":26,"updated":"2024-12-18 09:00:42.000000000","message":"Is it just me or are we templating all of them irrelevant if those are enabled or not?","commit_id":"b1e848e8e0b465bad903e84db9ff9f4fe693d616"},{"author":{"_account_id":35264,"name":"Alex Welsh","email":"alex@stackhpc.com","username":"alex-welsh"},"change_message_id":"d7f1096dbb09a01c088a39be6c69f63b03dc9bab","unresolved":false,"context_lines":[{"line_number":23,"context_line":"        - file_sd_configs"},{"line_number":24,"context_line":"        - file_sd_configs_staging"},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"    - name: Template file SD configuration files"},{"line_number":27,"context_line":"      include_tasks: file-sd-config.yml"},{"line_number":28,"context_line":"      loop_control:"},{"line_number":29,"context_line":"        loop_var: scrape_job"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"88a12a8c_7e3ad142","line":26,"in_reply_to":"648f1b09_290e8787","updated":"2024-12-18 14:42:20.000000000","message":"Yep, missed that. Fixed now","commit_id":"b1e848e8e0b465bad903e84db9ff9f4fe693d616"},{"author":{"_account_id":22629,"name":"Michal Nasiadka","email":"mnasiadka@gmail.com","username":"mnasiadka"},"change_message_id":"3d9020d9183d372bc9e3e1884f7c338352c0c195","unresolved":true,"context_lines":[{"line_number":23,"context_line":"        - file_sd_configs"},{"line_number":24,"context_line":"        - file_sd_configs_staging"},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"    - name: Template file SD configuration files"},{"line_number":27,"context_line":"      include_tasks: file-sd-config.yml"},{"line_number":28,"context_line":"      loop_control:"},{"line_number":29,"context_line":"        loop_var: scrape_job"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"faf202df_6b3d38e1","line":26,"in_reply_to":"88a12a8c_7e3ad142","updated":"2024-12-18 16:28:41.000000000","message":"Ok, what happens if somebody disables e.g. cadvisor - will the file get deleted?","commit_id":"b1e848e8e0b465bad903e84db9ff9f4fe693d616"},{"author":{"_account_id":35264,"name":"Alex Welsh","email":"alex@stackhpc.com","username":"alex-welsh"},"change_message_id":"1e5b7bc621ab23837d3356950eded9ca5ac4336a","unresolved":false,"context_lines":[{"line_number":23,"context_line":"        - file_sd_configs"},{"line_number":24,"context_line":"        - file_sd_configs_staging"},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"    - name: Template file SD configuration files"},{"line_number":27,"context_line":"      include_tasks: file-sd-config.yml"},{"line_number":28,"context_line":"      loop_control:"},{"line_number":29,"context_line":"        loop_var: scrape_job"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"5dcc4047_c97b592c","line":26,"in_reply_to":"faf202df_6b3d38e1","updated":"2025-01-10 10:48:31.000000000","message":"It will now :)","commit_id":"b1e848e8e0b465bad903e84db9ff9f4fe693d616"}],"ansible/roles/prometheus/templates/prometheus.yml.j2":[{"author":{"_account_id":17669,"name":"Doug Szumski","email":"doug@stackhpc.com","username":"DougSzumski"},"change_message_id":"2f4af2ea84a7acb1beefc527f776376bbefe3970","unresolved":true,"context_lines":[{"line_number":34,"context_line":"      - \u0027file_sd_configs/sd-prometheus_node_exporter.yaml\u0027"},{"line_number":35,"context_line":"{% endif %}"},{"line_number":36,"context_line":""},{"line_number":37,"context_line":"{% if enable_prometheus_mysqld_exporter | bool %}"},{"line_number":38,"context_line":"  - job_name: mysqld"},{"line_number":39,"context_line":"    file_sd_configs:"},{"line_number":40,"context_line":"    - refresh_interval: 30s"}],"source_content_type":"text/x-jinja2","patch_set":4,"id":"47879182_4bf33ba8","line":37,"updated":"2025-01-17 14:07:33.000000000","message":"Please can we move these service specific snippets into the individual config files and use a glob here to find them all? It seems this should be supported.\n\nThe advantage is that you don\u0027t need to reconfigure Prometheus server when you enable ElasticSearch for example. It should also be simpler.","commit_id":"99b6a3b1af074efbf351d62938a6e62786885836"},{"author":{"_account_id":35264,"name":"Alex Welsh","email":"alex@stackhpc.com","username":"alex-welsh"},"change_message_id":"774174d3fc652b7edb8e9eae84440373419c0ed1","unresolved":true,"context_lines":[{"line_number":34,"context_line":"      - \u0027file_sd_configs/sd-prometheus_node_exporter.yaml\u0027"},{"line_number":35,"context_line":"{% endif %}"},{"line_number":36,"context_line":""},{"line_number":37,"context_line":"{% if enable_prometheus_mysqld_exporter | bool %}"},{"line_number":38,"context_line":"  - job_name: mysqld"},{"line_number":39,"context_line":"    file_sd_configs:"},{"line_number":40,"context_line":"    - refresh_interval: 30s"}],"source_content_type":"text/x-jinja2","patch_set":4,"id":"4d8e5c8d_2eebfe0a","line":37,"in_reply_to":"47879182_4bf33ba8","updated":"2025-01-17 16:11:50.000000000","message":"I\u0027m not sure how that would be possible.\n\nThe docs say:\n\u003e It reads a set of files containing a list of zero or more \u003cstatic_config\u003es [1]\n\nStatic configs only seem to contain the targets for the job \n\u003e A static_config allows specifying a list of targets and a common label set for them [2]\n\nSo I don\u0027t think you can include full scrape configs (e.g. no relabeling config for blackbox, or scheme for etcd)\n\nWe could just make all of these list elements unconditional, and we just add the target files conditionally. This file then remains static and we don\u0027t have to restart prometheus. \n\n[1] https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config\n[2] https://prometheus.io/docs/prometheus/latest/configuration/configuration/#static_config","commit_id":"99b6a3b1af074efbf351d62938a6e62786885836"},{"author":{"_account_id":17669,"name":"Doug Szumski","email":"doug@stackhpc.com","username":"DougSzumski"},"change_message_id":"108dee68e2852fd3c1815fbb6a2f42ba7f5e1d5d","unresolved":true,"context_lines":[{"line_number":34,"context_line":"      - \u0027file_sd_configs/sd-prometheus_node_exporter.yaml\u0027"},{"line_number":35,"context_line":"{% endif %}"},{"line_number":36,"context_line":""},{"line_number":37,"context_line":"{% if enable_prometheus_mysqld_exporter | bool %}"},{"line_number":38,"context_line":"  - job_name: mysqld"},{"line_number":39,"context_line":"    file_sd_configs:"},{"line_number":40,"context_line":"    - refresh_interval: 30s"}],"source_content_type":"text/x-jinja2","patch_set":4,"id":"6f508150_959442d7","line":37,"in_reply_to":"4d8e5c8d_2eebfe0a","updated":"2025-01-17 17:03:22.000000000","message":"Thanks Alex, that\u0027s a pain. But yes, making this static sounds fine. What does it look like from Prometheus if you do that? Do the jobs for disabled services get listed in the UI?\n\nI also guess you might need to write all the config files out, but leave them empty for disabled services? Or will it be happy if some don\u0027t actually exist (when the related service is disabled).","commit_id":"99b6a3b1af074efbf351d62938a6e62786885836"},{"author":{"_account_id":35264,"name":"Alex Welsh","email":"alex@stackhpc.com","username":"alex-welsh"},"change_message_id":"77a58a5605661ae36f19ac3a58a0fd2385e77b0f","unresolved":false,"context_lines":[{"line_number":34,"context_line":"      - \u0027file_sd_configs/sd-prometheus_node_exporter.yaml\u0027"},{"line_number":35,"context_line":"{% endif %}"},{"line_number":36,"context_line":""},{"line_number":37,"context_line":"{% if enable_prometheus_mysqld_exporter | bool %}"},{"line_number":38,"context_line":"  - job_name: mysqld"},{"line_number":39,"context_line":"    file_sd_configs:"},{"line_number":40,"context_line":"    - refresh_interval: 30s"}],"source_content_type":"text/x-jinja2","patch_set":4,"id":"33e007ec_56b7ab80","line":37,"in_reply_to":"6f508150_959442d7","updated":"2025-01-21 16:01:03.000000000","message":"It appears to work well. There are no errors or warnings in the prometheus dashboard. The jobs just simply don\u0027t show up in the list if there are no targets.\n\nI\u0027ve removed all the conditional statements that I can.","commit_id":"99b6a3b1af074efbf351d62938a6e62786885836"},{"author":{"_account_id":17669,"name":"Doug Szumski","email":"doug@stackhpc.com","username":"DougSzumski"},"change_message_id":"0abdabc1f069ceffdfc10e16abcc203ea1648f1c","unresolved":true,"context_lines":[{"line_number":181,"context_line":"{% endif %}"},{"line_number":182,"context_line":""},{"line_number":183,"context_line":"{% if enable_prometheus_alertmanager | bool %}"},{"line_number":184,"context_line":"  - job_name: alertmanager"},{"line_number":185,"context_line":"    file_sd_configs:"},{"line_number":186,"context_line":"    - refresh_interval: 30s"},{"line_number":187,"context_line":"      files:"}],"source_content_type":"text/x-jinja2","patch_set":4,"id":"5389d689_99350051","line":184,"updated":"2025-01-29 11:33:13.000000000","message":"If this is disabled, won\u0027t the targets below show as down?","commit_id":"99b6a3b1af074efbf351d62938a6e62786885836"}]}