)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":32968,"name":"Juan Larriba","email":"jlarriba@redhat.com","username":"jlarriba"},"change_message_id":"7bb9d29246fe0c683896e3d317badfea92b05c89","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"34b2810c_97ce3105","updated":"2026-04-13 08:54:59.000000000","message":"After reading carefully Sean\u0027s comments and having a lenghty conversation with him about this topic, I decided to explore the possibility of adding the tracing code to the oslo.middleware library.\n\nMy exploration of that topic has rendered excellent results and I will be contributing tracing code there soon.\n\nAs this will be something that will live on an existing library requiring very little modification on the services themselves (with opt-in according to each service community timeframes), I dont think a community goal is required, so I am abandoning it. Please check my incoming patch to oslo.middleware.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":13252,"name":"Dr. Jens Harbott","display_name":"Jens Harbott (frickler)","email":"frickler@offenerstapel.de","username":"jrosenboom"},"change_message_id":"7eaae6254a253717a2950d46f96630946e1851bb","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"b3675c0b_ab4f7001","updated":"2026-03-30 12:16:57.000000000","message":"IIUC the future of the whole middleware framework is unclear, see the thread at https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/KK56O32VOJDMT5WTG5QANHYRYNGIELOG/ in particular. I don\u0027t think we can proceed with a task like this before we get the basics done.\n\nalso, do you volunteer to actually perform the implementation done for all the services that need it? most project teams are severely understaffed, I very much doubt most of them will have the capacity to do anything more than reviewing patches that get proposed to them - if even that.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"f9faab1205b061cc1a15df4813b1ac83b3b9214b","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"629d2b02_cf3ae917","in_reply_to":"7eb02b22_847eddd0","updated":"2026-03-30 18:21:48.000000000","message":"i kind of disagree witth the idea the the middelare woudl live in tree in each service\n\nfor what its worth watcher does not allow middeware confiutaon and i would not want to have ot carry this in tree it hsould be a lib.\n\n\nthe `opentelemetry-sdk`` and``opentelemetry-exporter-otlp-proto-http`` depenecies woudl ideally be optional depencies of oslo.midelware or osprofiel or whatever shared lib.\n\nthe service woudl jsut need to registere the config otpions form that lib and load the midelware into there pepilen\n\n\nif we go forwward with stephens propoasl ot create oslo.wsgi that could be where it lives but personally i see the current Paste/PasteDeploy stack as simialr technical debt to eventlet.\n\nnot your nova poc will need a nova spec to get agreement on if this is something the nova team want to suprpoted as well and how. same for all the other services.\n\nwithout having per service dicsussion on this topic last cycel this feels premature to propsoe as a comunity goal. what proably would make sense howver is to have a converstion about this at the upcoming PTG.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":32968,"name":"Juan Larriba","email":"jlarriba@redhat.com","username":"jlarriba"},"change_message_id":"5ba111b098b46c9d03ccfbec15d641b1b1aae1d4","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"7eb02b22_847eddd0","in_reply_to":"b3675c0b_ab4f7001","updated":"2026-03-30 15:34:01.000000000","message":"Thank you for your input Jens.\n\nObviously it is true that the proposed code would have a lot to do with the middleware, but I have done different services like Nova (which uses Paste/WSGI as framework) and Keystone (which uses Flask) and even if there are differences, it should be quite straightforward to migrate the tracing code from one framework to another.\n\nThis migration might still take some more cycles to be implemented, and during all that time our users could be enjoying tracing in their clouds.\n\nRegarding implementation, yes, I volunteer to contribute the needed code to any service, in phased approach. As stated in the goal, my intention is to start with the most critical services. That seems 80% of the gain with 20% of the effort for the users to be able to detect problems early. These services are Nova, Neutron, Keystone, Cinder and Glance. I will contribute the patches to them and also will be iteratively adding new services as time allows.\n\nI also commit to contribute the fixes to the tracing code when the middleware finally changes.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":32968,"name":"Juan Larriba","email":"jlarriba@redhat.com","username":"jlarriba"},"change_message_id":"5ba111b098b46c9d03ccfbec15d641b1b1aae1d4","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"2ca46a04_aef70cb6","in_reply_to":"b3675c0b_ab4f7001","updated":"2026-03-30 15:34:01.000000000","message":"Thank you for your reply!","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"}],"goals/proposed/tracing.rst":[{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8180229da07c63f5455cca6c795f53177e7ee42f","unresolved":true,"context_lines":[{"line_number":1,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":2,"context_line":"OpenTelemetry Tracing for OpenStack Services"},{"line_number":3,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":4,"context_line":""},{"line_number":5,"context_line":"Observability is a critical capability for operating and debugging distributed"},{"line_number":6,"context_line":"systems. OpenStack services currently lack standardized distributed tracing,"},{"line_number":7,"context_line":"making it difficult to follow a request as it traverses multiple services (e.g."},{"line_number":8,"context_line":"Nova calling Glance, Keystone, Neutron, and Cinder during instance creation)."},{"line_number":9,"context_line":""},{"line_number":10,"context_line":"OpenTelemetry (OTEL) is the industry-standard, vendor-neutral framework for"},{"line_number":11,"context_line":"distributed tracing, supported by the CNCF. By adding lightweight OTEL tracing"}],"source_content_type":"text/x-rst","patch_set":1,"id":"00e37b06_29449695","line":8,"range":{"start_line":4,"start_character":1,"end_line":8,"end_character":77},"updated":"2026-03-30 18:13:59.000000000","message":"so we supprot request-ids to enabel this today\n\ni.e. we proagate teh same rquest-id form the first service to recive the request to all other where possible.\n\nwhat woudl it look like to do the same for OTEL","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8be16d7badff263b11b70e6314adb962c40a8a60","unresolved":true,"context_lines":[{"line_number":1,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":2,"context_line":"OpenTelemetry Tracing for OpenStack Services"},{"line_number":3,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":4,"context_line":""},{"line_number":5,"context_line":"Observability is a critical capability for operating and debugging distributed"},{"line_number":6,"context_line":"systems. OpenStack services currently lack standardized distributed tracing,"},{"line_number":7,"context_line":"making it difficult to follow a request as it traverses multiple services (e.g."},{"line_number":8,"context_line":"Nova calling Glance, Keystone, Neutron, and Cinder during instance creation)."},{"line_number":9,"context_line":""},{"line_number":10,"context_line":"OpenTelemetry (OTEL) is the industry-standard, vendor-neutral framework for"},{"line_number":11,"context_line":"distributed tracing, supported by the CNCF. By adding lightweight OTEL tracing"}],"source_content_type":"text/x-rst","patch_set":1,"id":"11cf0ebf_dae0790a","line":8,"range":{"start_line":4,"start_character":1,"end_line":8,"end_character":77},"in_reply_to":"00e37b06_29449695","updated":"2026-03-30 20:10:16.000000000","message":"this is partly dicsed below but there is no dicussion of how inter serivice porpaation is exected to work adn intra service\n\n\ni.e. takeing a vm snapshot `openstack service image create`\n\n\ningernal the flow is like this at a high level\n\nosc has to first get a token form keystoen then it uses that token to call nova\nnova intrally does an rpc to the compute agent which will do the snapshot and call galnce to upload the imaage\n\n\ntoday we use the same request-id all the way form keyteon to nova to glance\nthat is transproted all the way form teh api header to the compute agent\nvia the context sent of rpc and then the same request id is used when calling glance\n\nso i woudl expect we woudl need to do something simiaar for the OTEL headers?","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8180229da07c63f5455cca6c795f53177e7ee42f","unresolved":true,"context_lines":[{"line_number":14,"context_line":"requiring any changes to deployment tooling or external dependencies beyond the"},{"line_number":15,"context_line":"OpenTelemetry SDK."},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"The proposed approach adds a configurable WSGI middleware to each service that:"},{"line_number":18,"context_line":""},{"line_number":19,"context_line":"* Creates a span for each incoming HTTP request, capturing method, URL, status"},{"line_number":20,"context_line":"  code, and other standard HTTP attributes."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9115bed6_f57a4b1e","line":17,"range":{"start_line":17,"start_character":29,"end_line":17,"end_character":41},"updated":"2026-03-30 18:13:59.000000000","message":"not all service support middelware confguretion\n\n\nso if this is configurble or supproted liekly shoudl be a per project decsion.\n\npast-deploy and api-paste.ini is semi defunct at this poitn so we shoudl be very carful with assuming othat mechanisuem will eb suprpoted in teh long term across all project.\n\ni woudl liek teh exact mahcnis of if/how the middelware is confiugrable to project\n\nsome might use api-paste.ini other might have have a cofnig option via oslo.cofnig in there api service ectra.\noters may not allow configurableity at all and eitehr supprot this alwasy or never.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8180229da07c63f5455cca6c795f53177e7ee42f","unresolved":true,"context_lines":[{"line_number":21,"context_line":"* Propagates trace context across services using the W3C TraceContext standard"},{"line_number":22,"context_line":"  (``traceparent`` / ``tracestate`` headers), enabling end-to-end distributed"},{"line_number":23,"context_line":"  traces."},{"line_number":24,"context_line":"* Exports spans via OTLP (OpenTelemetry Protocol) to any compatible backend"},{"line_number":25,"context_line":"  (Jaeger, Zipkin, Grafana Tempo, etc.)."},{"line_number":26,"context_line":"* Is fully disabled by default and can be enabled via oslo.config with zero"},{"line_number":27,"context_line":"  performance impact when disabled."},{"line_number":28,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"3d3bed1c_f9c9aad2","line":25,"range":{"start_line":24,"start_character":2,"end_line":25,"end_character":40},"updated":"2026-03-30 18:13:59.000000000","message":"this so me i think is soemwhat problmetaic unless that is doen entirly by the middleware via its own oslo.cofnig integration similar to how os-profile is integrated.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8be16d7badff263b11b70e6314adb962c40a8a60","unresolved":true,"context_lines":[{"line_number":21,"context_line":"* Propagates trace context across services using the W3C TraceContext standard"},{"line_number":22,"context_line":"  (``traceparent`` / ``tracestate`` headers), enabling end-to-end distributed"},{"line_number":23,"context_line":"  traces."},{"line_number":24,"context_line":"* Exports spans via OTLP (OpenTelemetry Protocol) to any compatible backend"},{"line_number":25,"context_line":"  (Jaeger, Zipkin, Grafana Tempo, etc.)."},{"line_number":26,"context_line":"* Is fully disabled by default and can be enabled via oslo.config with zero"},{"line_number":27,"context_line":"  performance impact when disabled."},{"line_number":28,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"029b45a4_10e742ec","line":25,"range":{"start_line":24,"start_character":2,"end_line":25,"end_character":40},"in_reply_to":"3d3bed1c_f9c9aad2","updated":"2026-03-30 20:10:16.000000000","message":"not that we will need to select a backend and devleop a devstack plugin and enable the refence backend in ci as part of this effort.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8180229da07c63f5455cca6c795f53177e7ee42f","unresolved":true,"context_lines":[{"line_number":24,"context_line":"* Exports spans via OTLP (OpenTelemetry Protocol) to any compatible backend"},{"line_number":25,"context_line":"  (Jaeger, Zipkin, Grafana Tempo, etc.)."},{"line_number":26,"context_line":"* Is fully disabled by default and can be enabled via oslo.config with zero"},{"line_number":27,"context_line":"  performance impact when disabled."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"This work complements the existing osprofiler library, which provides on-demand"},{"line_number":30,"context_line":"deep profiling (SQL queries, RPC calls) but requires explicit per-request"}],"source_content_type":"text/x-rst","patch_set":1,"id":"139e6fe6_bbf5eb35","line":27,"updated":"2026-03-30 18:13:59.000000000","message":"ack so this is following the os-profiler pattern it might help to make refence to that.\n\nhttps://github.com/openstack/osprofiler/blob/master/osprofiler/opts.py\n\nfor service that dont supprot configurable middelware it hink we woudl want to use this as the way to enabel it\n\nthis rasses another poirnt\n\nwe areadly have 2 tracing mechanisums today\n\nwe have request-id headars\nand we have osprofiler\n\ncoudl OTEL supprot just eb added to soprofiler so ww eget this capablity for free in all services that already suprpot it?\n\nthat feels like a beter way to do this integration since osprofiler is already supproted in a number of projects.\n\nif we coudl simple enable OTEL mode that woudl get you a long way to supproting it in many services.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8180229da07c63f5455cca6c795f53177e7ee42f","unresolved":true,"context_lines":[{"line_number":29,"context_line":"This work complements the existing osprofiler library, which provides on-demand"},{"line_number":30,"context_line":"deep profiling (SQL queries, RPC calls) but requires explicit per-request"},{"line_number":31,"context_line":"activation. OTEL tracing is designed for always-on observability of production"},{"line_number":32,"context_line":"traffic."},{"line_number":33,"context_line":""},{"line_number":34,"context_line":"A proof-of-concept has been implemented and benchmarked on a devstack"},{"line_number":35,"context_line":"environment across five services (Glance, Nova, Cinder, Neutron, Keystone),"}],"source_content_type":"text/x-rst","patch_set":1,"id":"aa54bc08_c2fb9ca2","line":32,"updated":"2026-03-30 18:13:59.000000000","message":"hum im not sure i agree. i think osprofiler coudl evlvoe to provide OTEL suppor without the \"heavy\" tracig by default so that it woudl provei alwasy-on observablity.\n\nbasiclly in otel mode it woudl jsut provide the alwasy on capabliteis and then if allowed in the cofnig you coudl also supprot the deep profiling on each request if enabled and requested.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8be16d7badff263b11b70e6314adb962c40a8a60","unresolved":true,"context_lines":[{"line_number":29,"context_line":"This work complements the existing osprofiler library, which provides on-demand"},{"line_number":30,"context_line":"deep profiling (SQL queries, RPC calls) but requires explicit per-request"},{"line_number":31,"context_line":"activation. OTEL tracing is designed for always-on observability of production"},{"line_number":32,"context_line":"traffic."},{"line_number":33,"context_line":""},{"line_number":34,"context_line":"A proof-of-concept has been implemented and benchmarked on a devstack"},{"line_number":35,"context_line":"environment across five services (Glance, Nova, Cinder, Neutron, Keystone),"}],"source_content_type":"text/x-rst","patch_set":1,"id":"44fa98a3_529469cc","line":32,"in_reply_to":"aa54bc08_c2fb9ca2","updated":"2026-03-30 20:10:16.000000000","message":"thinking about this a littel more i think oslo.middleware may be more correct then osprofiler or a future oslo.wsgi but this is definetly an open that we need to resolve i think before adopting this goal.\n\neven if the mechaniums are similar tracing and profileign have diffent intent so perhaps mixing both in osprofiler is not ideal espcially fi there is a delta in the security or performance of both.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8180229da07c63f5455cca6c795f53177e7ee42f","unresolved":true,"context_lines":[{"line_number":32,"context_line":"traffic."},{"line_number":33,"context_line":""},{"line_number":34,"context_line":"A proof-of-concept has been implemented and benchmarked on a devstack"},{"line_number":35,"context_line":"environment across five services (Glance, Nova, Cinder, Neutron, Keystone),"},{"line_number":36,"context_line":"confirming that the overhead is minimal and within normal latency variance for"},{"line_number":37,"context_line":"most services."},{"line_number":38,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"3d2df8c6_c540dd9c","line":35,"range":{"start_line":35,"start_character":19,"end_line":35,"end_character":33},"updated":"2026-03-30 18:13:59.000000000","message":"so even if the comunity goal was adopted you woudl have to follow those services spec/feature process to agree teh desgin and timeline so just notight that a comunity goal does not premetn the normal feature propal process.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8180229da07c63f5455cca6c795f53177e7ee42f","unresolved":true,"context_lines":[{"line_number":44,"context_line":"Gerrit Topic"},{"line_number":45,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"To facilitate tracking, commits related to this goal should use the"},{"line_number":48,"context_line":"gerrit topic::"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"  opentelemetry-tracing"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"Completion Criteria"},{"line_number":53,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ab71b9bb_2c7b6876","line":50,"range":{"start_line":47,"start_character":0,"end_line":50,"end_character":23},"updated":"2026-03-30 18:13:59.000000000","message":"there is a propasl to discontinue teh use fo topic for goals and replace them with the hashtag feature so if that is adopted this shoudl be updated.\n\ni.e. also note that may project ahve conventions for how topcis/branches shoudl be set\nsepcific bp/\u003cbluepirnt name\u003e for feature like this and usign a dedicated topci breaks that. \n\nif we do move to hastags we can properly track it with the project convetion and the goal convetion but in teh past it has been very annorying when folk dont follow the project convetion or worse chagne the review topic without talking to the person who wrote the patches so i would wonder if we should remove this section form the template entirly?","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8be16d7badff263b11b70e6314adb962c40a8a60","unresolved":true,"context_lines":[{"line_number":44,"context_line":"Gerrit Topic"},{"line_number":45,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"To facilitate tracking, commits related to this goal should use the"},{"line_number":48,"context_line":"gerrit topic::"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"  opentelemetry-tracing"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"Completion Criteria"},{"line_number":53,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"877d1b1e_929ea857","line":50,"range":{"start_line":47,"start_character":0,"end_line":50,"end_character":23},"in_reply_to":"ab71b9bb_2c7b6876","updated":"2026-03-30 20:10:16.000000000","message":"im refering to https://review.opendev.org/c/openstack/governance/+/981832\n\ni guess we can let that resolve independently\n\nyou didn\u0027t actually use this topic for this review and i don\u0027t see a link to\nany of the poc either\n\nhttps://review.opendev.org/q/topic:%22opentelemetry-tracing%22","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8180229da07c63f5455cca6c795f53177e7ee42f","unresolved":true,"context_lines":[{"line_number":52,"context_line":"Completion Criteria"},{"line_number":53,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":54,"context_line":""},{"line_number":55,"context_line":"#. All official OpenStack service projects that expose an HTTP API should"},{"line_number":56,"context_line":"   provide an OpenTelemetry tracing middleware that can be enabled via"},{"line_number":57,"context_line":"   oslo.config."},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"#. The middleware must use the W3C TraceContext standard for context"},{"line_number":60,"context_line":"   propagation across services."}],"source_content_type":"text/x-rst","patch_set":1,"id":"0660e7a3_d3a7f460","line":57,"range":{"start_line":55,"start_character":0,"end_line":57,"end_character":15},"updated":"2026-03-30 18:13:59.000000000","message":"so i think this is incorrect\n\nthere shoudl be 1 and only one midelware develop as part of https://opendev.org/openstack/oslo.middleware\n\nand all services shoudl reuse it.\n\ni dont belvie that we should develop one per service.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8180229da07c63f5455cca6c795f53177e7ee42f","unresolved":true,"context_lines":[{"line_number":59,"context_line":"#. The middleware must use the W3C TraceContext standard for context"},{"line_number":60,"context_line":"   propagation across services."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"#. The middleware must export spans via OTLP (HTTP or gRPC) to allow"},{"line_number":63,"context_line":"   integration with any OpenTelemetry-compatible backend."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"#. Tracing must be disabled by default. When disabled, the middleware must"}],"source_content_type":"text/x-rst","patch_set":1,"id":"292efc18_ae1e6d8a","line":62,"range":{"start_line":62,"start_character":54,"end_line":62,"end_character":58},"updated":"2026-03-30 18:13:59.000000000","message":"this im not sure i agree with\n\ngRPC will pull in a hard depency on google protocol buffers and the relevent language binding and gRPC also has implciations for request lifetime with websocket/grpc transprots that may or may not play nicely with wsgi implaitons.\n\nany long lived conencton to a tracing backend may have scalablity and uptime implcaitions for the services depending on hwo this works so unless this si operating in a purly stateless push model that could be problematic.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8be16d7badff263b11b70e6314adb962c40a8a60","unresolved":true,"context_lines":[{"line_number":59,"context_line":"#. The middleware must use the W3C TraceContext standard for context"},{"line_number":60,"context_line":"   propagation across services."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"#. The middleware must export spans via OTLP (HTTP or gRPC) to allow"},{"line_number":63,"context_line":"   integration with any OpenTelemetry-compatible backend."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"#. Tracing must be disabled by default. When disabled, the middleware must"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bd1fd4a9_7fa98848","line":62,"range":{"start_line":62,"start_character":54,"end_line":62,"end_character":58},"in_reply_to":"292efc18_ae1e6d8a","updated":"2026-03-30 20:10:16.000000000","message":"by the way if the suggestion is the middelware woudl expore a http/gRPC endpoint that an external service can call i.e. a pull model where somethign external scrapes the endpoint that is also problematic so this need more clarity on what exaction \"integration with any OpenTelemetry-compatible backend\" means.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8be16d7badff263b11b70e6314adb962c40a8a60","unresolved":true,"context_lines":[{"line_number":62,"context_line":"#. The middleware must export spans via OTLP (HTTP or gRPC) to allow"},{"line_number":63,"context_line":"   integration with any OpenTelemetry-compatible backend."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"#. Tracing must be disabled by default. When disabled, the middleware must"},{"line_number":66,"context_line":"   short-circuit with negligible overhead (no OpenTelemetry SDK imports in"},{"line_number":67,"context_line":"   the request path)."},{"line_number":68,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"996b9611_aa52e3fa","line":65,"range":{"start_line":65,"start_character":11,"end_line":65,"end_character":15},"updated":"2026-03-30 20:10:16.000000000","message":"\"must\" is probably overly prescriptive\n\nim not advocating for mandating that it should be enabled by default but im saying this is really something that project shoudl device unless we are using a shared lib.\n\ni agree that it should be disabled by default by the way just this feels like an implementation detail.\n\nfor SRBAC we specified a fased enabling approhc because it not safe to do that incrementally but i dont think that applies to this effort correct?","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8180229da07c63f5455cca6c795f53177e7ee42f","unresolved":true,"context_lines":[{"line_number":70,"context_line":"   ``[tracing]`` group in oslo.config, including at minimum: ``enabled``,"},{"line_number":71,"context_line":"   ``otlp_endpoint``, ``service_name``, and ``sampling_rate``."},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"#. The implementation should be contributed and completed for all services"},{"line_number":74,"context_line":"   by the end of the Hibiscus cycle."},{"line_number":75,"context_line":""},{"line_number":76,"context_line":"Current State / Anticipated Impact"},{"line_number":77,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"08c0255f_04020758","line":74,"range":{"start_line":73,"start_character":2,"end_line":74,"end_character":36},"updated":"2026-03-30 18:13:59.000000000","message":"i dont think that is a realist timelien for this i woudl set 2027.1 as the ealiest target.\n\nwe woudl need to build a concentiosn to do this and liekly discusion it in the upcomeing ptg before commiting to such an agressive timeline.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8180229da07c63f5455cca6c795f53177e7ee42f","unresolved":true,"context_lines":[{"line_number":81,"context_line":"* **Glance**: Paste pipeline filter in ``glance-api-paste.ini``, configuration"},{"line_number":82,"context_line":"  in ``glance/common/tracing.py``, middleware in"},{"line_number":83,"context_line":"  ``glance/api/middleware/tracing.py``."},{"line_number":84,"context_line":"* **Nova**: Paste pipeline filter in ``api-paste.ini``, configuration in"},{"line_number":85,"context_line":"  ``nova/common/tracing.py``, middleware in ``nova/api/middleware/tracing.py``."},{"line_number":86,"context_line":"* **Cinder**: Paste pipeline filter in ``api-paste.ini``, configuration in"},{"line_number":87,"context_line":"  ``cinder/common/tracing.py``, middleware in"},{"line_number":88,"context_line":"  ``cinder/api/middleware/tracing.py``."}],"source_content_type":"text/x-rst","patch_set":1,"id":"a13ee7de_628f257c","line":85,"range":{"start_line":84,"start_character":1,"end_line":85,"end_character":77},"updated":"2026-03-30 18:13:59.000000000","message":"-1\n\nagain we coudl do this but unless there is a very stong reason not to put it in \n\nhttps://opendev.org/openstack/oslo.middleware\nor \nhttps://github.com/openstack/osprofiler \n\ni think we shoudl not need per service version fo this.\n\nor first approch shoudl be to have a common moduel as aprt of oslo.\n\n\nby the way if the tracing is to work across servic ealls you woudl also need supprot in the openstacksdk for passing the relvent header the same way we propagate the requqstid.\n\nmay project specific cleine are frozen already which mean for tracing to work between nova and nueton for example we woudl need to complete the adotion of the SDk for talkign to neutron in nova and update the sdk to supprot passing the relvent headers.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"22c852dc40702592aa3a6573b4cb99dbac44f8c9","unresolved":true,"context_lines":[{"line_number":81,"context_line":"* **Glance**: Paste pipeline filter in ``glance-api-paste.ini``, configuration"},{"line_number":82,"context_line":"  in ``glance/common/tracing.py``, middleware in"},{"line_number":83,"context_line":"  ``glance/api/middleware/tracing.py``."},{"line_number":84,"context_line":"* **Nova**: Paste pipeline filter in ``api-paste.ini``, configuration in"},{"line_number":85,"context_line":"  ``nova/common/tracing.py``, middleware in ``nova/api/middleware/tracing.py``."},{"line_number":86,"context_line":"* **Cinder**: Paste pipeline filter in ``api-paste.ini``, configuration in"},{"line_number":87,"context_line":"  ``cinder/common/tracing.py``, middleware in"},{"line_number":88,"context_line":"  ``cinder/api/middleware/tracing.py``."}],"source_content_type":"text/x-rst","patch_set":1,"id":"5c691a16_061749f1","line":85,"range":{"start_line":84,"start_character":1,"end_line":85,"end_character":77},"in_reply_to":"a13ee7de_628f257c","updated":"2026-03-30 19:01:22.000000000","message":"by the way the there as an unnoffical config centralisation community goal/initiv in teh past to unify where and now service config regeistration is done that resulted in most service storing there config in \u003cservice\u003e/conf\n\nhttps://github.com/openstack/cyborg/tree/master/cyborg/conf\nhttps://github.com/openstack/watcher/tree/master/watcher/conf\nhttps://github.com/openstack/nova/tree/master/nova/conf\nhttps://github.com/openstack/placement/tree/master/placement/conf\nhttps://github.com/openstack/ironic/tree/master/ironic/conf\nhttps://github.com/openstack/neutron/tree/master/neutron/conf\nhttps://github.com/openstack/keystone/tree/master/keystone/conf\n\nconfig option shoudl not live in \u003cservice\u003e/common/tracing  if we needed to have them per service they shoudl follow the \n\n\nthe centralisation of the config generation \n\nhttps://governance.openstack.org/tc/goals/completed/queens/policy-in-code.html\n\nmost project did a consodlaition fo config option in the conf direfcty around the same tiem as we adotped policy in code as a community if they had not already done so before.\n\nhttps://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/centralize-config-options.html\nhttps://lists.openstack.org/pipermail/openstack-dev/2015-July/070306.html\n\n\nif we were to have in repo mideleware snadardising its localtion like we did for \n\nhttps://governance.openstack.org/tc/goals/completed/2025.2/migrate-from-wsgi-scripts-to-module-paths.html\nto \u003cservice\u003e/middleware/... might make sense but that only really was done in the wsgi case above because we wanted a standard entry proint to be supproted for each service and this middelware would not be a public part of the python interface to teh service\n\nso specifying it feel a littel odd\n\nfor nova we typiclly store the midelware in \n\nhttps://github.com/openstack/nova/tree/master/nova/api\n\nhttps://github.com/openstack/nova/blob/master/nova/api/openstack/requestlog.py\nor  \nhttps://github.com/openstack/nova/blob/master/nova/api/compute_req_id.py\n\nbut we try not to have custom middleware if we can avoid it.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"8180229da07c63f5455cca6c795f53177e7ee42f","unresolved":true,"context_lines":[{"line_number":126,"context_line":"Only Glance showed a measurable overhead (~36 ms per request). For the other"},{"line_number":127,"context_line":"services, the tracing cost was within their natural latency variance."},{"line_number":128,"context_line":""},{"line_number":129,"context_line":"The anticipated impact of this goal is minimal for individual projects. The"},{"line_number":130,"context_line":"middleware implementation is self-contained (two new files per service plus"},{"line_number":131,"context_line":"paste pipeline or Flask registration) and does not affect existing"},{"line_number":132,"context_line":"functionality when disabled. The ``opentelemetry-sdk`` and"}],"source_content_type":"text/x-rst","patch_set":1,"id":"89040b72_4ac89cd1","line":129,"updated":"2026-03-30 18:13:59.000000000","message":"i think its pretty high\n\nnot only does each project need to update there ci jobs to test this tracing end to end they need to adopt/supprot new midelware, change there request context to be ablet to propaget the header and change there rest client calls to propagatge them.\n\nthat not a minimal impact.","commit_id":"8ec1593f8a0497362086172973a309583eb601ce"}]}