)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":32177,"name":"Jacob Anders","email":"jacob-anders-dev@proton.me","username":"janders"},"change_message_id":"bd88c263ebae6309d737f368a4f7fe16ac1146da","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"b7bfabd3_bbbed5ed","updated":"2026-06-10 23:51:46.000000000","message":"Thank you for proposing this, Julia. I find it very interesting and very timely. Bunch of inline comments incoming, most are of food-for-thought kind as opposed to corrections.\n\nGeneral hint: perhaps it\u0027s worth splitting it into smaller specs for reviewability. Perhaps IPA or switch-side bits could be taken out and some of the redundant parts (e.g. list of steps needed for RDMA to work) could be deduped.\n\nThat said great work overall.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":24245,"name":"Harald Jensås","email":"hjensas@redhat.com","username":"harald.jensas"},"change_message_id":"344c16e83fc31206d478d6d5929151797c8f70af","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"f18f4c4b_6b1f712e","updated":"2026-06-15 12:21:01.000000000","message":"Thanks, overall I think this is good. I made some comments inline.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"e887b6f6a9a9defe0b96ba6410273e84de80cff4","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":9,"id":"42ab90dd_6f46fa48","updated":"2026-07-21 15:51:08.000000000","message":"Group review outcome: Nobody screamed. A question if we should have a separate binding information field came up, but consensus seemed to reveal that maybe that was \"overkill\" or \"redundant\"\n\nDmitry screams \"why is this so complicated! Give me my pigeon!\"","commit_id":"37463258819ff3343ac1c7b3b482691dc5ad89ba"}],"specs/approved/rdma-converged-networking-support.rst":[{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"e887b6f6a9a9defe0b96ba6410273e84de80cff4","unresolved":true,"context_lines":[{"line_number":309,"context_line":"    openstack network set rdma-net --mtu 9000"},{"line_number":310,"context_line":""},{"line_number":311,"context_line":"    # Tag for trait-based networking matching"},{"line_number":312,"context_line":"    openstack network set rdma-net --tag rdma"},{"line_number":313,"context_line":""},{"line_number":314,"context_line":"    # RDMA subnet: static IPs, no DHCP, no gateway"},{"line_number":315,"context_line":"    openstack subnet create rdma-subnet \\"}],"source_content_type":"text/x-rst","patch_set":9,"id":"3211a4e3_ea02c237","line":312,"updated":"2026-07-21 15:51:08.000000000","message":"The other question is how does anyone *ensure* this gets bound to a specific network fabric. This also overlaps into \"I want all my storage operations on the purple network fabric\" because there really is no way to force-weight short of the actual physical interface binding on nodes.","commit_id":"37463258819ff3343ac1c7b3b482691dc5ad89ba"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"e887b6f6a9a9defe0b96ba6410273e84de80cff4","unresolved":true,"context_lines":[{"line_number":315,"context_line":"    openstack subnet create rdma-subnet \\"},{"line_number":316,"context_line":"      --network rdma-net \\"},{"line_number":317,"context_line":"      --subnet-range 10.0.100.0/24 \\"},{"line_number":318,"context_line":"      --no-dhcp \\"},{"line_number":319,"context_line":"      --no-gateway \\"},{"line_number":320,"context_line":"      --allocation-pool start\u003d10.0.100.10,end\u003d10.0.100.254"},{"line_number":321,"context_line":""},{"line_number":322,"context_line":"**Step 2: Register nodes with RDMA capability**"}],"source_content_type":"text/x-rst","patch_set":9,"id":"ee125181_0aa8475f","line":319,"range":{"start_line":318,"start_character":0,"end_line":319,"end_character":20},"updated":"2026-07-21 15:51:08.000000000","message":"so, one aspect is we will need to reiterate this because gateway placement actually *schedules the network* on the binding operation in neutron. Then this will hash out on the actual bind operation.","commit_id":"37463258819ff3343ac1c7b3b482691dc5ad89ba"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"e887b6f6a9a9defe0b96ba6410273e84de80cff4","unresolved":true,"context_lines":[{"line_number":658,"context_line":"-----------"},{"line_number":659,"context_line":""},{"line_number":660,"context_line":"Primary assignee:"},{"line_number":661,"context_line":"  None"},{"line_number":662,"context_line":""},{"line_number":663,"context_line":"Other contributors:"},{"line_number":664,"context_line":"  None"}],"source_content_type":"text/x-rst","patch_set":9,"id":"9e891bfe_1f112e0a","line":661,"updated":"2026-07-21 15:51:08.000000000","message":"Maybe Julia?","commit_id":"37463258819ff3343ac1c7b3b482691dc5ad89ba"}],"specs/backlog/rdma-converged-networking-support.rst":[{"author":{"_account_id":32177,"name":"Jacob Anders","email":"jacob-anders-dev@proton.me","username":"janders"},"change_message_id":"bd88c263ebae6309d737f368a4f7fe16ac1146da","unresolved":true,"context_lines":[{"line_number":101,"context_line":"The ConnectX adapter must be set to Ethernet mode with RoCEv2"},{"line_number":102,"context_line":"enabled. The Mellanox hardware manager in ironic-python-agent"},{"line_number":103,"context_line":"(IPA) can do this via ``update_nvidia_nic_firmware_settings``"},{"line_number":104,"context_line":"using ``mstconfig``, but this capability is only available as a"},{"line_number":105,"context_line":"*clean step* and *service step*. It is **not registered as a"},{"line_number":106,"context_line":"deploy step**, so it cannot be triggered by a deploy template"},{"line_number":107,"context_line":"matched to a Nova flavor trait. The operator must either"},{"line_number":108,"context_line":"pre-configure every NIC manually or run a cleaning cycle before"},{"line_number":109,"context_line":"every deployment -- neither of which is acceptable at scale."},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"**Gap 2: No reference documentation for RDMA BIOS settings.**"},{"line_number":112,"context_line":""},{"line_number":113,"context_line":"The BIOS interface\u0027s ``apply_configuration`` already works as a"}],"source_content_type":"text/x-rst","patch_set":3,"id":"74dcee4e_c197bbeb","line":110,"range":{"start_line":104,"start_character":25,"end_line":110,"end_character":1},"updated":"2026-06-10 23:51:46.000000000","message":"totally valid concern - yet I wonder, would the operator want to use RDMA-enabled hardware for non-RDMA enabled workloads?\nIf not - perhaps this clean step could be ran once and RDMA-friendly settings could persist across workloads/tenants in the first iteration(s) of the system?","commit_id":"8d8c2cbd4eb52ab7b131ef5ecf1de469bf813cc5"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"62ee0f73389ddd0407135710fafc2a33eca96cfc","unresolved":false,"context_lines":[{"line_number":101,"context_line":"The ConnectX adapter must be set to Ethernet mode with RoCEv2"},{"line_number":102,"context_line":"enabled. The Mellanox hardware manager in ironic-python-agent"},{"line_number":103,"context_line":"(IPA) can do this via ``update_nvidia_nic_firmware_settings``"},{"line_number":104,"context_line":"using ``mstconfig``, but this capability is only available as a"},{"line_number":105,"context_line":"*clean step* and *service step*. It is **not registered as a"},{"line_number":106,"context_line":"deploy step**, so it cannot be triggered by a deploy template"},{"line_number":107,"context_line":"matched to a Nova flavor trait. The operator must either"},{"line_number":108,"context_line":"pre-configure every NIC manually or run a cleaning cycle before"},{"line_number":109,"context_line":"every deployment -- neither of which is acceptable at scale."},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"**Gap 2: No reference documentation for RDMA BIOS settings.**"},{"line_number":112,"context_line":""},{"line_number":113,"context_line":"The BIOS interface\u0027s ``apply_configuration`` already works as a"}],"source_content_type":"text/x-rst","patch_set":3,"id":"4676329f_bfa00028","line":110,"range":{"start_line":104,"start_character":25,"end_line":110,"end_character":1},"in_reply_to":"74dcee4e_c197bbeb","updated":"2026-06-11 18:50:47.000000000","message":"Would an user operator want to, likely or they may just not care in the slightest and just want the TPU or GPU in the host.\n\nBut the infrastructure operator can manage that with traits, since in the OpenStack context traits are admin role controlled.\n\nYes, you can have tenants with independent admins, but then they are operators of the infrastructure ... or should instead be a \"manager\".\n\nJust thinking from a larger standpoint, there may be a concern of entirely unwinding an RMDA specific setting on a card. I\u0027ll add a todo to at least keep it in mind.\n\nIn terms of the structure and the doc, It is already labeled as a deploy step. I guess claude missed it when I asked for it to update. :(\n\nI\u0027m going to mark this one as resolved.","commit_id":"8d8c2cbd4eb52ab7b131ef5ecf1de469bf813cc5"},{"author":{"_account_id":32177,"name":"Jacob Anders","email":"jacob-anders-dev@proton.me","username":"janders"},"change_message_id":"bd88c263ebae6309d737f368a4f7fe16ac1146da","unresolved":true,"context_lines":[{"line_number":112,"context_line":""},{"line_number":113,"context_line":"The BIOS interface\u0027s ``apply_configuration`` already works as a"},{"line_number":114,"context_line":"deploy step. However, RDMA-optimized deployments require"},{"line_number":115,"context_line":"specific BIOS settings (SR-IOV enablement, IOMMU, power"},{"line_number":116,"context_line":"management tuning) whose attribute names vary by server vendor"},{"line_number":117,"context_line":"(Dell, HPE, Cisco UCS, Supermicro). Without reference"},{"line_number":118,"context_line":"documentation, every operator must reverse-engineer these names"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5ec7f91e_1a4a2a5a","line":115,"range":{"start_line":115,"start_character":24,"end_line":115,"end_character":48},"updated":"2026-06-10 23:51:46.000000000","message":"do we need SRIOV and IOMMU for baremetal ROCE? I doubt SRIOV should be needed, less confident about IOMMU. If ROCE in VMs is needed, VT would likely be needed also.\nAgreed regarding performance-related settings","commit_id":"8d8c2cbd4eb52ab7b131ef5ecf1de469bf813cc5"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"62ee0f73389ddd0407135710fafc2a33eca96cfc","unresolved":false,"context_lines":[{"line_number":112,"context_line":""},{"line_number":113,"context_line":"The BIOS interface\u0027s ``apply_configuration`` already works as a"},{"line_number":114,"context_line":"deploy step. However, RDMA-optimized deployments require"},{"line_number":115,"context_line":"specific BIOS settings (SR-IOV enablement, IOMMU, power"},{"line_number":116,"context_line":"management tuning) whose attribute names vary by server vendor"},{"line_number":117,"context_line":"(Dell, HPE, Cisco UCS, Supermicro). Without reference"},{"line_number":118,"context_line":"documentation, every operator must reverse-engineer these names"}],"source_content_type":"text/x-rst","patch_set":3,"id":"615a9af6_8c4fd5bf","line":115,"range":{"start_line":115,"start_character":24,"end_line":115,"end_character":48},"in_reply_to":"5ec7f91e_1a4a2a5a","updated":"2026-06-11 18:50:47.000000000","message":"They would be needed in virtualization cases, as in if the host is used as a hypervisor. That is mainly in regards to SRIOV, somewhat in regards to IOMMU, however that can also be a security guard from what I understand so some operators may need to enable or disable it. I\u0027ve dialed this back in the text because it is *entirely* going to be based upon the operator preferences/needs.","commit_id":"8d8c2cbd4eb52ab7b131ef5ecf1de469bf813cc5"},{"author":{"_account_id":32177,"name":"Jacob Anders","email":"jacob-anders-dev@proton.me","username":"janders"},"change_message_id":"bd88c263ebae6309d737f368a4f7fe16ac1146da","unresolved":true,"context_lines":[{"line_number":216,"context_line":"The RDMA NIC is placed on a NUMA node local to the CPUs that"},{"line_number":217,"context_line":"will run the RDMA workload, ensuring the lowest latency and"},{"line_number":218,"context_line":"maximum throughput."},{"line_number":219,"context_line":""},{"line_number":220,"context_line":"**The RDMA and IP switch fabrics may be the same or separate.**"},{"line_number":221,"context_line":"Ideally the RDMA traffic runs on its own dedicated switches"},{"line_number":222,"context_line":"(or at minimum its own set of switch ports with a dedicated"},{"line_number":223,"context_line":"lossless QoS policy). The lossless behavior required by RoCEv2"}],"source_content_type":"text/x-rst","patch_set":4,"id":"4922b96d_b50c69ec","line":220,"range":{"start_line":219,"start_character":1,"end_line":220,"end_character":63},"updated":"2026-06-10 23:51:46.000000000","message":"nit: technically IP is used on the RDMA fabric, too. I couldn\u0027t come up with a better way of expressing this in reasonable time so I asked Claude, here\u0027s some suggestions, feel free to adopt or disregard.\n\nIf you want to tighten it, consider:\n\n\"The RDMA and general-purpose IP fabrics may be the same or separate.\"\n\nor purpose-oriented:\n\n\"The RDMA fabric and the management/provisioning network may be the same or separate.\"\n\nThat said, for the target audience (infra engineers deploying Ironic at scale), \"RDMA and IP fabrics\" is standard data center shorthand and unlikely to cause real confusion.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"62ee0f73389ddd0407135710fafc2a33eca96cfc","unresolved":true,"context_lines":[{"line_number":216,"context_line":"The RDMA NIC is placed on a NUMA node local to the CPUs that"},{"line_number":217,"context_line":"will run the RDMA workload, ensuring the lowest latency and"},{"line_number":218,"context_line":"maximum throughput."},{"line_number":219,"context_line":""},{"line_number":220,"context_line":"**The RDMA and IP switch fabrics may be the same or separate.**"},{"line_number":221,"context_line":"Ideally the RDMA traffic runs on its own dedicated switches"},{"line_number":222,"context_line":"(or at minimum its own set of switch ports with a dedicated"},{"line_number":223,"context_line":"lossless QoS policy). The lossless behavior required by RoCEv2"}],"source_content_type":"text/x-rst","patch_set":4,"id":"b9d39057_e79a6eac","line":220,"range":{"start_line":219,"start_character":1,"end_line":220,"end_character":63},"in_reply_to":"4922b96d_b50c69ec","updated":"2026-06-11 18:50:47.000000000","message":"This was more geared at switch fabrics in general. I\u0027ve revised the text accordingly.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":32177,"name":"Jacob Anders","email":"jacob-anders-dev@proton.me","username":"janders"},"change_message_id":"bd88c263ebae6309d737f368a4f7fe16ac1146da","unresolved":true,"context_lines":[{"line_number":241,"context_line":"it crosses the spine to reach another leaf. PFC is an L2"},{"line_number":242,"context_line":"mechanism that operates hop-by-hop -- it protects the access"},{"line_number":243,"context_line":"port link between the server and the leaf switch, but it does"},{"line_number":244,"context_line":"not extend through the VXLAN tunnel across the spine. This"},{"line_number":245,"context_line":"means that when two RDMA nodes are on different leaf"},{"line_number":246,"context_line":"switches, the VXLAN segment between leaves is **not"},{"line_number":247,"context_line":"inherently lossless**, and a congested spine link can drop"},{"line_number":248,"context_line":"RDMA packets, causing RoCE performance degradation."},{"line_number":249,"context_line":""},{"line_number":250,"context_line":"Operators should address this risk through one or more of"},{"line_number":251,"context_line":"the following strategies:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"d230002e_28e9ed0a","line":248,"range":{"start_line":244,"start_character":54,"end_line":248,"end_character":51},"updated":"2026-06-10 23:51:46.000000000","message":"General comment: this is the entire reason why RDMA over InfiniBand can be significantly superior to RDMA over Converged Ethernet (RoCE) in one sentence.\n\nIf you want it may be worth highlighting this specifically.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"62ee0f73389ddd0407135710fafc2a33eca96cfc","unresolved":false,"context_lines":[{"line_number":241,"context_line":"it crosses the spine to reach another leaf. PFC is an L2"},{"line_number":242,"context_line":"mechanism that operates hop-by-hop -- it protects the access"},{"line_number":243,"context_line":"port link between the server and the leaf switch, but it does"},{"line_number":244,"context_line":"not extend through the VXLAN tunnel across the spine. This"},{"line_number":245,"context_line":"means that when two RDMA nodes are on different leaf"},{"line_number":246,"context_line":"switches, the VXLAN segment between leaves is **not"},{"line_number":247,"context_line":"inherently lossless**, and a congested spine link can drop"},{"line_number":248,"context_line":"RDMA packets, causing RoCE performance degradation."},{"line_number":249,"context_line":""},{"line_number":250,"context_line":"Operators should address this risk through one or more of"},{"line_number":251,"context_line":"the following strategies:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"79d17109_b83a85d0","line":248,"range":{"start_line":244,"start_character":54,"end_line":248,"end_character":51},"in_reply_to":"d230002e_28e9ed0a","updated":"2026-06-11 18:50:47.000000000","message":"Done","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":32177,"name":"Jacob Anders","email":"jacob-anders-dev@proton.me","username":"janders"},"change_message_id":"bd88c263ebae6309d737f368a4f7fe16ac1146da","unresolved":true,"context_lines":[{"line_number":250,"context_line":"Operators should address this risk through one or more of"},{"line_number":251,"context_line":"the following strategies:"},{"line_number":252,"context_line":""},{"line_number":253,"context_line":"* **Co-locate RDMA nodes on the same leaf switch.** When"},{"line_number":254,"context_line":"  RDMA nodes are on the same leaf (or leaf pair), traffic"},{"line_number":255,"context_line":"  stays within the switch as a local VLAN and PFC covers"},{"line_number":256,"context_line":"  the entire path end-to-end. This eliminates the VXLAN"}],"source_content_type":"text/x-rst","patch_set":4,"id":"2871f79f_97f232ca","line":253,"range":{"start_line":253,"start_character":0,"end_line":253,"end_character":49},"updated":"2026-06-10 23:51:46.000000000","message":"further to my last comment - in this scenario RoCE can near-match Infinband in performance (if scale requirements allow it). It\u0027s quite an interesting area because if this is practical, operator can relatively easily achieve IB-grade performance with Ethernet-grade interop.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"62ee0f73389ddd0407135710fafc2a33eca96cfc","unresolved":true,"context_lines":[{"line_number":250,"context_line":"Operators should address this risk through one or more of"},{"line_number":251,"context_line":"the following strategies:"},{"line_number":252,"context_line":""},{"line_number":253,"context_line":"* **Co-locate RDMA nodes on the same leaf switch.** When"},{"line_number":254,"context_line":"  RDMA nodes are on the same leaf (or leaf pair), traffic"},{"line_number":255,"context_line":"  stays within the switch as a local VLAN and PFC covers"},{"line_number":256,"context_line":"  the entire path end-to-end. This eliminates the VXLAN"}],"source_content_type":"text/x-rst","patch_set":4,"id":"37e2565b_1a5219b1","line":253,"range":{"start_line":253,"start_character":0,"end_line":253,"end_character":49},"in_reply_to":"2871f79f_97f232ca","updated":"2026-06-11 18:50:47.000000000","message":"Agree, but its a trade-off between perfect design versus flexible design. The key is to make it an informed decision.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":32177,"name":"Jacob Anders","email":"jacob-anders-dev@proton.me","username":"janders"},"change_message_id":"bd88c263ebae6309d737f368a4f7fe16ac1146da","unresolved":true,"context_lines":[{"line_number":265,"context_line":"  nodes whose inventory structure reflects shared rack"},{"line_number":266,"context_line":"  or switch topology."},{"line_number":267,"context_line":""},{"line_number":268,"context_line":"* **Configure the entire RDMA underlay for lossless"},{"line_number":269,"context_line":"  behavior.** If RDMA nodes must span multiple leaf"},{"line_number":270,"context_line":"  switches, operators should configure PFC and ECN on all"},{"line_number":271,"context_line":"  spine and inter-switch links in the RDMA fabric, and"},{"line_number":272,"context_line":"  ensure that leaf VTEPs preserve the DSCP marking from"},{"line_number":273,"context_line":"  the inner RoCE header to the outer VXLAN header. This"},{"line_number":274,"context_line":"  is supported by modern EVPN-VXLAN fabrics (Cisco,"},{"line_number":275,"context_line":"  Arista, Nvidia, and others document \"RoCEv2 over"},{"line_number":276,"context_line":"  EVPN-VXLAN\" configurations) but is a fabric-wide QoS"},{"line_number":277,"context_line":"  design effort beyond access-port configuration."},{"line_number":278,"context_line":""},{"line_number":279,"context_line":"* **Monitor spine link utilization.** Regardless of"},{"line_number":280,"context_line":"  topology, operators should monitor spine link"},{"line_number":281,"context_line":"  utilization and congestion metrics (ECN marks, PFC"},{"line_number":282,"context_line":"  pause counts, buffer utilization) on the RDMA fabric."},{"line_number":283,"context_line":"  Congestion on the VXLAN underlay is the primary risk"},{"line_number":284,"context_line":"  to RDMA performance in multi-leaf deployments."},{"line_number":285,"context_line":""},{"line_number":286,"context_line":"**MTU considerations with VXLAN underlay.** VXLAN"},{"line_number":287,"context_line":"encapsulation adds approximately 50 bytes of overhead"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1fdb40d6_545031e9","line":284,"range":{"start_line":268,"start_character":0,"end_line":284,"end_character":48},"updated":"2026-06-10 23:51:46.000000000","message":"General comment: To some degree this is practical, but at a certain point (operator wishing to have fully-non blocking - think fat tree topology - or 2:1 blocking ratio) I am not sure what\u0027s possible with Ethernet and any constraints there may be on numbers of ports per aggregation group or number of redundant paths through different spine switches.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"62ee0f73389ddd0407135710fafc2a33eca96cfc","unresolved":true,"context_lines":[{"line_number":265,"context_line":"  nodes whose inventory structure reflects shared rack"},{"line_number":266,"context_line":"  or switch topology."},{"line_number":267,"context_line":""},{"line_number":268,"context_line":"* **Configure the entire RDMA underlay for lossless"},{"line_number":269,"context_line":"  behavior.** If RDMA nodes must span multiple leaf"},{"line_number":270,"context_line":"  switches, operators should configure PFC and ECN on all"},{"line_number":271,"context_line":"  spine and inter-switch links in the RDMA fabric, and"},{"line_number":272,"context_line":"  ensure that leaf VTEPs preserve the DSCP marking from"},{"line_number":273,"context_line":"  the inner RoCE header to the outer VXLAN header. This"},{"line_number":274,"context_line":"  is supported by modern EVPN-VXLAN fabrics (Cisco,"},{"line_number":275,"context_line":"  Arista, Nvidia, and others document \"RoCEv2 over"},{"line_number":276,"context_line":"  EVPN-VXLAN\" configurations) but is a fabric-wide QoS"},{"line_number":277,"context_line":"  design effort beyond access-port configuration."},{"line_number":278,"context_line":""},{"line_number":279,"context_line":"* **Monitor spine link utilization.** Regardless of"},{"line_number":280,"context_line":"  topology, operators should monitor spine link"},{"line_number":281,"context_line":"  utilization and congestion metrics (ECN marks, PFC"},{"line_number":282,"context_line":"  pause counts, buffer utilization) on the RDMA fabric."},{"line_number":283,"context_line":"  Congestion on the VXLAN underlay is the primary risk"},{"line_number":284,"context_line":"  to RDMA performance in multi-leaf deployments."},{"line_number":285,"context_line":""},{"line_number":286,"context_line":"**MTU considerations with VXLAN underlay.** VXLAN"},{"line_number":287,"context_line":"encapsulation adds approximately 50 bytes of overhead"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9f84a84f_ad1453a4","line":284,"range":{"start_line":268,"start_character":0,"end_line":284,"end_character":48},"in_reply_to":"1fdb40d6_545031e9","updated":"2026-06-11 18:50:47.000000000","message":"Yeah, I think the challenge is we have to make it as much of an informed decision as possible. For what it is worth, I\u0027m enhancing the paragraph a little bit more to stress the fact that an alternative model may be needed.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":32177,"name":"Jacob Anders","email":"jacob-anders-dev@proton.me","username":"janders"},"change_message_id":"bd88c263ebae6309d737f368a4f7fe16ac1146da","unresolved":true,"context_lines":[{"line_number":417,"context_line":"  deployment described in this specification."},{"line_number":418,"context_line":""},{"line_number":419,"context_line":".. _Change 4\\: Diskimage-Builder RDMA Element:"},{"line_number":420,"context_line":""},{"line_number":421,"context_line":"Change 4: Diskimage-Builder RDMA Element"},{"line_number":422,"context_line":"-----------------------------------------"},{"line_number":423,"context_line":""},{"line_number":424,"context_line":"**What**: Create a new ``rdma-roce`` element in"}],"source_content_type":"text/x-rst","patch_set":4,"id":"a4559d96_ef002504","line":421,"range":{"start_line":420,"start_character":1,"end_line":421,"end_character":40},"updated":"2026-06-10 23:51:46.000000000","message":"below looks about right - will certainly need a test at some point but this is a great start","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"62ee0f73389ddd0407135710fafc2a33eca96cfc","unresolved":true,"context_lines":[{"line_number":417,"context_line":"  deployment described in this specification."},{"line_number":418,"context_line":""},{"line_number":419,"context_line":".. _Change 4\\: Diskimage-Builder RDMA Element:"},{"line_number":420,"context_line":""},{"line_number":421,"context_line":"Change 4: Diskimage-Builder RDMA Element"},{"line_number":422,"context_line":"-----------------------------------------"},{"line_number":423,"context_line":""},{"line_number":424,"context_line":"**What**: Create a new ``rdma-roce`` element in"}],"source_content_type":"text/x-rst","patch_set":4,"id":"6e3cfb78_b3bb54c5","line":421,"range":{"start_line":420,"start_character":1,"end_line":421,"end_character":40},"in_reply_to":"a4559d96_ef002504","updated":"2026-06-11 18:50:47.000000000","message":"Yeah, we likely shouldn\u0027t block on merging, but yeah.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":32177,"name":"Jacob Anders","email":"jacob-anders-dev@proton.me","username":"janders"},"change_message_id":"bd88c263ebae6309d737f368a4f7fe16ac1146da","unresolved":true,"context_lines":[{"line_number":639,"context_line":"  classes. Storm control rate-limits broadcast/multicast"},{"line_number":640,"context_line":"  traffic, which can interfere with PFC pause frames."},{"line_number":641,"context_line":""},{"line_number":642,"context_line":"**Current state**: All of these settings must be manually"},{"line_number":643,"context_line":"pre-configured on every switch port that will carry RDMA"},{"line_number":644,"context_line":"traffic. This is acceptable for static environments where"},{"line_number":645,"context_line":"RDMA ports are known in advance, but it breaks down when"},{"line_number":646,"context_line":"nodes are dynamically provisioned -- a switch port that"},{"line_number":647,"context_line":"carried standard IP traffic yesterday may need lossless"},{"line_number":648,"context_line":"configuration today because Ironic scheduled an RDMA"},{"line_number":649,"context_line":"workload to that node."},{"line_number":650,"context_line":""},{"line_number":651,"context_line":"**Proposed approach**: This is a significant body of work"},{"line_number":652,"context_line":"that requires its own specification in the"},{"line_number":653,"context_line":"``networking-generic-switch`` project. The high-level"},{"line_number":654,"context_line":"approach would be:"},{"line_number":655,"context_line":""},{"line_number":656,"context_line":"1. **Extend the switch device driver interface** in"},{"line_number":657,"context_line":"   ``networking-generic-switch`` with new command templates"}],"source_content_type":"text/x-rst","patch_set":4,"id":"c9632272_947b54fb","line":654,"range":{"start_line":642,"start_character":0,"end_line":654,"end_character":18},"updated":"2026-06-10 23:51:46.000000000","message":"this is a real concern and may be the majority of the effort. Idea/suggestion: for MVP / version 1.0 we could assume that \n\n1) all the NIC firmware and BIOS settings\n2) all the switching\n\nare statically pre-configured for RDMA work and the key \"dynamic\" part is changing VXLAN/VLAN configuration.\n\nRDMA interconnects are expensive, I don\u0027t think a lot of cloud providers would want non-RDMA workloads to run on RDMA hardware. And this is... before we throw GPUs to the mix. Where AI workloads are involved, RDMA-enabled nodes will also be GPU-enabled which strengthens the case for only ever using them for GPU/RDMA workloads.\n\nWhile this is all great, I am concerned getting the time commitment to implement this part may be difficult and what I am trying to say here is we can probably deliver 80% of the value with 20% of the effort by making 1) switching and 2) possibly firmware/bios-settings side (less concern) semi static and mostly work with parts dictated by security requirements / tenant isolation.\n\nWDYT?","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"62ee0f73389ddd0407135710fafc2a33eca96cfc","unresolved":true,"context_lines":[{"line_number":639,"context_line":"  classes. Storm control rate-limits broadcast/multicast"},{"line_number":640,"context_line":"  traffic, which can interfere with PFC pause frames."},{"line_number":641,"context_line":""},{"line_number":642,"context_line":"**Current state**: All of these settings must be manually"},{"line_number":643,"context_line":"pre-configured on every switch port that will carry RDMA"},{"line_number":644,"context_line":"traffic. This is acceptable for static environments where"},{"line_number":645,"context_line":"RDMA ports are known in advance, but it breaks down when"},{"line_number":646,"context_line":"nodes are dynamically provisioned -- a switch port that"},{"line_number":647,"context_line":"carried standard IP traffic yesterday may need lossless"},{"line_number":648,"context_line":"configuration today because Ironic scheduled an RDMA"},{"line_number":649,"context_line":"workload to that node."},{"line_number":650,"context_line":""},{"line_number":651,"context_line":"**Proposed approach**: This is a significant body of work"},{"line_number":652,"context_line":"that requires its own specification in the"},{"line_number":653,"context_line":"``networking-generic-switch`` project. The high-level"},{"line_number":654,"context_line":"approach would be:"},{"line_number":655,"context_line":""},{"line_number":656,"context_line":"1. **Extend the switch device driver interface** in"},{"line_number":657,"context_line":"   ``networking-generic-switch`` with new command templates"}],"source_content_type":"text/x-rst","patch_set":4,"id":"c0cf6843_a32ed962","line":654,"range":{"start_line":642,"start_character":0,"end_line":654,"end_character":18},"in_reply_to":"c9632272_947b54fb","updated":"2026-06-11 18:50:47.000000000","message":"So, I think we\u0027re so close that really the bulk of the work is much more the later items of settings/modeling/substrate and ultimately switch port settings. I don\u0027t know how we really articulate that entirely. I\u0027m less concerned about the time commitment to get started since it already seems like folks are using some of what is already available to get there. They just have the required dib elements and some additional configuration.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":32177,"name":"Jacob Anders","email":"jacob-anders-dev@proton.me","username":"janders"},"change_message_id":"bd88c263ebae6309d737f368a4f7fe16ac1146da","unresolved":true,"context_lines":[{"line_number":1039,"context_line":"      --network rdma-net \\"},{"line_number":1040,"context_line":"      my-rdma-node"},{"line_number":1041,"context_line":""},{"line_number":1042,"context_line":"What happens behind the scenes:"},{"line_number":1043,"context_line":""},{"line_number":1044,"context_line":"1. Nova finds a node with ``CUSTOM_BAREMETAL_RDMA`` resource"},{"line_number":1045,"context_line":"   class and ``CUSTOM_RDMA_ROCE`` trait."}],"source_content_type":"text/x-rst","patch_set":4,"id":"3760cdfa_6b8ecd4c","line":1042,"range":{"start_line":1042,"start_character":0,"end_line":1042,"end_character":31},"updated":"2026-06-10 23:51:46.000000000","message":"this all LGTM","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"62ee0f73389ddd0407135710fafc2a33eca96cfc","unresolved":false,"context_lines":[{"line_number":1039,"context_line":"      --network rdma-net \\"},{"line_number":1040,"context_line":"      my-rdma-node"},{"line_number":1041,"context_line":""},{"line_number":1042,"context_line":"What happens behind the scenes:"},{"line_number":1043,"context_line":""},{"line_number":1044,"context_line":"1. Nova finds a node with ``CUSTOM_BAREMETAL_RDMA`` resource"},{"line_number":1045,"context_line":"   class and ``CUSTOM_RDMA_ROCE`` trait."}],"source_content_type":"text/x-rst","patch_set":4,"id":"3988b651_139764a2","line":1042,"range":{"start_line":1042,"start_character":0,"end_line":1042,"end_character":31},"in_reply_to":"3760cdfa_6b8ecd4c","updated":"2026-06-11 18:50:47.000000000","message":"Done","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":32177,"name":"Jacob Anders","email":"jacob-anders-dev@proton.me","username":"janders"},"change_message_id":"bd88c263ebae6309d737f368a4f7fe16ac1146da","unresolved":true,"context_lines":[{"line_number":1072,"context_line":"10. The node is marked ACTIVE."},{"line_number":1073,"context_line":""},{"line_number":1074,"context_line":""},{"line_number":1075,"context_line":"RDMA-Ready Image Requirements"},{"line_number":1076,"context_line":"------------------------------"},{"line_number":1077,"context_line":""},{"line_number":1078,"context_line":"The deployed OS image must contain the RDMA software stack."}],"source_content_type":"text/x-rst","patch_set":4,"id":"e29b5f5e_efe23df8","line":1075,"range":{"start_line":1075,"start_character":0,"end_line":1075,"end_character":29},"updated":"2026-06-10 23:51:46.000000000","message":"I think Claude is repeating himself here (this was covered in the earlier DIB section). Perhaps could be consolidated to condense the spec.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"62ee0f73389ddd0407135710fafc2a33eca96cfc","unresolved":false,"context_lines":[{"line_number":1072,"context_line":"10. The node is marked ACTIVE."},{"line_number":1073,"context_line":""},{"line_number":1074,"context_line":""},{"line_number":1075,"context_line":"RDMA-Ready Image Requirements"},{"line_number":1076,"context_line":"------------------------------"},{"line_number":1077,"context_line":""},{"line_number":1078,"context_line":"The deployed OS image must contain the RDMA software stack."}],"source_content_type":"text/x-rst","patch_set":4,"id":"12d3bb5b_c34710db","line":1075,"range":{"start_line":1075,"start_character":0,"end_line":1075,"end_character":29},"in_reply_to":"e29b5f5e_efe23df8","updated":"2026-06-11 18:50:47.000000000","message":"Yeah, it was already being a bit verbose. But it wants to burn the tokens!","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":32177,"name":"Jacob Anders","email":"jacob-anders-dev@proton.me","username":"janders"},"change_message_id":"bd88c263ebae6309d737f368a4f7fe16ac1146da","unresolved":true,"context_lines":[{"line_number":1113,"context_line":"Alternatives"},{"line_number":1114,"context_line":"------------"},{"line_number":1115,"context_line":""},{"line_number":1116,"context_line":"**Single converged adapter**: Use one physical NIC for both IP"},{"line_number":1117,"context_line":"and RDMA traffic. This forces RDMA and IP traffic to share"},{"line_number":1118,"context_line":"PCIe bandwidth, interrupt queues, and NUMA locality, degrading"},{"line_number":1119,"context_line":"RDMA performance. Not recommended for performance-sensitive"},{"line_number":1120,"context_line":"workloads."},{"line_number":1121,"context_line":""},{"line_number":1122,"context_line":"**Pre-configured NIC firmware on all nodes**: Flash every NIC"},{"line_number":1123,"context_line":"to RoCEv2 mode during initial rack-and-stack, eliminating the"}],"source_content_type":"text/x-rst","patch_set":4,"id":"4728da10_3e5b72d3","line":1120,"range":{"start_line":1116,"start_character":0,"end_line":1120,"end_character":10},"updated":"2026-06-10 23:51:46.000000000","message":"I think this may be a pain if people want to use \"mainstream\" networking with DHCP, neutron routers, floating IPs, security groups, etc. It can all be made work on the HCA *probably* but for most operators effort of doing this may involve larger cost than purchase and operations of a second basic/low-cost management network.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"62ee0f73389ddd0407135710fafc2a33eca96cfc","unresolved":true,"context_lines":[{"line_number":1113,"context_line":"Alternatives"},{"line_number":1114,"context_line":"------------"},{"line_number":1115,"context_line":""},{"line_number":1116,"context_line":"**Single converged adapter**: Use one physical NIC for both IP"},{"line_number":1117,"context_line":"and RDMA traffic. This forces RDMA and IP traffic to share"},{"line_number":1118,"context_line":"PCIe bandwidth, interrupt queues, and NUMA locality, degrading"},{"line_number":1119,"context_line":"RDMA performance. Not recommended for performance-sensitive"},{"line_number":1120,"context_line":"workloads."},{"line_number":1121,"context_line":""},{"line_number":1122,"context_line":"**Pre-configured NIC firmware on all nodes**: Flash every NIC"},{"line_number":1123,"context_line":"to RoCEv2 mode during initial rack-and-stack, eliminating the"}],"source_content_type":"text/x-rst","patch_set":4,"id":"c5eeeac3_4c205a37","line":1120,"range":{"start_line":1116,"start_character":0,"end_line":1120,"end_character":10},"in_reply_to":"4728da10_3e5b72d3","updated":"2026-06-11 18:50:47.000000000","message":"That was along the lines of what I was thinking. It would force the operator to get things \"just right\" and ultimately the networking models don\u0027t really match in general *either*.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":32177,"name":"Jacob Anders","email":"jacob-anders-dev@proton.me","username":"janders"},"change_message_id":"bd88c263ebae6309d737f368a4f7fe16ac1146da","unresolved":true,"context_lines":[{"line_number":1119,"context_line":"RDMA performance. Not recommended for performance-sensitive"},{"line_number":1120,"context_line":"workloads."},{"line_number":1121,"context_line":""},{"line_number":1122,"context_line":"**Pre-configured NIC firmware on all nodes**: Flash every NIC"},{"line_number":1123,"context_line":"to RoCEv2 mode during initial rack-and-stack, eliminating the"},{"line_number":1124,"context_line":"need for deploy-time firmware steps. This works for homogeneous"},{"line_number":1125,"context_line":"environments where every node will always be used for RDMA, but"},{"line_number":1126,"context_line":"prevents the same hardware from being used for non-RDMA"},{"line_number":1127,"context_line":"workloads without a manual firmware reset."},{"line_number":1128,"context_line":""},{"line_number":1129,"context_line":"**Post-deploy configuration management**: Use Ansible or"},{"line_number":1130,"context_line":"similar tools after deployment to configure RDMA. This creates"}],"source_content_type":"text/x-rst","patch_set":4,"id":"90221f5f_06e08e33","line":1127,"range":{"start_line":1122,"start_character":0,"end_line":1127,"end_character":42},"updated":"2026-06-10 23:51:46.000000000","message":"solid +1 from me on this especially for MVP and/or v1.0. Once adoption is there (and development effort is justified) by all means we can and should streamline all of this.\n\nThis is coming from a guy who\u0027s build multiple RDMA-enabled OpenStack PoCs which worked well validated underlying concepts and proved their capabilities but were deemed too complex and too costly to operate to go large-scale.\n\nSimplifying things at the beginning made these projects feasible at all - and not going through with full implementation scope early saved a lot of effort that would have otherwise been wasted due to lack of adoption beyond PoCs.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"62ee0f73389ddd0407135710fafc2a33eca96cfc","unresolved":false,"context_lines":[{"line_number":1119,"context_line":"RDMA performance. Not recommended for performance-sensitive"},{"line_number":1120,"context_line":"workloads."},{"line_number":1121,"context_line":""},{"line_number":1122,"context_line":"**Pre-configured NIC firmware on all nodes**: Flash every NIC"},{"line_number":1123,"context_line":"to RoCEv2 mode during initial rack-and-stack, eliminating the"},{"line_number":1124,"context_line":"need for deploy-time firmware steps. This works for homogeneous"},{"line_number":1125,"context_line":"environments where every node will always be used for RDMA, but"},{"line_number":1126,"context_line":"prevents the same hardware from being used for non-RDMA"},{"line_number":1127,"context_line":"workloads without a manual firmware reset."},{"line_number":1128,"context_line":""},{"line_number":1129,"context_line":"**Post-deploy configuration management**: Use Ansible or"},{"line_number":1130,"context_line":"similar tools after deployment to configure RDMA. This creates"}],"source_content_type":"text/x-rst","patch_set":4,"id":"0b153ab7_3e092b4a","line":1127,"range":{"start_line":1122,"start_character":0,"end_line":1127,"end_character":42},"in_reply_to":"90221f5f_06e08e33","updated":"2026-06-11 18:50:47.000000000","message":"Yeah, I think a solid starting point and really the base state is that they *could* potentially achieve most of this on an existing fabric if the switch config is happy in advance. The idea is to always make things a bit more dynamic 😉","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":32177,"name":"Jacob Anders","email":"jacob-anders-dev@proton.me","username":"janders"},"change_message_id":"bd88c263ebae6309d737f368a4f7fe16ac1146da","unresolved":true,"context_lines":[{"line_number":1131,"context_line":"a window where the node is ACTIVE but not functional, and"},{"line_number":1132,"context_line":"requires external orchestration."},{"line_number":1133,"context_line":""},{"line_number":1134,"context_line":"**InfiniBand instead of RoCE**: Use native InfiniBand with a"},{"line_number":1135,"context_line":"subnet manager. This provides better RDMA performance but"},{"line_number":1136,"context_line":"requires InfiniBand switches (not Ethernet) and a dedicated"},{"line_number":1137,"context_line":"fabric. Ironic has basic InfiniBand support (see"},{"line_number":1138,"context_line":"``specs/approved/add-infiniband-support.rst``) but it is"},{"line_number":1139,"context_line":"limited to PXE boot and MAC address handling."},{"line_number":1140,"context_line":""},{"line_number":1141,"context_line":"Data model impact"},{"line_number":1142,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"bd997223_633fc95d","line":1139,"range":{"start_line":1134,"start_character":0,"end_line":1139,"end_character":45},"updated":"2026-06-10 23:51:46.000000000","message":"IB-based implementation can be extremely powerful due to near-unlimited scaling (in a way this is the holy grail of BM HPC Cloud) but has compatibility quirks. Many tools used by end-users come with built in assumptions they are running on ethernet; think of difficulty of bridging standard TCPIP comms and IB fabric to do floating IP (EIPoIB topic comes back). And that\u0027s even before we get onto the SDN part. It all could be made work but I think it would be good to see some RoCE adoption first to justify development effort and get vendor buy-in (I\u0027d expect significant effort to be required from interconnect vendors to make IB implementation production-grade).","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"62ee0f73389ddd0407135710fafc2a33eca96cfc","unresolved":true,"context_lines":[{"line_number":1131,"context_line":"a window where the node is ACTIVE but not functional, and"},{"line_number":1132,"context_line":"requires external orchestration."},{"line_number":1133,"context_line":""},{"line_number":1134,"context_line":"**InfiniBand instead of RoCE**: Use native InfiniBand with a"},{"line_number":1135,"context_line":"subnet manager. This provides better RDMA performance but"},{"line_number":1136,"context_line":"requires InfiniBand switches (not Ethernet) and a dedicated"},{"line_number":1137,"context_line":"fabric. Ironic has basic InfiniBand support (see"},{"line_number":1138,"context_line":"``specs/approved/add-infiniband-support.rst``) but it is"},{"line_number":1139,"context_line":"limited to PXE boot and MAC address handling."},{"line_number":1140,"context_line":""},{"line_number":1141,"context_line":"Data model impact"},{"line_number":1142,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"f3f88e98_d7064dec","line":1139,"range":{"start_line":1134,"start_character":0,"end_line":1139,"end_character":45},"in_reply_to":"bd997223_633fc95d","updated":"2026-06-11 18:50:47.000000000","message":"I guess the prime blocker here is a lack of network model to manage it through Neutron in similar fashion. Not impossible, but also not in scope for a first pass at the topic.\n\nI guess the lack of direct backend matching is a bit of a limiting factor to such anyhow.","commit_id":"42fb97eb0b176aa68d099ef06192f634241e4cf3"},{"author":{"_account_id":24245,"name":"Harald Jensås","email":"hjensas@redhat.com","username":"harald.jensas"},"change_message_id":"344c16e83fc31206d478d6d5929151797c8f70af","unresolved":true,"context_lines":[{"line_number":8,"context_line":"RDMA over Converged Ethernet (RoCE) Bare Metal Provisioning"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://bugs.launchpad.net/ironic/+bug/XXXXXXX"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Operators want to offer bare metal nodes with RDMA over Converged"},{"line_number":14,"context_line":"Ethernet (RoCE) as a self-service resource alongside standard IP"}],"source_content_type":"text/x-rst","patch_set":6,"id":"cf61cbed_0ad9f547","line":11,"range":{"start_line":11,"start_character":39,"end_line":11,"end_character":46},"updated":"2026-06-15 12:21:01.000000000","message":"Is there a bug, or do we plan to create one and update this to reference it?","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"c67edf86775f61b46f5dc797c8c96c0341180b6e","unresolved":true,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://bugs.launchpad.net/ironic/+bug/XXXXXXX"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Operators want to offer bare metal nodes with RDMA over Converged"},{"line_number":14,"context_line":"Ethernet (RoCE) as a self-service resource alongside standard IP"},{"line_number":15,"context_line":"networking. In their operational model, the RDMA fabric is simply"},{"line_number":16,"context_line":"another Neutron network. The typical server has separate physical"}],"source_content_type":"text/x-rst","patch_set":6,"id":"c93502bf_ea67d0d4","line":13,"updated":"2026-07-14 18:07:04.000000000","message":"We should explain Remote Direct Memory Access","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":24245,"name":"Harald Jensås","email":"hjensas@redhat.com","username":"harald.jensas"},"change_message_id":"344c16e83fc31206d478d6d5929151797c8f70af","unresolved":true,"context_lines":[{"line_number":59,"context_line":""},{"line_number":60,"context_line":"   This specification refers to RoCE, RDMA over Converged Ethernet,"},{"line_number":61,"context_line":"   however Conveged adapters are not explicitly required. This document"},{"line_number":62,"context_line":"   refers to it as a shorthand for the RoCEv2 of use of UDP packets on"},{"line_number":63,"context_line":"   a common physical subnet. In other words, it is entirely possible"},{"line_number":64,"context_line":"   to use a non-conveged adapter, but a converged adapter with RDMA"},{"line_number":65,"context_line":"   offload would likely be preferred by infrastructure operators for"}],"source_content_type":"text/x-rst","patch_set":6,"id":"4fb40ced_6d5d355e","line":62,"range":{"start_line":62,"start_character":46,"end_line":62,"end_character":48},"updated":"2026-06-15 12:21:01.000000000","message":"nit, remove this `of`","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"c67edf86775f61b46f5dc797c8c96c0341180b6e","unresolved":true,"context_lines":[{"line_number":272,"context_line":"  directly and co-locate RDMA workloads by choosing"},{"line_number":273,"context_line":"  nodes whose inventory structure reflects shared rack"},{"line_number":274,"context_line":"  or switch topology."},{"line_number":275,"context_line":""},{"line_number":276,"context_line":"* **Configure the entire RDMA underlay for lossless"},{"line_number":277,"context_line":"  behavior.** If RDMA nodes must span multiple leaf"},{"line_number":278,"context_line":"  switches, operators should configure PFC and ECN on all"}],"source_content_type":"text/x-rst","patch_set":6,"id":"a6ad5c12_b56fddb1","line":275,"updated":"2026-07-14 18:07:04.000000000","message":"prime reviewer feedback from group review:\n1) Too much text to explain it, reviewers are basically punting because the size.\n2) dial back to the needful text.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":24245,"name":"Harald Jensås","email":"hjensas@redhat.com","username":"harald.jensas"},"change_message_id":"344c16e83fc31206d478d6d5929151797c8f70af","unresolved":true,"context_lines":[{"line_number":344,"context_line":""},{"line_number":345,"context_line":"The ports have completely different ``local_link_connection``"},{"line_number":346,"context_line":"values because they are different NICs on different cables to"},{"line_number":347,"context_line":"potentially different switches. They are associated with"},{"line_number":348,"context_line":"different ``physical_network`` values (``tenant-physnet`` vs"},{"line_number":349,"context_line":"``rdma-physnet``), which tells Neutron\u0027s ML2 plugin which"},{"line_number":350,"context_line":"provider network segment and which switch to configure."},{"line_number":351,"context_line":""},{"line_number":352,"context_line":"When Ironic plugs each port to its respective Neutron network,"}],"source_content_type":"text/x-rst","patch_set":6,"id":"51a7908f_98efcea1","line":349,"range":{"start_line":347,"start_character":32,"end_line":349,"end_character":17},"updated":"2026-06-15 12:21:01.000000000","message":"I think we should be more explicit here, we only want to use separate physnet if it is actually separate switches. If eth1/1 (IP traffic) and eth1/2 (RDMA traffic) is connected to the same switch - it should have the same physnet to avoid the \"foot gun\" where the same VLAN range can be configured for two physnets in neutron.\n\nSince RDMA networks (i.e VLANs) require a tag no the neutron network, we should be able to differentiate the config applied by ML2 plug-ins from that tag alone. \n\nAFICT the spec already has all the details to allow this to work with a single-physnet.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":24245,"name":"Harald Jensås","email":"hjensas@redhat.com","username":"harald.jensas"},"change_message_id":"04d2d8cc0747d3f142872f470cceda9ce0ba85f4","unresolved":true,"context_lines":[{"line_number":344,"context_line":""},{"line_number":345,"context_line":"The ports have completely different ``local_link_connection``"},{"line_number":346,"context_line":"values because they are different NICs on different cables to"},{"line_number":347,"context_line":"potentially different switches. They are associated with"},{"line_number":348,"context_line":"different ``physical_network`` values (``tenant-physnet`` vs"},{"line_number":349,"context_line":"``rdma-physnet``), which tells Neutron\u0027s ML2 plugin which"},{"line_number":350,"context_line":"provider network segment and which switch to configure."},{"line_number":351,"context_line":""},{"line_number":352,"context_line":"When Ironic plugs each port to its respective Neutron network,"}],"source_content_type":"text/x-rst","patch_set":6,"id":"44f3d1fa_e2221ee3","line":349,"range":{"start_line":347,"start_character":32,"end_line":349,"end_character":17},"in_reply_to":"0521b23f_d0c099f1","updated":"2026-07-14 16:52:54.000000000","message":"This may well be the physnet is human construct - and we prabably just think different. :)\n\nI read \"potentially different switches\" - and then \"are associated with different physical_network\". Basically they have to be separate physnet. I don\u0027t think they need to be on different physnet - the scheduling happens via the category \u003d\u003d \u0027CUSTOM_RDMA\u0027 and the networks \u0027rdma\u0027 tag. So it scheduling will work in either case?\n\nSo, maybe less explicit rather than more - if it can be the same or different physnet we may not want to just remove that wording.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"20e0fb13e4748f76fc778735c475cbac1a31b08c","unresolved":true,"context_lines":[{"line_number":344,"context_line":""},{"line_number":345,"context_line":"The ports have completely different ``local_link_connection``"},{"line_number":346,"context_line":"values because they are different NICs on different cables to"},{"line_number":347,"context_line":"potentially different switches. They are associated with"},{"line_number":348,"context_line":"different ``physical_network`` values (``tenant-physnet`` vs"},{"line_number":349,"context_line":"``rdma-physnet``), which tells Neutron\u0027s ML2 plugin which"},{"line_number":350,"context_line":"provider network segment and which switch to configure."},{"line_number":351,"context_line":""},{"line_number":352,"context_line":"When Ironic plugs each port to its respective Neutron network,"}],"source_content_type":"text/x-rst","patch_set":6,"id":"0521b23f_d0c099f1","line":349,"range":{"start_line":347,"start_character":32,"end_line":349,"end_character":17},"in_reply_to":"51a7908f_98efcea1","updated":"2026-07-14 14:08:08.000000000","message":"Regarding the footgun risk you highlighted, I think thats sort of outside our direct control because physnets are entirely a human construct to manage zoning and you can overlap them. That being said, its not a great idea to do so as we\u0027re all well aware. I guess the challenge is I don\u0027t really see an issue *unless* the operator has chosen to directly overlap \"physnets\" across the same switches.\n\nWhich makes me wonder what what should we really be more explicit about aside from highlighting the intent behind physnet model.\n\nAs for differentiating, I have been thinking binding profile data could be explicit. Relying purely on the physnet name being known across to the ml2 driver seems wrong, but then again maybe that migh tbe the right path becuase we make it explicit and physnets are now shipped across to allow additional logic to be performed on that side. The alternative is to have a flag in the binding profile data.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"c67edf86775f61b46f5dc797c8c96c0341180b6e","unresolved":true,"context_lines":[{"line_number":356,"context_line":"as an access port on VLAN 200."},{"line_number":357,"context_line":""},{"line_number":358,"context_line":""},{"line_number":359,"context_line":"Change 1: Promote NIC Firmware Steps to Deploy Steps (Done)"},{"line_number":360,"context_line":"------------------------------------------------------------"},{"line_number":361,"context_line":""},{"line_number":362,"context_line":".. note::"}],"source_content_type":"text/x-rst","patch_set":6,"id":"3a1363a6_1cff4c56","line":359,"updated":"2026-07-14 18:07:04.000000000","message":"Group review note: There is ongoing discussion fo retiring \"other\" firmware update models, this doesn\u0027t mandate firmware updates for devices. Likely need to drop change 1 entirely.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"c67edf86775f61b46f5dc797c8c96c0341180b6e","unresolved":true,"context_lines":[{"line_number":386,"context_line":"    # NIC firmware is configured automatically"},{"line_number":387,"context_line":"    openstack server create --flavor bm.rdma.large ..."},{"line_number":388,"context_line":""},{"line_number":389,"context_line":"**Reboot handling**: NIC firmware changes typically require a"},{"line_number":390,"context_line":"reboot to take effect. The existing methods already set"},{"line_number":391,"context_line":"``reboot_requested\u003dTrue`` in their step definition. The deploy"},{"line_number":392,"context_line":"step framework handles this by rebooting the node and waiting"},{"line_number":393,"context_line":"for the IPA ramdisk to reconnect before proceeding to the next"},{"line_number":394,"context_line":"step."},{"line_number":395,"context_line":""},{"line_number":396,"context_line":"Change 2: Verify and Document Reboot Coalescing"},{"line_number":397,"context_line":"------------------------------------------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"0ab8c1a1_d1ea8d48","line":394,"range":{"start_line":389,"start_character":0,"end_line":394,"end_character":0},"updated":"2026-07-14 18:07:04.000000000","message":"this is informational and guarding context, and can be dropped.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"c67edf86775f61b46f5dc797c8c96c0341180b6e","unresolved":true,"context_lines":[{"line_number":439,"context_line":"This has already been proposed as"},{"line_number":440,"context_line":"`change 990417 \u003chttps://review.opendev.org/c/openstack/diskimage-builder/+/990417\u003e`_."},{"line_number":441,"context_line":""},{"line_number":442,"context_line":"Change 5: Switch Port Configuration for Lossless Ethernet"},{"line_number":443,"context_line":"----------------------------------------------------------"},{"line_number":444,"context_line":""},{"line_number":445,"context_line":"**What**: Propose extensions to the switch management plugin"}],"source_content_type":"text/x-rst","patch_set":6,"id":"b72ddc0b_44e05d5e","line":442,"range":{"start_line":442,"start_character":0,"end_line":442,"end_character":57},"updated":"2026-07-14 18:07:04.000000000","message":"This is really the important bit of content for the specification.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"c67edf86775f61b46f5dc797c8c96c0341180b6e","unresolved":true,"context_lines":[{"line_number":533,"context_line":"their own and would need some initial prior art to exist in"},{"line_number":534,"context_line":"order to make progress. A future high-level approach for"},{"line_number":535,"context_line":"that project might be:"},{"line_number":536,"context_line":""},{"line_number":537,"context_line":"1. **Extend the switch device driver interface** in"},{"line_number":538,"context_line":"   ``networking-generic-switch`` with new command templates"},{"line_number":539,"context_line":"   for MTU, DSCP-to-CoS mapping, PFC, ETS, ECN, DCBX,"}],"source_content_type":"text/x-rst","patch_set":6,"id":"a0f9c9ad_4a045a98","line":536,"updated":"2026-07-14 18:07:04.000000000","message":"frame this in part that this needs to be additional detail which needs to be unwound on the port binding in n-g-s on the binding profile.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":24245,"name":"Harald Jensås","email":"hjensas@redhat.com","username":"harald.jensas"},"change_message_id":"344c16e83fc31206d478d6d5929151797c8f70af","unresolved":true,"context_lines":[{"line_number":541,"context_line":"   (Cisco NX-OS, Arista EOS, Dell OS10, Juniper Junos,"},{"line_number":542,"context_line":"   etc.) would need its own command implementations."},{"line_number":543,"context_line":""},{"line_number":544,"context_line":"2. **Propagate network MTU to the switch port**. The L2VNI"},{"line_number":545,"context_line":"   driver already knows the Neutron network\u0027s MTU. The"},{"line_number":546,"context_line":"   switch management plugin should receive this value and"},{"line_number":547,"context_line":"   configure the physical switch port accordingly."},{"line_number":548,"context_line":"   As a note, there is already a patch to do this in upstream"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7d7f083e_f3121309","line":545,"range":{"start_line":544,"start_character":49,"end_line":545,"end_character":9},"updated":"2026-06-15 12:21:01.000000000","message":"Mentioning L2VNI here is confusing, the MTU is part of the NetworkContext so setting the MTU is plain ML2 (ngs) port plugging?","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"20e0fb13e4748f76fc778735c475cbac1a31b08c","unresolved":true,"context_lines":[{"line_number":541,"context_line":"   (Cisco NX-OS, Arista EOS, Dell OS10, Juniper Junos,"},{"line_number":542,"context_line":"   etc.) would need its own command implementations."},{"line_number":543,"context_line":""},{"line_number":544,"context_line":"2. **Propagate network MTU to the switch port**. The L2VNI"},{"line_number":545,"context_line":"   driver already knows the Neutron network\u0027s MTU. The"},{"line_number":546,"context_line":"   switch management plugin should receive this value and"},{"line_number":547,"context_line":"   configure the physical switch port accordingly."},{"line_number":548,"context_line":"   As a note, there is already a patch to do this in upstream"}],"source_content_type":"text/x-rst","patch_set":6,"id":"197eaa4f_2a469672","line":545,"range":{"start_line":544,"start_character":49,"end_line":545,"end_character":9},"in_reply_to":"7d7f083e_f3121309","updated":"2026-07-14 14:08:08.000000000","message":"I can see the L2VNI driver being jarring ot mention, really, it is just the port context also handing across the binding profile data on the binding action which includes an MTU. Outside of the port, I don\u0027t think there is any need to set an MTU.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":24245,"name":"Harald Jensås","email":"hjensas@redhat.com","username":"harald.jensas"},"change_message_id":"344c16e83fc31206d478d6d5929151797c8f70af","unresolved":true,"context_lines":[{"line_number":549,"context_line":"   review, as"},{"line_number":550,"context_line":"   `change 984832 \u003chttps://review.opendev.org/c/openstack/networking-generic-switch/+/984832\u003e`_."},{"line_number":551,"context_line":""},{"line_number":552,"context_line":"3. **Use Ironic port metadata to signal lossless"},{"line_number":553,"context_line":"   requirements**. The Ironic structure data or network tag"},{"line_number":554,"context_line":"   field carry a flag indicating that the port requires"},{"line_number":555,"context_line":"   lossless switch configuration (PFC,"}],"source_content_type":"text/x-rst","patch_set":6,"id":"22ed51a7_59eb81e0","line":552,"range":{"start_line":552,"start_character":5,"end_line":552,"end_character":48},"updated":"2026-06-15 12:21:01.000000000","message":"The Ironic port metadate is not directly available to Neutron ML2.\n\nWe can use binding:profile as it is an opaque JSON dict to hold this data, Ironic\u0027s `plug_port_to_tenant_network` can add it. But worth mentioning in the spec.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"20e0fb13e4748f76fc778735c475cbac1a31b08c","unresolved":true,"context_lines":[{"line_number":549,"context_line":"   review, as"},{"line_number":550,"context_line":"   `change 984832 \u003chttps://review.opendev.org/c/openstack/networking-generic-switch/+/984832\u003e`_."},{"line_number":551,"context_line":""},{"line_number":552,"context_line":"3. **Use Ironic port metadata to signal lossless"},{"line_number":553,"context_line":"   requirements**. The Ironic structure data or network tag"},{"line_number":554,"context_line":"   field carry a flag indicating that the port requires"},{"line_number":555,"context_line":"   lossless switch configuration (PFC,"}],"source_content_type":"text/x-rst","patch_set":6,"id":"fb964b55_61c94807","line":552,"range":{"start_line":552,"start_character":5,"end_line":552,"end_character":48},"in_reply_to":"22ed51a7_59eb81e0","updated":"2026-07-14 14:08:08.000000000","message":"Ahh, yeah, we should likely re-frame this as \"Add additional metadata to the binding profile to signal lossless\"","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":24245,"name":"Harald Jensås","email":"hjensas@redhat.com","username":"harald.jensas"},"change_message_id":"04d2d8cc0747d3f142872f470cceda9ce0ba85f4","unresolved":true,"context_lines":[{"line_number":549,"context_line":"   review, as"},{"line_number":550,"context_line":"   `change 984832 \u003chttps://review.opendev.org/c/openstack/networking-generic-switch/+/984832\u003e`_."},{"line_number":551,"context_line":""},{"line_number":552,"context_line":"3. **Use Ironic port metadata to signal lossless"},{"line_number":553,"context_line":"   requirements**. The Ironic structure data or network tag"},{"line_number":554,"context_line":"   field carry a flag indicating that the port requires"},{"line_number":555,"context_line":"   lossless switch configuration (PFC,"}],"source_content_type":"text/x-rst","patch_set":6,"id":"5123b62a_dd965de4","line":552,"range":{"start_line":552,"start_character":5,"end_line":552,"end_character":48},"in_reply_to":"fb964b55_61c94807","updated":"2026-07-14 16:52:54.000000000","message":"+1","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":24245,"name":"Harald Jensås","email":"hjensas@redhat.com","username":"harald.jensas"},"change_message_id":"344c16e83fc31206d478d6d5929151797c8f70af","unresolved":true,"context_lines":[{"line_number":566,"context_line":""},{"line_number":567,"context_line":"   .. code-block::"},{"line_number":568,"context_line":""},{"line_number":569,"context_line":"       ! Classify RoCE traffic by DSCP"},{"line_number":570,"context_line":"       class-map type qos match-all roce-traffic"},{"line_number":571,"context_line":"         match dscp 26"},{"line_number":572,"context_line":"       policy-map type qos roce-ingress"},{"line_number":573,"context_line":"         class roce-traffic"},{"line_number":574,"context_line":"           set qos-group 3"},{"line_number":575,"context_line":""},{"line_number":576,"context_line":"       ! Apply QoS and lossless settings to the port"},{"line_number":577,"context_line":"       interface Ethernet1/5"}],"source_content_type":"text/x-rst","patch_set":6,"id":"825c0c02_04911b76","line":574,"range":{"start_line":569,"start_character":0,"end_line":574,"end_character":26},"updated":"2026-06-15 12:21:01.000000000","message":"These a re one time config, should we be explicit and say this is something that must be pre-configured by the operator? Or do we want to automate it, apply on startup only or do a get_config -\u003e apply if needed dance?","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"20e0fb13e4748f76fc778735c475cbac1a31b08c","unresolved":true,"context_lines":[{"line_number":566,"context_line":""},{"line_number":567,"context_line":"   .. code-block::"},{"line_number":568,"context_line":""},{"line_number":569,"context_line":"       ! Classify RoCE traffic by DSCP"},{"line_number":570,"context_line":"       class-map type qos match-all roce-traffic"},{"line_number":571,"context_line":"         match dscp 26"},{"line_number":572,"context_line":"       policy-map type qos roce-ingress"},{"line_number":573,"context_line":"         class roce-traffic"},{"line_number":574,"context_line":"           set qos-group 3"},{"line_number":575,"context_line":""},{"line_number":576,"context_line":"       ! Apply QoS and lossless settings to the port"},{"line_number":577,"context_line":"       interface Ethernet1/5"}],"source_content_type":"text/x-rst","patch_set":6,"id":"b68d4cd2_553a9125","line":574,"range":{"start_line":569,"start_character":0,"end_line":574,"end_character":26},"in_reply_to":"825c0c02_04911b76","updated":"2026-07-14 14:08:08.000000000","message":"I was thinking more as the one time operator configuration, that we shouldn\u0027t really expect to set that up for folks. Maybe we could though?\n\nI guess the challenge is operators might be doing some of this already and the best pattern we can likely ship is one where it integrates into what they are doing?","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":24245,"name":"Harald Jensås","email":"hjensas@redhat.com","username":"harald.jensas"},"change_message_id":"04d2d8cc0747d3f142872f470cceda9ce0ba85f4","unresolved":true,"context_lines":[{"line_number":566,"context_line":""},{"line_number":567,"context_line":"   .. code-block::"},{"line_number":568,"context_line":""},{"line_number":569,"context_line":"       ! Classify RoCE traffic by DSCP"},{"line_number":570,"context_line":"       class-map type qos match-all roce-traffic"},{"line_number":571,"context_line":"         match dscp 26"},{"line_number":572,"context_line":"       policy-map type qos roce-ingress"},{"line_number":573,"context_line":"         class roce-traffic"},{"line_number":574,"context_line":"           set qos-group 3"},{"line_number":575,"context_line":""},{"line_number":576,"context_line":"       ! Apply QoS and lossless settings to the port"},{"line_number":577,"context_line":"       interface Ethernet1/5"}],"source_content_type":"text/x-rst","patch_set":6,"id":"dfee9bbb_65814ed1","line":574,"range":{"start_line":569,"start_character":0,"end_line":574,"end_character":26},"in_reply_to":"b68d4cd2_553a9125","updated":"2026-07-14 16:52:54.000000000","message":"Yeah, I am leaning on the fact operators already do this and probably have some management of \"static\" config already so I think it is good enough we document these must exist. At least in the first iteration.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"3198509cf74f2fd5afcd5f09a6aadda77c46d743","unresolved":true,"context_lines":[{"line_number":566,"context_line":""},{"line_number":567,"context_line":"   .. code-block::"},{"line_number":568,"context_line":""},{"line_number":569,"context_line":"       ! Classify RoCE traffic by DSCP"},{"line_number":570,"context_line":"       class-map type qos match-all roce-traffic"},{"line_number":571,"context_line":"         match dscp 26"},{"line_number":572,"context_line":"       policy-map type qos roce-ingress"},{"line_number":573,"context_line":"         class roce-traffic"},{"line_number":574,"context_line":"           set qos-group 3"},{"line_number":575,"context_line":""},{"line_number":576,"context_line":"       ! Apply QoS and lossless settings to the port"},{"line_number":577,"context_line":"       interface Ethernet1/5"}],"source_content_type":"text/x-rst","patch_set":6,"id":"2f0b3f6f_1a1fb84a","line":574,"range":{"start_line":569,"start_character":0,"end_line":574,"end_character":26},"in_reply_to":"dfee9bbb_65814ed1","updated":"2026-07-14 18:10:29.000000000","message":"I did a group review upstream with some folks and basically everyone \"oh, we\u0027re surely already doing this manually\"","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":24245,"name":"Harald Jensås","email":"hjensas@redhat.com","username":"harald.jensas"},"change_message_id":"344c16e83fc31206d478d6d5929151797c8f70af","unresolved":true,"context_lines":[{"line_number":586,"context_line":""},{"line_number":587,"context_line":"   .. code-block::"},{"line_number":588,"context_line":""},{"line_number":589,"context_line":"       ! DSCP-to-traffic-class mapping"},{"line_number":590,"context_line":"       ip access-list roce-traffic"},{"line_number":591,"context_line":"         permit udp any any eq 4791"},{"line_number":592,"context_line":"       class-map type qos roce"},{"line_number":593,"context_line":"         match ip access-group roce-traffic"},{"line_number":594,"context_line":"       policy-map type quality-of-service roce-qos"},{"line_number":595,"context_line":"         class roce"},{"line_number":596,"context_line":"           set traffic-class 3"},{"line_number":597,"context_line":""},{"line_number":598,"context_line":"       interface Ethernet5"},{"line_number":599,"context_line":"         mtu 9214"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9829bbb3_db8989fa","line":596,"range":{"start_line":589,"start_character":0,"end_line":596,"end_character":30},"updated":"2026-06-15 12:21:01.000000000","message":"Ditto.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":24245,"name":"Harald Jensås","email":"hjensas@redhat.com","username":"harald.jensas"},"change_message_id":"344c16e83fc31206d478d6d5929151797c8f70af","unresolved":true,"context_lines":[{"line_number":708,"context_line":"    openstack network set rdma-net --mtu 9000"},{"line_number":709,"context_line":""},{"line_number":710,"context_line":"    # Tag the network so trait-based networking can match it"},{"line_number":711,"context_line":"    openstack network set rdma-net --tag rdma"},{"line_number":712,"context_line":""},{"line_number":713,"context_line":"    # RDMA subnet: static IPs, no DHCP, no gateway"},{"line_number":714,"context_line":"    # (RDMA fabric is non-routed)"}],"source_content_type":"text/x-rst","patch_set":6,"id":"54f38716_da9091df","line":711,"range":{"start_line":711,"start_character":41,"end_line":711,"end_character":45},"updated":"2026-06-15 12:21:01.000000000","message":"This may be a bit of over-engeineering - but should we make plans for users that need different PFC and QoS priorities for different networks? We could make the tag configurable (example: `lossless_network_tags \u003d rdma, rdma-gpu, rdma-storage`) from the start, and later use this to apply different policies based on the tag and potentially also map this to selecting \"inter-switch-links\" in case the operator want \"no noisy neighbor\" for rdma and rdma-storage. Ther is also the case where one baremetal node may need eth1 for rdma and eth2 for rdma-storage and those NIC\u0027s may need a slightly different profile/config as well, AFICT that would work nicely with trait based networking and deploy templates as well.\n\nthe option lossless_network_tags as a list might not be ideal, we may want to map that to the profiles like below, or just `lossless_networks_types \u003d \u003ctag\u003e:\u003cprofile\u003e,\u003ctag\u003e:\u003cprofile\u003e`  ?\n\n```\n[lossless_profiles]\n# Tags that signal lossless switch port config\n# tag \u003d QoS profile name (references a pre-provisioned policy on the switch)\nrdma-gpu \u003d roce-gpu-qos\nrdma-storage \u003d roce-storage-qos\nrdma \u003d roce-default-qos\n```","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"20e0fb13e4748f76fc778735c475cbac1a31b08c","unresolved":true,"context_lines":[{"line_number":708,"context_line":"    openstack network set rdma-net --mtu 9000"},{"line_number":709,"context_line":""},{"line_number":710,"context_line":"    # Tag the network so trait-based networking can match it"},{"line_number":711,"context_line":"    openstack network set rdma-net --tag rdma"},{"line_number":712,"context_line":""},{"line_number":713,"context_line":"    # RDMA subnet: static IPs, no DHCP, no gateway"},{"line_number":714,"context_line":"    # (RDMA fabric is non-routed)"}],"source_content_type":"text/x-rst","patch_set":6,"id":"df1c8422_aa3f88f6","line":711,"range":{"start_line":711,"start_character":41,"end_line":711,"end_character":45},"in_reply_to":"54f38716_da9091df","updated":"2026-07-14 14:08:08.000000000","message":"How would we translate the profile?\n\nStrictly speaking, we don\u0027t need RDMA for NVMe, but I guess the quesiton is what does the profiles match to and what are we doing there. Hwo does that get turned into config? Is it just another option we\u0027re shipping along in port binding when the time comes?","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"3198509cf74f2fd5afcd5f09a6aadda77c46d743","unresolved":true,"context_lines":[{"line_number":708,"context_line":"    openstack network set rdma-net --mtu 9000"},{"line_number":709,"context_line":""},{"line_number":710,"context_line":"    # Tag the network so trait-based networking can match it"},{"line_number":711,"context_line":"    openstack network set rdma-net --tag rdma"},{"line_number":712,"context_line":""},{"line_number":713,"context_line":"    # RDMA subnet: static IPs, no DHCP, no gateway"},{"line_number":714,"context_line":"    # (RDMA fabric is non-routed)"}],"source_content_type":"text/x-rst","patch_set":6,"id":"6ea73eea_ed52f98a","line":711,"range":{"start_line":711,"start_character":41,"end_line":711,"end_character":45},"in_reply_to":"85e47a35_84f8b28f","updated":"2026-07-14 18:10:29.000000000","message":"I\u0027ll ping you on slack, but a question is how do we want to map/flag the features. I\u0027m wondering if we add extra values to port_binding for each specific high level flag, and potentially enable things like dscp, and qos grouping to be explicitly settable via the trait based networking configuration.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":24245,"name":"Harald Jensås","email":"hjensas@redhat.com","username":"harald.jensas"},"change_message_id":"04d2d8cc0747d3f142872f470cceda9ce0ba85f4","unresolved":true,"context_lines":[{"line_number":708,"context_line":"    openstack network set rdma-net --mtu 9000"},{"line_number":709,"context_line":""},{"line_number":710,"context_line":"    # Tag the network so trait-based networking can match it"},{"line_number":711,"context_line":"    openstack network set rdma-net --tag rdma"},{"line_number":712,"context_line":""},{"line_number":713,"context_line":"    # RDMA subnet: static IPs, no DHCP, no gateway"},{"line_number":714,"context_line":"    # (RDMA fabric is non-routed)"}],"source_content_type":"text/x-rst","patch_set":6,"id":"85e47a35_84f8b28f","line":711,"range":{"start_line":711,"start_character":41,"end_line":711,"end_character":45},"in_reply_to":"df1c8422_aa3f88f6","updated":"2026-07-14 16:52:54.000000000","message":"``[lossless_profieles]/rdma-gpu \u003d roce-gpu-qos`` - this would map to a operator defined pre-configured policy.\n\n```\n       ! Classify RoCE traffic by DSCP\n       class-map type qos match-all roce-traffic\n         match dscp 26\n       policy-map type qos roce-gpu-qos\n         class roce-traffic\n           set qos-group 3\n```\n\nWhen ngs binds the port it has the tag on the network, and it has the config - so it ``service-policy type qos input \u003cpolicy-picked-by-\"tag + config\"-lookup\u003e``.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"c67edf86775f61b46f5dc797c8c96c0341180b6e","unresolved":true,"context_lines":[{"line_number":737,"context_line":"cloud-init on the deployed OS will configure the interface"},{"line_number":738,"context_line":"with that static IP. No DHCP server is needed."},{"line_number":739,"context_line":""},{"line_number":740,"context_line":"**Why tag the network?** The tag ``rdma`` is used in the"},{"line_number":741,"context_line":"trait-based networking configuration (Step 4) to match this"},{"line_number":742,"context_line":"network when deciding which physical port to attach."},{"line_number":743,"context_line":""},{"line_number":744,"context_line":"**Step 2: Register nodes with RDMA capability**"},{"line_number":745,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"d9bee4a6_bc0c40ce","line":742,"range":{"start_line":740,"start_character":0,"end_line":742,"end_character":52},"updated":"2026-07-14 18:07:04.000000000","message":"All we need to be able to do is match and *signal* across the fact that the network is being used for RDMA.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"c67edf86775f61b46f5dc797c8c96c0341180b6e","unresolved":true,"context_lines":[{"line_number":858,"context_line":""},{"line_number":859,"context_line":".. code-block:: yaml"},{"line_number":860,"context_line":""},{"line_number":861,"context_line":"    # /etc/ironic/trait_networking.yaml"},{"line_number":862,"context_line":"    CUSTOM_RDMA_ROCE:"},{"line_number":863,"context_line":"      actions:"},{"line_number":864,"context_line":"        - action: attach_port"},{"line_number":865,"context_line":"          filter: \u003e-"},{"line_number":866,"context_line":"            port.category \u003d\u003d \u0027CUSTOM_RDMA\u0027"},{"line_number":867,"context_line":"            \u0026\u0026 network.tags \u003d\u003d \u0027rdma\u0027"},{"line_number":868,"context_line":""},{"line_number":869,"context_line":"This rule says: when a VIF attachment is requested for a node"},{"line_number":870,"context_line":"with the ``CUSTOM_RDMA_ROCE`` trait, find a port whose category"},{"line_number":871,"context_line":"is ``CUSTOM_RDMA`` and attach it to the network tagged ``rdma``."}],"source_content_type":"text/x-rst","patch_set":6,"id":"882c5524_482ee9ed","line":868,"range":{"start_line":861,"start_character":19,"end_line":868,"end_character":0},"updated":"2026-07-14 18:07:04.000000000","message":"We likely need a way to *explicitly* signal an additional action before attach_port, to set or ensure the required networking properties ECN/etc.\n\nThese wouldn\u0027t be settings on a network, but settings on the port which need to translate across in port binding.\n\ni.e. they need to be picked up in the ATTACH.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"},{"author":{"_account_id":11655,"name":"Julia Kreger","email":"juliaashleykreger@gmail.com","username":"jkreger","status":"Flying to the moon with a Jetpack!"},"change_message_id":"c67edf86775f61b46f5dc797c8c96c0341180b6e","unresolved":true,"context_lines":[{"line_number":1192,"context_line":"   This would also enable proper RBAC, input validation, and"},{"line_number":1193,"context_line":"   change auditing."},{"line_number":1194,"context_line":""},{"line_number":1195,"context_line":"6. **Scheduling affinity for RDMA co-location**: When RDMA"},{"line_number":1196,"context_line":"   nodes are on the same leaf switch, traffic stays within"},{"line_number":1197,"context_line":"   the switch as a local VLAN and PFC covers the entire"},{"line_number":1198,"context_line":"   path end-to-end, eliminating the VXLAN underlay"},{"line_number":1199,"context_line":"   lossless gap. Co-locating RDMA workloads on the same"},{"line_number":1200,"context_line":"   leaf (or leaf pair) is the recommended topology for"},{"line_number":1201,"context_line":"   latency-sensitive deployments."},{"line_number":1202,"context_line":""},{"line_number":1203,"context_line":"   There is a scale modeling challenge here. Nova\u0027s"},{"line_number":1204,"context_line":"   scheduling primitives (host aggregates, availability"}],"source_content_type":"text/x-rst","patch_set":6,"id":"76c9bd1a_25440dd5","line":1201,"range":{"start_line":1195,"start_character":0,"end_line":1201,"end_character":33},"updated":"2026-07-14 18:07:04.000000000","message":"Ironic has had this ask for approximately: forever, per Jay.","commit_id":"a6236155700f65477d5baf995bf71688a280111f"}]}