)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"0711201369b3a582d320f566137679009f47cc67","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":6,"id":"6a190b95_773b63e1","updated":"2026-07-23 22:46:39.000000000","message":"The root cause confirmed: with RemoteIPProxyProtocol On, mod_remoteip drops any header-less connection whose source isn\u0027t in  RemoteIPProxyProtocolExceptions. The functional tests hit the vhost directly over localhost  - ::1, so those connections carry no PROXY header and get dropped - which is what fails system-config-run-gitea. Excepting ::1 alongside 127.0.0.1 (this patchset) fixes it; haproxy stays un-excepted so mod_remoteip still recovers the real client address for proxied traffic.","commit_id":"55c55bb66e4b25db9adb1c609d5eea190a4651b2"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"5b3afa28d955cb62e33f104a50a3a50c13cba5a6","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":6,"id":"96ea7fe1_7fa047fb","in_reply_to":"6a190b95_773b63e1","updated":"2026-07-23 22:54:07.000000000","message":"Verified two ways in a standalone reproduction of the chain (haproxy -\u003e apache/mod_remoteip -\u003e anubis -\u003e gitea):\n\n1. Deterministic toggle - exceptions  127.0.0.1 : request to  [::1]  gets an empty reply (dropped); exceptions 127.0.0.1 ::1 : both [::1] and 127.0.0.1 return 200. That\u0027s exactly the gate\u0027s localhost path.\n\n2. Across two separate hosts (haproxy and the backend on different machines, PROXY protocol over the wire): the backend recovered the client address from the PROXY header and it landed in Gitea\u0027s access log -\n10.77.0.100 XFF\u003d[...] XRI\u003d[10.77.0.100]  \"GET /...\"\ni.e. mod_remoteip does surface the client info this change is after.\n\nReproduction: https://github.com/askb/gitea-proxy-poc - the bug-missing-ipv6  (RED) and  fix-with-ipv6  (GREEN) workflows are the ::1 toggle; multi-runner-chain is the two-host proof.","commit_id":"55c55bb66e4b25db9adb1c609d5eea190a4651b2"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"05f4f35ffd08719f03b08722b74cdc9e2ae60729","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"648f09f1_4d075b4c","in_reply_to":"96ea7fe1_7fa047fb","updated":"2026-07-27 09:36:09.000000000","message":"Done","commit_id":"55c55bb66e4b25db9adb1c609d5eea190a4651b2"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"2cd2c54699c97887b1b0b142e8403ddd1b0c5a4c","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":8,"id":"9cb3bf84_b691fb0b","updated":"2026-07-27 07:15:17.000000000","message":"The bind fix worked. AH03500 is now zero occurrences in gitea-ssl-error.log, and haproxy logs the client as native 127.0.0.1 (vs ::ffff: on the :80 listener, which I left dual-stack - a nice built-in control).\n\nA second bug was hiding underneath it. The same two tests now fail with curl exit 35 (SSL connect error) - and this time Apache logged nothing at all. haproxy logged all 4101 connections as CD (client abort) after 494 bytes.\n\nCause:  proxy_string: \u0027ssl send-proxy\u0027 .\n\nserver gitea99 \u003cip\u003e:3081 check check-ssl ca-file … ssl send-proxy\n                                                   ^^^","commit_id":"990bd31615fec48ba7b5b5ccbe8352dc3d8d17a1"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"4893723d6d700eda706d4d2445e3a4d3f814e183","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":8,"id":"f60eefd2_0c09d702","in_reply_to":"9cb3bf84_b691fb0b","updated":"2026-07-27 09:36:38.000000000","message":"Done","commit_id":"990bd31615fec48ba7b5b5ccbe8352dc3d8d17a1"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"05f4f35ffd08719f03b08722b74cdc9e2ae60729","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":9,"id":"d4f79448_5707de6a","updated":"2026-07-27 09:36:09.000000000","message":"PS9 is Verified+1 -- system-config-run-gitea SUCCESS in 26m30s.\n\nMy PS8 comment got truncated mid way, so to finish it: the second bug was\nthe \u0027ssl\u0027 keyword in proxy_string.\n\n    server gitea99 \u003cip\u003e:3081 check check-ssl ca-file ... ssl send-proxy\n                                                        ^^^\n\nThe frontend bind has no ssl/crt, so haproxy never terminates TLS -- in mode\ntcp it relays bytes and TLS is end-to-end client \u003c-\u003e gitea\u0027s Apache. \u0027ssl\u0027 on\nthe *server* line made haproxy open a second TLS session to the backend and\nwrite the client\u0027s already-encrypted bytes into it, so Apache decrypted the\nouter layer, read the ClientHello as a plaintext request and answered 400 in\ncleartext. Hence curl exit 35 and a silent error log -- the PROXY header itself\nparsed fine. master never had \u0027ssl\u0027 here either. PS9 drops it; check-ssl is\nuntouched, so health checks still speak TLS.\n\nClient-IP recovery demonstrably works now, not just \"tests stopped failing\":\nPS9\u0027s gitea-ssl-access.log has 4098 requests that reached gitea99 on its\nprivate address (10.0.16.122, i.e. via the LB) logged with client 127.0.0.1 --\nfrom the PROXY header, not the LB\u0027s own 10.0.17.188 -- all 200. Same query on\nPS8 returns 0.\n\nBoth bugs reproduced standalone, one variable each:\n  https://github.com/askb/gitea-proxy-poc/actions/workflows/bind-bug-ah03500.yaml\n  https://github.com/askb/gitea-proxy-poc/actions/workflows/tls-doublewrap.yaml\n\nOne open question before this merges: no RemoteIPHeader is set anywhere in the\nvhost, so the RemoteIPTrustedProxy lines this change adds are inert -- they\nonly drive the XFF chain-walk, which the PROXY path never touches. Drop them,\nor keep as groundwork for a future XFF setup? I\u0027ll respin either way.\n\n(system-config-run-base-ansible-devel POST_FAILURE is non-voting and identical\non PS6-9, unrelated.)","commit_id":"f7eb1ac4a1314f989fab0a89e71961ff77c75dcb"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"8287d41650a5ccfcd4d4b14a4b19d55e1c97aaeb","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":9,"id":"2e2eb8ed_db9191ba","updated":"2026-07-27 21:59:34.000000000","message":"To trace a connection and confirm this seems to improve the logging situation we can look at the haproxy log here: https://zuul.opendev.org/t/openstack/build/adfe0fbeb9a844a9975aace1b60f5160/log/gitea-lb03.opendev.org/haproxy.log#3964 and look at this specific request from 127.0.0.1 on the load balancer to apache using the client side port 46216. Then in the apache log we can see that the client is logged as 127.0.0.1:46216 here https://zuul.opendev.org/t/openstack/build/adfe0fbeb9a844a9975aace1b60f5160/log/gitea99.opendev.org/apache2/gitea-ssl-access.log#3752 rather than 50.56.157.54 or 10.0.17.188 which are the ip addresses of the load balancer.\n\nI think that shows this is working.","commit_id":"f7eb1ac4a1314f989fab0a89e71961ff77c75dcb"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"a8299b1a57b444556351a25089bebace49e9fcaf","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":9,"id":"90be6085_8c1e0d18","in_reply_to":"d4f79448_5707de6a","updated":"2026-07-28 11:47:11.000000000","message":"Done","commit_id":"f7eb1ac4a1314f989fab0a89e71961ff77c75dcb"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"3062c63460438ecdc35ac9f550242f0923738456","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":10,"id":"6dd77b26_0520450c","updated":"2026-07-28 06:47:41.000000000","message":"recheck","commit_id":"8a859927556d923c7e43f486aa070d542845d9f4"}],"playbooks/roles/gitea-lb/templates/gitea-haproxy.cfg.j2":[{"author":{"_account_id":22629,"name":"Michal Nasiadka","email":"mnasiadka@gmail.com","username":"mnasiadka"},"change_message_id":"47be615647789954101cc44a13ec961eaeef2fd0","unresolved":true,"context_lines":[{"line_number":30,"context_line":"  option httpchk GET /api/healthz"},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"  {% for server in listener.servers %}"},{"line_number":33,"context_line":"  server {{ server.name }} {{ server.address }} {{ server.check_method }} {{ server.proxy_string }}"},{"line_number":34,"context_line":"  {% endfor %}"},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"{% endfor %}"}],"source_content_type":"text/x-jinja2","patch_set":10,"id":"14205be0_cdc54ad9","line":33,"range":{"start_line":33,"start_character":74,"end_line":33,"end_character":99},"updated":"2026-07-28 09:17:18.000000000","message":"If we add | default(\u0027\u0027) then we can skip the empty declaration in group_vars","commit_id":"8a859927556d923c7e43f486aa070d542845d9f4"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"3d9f024e98a2233b274b45e673c485393fb84097","unresolved":false,"context_lines":[{"line_number":30,"context_line":"  option httpchk GET /api/healthz"},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"  {% for server in listener.servers %}"},{"line_number":33,"context_line":"  server {{ server.name }} {{ server.address }} {{ server.check_method }} {{ server.proxy_string }}"},{"line_number":34,"context_line":"  {% endfor %}"},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"{% endfor %}"}],"source_content_type":"text/x-jinja2","patch_set":10,"id":"53086dae_de10df81","line":33,"range":{"start_line":33,"start_character":74,"end_line":33,"end_character":99},"in_reply_to":"14205be0_cdc54ad9","updated":"2026-07-28 11:46:40.000000000","message":"Good call, done in PS11.\n\nAdded `| default(\u0027\u0027)` to the server line and dropped the seven now\nredundant empty declarations: six in the gitea-lb group_vars and one in\nthe Zuul test group_vars template. Net -7 lines, and the http listener\nblock is back to being a plain list of hosts.\n\nVerified the rendered haproxy.cfg is byte-identical before and after,\nfor both the production inventory and the test group_vars, so this is a\npure readability change with no behaviour delta.","commit_id":"8a859927556d923c7e43f486aa070d542845d9f4"}],"playbooks/roles/gitea/templates/gitea.vhost.j2":[{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"c10667793e4c949f2820a0c7d7df7e2e0d5fbf76","unresolved":true,"context_lines":[{"line_number":31,"context_line":"  # because `send-proxy` also prepends the PROXY header to httpchk"},{"line_number":32,"context_line":"  # connections."},{"line_number":33,"context_line":"  RemoteIPProxyProtocol On"},{"line_number":34,"context_line":"  RemoteIPProxyProtocolExceptions 127.0.0.1 ::1"},{"line_number":35,"context_line":"  {% for addr in groups.get(\u0027gitea-lb\u0027) | map(\u0027extract\u0027, hostvars, \u0027public_v4\u0027) -%}"},{"line_number":36,"context_line":"  {% if addr -%}"},{"line_number":37,"context_line":"  RemoteIPTrustedProxy {{ addr }}"}],"source_content_type":"text/x-jinja2","patch_set":7,"id":"cfd212be_27c6dc49","line":34,"updated":"2026-07-26 22:59:04.000000000","message":"We are still getting errors like AH03500: RemoteIPProxyProtocol: invalid client-address \u0027::ffff:\n127.0.0.1\u0027 found in header \u0027PROXY TCP6 ::ffff:127.0.0.1 ::ffff:127.0.0.1 52380 443\u0027 in https://zuul.opendev.org/t/openstack/build/fa8b83088f6143539657f3b142886a93/log/gitea99.opendev.org/apache2/gitea-ssl-error.log.\n\nI think the issue is that when we run the functional testinfra tests against the haproxy load balancer (test cases like test_haproxy_gitea_connection and test_more_than_haproxy_maxconn_conns) we are running curl against 127.0.0.1:443 on the haproxy server. This sets the client ip to 127.0.0.1 which is blocked by default by RemoteIPTrustedProxy. Instead we would need to use RemoteIPInternalProxy to trust clients coming from private ranges. But that may not be appropriate in the production deployment so I\u0027m not sure we want to use RemoteIPInternalProxy here.\n\nInstead I wonder if we can run the curl requests in those two test cases against the public IP address so that our client IP address is also a public IP address? This may not work in clouds that use floating IPs though. Maybe it is safe to use RemoteIPInternalProxy? https://httpd.apache.org/docs/current/mod/mod_remoteip.html#remoteipinternalproxy doesn\u0027t really explain what the main concerns are there.\n\nAs a fallback we could set RemoteIPInternalProxy only if deployed within the test environment.","commit_id":"f3e9afe533940c3bbecaed53b1134854a4ffe6ad"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"4893723d6d700eda706d4d2445e3a4d3f814e183","unresolved":false,"context_lines":[{"line_number":31,"context_line":"  # because `send-proxy` also prepends the PROXY header to httpchk"},{"line_number":32,"context_line":"  # connections."},{"line_number":33,"context_line":"  RemoteIPProxyProtocol On"},{"line_number":34,"context_line":"  RemoteIPProxyProtocolExceptions 127.0.0.1 ::1"},{"line_number":35,"context_line":"  {% for addr in groups.get(\u0027gitea-lb\u0027) | map(\u0027extract\u0027, hostvars, \u0027public_v4\u0027) -%}"},{"line_number":36,"context_line":"  {% if addr -%}"},{"line_number":37,"context_line":"  RemoteIPTrustedProxy {{ addr }}"}],"source_content_type":"text/x-jinja2","patch_set":7,"id":"95cb449a_271fefd7","line":34,"in_reply_to":"45b11aa2_04a9c112","updated":"2026-07-27 09:36:38.000000000","message":"Done","commit_id":"f3e9afe533940c3bbecaed53b1134854a4ffe6ad"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"d16f8a5dba469841def37661c4cb22723775f3a9","unresolved":false,"context_lines":[{"line_number":31,"context_line":"  # because `send-proxy` also prepends the PROXY header to httpchk"},{"line_number":32,"context_line":"  # connections."},{"line_number":33,"context_line":"  RemoteIPProxyProtocol On"},{"line_number":34,"context_line":"  RemoteIPProxyProtocolExceptions 127.0.0.1 ::1"},{"line_number":35,"context_line":"  {% for addr in groups.get(\u0027gitea-lb\u0027) | map(\u0027extract\u0027, hostvars, \u0027public_v4\u0027) -%}"},{"line_number":36,"context_line":"  {% if addr -%}"},{"line_number":37,"context_line":"  RemoteIPTrustedProxy {{ addr }}"}],"source_content_type":"text/x-jinja2","patch_set":7,"id":"c236ad6d_a511404b","line":34,"in_reply_to":"95cb449a_271fefd7","updated":"2026-07-27 22:01:00.000000000","message":"\u003e One follow-up worth a decision: with no RemoteIPHeader set, the RemoteIPTrustedProxy lines this change adds are effectively inert. Happy to drop them or keep them for a future XFF setup - whichever you prefer.\n\nDoes this mean that `RemoteIPProtocolProxy On` is sufficient to get the logging we want? If so then I think cleaning this up to avoid future confusion is a good idea.","commit_id":"f3e9afe533940c3bbecaed53b1134854a4ffe6ad"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"fd29400edf5cd752e63068e9d980683abc1a21bd","unresolved":false,"context_lines":[{"line_number":31,"context_line":"  # because `send-proxy` also prepends the PROXY header to httpchk"},{"line_number":32,"context_line":"  # connections."},{"line_number":33,"context_line":"  RemoteIPProxyProtocol On"},{"line_number":34,"context_line":"  RemoteIPProxyProtocolExceptions 127.0.0.1 ::1"},{"line_number":35,"context_line":"  {% for addr in groups.get(\u0027gitea-lb\u0027) | map(\u0027extract\u0027, hostvars, \u0027public_v4\u0027) -%}"},{"line_number":36,"context_line":"  {% if addr -%}"},{"line_number":37,"context_line":"  RemoteIPTrustedProxy {{ addr }}"}],"source_content_type":"text/x-jinja2","patch_set":7,"id":"eae95d55_30b374c0","line":34,"in_reply_to":"c236ad6d_a511404b","updated":"2026-07-28 04:11:58.000000000","message":"Yes, RemoteIPProxyProtocol On is sufficient - dropped the\nRemoteIPTrustedProxy lines in PS10.\n\nRemoteIPTrustedProxy only gates the header-based path (RemoteIPHeader,\ne.g. X-Forwarded-For), which we don\u0027t set. The PROXY protocol path is\ngated by RemoteIPProxyProtocolExceptions instead, so those lines were\nnever consulted.\n\nChecked it rather than trusting the docs — same Apache four ways, real\nclient 172.30.1.100, haproxy 172.30.1.13, logging %a:\n\n  ProxyProtocol On, no TrustedProxy        -\u003e 172.30.1.100\n  ProxyProtocol On + TrustedProxy \u003clb\u003e     -\u003e 172.30.1.100\n  ProxyProtocol On + TrustedProxy 10.99.99.99  -\u003e 172.30.1.100 (!)\n  TrustedProxy only, no ProxyProtocol      -\u003e connection fails\n\nThe wrong-value case is the conclusive one. Details:\nhttps://github.com/askb/gitea-proxy-poc/tree/main/remoteip-trustedproxy\n\nPS10 leaves a comment explaining why there\u0027s no RemoteIPTrustedProxy so\nit doesn\u0027t come back. Trust is still scoped:\niptables_extra_allowed_groups already limits 3081 to the gitea-lb group.","commit_id":"f3e9afe533940c3bbecaed53b1134854a4ffe6ad"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"d9ca3fad7a75bb1bfc95dff8aebca51c108e8339","unresolved":true,"context_lines":[{"line_number":31,"context_line":"  # because `send-proxy` also prepends the PROXY header to httpchk"},{"line_number":32,"context_line":"  # connections."},{"line_number":33,"context_line":"  RemoteIPProxyProtocol On"},{"line_number":34,"context_line":"  RemoteIPProxyProtocolExceptions 127.0.0.1 ::1"},{"line_number":35,"context_line":"  {% for addr in groups.get(\u0027gitea-lb\u0027) | map(\u0027extract\u0027, hostvars, \u0027public_v4\u0027) -%}"},{"line_number":36,"context_line":"  {% if addr -%}"},{"line_number":37,"context_line":"  RemoteIPTrustedProxy {{ addr }}"}],"source_content_type":"text/x-jinja2","patch_set":7,"id":"45b11aa2_04a9c112","line":34,"in_reply_to":"cfd212be_27c6dc49","updated":"2026-07-27 03:55:31.000000000","message":"Good catch, and thanks for digging into the logs.\n\nI went through mod_remoteip\u0027s source and I think the mechanism is a bit different from the trust one, which changes what the fix needs to be.\n\nAH03500 is raised in remoteip_process_v1_header() while parsing the PROXY header, before any trust evaluation happens. For a TCP6 header, the address is validated with strspn() against \"0123456789abcdefABCDEF:\" - no  . . So ::ffff:127.0.0.1 fails purely on the charset, and the connection is aborted (f-\u003ec-\u003eaborted \u003d 1, APR_ECONNABORTED), which is why we see an empty \nreply rather than a wrong client IP. Only unreleased httpd trunk adds . to that set; every released 2.4.x rejects it.\n\nThat also means RemoteIPTrustedProxy / RemoteIPInternalProxy can\u0027t rescue this: those lists drive the RemoteIPHeader (X-Forwarded-For) chain-walk. The PROXY-protocol path sets client_addr / client_ip directly from the parsed header and never consults them, so there\u0027s no private-range rejection to relax.\n\nThe reason a v4 client produces a TCP6 header at all is the load balancer bind: balance_git_https binds a single dual-stack :::443, so the kernel presents IPv4 peers as IPv4-mapped IPv6 and send-proxy forwards that verbatim.\n\nWhich also means this isn\u0027t test-only - every real IPv4 visitor arrives as  ::ffff:\u003caddr\u003e  too, so as written this change would abort IPv4 clients to opendev.org in production. The failing testinfra cases are the canary rather than the problem.\nPS8 fixes it at the bind instead:\n-    bind :::443\n+    bind 0.0.0.0:443\n+    bind :::443 v6only\n\nv4 peers now arrive on a native v4 socket, so haproxy emits  PROXY TCP4 127.0.0.1 ...  which mod_remoteip accepts ( .  is valid in the TCP4 charset); v6 peers still get a native TCP6 header. No testinfra changes, no InternalProxy, and no dependency on public vs private addressing or floating IPs.\n\nVerified in a standalone repro where the only variable is the bind style: the dual-stack bind reproduces the exact AH03500 with the  ::ffff:  header and an aborted request, and the split bind returns 200 with the real client IP recovered — https://github.com/askb/gitea-proxy-poc/actions/workflows/bind-bug-ah03500.yaml\n\nOne follow-up worth a decision: with no RemoteIPHeader set, the RemoteIPTrustedProxy lines this change adds are effectively inert. Happy to drop them or keep them for a future XFF setup - whichever you prefer.","commit_id":"f3e9afe533940c3bbecaed53b1134854a4ffe6ad"}]}