)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":1955,"name":"Alan Pevec","email":"alan.pevec@redhat.com","username":"apevec"},"change_message_id":"196f9f429a30838cbad2e0573092a98f862d4bbd","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"d66be9e7_1248099f","updated":"2022-01-21 11:48:46.000000000","message":"\u003e I\u0027ve seen no indication that switching to Facebook\u0027s mirror would have made any difference.\n\nwe do, please see Alfredo\u0027s comment https://bugs.launchpad.net/tripleo/+bug/1958510/comments/5\nAlso for consistency with CS9 mirroring.\n\n","commit_id":"3fd5bba62b81767ddb7bdd3365d8d971c7ea3d7e"},{"author":{"_account_id":12393,"name":"chandan kumar","display_name":"Chandan Kumar","email":"chkumar@redhat.com","username":"chkumar246"},"change_message_id":"b9296ec93ad24b3a7a7bd5b28298b3d0a47524a9","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"ba1e3137_5d8df021","updated":"2022-01-21 07:22:05.000000000","message":"\u003e Patch Set 2:\n\u003e \n\u003e To all those +1\u0027ing this change, please note that it\u0027s currently suspected the problem was global across all CentOS repositories. We have no reason to believe that there\u0027s anything going wrong wit the Leaseweb mirror we\u0027re syncing from, it looks like it simply copied an index change from wherever it\u0027s synced, and then copied a revert undoing that index change roughly 6 hours later. I\u0027ve seen no indication that switching to Facebook\u0027s mirror would have made any difference.\n\nThank you Jeremy for looking into this, based on this https://review.opendev.org/c/opendev/system-config/+/825446/2#message-9bc5359ea0811ef3191fbf2c4ebe41e135003182 something strange happened on CentOS side.\n\nOne suggestion is to use known and validated mirror. May be we can rely on facebook mirror.","commit_id":"3fd5bba62b81767ddb7bdd3365d8d971c7ea3d7e"},{"author":{"_account_id":12393,"name":"chandan kumar","display_name":"Chandan Kumar","email":"chkumar@redhat.com","username":"chkumar246"},"change_message_id":"4658b465d74f3c7bdd21726e02c956c81e7055af","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"ddaafdda_1b478a63","updated":"2022-01-21 08:27:20.000000000","message":"Added https://pagure.io/centos-infra/issue/609 to track it on centos side","commit_id":"3fd5bba62b81767ddb7bdd3365d8d971c7ea3d7e"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"4ff38c32a54a51b9388d2fb0e8967e53901fce71","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"9c9908d7_46375166","updated":"2022-01-21 18:46:48.000000000","message":"As noted in IRC I\u0027m not sure facebook will be significantly better? When I checked its contents against the limestone repo to followup on the systemd-239-55 update they seemed to be in sync with each other. That said I\u0027m happy to update this and get some hard data.","commit_id":"3fd5bba62b81767ddb7bdd3365d8d971c7ea3d7e"},{"author":{"_account_id":12393,"name":"chandan kumar","display_name":"Chandan Kumar","email":"chkumar@redhat.com","username":"chkumar246"},"change_message_id":"9bc5359ea0811ef3191fbf2c4ebe41e135003182","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"cf508cea_e4446fed","updated":"2022-01-21 07:20:11.000000000","message":"Copying the discussion from #centos-devel\n```\n08:09 \u003craukadah\u003e Hello #centos-devel\n08:09 \u003craukadah\u003e on OpenStack side, we rsync cs8 content to openstack afs mirrors\n08:09 \u003craukadah\u003e we are seeing 404 on  Status code: 404 for https://mirror-int.ord.rax.opendev.org/centos/8-stream/AppStream/x86_64/os/repodata/8436c3dcc9cb11facf086b1da2dcc\n1be03e96045dd53c872486e9044ec58d95b-primary.xml.gz (IP: 10.209.128.57)\n08:10 \u003c-- RemiFedora (~RemiFedor@fedora/remi) has quit (Read error: Connection reset by peer)\n08:10 \u003craukadah\u003e It rsyncs from mirror.dal10.us.leaseweb.net\n08:10 \u003craukadah\u003e we have seen this issue and it got resolved automatically but it is coming again today\n08:10 --\u003e RemiFedora (~RemiFedor@fedora/remi) has joined #centos-devel\n08:10 \u003craukadah\u003e can someone please look at mirror side\n08:10 \u003craukadah\u003e ? Thanks!\n08:12 \u003carrfab\u003e raukadah: we don\u0027t control when external mirrors are pulling from official mirrors either\n08:13 \u003craukadah\u003e arrfab: any specific offical mirror we can use to pull the content from?\n08:13 \u003craukadah\u003e to avoid the issue\n08:13 \u003carrfab\u003e we authorize public mirror to pull from us, but restricted for the others\n08:13 --\u003e vbenes (~vbenes@31.96.broadband16.iol.cz) has joined #centos-devel\n08:13 \u003carrfab\u003e why are you pulling from that particular leaseweb.net node ?\n08:14 \u003carrfab\u003e something you decided ?\n08:14 \u003craukadah\u003e based on this https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update#L42\n08:15 \u003craukadah\u003e there is no reason given but it somewhere in TX so we are using it\n08:15 \u003craukadah\u003e recently we have switched to facebook mirror for centos stream 9 content\n08:15 \u003carrfab\u003e probably worth sticking to one known and validated mirror that wouldn\u0027t have issues\n08:16 \u003carrfab\u003e but it seems there was a push during the night too\n08:16 \u003carrfab\u003e well, I take that back\n08:16 \u003carrfab\u003e nothing was pushed it seems\n08:16 \u003carrfab\u003e http://mirror.centos.org/centos/8-stream/COMPOSE_ID\n08:17 \u003craukadah\u003e http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026831.html \u003c- there was a  upheaval for CentOS Stream 8 package indices\n08:17 \u003carrfab\u003e which is strange .. as some pkgs clearly landed for BaseOS so worth verifying with hughesjr\n08:19 \u003craukadah\u003e arrfab: thank you for looking into this, will wait for hughesjr \u0027s reply :-) \n08:19 \u003carrfab\u003e something is b0rken, compose_id is from the past but even newer .iso images are pushed so you should ask hughesjr\n08:19 \u003craukadah\u003e In the mean time, we will ask openstack infra to switch to facebook mirror.\n\n```","commit_id":"3fd5bba62b81767ddb7bdd3365d8d971c7ea3d7e"},{"author":{"_account_id":16312,"name":"Alfredo Moralejo","email":"amoralej@redhat.com","username":"amoralej"},"change_message_id":"1d350531b3f9afa9dd689995c7d123c065dda75e","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"5d73badc_72069dc6","in_reply_to":"d66be9e7_1248099f","updated":"2022-01-21 11:57:37.000000000","message":"To be clear, i don\u0027t have evidences that it\u0027s not happening in other mirrors other that if that issue would be spread accross the entire network of mirrors, there would be many more complains (i guess). I sent a mail to the mirror admin of leaseweb but didn\u0027t receive any response.\n\nAlso, i agree that moving to facebook mirror would also be good for consistency with cs9.","commit_id":"3fd5bba62b81767ddb7bdd3365d8d971c7ea3d7e"}],"playbooks/roles/mirror-update/files/centos-mirror-update":[{"author":{"_account_id":16312,"name":"Alfredo Moralejo","email":"amoralej@redhat.com","username":"amoralej"},"change_message_id":"b6bdbec93c085d5e85bb29387c7b55b25dc04919","unresolved":true,"context_lines":[{"line_number":39,"context_line":"# -- Centos 8-stream --"},{"line_number":40,"context_line":"#"},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"MIRROR\u003drsync://https://mirror.facebook.net/centos/8-stream/centos/8-stream/ # somewhere in US"},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"if ! [ -f $BASE/8-stream ]; then"},{"line_number":45,"context_line":"    $K5START mkdir -p $BASE/8-stream"}],"source_content_type":"application/x-shellscript","patch_set":1,"id":"087cd75c_8478546b","line":42,"range":{"start_line":42,"start_character":43,"end_line":42,"end_character":58},"updated":"2022-01-20 10:01:01.000000000","message":"also this","commit_id":"9246324c24a6781e3d020999f05ec9e78324df57"},{"author":{"_account_id":16312,"name":"Alfredo Moralejo","email":"amoralej@redhat.com","username":"amoralej"},"change_message_id":"b6bdbec93c085d5e85bb29387c7b55b25dc04919","unresolved":true,"context_lines":[{"line_number":39,"context_line":"# -- Centos 8-stream --"},{"line_number":40,"context_line":"#"},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"MIRROR\u003drsync://https://mirror.facebook.net/centos/8-stream/centos/8-stream/ # somewhere in US"},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"if ! [ -f $BASE/8-stream ]; then"},{"line_number":45,"context_line":"    $K5START mkdir -p $BASE/8-stream"}],"source_content_type":"application/x-shellscript","patch_set":1,"id":"0d1f8dff_999702a3","line":42,"range":{"start_line":42,"start_character":15,"end_line":42,"end_character":23},"updated":"2022-01-20 10:01:01.000000000","message":"typo, remove \"https://\"","commit_id":"9246324c24a6781e3d020999f05ec9e78324df57"},{"author":{"_account_id":12393,"name":"chandan kumar","display_name":"Chandan Kumar","email":"chkumar@redhat.com","username":"chkumar246"},"change_message_id":"bb3a222a63bd44f5114bd4e0d5c2b807e439e082","unresolved":false,"context_lines":[{"line_number":39,"context_line":"# -- Centos 8-stream --"},{"line_number":40,"context_line":"#"},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"MIRROR\u003drsync://https://mirror.facebook.net/centos/8-stream/centos/8-stream/ # somewhere in US"},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"if ! [ -f $BASE/8-stream ]; then"},{"line_number":45,"context_line":"    $K5START mkdir -p $BASE/8-stream"}],"source_content_type":"application/x-shellscript","patch_set":1,"id":"4c00b58e_0ce8e8a1","line":42,"range":{"start_line":42,"start_character":43,"end_line":42,"end_character":58},"in_reply_to":"087cd75c_8478546b","updated":"2022-01-20 10:03:09.000000000","message":"Done","commit_id":"9246324c24a6781e3d020999f05ec9e78324df57"},{"author":{"_account_id":12393,"name":"chandan kumar","display_name":"Chandan Kumar","email":"chkumar@redhat.com","username":"chkumar246"},"change_message_id":"bb3a222a63bd44f5114bd4e0d5c2b807e439e082","unresolved":false,"context_lines":[{"line_number":39,"context_line":"# -- Centos 8-stream --"},{"line_number":40,"context_line":"#"},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"MIRROR\u003drsync://https://mirror.facebook.net/centos/8-stream/centos/8-stream/ # somewhere in US"},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"if ! [ -f $BASE/8-stream ]; then"},{"line_number":45,"context_line":"    $K5START mkdir -p $BASE/8-stream"}],"source_content_type":"application/x-shellscript","patch_set":1,"id":"f46c4d4c_89fa9818","line":42,"range":{"start_line":42,"start_character":15,"end_line":42,"end_character":23},"in_reply_to":"0d1f8dff_999702a3","updated":"2022-01-20 10:03:09.000000000","message":"Done","commit_id":"9246324c24a6781e3d020999f05ec9e78324df57"}]}