)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"d3f3f7ed1338e37bdb8101ca9b15f8f4eadfb285","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"52e71776_ab54daed","updated":"2022-08-11 23:47:24.000000000","message":"Confirmed with clarkb the concerns are mostly around just blindly switching things.  In this case I think we\u0027ve checked it out, tried to report it as best we can, so moving on with this.","commit_id":"cc2dd16d3a7194a4185ad6e1da854cb4fde01b1c"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"42441342ac40a6bf09bed97ef9ca59b6ca30b6ae","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"ad0e5d04_e0e423f6","updated":"2022-08-10 21:44:56.000000000","message":"Consider this half a -1. We\u0027ve changed these upstreams constantly in order to try and avoid these problems. We were assured that facebook was as stable as it gets. It would be helpful if we could make a deliberate decision rather than just changing them periodically.\n\nWould it be possible for those that need these packages to examine the mirrors and try to make an informed decision on which would be best for us to consume from? I thought this had already happened with the switch to facebook but either it didn\u0027t or facebook is suddenly doing less well than previously expected.","commit_id":"cc2dd16d3a7194a4185ad6e1da854cb4fde01b1c"},{"author":{"_account_id":8449,"name":"Marios Andreou","email":"marios.andreou@gmail.com","username":"marios"},"change_message_id":"accd11700407b01f05a1ff3d994735eb5bd5910c","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"fd4ef013_9836b294","in_reply_to":"6dee4a51_1c90cee6","updated":"2022-08-11 09:00:55.000000000","message":"+1 ack on clark\u0027s concerns but not sure what else we can do right now to unblock ourselves.\n\nthe situation was obviously different when our team proposed the patch that this reverts with rax \u003c facebook. That is now flipped and even now hours after we filed the bug the issue is ongoing and blocks the tripleo master gate with https://bugs.launchpad.net/tripleo/+bug/1984175/comments/9\n\nthere is no guarantee that we won\u0027t have this issue again if rax is out of sync. A few hours is OK but anything close to 1 day and more is always going to be problematic with folks needing to merge things for deadlines/releases etc. \n\nRight now rax seems to be OK so can we please merge this to unblock us? Moving forward is it only those two facebook vs rax  - are there any others we can try?","commit_id":"cc2dd16d3a7194a4185ad6e1da854cb4fde01b1c"},{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"9f3625698d6f8717d1e61acfb92ab4f620a75284","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"df3992cc_5273d1e7","in_reply_to":"ad0e5d04_e0e423f6","updated":"2022-08-10 22:17:11.000000000","message":"\u003e We were assured that facebook was as stable as it gets.\n\nHonestly I\u0027m not sure that was ever assured; we don\u0027t have any insight into how it is run.  It just seemed to work at the time better than others, and was fast.  We were using the utexas one successfully for quite a while, where we at least had a sometimes on IRC contact, but that had some sort of disk issue and was broken until they could get more hardware.  I may have expressed my feeling that fb may be many things, but running out of disks probably isn\u0027t one of them 😊\n\nI haven\u0027t looked and will try to find time, but last time thought the mirror was out of sync, it was actually an issue with upstream (https://pagure.io/centos-infra/issue/814).  So it\u0027s worth digging a bit to understand what\u0027s going on.","commit_id":"cc2dd16d3a7194a4185ad6e1da854cb4fde01b1c"},{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"e201f311946164dd85e1b2cae803f1d796994fc2","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"6dee4a51_1c90cee6","in_reply_to":"df3992cc_5273d1e7","updated":"2022-08-11 02:22:48.000000000","message":"After investigation, it does look like the mirror is out of sync.\n\nhttps://meetings.opendev.org/irclogs/%23opendev/%23opendev.2022-08-11.log.html#t2022-08-11T01:10:44\n\nWhen the RAX mirrors weren\u0027t responding to rsync in april, I sent mail to the mirror-admin list.  I never got a response and I don\u0027t know if it got moderated out, it doesn\u0027t appear in the archive\n\nhttps://lists.fedoraproject.org/archives/list/mirror-admin@lists.fedoraproject.org/2022/4/\n\nSo I remain unsure if anyone noticed that and got it fixed, or it just got fixed on its own.  On the off chance someone did do something, I have sent an email about this situation to the same list.\n\nWe don\u0027t have the space or the need to run a full mirror, so we are at the mercy of other peoples admin of their mirrors.  I do admit communication around all this is terrible.  But I\u0027m not sure what else we can do but try to reach out, and switch ourselves in situations like this.  \n\nAnd to correct myself, it was actually math.uh.edu, the conversation doesn\u0027t seem logged but it was around\n\n Apr 13 12:42:27 \u003ctibbs\u003e ianw:  Sadly pubmirror1 is out of date for alt but up to date for everything else, because the repository got way too big.  New disks are on the way.\n \nthat may also be back now; it was stable before this.","commit_id":"cc2dd16d3a7194a4185ad6e1da854cb4fde01b1c"},{"author":{"_account_id":16312,"name":"Alfredo Moralejo","email":"amoralej@redhat.com","username":"amoralej"},"change_message_id":"bfb6ad40ae09fdb3a38b326f6b62346553f70521","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"3b15cf3d_078406a0","in_reply_to":"fd4ef013_9836b294","updated":"2022-08-11 09:52:17.000000000","message":"I understand the concern but mirrors do not provide SLAs and so it\u0027s hard to look for any warranty. Facebook mirror has been working pretty well for some time, they are commited to CentOS so we could expect to be a healthy one but that\u0027s not the case now. \nI\u0027m trying to contact the facebook engineer who is part of CentOS Board as i expect he can help us to reach the right team maintaining the mirror.","commit_id":"cc2dd16d3a7194a4185ad6e1da854cb4fde01b1c"}]}