)]}'
{"doc/source/troubleshooting_ceph.rst":[{"author":{"_account_id":28618,"name":"Drew Walters","email":"drewwalters@microsoft.com","username":"drewwalters96"},"change_message_id":"61d66443088207ce90edef01d26b39e65c231b7b","unresolved":false,"context_lines":[{"line_number":12,"context_line":"      under the License."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"Troubleshooting Ceph"},{"line_number":15,"context_line":"####################"},{"line_number":16,"context_line":""},{"line_number":17,"context_line":".. contents:: Table of Contents"},{"line_number":18,"context_line":"    :depth: 3"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bf51134e_afc3bb37","line":15,"range":{"start_line":15,"start_character":0,"end_line":15,"end_character":20},"updated":"2020-06-22 19:56:52.000000000","message":"It seems like these headings should be reversed, but it renders nicely. What is this accounting for?","commit_id":"dbcf10f8d9df2fb23a4a52535d0fe4e91157b07c"},{"author":{"_account_id":28618,"name":"Drew Walters","email":"drewwalters@microsoft.com","username":"drewwalters96"},"change_message_id":"61d66443088207ce90edef01d26b39e65c231b7b","unresolved":false,"context_lines":[{"line_number":88,"context_line":""},{"line_number":89,"context_line":"You should keep in mind that losing a monitor, or a bunch of them, doesn\u0027t"},{"line_number":90,"context_line":"necessarily mean that your cluster is down, as long as a majority of them remain"},{"line_number":91,"context_line":"running and have a formed quorum."},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"Identifying Problems"},{"line_number":94,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bf51134e_afd8db1d","line":91,"range":{"start_line":91,"start_character":17,"end_line":91,"end_character":33},"updated":"2020-06-22 19:56:52.000000000","message":"I think it\u0027s pretty standard to just say quorum.\n\n  \"and are in quorum.\"","commit_id":"dbcf10f8d9df2fb23a4a52535d0fe4e91157b07c"},{"author":{"_account_id":28618,"name":"Drew Walters","email":"drewwalters@microsoft.com","username":"drewwalters96"},"change_message_id":"61d66443088207ce90edef01d26b39e65c231b7b","unresolved":false,"context_lines":[{"line_number":90,"context_line":"necessarily mean that your cluster is down, as long as a majority of them remain"},{"line_number":91,"context_line":"running and have a formed quorum."},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"Identifying Problems"},{"line_number":94,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":95,"context_line":""},{"line_number":96,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bf51134e_0f38e73a","line":93,"range":{"start_line":93,"start_character":0,"end_line":93,"end_character":20},"updated":"2020-06-22 19:56:52.000000000","message":"These are really nice sections below. Nice work.","commit_id":"dbcf10f8d9df2fb23a4a52535d0fe4e91157b07c"},{"author":{"_account_id":28618,"name":"Drew Walters","email":"drewwalters@microsoft.com","username":"drewwalters96"},"change_message_id":"61d66443088207ce90edef01d26b39e65c231b7b","unresolved":false,"context_lines":[{"line_number":159,"context_line":""},{"line_number":160,"context_line":"mon_status can be obtained through the ceph tool when you have a formed quorum."},{"line_number":161,"context_line":"This command will output a multitude of information about the monitor, including"},{"line_number":162,"context_line":"the same output you would get with quorum_status. TODO Exact Commands?"},{"line_number":163,"context_line":""},{"line_number":164,"context_line":"Take the following example of mon_status:"},{"line_number":165,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"bf51134e_8f1cf7c1","line":162,"range":{"start_line":162,"start_character":50,"end_line":162,"end_character":69},"updated":"2020-06-22 19:56:52.000000000","message":"I think this may be left behind :)","commit_id":"dbcf10f8d9df2fb23a4a52535d0fe4e91157b07c"},{"author":{"_account_id":28618,"name":"Drew Walters","email":"drewwalters@microsoft.com","username":"drewwalters96"},"change_message_id":"61d66443088207ce90edef01d26b39e65c231b7b","unresolved":false,"context_lines":[{"line_number":194,"context_line":"    }"},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"A couple of things stand out: there are three monitors in the monmap (a, b and"},{"line_number":197,"context_line":"c), the quorum is formed by only two monitors, and c is in the quorum as a peon."},{"line_number":198,"context_line":""},{"line_number":199,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"},{"line_number":200,"context_line":"Which monitor is out of the quorum?"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bf51134e_0f8627dc","line":197,"range":{"start_line":197,"start_character":73,"end_line":197,"end_character":80},"updated":"2020-06-22 19:56:52.000000000","message":"Is there an external resource that explains this? As someone less familiar with ceph, I am lost.","commit_id":"dbcf10f8d9df2fb23a4a52535d0fe4e91157b07c"},{"author":{"_account_id":28618,"name":"Drew Walters","email":"drewwalters@microsoft.com","username":"drewwalters96"},"change_message_id":"61d66443088207ce90edef01d26b39e65c231b7b","unresolved":false,"context_lines":[{"line_number":241,"context_line":"one of probing, electing or synchronizing. If it happens to be either leader or"},{"line_number":242,"context_line":"peon, then the monitor believes to be in quorum, while the remaining cluster is"},{"line_number":243,"context_line":"sure it is not; or maybe it got into the quorum while troubleshooting of"},{"line_number":244,"context_line":"monitor goes on, so check you ceph -s again just to make sure. Proceed if the"},{"line_number":245,"context_line":"monitor is not yet in the quorum."},{"line_number":246,"context_line":""},{"line_number":247,"context_line":"What if the state is probing?"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bf51134e_2f5a0b82","line":244,"range":{"start_line":244,"start_character":17,"end_line":244,"end_character":35},"updated":"2020-06-22 19:56:52.000000000","message":"so check you?\n\n:)","commit_id":"dbcf10f8d9df2fb23a4a52535d0fe4e91157b07c"},{"author":{"_account_id":28618,"name":"Drew Walters","email":"drewwalters@microsoft.com","username":"drewwalters96"},"change_message_id":"61d66443088207ce90edef01d26b39e65c231b7b","unresolved":false,"context_lines":[{"line_number":313,"context_line":"In this sort of situations, you have two possible solutions:"},{"line_number":314,"context_line":""},{"line_number":315,"context_line":"* Scrap the monitor and create a new one. See Removing a Monitor. See Adding a"},{"line_number":316,"context_line":"  Monitor (TODO Links)"},{"line_number":317,"context_line":"* Inject a valid monmap into the monitor. Not practical in a Kubernetes"},{"line_number":318,"context_line":"  environment (for now unless I find a way to stop and restart the MON)."},{"line_number":319,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"bf51134e_efc81323","line":316,"range":{"start_line":316,"start_character":10,"end_line":316,"end_character":22},"updated":"2020-06-22 19:56:52.000000000","message":"todo","commit_id":"dbcf10f8d9df2fb23a4a52535d0fe4e91157b07c"},{"author":{"_account_id":28618,"name":"Drew Walters","email":"drewwalters@microsoft.com","username":"drewwalters96"},"change_message_id":"61d66443088207ce90edef01d26b39e65c231b7b","unresolved":false,"context_lines":[{"line_number":315,"context_line":"* Scrap the monitor and create a new one. See Removing a Monitor. See Adding a"},{"line_number":316,"context_line":"  Monitor (TODO Links)"},{"line_number":317,"context_line":"* Inject a valid monmap into the monitor. Not practical in a Kubernetes"},{"line_number":318,"context_line":"  environment (for now unless I find a way to stop and restart the MON)."},{"line_number":319,"context_line":""},{"line_number":320,"context_line":"^^^^^^^^^^^"},{"line_number":321,"context_line":"Clock Skews"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bf51134e_2fcfab29","line":318,"range":{"start_line":318,"start_character":15,"end_line":318,"end_character":71},"updated":"2020-06-22 19:56:52.000000000","message":"todo?","commit_id":"dbcf10f8d9df2fb23a4a52535d0fe4e91157b07c"},{"author":{"_account_id":28618,"name":"Drew Walters","email":"drewwalters@microsoft.com","username":"drewwalters96"},"change_message_id":"61d66443088207ce90edef01d26b39e65c231b7b","unresolved":false,"context_lines":[{"line_number":431,"context_line":"The amount of information in the logs are subject to the debug levels being"},{"line_number":432,"context_line":"enforced by your configuration files. If you have not enforced a specific debug"},{"line_number":433,"context_line":"level then Ceph is using the default levels and your logs may not contain"},{"line_number":434,"context_line":"important information to track down you issue. A first step in getting relevant"},{"line_number":435,"context_line":"information into your logs will be to raise debug levels. In this case we will"},{"line_number":436,"context_line":"be interested in the information from the monitor. Similarly, to what happens on"},{"line_number":437,"context_line":"other components, different parts of the monitor will output their debug"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bf51134e_affbbb44","line":434,"range":{"start_line":434,"start_character":36,"end_line":434,"end_character":39},"updated":"2020-06-22 19:56:52.000000000","message":"your","commit_id":"dbcf10f8d9df2fb23a4a52535d0fe4e91157b07c"},{"author":{"_account_id":28618,"name":"Drew Walters","email":"drewwalters@microsoft.com","username":"drewwalters96"},"change_message_id":"61d66443088207ce90edef01d26b39e65c231b7b","unresolved":false,"context_lines":[{"line_number":489,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"},{"line_number":490,"context_line":""},{"line_number":491,"context_line":"Ideally you would send to support only the relevant portions of your logs. Your"},{"line_number":492,"context_line":"support organization knows that figuring out the corresponding portion may not"},{"line_number":493,"context_line":"be the easiest of tasks. Therefore, they won\u0027t hold it to you if you provide the"},{"line_number":494,"context_line":"full log, but common sense should be employed. If your log has hundreds of"},{"line_number":495,"context_line":"thousands of lines, it may get tricky to go through the whole thing, especially"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bf51134e_efbf735e","line":492,"range":{"start_line":492,"start_character":0,"end_line":492,"end_character":19},"updated":"2020-06-22 19:56:52.000000000","message":"Is this the community? Maybe a link to the mailing list or slack would help.","commit_id":"dbcf10f8d9df2fb23a4a52535d0fe4e91157b07c"}]}