)]}'
{"specs/rocky/regions.rst":[{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"588153ced201da274a0b6619185b7557e3ad0c94","unresolved":false,"context_lines":[{"line_number":20,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":21,"context_line":""},{"line_number":22,"context_line":"OpenStack has supported Regions since before Tripleo existed.  However, Tripleo"},{"line_number":23,"context_line":"has thus far restricted itself to a single region."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":""},{"line_number":26,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"5f7c97a3_34d1a0ce","line":23,"updated":"2018-05-08 14:04:20.000000000","message":"I\u0027ve been pointed here because I\u0027m actually working on Keystone services that are stretched between multiple overclouds.   However, I\u0027m not sure what use case this blueprint is addressing.     The use case I have is, there are two datacenters.  Each has its own undercloud and overcloud.   The keystone services on the *overcloud* for each of these share a common Galera database that is stretched between the two datacenters, and they make use of regions to get at their datacenter-local service information.\n\nI already have this working as a POC and am working into tripleo how this would have to work.  \n\nis this blueprint addressing the same use case?    I am not familiar with how the keystone database on the undercloud interacts with that of the overcloud.  I haven\u0027t had to touch the undercloud database at all, I just build out the two overclouds and then merge the DBs together on the overcloud.","commit_id":"545ef61da6fc83469f5c29064395dc827d297235"},{"author":{"_account_id":10873,"name":"Juan Antonio Osorio Robles","email":"jaosorior@redhat.com","username":"ejuaoso"},"change_message_id":"611ddb0631eb411e89d71e7fdf1a0748bdb6eaba","unresolved":false,"context_lines":[{"line_number":20,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":21,"context_line":""},{"line_number":22,"context_line":"OpenStack has supported Regions since before Tripleo existed.  However, Tripleo"},{"line_number":23,"context_line":"has thus far restricted itself to a single region."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":""},{"line_number":26,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"5f7c97a3_d4f86c4a","line":23,"in_reply_to":"5f7c97a3_34d1a0ce","updated":"2018-05-08 14:06:07.000000000","message":"The undercloud keystone is not linked at all with the overcloud keystone. So that\u0027s not a problem. from my point of view, this would be addressed exactly by what you\u0027re working with :D","commit_id":"545ef61da6fc83469f5c29064395dc827d297235"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"25f6a256a899f9cb15b38f4506c5a81edb5b7063","unresolved":false,"context_lines":[{"line_number":20,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":21,"context_line":""},{"line_number":22,"context_line":"OpenStack has supported Regions since before Tripleo existed.  However, Tripleo"},{"line_number":23,"context_line":"has thus far restricted itself to a single region."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":""},{"line_number":26,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"5f7c97a3_f45fa83e","line":23,"in_reply_to":"5f7c97a3_d4f86c4a","updated":"2018-05-08 14:09:48.000000000","message":"OK so...that means you are equally puzzled why we are replicating keystone DBs across underclouds.  FWIW here is the work I\u0027m doing: https://github.com/zzzeek/stretch_cluster/tree/standard_tripleo_version that\u0027s the branch that does two tripleo overclouds and merges them together.  in master I\u0027m starting to work on getting ooo to build up the second galera instance automatically.","commit_id":"545ef61da6fc83469f5c29064395dc827d297235"},{"author":{"_account_id":10873,"name":"Juan Antonio Osorio Robles","email":"jaosorior@redhat.com","username":"ejuaoso"},"change_message_id":"c2c38447c33cc987b2e7d818109c48e32025bd7a","unresolved":false,"context_lines":[{"line_number":20,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":21,"context_line":""},{"line_number":22,"context_line":"OpenStack has supported Regions since before Tripleo existed.  However, Tripleo"},{"line_number":23,"context_line":"has thus far restricted itself to a single region."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":""},{"line_number":26,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"5f7c97a3_5f3795b1","line":23,"in_reply_to":"5f7c97a3_f45fa83e","updated":"2018-05-08 14:21:21.000000000","message":"Took a brief look at your playbooks and it looks quite cool! Let me know if you need help with the TripleO integration of this. Ideally this would be integrated as part of the overcloud deployment of the second region.","commit_id":"545ef61da6fc83469f5c29064395dc827d297235"},{"author":{"_account_id":10873,"name":"Juan Antonio Osorio Robles","email":"jaosorior@redhat.com","username":"ejuaoso"},"change_message_id":"b492806b9637faef33a795b2aedbfb9404c67c39","unresolved":false,"context_lines":[{"line_number":31,"context_line":"--------"},{"line_number":32,"context_line":""},{"line_number":33,"context_line":""},{"line_number":34,"context_line":"1.  Add a new undercloud to the existing overcloud."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"    `openstack undercloud add  --config \u003cfile\u003e`"},{"line_number":37,"context_line":""},{"line_number":38,"context_line":"2. Add a region on the new undercloud"},{"line_number":39,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"5f7c97a3_10811590","line":36,"range":{"start_line":34,"start_character":0,"end_line":36,"end_character":47},"updated":"2018-05-08 09:59:04.000000000","message":"why would this be needed? probably having a completely separate undercloud is just fine. The main thing is to be able to configure the overcloud in a certain way. You could even trigger the new region deployment from the same undercloud.","commit_id":"545ef61da6fc83469f5c29064395dc827d297235"},{"author":{"_account_id":10873,"name":"Juan Antonio Osorio Robles","email":"jaosorior@redhat.com","username":"ejuaoso"},"change_message_id":"b492806b9637faef33a795b2aedbfb9404c67c39","unresolved":false,"context_lines":[{"line_number":35,"context_line":""},{"line_number":36,"context_line":"    `openstack undercloud add  --config \u003cfile\u003e`"},{"line_number":37,"context_line":""},{"line_number":38,"context_line":"2. Add a region on the new undercloud"},{"line_number":39,"context_line":""},{"line_number":40,"context_line":"    `openstack overcloud extend \u003cundercloud\u003e  \u003cregion`"},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"    A. Create a new region in the existing OverCLoud Keystone"},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"    B. Create a new database server on the new undercloud"},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"    C. Replicate the existing Keystone Database to the new undercloud."},{"line_number":47,"context_line":""},{"line_number":48,"context_line":"3.  Add new servers to the region.  Each of these command can be performed separately."},{"line_number":49,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"5f7c97a3_b049418c","line":46,"range":{"start_line":38,"start_character":0,"end_line":46,"end_character":70},"updated":"2018-05-08 09:59:04.000000000","message":"This part was a little bit confusing for me, can you rephrase?\n\nFrom what I understand, what we really need to do is expand the database (mostly just the keystone one) and the keystone servers to towards the new region.\n\nThe overcloud deployment would be pretty completely separate, except for the fact that the keystone instances in both regions would share the same data, and the new overcloud would have the services configured with the other region name.\n\nThis is the part that I\u0027m not so sure about, how do we replicate the keystone database to the other region. For that. I would need some input from Michele and Damien.","commit_id":"545ef61da6fc83469f5c29064395dc827d297235"},{"author":{"_account_id":10873,"name":"Juan Antonio Osorio Robles","email":"jaosorior@redhat.com","username":"ejuaoso"},"change_message_id":"b492806b9637faef33a795b2aedbfb9404c67c39","unresolved":false,"context_lines":[{"line_number":59,"context_line":"Alternatives"},{"line_number":60,"context_line":"------------"},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"Spin uip multiple Overcloud and use Keystone-to-Keystone (K2K) Federation is the only"},{"line_number":63,"context_line":"existing option today."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":""},{"line_number":66,"context_line":"Security Impact"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5f7c97a3_906d2506","line":63,"range":{"start_line":62,"start_character":0,"end_line":63,"end_character":22},"updated":"2018-05-08 09:59:04.000000000","message":"this seems like the most viable option right now","commit_id":"545ef61da6fc83469f5c29064395dc827d297235"},{"author":{"_account_id":10873,"name":"Juan Antonio Osorio Robles","email":"jaosorior@redhat.com","username":"ejuaoso"},"change_message_id":"6c1ab0e973088de7cf23bc0b7d2028ea23584ec7","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"5f7c97a3_bbb8c174","updated":"2018-05-31 06:24:29.000000000","message":"And now I\u0027m wondering if we should be looking at the Tricircle project instead of multi-site https://wiki.openstack.org/wiki/Tricircle\n\nuhm...","commit_id":"973845be5d56939ff8d4a43b59500341b672332c"},{"author":{"_account_id":10873,"name":"Juan Antonio Osorio Robles","email":"jaosorior@redhat.com","username":"ejuaoso"},"change_message_id":"22ba94e44512212ff433e13dc489d23c2bbfb646","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"5f7c97a3_1e5203df","in_reply_to":"5f7c97a3_bbb8c174","updated":"2018-05-31 06:50:37.000000000","message":"The project I should have pointed at would have been this one https://wiki.openstack.org/wiki/Trio2o","commit_id":"973845be5d56939ff8d4a43b59500341b672332c"},{"author":{"_account_id":10873,"name":"Juan Antonio Osorio Robles","email":"jaosorior@redhat.com","username":"ejuaoso"},"change_message_id":"deb2236ae5c5dbb3416364c607cece620552005e","unresolved":false,"context_lines":[{"line_number":146,"context_line":"can decide that there is already a bootstrap node remote to the local Pacemaker"},{"line_number":147,"context_line":"cluster, it simply starts up its local Galera nodes without specifying any"},{"line_number":148,"context_line":"of them as the bootstrap."},{"line_number":149,"context_line":""},{"line_number":150,"context_line":""},{"line_number":151,"context_line":"Deploy One Overcloud at a Time"},{"line_number":152,"context_line":"------------------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5f7c97a3_fb3b9998","line":149,"updated":"2018-05-31 05:38:03.000000000","message":"In the edge case, the clusters might be quite close to each other, but in other multi-site cases, the regions might be quite far and have considerable network latency. Would this be a problem for the resource agent?","commit_id":"973845be5d56939ff8d4a43b59500341b672332c"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"927500dc7f2cfe6db21a8ceb03e6a99b24fbd3b6","unresolved":false,"context_lines":[{"line_number":146,"context_line":"can decide that there is already a bootstrap node remote to the local Pacemaker"},{"line_number":147,"context_line":"cluster, it simply starts up its local Galera nodes without specifying any"},{"line_number":148,"context_line":"of them as the bootstrap."},{"line_number":149,"context_line":""},{"line_number":150,"context_line":""},{"line_number":151,"context_line":"Deploy One Overcloud at a Time"},{"line_number":152,"context_line":"------------------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5f7c97a3_b7d41249","line":149,"in_reply_to":"5f7c97a3_fb3b9998","updated":"2018-06-04 17:13:27.000000000","message":"it\u0027s not an issue for the resource agent, as it only looks at the other nodes during a bootstrap process but when the cluster is normally running it only looks at its local mysql instance.  it does so using only very simple commands that exchange a few hundred bytes of status commands/information.   there\u0027s more network happening between galera nodes themselves, but again that communication does not impact the responsiveness of read-only database operations.","commit_id":"973845be5d56939ff8d4a43b59500341b672332c"},{"author":{"_account_id":28223,"name":"Cedric Jeanneret","display_name":"cjeanner (Tengu)","email":"cjeanner@redhat.com","username":"cjeanner"},"change_message_id":"268479a279b79292f1d78bff9f00e3a86548c87b","unresolved":false,"context_lines":[{"line_number":200,"context_line":"Alternatives"},{"line_number":201,"context_line":"------------"},{"line_number":202,"context_line":""},{"line_number":203,"context_line":"Spin uip multiple Overcloud and use Keystone-to-Keystone (K2K) Federation is the only"},{"line_number":204,"context_line":"existing option today."},{"line_number":205,"context_line":""},{"line_number":206,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5f7c97a3_fb0959dc","line":203,"range":{"start_line":203,"start_character":5,"end_line":203,"end_character":8},"updated":"2018-05-31 05:32:14.000000000","message":"\"up\"?","commit_id":"973845be5d56939ff8d4a43b59500341b672332c"},{"author":{"_account_id":28223,"name":"Cedric Jeanneret","display_name":"cjeanner (Tengu)","email":"cjeanner@redhat.com","username":"cjeanner"},"change_message_id":"3bafbbd970304a85086f6688c65bc8efc8bfad18","unresolved":false,"context_lines":[{"line_number":200,"context_line":"Alternatives"},{"line_number":201,"context_line":"------------"},{"line_number":202,"context_line":""},{"line_number":203,"context_line":"Spin uip multiple Overcloud and use Keystone-to-Keystone (K2K) Federation is the only"},{"line_number":204,"context_line":"existing option today."},{"line_number":205,"context_line":""},{"line_number":206,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5f7c97a3_fbc07952","line":203,"range":{"start_line":203,"start_character":58,"end_line":203,"end_character":61},"updated":"2018-05-31 05:41:49.000000000","message":"What are the drawback of this solution? Might be good to list its \"cons\" vs the \"pros\" of the proposed solution.","commit_id":"973845be5d56939ff8d4a43b59500341b672332c"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"927500dc7f2cfe6db21a8ceb03e6a99b24fbd3b6","unresolved":false,"context_lines":[{"line_number":200,"context_line":"Alternatives"},{"line_number":201,"context_line":"------------"},{"line_number":202,"context_line":""},{"line_number":203,"context_line":"Spin uip multiple Overcloud and use Keystone-to-Keystone (K2K) Federation is the only"},{"line_number":204,"context_line":"existing option today."},{"line_number":205,"context_line":""},{"line_number":206,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5f7c97a3_57c9fea9","line":203,"range":{"start_line":203,"start_character":58,"end_line":203,"end_character":61},"in_reply_to":"5f7c97a3_fbc07952","updated":"2018-06-04 17:13:27.000000000","message":"This was @ayoung\u0027s original comment so if some keystone experts can discuss this, that would be helpful.","commit_id":"973845be5d56939ff8d4a43b59500341b672332c"},{"author":{"_account_id":10873,"name":"Juan Antonio Osorio Robles","email":"jaosorior@redhat.com","username":"ejuaoso"},"change_message_id":"deb2236ae5c5dbb3416364c607cece620552005e","unresolved":false,"context_lines":[{"line_number":208,"context_line":"---------------"},{"line_number":209,"context_line":""},{"line_number":210,"context_line":"* security of the network in which the galera nodes are \"stretched\", Galera"},{"line_number":211,"context_line":"can use SSL + SSL rsync replication"},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"* issues of passwords shared across overclouds in Keystone"},{"line_number":214,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5f7c97a3_7b70e972","line":211,"updated":"2018-05-31 05:38:03.000000000","message":"Right, there are two alternatives here really. One of them being to use internal TLS (we can actually set up FreeIPA in both clouds, and have both clouds trust each other\u0027s FreeIPA root certificate (we would need to have a PEM file with both FreeIPAs certs that the stretched Galera will use for trust).\n\nThe other option is to use IPSec in the network where the stretch galera is deployed. This would probably require us to use a dedicated network so we don\u0027t run the tunnels over the external network that the users of the cloud will be accessing.","commit_id":"973845be5d56939ff8d4a43b59500341b672332c"},{"author":{"_account_id":10873,"name":"Juan Antonio Osorio Robles","email":"jaosorior@redhat.com","username":"ejuaoso"},"change_message_id":"deb2236ae5c5dbb3416364c607cece620552005e","unresolved":false,"context_lines":[{"line_number":210,"context_line":"* security of the network in which the galera nodes are \"stretched\", Galera"},{"line_number":211,"context_line":"can use SSL + SSL rsync replication"},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"* issues of passwords shared across overclouds in Keystone"},{"line_number":214,"context_line":""},{"line_number":215,"context_line":""},{"line_number":216,"context_line":"Other End User Impact"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5f7c97a3_9b0ce5ec","line":213,"updated":"2018-05-31 05:38:03.000000000","message":"This we need to fix. Ideally the OpenStack service users in keystone of both regions are different. We can achieve this two ways:\n\n* Having a region prefix in the OpenStack service users for each region.\n\n* Having the service users be in different domains for each region.","commit_id":"973845be5d56939ff8d4a43b59500341b672332c"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"927500dc7f2cfe6db21a8ceb03e6a99b24fbd3b6","unresolved":false,"context_lines":[{"line_number":210,"context_line":"* security of the network in which the galera nodes are \"stretched\", Galera"},{"line_number":211,"context_line":"can use SSL + SSL rsync replication"},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"* issues of passwords shared across overclouds in Keystone"},{"line_number":214,"context_line":""},{"line_number":215,"context_line":""},{"line_number":216,"context_line":"Other End User Impact"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5f7c97a3_f7b92a15","line":213,"in_reply_to":"5f7c97a3_9b0ce5ec","updated":"2018-06-04 17:13:27.000000000","message":"I will see if I can try to start with \"region prefix\" in the tripleo feature I\u0027m doing but it might be tricky.","commit_id":"973845be5d56939ff8d4a43b59500341b672332c"},{"author":{"_account_id":10873,"name":"Juan Antonio Osorio Robles","email":"jaosorior@redhat.com","username":"ejuaoso"},"change_message_id":"deb2236ae5c5dbb3416364c607cece620552005e","unresolved":false,"context_lines":[{"line_number":237,"context_line":"Developer Impact"},{"line_number":238,"context_line":"----------------"},{"line_number":239,"context_line":""},{"line_number":240,"context_line":"Discuss things that will affect other developers working on OpenStack."},{"line_number":241,"context_line":""},{"line_number":242,"context_line":""},{"line_number":243,"context_line":"Implementation"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5f7c97a3_1bee95ec","line":240,"updated":"2018-05-31 05:38:03.000000000","message":"Here, and related to security:\n\nWe need to figure out what are folks doing in the field... Do we have a control plane network that spawns between datacenters? or are they separate? what assumptions can we make here?\n\nThis would affect how the stretch script is written. Is it stretching by referencing all clouds at the same time (all nodes in the ansible inventory)? Or do we execute the stretch one cloud at a time?\n\nI would rather have data from folks in the field than making a wrong asumption :)","commit_id":"973845be5d56939ff8d4a43b59500341b672332c"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"927500dc7f2cfe6db21a8ceb03e6a99b24fbd3b6","unresolved":false,"context_lines":[{"line_number":237,"context_line":"Developer Impact"},{"line_number":238,"context_line":"----------------"},{"line_number":239,"context_line":""},{"line_number":240,"context_line":"Discuss things that will affect other developers working on OpenStack."},{"line_number":241,"context_line":""},{"line_number":242,"context_line":""},{"line_number":243,"context_line":"Implementation"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5f7c97a3_573ede8b","line":240,"in_reply_to":"5f7c97a3_1bee95ec","updated":"2018-06-04 17:13:27.000000000","message":"as we discussed, if this can work by working on only one overcloud at a time (not totally sure), that is the more conservative approach which would accommodate for both workflows.","commit_id":"973845be5d56939ff8d4a43b59500341b672332c"},{"author":{"_account_id":17517,"name":"Andrew Beekhof","email":"abeekhof@redhat.com","username":"beekhof"},"change_message_id":"24e71a110f8741f647b81970db2a17e533533c1f","unresolved":false,"context_lines":[{"line_number":53,"context_line":"in a second step; in the \"merge\" scenario, the Keystone database that is local"},{"line_number":54,"context_line":"to a particular overcloud is copied and merged into the larger Galera cluster."},{"line_number":55,"context_line":""},{"line_number":56,"context_line":"Second Galera Database, a.k.a. \"Stretch\""},{"line_number":57,"context_line":"----------------------------------------"},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"For the initial deployment of a particular overcloud, the existing overcloud"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_21c24035","line":56,"range":{"start_line":56,"start_character":0,"end_line":56,"end_character":22},"updated":"2018-06-13 01:03:17.000000000","message":"How many copies at each site?","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"ecfe536fb6941a6e870335348b5da61b0dffd0f9","unresolved":false,"context_lines":[{"line_number":53,"context_line":"in a second step; in the \"merge\" scenario, the Keystone database that is local"},{"line_number":54,"context_line":"to a particular overcloud is copied and merged into the larger Galera cluster."},{"line_number":55,"context_line":""},{"line_number":56,"context_line":"Second Galera Database, a.k.a. \"Stretch\""},{"line_number":57,"context_line":"----------------------------------------"},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"For the initial deployment of a particular overcloud, the existing overcloud"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_44266acd","line":56,"range":{"start_line":56,"start_character":0,"end_line":56,"end_character":22},"in_reply_to":"5f7c97a3_21c24035","updated":"2018-06-13 02:57:22.000000000","message":"at the moment we\u0027re just mirroring what the existing galera deployment does, deploying a galera node per controller.  so three per site for a three-node cluster.  obv this can be whatever.","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":17517,"name":"Andrew Beekhof","email":"abeekhof@redhat.com","username":"beekhof"},"change_message_id":"b4481374bfb7fb67ef4cd2a2d99dc6d60271dbc5","unresolved":false,"context_lines":[{"line_number":53,"context_line":"in a second step; in the \"merge\" scenario, the Keystone database that is local"},{"line_number":54,"context_line":"to a particular overcloud is copied and merged into the larger Galera cluster."},{"line_number":55,"context_line":""},{"line_number":56,"context_line":"Second Galera Database, a.k.a. \"Stretch\""},{"line_number":57,"context_line":"----------------------------------------"},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"For the initial deployment of a particular overcloud, the existing overcloud"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_a431630d","line":56,"range":{"start_line":56,"start_character":0,"end_line":56,"end_character":22},"in_reply_to":"5f7c97a3_44266acd","updated":"2018-06-14 00:35:55.000000000","message":"Minimum is 2 per site, possibly 3. Otherwise the loss of a single node (if it was the one with stretch galera) would take that site offline.\n\nUnless you arrange for the keystone traffic to fall back to using the other site but I\u0027d assume the performance would be prohibitive.","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":17517,"name":"Andrew Beekhof","email":"abeekhof@redhat.com","username":"beekhof"},"change_message_id":"24e71a110f8741f647b81970db2a17e533533c1f","unresolved":false,"context_lines":[{"line_number":128,"context_line":"literally just \"one job\", to figure out if that flag should be set, or not."},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"The resource agent publishes information about its own MySQL service using"},{"line_number":131,"context_line":"pacemaker attribute commands, and it learns about other nodes only by"},{"line_number":132,"context_line":"inspecting these attributes using those same commands.  The commands used are"},{"line_number":133,"context_line":"the \"crm_attribute\", \"crm_mon\" and \"crm_master\" commands.  In order to allow"},{"line_number":134,"context_line":"the resource agent to coordinate with Galera nodes in other overclouds, the"},{"line_number":135,"context_line":"initial proof of concept uses ssh to run the same commands remotely on other"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_819eb451","line":132,"range":{"start_line":131,"start_character":34,"end_line":132,"end_character":53},"updated":"2018-06-13 01:03:17.000000000","message":"But only of those in the same pacemaker cluster right? You\u0027re not cross populating node info from other sites are you?","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"cfb10fa5ed09396615a595e31c46c386dddfd7d8","unresolved":false,"context_lines":[{"line_number":128,"context_line":"literally just \"one job\", to figure out if that flag should be set, or not."},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"The resource agent publishes information about its own MySQL service using"},{"line_number":131,"context_line":"pacemaker attribute commands, and it learns about other nodes only by"},{"line_number":132,"context_line":"inspecting these attributes using those same commands.  The commands used are"},{"line_number":133,"context_line":"the \"crm_attribute\", \"crm_mon\" and \"crm_master\" commands.  In order to allow"},{"line_number":134,"context_line":"the resource agent to coordinate with Galera nodes in other overclouds, the"},{"line_number":135,"context_line":"initial proof of concept uses ssh to run the same commands remotely on other"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_3727b965","line":132,"range":{"start_line":131,"start_character":34,"end_line":132,"end_character":53},"in_reply_to":"5f7c97a3_05e7f013","updated":"2018-06-14 14:07:05.000000000","message":"I understand that Booth https://github.com/ClusterLabs/booth has been discussed as a more comprehensive solution to coordination of multiple pacemakers.  IIUC we would have to alter the resource agent to rely upon receiving a booth ticket that identifies it as the current leader using raft, so that it knows it has the authority to launch the bootstrap; that way it doesnt need to reach out to the other pacemakers itself.   this would also be a very nice architecture, so what I will do is try to add some extended discussion here about the RA\u0027s options.  However, the downside of booth is that we again need to have an odd number of nodes not unlike if we have garbd in place, and we then have to get booth integrated into tripleo as well and deal with all of that complexity.  So I think the \"RA direct access to the other pacemaker\" version I have now should be what we start out with and possibly have as an option even if we do get a more orchestrated booth version going, because not every deployer wants to have a third datacenter.  for the two-datacenter issue, I will also discuss the possibility of allowing the galera cluster on the losing side to work in a \"read only\" mode, which also would require some changes.   Both \"booth\" and \"read only split brain\" I think should be phased in on top of an initial proof of concept version.","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":17517,"name":"Andrew Beekhof","email":"abeekhof@redhat.com","username":"beekhof"},"change_message_id":"fa5a8d6887d8371f719c61b230b17c5c4c5cab2e","unresolved":false,"context_lines":[{"line_number":128,"context_line":"literally just \"one job\", to figure out if that flag should be set, or not."},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"The resource agent publishes information about its own MySQL service using"},{"line_number":131,"context_line":"pacemaker attribute commands, and it learns about other nodes only by"},{"line_number":132,"context_line":"inspecting these attributes using those same commands.  The commands used are"},{"line_number":133,"context_line":"the \"crm_attribute\", \"crm_mon\" and \"crm_master\" commands.  In order to allow"},{"line_number":134,"context_line":"the resource agent to coordinate with Galera nodes in other overclouds, the"},{"line_number":135,"context_line":"initial proof of concept uses ssh to run the same commands remotely on other"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_f62507c0","line":132,"range":{"start_line":131,"start_character":34,"end_line":132,"end_character":53},"in_reply_to":"5f7c97a3_05e7f013","updated":"2018-06-25 02:42:41.000000000","message":"I was looking at the loop that didn\u0027t error out when retrieving a remote peer\u0027s seqno and anticipating problems, but I\u0027d missed the other loop where the agent verifies every peer has a seqno set before bootstrapping the cluster.   So I was way off there.","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":17517,"name":"Andrew Beekhof","email":"abeekhof@redhat.com","username":"beekhof"},"change_message_id":"fa5a8d6887d8371f719c61b230b17c5c4c5cab2e","unresolved":false,"context_lines":[{"line_number":128,"context_line":"literally just \"one job\", to figure out if that flag should be set, or not."},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"The resource agent publishes information about its own MySQL service using"},{"line_number":131,"context_line":"pacemaker attribute commands, and it learns about other nodes only by"},{"line_number":132,"context_line":"inspecting these attributes using those same commands.  The commands used are"},{"line_number":133,"context_line":"the \"crm_attribute\", \"crm_mon\" and \"crm_master\" commands.  In order to allow"},{"line_number":134,"context_line":"the resource agent to coordinate with Galera nodes in other overclouds, the"},{"line_number":135,"context_line":"initial proof of concept uses ssh to run the same commands remotely on other"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_9935e4aa","line":132,"range":{"start_line":131,"start_character":34,"end_line":132,"end_character":53},"in_reply_to":"5f7c97a3_3727b965","updated":"2018-06-25 02:42:41.000000000","message":"\u003e  IIUC we would have to alter the resource agent to rely upon receiving a booth ticket\n\nNo. The agent has no knowledge of tickets, thats handled at the pacemaker level\n\n\u003e the downside of booth is that we again need to have an odd number of nodes not unlike if we have garbd in place\n\nYou can have 2 booth nodes but that would defeat the point and even numbers of nodes are also fine, just like a normal cluster,  provided you don\u0027t arrange them such that a single network failure can create an even split.\n\nFrom what I\u0027m told, the big difference is that all DB (update) traffic goes through the garbd nodes.  It is perhaps helpful to think of Booth as a light weight corosync that can handle much higher latencies and instead of handing out a single quorum value, it hands out tickets that can represent an application specific kind of quorum.\n\nWithout booth, how would you decide which site boostraps the DB in the case that all nodes have identical seqnos?","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":17517,"name":"Andrew Beekhof","email":"abeekhof@redhat.com","username":"beekhof"},"change_message_id":"3bfb03ceaad2bc67ffefd7b5477b2bfc4f2ee1e5","unresolved":false,"context_lines":[{"line_number":128,"context_line":"literally just \"one job\", to figure out if that flag should be set, or not."},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"The resource agent publishes information about its own MySQL service using"},{"line_number":131,"context_line":"pacemaker attribute commands, and it learns about other nodes only by"},{"line_number":132,"context_line":"inspecting these attributes using those same commands.  The commands used are"},{"line_number":133,"context_line":"the \"crm_attribute\", \"crm_mon\" and \"crm_master\" commands.  In order to allow"},{"line_number":134,"context_line":"the resource agent to coordinate with Galera nodes in other overclouds, the"},{"line_number":135,"context_line":"initial proof of concept uses ssh to run the same commands remotely on other"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_62401a07","line":132,"range":{"start_line":131,"start_character":34,"end_line":132,"end_character":53},"in_reply_to":"5f7c97a3_5f3dba8b","updated":"2018-06-14 02:56:14.000000000","message":"\u003e it\u0027s one big cluster \n\nOnly at the galera level\n\n\u003e so the resource agent wouldn\u0027t do that\n\nAs the code works today, it totally would if you have a network split (and you have no way to know if what you\u0027re in is a network split or a site failure)\n\n\u003e should make for some spectacular support calls :).\n\nThat is my concern","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"d27d4e465f8aa894d5e5d385f2b61f82719da0ae","unresolved":false,"context_lines":[{"line_number":128,"context_line":"literally just \"one job\", to figure out if that flag should be set, or not."},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"The resource agent publishes information about its own MySQL service using"},{"line_number":131,"context_line":"pacemaker attribute commands, and it learns about other nodes only by"},{"line_number":132,"context_line":"inspecting these attributes using those same commands.  The commands used are"},{"line_number":133,"context_line":"the \"crm_attribute\", \"crm_mon\" and \"crm_master\" commands.  In order to allow"},{"line_number":134,"context_line":"the resource agent to coordinate with Galera nodes in other overclouds, the"},{"line_number":135,"context_line":"initial proof of concept uses ssh to run the same commands remotely on other"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_05e7f013","line":132,"range":{"start_line":131,"start_character":34,"end_line":132,"end_character":53},"in_reply_to":"5f7c97a3_62401a07","updated":"2018-06-14 04:21:21.000000000","message":"\u003e As the code works today, it totally would if you have a network split (and you have no way to know if what you\u0027re in is a network split or a site failure)\n\nkeep in mind that the resource agent and the galera cluster address has *all* the nodes in its list up front, across all datacenters.   it isn\u0027t just starting itself up and assuming some other remote galera cluster is doing whatever it wants,  from a galera / RA point of view this is one big galera cluster, it\u0027s just the way it sends and retrieves keys in order to solve the bootstrap issue has been modified to be able to store those keys amongst multiple pacemaker registries, but in order to bootstrap, it needs positive confirmation from all Galera nodes.  It won\u0027t emit --wsrep-new-cluster for a bootstrap if it can\u0027t get to a particular node or that particular node\u0027s pacemaker registry.  The fire drill we\u0027re always doing with customers is that one of their controllers wont come on and they want to bootstrap the rest of the cluster wtihout it.  The resource agent won\u0027t start unless we explicitly tell it to ignore that missing nodes.  *that\u0027s* the support situation I\u0027m referring towards, that because they want to share their keystone across two datacenters, they have to keep that link up and if they can\u0027t, then they have to go without one of those datacenters being functional, Galera is not going to let it happen and the RA won\u0027t bootstrap a totally-down cluster without it.   There is no concern about an actual split-brain-writer situation happening.  \n\nThe concern I think you have is regarding the split brain issue, e.g. all the nodes are up but the network splits the galera cluster in half.  Galera doesn\u0027t keep any nodes in \"primary\" state if you don\u0027t have enough nodes for quorum, so a 3 + 3 situation plus a netsplit means both go down by default.   The idea, which I didn\u0027t get into in this document yet, is that you either are using an odd number of datacenters, or you\u0027ve picked a winning datacenter up front in a 2-datacenter situation where you set galera\u0027s \"weight\" parameters to favor one side over the other in the case of a split.","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"d4c0abe75908cecbc6f02ec617891996059b3c2a","unresolved":false,"context_lines":[{"line_number":128,"context_line":"literally just \"one job\", to figure out if that flag should be set, or not."},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"The resource agent publishes information about its own MySQL service using"},{"line_number":131,"context_line":"pacemaker attribute commands, and it learns about other nodes only by"},{"line_number":132,"context_line":"inspecting these attributes using those same commands.  The commands used are"},{"line_number":133,"context_line":"the \"crm_attribute\", \"crm_mon\" and \"crm_master\" commands.  In order to allow"},{"line_number":134,"context_line":"the resource agent to coordinate with Galera nodes in other overclouds, the"},{"line_number":135,"context_line":"initial proof of concept uses ssh to run the same commands remotely on other"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_5f3dba8b","line":132,"range":{"start_line":131,"start_character":34,"end_line":132,"end_character":53},"in_reply_to":"5f7c97a3_6437cbff","updated":"2018-06-14 01:07:25.000000000","message":"\u003e Is there a problem if site A starts with --wsrep-new-cluster when site B is already running?\n\nit\u0027s one big cluster so the resource agent wouldn\u0027t do that, in order to know if it can boot the cluster it needs to get an answer from all the nodes just like is the case now, which in this case would include nodes in the other overcloud.\n\nSoooooo......if your other datacenter is down and you want to fully reboot this datacenter without the other one coming online, having this one become the master and you definitely want the local DB to be the source of truth and the other datacenter can catch up later ,  you need to change the pacemaker config to no longer look for those remote nodes so that it can bootstrap.\n\nshould make for some spectacular support calls :).","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":17517,"name":"Andrew Beekhof","email":"abeekhof@redhat.com","username":"beekhof"},"change_message_id":"6a9e66b8f6167cab9711c4aed732f260f5c2cf8b","unresolved":false,"context_lines":[{"line_number":128,"context_line":"literally just \"one job\", to figure out if that flag should be set, or not."},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"The resource agent publishes information about its own MySQL service using"},{"line_number":131,"context_line":"pacemaker attribute commands, and it learns about other nodes only by"},{"line_number":132,"context_line":"inspecting these attributes using those same commands.  The commands used are"},{"line_number":133,"context_line":"the \"crm_attribute\", \"crm_mon\" and \"crm_master\" commands.  In order to allow"},{"line_number":134,"context_line":"the resource agent to coordinate with Galera nodes in other overclouds, the"},{"line_number":135,"context_line":"initial proof of concept uses ssh to run the same commands remotely on other"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_b479c793","line":132,"range":{"start_line":131,"start_character":34,"end_line":132,"end_character":53},"in_reply_to":"5f7c97a3_796c8806","updated":"2018-06-25 03:40:09.000000000","message":"\u003e  it is using crm_mon and parsing the XML to return this.\n\nBut that only tells you if someone at the local site is bootstrapping doesn\u0027t it?  What stops the other site from querying its local cib and also determining that \"nobody is a master and the node is safe to bootstrap\"?\n\nAlso, can we switch to email? This is a terrible forum for a conversation.  I can\u0027t even copy text in order to reply to it :-(","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"ecfe536fb6941a6e870335348b5da61b0dffd0f9","unresolved":false,"context_lines":[{"line_number":128,"context_line":"literally just \"one job\", to figure out if that flag should be set, or not."},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"The resource agent publishes information about its own MySQL service using"},{"line_number":131,"context_line":"pacemaker attribute commands, and it learns about other nodes only by"},{"line_number":132,"context_line":"inspecting these attributes using those same commands.  The commands used are"},{"line_number":133,"context_line":"the \"crm_attribute\", \"crm_mon\" and \"crm_master\" commands.  In order to allow"},{"line_number":134,"context_line":"the resource agent to coordinate with Galera nodes in other overclouds, the"},{"line_number":135,"context_line":"initial proof of concept uses ssh to run the same commands remotely on other"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_8f533b16","line":132,"range":{"start_line":131,"start_character":34,"end_line":132,"end_character":53},"in_reply_to":"5f7c97a3_819eb451","updated":"2018-06-13 02:57:22.000000000","message":"*thinks*.... yeah there\u0027s no cross-population.   if im a node, and I want to know about node X that I know is local, I look in my local attributes.  If I want to know about node Y that I know is somewhere else, I ssh to that cloud to run the pcs command.   When I *write* data to pacemaker, it is always in terms of a specific node; like, \"node X is the bootstrap node\", and we always write to the pacemaker cluster that is local to that specific node.     That is, no pacemaker attribute registry has keys in it about nodes that aren\u0027t local.\n\nI cc\u0027ed you on the PR about 6 weeks ago looking for your comments: https://github.com/ClusterLabs/resource-agents/pull/1141","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":17517,"name":"Andrew Beekhof","email":"abeekhof@redhat.com","username":"beekhof"},"change_message_id":"b4481374bfb7fb67ef4cd2a2d99dc6d60271dbc5","unresolved":false,"context_lines":[{"line_number":128,"context_line":"literally just \"one job\", to figure out if that flag should be set, or not."},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"The resource agent publishes information about its own MySQL service using"},{"line_number":131,"context_line":"pacemaker attribute commands, and it learns about other nodes only by"},{"line_number":132,"context_line":"inspecting these attributes using those same commands.  The commands used are"},{"line_number":133,"context_line":"the \"crm_attribute\", \"crm_mon\" and \"crm_master\" commands.  In order to allow"},{"line_number":134,"context_line":"the resource agent to coordinate with Galera nodes in other overclouds, the"},{"line_number":135,"context_line":"initial proof of concept uses ssh to run the same commands remotely on other"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_6437cbff","line":132,"range":{"start_line":131,"start_character":34,"end_line":132,"end_character":53},"in_reply_to":"5f7c97a3_8f533b16","updated":"2018-06-14 00:35:55.000000000","message":"Ah sorry, github notifications got too noisey so they don\u0027t land in my inbox anymore.\n\nIs there a problem if site A starts with --wsrep-new-cluster when site B is already running?","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"9903430203a51fac1d25dd9c8bea27b358cd78d1","unresolved":false,"context_lines":[{"line_number":128,"context_line":"literally just \"one job\", to figure out if that flag should be set, or not."},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"The resource agent publishes information about its own MySQL service using"},{"line_number":131,"context_line":"pacemaker attribute commands, and it learns about other nodes only by"},{"line_number":132,"context_line":"inspecting these attributes using those same commands.  The commands used are"},{"line_number":133,"context_line":"the \"crm_attribute\", \"crm_mon\" and \"crm_master\" commands.  In order to allow"},{"line_number":134,"context_line":"the resource agent to coordinate with Galera nodes in other overclouds, the"},{"line_number":135,"context_line":"initial proof of concept uses ssh to run the same commands remotely on other"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_796c8806","line":132,"range":{"start_line":131,"start_character":34,"end_line":132,"end_character":53},"in_reply_to":"5f7c97a3_9935e4aa","updated":"2018-06-25 03:20:33.000000000","message":"\u003e Without booth, how would you decide which site boostraps the DB in the case that all nodes have identical seqnos?\n\nHere\u0027s how it works without my changes:  each RA reads it\u0027s local grastate.dat for the \"safe_to_bootstrap\" flag (see http://galeracluster.com/2016/11/introducing-the-safe-to-bootstrap-feature-in-galera-cluster/, this feature first added to the RA in https://github.com/ClusterLabs/resource-agents/commit/16dee87e24ee1a0d6e37b5fa7b91c303f7c912db ), then the \"master_exists\" function is called which calls out to pacemaker and asks, \"has someone already bootstrapped?\".  it is using crm_mon and parsing the XML to return this.  if nobody is a master and the node is safe to bootstrap, it starts up with \"--wsrep-cluster-address\u003dgcomm://\", e.g. that this is the new cluster.  I\u0027m not really sure why it isn\u0027t using --wsrep-new-cluster, I have a feeling this might be part of a strategy where it will try to resolve for the race condition that two nodes try to bootstrap at the same time.  Need to get more details from Damien.\n\nAnyway, that is how the RA works today.   With the changes I made, everything above is exactly the same, except the step that does the crm_mon + parsing the XML also calls out to the remote pacemakers over ssh.   If we are relying upon some pacemaker behavior where it definitely only allows one node to start at a time and that\u0027s how we avoid a race, then that might be an issue.   This is more a Q for Damien.","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":17517,"name":"Andrew Beekhof","email":"abeekhof@redhat.com","username":"beekhof"},"change_message_id":"24e71a110f8741f647b81970db2a17e533533c1f","unresolved":false,"context_lines":[{"line_number":238,"context_line":"Alternatives"},{"line_number":239,"context_line":"------------"},{"line_number":240,"context_line":""},{"line_number":241,"context_line":"Spin up multiple Overcloud and use Keystone-to-Keystone (K2K) Federation is the only"},{"line_number":242,"context_line":"existing option today."},{"line_number":243,"context_line":""},{"line_number":244,"context_line":""},{"line_number":245,"context_line":"Security Impact"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_e1068886","line":242,"range":{"start_line":241,"start_character":0,"end_line":242,"end_character":22},"updated":"2018-06-13 01:03:17.000000000","message":"Would be worth at least 1 line explaining why we\u0027re not taking this approach","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"ecfe536fb6941a6e870335348b5da61b0dffd0f9","unresolved":false,"context_lines":[{"line_number":238,"context_line":"Alternatives"},{"line_number":239,"context_line":"------------"},{"line_number":240,"context_line":""},{"line_number":241,"context_line":"Spin up multiple Overcloud and use Keystone-to-Keystone (K2K) Federation is the only"},{"line_number":242,"context_line":"existing option today."},{"line_number":243,"context_line":""},{"line_number":244,"context_line":""},{"line_number":245,"context_line":"Security Impact"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5f7c97a3_6f657fe0","line":242,"range":{"start_line":241,"start_character":0,"end_line":242,"end_character":22},"in_reply_to":"5f7c97a3_e1068886","updated":"2018-06-13 02:57:22.000000000","message":"I might have mentioned somewhere that I need help on this as this line came from the original document here, I didn\u0027t write this.   I did hear that someone did a talk at the last summit that compared a few different approaches like using cockroachDB, jaosorior also referred to some other product on this review.","commit_id":"bdc07ee024586c59e41d56485006100aa17aa2bb"}]}