)]}'
{"specs/branchless-tempest.rst":[{"author":{"_account_id":1192,"name":"David Kranz","email":"dkranz@redhat.com","username":"david-kranz"},"change_message_id":"dba55bda8e59a0935028c501933969bee6e9e13b","unresolved":false,"context_lines":[{"line_number":25,"context_line":"Tempest is largely black box testing for the OpenStack API. The API is"},{"line_number":26,"context_line":"represented by major versions, not by release tags, so Tempest should"},{"line_number":27,"context_line":"not need branches. This change would have a number of interesting"},{"line_number":28,"context_line":"cascading impacts, all of which should be possitive for the health of"},{"line_number":29,"context_line":"the project."},{"line_number":30,"context_line":""},{"line_number":31,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":1,"id":"AAAAW3%2F%2FyB8%3D","line":28,"updated":"2014-04-10 13:29:26.000000000","message":"positive","commit_id":"5d88e10900d8695c317668ceba876e4bf0d8af63"},{"author":{"_account_id":5196,"name":"Matthew Treinish","email":"mtreinish@kortar.org","username":"treinish"},"change_message_id":"9eb326d65c2a068a3e757148314407f677a80008","unresolved":false,"context_lines":[{"line_number":100,"context_line":"- land changed test in Tempest"},{"line_number":101,"context_line":""},{"line_number":102,"context_line":"In this new model the change *will also* need to be backported to"},{"line_number":103,"context_line":"stable/icehouse simultaneously. Or we decide that this is behavior"},{"line_number":104,"context_line":"that should not be tested for, and drop the test entirely."},{"line_number":105,"context_line":""},{"line_number":106,"context_line":"Scenario 3: New Tests for existing features"}],"source_content_type":"text/x-rst","patch_set":1,"id":"AAAAW3%2F%2Fwws%3D","line":103,"updated":"2014-04-10 20:15:48.000000000","message":"I\u0027d probably reword this to say before we land the changed tempest test just to make it clear we can\u0027t land the unskip until everything is backported.","commit_id":"5d88e10900d8695c317668ceba876e4bf0d8af63"},{"author":{"_account_id":1192,"name":"David Kranz","email":"dkranz@redhat.com","username":"david-kranz"},"change_message_id":"dba55bda8e59a0935028c501933969bee6e9e13b","unresolved":false,"context_lines":[{"line_number":107,"context_line":"-------------------------------------------"},{"line_number":108,"context_line":"When adding new tests for existing features the new tests will need to"},{"line_number":109,"context_line":"simultaneously pass when tested against nova master and nova"},{"line_number":110,"context_line":"stable/havana. If we discover there is a difference of behavior"},{"line_number":111,"context_line":"between the two, we\u0027ll need to harmonize that behavior before landing"},{"line_number":112,"context_line":"the test. We end up with 3 options:"},{"line_number":113,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"AAAAW3%2F%2Fx%2Fk%3D","line":110,"updated":"2014-04-10 13:29:26.000000000","message":"stable/icehouse","commit_id":"5d88e10900d8695c317668ceba876e4bf0d8af63"},{"author":{"_account_id":1192,"name":"David Kranz","email":"dkranz@redhat.com","username":"david-kranz"},"change_message_id":"dba55bda8e59a0935028c501933969bee6e9e13b","unresolved":false,"context_lines":[{"line_number":146,"context_line":"  API does what we expect in Tempest, but also that it acts the same"},{"line_number":147,"context_line":"  across multiple versions. The added friction to breaking that kind"},{"line_number":148,"context_line":"  of behavior will slow down any future breaks there."},{"line_number":149,"context_line":""},{"line_number":150,"context_line":"Alternatives"},{"line_number":151,"context_line":"------------"},{"line_number":152,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"AAAAW3%2F%2Fx30%3D","line":149,"updated":"2014-04-10 13:29:26.000000000","message":"This is also a major change to the API stability guidelines (https://wiki.openstack.org/wiki/APIChangeGuidelines) which are based on defining an api change as something that could break existing apps, but allows behavior changes that are not discoverable under some circumstances.  This proposal would effectively eliminate the first four items from the Acceptable list.\n\nThis change will also allow all work to improve Tempest in ways other than adding tests to happen on master, but still be useful when testing older versions.\n\nIt may be that some people will continue to want to run tempest against no-longer-supported releases. In that case a branch or fork could be created. This was the original motivation of the current stable branch system.","commit_id":"5d88e10900d8695c317668ceba876e4bf0d8af63"},{"author":{"_account_id":1192,"name":"David Kranz","email":"dkranz@redhat.com","username":"david-kranz"},"change_message_id":"dba55bda8e59a0935028c501933969bee6e9e13b","unresolved":false,"context_lines":[{"line_number":147,"context_line":"  across multiple versions. The added friction to breaking that kind"},{"line_number":148,"context_line":"  of behavior will slow down any future breaks there."},{"line_number":149,"context_line":""},{"line_number":150,"context_line":"Alternatives"},{"line_number":151,"context_line":"------------"},{"line_number":152,"context_line":""},{"line_number":153,"context_line":"The alternative is to continue with the current approach which uses"}],"source_content_type":"text/x-rst","patch_set":1,"id":"AAAAW3%2F%2Fx1A%3D","line":150,"updated":"2014-04-10 13:29:26.000000000","message":"Status of Havana. Given that refstack is going to use havana to begin with, it would be beneficial to be able to run master tempset against havana. The problem is that there have been a large number of api changes that would not be allowed, or would have had to be backported, under the proposed scheme. Thus there are a lot of errors when running master against havana. Here is an example: http://logs.openstack.org/16/86416/2/check/check-tempest-master-dsvm-full-havana/4d1fffb/console.html\n\nSome of the errors may be bugs but most are just api changes. We could have a mechanism in master to skip these tests when running against havana.","commit_id":"5d88e10900d8695c317668ceba876e4bf0d8af63"},{"author":{"_account_id":1192,"name":"David Kranz","email":"dkranz@redhat.com","username":"david-kranz"},"change_message_id":"dba55bda8e59a0935028c501933969bee6e9e13b","unresolved":false,"context_lines":[{"line_number":153,"context_line":"The alternative is to continue with the current approach which uses"},{"line_number":154,"context_line":"stable/* branches in Tempest. However that has caused more"},{"line_number":155,"context_line":"unintentional coupling with devstack that we\u0027d like, and I believe"},{"line_number":156,"context_line":"that long term it\u0027s the wrong approach for a test suite like Tempest."},{"line_number":157,"context_line":""},{"line_number":158,"context_line":"Implementation"},{"line_number":159,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"AAAAW3%2F%2Fxys%3D","line":156,"updated":"2014-04-10 13:29:26.000000000","message":"It also causes massive pain as people roll out real deployments of a release well after the release date. Testing of these deployments cannot take advantage of new tests and features that have appeared in tempest, and cannot fix problems in tempest because we do not want to deal with churn on stable branches. This is a particular problem when the tempest issue causing problems is due to a deployment that uses a configuration not tested in the upstream gate: nova-network with vlans for example.","commit_id":"5d88e10900d8695c317668ceba876e4bf0d8af63"},{"author":{"_account_id":1192,"name":"David Kranz","email":"dkranz@redhat.com","username":"david-kranz"},"change_message_id":"f80bd841c47e46d072c775718f7a4a4f52255646","unresolved":false,"context_lines":[{"line_number":91,"context_line":""},{"line_number":92,"context_line":"In this new model the change *will also* need to be backported to"},{"line_number":93,"context_line":"stable/icehouse simultaneously. Or we decide that this is behavior"},{"line_number":94,"context_line":"that should not be tested for, and drop the test entirely."},{"line_number":95,"context_line":""},{"line_number":96,"context_line":"Scenario 3: New Tests for existing features"},{"line_number":97,"context_line":"-------------------------------------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"AAAAW3%2F%2FS3I%3D","line":94,"updated":"2014-04-15 14:30:15.000000000","message":"I still don\u0027t like saying this is so binary: in order to fix a bug in icehouse, it must be fixed in havana too, and the same going forward. It would be good to backport a fix if possible, but the folks adding tests have no control over whether something gets backported and I see no reason to block test improvements over this issue. I also think a lot of folks will want to continue to run master even after a stable/branch is no longer \"supported\" by upstream. A compromise is to provide a conf flag with a list of tests that might be skipped, and a decorator that says if a test should consult that list of tests. So if a new test, or bug fix, does not work for a stable branch (whether gated on or not), it could be tagged and then people configuring tempest (or devstack for stable branch if supported), could set it to skip. Doing this would also address the CD issue.\n\nThis mechanism would also make it easy to get stable/havana in the mix quickly.","commit_id":"a1b67cb5f833256247e5defbd0851202fa62d1a6"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"e7455f3644c80906e83c450cb70906d46ebc045f","unresolved":false,"context_lines":[{"line_number":91,"context_line":""},{"line_number":92,"context_line":"In this new model the change *will also* need to be backported to"},{"line_number":93,"context_line":"stable/icehouse simultaneously. Or we decide that this is behavior"},{"line_number":94,"context_line":"that should not be tested for, and drop the test entirely."},{"line_number":95,"context_line":""},{"line_number":96,"context_line":"Scenario 3: New Tests for existing features"},{"line_number":97,"context_line":"-------------------------------------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"AAAAW3%2F%2FSgw%3D","line":94,"in_reply_to":"AAAAW3%2F%2FS3I%3D","updated":"2014-04-15 14:52:03.000000000","message":"I\u0027m very much -1 about branch based skipping. Branched based skipping means that 2 openstack installations are different in no user detectable way. Which is one of the things this is trying to avoid.","commit_id":"a1b67cb5f833256247e5defbd0851202fa62d1a6"},{"author":{"_account_id":1192,"name":"David Kranz","email":"dkranz@redhat.com","username":"david-kranz"},"change_message_id":"44b40ecc2df0a6763f6ac0f96ce16d66204cba4f","unresolved":false,"context_lines":[{"line_number":91,"context_line":""},{"line_number":92,"context_line":"In this new model the change *will also* need to be backported to"},{"line_number":93,"context_line":"stable/icehouse simultaneously. Or we decide that this is behavior"},{"line_number":94,"context_line":"that should not be tested for, and drop the test entirely."},{"line_number":95,"context_line":""},{"line_number":96,"context_line":"Scenario 3: New Tests for existing features"},{"line_number":97,"context_line":"-------------------------------------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"AAAAW3%2F%2FQ68%3D","line":94,"in_reply_to":"AAAAW3%2F%2FSgw%3D","updated":"2014-04-15 17:26:21.000000000","message":"What I am proposing here is not branch-based skipping, and is a suggested addendum to this proposal. It is about providing a way to skip tests based on the conf file rather than having to check in skip annotations or use absurd regexps to configure a test execution, and thus being able to say \"here is the tempest.conf that corresponds to some deployment\" but you can run it on master. I","commit_id":"a1b67cb5f833256247e5defbd0851202fa62d1a6"},{"author":{"_account_id":1192,"name":"David Kranz","email":"dkranz@redhat.com","username":"david-kranz"},"change_message_id":"f80bd841c47e46d072c775718f7a4a4f52255646","unresolved":false,"context_lines":[{"line_number":105,"context_line":"- fix nova stable/icehouse to reflect nova master behavior (missing"},{"line_number":106,"context_line":"  backport)"},{"line_number":107,"context_line":"- decide to not land the Tempest test as it is attempting to verify"},{"line_number":108,"context_line":"  behavior we don\u0027t consider stable"},{"line_number":109,"context_line":""},{"line_number":110,"context_line":"Scenario 4: Obsolete Tests"},{"line_number":111,"context_line":"--------------------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"AAAAW3%2F%2FS2w%3D","line":108,"updated":"2014-04-15 14:30:15.000000000","message":"Same comment as scenario 2.","commit_id":"a1b67cb5f833256247e5defbd0851202fa62d1a6"},{"author":{"_account_id":1192,"name":"David Kranz","email":"dkranz@redhat.com","username":"david-kranz"},"change_message_id":"f80bd841c47e46d072c775718f7a4a4f52255646","unresolved":false,"context_lines":[{"line_number":132,"context_line":"  of tempest patches that will require a working stable/* gate to"},{"line_number":133,"context_line":"  land, stable/* will get a lot more regular testing, and be in a"},{"line_number":134,"context_line":"  working state more often"},{"line_number":135,"context_line":"- API guaruntees will tighten up. We\u0027ll not only be testing that the"},{"line_number":136,"context_line":"  API does what we expect in Tempest, but also that it acts the same"},{"line_number":137,"context_line":"  across multiple versions. The added friction to breaking that kind"},{"line_number":138,"context_line":"  of behavior will slow down any future breaks there. This means a"}],"source_content_type":"text/x-rst","patch_set":2,"id":"AAAAW3%2F%2FS08%3D","line":135,"updated":"2014-04-15 14:30:15.000000000","message":"guarantees","commit_id":"a1b67cb5f833256247e5defbd0851202fa62d1a6"}]}