)]}'
{"specs/tempest/add-interop-tag-to-defcore-tests.rst":[{"author":{"_account_id":97,"name":"Daryl Walleck","email":"daryl.walleck@rackspace.com","username":"dwalleck"},"change_message_id":"fdce67f378226971688db305cb58152f8384a0e5","unresolved":false,"context_lines":[{"line_number":28,"context_line":"- Do not require administrator access."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Currently, there is no way to directly identify which tests are defined to"},{"line_number":31,"context_line":"DefCore tests, leading to a ad-hoc and volatile methods for maintaining"},{"line_number":32,"context_line":"lists and tracking movement. By adding tags identifying DefCore tests,"},{"line_number":33,"context_line":"DefCore consumers can quickly identify which tests are required directly."},{"line_number":34,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a061dce_69930850","line":31,"updated":"2016-04-05 23:22:16.000000000","message":"Wasn\u0027t this the problem that the DefCore spec was designed to solve? I understand that there\u0027s no visibility directly in the tests, but could this be solved via tooling without duplicate effort?","commit_id":"5fb2b348aeaeee46eb81bf29e0e738d2fae92588"},{"author":{"_account_id":11797,"name":"Gema Gomez-Solano","email":"gema.gomez-solano@linaro.org","username":"gema"},"change_message_id":"73810d4f8ceca23e776b65193ede4733943f84e9","unresolved":false,"context_lines":[{"line_number":28,"context_line":"- Do not require administrator access."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Currently, there is no way to directly identify which tests are defined to"},{"line_number":31,"context_line":"DefCore tests, leading to a ad-hoc and volatile methods for maintaining"},{"line_number":32,"context_line":"lists and tracking movement. By adding tags identifying DefCore tests,"},{"line_number":33,"context_line":"DefCore consumers can quickly identify which tests are required directly."},{"line_number":34,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a061dce_d96938c0","line":31,"in_reply_to":"9a061dce_69930850","updated":"2016-04-06 07:14:48.000000000","message":"Adding a tag solves the issue, because then that test has an explicit dependency on DefCore and won\u0027t be changed without involving the DefCore committee, it solves the issue and makes test developers think about the implications of their changes, imo.","commit_id":"5fb2b348aeaeee46eb81bf29e0e738d2fae92588"},{"author":{"_account_id":6167,"name":"Ken\u0027ichi Ohmichi","email":"ken1ohmichi@gmail.com","username":"oomichi"},"change_message_id":"dd624694375ef7ddcaac38048e9230d63c0fbac3","unresolved":false,"context_lines":[{"line_number":38,"context_line":"(with that specification and associated tests maintained by the DefCore"},{"line_number":39,"context_line":"commitee in collaboration with the QA team)."},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"Finally, tags on tests will serve as markers to Tempest contributors"},{"line_number":42,"context_line":"to treat such tests with care during refactoring, renaming, and removal,"},{"line_number":43,"context_line":"and give an additional method for allowing the community to give feedback"},{"line_number":44,"context_line":"on the DefCore interoperability tests."},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"Proposed change"},{"line_number":47,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a061dce_725d7a14","line":44,"range":{"start_line":41,"start_character":0,"end_line":44,"end_character":38},"updated":"2016-04-06 18:59:53.000000000","message":"Now defcore contains the list of Tempest tests.\n\nAfter this work, I guess we can remove the list from defcore for avoiding maintaining burden on defcore side and we(including defcore people) can work together to improve Tempest. Is that right?\n\nSo maybe we need some tool also for getting the list from Tempest code, I am imaging.","commit_id":"5fb2b348aeaeee46eb81bf29e0e738d2fae92588"},{"author":{"_account_id":5196,"name":"Matthew Treinish","email":"mtreinish@kortar.org","username":"treinish"},"change_message_id":"c22ab51f4021ad0dac912c574eed64a102fe25f7","unresolved":false,"context_lines":[{"line_number":49,"context_line":"To add a new set of tags (in the form of a decorator) that will identify"},{"line_number":50,"context_line":"interoprability tests. Initially there will be three decorators:"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"* defcore-required, for existing DefCore tests."},{"line_number":53,"context_line":"* defcore-flagged, for existing DefCore tests that have been flagged."},{"line_number":54,"context_line":"* defcore-advisory, for advisory DefCore tests."},{"line_number":55,"context_line":"* interop, for tests that meet a DefCore defined interoperability spec."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a061dce_98ea280e","line":52,"updated":"2016-04-05 21:32:49.000000000","message":"So personally I don\u0027t think we want tags here that are based on the set of things in the defcore list. I\u0027d rather see the tags based on the things you\u0027re outlining below in the implementation section. Like single tenant, admin or not (which is already implicitly tagged by the admin dir), etc.\n\nI\u0027d rather see us provide this extra annotation to all users and then have the defcore committee use this as an input in defining the set of tests for it\u0027s needs.","commit_id":"5fb2b348aeaeee46eb81bf29e0e738d2fae92588"},{"author":{"_account_id":10526,"name":"Catherine Diep","email":"cdiep@us.ibm.com","username":"cdiep"},"change_message_id":"30174ddbd455fddf97bef2d21fdc9b7a6f80fd07","unresolved":false,"context_lines":[{"line_number":49,"context_line":"To add a new set of tags (in the form of a decorator) that will identify"},{"line_number":50,"context_line":"interoprability tests. Initially there will be three decorators:"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"* defcore-required, for existing DefCore tests."},{"line_number":53,"context_line":"* defcore-flagged, for existing DefCore tests that have been flagged."},{"line_number":54,"context_line":"* defcore-advisory, for advisory DefCore tests."},{"line_number":55,"context_line":"* interop, for tests that meet a DefCore defined interoperability spec."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a061dce_c9de943e","line":52,"in_reply_to":"9a061dce_98ea280e","updated":"2016-04-05 23:16:24.000000000","message":"Agree on using tags to describe things such as single tenant, admin ...  \n\nCurrently, there are tests that require admin credentials but are not tagged by the admin dir.  Please see examples in [1].\n\n[1] https://review.openstack.org/#/c/299491/","commit_id":"5fb2b348aeaeee46eb81bf29e0e738d2fae92588"},{"author":{"_account_id":97,"name":"Daryl Walleck","email":"daryl.walleck@rackspace.com","username":"dwalleck"},"change_message_id":"fdce67f378226971688db305cb58152f8384a0e5","unresolved":false,"context_lines":[{"line_number":49,"context_line":"To add a new set of tags (in the form of a decorator) that will identify"},{"line_number":50,"context_line":"interoprability tests. Initially there will be three decorators:"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"* defcore-required, for existing DefCore tests."},{"line_number":53,"context_line":"* defcore-flagged, for existing DefCore tests that have been flagged."},{"line_number":54,"context_line":"* defcore-advisory, for advisory DefCore tests."},{"line_number":55,"context_line":"* interop, for tests that meet a DefCore defined interoperability spec."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a061dce_8e2ca203","line":52,"in_reply_to":"9a061dce_98ea280e","updated":"2016-04-05 23:22:16.000000000","message":"I agree with the idea using tags specific to what rules should be followed by the test would be more clear than using defcore terminology. The rules aren\u0027t enforced by the test runner, but there\u0027s no easy solution to that problem","commit_id":"5fb2b348aeaeee46eb81bf29e0e738d2fae92588"},{"author":{"_account_id":11797,"name":"Gema Gomez-Solano","email":"gema.gomez-solano@linaro.org","username":"gema"},"change_message_id":"73810d4f8ceca23e776b65193ede4733943f84e9","unresolved":false,"context_lines":[{"line_number":49,"context_line":"To add a new set of tags (in the form of a decorator) that will identify"},{"line_number":50,"context_line":"interoprability tests. Initially there will be three decorators:"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"* defcore-required, for existing DefCore tests."},{"line_number":53,"context_line":"* defcore-flagged, for existing DefCore tests that have been flagged."},{"line_number":54,"context_line":"* defcore-advisory, for advisory DefCore tests."},{"line_number":55,"context_line":"* interop, for tests that meet a DefCore defined interoperability spec."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a061dce_999a2062","line":52,"in_reply_to":"9a061dce_98ea280e","updated":"2016-04-06 07:14:48.000000000","message":"Maybe we could have tags for single tenant, admin or not, etc, which are test description tags as well as having one that marks the test as DefCore relevant, to make sure any change to that test involves DefCore in the discussion.\n\nI think distinguishing between required, advisory and flagged may be confusing for test developers and generate doubt as to whether the test can be changed or not and why, whilst if we only have one tag, it is clear that they need external input because we rely, one way or another, on that test being there doing what it does.","commit_id":"5fb2b348aeaeee46eb81bf29e0e738d2fae92588"},{"author":{"_account_id":11797,"name":"Gema Gomez-Solano","email":"gema.gomez-solano@linaro.org","username":"gema"},"change_message_id":"73810d4f8ceca23e776b65193ede4733943f84e9","unresolved":false,"context_lines":[{"line_number":50,"context_line":"interoprability tests. Initially there will be three decorators:"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"* defcore-required, for existing DefCore tests."},{"line_number":53,"context_line":"* defcore-flagged, for existing DefCore tests that have been flagged."},{"line_number":54,"context_line":"* defcore-advisory, for advisory DefCore tests."},{"line_number":55,"context_line":"* interop, for tests that meet a DefCore defined interoperability spec."},{"line_number":56,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a061dce_d9cbf851","line":53,"updated":"2016-04-06 07:14:48.000000000","message":"What would flagged mean? that the test needs to be fixed or made more stable? doesn\u0027t that information belong in a bug with logs and further explanation?","commit_id":"5fb2b348aeaeee46eb81bf29e0e738d2fae92588"},{"author":{"_account_id":11797,"name":"Gema Gomez-Solano","email":"gema.gomez-solano@linaro.org","username":"gema"},"change_message_id":"73810d4f8ceca23e776b65193ede4733943f84e9","unresolved":false,"context_lines":[{"line_number":52,"context_line":"* defcore-required, for existing DefCore tests."},{"line_number":53,"context_line":"* defcore-flagged, for existing DefCore tests that have been flagged."},{"line_number":54,"context_line":"* defcore-advisory, for advisory DefCore tests."},{"line_number":55,"context_line":"* interop, for tests that meet a DefCore defined interoperability spec."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"These tags will use the existing decorator functionality within tempest,"},{"line_number":58,"context_line":"using the test.attr type attribute::"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a061dce_9936a050","line":55,"updated":"2016-04-06 07:14:48.000000000","message":"I think this one would be very useful for DefCore, not sure how it\u0027d impact QA\u0027s way of dealing with these tests, though.","commit_id":"5fb2b348aeaeee46eb81bf29e0e738d2fae92588"},{"author":{"_account_id":11797,"name":"Gema Gomez-Solano","email":"gema.gomez-solano@linaro.org","username":"gema"},"change_message_id":"73810d4f8ceca23e776b65193ede4733943f84e9","unresolved":false,"context_lines":[{"line_number":79,"context_line":"* The test must not require admin access."},{"line_number":80,"context_line":"* The test must not require multi-tenancy."},{"line_number":81,"context_line":"* The test must be an API test."},{"line_number":82,"context_line":"* The test should be as atomic as possible."},{"line_number":83,"context_line":""},{"line_number":84,"context_line":"The DefCore Committee will be responsible for maintaing the tags."},{"line_number":85,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a061dce_39cb1420","line":82,"updated":"2016-04-06 07:14:48.000000000","message":"Can we have an explanatory note on what \"atomic\" means in this context? I had the impression at the midcycle that we all meant subtly different things when we say \"atomic test\".","commit_id":"5fb2b348aeaeee46eb81bf29e0e738d2fae92588"},{"author":{"_account_id":97,"name":"Daryl Walleck","email":"daryl.walleck@rackspace.com","username":"dwalleck"},"change_message_id":"82a809f28c7e6fac36bc29bae3ca31052cb24718","unresolved":false,"context_lines":[{"line_number":79,"context_line":"* The test must not require admin access."},{"line_number":80,"context_line":"* The test must not require multi-tenancy."},{"line_number":81,"context_line":"* The test must be an API test."},{"line_number":82,"context_line":"* The test should be as atomic as possible."},{"line_number":83,"context_line":""},{"line_number":84,"context_line":"The DefCore Committee will be responsible for maintaing the tags."},{"line_number":85,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a061dce_13eed2b3","line":82,"in_reply_to":"9a061dce_39cb1420","updated":"2016-04-06 14:59:51.000000000","message":"To me, atomic means that the test should be targeted to one bit of functionality, since the capabilities are tied to the implementation of the test. This makes it very clear to determine what functionality is not DefCore compliant when a test fails (e.x test_get_image would be a good example of an atomic test. test_create_delete_image is a good example of a non-atomic, scenario-like test)","commit_id":"5fb2b348aeaeee46eb81bf29e0e738d2fae92588"}]}