)]}'
{"specs/testing-guidelines.rst":[{"author":{"_account_id":308,"name":"Thierry Carrez","email":"thierry@openstack.org","username":"ttx"},"change_message_id":"11060b3e2e5b96f2735f2e8c3f1a74b03df2ec19","unresolved":false,"context_lines":[{"line_number":2,"context_line":""},{"line_number":3,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":4,"context_line":" OpenStack Theory of Testing"},{"line_number":5,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"The following is an attempt to provide a \"theory of testing\" to be"},{"line_number":8,"context_line":"used in OpenStack projects. It accumulates what we\u0027ve learned over the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_e6d689f0","line":5,"updated":"2015-02-04 13:40:50.000000000","message":"Align \u003d\u003d\u003d with title","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":308,"name":"Thierry Carrez","email":"thierry@openstack.org","username":"ttx"},"change_message_id":"11060b3e2e5b96f2735f2e8c3f1a74b03df2ec19","unresolved":false,"context_lines":[{"line_number":4,"context_line":" OpenStack Theory of Testing"},{"line_number":5,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"The following is an attempt to provide a \"theory of testing\" to be"},{"line_number":8,"context_line":"used in OpenStack projects. It accumulates what we\u0027ve learned over the"},{"line_number":9,"context_line":"past few years of the OpenStack project in various iterations of"},{"line_number":10,"context_line":"automated testing. It also attempts to provide guidance for projects"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_c6f0a593","line":7,"updated":"2015-02-04 13:40:50.000000000","message":"I think this is sufficiently solid to be presented as a spec, rather than as a braindump. So maybe you can remove this warning boilerplate.","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":308,"name":"Thierry Carrez","email":"thierry@openstack.org","username":"ttx"},"change_message_id":"11060b3e2e5b96f2735f2e8c3f1a74b03df2ec19","unresolved":false,"context_lines":[{"line_number":11,"context_line":"setting up testing in the future."},{"line_number":12,"context_line":""},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":15,"context_line":" Things we\u0027ve learned - usually the hard way"},{"line_number":16,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":17,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_4634b549","line":14,"updated":"2015-02-04 13:40:50.000000000","message":"Should be level 2 title, not level 1","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":308,"name":"Thierry Carrez","email":"thierry@openstack.org","username":"ttx"},"change_message_id":"11060b3e2e5b96f2735f2e8c3f1a74b03df2ec19","unresolved":false,"context_lines":[{"line_number":12,"context_line":""},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":15,"context_line":" Things we\u0027ve learned - usually the hard way"},{"line_number":16,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"1. If it\u0027s not tested, it\u0027s probably broken"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_067c7deb","line":15,"updated":"2015-02-04 13:40:50.000000000","message":"Maybe use the spec-classic \"Rationale\" section and wrap \"things we learned\" into it ?","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":308,"name":"Thierry Carrez","email":"thierry@openstack.org","username":"ttx"},"change_message_id":"11060b3e2e5b96f2735f2e8c3f1a74b03df2ec19","unresolved":false,"context_lines":[{"line_number":22,"context_line":"deployed. Live migration, Cells, and AWS interfaces are often broken"},{"line_number":23,"context_line":"in Nova. This is because of little to no testing on them. Most"},{"line_number":24,"context_line":"projects broke the first time they were hit with multiple API requests"},{"line_number":25,"context_line":"at the same time on the same API end point."},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"2. It is important to be skeptical of passing tests"},{"line_number":28,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_4659158d","line":25,"updated":"2015-02-04 13:40:50.000000000","message":"Furthermore, even if it\u0027s accidentally working, absence of tests will guarantee that it will break in the (near) future.","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":4715,"name":"Angus Salkeld","email":"asalkeld@redhat.com","username":"asalkeld"},"change_message_id":"421464f21952da7d96aa9e346c0a03b3771007fa","unresolved":false,"context_lines":[{"line_number":26,"context_line":""},{"line_number":27,"context_line":"2. It is important to be skeptical of passing tests"},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Because you wrote a test case, and it passes, doesn\u0027t mean anything is"},{"line_number":30,"context_line":"working correctly. In the past configuration changes have happened"},{"line_number":31,"context_line":"that made test scenarios run in ways that were unexpected, and no"},{"line_number":32,"context_line":"longer testing what we believed they were. This has happened with"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa81d914_1f10c64d","line":29,"updated":"2015-01-28 06:00:22.000000000","message":"s/anything/everything ?","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"cf153fb8fba2b7dab8502e682afd384bcb223dd8","unresolved":false,"context_lines":[{"line_number":26,"context_line":""},{"line_number":27,"context_line":"2. It is important to be skeptical of passing tests"},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Because you wrote a test case, and it passes, doesn\u0027t mean anything is"},{"line_number":30,"context_line":"working correctly. In the past configuration changes have happened"},{"line_number":31,"context_line":"that made test scenarios run in ways that were unexpected, and no"},{"line_number":32,"context_line":"longer testing what we believed they were. This has happened with"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa81d914_db953741","line":29,"in_reply_to":"fa81d914_1f10c64d","updated":"2015-01-28 22:14:56.000000000","message":"actually, anything is what I intended.","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":308,"name":"Thierry Carrez","email":"thierry@openstack.org","username":"ttx"},"change_message_id":"11060b3e2e5b96f2735f2e8c3f1a74b03df2ec19","unresolved":false,"context_lines":[{"line_number":31,"context_line":"that made test scenarios run in ways that were unexpected, and no"},{"line_number":32,"context_line":"longer testing what we believed they were. This has happened with"},{"line_number":33,"context_line":"PostgreSQL and Cells in the past. It will happen with other things in"},{"line_number":34,"context_line":"the futue."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"Working tests should always have a documented way to show that they"},{"line_number":37,"context_line":"correctly fail when something is wrong."}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_09500e0a","line":34,"updated":"2015-02-04 13:40:50.000000000","message":"typo: future","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":308,"name":"Thierry Carrez","email":"thierry@openstack.org","username":"ttx"},"change_message_id":"11060b3e2e5b96f2735f2e8c3f1a74b03df2ec19","unresolved":false,"context_lines":[{"line_number":58,"context_line":"any of a large number of source trees, all with different review"},{"line_number":59,"context_line":"teams, review throughput, and cultures."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"These sorts of situations lead to lockups in the gate, and a fealing"},{"line_number":62,"context_line":"of powerlessness from team members. If the Neutron team can\u0027t merge"},{"line_number":63,"context_line":"bug fixes because of a bug in another repository, the team gets"},{"line_number":64,"context_line":"frustrated and gives up. If the Nova team\u0027s unit tests get broken by"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_896a3eb1","line":61,"updated":"2015-02-04 13:40:50.000000000","message":"Typo: feeling","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":6547,"name":"Andreas Jaeger","email":"jaegerandi@gmail.com","username":"jaegerandi"},"change_message_id":"2f1f8c4534041957de538a3f5b49a3f51a57c0da","unresolved":false,"context_lines":[{"line_number":73,"context_line":"bugs, some times this is the consequence of a complex system that has"},{"line_number":74,"context_line":"global state."},{"line_number":75,"context_line":""},{"line_number":76,"context_line":"This means failure or success is not binary. It\u0027s statisically"},{"line_number":77,"context_line":"significant. A failed test does not mean the code that was tested was"},{"line_number":78,"context_line":"at fault for the failure. It also means that the emergence of a new"},{"line_number":79,"context_line":"catostrophic break takes time to see. 1 test failure is not"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa81d914_d4b6efdb","line":76,"updated":"2015-01-28 19:44:04.000000000","message":"typo: statistically","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":6547,"name":"Andreas Jaeger","email":"jaegerandi@gmail.com","username":"jaegerandi"},"change_message_id":"2f1f8c4534041957de538a3f5b49a3f51a57c0da","unresolved":false,"context_lines":[{"line_number":76,"context_line":"This means failure or success is not binary. It\u0027s statisically"},{"line_number":77,"context_line":"significant. A failed test does not mean the code that was tested was"},{"line_number":78,"context_line":"at fault for the failure. It also means that the emergence of a new"},{"line_number":79,"context_line":"catostrophic break takes time to see. 1 test failure is not"},{"line_number":80,"context_line":"useful. Many test failures becomes useful."},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"False negatives are dangerous. They are dangerous in the same way that"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa81d914_74ce235c","line":79,"updated":"2015-01-28 19:44:04.000000000","message":"Typo: catastrophic. \n\nUse \"One test\"","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":6547,"name":"Andreas Jaeger","email":"jaegerandi@gmail.com","username":"jaegerandi"},"change_message_id":"2f1f8c4534041957de538a3f5b49a3f51a57c0da","unresolved":false,"context_lines":[{"line_number":89,"context_line":""},{"line_number":90,"context_line":"1. Test in a pyramid"},{"line_number":91,"context_line":""},{"line_number":92,"context_line":"Tests should start at the lowest possible level and guaruntee"},{"line_number":93,"context_line":"interfaces and behavior. This establishes a contract, which should"},{"line_number":94,"context_line":"carry to the next level."},{"line_number":95,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa81d914_74f54382","line":92,"updated":"2015-01-28 19:44:04.000000000","message":"typo: guarantee","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":6547,"name":"Andreas Jaeger","email":"jaegerandi@gmail.com","username":"jaegerandi"},"change_message_id":"2f1f8c4534041957de538a3f5b49a3f51a57c0da","unresolved":false,"context_lines":[{"line_number":103,"context_line":"Projects that can run on their own, should be tested on their own."},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"Projects that are tightly coupled (such as the compute kernel) can be"},{"line_number":106,"context_line":"cogated on each other."},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"Projects that rely on the compute kernel should assume it is"},{"line_number":109,"context_line":"functioning correctly, and test above it."}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa81d914_944e37c0","line":106,"updated":"2015-01-28 19:44:04.000000000","message":"You use further down \"co-gated\", let\u0027s be consistent","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":308,"name":"Thierry Carrez","email":"thierry@openstack.org","username":"ttx"},"change_message_id":"11060b3e2e5b96f2735f2e8c3f1a74b03df2ec19","unresolved":false,"context_lines":[{"line_number":117,"context_line":"This violates the ability to isolate failures or know whether or not"},{"line_number":118,"context_line":"those clients work in the general case."},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"4. Treat substantial false negatives at critical bugs"},{"line_number":121,"context_line":""},{"line_number":122,"context_line":"False negatives make people ignore the system, and rage"},{"line_number":123,"context_line":"quit. Addressing false negatives should be a high priority issue. If a"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_298a326a","line":120,"updated":"2015-02-04 13:40:50.000000000","message":"I suspect you meant \"as critical bugs\"","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":308,"name":"Thierry Carrez","email":"thierry@openstack.org","username":"ttx"},"change_message_id":"11060b3e2e5b96f2735f2e8c3f1a74b03df2ec19","unresolved":false,"context_lines":[{"line_number":122,"context_line":"False negatives make people ignore the system, and rage"},{"line_number":123,"context_line":"quit. Addressing false negatives should be a high priority issue. If a"},{"line_number":124,"context_line":"test for a feature can\u0027t be made reliable we shouldn\u0027t keep self"},{"line_number":125,"context_line":"inflicting damage on ourselves."},{"line_number":126,"context_line":""},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"Proposed changes"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_66ec5990","line":125,"updated":"2015-02-04 13:40:50.000000000","message":"It is a culture issue though. Setting bugs to \"critical\" never got them fixed faster. We need to educate people why false negatives are a priority to fix, so that they self-prioritize correctly.","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":5196,"name":"Matthew Treinish","email":"mtreinish@kortar.org","username":"treinish"},"change_message_id":"307ec864acc4b65424c09f5944c05e6939b47682","unresolved":false,"context_lines":[{"line_number":132,"context_line":"---------------"},{"line_number":133,"context_line":""},{"line_number":134,"context_line":"Adjust the Tempest testing and co-gating of projects in the following"},{"line_number":135,"context_line":"way:"},{"line_number":136,"context_line":""},{"line_number":137,"context_line":"1. Keystone"},{"line_number":138,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_690b418e","line":135,"updated":"2015-02-04 17:17:27.000000000","message":"Making this reasonable will require some additional effort in tempest to improve our per service tagging. It might be worth putting this detail somewhere. (or not, it doesn\u0027t really matter for this spec I guess)","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":6547,"name":"Andreas Jaeger","email":"jaegerandi@gmail.com","username":"jaegerandi"},"change_message_id":"2f1f8c4534041957de538a3f5b49a3f51a57c0da","unresolved":false,"context_lines":[{"line_number":137,"context_line":"1. Keystone"},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"   - runs only keystone"},{"line_number":140,"context_line":"   - runs only indentity tempest tests"},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"2. Swift"},{"line_number":143,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa81d914_9420f7f8","line":140,"updated":"2015-01-28 19:44:04.000000000","message":"identity\n\nShould there be some way of asking for a good coverage?","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":5196,"name":"Matthew Treinish","email":"mtreinish@kortar.org","username":"treinish"},"change_message_id":"307ec864acc4b65424c09f5944c05e6939b47682","unresolved":false,"context_lines":[{"line_number":156,"context_line":"4. Ironic"},{"line_number":157,"context_line":""},{"line_number":158,"context_line":"   - runs only ironic + keystone"},{"line_number":159,"context_line":"   - runs only ironic tests"},{"line_number":160,"context_line":""},{"line_number":161,"context_line":"5. All additional OpenStack projects"},{"line_number":162,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_89e6b5ae","line":159,"updated":"2015-02-04 17:17:27.000000000","message":"I get what you\u0027re going for here about the testing separation along api boundaries but 2 things here:\n\n1. Ironic\u0027s api is admin only which goes against your push to remove admn tests from tempest below.\n2. AIUI ironic is only really useful when used in conjunction with nova and the other services. I think you can take nova out of the equation but it still needs other projects to start a baremetal guest for example. (at least from what I\u0027ve been told)","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":308,"name":"Thierry Carrez","email":"thierry@openstack.org","username":"ttx"},"change_message_id":"11060b3e2e5b96f2735f2e8c3f1a74b03df2ec19","unresolved":false,"context_lines":[{"line_number":164,"context_line":"     service + keystone, or service + compute kernel)"},{"line_number":165,"context_line":"   - assume services outside your project work correctly for the"},{"line_number":166,"context_line":"     purposes of the tests"},{"line_number":167,"context_line":"   - only runs tests specific to their project"},{"line_number":168,"context_line":""},{"line_number":169,"context_line":"6. Allow for possible voluntary project co-gating"},{"line_number":170,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_6671b9df","line":167,"updated":"2015-02-04 13:40:50.000000000","message":"We have a number of projects (like Magnum) which depend on a compute-kernel + heat (mostly because the TC encouraged them to not reinvent orchestration). Would that form another possible \"base stack\" ?","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":5196,"name":"Matthew Treinish","email":"mtreinish@kortar.org","username":"treinish"},"change_message_id":"307ec864acc4b65424c09f5944c05e6939b47682","unresolved":false,"context_lines":[{"line_number":164,"context_line":"     service + keystone, or service + compute kernel)"},{"line_number":165,"context_line":"   - assume services outside your project work correctly for the"},{"line_number":166,"context_line":"     purposes of the tests"},{"line_number":167,"context_line":"   - only runs tests specific to their project"},{"line_number":168,"context_line":""},{"line_number":169,"context_line":"6. Allow for possible voluntary project co-gating"},{"line_number":170,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_de5ce54d","line":167,"in_reply_to":"da86d52c_6671b9df","updated":"2015-02-04 17:17:27.000000000","message":"I think the first bullet here covers that. So for magnum they would run the compute-kernel + heat because they require it.","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":4715,"name":"Angus Salkeld","email":"asalkeld@redhat.com","username":"asalkeld"},"change_message_id":"421464f21952da7d96aa9e346c0a03b3771007fa","unresolved":false,"context_lines":[{"line_number":169,"context_line":"6. Allow for possible voluntary project co-gating"},{"line_number":170,"context_line":""},{"line_number":171,"context_line":"   - a set of projects can decide they must be coupled together"},{"line_number":172,"context_line":"   - you can only be part of one coupling (to prevent transient coupling)"},{"line_number":173,"context_line":""},{"line_number":174,"context_line":"A bug in a project that is surfaced by another project (i.e. a Nova"},{"line_number":175,"context_line":"bug that breaks Heat) should be considered a high priority bug by both"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa81d914_5f007eaa","line":172,"updated":"2015-01-28 06:00:22.000000000","message":"would something like mimic be useful to mention here https://github.com/rackerlabs/mimic (once compute is tested, higher layers can \"mock\" the entire service). I haven\u0027t tried it, but seems interesting, and might help reduce load on infra.","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":6547,"name":"Andreas Jaeger","email":"jaegerandi@gmail.com","username":"jaegerandi"},"change_message_id":"2f1f8c4534041957de538a3f5b49a3f51a57c0da","unresolved":false,"context_lines":[{"line_number":170,"context_line":""},{"line_number":171,"context_line":"   - a set of projects can decide they must be coupled together"},{"line_number":172,"context_line":"   - you can only be part of one coupling (to prevent transient coupling)"},{"line_number":173,"context_line":""},{"line_number":174,"context_line":"A bug in a project that is surfaced by another project (i.e. a Nova"},{"line_number":175,"context_line":"bug that breaks Heat) should be considered a high priority bug by both"},{"line_number":176,"context_line":"teams. The fix and associated test case should only be done in the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa81d914_d4eeef8a","line":173,"updated":"2015-01-28 19:44:04.000000000","message":"Should we ask that these couplings (from 5 and 6) get documented as patch for this spec?","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":6547,"name":"Andreas Jaeger","email":"jaegerandi@gmail.com","username":"jaegerandi"},"change_message_id":"2f1f8c4534041957de538a3f5b49a3f51a57c0da","unresolved":false,"context_lines":[{"line_number":174,"context_line":"A bug in a project that is surfaced by another project (i.e. a Nova"},{"line_number":175,"context_line":"bug that breaks Heat) should be considered a high priority bug by both"},{"line_number":176,"context_line":"teams. The fix and associated test case should only be done in the"},{"line_number":177,"context_line":"consumed project (in this case Nova)."},{"line_number":178,"context_line":""},{"line_number":179,"context_line":"Prereqs: None"},{"line_number":180,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa81d914_d4bc2f6f","line":177,"updated":"2015-01-28 19:44:04.000000000","message":"consumed project is Heat in the above example.","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":5196,"name":"Matthew Treinish","email":"mtreinish@kortar.org","username":"treinish"},"change_message_id":"307ec864acc4b65424c09f5944c05e6939b47682","unresolved":false,"context_lines":[{"line_number":174,"context_line":"A bug in a project that is surfaced by another project (i.e. a Nova"},{"line_number":175,"context_line":"bug that breaks Heat) should be considered a high priority bug by both"},{"line_number":176,"context_line":"teams. The fix and associated test case should only be done in the"},{"line_number":177,"context_line":"consumed project (in this case Nova)."},{"line_number":178,"context_line":""},{"line_number":179,"context_line":"Prereqs: None"},{"line_number":180,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_1ea59d46","line":177,"in_reply_to":"fa81d914_d4bc2f6f","updated":"2015-02-04 17:17:27.000000000","message":"I think he meant that Nova is being consumed by Heat. So Heat is the consumer and Nova is consumed","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":6547,"name":"Andreas Jaeger","email":"jaegerandi@gmail.com","username":"jaegerandi"},"change_message_id":"2f1f8c4534041957de538a3f5b49a3f51a57c0da","unresolved":false,"context_lines":[{"line_number":182,"context_line":"changes in tests."},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Create Client Library Functional Testing"},{"line_number":185,"context_line":"-----------------------------"},{"line_number":186,"context_line":""},{"line_number":187,"context_line":"Client libraries should have their own functional test suites that"},{"line_number":188,"context_line":"expect to run against real end points. These should assume a working"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa81d914_94aab717","line":185,"updated":"2015-01-28 19:44:04.000000000","message":"Use same amount of \"-\" as line above.","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":5196,"name":"Matthew Treinish","email":"mtreinish@kortar.org","username":"treinish"},"change_message_id":"307ec864acc4b65424c09f5944c05e6939b47682","unresolved":false,"context_lines":[{"line_number":182,"context_line":"changes in tests."},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Create Client Library Functional Testing"},{"line_number":185,"context_line":"-----------------------------"},{"line_number":186,"context_line":""},{"line_number":187,"context_line":"Client libraries should have their own functional test suites that"},{"line_number":188,"context_line":"expect to run against real end points. These should assume a working"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_49a3ddb1","line":185,"in_reply_to":"fa81d914_94aab717","updated":"2015-02-04 17:17:27.000000000","message":"IIRC, it\u0027s supposed to match the length of the title strings. Gerrit sucks at showing this though so I can\u0027t tell whether it\u0027s correct here or not.","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":5196,"name":"Matthew Treinish","email":"mtreinish@kortar.org","username":"treinish"},"change_message_id":"307ec864acc4b65424c09f5944c05e6939b47682","unresolved":false,"context_lines":[{"line_number":197,"context_line":"-----------------------------------"},{"line_number":198,"context_line":""},{"line_number":199,"context_line":"Tempest should only be testing the public, user facing REST API. The"},{"line_number":200,"context_line":"admin APIs should be left to internal project testing."},{"line_number":201,"context_line":""},{"line_number":202,"context_line":"Prereqs: projects must spin up functional tests that cover this"},{"line_number":203,"context_line":"functionality."}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_fef9a94e","line":200,"updated":"2015-02-04 17:17:27.000000000","message":"When was this decided? I view an admin REST API as still being part of the public interface and having to adhere to the same contract as any other REST API. Especially because which API actions require admin is 100% configurable this distinction doesn\u0027t make much sense to me.\n\nThat being said I would be in favor of working to remove a large number of admin tests from tempest (especially for things like aggregates) probably for the same reason you do. The global state, failure and debug story is a nightmare for a lot of them. But, I view this sentence as drawing an arbitrary distinction about the API which I don\u0027t think was ever really discussed.","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":5196,"name":"Matthew Treinish","email":"mtreinish@kortar.org","username":"treinish"},"change_message_id":"307ec864acc4b65424c09f5944c05e6939b47682","unresolved":false,"context_lines":[{"line_number":228,"context_line":""},{"line_number":229,"context_line":"- Removal of admin tests from Tempest"},{"line_number":230,"context_line":""},{"line_number":231,"context_line":"  - TBD"},{"line_number":232,"context_line":""},{"line_number":233,"context_line":"Work Items"},{"line_number":234,"context_line":"----------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da86d52c_8455bcfb","line":231,"updated":"2015-02-04 17:17:27.000000000","message":"So I\u0027m fine with owning this, but I feel the 3 pronged rule and process that was agreed on for test removal of tempest testing should still apply here. Which means comparable testing will still have to exist elsewhere before we start removing things.","commit_id":"e438df731495405d8bf12bfc1ffbc484a2611242"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"b8e75ec8e66bf305c3d683dfb68ba5574dfbed1d","unresolved":false,"context_lines":[{"line_number":19,"context_line":"reasonable testing, it turned out that it didn\u0027t work when"},{"line_number":20,"context_line":"deployed. Live migration, Cells, and AWS interfaces are often broken"},{"line_number":21,"context_line":"in Nova. This is because of little to no testing on them. Most"},{"line_number":22,"context_line":"projects broke the first time they were hit with multiple API requests"},{"line_number":23,"context_line":"at the same time on the same API end point."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"Most active OpenStack projects see a 15-30% change in code (by git"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_ae02f606","line":22,"updated":"2015-02-09 21:53:33.000000000","message":"OpenStack has 2 well integrated testing frameworks currently in CI. Tempest and Rally. Rally is perfectly designed to catch issues under load and make races reproducible in gates. \n\nI think it makes sense to put here such details.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":20,"context_line":"deployed. Live migration, Cells, and AWS interfaces are often broken"},{"line_number":21,"context_line":"in Nova. This is because of little to no testing on them. Most"},{"line_number":22,"context_line":"projects broke the first time they were hit with multiple API requests"},{"line_number":23,"context_line":"at the same time on the same API end point."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"Most active OpenStack projects see a 15-30% change in code (by git"},{"line_number":26,"context_line":"diff) in each release, which means that even if a feature is working"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_2758a889","line":23,"updated":"2015-02-11 07:39:54.000000000","message":"The latter sentence probably deserves a \"lesson learned\" bullet of its own: You should test a component in the same way as it is used (or if you want, serial testing is handwave testing). We have plenty of experience from Neutron, Nova and other projects with races which went unnoticed with serial tests.\n\nAnd yeah Rally does that. But that\u0027s not the point of this paragraph in my opinion.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"481ddeb4a0088af22f91f7af6f6cf2466a9ecca9","unresolved":false,"context_lines":[{"line_number":21,"context_line":"in Nova. This is because of little to no testing on them. Most"},{"line_number":22,"context_line":"projects broke the first time they were hit with multiple API requests"},{"line_number":23,"context_line":"at the same time on the same API end point."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"Most active OpenStack projects see a 15-30% change in code (by git"},{"line_number":26,"context_line":"diff) in each release, which means that even if a feature is working"},{"line_number":27,"context_line":"today, if it\u0027s not tested, it\u0027s very likely to get accidentally broken"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_b2c32b89","line":24,"updated":"2015-02-10 06:30:58.000000000","message":"An interesting question; how do we avoid these types of features that aren\u0027t tested from staying in the project (and rotting)... It seems like some kind of reactive/administrative change should of been done for these (and if the situation didn\u0027t get better, delete the feature).","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"798818f899d6a96640ae06a19e2b937da02cf602","unresolved":false,"context_lines":[{"line_number":21,"context_line":"in Nova. This is because of little to no testing on them. Most"},{"line_number":22,"context_line":"projects broke the first time they were hit with multiple API requests"},{"line_number":23,"context_line":"at the same time on the same API end point."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"Most active OpenStack projects see a 15-30% change in code (by git"},{"line_number":26,"context_line":"diff) in each release, which means that even if a feature is working"},{"line_number":27,"context_line":"today, if it\u0027s not tested, it\u0027s very likely to get accidentally broken"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_c549a9b8","line":24,"in_reply_to":"da86d52c_b2c32b89","updated":"2015-02-10 13:44:56.000000000","message":"Honestly, beyond scope of this spec.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"b8e75ec8e66bf305c3d683dfb68ba5574dfbed1d","unresolved":false,"context_lines":[{"line_number":24,"context_line":""},{"line_number":25,"context_line":"Most active OpenStack projects see a 15-30% change in code (by git"},{"line_number":26,"context_line":"diff) in each release, which means that even if a feature is working"},{"line_number":27,"context_line":"today, if it\u0027s not tested, it\u0027s very likely to get accidentally broken"},{"line_number":28,"context_line":"tomorrow."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"2. It is important to be skeptical of passing tests"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_ee0cee3d","line":27,"updated":"2015-02-09 21:53:33.000000000","message":"+1","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"180572a10bc22adb3d9a60a0614fe5b36319ae3b","unresolved":false,"context_lines":[{"line_number":29,"context_line":""},{"line_number":30,"context_line":"2. It is important to be skeptical of passing tests"},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"Because you wrote a test case, and it passes, doesn\u0027t mean anything is"},{"line_number":33,"context_line":"actually working correctly. In the past configuration changes have"},{"line_number":34,"context_line":"happened that made test scenarios run in ways that were unexpected,"},{"line_number":35,"context_line":"and no longer testing what we believed they were. This has happened"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_681aa87d","line":32,"updated":"2015-02-16 20:05:22.000000000","message":"The wording here is a little awkward. I think you\u0027re saying something like:\n\n  Writing a test case and having it pass doesn\u0027t mean that\n  *anything* is actually working correctly.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"65ffaddb898237bffafdeade6eed5307e4a1d7e2","unresolved":false,"context_lines":[{"line_number":29,"context_line":""},{"line_number":30,"context_line":"2. It is important to be skeptical of passing tests"},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"Because you wrote a test case, and it passes, doesn\u0027t mean anything is"},{"line_number":33,"context_line":"actually working correctly. In the past configuration changes have"},{"line_number":34,"context_line":"happened that made test scenarios run in ways that were unexpected,"},{"line_number":35,"context_line":"and no longer testing what we believed they were. This has happened"}],"source_content_type":"text/x-rst","patch_set":3,"id":"ba7be1f8_86ffdb00","line":32,"in_reply_to":"da86d52c_681aa87d","updated":"2015-02-28 16:21:37.000000000","message":"Actually, I think Sean meant:\n\n Just because you wrote a passing test case does\n not mean everything is working correctly.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"65ffaddb898237bffafdeade6eed5307e4a1d7e2","unresolved":false,"context_lines":[{"line_number":30,"context_line":"2. It is important to be skeptical of passing tests"},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"Because you wrote a test case, and it passes, doesn\u0027t mean anything is"},{"line_number":33,"context_line":"actually working correctly. In the past configuration changes have"},{"line_number":34,"context_line":"happened that made test scenarios run in ways that were unexpected,"},{"line_number":35,"context_line":"and no longer testing what we believed they were. This has happened"},{"line_number":36,"context_line":"with PostgreSQL and Cells in the past. It will happen with other"}],"source_content_type":"text/x-rst","patch_set":3,"id":"ba7be1f8_6b01d205","line":33,"updated":"2015-02-28 16:21:37.000000000","message":"s/past configuration/past, configuration/","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":34,"context_line":"happened that made test scenarios run in ways that were unexpected,"},{"line_number":35,"context_line":"and no longer testing what we believed they were. This has happened"},{"line_number":36,"context_line":"with PostgreSQL and Cells in the past. It will happen with other"},{"line_number":37,"context_line":"things in the future."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"Working tests should always have a documented way to show that they"},{"line_number":40,"context_line":"correctly fail when something is wrong, otherwise their is no value in"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_67038042","line":37,"updated":"2015-02-11 07:39:54.000000000","message":"I think that a reference to the stories you\u0027re quoting here might be helpful for readers, if available. A mailing list thread, a bug report, or a review could be enough.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"180572a10bc22adb3d9a60a0614fe5b36319ae3b","unresolved":false,"context_lines":[{"line_number":34,"context_line":"happened that made test scenarios run in ways that were unexpected,"},{"line_number":35,"context_line":"and no longer testing what we believed they were. This has happened"},{"line_number":36,"context_line":"with PostgreSQL and Cells in the past. It will happen with other"},{"line_number":37,"context_line":"things in the future."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"Working tests should always have a documented way to show that they"},{"line_number":40,"context_line":"correctly fail when something is wrong, otherwise their is no value in"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_d55b9b83","line":37,"in_reply_to":"da86d52c_67038042","updated":"2015-02-16 20:05:22.000000000","message":"+1, if it\u0027s easy to find","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"b8e75ec8e66bf305c3d683dfb68ba5574dfbed1d","unresolved":false,"context_lines":[{"line_number":36,"context_line":"with PostgreSQL and Cells in the past. It will happen with other"},{"line_number":37,"context_line":"things in the future."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"Working tests should always have a documented way to show that they"},{"line_number":40,"context_line":"correctly fail when something is wrong, otherwise their is no value in"},{"line_number":41,"context_line":"them succeeding."},{"line_number":42,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_8e37f2af","line":39,"updated":"2015-02-09 21:53:33.000000000","message":"I agree with idea of this paragraph absolutely. But have 2 questions: \n\n  1) When it happens. Like we are writing test with assertions and it will fail if something went wrong? \n  2) Assuming that point 1 become clear after your reply, how can we avoid such situation?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"2f3b5e1ceea4c4a588ba6f1b4307f6aa7ac4d479","unresolved":false,"context_lines":[{"line_number":37,"context_line":"things in the future."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"Working tests should always have a documented way to show that they"},{"line_number":40,"context_line":"correctly fail when something is wrong, otherwise their is no value in"},{"line_number":41,"context_line":"them succeeding."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"3. The value of a Testing scenario is: when it breaks you know why"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_464a5128","line":40,"updated":"2015-02-09 16:48:16.000000000","message":"there","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"7df6fa8fb3013a54dfba76baf9e722fbba1a01e1","unresolved":false,"context_lines":[{"line_number":37,"context_line":"things in the future."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"Working tests should always have a documented way to show that they"},{"line_number":40,"context_line":"correctly fail when something is wrong, otherwise their is no value in"},{"line_number":41,"context_line":"them succeeding."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"3. The value of a Testing scenario is: when it breaks you know why"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_a5abfe87","line":40,"in_reply_to":"da86d52c_464a5128","updated":"2015-02-09 18:38:28.000000000","message":"thanks!","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":38,"context_line":""},{"line_number":39,"context_line":"Working tests should always have a documented way to show that they"},{"line_number":40,"context_line":"correctly fail when something is wrong, otherwise their is no value in"},{"line_number":41,"context_line":"them succeeding."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"3. The value of a Testing scenario is: when it breaks you know why"},{"line_number":44,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_8775d4cc","line":41,"updated":"2015-02-11 07:39:54.000000000","message":"Is this paragraph part of the lessons learned, or of the proposed test philosophy? \n\nI think this is a statement regarding how tests should be designed. For every step where a fault can occur, add an assertion to trigger a failure if the fault was observed, even if it did not trigger errors which caused the rest of the test to fail. If I did not understand your statement, probably some rewording might help. If I did understand it, probably some rewording might help anyway!","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2035,"name":"Maru Newby","email":"marun@redhat.com","username":"maru"},"change_message_id":"3b10d898015c2a3ca62fc305a1585ac40c47e65a","unresolved":false,"context_lines":[{"line_number":38,"context_line":""},{"line_number":39,"context_line":"Working tests should always have a documented way to show that they"},{"line_number":40,"context_line":"correctly fail when something is wrong, otherwise their is no value in"},{"line_number":41,"context_line":"them succeeding."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"3. The value of a Testing scenario is: when it breaks you know why"},{"line_number":44,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_d3e1802b","line":41,"in_reply_to":"da86d52c_8775d4cc","updated":"2015-02-17 23:35:44.000000000","message":"I read this sentence as \u0027Tests should be falsifiable; it should be obvious how they can fail\u0027.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"2f3b5e1ceea4c4a588ba6f1b4307f6aa7ac4d479","unresolved":false,"context_lines":[{"line_number":40,"context_line":"correctly fail when something is wrong, otherwise their is no value in"},{"line_number":41,"context_line":"them succeeding."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"3. The value of a Testing scenario is: when it breaks you know why"},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"Tests that always pass are uninteresting. They demonstrate that"},{"line_number":46,"context_line":"certain parts of the code are highly unlikely to be broken because"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_66e3d505","line":43,"updated":"2015-02-09 16:48:16.000000000","message":"This chunk seems a bit weak to me, in the sense that it is not clear how each of the paragraphs below connect to reinforce this statement above.\n\n(Sorry this is such a lame comment, I can\u0027t seem to capture what I really want to say.)","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"180572a10bc22adb3d9a60a0614fe5b36319ae3b","unresolved":false,"context_lines":[{"line_number":40,"context_line":"correctly fail when something is wrong, otherwise their is no value in"},{"line_number":41,"context_line":"them succeeding."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"3. The value of a Testing scenario is: when it breaks you know why"},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"Tests that always pass are uninteresting. They demonstrate that"},{"line_number":46,"context_line":"certain parts of the code are highly unlikely to be broken because"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_b5e627ae","line":43,"in_reply_to":"da86d52c_25c8ee73","updated":"2015-02-16 20:05:22.000000000","message":"Is this rewording capturing what you mean?\n\n Tests are most valuable when it is easy to understand why they fail.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"b8e75ec8e66bf305c3d683dfb68ba5574dfbed1d","unresolved":false,"context_lines":[{"line_number":40,"context_line":"correctly fail when something is wrong, otherwise their is no value in"},{"line_number":41,"context_line":"them succeeding."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"3. The value of a Testing scenario is: when it breaks you know why"},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"Tests that always pass are uninteresting. They demonstrate that"},{"line_number":46,"context_line":"certain parts of the code are highly unlikely to be broken because"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_4e1a5a18","line":43,"in_reply_to":"da86d52c_25c8ee73","updated":"2015-02-09 21:53:33.000000000","message":"It actually depends more on project then on testing tool. \n\nI mean to make it clear why some test fail you should have at least all logs related to all requests that was done during the test.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":40,"context_line":"correctly fail when something is wrong, otherwise their is no value in"},{"line_number":41,"context_line":"them succeeding."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"3. The value of a Testing scenario is: when it breaks you know why"},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"Tests that always pass are uninteresting. They demonstrate that"},{"line_number":46,"context_line":"certain parts of the code are highly unlikely to be broken because"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_4a5be1f5","line":43,"in_reply_to":"da86d52c_4e1a5a18","updated":"2015-02-11 07:39:54.000000000","message":"There is also a \"reverse\" lesson learned here. A test which does verify the correctness of operations in all components exercised is pointless and generates false negatives (or false positive if by positive you mean a success outcome)","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"7df6fa8fb3013a54dfba76baf9e722fbba1a01e1","unresolved":false,"context_lines":[{"line_number":40,"context_line":"correctly fail when something is wrong, otherwise their is no value in"},{"line_number":41,"context_line":"them succeeding."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"3. The value of a Testing scenario is: when it breaks you know why"},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"Tests that always pass are uninteresting. They demonstrate that"},{"line_number":46,"context_line":"certain parts of the code are highly unlikely to be broken because"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_25c8ee73","line":43,"in_reply_to":"da86d52c_66e3d505","updated":"2015-02-09 18:38:28.000000000","message":"so the 3rd paragraph more directly enforces this statement. I agree this is probably one of the weaker prose sections of the document.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"65ffaddb898237bffafdeade6eed5307e4a1d7e2","unresolved":false,"context_lines":[{"line_number":42,"context_line":""},{"line_number":43,"context_line":"3. The value of a Testing scenario is: when it breaks you know why"},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"Tests that always pass are uninteresting. They demonstrate that"},{"line_number":46,"context_line":"certain parts of the code are highly unlikely to be broken because"},{"line_number":47,"context_line":"they are too stable or too uninteresting to see changes."},{"line_number":48,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"ba7be1f8_461de365","line":45,"updated":"2015-02-28 16:21:37.000000000","message":"Tests that may *seem* uninteresting because they test parts of the code base that are \"too stable\" actually do serve a purpose: they make changing the behaviour of the code generate a test failure, and thus enforce a kind of checks-and-balances system in the code base.\n\nPut another way: if you add (or, particularly, change the behaviour of) a bunch of code, and you re-run your tests, and nothing fails, it is likely that you do not have adequate unit test coverage.\n\nHaving seemingly uninteresting tests in your test arsenal provide some rigidity an robustness to the code base by making it more obvious when changes are made in the code base.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":44,"context_line":""},{"line_number":45,"context_line":"Tests that always pass are uninteresting. They demonstrate that"},{"line_number":46,"context_line":"certain parts of the code are highly unlikely to be broken because"},{"line_number":47,"context_line":"they are too stable or too uninteresting to see changes."},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"Tests that fail, especially on initial patch submission, are very"},{"line_number":50,"context_line":"interesting because these are preventing people from changing"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_2af14d01","line":47,"updated":"2015-02-11 07:39:54.000000000","message":"what do you mean for \"uninteresting test\"? The fact that a code path is \"too stable\" or \"too uninteresting to see change\" obviously does not mean that there should be no interest in testing that.\n\nI\u0027m sure this is not what you meant, but to some readers, especially the non-english speaking one, it might sound that way.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2035,"name":"Maru Newby","email":"marun@redhat.com","username":"maru"},"change_message_id":"3b10d898015c2a3ca62fc305a1585ac40c47e65a","unresolved":false,"context_lines":[{"line_number":48,"context_line":""},{"line_number":49,"context_line":"Tests that fail, especially on initial patch submission, are very"},{"line_number":50,"context_line":"interesting because these are preventing people from changing"},{"line_number":51,"context_line":"something."},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"Tests that fail, but don\u0027t provide a reasonable set of bread crumbs to"},{"line_number":54,"context_line":"debug aren\u0027t very helpful. They demonstrate long standing issues that"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_7845f641","line":51,"updated":"2015-02-17 23:35:44.000000000","message":"I think there is a caveat to this statement: tests whose only purpose is to detect when the targeted code changes - as opposed to when behavior changes - are an anti-pattern.\n\nReference: http://googletesting.blogspot.com/2015/01/testing-on-toilet-change-detector-tests.html","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":48,"context_line":""},{"line_number":49,"context_line":"Tests that fail, especially on initial patch submission, are very"},{"line_number":50,"context_line":"interesting because these are preventing people from changing"},{"line_number":51,"context_line":"something."},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"Tests that fail, but don\u0027t provide a reasonable set of bread crumbs to"},{"line_number":54,"context_line":"debug aren\u0027t very helpful. They demonstrate long standing issues that"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_8adb197b","line":51,"updated":"2015-02-11 07:39:54.000000000","message":"well, technically that is true but when the patch is merged, not submitted, I think.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":53,"context_line":"Tests that fail, but don\u0027t provide a reasonable set of bread crumbs to"},{"line_number":54,"context_line":"debug aren\u0027t very helpful. They demonstrate long standing issues that"},{"line_number":55,"context_line":"we can\u0027t fix. This can often come from having too many moving parts,"},{"line_number":56,"context_line":"and not enough isolation on the scenarios being tested."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"4. Co-gating and the heavy reliance on Tempest jobs makes the system"},{"line_number":59,"context_line":"   fragile"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_cab8b113","line":56,"updated":"2015-02-11 07:39:54.000000000","message":"You can quote the tests which fail with \"unable to SSH into x.x.x.x\" as an example. It lead to use to use the same bug report to fix at least 10 different bugs, some of them in neutron, some of them in tempest.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":8788,"name":"Miguel Angel Ajo","email":"mangelajo@redhat.com","username":"mangelajo"},"change_message_id":"550f1ae5b37635876addf03df3357a31e70b4c32","unresolved":false,"context_lines":[{"line_number":53,"context_line":"Tests that fail, but don\u0027t provide a reasonable set of bread crumbs to"},{"line_number":54,"context_line":"debug aren\u0027t very helpful. They demonstrate long standing issues that"},{"line_number":55,"context_line":"we can\u0027t fix. This can often come from having too many moving parts,"},{"line_number":56,"context_line":"and not enough isolation on the scenarios being tested."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"4. Co-gating and the heavy reliance on Tempest jobs makes the system"},{"line_number":59,"context_line":"   fragile"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_f04f2332","line":56,"in_reply_to":"da86d52c_cab8b113","updated":"2015-02-11 10:54:02.000000000","message":"That\u0027s a good example, yes.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2035,"name":"Maru Newby","email":"marun@redhat.com","username":"maru"},"change_message_id":"3b10d898015c2a3ca62fc305a1585ac40c47e65a","unresolved":false,"context_lines":[{"line_number":53,"context_line":"Tests that fail, but don\u0027t provide a reasonable set of bread crumbs to"},{"line_number":54,"context_line":"debug aren\u0027t very helpful. They demonstrate long standing issues that"},{"line_number":55,"context_line":"we can\u0027t fix. This can often come from having too many moving parts,"},{"line_number":56,"context_line":"and not enough isolation on the scenarios being tested."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"4. Co-gating and the heavy reliance on Tempest jobs makes the system"},{"line_number":59,"context_line":"   fragile"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_f87e4669","line":56,"in_reply_to":"da86d52c_f04f2332","updated":"2015-02-17 23:35:44.000000000","message":"Does that suggest that non-trivial (probably everything above unit) should make a best-effort attempt to provide an explanation of possible failure?  Via assertion messages or otherwise?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"65ffaddb898237bffafdeade6eed5307e4a1d7e2","unresolved":false,"context_lines":[{"line_number":53,"context_line":"Tests that fail, but don\u0027t provide a reasonable set of bread crumbs to"},{"line_number":54,"context_line":"debug aren\u0027t very helpful. They demonstrate long standing issues that"},{"line_number":55,"context_line":"we can\u0027t fix. This can often come from having too many moving parts,"},{"line_number":56,"context_line":"and not enough isolation on the scenarios being tested."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"4. Co-gating and the heavy reliance on Tempest jobs makes the system"},{"line_number":59,"context_line":"   fragile"}],"source_content_type":"text/x-rst","patch_set":3,"id":"ba7be1f8_864d1b75","line":56,"in_reply_to":"da86d52c_f87e4669","updated":"2015-02-28 16:21:37.000000000","message":"Yes, Maru, I think that is a good suggestion.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6486,"name":"Brant Knudson","email":"blk@acm.org","username":"blk-u"},"change_message_id":"bda4231184a5c492d90f9f93ce50297b42675618","unresolved":false,"context_lines":[{"line_number":56,"context_line":"and not enough isolation on the scenarios being tested."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"4. Co-gating and the heavy reliance on Tempest jobs makes the system"},{"line_number":59,"context_line":"   fragile"},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"Fixing problems becomes much harder when the potential bug can be in"},{"line_number":62,"context_line":"any of a large number of source trees, all with different review"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_80f1886c","line":59,"updated":"2015-02-09 19:19:25.000000000","message":"What system? This might be confusing cause with effect. In other words, the system is fragile, but it\u0027s not because of co-gating. Isn\u0027t co-gating catching legitimate problems?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2035,"name":"Maru Newby","email":"marun@redhat.com","username":"maru"},"change_message_id":"3b10d898015c2a3ca62fc305a1585ac40c47e65a","unresolved":false,"context_lines":[{"line_number":56,"context_line":"and not enough isolation on the scenarios being tested."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"4. Co-gating and the heavy reliance on Tempest jobs makes the system"},{"line_number":59,"context_line":"   fragile"},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"Fixing problems becomes much harder when the potential bug can be in"},{"line_number":62,"context_line":"any of a large number of source trees, all with different review"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_3890de95","line":59,"in_reply_to":"da86d52c_7350b9d4","updated":"2015-02-17 23:35:44.000000000","message":"It might be worth mentioning how co-gating allows applications to increasingly rely on the implicit behavior of their co-gating siblings rather than forcing all involved to rely only on explicit API contracts.  The resulting combinatorial increase of interaction complexity would appear to be having a severe impact on the velocity of OpenStack as a whole.\n\nThen again, the need to move towards reliance on explicit contracts between applications would appear to be an issue larger than just testing.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"b32ef34d2fcd60b1f5545f84034fe778840b0cf0","unresolved":false,"context_lines":[{"line_number":56,"context_line":"and not enough isolation on the scenarios being tested."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"4. Co-gating and the heavy reliance on Tempest jobs makes the system"},{"line_number":59,"context_line":"   fragile"},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"Fixing problems becomes much harder when the potential bug can be in"},{"line_number":62,"context_line":"any of a large number of source trees, all with different review"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_7350b9d4","line":59,"in_reply_to":"da86d52c_80f1886c","updated":"2015-02-10 18:57:11.000000000","message":"The system \u003d \"the productivity of the OpenStack dev community\"?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"d249a018d1a7229133127738c5805695bd3410b1","unresolved":false,"context_lines":[{"line_number":56,"context_line":"and not enough isolation on the scenarios being tested."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"4. Co-gating and the heavy reliance on Tempest jobs makes the system"},{"line_number":59,"context_line":"   fragile"},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"Fixing problems becomes much harder when the potential bug can be in"},{"line_number":62,"context_line":"any of a large number of source trees, all with different review"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_9dd64e26","line":59,"in_reply_to":"da86d52c_80f1886c","updated":"2015-02-09 20:23:53.000000000","message":"This is a very good point and a worthwhile point of dicussion: Does the gate fail because co-gating is fragile or because our code, en-masse, ain\u0027t that good? If we attempt to solve the co-gating issues with a technical efforts (a sort of divide and maybe conquer) will we run the risk of not fixing things?\n\nIs there any way to address the social problem which is that so much code is weak and not fixed?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"5940048e53e33905bb1aa62e671da881d123e5b0","unresolved":false,"context_lines":[{"line_number":56,"context_line":"and not enough isolation on the scenarios being tested."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"4. Co-gating and the heavy reliance on Tempest jobs makes the system"},{"line_number":59,"context_line":"   fragile"},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"Fixing problems becomes much harder when the potential bug can be in"},{"line_number":62,"context_line":"any of a large number of source trees, all with different review"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_cd1e1970","line":59,"in_reply_to":"da86d52c_9dd64e26","updated":"2015-02-10 22:29:51.000000000","message":"I think its a bit of both: I\u0027m  reviewing a lot of bugs at the moment and we\u0027ve had failures because of incorrect assumptions about runtime environments (e.g. that nova is the exclusive user of libvirt on a system), because of the lack of examination of code paths as APIs change (e.g. the structure of neutron ports was changed in the Nova codebase, and a rarely-called function wasn\u0027t updated [you can argue that static type checking would have caught this, or more testing, or more awareness of the code, or ..., but the key thing is that that code path was not inspected by anything, human or machine, in the change]. We\u0027ve had bugs where our code was just confusing and hard to reason about. We\u0027ve bugs where we\u0027re missing support in our code to ensure valid states, and so on.\n\nIMO co-gating is catching legitimate problems, but its also making fixing those problems very very hard. It turns localised non-deterministic failures (e.g. race conditions on integration points) into a hard stop for all development, not just the code with the bug, and if you recall the formula for uptime in a cluster its the product of the uptime of all the components. So when we co-gate lets say just the big ones - thats nova, cinder, swift, keystone, neutron, and their clients, so 10 projects - if they each have their trunk passing tests 99.9% of the time, the total co-gate reliability will be approximately 99.1%, which is 8 times more failures experienced as a whole. If the reliability rate for the projects is lower - say the clients are 100% but the APIs are down at 99% - we would expect approximately 95% - and it gets rapidly worse. If we want to support a high rate of change, we need highly reliable CI - that is a very low test failure rate on pre-commit tests (except for the case where the patch actually is at fault of course!). With our pipeline strategy for test execution, each failure costs use approximately an hour - 24 failures a day will completely stall the patches behind those failures.\n\nIn terms of making our code less weak and better - I am putting energy into that sort of direction at the moment, and I\u0027d be delighted if you want to help :)","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"481ddeb4a0088af22f91f7af6f6cf2466a9ecca9","unresolved":false,"context_lines":[{"line_number":56,"context_line":"and not enough isolation on the scenarios being tested."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"4. Co-gating and the heavy reliance on Tempest jobs makes the system"},{"line_number":59,"context_line":"   fragile"},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"Fixing problems becomes much harder when the potential bug can be in"},{"line_number":62,"context_line":"any of a large number of source trees, all with different review"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_b2716b79","line":59,"in_reply_to":"da86d52c_9dd64e26","updated":"2015-02-10 06:30:58.000000000","message":"Seems like we need a TC/other that unites people behind common causes and leads everyone to a better land (not to be mistaken with moses).","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":67,"context_line":"bug fixes because of a bug in another repository, the team gets"},{"line_number":68,"context_line":"frustrated and gives up. If the Nova team\u0027s unit tests get broken by"},{"line_number":69,"context_line":"an Oslo library release, the team is frustrated and productivity goes"},{"line_number":70,"context_line":"way down."},{"line_number":71,"context_line":""},{"line_number":72,"context_line":"5. Test results in a complex system aren\u0027t binary, they are"},{"line_number":73,"context_line":"   statistical"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_8a7a3922","line":70,"updated":"2015-02-11 07:39:54.000000000","message":"I believe the analysis above is correct.\nBut I don\u0027t think it\u0027s fragility we should be talking about here.\nExcessive co-gating increases entropy, intended as dissipation of energy, both from resources doings gate testing, and from team members forced to debug failures they have no idea about.\n\nThe lesson learned here in my opinion is that co-gating uncorrelated systems (eg: cinder and trove) does not enable us to discover any new failure mode, but just annoys cinder users with trove failures and vice versa.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"65ffaddb898237bffafdeade6eed5307e4a1d7e2","unresolved":false,"context_lines":[{"line_number":67,"context_line":"bug fixes because of a bug in another repository, the team gets"},{"line_number":68,"context_line":"frustrated and gives up. If the Nova team\u0027s unit tests get broken by"},{"line_number":69,"context_line":"an Oslo library release, the team is frustrated and productivity goes"},{"line_number":70,"context_line":"way down."},{"line_number":71,"context_line":""},{"line_number":72,"context_line":"5. Test results in a complex system aren\u0027t binary, they are"},{"line_number":73,"context_line":"   statistical"}],"source_content_type":"text/x-rst","patch_set":3,"id":"ba7be1f8_464b2359","line":70,"in_reply_to":"da86d52c_8a7a3922","updated":"2015-02-28 16:21:37.000000000","message":"@Sal: ++, this is exactly correct.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"0758c964bba35041761ceb442a59acac09971f91","unresolved":false,"context_lines":[{"line_number":70,"context_line":"way down."},{"line_number":71,"context_line":""},{"line_number":72,"context_line":"5. Test results in a complex system aren\u0027t binary, they are"},{"line_number":73,"context_line":"   statistical"},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"Because we have shared global state in OpenStack services,"},{"line_number":76,"context_line":"interactions between tests are possible. Some times these are test"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_07ec9b8e","line":73,"updated":"2015-02-10 17:41:33.000000000","message":"I had difficulty trying to parse this point.\n\nIs what you\u0027re getting at here more like ...\n\n \"Individual test results are binary pass/fail, but their true value is often only\n  revealed in the accumulation of a certain failure pattern over many test runs\"","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"b32ef34d2fcd60b1f5545f84034fe778840b0cf0","unresolved":false,"context_lines":[{"line_number":70,"context_line":"way down."},{"line_number":71,"context_line":""},{"line_number":72,"context_line":"5. Test results in a complex system aren\u0027t binary, they are"},{"line_number":73,"context_line":"   statistical"},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"Because we have shared global state in OpenStack services,"},{"line_number":76,"context_line":"interactions between tests are possible. Some times these are test"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_38ab7e1f","line":73,"in_reply_to":"da86d52c_07ec9b8e","updated":"2015-02-10 18:57:11.000000000","message":"yeh, that\u0027s probably closer. I\u0027ll use some of that language in a follow up. Thanks.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2035,"name":"Maru Newby","email":"marun@redhat.com","username":"maru"},"change_message_id":"3b10d898015c2a3ca62fc305a1585ac40c47e65a","unresolved":false,"context_lines":[{"line_number":72,"context_line":"5. Test results in a complex system aren\u0027t binary, they are"},{"line_number":73,"context_line":"   statistical"},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"Because we have shared global state in OpenStack services,"},{"line_number":76,"context_line":"interactions between tests are possible. Some times these are test"},{"line_number":77,"context_line":"bugs, some times this is the consequence of a complex system that has"},{"line_number":78,"context_line":"global state."}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_186cfa35","line":75,"updated":"2015-02-17 23:35:44.000000000","message":"Why do we have shared global state at all?  Isn\u0027t this a huge anti-pattern wrt distributed systems, and one unlikely to be solved by any improvement in testing policy?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"65ffaddb898237bffafdeade6eed5307e4a1d7e2","unresolved":false,"context_lines":[{"line_number":73,"context_line":"   statistical"},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"Because we have shared global state in OpenStack services,"},{"line_number":76,"context_line":"interactions between tests are possible. Some times these are test"},{"line_number":77,"context_line":"bugs, some times this is the consequence of a complex system that has"},{"line_number":78,"context_line":"global state."},{"line_number":79,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"ba7be1f8_a6777f9e","line":76,"updated":"2015-02-28 16:21:37.000000000","message":"Here and the next line, \"Some times\" should be \"Sometimes\" :)","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"b8e75ec8e66bf305c3d683dfb68ba5574dfbed1d","unresolved":false,"context_lines":[{"line_number":80,"context_line":"This means failure or success is not binary. It\u0027s statistically"},{"line_number":81,"context_line":"significant. A failed test does not mean the code that was tested was"},{"line_number":82,"context_line":"at fault for the failure. It also means that the emergence of a new"},{"line_number":83,"context_line":"catastrophic break takes time to see. One test failure is not"},{"line_number":84,"context_line":"useful. Many test failures becomes useful."},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"False negatives are dangerous. They are dangerous in the same way that"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_2e8446b0","line":83,"updated":"2015-02-09 21:53:33.000000000","message":"Actually I see here a good point for collaboration between Tempest and Rally teams. \n\nAs far as Tempest is run much more often it collects more data. So we know what tests are failing often in gates. After that we can run load in gates of project and make this race 100% reproducible, make a fix with this check that will ensure that race is fixed.\n\nSo we don\u0027t need to merge \"possible\" fixes like now.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"720c2e407281546f73e31e56b136434710f2052a","unresolved":false,"context_lines":[{"line_number":80,"context_line":"This means failure or success is not binary. It\u0027s statistically"},{"line_number":81,"context_line":"significant. A failed test does not mean the code that was tested was"},{"line_number":82,"context_line":"at fault for the failure. It also means that the emergence of a new"},{"line_number":83,"context_line":"catastrophic break takes time to see. One test failure is not"},{"line_number":84,"context_line":"useful. Many test failures becomes useful."},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"False negatives are dangerous. They are dangerous in the same way that"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_84bc336e","line":83,"in_reply_to":"da86d52c_24f467ce","updated":"2015-02-09 22:45:03.000000000","message":"yep just fully separated job. Like now in Cinder we have rally and tempest jobs.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"8d1a491d1a61d9f0834cc0904e2b029bf1fe56ed","unresolved":false,"context_lines":[{"line_number":80,"context_line":"This means failure or success is not binary. It\u0027s statistically"},{"line_number":81,"context_line":"significant. A failed test does not mean the code that was tested was"},{"line_number":82,"context_line":"at fault for the failure. It also means that the emergence of a new"},{"line_number":83,"context_line":"catastrophic break takes time to see. One test failure is not"},{"line_number":84,"context_line":"useful. Many test failures becomes useful."},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"False negatives are dangerous. They are dangerous in the same way that"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_24f467ce","line":83,"in_reply_to":"da86d52c_2e8446b0","updated":"2015-02-09 22:30:18.000000000","message":"if they can be isolated, sure.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"798818f899d6a96640ae06a19e2b937da02cf602","unresolved":false,"context_lines":[{"line_number":80,"context_line":"This means failure or success is not binary. It\u0027s statistically"},{"line_number":81,"context_line":"significant. A failed test does not mean the code that was tested was"},{"line_number":82,"context_line":"at fault for the failure. It also means that the emergence of a new"},{"line_number":83,"context_line":"catastrophic break takes time to see. One test failure is not"},{"line_number":84,"context_line":"useful. Many test failures becomes useful."},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"False negatives are dangerous. They are dangerous in the same way that"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_c5f7e957","line":83,"in_reply_to":"da86d52c_84bc336e","updated":"2015-02-10 13:44:56.000000000","message":"No, you miss the point. If you can isolate the bug into a reproduce scenario, then that\u0027s a great place to build a repeat scenario. However, that\u0027s not always easy to do.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"b8e75ec8e66bf305c3d683dfb68ba5574dfbed1d","unresolved":false,"context_lines":[{"line_number":83,"context_line":"catastrophic break takes time to see. One test failure is not"},{"line_number":84,"context_line":"useful. Many test failures becomes useful."},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"False negatives are dangerous. They are dangerous in the same way that"},{"line_number":87,"context_line":"extra ERRORs in logs are dangerous, as they remove trust in the"},{"line_number":88,"context_line":"system, and make people ignore real errors."},{"line_number":89,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_2eab661f","line":86,"updated":"2015-02-09 21:53:33.000000000","message":"The one issue of Tempest that doesn\u0027t have Rally is next: \n\nIf we have test that faces race in tempest we have to remove it. \nIn Rally we can still run it, but make single test non voting (and keep collecting stats).","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"8d1a491d1a61d9f0834cc0904e2b029bf1fe56ed","unresolved":false,"context_lines":[{"line_number":83,"context_line":"catastrophic break takes time to see. One test failure is not"},{"line_number":84,"context_line":"useful. Many test failures becomes useful."},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"False negatives are dangerous. They are dangerous in the same way that"},{"line_number":87,"context_line":"extra ERRORs in logs are dangerous, as they remove trust in the"},{"line_number":88,"context_line":"system, and make people ignore real errors."},{"line_number":89,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_c4262b68","line":86,"in_reply_to":"da86d52c_2eab661f","updated":"2015-02-09 22:30:18.000000000","message":"I would agree that having tempest support XFAIL would be good, however I\u0027m not sure that really impacts the preamble here. This is about setting up what we know.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":83,"context_line":"catastrophic break takes time to see. One test failure is not"},{"line_number":84,"context_line":"useful. Many test failures becomes useful."},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"False negatives are dangerous. They are dangerous in the same way that"},{"line_number":87,"context_line":"extra ERRORs in logs are dangerous, as they remove trust in the"},{"line_number":88,"context_line":"system, and make people ignore real errors."},{"line_number":89,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_ea2c9504","line":86,"in_reply_to":"da86d52c_c4262b68","updated":"2015-02-11 07:39:54.000000000","message":"Is this patch about how Tempest differs from Rally? (and more specifically why Rally is better than tempest?)\n\nOn another note, I am not sure I follow your point on false negatives - mostly because I don\u0027t know if you\u0027re referring to test which fail because of bad test design, or test which succeed for the same reason. I agree however on your point on extra ERROR level log statements.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":8788,"name":"Miguel Angel Ajo","email":"mangelajo@redhat.com","username":"mangelajo"},"change_message_id":"550f1ae5b37635876addf03df3357a31e70b4c32","unresolved":false,"context_lines":[{"line_number":85,"context_line":""},{"line_number":86,"context_line":"False negatives are dangerous. They are dangerous in the same way that"},{"line_number":87,"context_line":"extra ERRORs in logs are dangerous, as they remove trust in the"},{"line_number":88,"context_line":"system, and make people ignore real errors."},{"line_number":89,"context_line":""},{"line_number":90,"context_line":""},{"line_number":91,"context_line":"Proposed Test Philosophy"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_d0ee9fec","line":88,"updated":"2015-02-11 10:54:02.000000000","message":"Yes, we should avoid ERRORS in favor of WARNINGS when the failure is not fatal and can be eventually recovered.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2035,"name":"Maru Newby","email":"marun@redhat.com","username":"maru"},"change_message_id":"3b10d898015c2a3ca62fc305a1585ac40c47e65a","unresolved":false,"context_lines":[{"line_number":95,"context_line":""},{"line_number":96,"context_line":"Tests should start at the lowest possible level and guarantee"},{"line_number":97,"context_line":"interfaces and behavior. This establishes a contract, which should"},{"line_number":98,"context_line":"carry to the next level."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"For instance, Nova has a database API. We should test the database API"},{"line_number":101,"context_line":"against a variety of backends. Any tests above the DB API (like the"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_d31fa559","line":98,"updated":"2015-02-17 23:35:44.000000000","message":"I think it bears mentioning that testing at a higher level should generally only be approached once lower-level tests have been written.  You\u0027ve captured this implicitly - \u0027contract, which should carry to the next level\u0027, but the \u0027why\u0027 of this statement might benefit from elaboration, e.g.\n\nHigher-level tests written in advance of lower-level testing being completed are likely to be more costly to write.  Failures are more likely since there won\u0027t be sufficient coverage of the lower-level details. The cycle time for getting feedback on those failures is going to be longer due to operating at a greater degree of abstraction, and it will often be more costly to debug those failures as a result.  Writing higher-level tests too soon also increases the possibility of covering too much lower-level functionality at an inappropriate level of abstraction.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6486,"name":"Brant Knudson","email":"blk@acm.org","username":"blk-u"},"change_message_id":"bda4231184a5c492d90f9f93ce50297b42675618","unresolved":false,"context_lines":[{"line_number":100,"context_line":"For instance, Nova has a database API. We should test the database API"},{"line_number":101,"context_line":"against a variety of backends. Any tests above the DB API (like the"},{"line_number":102,"context_line":"Object or REST API) should assume db API backends are now tested and"},{"line_number":103,"context_line":"only one database is used in higher level tests."},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"2. Test components in minimum linked sets"},{"line_number":106,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_e00b2429","line":103,"updated":"2015-02-09 19:19:25.000000000","message":"Are these functional tests or unit tests?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"2f3b5e1ceea4c4a588ba6f1b4307f6aa7ac4d479","unresolved":false,"context_lines":[{"line_number":100,"context_line":"For instance, Nova has a database API. We should test the database API"},{"line_number":101,"context_line":"against a variety of backends. Any tests above the DB API (like the"},{"line_number":102,"context_line":"Object or REST API) should assume db API backends are now tested and"},{"line_number":103,"context_line":"only one database is used in higher level tests."},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"2. Test components in minimum linked sets"},{"line_number":106,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_66911552","line":103,"updated":"2015-02-09 16:48:16.000000000","message":"Yes!","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":9555,"name":"Matthew Booth","email":"mbooth@redhat.com","username":"MatthewBooth"},"change_message_id":"1c0a2363f4052664d2ba79d3a6c90b7ea20cd9be","unresolved":false,"context_lines":[{"line_number":100,"context_line":"For instance, Nova has a database API. We should test the database API"},{"line_number":101,"context_line":"against a variety of backends. Any tests above the DB API (like the"},{"line_number":102,"context_line":"Object or REST API) should assume db API backends are now tested and"},{"line_number":103,"context_line":"only one database is used in higher level tests."},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"2. Test components in minimum linked sets"},{"line_number":106,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_d615eee5","line":103,"in_reply_to":"da86d52c_64b49f35","updated":"2015-02-10 09:25:48.000000000","message":"I\u0027m also confused about the intention of this paragraph. It says to me that we\u0027re only interested in unit tests, not functional tests. I\u0027m sure that\u0027s not what\u0027s intended. Functional tests catch many interesting bugs, for example when the unit test author has made invalid assumptions about how a unit is actually called.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2035,"name":"Maru Newby","email":"marun@redhat.com","username":"maru"},"change_message_id":"3b10d898015c2a3ca62fc305a1585ac40c47e65a","unresolved":false,"context_lines":[{"line_number":100,"context_line":"For instance, Nova has a database API. We should test the database API"},{"line_number":101,"context_line":"against a variety of backends. Any tests above the DB API (like the"},{"line_number":102,"context_line":"Object or REST API) should assume db API backends are now tested and"},{"line_number":103,"context_line":"only one database is used in higher level tests."},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"2. Test components in minimum linked sets"},{"line_number":106,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_d3408509","line":103,"in_reply_to":"da86d52c_aaf9bd4f","updated":"2015-02-17 23:35:44.000000000","message":"The question \u0027unit or functional\u0027 would seem to miss the point of the idea of a \u0027pyramid\u0027.  Consider adding a reference to Martin Fowler\u0027s original treatise:\n\nhttp://martinfowler.com/bliki/TestPyramid.html\n\nGiven the comments here, I don\u0027t think this section in its current form provides enough justification for the need to test things at as low a level as possible and have each successive layer build upon the foundation of trust achieved at the level beneath.  Given how important I believe this philosophy to be to an effective testing regime, more detail is suggested.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":100,"context_line":"For instance, Nova has a database API. We should test the database API"},{"line_number":101,"context_line":"against a variety of backends. Any tests above the DB API (like the"},{"line_number":102,"context_line":"Object or REST API) should assume db API backends are now tested and"},{"line_number":103,"context_line":"only one database is used in higher level tests."},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"2. Test components in minimum linked sets"},{"line_number":106,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_aaf9bd4f","line":103,"in_reply_to":"da86d52c_d615eee5","updated":"2015-02-11 07:39:54.000000000","message":"If the components have a well defined interface, you can do functional testing as well. The difference between unit and functional testing in my view, is that the former should pursue code coverage whereas the latter should validate the functionality of the component you\u0027re testing.\n\nI think in the openstack world we tend to develop unit tests as if they were actually functional tests, and that\u0027s where the confusion perhaps starts from. With reference to the DB API example, unit tests (white box) should aim at 100% code coverage, where functional tests (black box wrt the component under test) should verify the system behaves correctly. To this aim several backends might be used; fault injection to verify behaviour in failure conditions might be considered as well.\n\nAnyway, I\u0027m no testing expert so the above paragraph might just be a bunch of bovine excrements.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"2b5979341aeb43151a6f5c2ef1dd2fcab0fc63d9","unresolved":false,"context_lines":[{"line_number":100,"context_line":"For instance, Nova has a database API. We should test the database API"},{"line_number":101,"context_line":"against a variety of backends. Any tests above the DB API (like the"},{"line_number":102,"context_line":"Object or REST API) should assume db API backends are now tested and"},{"line_number":103,"context_line":"only one database is used in higher level tests."},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"2. Test components in minimum linked sets"},{"line_number":106,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_64b49f35","line":103,"in_reply_to":"da86d52c_e00b2429","updated":"2015-02-09 22:24:56.000000000","message":"All of the above. It\u0027s the general theory of set a contract, ensure all implementations of that contract are the same, don\u0027t care which one you use at the next level.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"5940048e53e33905bb1aa62e671da881d123e5b0","unresolved":false,"context_lines":[{"line_number":101,"context_line":"against a variety of backends. Any tests above the DB API (like the"},{"line_number":102,"context_line":"Object or REST API) should assume db API backends are now tested and"},{"line_number":103,"context_line":"only one database is used in higher level tests."},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"2. Test components in minimum linked sets"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"Projects that can run on their own, should be tested on their own."}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_4894d78a","line":104,"updated":"2015-02-10 22:29:51.000000000","message":"Let me try and say what I think this is saying:\n\nFor a given layer (whether internal, external, whatever), we:\n - 1) have its internal unit tests that may use mocks etc\n - 2) have functional tests that exercise the layer in anger, probing for race conditions, concurrency support, scale, performance, correctness against all backends that that layer supports. E.G. for messaging we test all backends. For db we test all backends. And those tests need to be really serious and rigorous.\n - 3) have a fast-and-correct backend that we then use *when using the layer in the functional tests of other layers that depend on it*.\n\nWe used this exact strategy in bzr to great effect - we had very very few bugs that snuck up layers in the way they do today in OpenStack, over 10 distinct scenarios executed in CI runs, and it was very easy to reason about where failures were occuring (because you can look for the lowest of 1/2/3 to see whats close to the problem).","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2035,"name":"Maru Newby","email":"marun@redhat.com","username":"maru"},"change_message_id":"3b10d898015c2a3ca62fc305a1585ac40c47e65a","unresolved":false,"context_lines":[{"line_number":101,"context_line":"against a variety of backends. Any tests above the DB API (like the"},{"line_number":102,"context_line":"Object or REST API) should assume db API backends are now tested and"},{"line_number":103,"context_line":"only one database is used in higher level tests."},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"2. Test components in minimum linked sets"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"Projects that can run on their own, should be tested on their own."}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_73eb59dc","line":104,"in_reply_to":"da86d52c_4894d78a","updated":"2015-02-17 23:35:44.000000000","message":"+1","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2035,"name":"Maru Newby","email":"marun@redhat.com","username":"maru"},"change_message_id":"3b10d898015c2a3ca62fc305a1585ac40c47e65a","unresolved":false,"context_lines":[{"line_number":102,"context_line":"Object or REST API) should assume db API backends are now tested and"},{"line_number":103,"context_line":"only one database is used in higher level tests."},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"2. Test components in minimum linked sets"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"Projects that can run on their own, should be tested on their own."},{"line_number":108,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_5bce5e5b","line":105,"updated":"2015-02-17 23:35:44.000000000","message":"Consider adding mention of using fakes that conform minimally to the contract between an application and the real component.  It shouldn\u0027t be necessary to require nova to validate that neutron can provide connectivity between endpoints, for instance, and a fake would be a good way of validating neutron\u0027s behavior in isolation.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":5263,"name":"Jeremy Stanley","display_name":"fungi","email":"fungi@yuggoth.org","username":"fungi","status":"missing, presumed fed"},"change_message_id":"ab35ce320b48687641c1c7498389f842b5467fb3","unresolved":false,"context_lines":[{"line_number":106,"context_line":""},{"line_number":107,"context_line":"Projects that can run on their own, should be tested on their own."},{"line_number":108,"context_line":""},{"line_number":109,"context_line":"Projects that are tightly coupled (such as the compute kernel) can be"},{"line_number":110,"context_line":"co-gated on each other."},{"line_number":111,"context_line":""},{"line_number":112,"context_line":"Projects that rely on the compute kernel should assume it is"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_d1535c6a","line":109,"updated":"2015-02-10 01:20:12.000000000","message":"The term \"compute kernel\" might need expanding here at its first use, as otherwise the reader doesn\u0027t get an explanation until way down at item #3 in the proposed changes section.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"65ffaddb898237bffafdeade6eed5307e4a1d7e2","unresolved":false,"context_lines":[{"line_number":106,"context_line":""},{"line_number":107,"context_line":"Projects that can run on their own, should be tested on their own."},{"line_number":108,"context_line":""},{"line_number":109,"context_line":"Projects that are tightly coupled (such as the compute kernel) can be"},{"line_number":110,"context_line":"co-gated on each other."},{"line_number":111,"context_line":""},{"line_number":112,"context_line":"Projects that rely on the compute kernel should assume it is"}],"source_content_type":"text/x-rst","patch_set":3,"id":"ba7be1f8_86827b50","line":109,"in_reply_to":"da86d52c_d1535c6a","updated":"2015-02-28 16:21:37.000000000","message":"@fungi: ++","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"180572a10bc22adb3d9a60a0614fe5b36319ae3b","unresolved":false,"context_lines":[{"line_number":112,"context_line":"Projects that rely on the compute kernel should assume it is"},{"line_number":113,"context_line":"functioning correctly, and test above it."},{"line_number":114,"context_line":""},{"line_number":115,"context_line":"3. Do not intentionally perform secondary testing"},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"OpenStack client libraries are not currently directly tested, they are"},{"line_number":118,"context_line":"only tested by secondary interaction (a nova boot triggers neutron"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_6b91c245","line":115,"updated":"2015-02-16 20:05:22.000000000","message":"Do you mean this literally as written, or do you mean that we shouldn\u0027t be relying on secondary testing as the sole way to test that code? The former may imply a lot more mocking than we actually want to do.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":13734,"name":"Nell Jerram","email":"nell@tigera.io","username":"neiljerram"},"change_message_id":"49355fb7bd0db0546e05a50f949e99c92d028591","unresolved":false,"context_lines":[{"line_number":112,"context_line":"Projects that rely on the compute kernel should assume it is"},{"line_number":113,"context_line":"functioning correctly, and test above it."},{"line_number":114,"context_line":""},{"line_number":115,"context_line":"3. Do not intentionally perform secondary testing"},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"OpenStack client libraries are not currently directly tested, they are"},{"line_number":118,"context_line":"only tested by secondary interaction (a nova boot triggers neutron"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a80dd14_d0eadb0d","line":115,"updated":"2015-03-20 14:40:27.000000000","message":"What is \"secondary\" testing?  I\u0027ve worked in software for 20 years but am not familiar with this term, so please define it.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2035,"name":"Maru Newby","email":"marun@redhat.com","username":"maru"},"change_message_id":"3b10d898015c2a3ca62fc305a1585ac40c47e65a","unresolved":false,"context_lines":[{"line_number":112,"context_line":"Projects that rely on the compute kernel should assume it is"},{"line_number":113,"context_line":"functioning correctly, and test above it."},{"line_number":114,"context_line":""},{"line_number":115,"context_line":"3. Do not intentionally perform secondary testing"},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"OpenStack client libraries are not currently directly tested, they are"},{"line_number":118,"context_line":"only tested by secondary interaction (a nova boot triggers neutron"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_9b19b6d3","line":115,"in_reply_to":"da86d52c_6b91c245","updated":"2015-02-17 23:35:44.000000000","message":"+1 to making it clear that it is not acceptable to rely on secondary testing as the only way to have something tested.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"b8e75ec8e66bf305c3d683dfb68ba5574dfbed1d","unresolved":false,"context_lines":[{"line_number":114,"context_line":""},{"line_number":115,"context_line":"3. Do not intentionally perform secondary testing"},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"OpenStack client libraries are not currently directly tested, they are"},{"line_number":118,"context_line":"only tested by secondary interaction (a nova boot triggers neutron"},{"line_number":119,"context_line":"client calls, for instance)."},{"line_number":120,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_ae4d969b","line":117,"updated":"2015-02-09 21:53:33.000000000","message":"Actually Rally is using client libraries. So we can say that they ARE tested. \n\nSo why not using Rally for testing them?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":5263,"name":"Jeremy Stanley","display_name":"fungi","email":"fungi@yuggoth.org","username":"fungi","status":"missing, presumed fed"},"change_message_id":"ab35ce320b48687641c1c7498389f842b5467fb3","unresolved":false,"context_lines":[{"line_number":114,"context_line":""},{"line_number":115,"context_line":"3. Do not intentionally perform secondary testing"},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"OpenStack client libraries are not currently directly tested, they are"},{"line_number":118,"context_line":"only tested by secondary interaction (a nova boot triggers neutron"},{"line_number":119,"context_line":"client calls, for instance)."},{"line_number":120,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_a801053b","line":117,"in_reply_to":"da86d52c_6471ff43","updated":"2015-02-10 01:20:12.000000000","message":"I think the point is that Rally (like Tempest) is not currently _testing_ the clients sufficiently, it\u0027s merely _using_ the clients to run other tests. As Sean states it isn\u0027t specifically and intentionally testing all code paths through the clients: all command-line and configurable options, all class methods and functions, et cetera. This is what he means by \"secondary testing,\" e.g. testing as a scenario byproduct of other more intentional tests.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":114,"context_line":""},{"line_number":115,"context_line":"3. Do not intentionally perform secondary testing"},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"OpenStack client libraries are not currently directly tested, they are"},{"line_number":118,"context_line":"only tested by secondary interaction (a nova boot triggers neutron"},{"line_number":119,"context_line":"client calls, for instance)."},{"line_number":120,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_ca7b51bc","line":117,"in_reply_to":"da86d52c_a801053b","updated":"2015-02-11 07:39:54.000000000","message":"Indeed. For instance all openstack projects heavily use oslo libraries, but I don\u0027t think we\u0027re suggesting that, for instance, scenario tests are considered a valid tests for oslo libraries.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"2b5979341aeb43151a6f5c2ef1dd2fcab0fc63d9","unresolved":false,"context_lines":[{"line_number":114,"context_line":""},{"line_number":115,"context_line":"3. Do not intentionally perform secondary testing"},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"OpenStack client libraries are not currently directly tested, they are"},{"line_number":118,"context_line":"only tested by secondary interaction (a nova boot triggers neutron"},{"line_number":119,"context_line":"client calls, for instance)."},{"line_number":120,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_e45a4f29","line":117,"in_reply_to":"da86d52c_ae4d969b","updated":"2015-02-09 22:24:56.000000000","message":"They are tested via secondary testing, which isn\u0027t the same thing. For instance, rally does not call every possible library call, that\u0027s not what it does.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"1c3781a19773b4874c14d3bfdcecda0db021b1b9","unresolved":false,"context_lines":[{"line_number":114,"context_line":""},{"line_number":115,"context_line":"3. Do not intentionally perform secondary testing"},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"OpenStack client libraries are not currently directly tested, they are"},{"line_number":118,"context_line":"only tested by secondary interaction (a nova boot triggers neutron"},{"line_number":119,"context_line":"client calls, for instance)."},{"line_number":120,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_6471ff43","line":117,"in_reply_to":"da86d52c_e45a4f29","updated":"2015-02-09 22:39:11.000000000","message":"Actually it\u0027s not \"tested\" via secondary testing. It\u0027s called directly. More over we you can pass any argument that python clients accept. \n\nRally use data driven approach. For example in scenario NovaServers.boot_and_delete in args section you can pass any arguments that accept nova python client create server method. \n\nhttps://github.com/stackforge/rally/blob/master/samples/tasks/scenarios/nova/boot-and-delete.json#L4\n\nIt works regarding to next 2 piece of code:\n\n\n  https://github.com/stackforge/rally/blob/master/rally/benchmark/scenarios/nova/servers.py#L63\n\nand\n  https://github.com/stackforge/rally/blob/master/rally/benchmark/scenarios/nova/utils.py#L118-L119","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":13734,"name":"Nell Jerram","email":"nell@tigera.io","username":"neiljerram"},"change_message_id":"49355fb7bd0db0546e05a50f949e99c92d028591","unresolved":false,"context_lines":[{"line_number":124,"context_line":"4. Treat substantial false negatives as critical bugs"},{"line_number":125,"context_line":""},{"line_number":126,"context_line":"False negatives make people ignore the system, and rage"},{"line_number":127,"context_line":"quit. Addressing false negatives should be a high priority issue. If a"},{"line_number":128,"context_line":"test for a feature can\u0027t be made reliable we shouldn\u0027t keep self"},{"line_number":129,"context_line":"inflicting damage on ourselves."},{"line_number":130,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a80dd14_b04d673d","line":127,"updated":"2015-03-20 14:40:27.000000000","message":"\"rage quit\"?  Do you mean \"rage and quit\"?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"481ddeb4a0088af22f91f7af6f6cf2466a9ecca9","unresolved":false,"context_lines":[{"line_number":125,"context_line":""},{"line_number":126,"context_line":"False negatives make people ignore the system, and rage"},{"line_number":127,"context_line":"quit. Addressing false negatives should be a high priority issue. If a"},{"line_number":128,"context_line":"test for a feature can\u0027t be made reliable we shouldn\u0027t keep self"},{"line_number":129,"context_line":"inflicting damage on ourselves."},{"line_number":130,"context_line":""},{"line_number":131,"context_line":"This will be a cultural change for many teams, and will require buy in"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_f20203b3","line":128,"updated":"2015-02-10 06:30:58.000000000","message":"+1 cut off the limb if its rotting","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":9555,"name":"Matthew Booth","email":"mbooth@redhat.com","username":"MatthewBooth"},"change_message_id":"1c0a2363f4052664d2ba79d3a6c90b7ea20cd9be","unresolved":false,"context_lines":[{"line_number":126,"context_line":"False negatives make people ignore the system, and rage"},{"line_number":127,"context_line":"quit. Addressing false negatives should be a high priority issue. If a"},{"line_number":128,"context_line":"test for a feature can\u0027t be made reliable we shouldn\u0027t keep self"},{"line_number":129,"context_line":"inflicting damage on ourselves."},{"line_number":130,"context_line":""},{"line_number":131,"context_line":"This will be a cultural change for many teams, and will require buy in"},{"line_number":132,"context_line":"and drum beating by PTLs."}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_369aca63","line":129,"updated":"2015-02-10 09:25:48.000000000","message":"+1","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"2f3b5e1ceea4c4a588ba6f1b4307f6aa7ac4d479","unresolved":false,"context_lines":[{"line_number":129,"context_line":"inflicting damage on ourselves."},{"line_number":130,"context_line":""},{"line_number":131,"context_line":"This will be a cultural change for many teams, and will require buy in"},{"line_number":132,"context_line":"and drum beating by PTLs."},{"line_number":133,"context_line":""},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"Proposed changes"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_86752934","line":132,"updated":"2015-02-09 16:48:16.000000000","message":"I\u0027d appreciate more context here.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2035,"name":"Maru Newby","email":"marun@redhat.com","username":"maru"},"change_message_id":"3b10d898015c2a3ca62fc305a1585ac40c47e65a","unresolved":false,"context_lines":[{"line_number":129,"context_line":"inflicting damage on ourselves."},{"line_number":130,"context_line":""},{"line_number":131,"context_line":"This will be a cultural change for many teams, and will require buy in"},{"line_number":132,"context_line":"and drum beating by PTLs."},{"line_number":133,"context_line":""},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"Proposed changes"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_bb63da18","line":132,"in_reply_to":"da86d52c_6d97df90","updated":"2015-02-17 23:35:44.000000000","message":"The issue of skipping failing tests due to a lack of action from affected projects is indicative to me of a culture that has not provided sufficient incentive for projects to care about the quality of OpenStack as a whole.  Relegating the bulk of testing effort to a post-dev and supra-project effort was a major contributing factor, and ongoing initiatives to push the responsibility for testing back on the projects will go a long way to solving this problem of incentives such that core reviewers can mandate a much higher standard of testing pre-merge and even be able to push back on shipping merged features that prove themselves unstable.\n\nAll this to say that I think it falls on the entire core review team to hold contributors accountable in a way that supra-project teams like infra and qa have historically had a difficult time doing.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":129,"context_line":"inflicting damage on ourselves."},{"line_number":130,"context_line":""},{"line_number":131,"context_line":"This will be a cultural change for many teams, and will require buy in"},{"line_number":132,"context_line":"and drum beating by PTLs."},{"line_number":133,"context_line":""},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"Proposed changes"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_6d97df90","line":132,"in_reply_to":"da86d52c_858411b2","updated":"2015-02-11 07:39:54.000000000","message":"I would like to add that I don\u0027t like the idea of putting as a part of the test philosophy that we need buy-in from PTLs.\n\nI think this kind of issues should be addressed more from a release management perspective. Openstack has a release phylosophy which does little to ensure critical bugs are fixed. We have fairly strict deadlines and fairly tight release cycles. Critical bugs tend to be ignored in the initial milestones of a release cycles, because the majority of the teams are working on addressing larger feature works in order to ensure code is ready before the feature proposal freeze. Then bug fixing activity starts to increase and peaks in the RC period. Once the release is out, the planning phase starts again and bug fix activity goes down again.\n\nAn exception are severe gate breaking bugs, where often team members step up to diagnose and fix bugs. But it is not uncommon for the QA team to just skip failing tests until someone from the affected project teams steps up.\n\nI\u0027m pretty sure you\u0027d like me to provide some data to corroborate this view, and if you think it might be useful I\u0027ll try and do that. That\u0027s of course way out of the scope of this patch. My point here is that rather than hope that the culture might be improved by exercising pressure on PTLs and their teams, we might be better off by tweaking the system so it adapts to the people and entices or forces them to behave better. \nI am not sure about what we can do, mostly because I have 0 experience in release management. Flexible release dates is an option; enforcing a maximum bug/blueprint ratio per project is another option... but I\u0027ll leave the judgment to people who have more experience than me.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"798818f899d6a96640ae06a19e2b937da02cf602","unresolved":false,"context_lines":[{"line_number":129,"context_line":"inflicting damage on ourselves."},{"line_number":130,"context_line":""},{"line_number":131,"context_line":"This will be a cultural change for many teams, and will require buy in"},{"line_number":132,"context_line":"and drum beating by PTLs."},{"line_number":133,"context_line":""},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"Proposed changes"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_858411b2","line":132,"in_reply_to":"da86d52c_858b5a1d","updated":"2015-02-10 13:44:56.000000000","message":"I\u0027ll soften this in next revision, I think I swung too far based on ttx\u0027s feedback.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"7df6fa8fb3013a54dfba76baf9e722fbba1a01e1","unresolved":false,"context_lines":[{"line_number":129,"context_line":"inflicting damage on ourselves."},{"line_number":130,"context_line":""},{"line_number":131,"context_line":"This will be a cultural change for many teams, and will require buy in"},{"line_number":132,"context_line":"and drum beating by PTLs."},{"line_number":133,"context_line":""},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"Proposed changes"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_858b5a1d","line":132,"in_reply_to":"da86d52c_86752934","updated":"2015-02-09 18:38:28.000000000","message":"critical bugs have often gone multiple releases before being fixed. Our turn around time on critical bugs is... lacking (see ttx\u0027s comments on PS1).","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":5263,"name":"Jeremy Stanley","display_name":"fungi","email":"fungi@yuggoth.org","username":"fungi","status":"missing, presumed fed"},"change_message_id":"ab35ce320b48687641c1c7498389f842b5467fb3","unresolved":false,"context_lines":[{"line_number":139,"context_line":"---------------"},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"Adjust the Tempest testing and co-gating of projects in the following"},{"line_number":142,"context_line":"way:"},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"1. Keystone"},{"line_number":145,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_a87665d6","line":142,"updated":"2015-02-10 01:20:12.000000000","message":"This looks like wayS (plural).","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6486,"name":"Brant Knudson","email":"blk@acm.org","username":"blk-u"},"change_message_id":"bda4231184a5c492d90f9f93ce50297b42675618","unresolved":false,"context_lines":[{"line_number":144,"context_line":"1. Keystone"},{"line_number":145,"context_line":""},{"line_number":146,"context_line":"   - runs only keystone"},{"line_number":147,"context_line":"   - runs only indentity tempest tests"},{"line_number":148,"context_line":""},{"line_number":149,"context_line":"2. Swift"},{"line_number":150,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_2b1d1d67","line":147,"updated":"2015-02-09 19:19:25.000000000","message":"change \"indentity\" to \"identity\".\n\nAre there going to be identity tempest tests if we have functional tests in keystone?\n\nKeystone supports different backends, so we should consider what backends are used (sql vs ldap vs federation).\n\nAlso, we should consider whether auth_token middleware is included in keystone testing. For example, we could test auth_token middleware not by using a real server like glance but instead have a fake server that\u0027s running middleware.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"2b5979341aeb43151a6f5c2ef1dd2fcab0fc63d9","unresolved":false,"context_lines":[{"line_number":144,"context_line":"1. Keystone"},{"line_number":145,"context_line":""},{"line_number":146,"context_line":"   - runs only keystone"},{"line_number":147,"context_line":"   - runs only indentity tempest tests"},{"line_number":148,"context_line":""},{"line_number":149,"context_line":"2. Swift"},{"line_number":150,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_2491c79a","line":147,"in_reply_to":"da86d52c_2b1d1d67","updated":"2015-02-09 22:24:56.000000000","message":"I think the multiple backends question is a good one, but I assume like db drivers, that would be done at a keystone functional level, and not at the tempest level.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"65ffaddb898237bffafdeade6eed5307e4a1d7e2","unresolved":false,"context_lines":[{"line_number":151,"context_line":"   - runs only swift + keystone"},{"line_number":152,"context_line":"   - runs only swift tempest tests"},{"line_number":153,"context_line":""},{"line_number":154,"context_line":"3. Compute Kernel (Glance, Nova, Cinder, Neutron (base services))"},{"line_number":155,"context_line":""},{"line_number":156,"context_line":"   - assumes Keystone works, uses keystone master"},{"line_number":157,"context_line":"   - use Filesystem backend for Glance"}],"source_content_type":"text/x-rst","patch_set":3,"id":"ba7be1f8_66b90715","line":154,"updated":"2015-02-28 16:21:37.000000000","message":"OK, so here, I actually think that there are a number of functional and integration tests that should come *before* here in the pyramid:\n\na) Testing of Glance and Nova interaction, in isolation.\n\nFor example, we can run a set of functional tests that stress the interaction between Nova and Glance -- running glance image-create; nova image-list, for example. If this simple functional tests does not pass, there\u0027s no reason to try running an integration test that uses Nova, Glance, Cinder *and* Neutron.\n\nThese functional tests should be set up as test scenarios on all the Glance API versions that we, in Nova, say we support: v1 and v2. If either fails, what is the point of continuing to test further from an integration perspective.\n\nb) Testing of Nova and Neutron interaction, in isolation.\n\nSimilarly, we can run simple functional tests that do things like neutron net-create; neutron subnet-create; nova network-list; etc. Again, if these simple tests don\u0027t pass, then there\u0027s no need to proceed further with more complex functional integration tests that, say, stress the entire boot instance sequence.\n\nc) Testing of Nova and Cinder interaction, in isolation.\n\nSame thing here. Do a simple cinder volume-create; nova volume-list operation to verify baseline interaction between Nova and Cinder. If this fails, then fail-fast and don\u0027t bother running the full integration test suites.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":9555,"name":"Matthew Booth","email":"mbooth@redhat.com","username":"MatthewBooth"},"change_message_id":"1c0a2363f4052664d2ba79d3a6c90b7ea20cd9be","unresolved":false,"context_lines":[{"line_number":154,"context_line":"3. Compute Kernel (Glance, Nova, Cinder, Neutron (base services))"},{"line_number":155,"context_line":""},{"line_number":156,"context_line":"   - assumes Keystone works, uses keystone master"},{"line_number":157,"context_line":"   - use Filesystem backend for Glance"},{"line_number":158,"context_line":"   - uses only mysql backend"},{"line_number":159,"context_line":"   - runs only keystone + glance + nova + cinder + neutron (basic"},{"line_number":160,"context_line":"     services only)"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_36ff2a69","line":157,"updated":"2015-02-10 09:25:48.000000000","message":"Which nova, cinder and neutron drivers should be used?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"798818f899d6a96640ae06a19e2b937da02cf602","unresolved":false,"context_lines":[{"line_number":154,"context_line":"3. Compute Kernel (Glance, Nova, Cinder, Neutron (base services))"},{"line_number":155,"context_line":""},{"line_number":156,"context_line":"   - assumes Keystone works, uses keystone master"},{"line_number":157,"context_line":"   - use Filesystem backend for Glance"},{"line_number":158,"context_line":"   - uses only mysql backend"},{"line_number":159,"context_line":"   - runs only keystone + glance + nova + cinder + neutron (basic"},{"line_number":160,"context_line":"     services only)"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_65a39d9e","line":157,"in_reply_to":"da86d52c_36ff2a69","updated":"2015-02-10 13:44:56.000000000","message":"The same defaults we are using today. Because that was not a change from the current situation that wasn\u0027t explicitly mentioned.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"5940048e53e33905bb1aa62e671da881d123e5b0","unresolved":false,"context_lines":[{"line_number":163,"context_line":""},{"line_number":164,"context_line":"4. Ironic"},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"   - runs only ironic + keystone"},{"line_number":167,"context_line":"   - runs only ironic tests"},{"line_number":168,"context_line":""},{"line_number":169,"context_line":"5. All additional OpenStack projects"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_08449f19","line":166,"updated":"2015-02-10 22:29:51.000000000","message":"Ironic has a functional dependency on Neutron, shouldn\u0027t it have it there? (It calls out to Neutron to configure PXE booting of ports).","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"59c4d9587d6adc54a914929f03683efe54d97523","unresolved":false,"context_lines":[{"line_number":163,"context_line":""},{"line_number":164,"context_line":"4. Ironic"},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"   - runs only ironic + keystone"},{"line_number":167,"context_line":"   - runs only ironic tests"},{"line_number":168,"context_line":""},{"line_number":169,"context_line":"5. All additional OpenStack projects"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_48cf7756","line":166,"in_reply_to":"da86d52c_08449f19","updated":"2015-02-10 22:36:15.000000000","message":"No. That should be a well defined behavior in Neutron, tested by Neutron. It\u0027s testing shouldn\u0027t be in the Ironic scenario. If it doesn\u0027t work reliably, that needs to be fixed and tested in Neutron.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"e9270d6df18512d1c139ebbba4a98b4d315fdf85","unresolved":false,"context_lines":[{"line_number":163,"context_line":""},{"line_number":164,"context_line":"4. Ironic"},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"   - runs only ironic + keystone"},{"line_number":167,"context_line":"   - runs only ironic tests"},{"line_number":168,"context_line":""},{"line_number":169,"context_line":"5. All additional OpenStack projects"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_62cff756","line":166,"in_reply_to":"da86d52c_48cf7756","updated":"2015-02-11 00:09:55.000000000","message":"I was not suggesting that it test Neutron. Why is keystone there? We aren\u0027t testing keystone either....","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"4da2d6dd719972f86527e1cfd7cc55538447e3ae","unresolved":false,"context_lines":[{"line_number":163,"context_line":""},{"line_number":164,"context_line":"4. Ironic"},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"   - runs only ironic + keystone"},{"line_number":167,"context_line":"   - runs only ironic tests"},{"line_number":168,"context_line":""},{"line_number":169,"context_line":"5. All additional OpenStack projects"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_a96622eb","line":166,"in_reply_to":"da86d52c_62cff756","updated":"2015-02-11 12:01:22.000000000","message":"oh... my bad. Sure, neutron should be in that list.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":10343,"name":"Jim Rollenhagen","email":"jim@jimrollenhagen.com","username":"jimrollenhagen"},"change_message_id":"1c9a468c190090a92eaaff8ab1b541e459574bb0","unresolved":false,"context_lines":[{"line_number":163,"context_line":""},{"line_number":164,"context_line":"4. Ironic"},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"   - runs only ironic + keystone"},{"line_number":167,"context_line":"   - runs only ironic tests"},{"line_number":168,"context_line":""},{"line_number":169,"context_line":"5. All additional OpenStack projects"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_53c1ec7e","line":166,"in_reply_to":"da86d52c_a96622eb","updated":"2015-02-17 21:10:14.000000000","message":"FWIW, we could use a static DHCP/PXE configuration and skip neutron.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"65ffaddb898237bffafdeade6eed5307e4a1d7e2","unresolved":false,"context_lines":[{"line_number":169,"context_line":"5. All additional OpenStack projects"},{"line_number":170,"context_line":""},{"line_number":171,"context_line":"   - runs minimum viable OpenStack stack below them (possibly"},{"line_number":172,"context_line":"     service + keystone, or service + compute kernel)"},{"line_number":173,"context_line":"   - assume services outside your project work correctly for the"},{"line_number":174,"context_line":"     purposes of the tests"},{"line_number":175,"context_line":"   - only runs tests specific to their project"}],"source_content_type":"text/x-rst","patch_set":3,"id":"ba7be1f8_a63a1f96","line":172,"updated":"2015-02-28 16:21:37.000000000","message":"Tests that functionally test Keystone integration with only the service endpoint itself should be done *before* this step in the testing pyramid, no?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"5940048e53e33905bb1aa62e671da881d123e5b0","unresolved":false,"context_lines":[{"line_number":177,"context_line":"6. Allow for possible voluntary project co-gating"},{"line_number":178,"context_line":""},{"line_number":179,"context_line":"   - a set of projects can decide they must be coupled together"},{"line_number":180,"context_line":"   - you can only be part of one coupling (to prevent transient"},{"line_number":181,"context_line":"     coupling)"},{"line_number":182,"context_line":"   - Note: any additional future coupling between projects should"},{"line_number":183,"context_line":"     create an update to this specification."}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_884faf33","line":180,"updated":"2015-02-10 22:29:51.000000000","message":"s/transient/transitive/ ?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"59c4d9587d6adc54a914929f03683efe54d97523","unresolved":false,"context_lines":[{"line_number":177,"context_line":"6. Allow for possible voluntary project co-gating"},{"line_number":178,"context_line":""},{"line_number":179,"context_line":"   - a set of projects can decide they must be coupled together"},{"line_number":180,"context_line":"   - you can only be part of one coupling (to prevent transient"},{"line_number":181,"context_line":"     coupling)"},{"line_number":182,"context_line":"   - Note: any additional future coupling between projects should"},{"line_number":183,"context_line":"     create an update to this specification."}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_a805537b","line":180,"in_reply_to":"da86d52c_884faf33","updated":"2015-02-10 22:36:15.000000000","message":"yep","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":10980,"name":"Doug Wiegley","email":"dougwig@parkside.io","username":"dougw"},"change_message_id":"b2c8e88eec9161a7dd513fd0a61e4006297f3dcb","unresolved":false,"context_lines":[{"line_number":180,"context_line":"   - you can only be part of one coupling (to prevent transient"},{"line_number":181,"context_line":"     coupling)"},{"line_number":182,"context_line":"   - Note: any additional future coupling between projects should"},{"line_number":183,"context_line":"     create an update to this specification."},{"line_number":184,"context_line":""},{"line_number":185,"context_line":"A bug in a project that is surfaced by another project (i.e. a Nova"},{"line_number":186,"context_line":"bug that breaks Heat) should be considered a high priority bug by both"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_45975d90","line":183,"updated":"2015-02-11 00:28:45.000000000","message":"If you\u0027re documenting all coupling here, currently neutron and the new services repos co-gate, because the internal interfaces being used in neutron have no other tests to prevent that breakage. Yes, it\u0027s madness, yes, those libraries calls need to be librarified and their interfaces regressed, and yes we are headed that way. But it\u0027s not going to be quick.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"65ffaddb898237bffafdeade6eed5307e4a1d7e2","unresolved":false,"context_lines":[{"line_number":185,"context_line":"A bug in a project that is surfaced by another project (i.e. a Nova"},{"line_number":186,"context_line":"bug that breaks Heat) should be considered a high priority bug by both"},{"line_number":187,"context_line":"teams. The fix and associated test case should only be done in the"},{"line_number":188,"context_line":"consumed project (in this case Nova)."},{"line_number":189,"context_line":""},{"line_number":190,"context_line":"Example: Magnum project wants full stack testing. They fall into"},{"line_number":191,"context_line":"rule 5. Their setup would be a compute kernel + heat, and assume all"}],"source_content_type":"text/x-rst","patch_set":3,"id":"ba7be1f8_064beb2b","line":188,"updated":"2015-02-28 16:21:37.000000000","message":"++","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":9555,"name":"Matthew Booth","email":"mbooth@redhat.com","username":"MatthewBooth"},"change_message_id":"1c0a2363f4052664d2ba79d3a6c90b7ea20cd9be","unresolved":false,"context_lines":[{"line_number":191,"context_line":"rule 5. Their setup would be a compute kernel + heat, and assume all"},{"line_number":192,"context_line":"those components are working correctly. If they find a behavior in one"},{"line_number":193,"context_line":"of those components has an ill defined contract, they should submit"},{"line_number":194,"context_line":"upstream tests to the relevant project to nail down that behavior."},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"Prereqs: None"},{"line_number":197,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_f6bcb26e","line":194,"updated":"2015-02-10 09:25:48.000000000","message":"Do we need to run the tip of all projects? i.e. when testing Nova, do we need to run the tip of Cinder? For example, could we instead check out release tags which would isolate projects from each other during periods of high churn/instability. These would be development-focussed release tags, maybe made weekly. In fact, Nova could even maintain its own list of commits it wants to use from other projects, and bump it regularly. It would then be free to bump it down again if there was a temporary problem in another project (and vice-versa, of course).","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":9555,"name":"Matthew Booth","email":"mbooth@redhat.com","username":"MatthewBooth"},"change_message_id":"7670f8b2946a6c8e107b0605f63697d7dd8ec5d7","unresolved":false,"context_lines":[{"line_number":191,"context_line":"rule 5. Their setup would be a compute kernel + heat, and assume all"},{"line_number":192,"context_line":"those components are working correctly. If they find a behavior in one"},{"line_number":193,"context_line":"of those components has an ill defined contract, they should submit"},{"line_number":194,"context_line":"upstream tests to the relevant project to nail down that behavior."},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"Prereqs: None"},{"line_number":197,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_00ed6f25","line":194,"in_reply_to":"da86d52c_45869914","updated":"2015-02-10 13:52:23.000000000","message":"\u0027release tags\u0027 was an unfortunate choice of words. I meant that rather than running tip, we could run last week\u0027s tip. We would bump it to today\u0027s tip on a regular basis, but if the co-gated project was temporarily broken we\u0027d have the mechanism in place to avoid the breakage everywhere else while it\u0027s fixed.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"798818f899d6a96640ae06a19e2b937da02cf602","unresolved":false,"context_lines":[{"line_number":191,"context_line":"rule 5. Their setup would be a compute kernel + heat, and assume all"},{"line_number":192,"context_line":"those components are working correctly. If they find a behavior in one"},{"line_number":193,"context_line":"of those components has an ill defined contract, they should submit"},{"line_number":194,"context_line":"upstream tests to the relevant project to nail down that behavior."},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"Prereqs: None"},{"line_number":197,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_45869914","line":194,"in_reply_to":"da86d52c_f6bcb26e","updated":"2015-02-10 13:44:56.000000000","message":"For now, yes. Unwinding that assumption requires a ton of additional work, and makes a pretty invalid assumption that release tags are more stable than the project in general. Often times they are less so as bugs are fixed primarily in master.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"5940048e53e33905bb1aa62e671da881d123e5b0","unresolved":false,"context_lines":[{"line_number":192,"context_line":"those components are working correctly. If they find a behavior in one"},{"line_number":193,"context_line":"of those components has an ill defined contract, they should submit"},{"line_number":194,"context_line":"upstream tests to the relevant project to nail down that behavior."},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"Prereqs: None"},{"line_number":197,"context_line":""},{"line_number":198,"context_line":"This decoupling should be able to happen without substantial coverage"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_e823cb55","line":195,"updated":"2015-02-10 22:29:51.000000000","message":"I think last weeks tip would be even harder to reason about than either a release or tip.\n\nI suggest that we revisit what-revision-to-test-against *after* implementing the change proposed here, because they are almost entirely orthogonal IMO - we need tip+tip to work for developer sanity, so I see any discussion about other combinations as additive, not subtractive.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":1192,"name":"David Kranz","email":"dkranz@redhat.com","username":"david-kranz"},"change_message_id":"807ccddda16aeac1a2063c584862376da2d9dcc4","unresolved":false,"context_lines":[{"line_number":192,"context_line":"those components are working correctly. If they find a behavior in one"},{"line_number":193,"context_line":"of those components has an ill defined contract, they should submit"},{"line_number":194,"context_line":"upstream tests to the relevant project to nail down that behavior."},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"Prereqs: None"},{"line_number":197,"context_line":""},{"line_number":198,"context_line":"This decoupling should be able to happen without substantial coverage"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_9b0eff8a","line":195,"in_reply_to":"da86d52c_e823cb55","updated":"2015-02-11 14:58:38.000000000","message":"Yes, revisiting makes sense. Right now the testing done before exposing a merge to other projects is limited to the budget of what can be done on a per-commit basis. As projects become more focused on the testing and stability of their own apis, with lots more tests, using a release model will be more attractive as it will allow much more testing of changes.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":196,"context_line":"Prereqs: None"},{"line_number":197,"context_line":""},{"line_number":198,"context_line":"This decoupling should be able to happen without substantial coverage"},{"line_number":199,"context_line":"changes in tests."},{"line_number":200,"context_line":""},{"line_number":201,"context_line":"Create Client Library Functional Testing"},{"line_number":202,"context_line":"----------------------------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_ad52b7db","line":199,"updated":"2015-02-11 07:39:54.000000000","message":"The plan is solid but I think it\u0027s leaving out something like:\n- testing the kernel in different configurations\nFor instance nova w/cells, or nova/neutron with config drive vs metadata\nshould this be done as part of #3, or is it a case falling in the #5 category?\n- testing additional services from a project\nfor instance, would neutron advanced services be considered a case falling into the #5 category?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"b8e75ec8e66bf305c3d683dfb68ba5574dfbed1d","unresolved":false,"context_lines":[{"line_number":198,"context_line":"This decoupling should be able to happen without substantial coverage"},{"line_number":199,"context_line":"changes in tests."},{"line_number":200,"context_line":""},{"line_number":201,"context_line":"Create Client Library Functional Testing"},{"line_number":202,"context_line":"----------------------------------------"},{"line_number":203,"context_line":""},{"line_number":204,"context_line":"Client libraries should have their own functional test suites that"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_ce3baae9","line":201,"updated":"2015-02-09 21:53:33.000000000","message":"Why duplicating effort of Rally hm?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":1192,"name":"David Kranz","email":"dkranz@redhat.com","username":"david-kranz"},"change_message_id":"807ccddda16aeac1a2063c584862376da2d9dcc4","unresolved":false,"context_lines":[{"line_number":198,"context_line":"This decoupling should be able to happen without substantial coverage"},{"line_number":199,"context_line":"changes in tests."},{"line_number":200,"context_line":""},{"line_number":201,"context_line":"Create Client Library Functional Testing"},{"line_number":202,"context_line":"----------------------------------------"},{"line_number":203,"context_line":""},{"line_number":204,"context_line":"Client libraries should have their own functional test suites that"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_bb0ca343","line":201,"in_reply_to":"da86d52c_2d0de7e8","updated":"2015-02-11 14:58:38.000000000","message":"I don\u0027t think you are misunderstanding. This has nothing to do with Rally, though a project might choose to use rally for functional testing. The only reason tempest is mentioned so much here is that tempest tests are an integral part of our monolithic gating process and this document is about untangling that.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"798818f899d6a96640ae06a19e2b937da02cf602","unresolved":false,"context_lines":[{"line_number":198,"context_line":"This decoupling should be able to happen without substantial coverage"},{"line_number":199,"context_line":"changes in tests."},{"line_number":200,"context_line":""},{"line_number":201,"context_line":"Create Client Library Functional Testing"},{"line_number":202,"context_line":"----------------------------------------"},{"line_number":203,"context_line":""},{"line_number":204,"context_line":"Client libraries should have their own functional test suites that"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_a5c4b540","line":201,"in_reply_to":"da86d52c_44207bf7","updated":"2015-02-10 13:44:56.000000000","message":"Boris, you completely don\u0027t understand what this paragraph is all about. Rally isn\u0027t testing clients, it\u0027s using client. There is a massive difference there.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":198,"context_line":"This decoupling should be able to happen without substantial coverage"},{"line_number":199,"context_line":"changes in tests."},{"line_number":200,"context_line":""},{"line_number":201,"context_line":"Create Client Library Functional Testing"},{"line_number":202,"context_line":"----------------------------------------"},{"line_number":203,"context_line":""},{"line_number":204,"context_line":"Client libraries should have their own functional test suites that"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_2d0de7e8","line":201,"in_reply_to":"da86d52c_a5c4b540","updated":"2015-02-11 07:39:54.000000000","message":"I think this document is not at all about either Tempest or Rally, but more about testing phylosophy and strategy which might be adopted by these two project (or is already adopted) but might be adopted also for in-tree testing.\n\nAs I see several comments discussing tempest vs. Rally, I\u0027m starting to suspect I\u0027m failing to understand the spirit of this document.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"1c3781a19773b4874c14d3bfdcecda0db021b1b9","unresolved":false,"context_lines":[{"line_number":198,"context_line":"This decoupling should be able to happen without substantial coverage"},{"line_number":199,"context_line":"changes in tests."},{"line_number":200,"context_line":""},{"line_number":201,"context_line":"Create Client Library Functional Testing"},{"line_number":202,"context_line":"----------------------------------------"},{"line_number":203,"context_line":""},{"line_number":204,"context_line":"Client libraries should have their own functional test suites that"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_44207bf7","line":201,"in_reply_to":"da86d52c_c46e2bc9","updated":"2015-02-09 22:39:11.000000000","message":"I agree that CLI interactions is not tested by Rally. But it\u0027s only the smart part of library. \n\nI am trying to say that not using Rally for testing python clients as a library is duplicating effort of Rally.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"2b5979341aeb43151a6f5c2ef1dd2fcab0fc63d9","unresolved":false,"context_lines":[{"line_number":198,"context_line":"This decoupling should be able to happen without substantial coverage"},{"line_number":199,"context_line":"changes in tests."},{"line_number":200,"context_line":""},{"line_number":201,"context_line":"Create Client Library Functional Testing"},{"line_number":202,"context_line":"----------------------------------------"},{"line_number":203,"context_line":""},{"line_number":204,"context_line":"Client libraries should have their own functional test suites that"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_c46e2bc9","line":201,"in_reply_to":"da86d52c_ce3baae9","updated":"2015-02-09 22:24:56.000000000","message":"This is not duplicating Rally. Rally doesn\u0027t cover the execution space of all library calls or cli interactions.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6486,"name":"Brant Knudson","email":"blk@acm.org","username":"blk-u"},"change_message_id":"bda4231184a5c492d90f9f93ce50297b42675618","unresolved":false,"context_lines":[{"line_number":209,"context_line":"not lose test coverage."},{"line_number":210,"context_line":""},{"line_number":211,"context_line":"This should include both testing the library internal interfaces, as"},{"line_number":212,"context_line":"well as testing the the CLI operation through an exec interface."},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"Removal of Admin Tests from Tempest"},{"line_number":215,"context_line":"-----------------------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_8b08c9da","line":212,"updated":"2015-02-09 19:19:25.000000000","message":"change \"the the\" to \"the\".","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"2b5979341aeb43151a6f5c2ef1dd2fcab0fc63d9","unresolved":false,"context_lines":[{"line_number":209,"context_line":"not lose test coverage."},{"line_number":210,"context_line":""},{"line_number":211,"context_line":"This should include both testing the library internal interfaces, as"},{"line_number":212,"context_line":"well as testing the the CLI operation through an exec interface."},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"Removal of Admin Tests from Tempest"},{"line_number":215,"context_line":"-----------------------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_c4f38b08","line":212,"in_reply_to":"da86d52c_8b08c9da","updated":"2015-02-09 22:24:56.000000000","message":"thanks","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":8788,"name":"Miguel Angel Ajo","email":"mangelajo@redhat.com","username":"mangelajo"},"change_message_id":"550f1ae5b37635876addf03df3357a31e70b4c32","unresolved":false,"context_lines":[{"line_number":209,"context_line":"not lose test coverage."},{"line_number":210,"context_line":""},{"line_number":211,"context_line":"This should include both testing the library internal interfaces, as"},{"line_number":212,"context_line":"well as testing the the CLI operation through an exec interface."},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"Removal of Admin Tests from Tempest"},{"line_number":215,"context_line":"-----------------------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_f0ec83bd","line":212,"in_reply_to":"da86d52c_c4f38b08","updated":"2015-02-11 10:54:02.000000000","message":"I like this point, probably the work should start at the with new openstack sdk, so it becomes the new de-facto standard for client libraries.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"b8e75ec8e66bf305c3d683dfb68ba5574dfbed1d","unresolved":false,"context_lines":[{"line_number":211,"context_line":"This should include both testing the library internal interfaces, as"},{"line_number":212,"context_line":"well as testing the the CLI operation through an exec interface."},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"Removal of Admin Tests from Tempest"},{"line_number":215,"context_line":"-----------------------------------"},{"line_number":216,"context_line":""},{"line_number":217,"context_line":"Tempest should only be testing the public, user facing REST API. The"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_0e0ce241","line":214,"updated":"2015-02-09 21:53:33.000000000","message":"Rally has separated tests that requires admin and that doesn\u0027t require. So if Tempest would like to *not* test this Rally can do it.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"8d1a491d1a61d9f0834cc0904e2b029bf1fe56ed","unresolved":false,"context_lines":[{"line_number":211,"context_line":"This should include both testing the library internal interfaces, as"},{"line_number":212,"context_line":"well as testing the the CLI operation through an exec interface."},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"Removal of Admin Tests from Tempest"},{"line_number":215,"context_line":"-----------------------------------"},{"line_number":216,"context_line":""},{"line_number":217,"context_line":"Tempest should only be testing the public, user facing REST API. The"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_a4bdb7d9","line":214,"in_reply_to":"da86d52c_0e0ce241","updated":"2015-02-09 22:30:18.000000000","message":"The point is to not full stack test them, because they should be functional tests done at a lower level.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"1a42afc1623505f7950591db46d1bb5fea77349b","unresolved":false,"context_lines":[{"line_number":211,"context_line":"This should include both testing the library internal interfaces, as"},{"line_number":212,"context_line":"well as testing the the CLI operation through an exec interface."},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"Removal of Admin Tests from Tempest"},{"line_number":215,"context_line":"-----------------------------------"},{"line_number":216,"context_line":""},{"line_number":217,"context_line":"Tempest should only be testing the public, user facing REST API. The"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_0abae56d","line":214,"in_reply_to":"da86d52c_3b72a6ad","updated":"2015-02-10 16:31:55.000000000","message":"Let me try to rephrase without word \"Rally\". \n\nWhat I am interested is to see \"HOW\" we are going to test everything and how it is aligned with QA, Developers, Operators and CI that we have. \n\nIf it is out of the scope of this spec let\u0027s make another that will describe this in details and just put here link to it.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":5263,"name":"Jeremy Stanley","display_name":"fungi","email":"fungi@yuggoth.org","username":"fungi","status":"missing, presumed fed"},"change_message_id":"ab35ce320b48687641c1c7498389f842b5467fb3","unresolved":false,"context_lines":[{"line_number":211,"context_line":"This should include both testing the library internal interfaces, as"},{"line_number":212,"context_line":"well as testing the the CLI operation through an exec interface."},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"Removal of Admin Tests from Tempest"},{"line_number":215,"context_line":"-----------------------------------"},{"line_number":216,"context_line":""},{"line_number":217,"context_line":"Tempest should only be testing the public, user facing REST API. The"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_d3618ed4","line":214,"in_reply_to":"da86d52c_44815b8e","updated":"2015-02-10 01:20:12.000000000","message":"Yes, a separate functional test job. Asking \"what tool should do that\" is like asking \"what tool should run your unit tests?\" It should be up to those projects to decide (within a reasonable scope of things which can run within our project testing infrastructure) and, if they want, to collaborate on some common framework. But that\u0027s outside the scope of this spec.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"798818f899d6a96640ae06a19e2b937da02cf602","unresolved":false,"context_lines":[{"line_number":211,"context_line":"This should include both testing the library internal interfaces, as"},{"line_number":212,"context_line":"well as testing the the CLI operation through an exec interface."},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"Removal of Admin Tests from Tempest"},{"line_number":215,"context_line":"-----------------------------------"},{"line_number":216,"context_line":""},{"line_number":217,"context_line":"Tempest should only be testing the public, user facing REST API. The"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_a5775501","line":214,"in_reply_to":"da86d52c_6eeab5ad","updated":"2015-02-10 13:44:56.000000000","message":"I actually don\u0027t understand your point at all. The point of the functional tests is to ensure that each project is substantially tested in it\u0027s function separate from other projects. This isn\u0027t full stack testing, it\u0027s not a Rally thing. \n\nIt would be really nice if you took the time to understand what\u0027s going on here instead of apparently looking for every possible opportunity to just say \"people should use Rally\".","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"720c2e407281546f73e31e56b136434710f2052a","unresolved":false,"context_lines":[{"line_number":211,"context_line":"This should include both testing the library internal interfaces, as"},{"line_number":212,"context_line":"well as testing the the CLI operation through an exec interface."},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"Removal of Admin Tests from Tempest"},{"line_number":215,"context_line":"-----------------------------------"},{"line_number":216,"context_line":""},{"line_number":217,"context_line":"Tempest should only be testing the public, user facing REST API. The"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_44815b8e","line":214,"in_reply_to":"da86d52c_a4bdb7d9","updated":"2015-02-09 22:45:03.000000000","message":"Could you elaborate? You mean \"admin\" functionality should be tested only in project gates? By some separated functional job?\n\nWhat tool should do that?","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"4d9316d4f1331989bdfe738cdd881656f9dffbe8","unresolved":false,"context_lines":[{"line_number":211,"context_line":"This should include both testing the library internal interfaces, as"},{"line_number":212,"context_line":"well as testing the the CLI operation through an exec interface."},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"Removal of Admin Tests from Tempest"},{"line_number":215,"context_line":"-----------------------------------"},{"line_number":216,"context_line":""},{"line_number":217,"context_line":"Tempest should only be testing the public, user facing REST API. The"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_e09bfbc6","line":214,"in_reply_to":"da86d52c_a5775501","updated":"2015-02-10 13:51:14.000000000","message":"It would be better to stop thinking that I am forcing everybody to use Rally and re-read my paragraph. \n\nProposed change in \"rules\" is going to add really HUGE amount of useless duplication of efforts. \n\nI don\u0027t think that making separated tests (tools) for: \n\n  full stack testing\n  performance testing\n  load testing\n  stress testing\n  negative testing\n  functional testing\n  any other kind of testing...\n\nIs a good approach at all, because you are writing the same code N times (which is one of the worst pattern in development aka \"copy-pasting\") \n\n\nAnd my request to describe better amount of \"copy-pasted\" code and \"duplication of effort\" here is reasonable.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"f204e675ef81a5faacfdaeaee1de873ed61895bf","unresolved":false,"context_lines":[{"line_number":211,"context_line":"This should include both testing the library internal interfaces, as"},{"line_number":212,"context_line":"well as testing the the CLI operation through an exec interface."},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"Removal of Admin Tests from Tempest"},{"line_number":215,"context_line":"-----------------------------------"},{"line_number":216,"context_line":""},{"line_number":217,"context_line":"Tempest should only be testing the public, user facing REST API. The"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_6eeab5ad","line":214,"in_reply_to":"da86d52c_d3618ed4","updated":"2015-02-10 02:10:32.000000000","message":"Actually this should be better described in this spec. \nIt\u0027s actually big decision that has huge impact. \n\nFrom one side having separated functional jobs and tests in projects tree makes things simpler. E.g. we can write code \u0026 functional tests together. \n\nFrom other side we have end users, operators and so on, that would like to have simple way to run all tests and check that everything works (functional tests have imho not only dev purpose)\n\nBefore this decision we have Tempest (as the set of tests for everybody), that with a bit practice and automation can be used. If most tests move to projects it will be really hard for Operators/Users to check their clouds. \n\n\nOne possible solution is to do like in Rally: \n\n  1) Support of in-tree plugins\n  2) Project is able to use tests that are in Rally or their source tree\n  3) Move all interesting tests from in-tree plugins to Rally tree\n  4) Keep all only interesting for dev process tests as in tree plugin\n\n\nJust getting rid of tests from tempest wont\u0027 make our users happier..","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"0d4e3d253422072a2c5f9ec2a42a1223e9f52e7a","unresolved":false,"context_lines":[{"line_number":211,"context_line":"This should include both testing the library internal interfaces, as"},{"line_number":212,"context_line":"well as testing the the CLI operation through an exec interface."},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"Removal of Admin Tests from Tempest"},{"line_number":215,"context_line":"-----------------------------------"},{"line_number":216,"context_line":""},{"line_number":217,"context_line":"Tempest should only be testing the public, user facing REST API. The"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_3b72a6ad","line":214,"in_reply_to":"da86d52c_e09bfbc6","updated":"2015-02-10 14:09:19.000000000","message":"I\u0027ll happily go on record that DRY is anti-pattern in testing. In the proposed model you are not \"writing the same code N times\". Instead you are writing different tests that cross N paths many different times. By crossing those paths from different directions you are increasing the chance that the tests catch (and eventually resolve) bugs which are the result of multi-variant interactions.\n\nWhat\u0027s important is that all the various testing suites make it as easy as possible to replicate a failure. Thus you need to be sure any given suite has properly stated its assumptions and variables and made itself traceable.\n\nIt is a good thing that there is tempest, rally and that there eventually be in-tree functional testing across all the projects. People complain that this causes noise, but it is noise that finds bugs.\n\nI\u0027m less certain that there needs to be a tempest-lib which codifies how the different projects test things. What\u0027s important is that stuff gets tested.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":6486,"name":"Brant Knudson","email":"blk@acm.org","username":"blk-u"},"change_message_id":"bda4231184a5c492d90f9f93ce50297b42675618","unresolved":false,"context_lines":[{"line_number":217,"context_line":"Tempest should only be testing the public, user facing REST API. The"},{"line_number":218,"context_line":"admin APIs should be left to internal project testing."},{"line_number":219,"context_line":""},{"line_number":220,"context_line":"Rationale: The administrative API in OpenStack is a relative new"},{"line_number":221,"context_line":"construct that was created in order to remove the various foo-manage"},{"line_number":222,"context_line":"commands. Most projects have never considered their admin API entirely"},{"line_number":223,"context_line":"stable, and these tests are specifically excluded from all interop"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_cb12c108","line":220,"updated":"2015-02-09 19:19:25.000000000","message":"change \"relative\" to \"relatively\"","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"2b5979341aeb43151a6f5c2ef1dd2fcab0fc63d9","unresolved":false,"context_lines":[{"line_number":217,"context_line":"Tempest should only be testing the public, user facing REST API. The"},{"line_number":218,"context_line":"admin APIs should be left to internal project testing."},{"line_number":219,"context_line":""},{"line_number":220,"context_line":"Rationale: The administrative API in OpenStack is a relative new"},{"line_number":221,"context_line":"construct that was created in order to remove the various foo-manage"},{"line_number":222,"context_line":"commands. Most projects have never considered their admin API entirely"},{"line_number":223,"context_line":"stable, and these tests are specifically excluded from all interop"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_e4f6cff8","line":220,"in_reply_to":"da86d52c_cb12c108","updated":"2015-02-09 22:24:56.000000000","message":"thanks","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":229,"context_line":"Tempest should make solid coverage of the public API much easier."},{"line_number":230,"context_line":""},{"line_number":231,"context_line":"Prereqs: projects must spin up functional tests that cover this"},{"line_number":232,"context_line":"functionality."},{"line_number":233,"context_line":""},{"line_number":234,"context_line":""},{"line_number":235,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_8d1ed30a","line":232,"updated":"2015-02-11 07:39:54.000000000","message":"Another prereq is probably that projects should clearly separate the admin APIs from the public facing ones. Neutron doesn\u0027t for instance. I\u0027m not sure Nova has a clear separation either.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"65ffaddb898237bffafdeade6eed5307e4a1d7e2","unresolved":false,"context_lines":[{"line_number":229,"context_line":"Tempest should make solid coverage of the public API much easier."},{"line_number":230,"context_line":""},{"line_number":231,"context_line":"Prereqs: projects must spin up functional tests that cover this"},{"line_number":232,"context_line":"functionality."},{"line_number":233,"context_line":""},{"line_number":234,"context_line":""},{"line_number":235,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":3,"id":"ba7be1f8_862c3b4c","line":232,"in_reply_to":"da86d52c_8d1ed30a","updated":"2015-02-28 16:21:37.000000000","message":"@Sal, no, Nova\u0027s \"admin API\" doesn\u0027t have any separation either. :( Muddying this arena is the policy.json framework stuff that allows any deployer to modify what is or isn\u0027t an \"admin API\" call, making it virtually impossible to count on any API interoperability between OpenStack deployments. :(","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":8788,"name":"Miguel Angel Ajo","email":"mangelajo@redhat.com","username":"mangelajo"},"change_message_id":"550f1ae5b37635876addf03df3357a31e70b4c32","unresolved":false,"context_lines":[{"line_number":229,"context_line":"Tempest should make solid coverage of the public API much easier."},{"line_number":230,"context_line":""},{"line_number":231,"context_line":"Prereqs: projects must spin up functional tests that cover this"},{"line_number":232,"context_line":"functionality."},{"line_number":233,"context_line":""},{"line_number":234,"context_line":""},{"line_number":235,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_906ee72d","line":232,"in_reply_to":"da86d52c_8d1ed30a","updated":"2015-02-11 10:54:02.000000000","message":"Separate the Admin APIs? , why?, that may lead to code / API replication in many cases (thinking of neutron). I\u0027m not following :)","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2035,"name":"Maru Newby","email":"marun@redhat.com","username":"maru"},"change_message_id":"3b10d898015c2a3ca62fc305a1585ac40c47e65a","unresolved":false,"context_lines":[{"line_number":229,"context_line":"Tempest should make solid coverage of the public API much easier."},{"line_number":230,"context_line":""},{"line_number":231,"context_line":"Prereqs: projects must spin up functional tests that cover this"},{"line_number":232,"context_line":"functionality."},{"line_number":233,"context_line":""},{"line_number":234,"context_line":""},{"line_number":235,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_564b952a","line":232,"in_reply_to":"da86d52c_906ee72d","updated":"2015-02-17 23:35:44.000000000","message":"If that\u0027s true, it\u0027s an accident of implementation.  Any modern wsgi framework can trivially provide the kind of routing isolation we\u0027re talking about.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":5263,"name":"Jeremy Stanley","display_name":"fungi","email":"fungi@yuggoth.org","username":"fungi","status":"missing, presumed fed"},"change_message_id":"ab35ce320b48687641c1c7498389f842b5467fb3","unresolved":false,"context_lines":[{"line_number":235,"context_line":"Alternatives"},{"line_number":236,"context_line":"------------"},{"line_number":237,"context_line":""},{"line_number":238,"context_line":"Become goat herders"},{"line_number":239,"context_line":""},{"line_number":240,"context_line":""},{"line_number":241,"context_line":"Implementation"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_d3e6ee43","line":238,"updated":"2015-02-10 01:20:12.000000000","message":"I recommend deep sea fishing or perhaps long-haul trucking.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":5997,"name":"Walt","display_name":"Hemna","email":"waboring@hemna.com","username":"walter-boring","status":"SAP"},"change_message_id":"ce57094ad51f2de5f4814aafd19e58fbbebaf133","unresolved":false,"context_lines":[{"line_number":235,"context_line":"Alternatives"},{"line_number":236,"context_line":"------------"},{"line_number":237,"context_line":""},{"line_number":238,"context_line":"Become goat herders"},{"line_number":239,"context_line":""},{"line_number":240,"context_line":""},{"line_number":241,"context_line":"Implementation"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_b0794cfe","line":238,"in_reply_to":"da86d52c_d3e6ee43","updated":"2015-02-10 17:04:16.000000000","message":"Alaskan gold miners, or pawn shop owners.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"5940048e53e33905bb1aa62e671da881d123e5b0","unresolved":false,"context_lines":[{"line_number":236,"context_line":"------------"},{"line_number":237,"context_line":""},{"line_number":238,"context_line":"Become goat herders"},{"line_number":239,"context_line":""},{"line_number":240,"context_line":""},{"line_number":241,"context_line":"Implementation"},{"line_number":242,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_a8d47320","line":239,"updated":"2015-02-10 22:29:51.000000000","message":"Pig farmers. https://www.youtube.com/watch?v\u003db2F-DItXtZs","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":261,"name":"Salvatore Orlando","email":"salv.orlando@gmail.com","username":"salvatore-orlando"},"change_message_id":"3edb7ef660ce0951ea7df453fa792d31feaced3b","unresolved":false,"context_lines":[{"line_number":242,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":243,"context_line":""},{"line_number":244,"context_line":"Assignee(s)"},{"line_number":245,"context_line":"-----------"},{"line_number":246,"context_line":""},{"line_number":247,"context_line":"- Gate Break Down"},{"line_number":248,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_0dc10354","line":245,"updated":"2015-02-11 07:39:54.000000000","message":"I think that the document at some point mentioned component-level functional testing as something that should be done as part of the \"testing pyramid\". It should probably also be captured here.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":5263,"name":"Jeremy Stanley","display_name":"fungi","email":"fungi@yuggoth.org","username":"fungi","status":"missing, presumed fed"},"change_message_id":"ab35ce320b48687641c1c7498389f842b5467fb3","unresolved":false,"context_lines":[{"line_number":250,"context_line":""},{"line_number":251,"context_line":"- Ensure service tags are correct in Tempest"},{"line_number":252,"context_line":""},{"line_number":253,"context_line":"  - TBD"},{"line_number":254,"context_line":""},{"line_number":255,"context_line":"- Client Functional Testing"},{"line_number":256,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_73cd42b7","line":253,"updated":"2015-02-10 01:20:12.000000000","message":"There may be some interest from the refstack crowd in stepping up for this--might want to reach out to them on it.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":5196,"name":"Matthew Treinish","email":"mtreinish@kortar.org","username":"treinish"},"change_message_id":"577906ab619ee7049d7897449592aaa2e69cac3b","unresolved":false,"context_lines":[{"line_number":258,"context_line":""},{"line_number":259,"context_line":"- Removal of admin tests from Tempest"},{"line_number":260,"context_line":""},{"line_number":261,"context_line":"  - TBD"},{"line_number":262,"context_line":""},{"line_number":263,"context_line":"Work Items"},{"line_number":264,"context_line":"----------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_53ae4c2f","line":261,"updated":"2015-02-10 23:09:29.000000000","message":"Like I said in an earlier rev I\u0027d be fine shepherding this when the time is right. \n\nIt will really require co-operation of all parties to ensure we have adequate coverage in the projects first. But I\u0027m fine having my name attached to this.","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":7020,"name":"Joe Hakim Rahme","email":"jhakimra@redhat.com","username":"rahmu"},"change_message_id":"af307261ff94e1e5466235a9349355b8ead0c4a8","unresolved":false,"context_lines":[{"line_number":277,"context_line":"---------"},{"line_number":278,"context_line":""},{"line_number":279,"context_line":"For people new to the OpenStack project, co-gating is the practice of"},{"line_number":280,"context_line":"coupling 2 or more project\u0027s git trees in testing. The theory holds"},{"line_number":281,"context_line":"that the 2 projects are too interconnected to ensure that if project A"},{"line_number":282,"context_line":"passes tests, and project B passes tests, that when deployed in a"},{"line_number":283,"context_line":"combined way they will both pass tests together. So changes to project"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_68caada2","line":280,"range":{"start_line":280,"start_character":14,"end_line":280,"end_character":18},"updated":"2015-02-10 00:48:10.000000000","message":"\"2 or more projects\u0027 git trees\"","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"},{"author":{"_account_id":2750,"name":"Sean Dague","email":"sean@dague.net","username":"sdague"},"change_message_id":"798818f899d6a96640ae06a19e2b937da02cf602","unresolved":false,"context_lines":[{"line_number":277,"context_line":"---------"},{"line_number":278,"context_line":""},{"line_number":279,"context_line":"For people new to the OpenStack project, co-gating is the practice of"},{"line_number":280,"context_line":"coupling 2 or more project\u0027s git trees in testing. The theory holds"},{"line_number":281,"context_line":"that the 2 projects are too interconnected to ensure that if project A"},{"line_number":282,"context_line":"passes tests, and project B passes tests, that when deployed in a"},{"line_number":283,"context_line":"combined way they will both pass tests together. So changes to project"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da86d52c_65547d54","line":280,"range":{"start_line":280,"start_character":14,"end_line":280,"end_character":18},"in_reply_to":"da86d52c_68caada2","updated":"2015-02-10 13:44:56.000000000","message":"Thanks","commit_id":"4e6bb1d1eea68735d33db2b75a8fecf5d2719150"}]}