)]}'
{"HACKING.rst":[{"author":{"_account_id":5196,"name":"Matthew Treinish","email":"mtreinish@kortar.org","username":"treinish"},"change_message_id":"d0277528e93edd4d0c28c216d5000e3b3f6e01fa","unresolved":false,"context_lines":[{"line_number":167,"context_line":"the wide variety and complexity of negative tests can lead to long test"},{"line_number":168,"context_line":"runs and knowledge of internal implementation details. The bulk of"},{"line_number":169,"context_line":"negative testing should be handled with project function tests. The"},{"line_number":170,"context_line":"exception to this rule is API tests used for interoperability testing."},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"Test skips because of Known Bugs"},{"line_number":173,"context_line":"--------------------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a98d1f4_1848012d","side":"PARENT","line":170,"updated":"2016-09-29 19:42:05.000000000","message":"I actually liked this little bit you\u0027re removing here. It provides a little more background detail and also suggests that the first place to try adding tests is in tree testing","commit_id":"6728df8792af7fa83f222bb83850643c12d5fe95"},{"author":{"_account_id":6167,"name":"Ken\u0027ichi Ohmichi","email":"ken1ohmichi@gmail.com","username":"oomichi"},"change_message_id":"ab856589a380303ecd4e72821b2090a038586386","unresolved":false,"context_lines":[{"line_number":167,"context_line":"the wide variety and complexity of negative tests can lead to long test"},{"line_number":168,"context_line":"runs and knowledge of internal implementation details. The bulk of"},{"line_number":169,"context_line":"negative testing should be handled with project function tests. The"},{"line_number":170,"context_line":"exception to this rule is API tests used for interoperability testing."},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"Test skips because of Known Bugs"},{"line_number":173,"context_line":"--------------------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a98d1f4_ecbab76d","side":"PARENT","line":170,"in_reply_to":"3a98d1f4_1848012d","updated":"2016-09-29 21:31:04.000000000","message":"oh, I see.\n\nI cannot remind why I removed the first place..","commit_id":"6728df8792af7fa83f222bb83850643c12d5fe95"},{"author":{"_account_id":5196,"name":"Matthew Treinish","email":"mtreinish@kortar.org","username":"treinish"},"change_message_id":"d0277528e93edd4d0c28c216d5000e3b3f6e01fa","unresolved":false,"context_lines":[{"line_number":171,"context_line":"Please do not add negative tests which are not based on `API-WG guideline`_"},{"line_number":172,"context_line":"because each project(Nova, Cinder, etc.) has their own manners for fixing"},{"line_number":173,"context_line":"wrong behaviors which are against `API-WG guideline`_ . So Tempest should"},{"line_number":174,"context_line":"not block such changes on each project side."},{"line_number":175,"context_line":""},{"line_number":176,"context_line":".. API-WG guideline: https://github.com/openstack/api-wg/blob/master/guidelines/http.rst#failure-code-clarifications"},{"line_number":177,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a98d1f4_7883c556","line":174,"updated":"2016-09-29 19:42:05.000000000","message":"So I actually have an issue with this section. All APIs have warts and the API-WG guidelines are just that guidelines. They don\u0027t actually reflect the reality of how the apis work. While most of what the guidelines say for error responses are just basic http stuff, and I agree that\u0027s a better state for apis to be in, I don\u0027t think we can just ignore the things that fall outside of this.\n\nFor example, if a response code changes from 404 to 400 to be be more correct that still has the potential to break end users. The same way a switch from a 200 to a 204 for a response with an empty body would.","commit_id":"395e10ccb8ed0b31da657f7ad27f32efb21ee8d9"},{"author":{"_account_id":6167,"name":"Ken\u0027ichi Ohmichi","email":"ken1ohmichi@gmail.com","username":"oomichi"},"change_message_id":"ab856589a380303ecd4e72821b2090a038586386","unresolved":false,"context_lines":[{"line_number":171,"context_line":"Please do not add negative tests which are not based on `API-WG guideline`_"},{"line_number":172,"context_line":"because each project(Nova, Cinder, etc.) has their own manners for fixing"},{"line_number":173,"context_line":"wrong behaviors which are against `API-WG guideline`_ . So Tempest should"},{"line_number":174,"context_line":"not block such changes on each project side."},{"line_number":175,"context_line":""},{"line_number":176,"context_line":".. API-WG guideline: https://github.com/openstack/api-wg/blob/master/guidelines/http.rst#failure-code-clarifications"},{"line_number":177,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a98d1f4_4c956bdc","line":174,"in_reply_to":"3a98d1f4_7883c556","updated":"2016-09-29 21:31:04.000000000","message":"Matthew,\n\n \u003e if a response code changes from 404 to 400 to be be more\n \u003e correct that still has the potential to break end users\n\nYeah, this is a nice point. and now each project is doing such changes without bumping a new microversion even if having microversions mechanism sometimes. Such changes are handled case by case during reviews, ml or irc discussions on each project side.\n\nI still think it is not ideal to verify non-ideal error behavior as QA team. As you said, these error behavior changes should be handled as the same as success behavior from the other strict viewpoint. But actually, these changes are handled case by case.\n\nIn addition, we said it was fine to change error status code if we found more accurate code as our history even but we said we should not change success code at the same time.\nSo we understate error behaviors than success ones, and I think each project can have decision right for such corner cases(against API-WG guideline) without Tempest testing.","commit_id":"395e10ccb8ed0b31da657f7ad27f32efb21ee8d9"},{"author":{"_account_id":6167,"name":"Ken\u0027ichi Ohmichi","email":"ken1ohmichi@gmail.com","username":"oomichi"},"change_message_id":"300682c7329209fe6f893792447ecc0658fb952b","unresolved":false,"context_lines":[{"line_number":169,"context_line":"negative testing should be handled with project function tests."},{"line_number":170,"context_line":"All negative tests should be based on `API-WG guideline`_ . Such negative"},{"line_number":171,"context_line":"tests can block any changes from accurate failure code to invalid one."},{"line_number":172,"context_line":"Please do not add negative tests which are not based on `API-WG guideline`_"},{"line_number":173,"context_line":"because each project(Nova, Cinder, etc.) has their own manners for fixing"},{"line_number":174,"context_line":"wrong behaviors which are against `API-WG guideline`_ . So Tempest should"},{"line_number":175,"context_line":"not block such changes on each project side."},{"line_number":176,"context_line":""},{"line_number":177,"context_line":".. _API-WG guideline: https://github.com/openstack/api-wg/blob/master/guidelines/http.rst#failure-code-clarifications"},{"line_number":178,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a95cdbc_8c209c4b","line":175,"range":{"start_line":172,"start_character":0,"end_line":175,"end_character":44},"updated":"2016-10-04 22:22:19.000000000","message":"We discuss here on IRC and some project don\u0027t follow api-wg guideline at all and these cases contain common error cases.\n\nSo it is nice to block common error cases\u0027 changes on Tempes side even if original errors don\u0027t follow the guideline.\nI will remove this block at this time and it is nice to discuss it when we face some problems in the future.","commit_id":"18e48c62a9bf566e6058352a7a1607d10b6ffee4"},{"author":{"_account_id":5196,"name":"Matthew Treinish","email":"mtreinish@kortar.org","username":"treinish"},"change_message_id":"356e4ed19f86c8b6fb341d8476bdc936df428b8f","unresolved":false,"context_lines":[{"line_number":175,"context_line":"If facing some gray area which is not clarified on the above guideline,"},{"line_number":176,"context_line":"please propose a new guideline to `API-WG guideline`_ . We will be able"},{"line_number":177,"context_line":"to build a consensus on whole OpenStack projects and that will make the"},{"line_number":178,"context_line":"quality better actually."},{"line_number":179,"context_line":"In addition, we have some guidelines for additional negative tests."},{"line_number":180,"context_line":""},{"line_number":181,"context_line":"- About BadRequest(HTTP400) case: We can add a single negative tests of"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1a95cdbc_8cf89c7e","line":178,"updated":"2016-10-04 22:30:57.000000000","message":"This last sentence is worded a bit awkwardly. I would write it as something like:\n\n  With a proposal to the API-WG we will be able to build a \n  consensus across all OpenStack projects and improve the\n  quality and consistency of all the APIs.","commit_id":"d0a825a5d690a95c303b0cb058cf6657e7f7ad8f"},{"author":{"_account_id":6167,"name":"Ken\u0027ichi Ohmichi","email":"ken1ohmichi@gmail.com","username":"oomichi"},"change_message_id":"91d78908bc61985c9f9ecef5c0a1e1a4f4d739b2","unresolved":false,"context_lines":[{"line_number":175,"context_line":"If facing some gray area which is not clarified on the above guideline,"},{"line_number":176,"context_line":"please propose a new guideline to `API-WG guideline`_ . We will be able"},{"line_number":177,"context_line":"to build a consensus on whole OpenStack projects and that will make the"},{"line_number":178,"context_line":"quality better actually."},{"line_number":179,"context_line":"In addition, we have some guidelines for additional negative tests."},{"line_number":180,"context_line":""},{"line_number":181,"context_line":"- About BadRequest(HTTP400) case: We can add a single negative tests of"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1a95cdbc_cc81c491","line":178,"in_reply_to":"1a95cdbc_8cf89c7e","updated":"2016-10-04 22:41:28.000000000","message":"Cool, that seems stylish.","commit_id":"d0a825a5d690a95c303b0cb058cf6657e7f7ad8f"},{"author":{"_account_id":5196,"name":"Matthew Treinish","email":"mtreinish@kortar.org","username":"treinish"},"change_message_id":"356e4ed19f86c8b6fb341d8476bdc936df428b8f","unresolved":false,"context_lines":[{"line_number":176,"context_line":"please propose a new guideline to `API-WG guideline`_ . We will be able"},{"line_number":177,"context_line":"to build a consensus on whole OpenStack projects and that will make the"},{"line_number":178,"context_line":"quality better actually."},{"line_number":179,"context_line":"In addition, we have some guidelines for additional negative tests."},{"line_number":180,"context_line":""},{"line_number":181,"context_line":"- About BadRequest(HTTP400) case: We can add a single negative tests of"},{"line_number":182,"context_line":"  BadRequest for each resource and method(POST, PUT)."}],"source_content_type":"text/x-rst","patch_set":6,"id":"1a95cdbc_2c9a2815","line":179,"updated":"2016-10-04 22:30:57.000000000","message":"I think we should talk about how to expand this list just to cover all the bases. Also, as I said in an earlier review I do think it would be good to mention reviewer discretion, since there are a lot of cases outside of 400 and 404. But, I won\u0027t block on that because it is kinda implied in every review.","commit_id":"d0a825a5d690a95c303b0cb058cf6657e7f7ad8f"},{"author":{"_account_id":6167,"name":"Ken\u0027ichi Ohmichi","email":"ken1ohmichi@gmail.com","username":"oomichi"},"change_message_id":"f5a67bcde35032f63f124276ac53d00478ad4923","unresolved":false,"context_lines":[{"line_number":191,"context_line":""},{"line_number":192,"context_line":"The above guidelines don\u0027t cover all cases and we will grow these guidelines"},{"line_number":193,"context_line":"organically over time. Patches outside of the above guidelines are left up to"},{"line_number":194,"context_line":"the reviewers\u0027 discretion and if we face some conflicts between reviewers, we will expand the guideline based on our discussion and experience."},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"Test skips because of Known Bugs"},{"line_number":197,"context_line":"--------------------------------"}],"source_content_type":"text/x-rst","patch_set":8,"id":"1a95cdbc_8f7c0e22","line":194,"range":{"start_line":194,"start_character":77,"end_line":194,"end_character":78},"updated":"2016-10-04 22:51:41.000000000","message":"here should put on a new line","commit_id":"c7586b41fb3e444016ad16da403183df46f7b065"},{"author":{"_account_id":10526,"name":"Catherine Diep","email":"cdiep@us.ibm.com","username":"cdiep"},"change_message_id":"dd1ac25d8e5ddd3ffd1592f5b9e5f4c5a7067ee9","unresolved":false,"context_lines":[{"line_number":166,"context_line":"suite, Tempest is not suitable for handling all negative test cases, as"},{"line_number":167,"context_line":"the wide variety and complexity of negative tests can lead to long test"},{"line_number":168,"context_line":"runs and knowledge of internal implementation details. The bulk of"},{"line_number":169,"context_line":"negative testing should be handled with project function tests."},{"line_number":170,"context_line":"All negative tests should be based on `API-WG guideline`_ . Such negative"},{"line_number":171,"context_line":"tests can block any changes from accurate failure code to invalid one."},{"line_number":172,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"1a95cdbc_82ddc044","line":169,"range":{"start_line":169,"start_character":62,"end_line":169,"end_character":63},"updated":"2016-10-05 04:37:24.000000000","message":"Just want to confirm that since the sentence [1] (from the existing hacking file) is removed.  Does this mean that ALL negative tests should be handled with project function tests?\n\n[1] \"The exception to this rule is API tests used for interoperability testing.\"","commit_id":"8db407576b46e3ec9727ae9127e805b76296fa66"},{"author":{"_account_id":6167,"name":"Ken\u0027ichi Ohmichi","email":"ken1ohmichi@gmail.com","username":"oomichi"},"change_message_id":"9612487dc2320f9e7a127e85e3f9c583c485625e","unresolved":false,"context_lines":[{"line_number":166,"context_line":"suite, Tempest is not suitable for handling all negative test cases, as"},{"line_number":167,"context_line":"the wide variety and complexity of negative tests can lead to long test"},{"line_number":168,"context_line":"runs and knowledge of internal implementation details. The bulk of"},{"line_number":169,"context_line":"negative testing should be handled with project function tests."},{"line_number":170,"context_line":"All negative tests should be based on `API-WG guideline`_ . Such negative"},{"line_number":171,"context_line":"tests can block any changes from accurate failure code to invalid one."},{"line_number":172,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"1a95cdbc_06cfee7f","line":169,"range":{"start_line":169,"start_character":62,"end_line":169,"end_character":63},"in_reply_to":"1a95cdbc_82ddc044","updated":"2016-10-05 13:44:39.000000000","message":"Full coverage of negative tests should be implemented with project side.\nBut a part of negative tests can be still implemented in Tempest like current implementation as the following said.","commit_id":"8db407576b46e3ec9727ae9127e805b76296fa66"}]}