)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":1669,"name":"Julien Danjou","display_name":"jd","email":"julien@danjou.info","username":"jdanjou"},"change_message_id":"4c672f7865bbb1bb134233946ba2521d19907be8","unresolved":false,"context_lines":[{"line_number":10,"context_line":"terms of coverage and readability, a suite of readable declarative"},{"line_number":11,"context_line":"tests is proposed."},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Note: The forms and formats proposed here are fully up for"},{"line_number":14,"context_line":"discussion. I think this is a thing we need but the form it takes is"},{"line_number":15,"context_line":"less important."},{"line_number":16,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"9aa7fdbe_5a10854e","line":13,"updated":"2014-10-20 17:51:59.000000000","message":"Not really helpful to have this once merged, and you know, we all know that patches on Gerrit are up to discussion. ;)","commit_id":"5f4e9dafe38537a29348a3d7c60d0a595e52764c"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"39358e5c372a0e4bcddabf20a399f51b1dfd21e7","unresolved":false,"context_lines":[{"line_number":10,"context_line":"terms of coverage and readability, a suite of readable declarative"},{"line_number":11,"context_line":"tests is proposed."},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Note: The forms and formats proposed here are fully up for"},{"line_number":14,"context_line":"discussion. I think this is a thing we need but the form it takes is"},{"line_number":15,"context_line":"less important."},{"line_number":16,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"9aa7fdbe_9482c150","line":13,"in_reply_to":"9aa7fdbe_5a10854e","updated":"2014-10-20 19:52:14.000000000","message":"Sure, just a friendly reminder because we\u0027ve seen some people be reluctant to either challenge a spec or get upset when their spec is challenged. I\u0027ll remove it soon enough.","commit_id":"5f4e9dafe38537a29348a3d7c60d0a595e52764c"}],"specs/kilo/declarative-http-tests.rst":[{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"ee8473ae8b4ba4345a2870168c5de5bf62cecd60","unresolved":false,"context_lines":[{"line_number":87,"context_line":""},{"line_number":88,"context_line":"* A directory including multiple YAML files each representing a"},{"line_number":89,"context_line":"  transaction/scenario of some kind in the API *or* all the routes"},{"line_number":90,"context_line":"  in one version of the API."},{"line_number":91,"context_line":""},{"line_number":92,"context_line":"Note that this proposal make no mention of client code because it is"},{"line_number":93,"context_line":"direct HTTP requests that are being tested, not calls to a client"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9aa7fdbe_ad04d815","line":90,"updated":"2014-10-20 20:08:31.000000000","message":"How would test setup be handled? (i.e. to ensure that the expected datapoints are present to be returned by the API)\n\nOr is the expectation that the initial sequence of requests are POSTs of samples/alarms/etc. so that the data store is populated via the API before being queried?\n\nWould that imply that the state is cumulative as the we proceed through the set of testcases on an individual test run? (or just accumulate for an individual \"transaction/scenario\"?)\n\nWhat about setup for the tests of endpoints where isn\u0027t a direct POST API, such as events or alarm history?","commit_id":"f87c46d72295fbf8ea5a7ffb3659ee32e32194b9"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"5b8b4620b4eb333a315d001cedb6cbe4d723e7b3","unresolved":false,"context_lines":[{"line_number":87,"context_line":""},{"line_number":88,"context_line":"* A directory including multiple YAML files each representing a"},{"line_number":89,"context_line":"  transaction/scenario of some kind in the API *or* all the routes"},{"line_number":90,"context_line":"  in one version of the API."},{"line_number":91,"context_line":""},{"line_number":92,"context_line":"Note that this proposal make no mention of client code because it is"},{"line_number":93,"context_line":"direct HTTP requests that are being tested, not calls to a client"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9aa7fdbe_2de4680c","line":90,"in_reply_to":"9aa7fdbe_ad04d815","updated":"2014-10-20 20:22:42.000000000","message":"When I\u0027ve done this before setup is handled, as needed, in the runner (the thing that loads the YAML files). We can control additional prerequisites however we like: fixtures in the runner, fixtures declared in the YAML, code snippets that need to run.\n\nI\u0027ll add a section that makes some hand-wavey gestures about that. Hand-wavey because I don\u0027t know yet. I feel there are several options and which is best may not be clear until the entire set of needs has been enumerated. It may be that we need different things for different parts of the API. At the stage I simply don\u0027t know (which is either a bug or a feature in the spec process, I\u0027m not sure which ).\n\nWhen possible it makes sense that the API be used to set up the data for the rest of the tests but it may be that I think that is okay because I prefer tests to exit on their first failure. If these tests don\u0027t do that they some kind of out of band setup will be required. I feel pretty strongly that that would be the wrong way to go.","commit_id":"f87c46d72295fbf8ea5a7ffb3659ee32e32194b9"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"8655edbe979fc1965ea5f120b943f4de63c61479","unresolved":false,"context_lines":[{"line_number":43,"context_line":"To address the shortcomings in HTTP testing (readability and"},{"line_number":44,"context_line":"completeness) a new target for ``tox`` will be defined that runs a"},{"line_number":45,"context_line":"suite of HTTP tests which are expressed as HTTP requests with"},{"line_number":46,"context_line":"expected responses. The exact format should be determined from"},{"line_number":47,"context_line":"discussion but some examples are listed in the further discussion"},{"line_number":48,"context_line":"below."},{"line_number":49,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"9aa7fdbe_8300c5ab","line":46,"updated":"2014-10-20 21:53:06.000000000","message":"Just wondering about the style of matching against the expected responses: literal, wildcarded, full regex-based?\n\nI\u0027m thinking in terms of elements of the API responses that may not be predictable in advance, such as say a timestamp that wasn\u0027t explicitly specified in the corresponding request, so defaulted to the current time when the sample was processed.\n\nAlso for highly repetitive response payloads (containing say multiple samples, or events, or alarm history items), would it be useful to be able to express a constraint on the response in terms of some multiple of a pattern?","commit_id":"cbe1d2abfa2f3be1d938d57a87f2ce07cdc215a1"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"9f7ff6c1bb006f6ff007cd95ae3de9fb1cebb07b","unresolved":false,"context_lines":[{"line_number":43,"context_line":"To address the shortcomings in HTTP testing (readability and"},{"line_number":44,"context_line":"completeness) a new target for ``tox`` will be defined that runs a"},{"line_number":45,"context_line":"suite of HTTP tests which are expressed as HTTP requests with"},{"line_number":46,"context_line":"expected responses. The exact format should be determined from"},{"line_number":47,"context_line":"discussion but some examples are listed in the further discussion"},{"line_number":48,"context_line":"below."},{"line_number":49,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"9aa7fdbe_436a5d1b","line":46,"in_reply_to":"9aa7fdbe_8300c5ab","updated":"2014-10-20 22:06:35.000000000","message":"The sample linked as \u0027good success\u0027 below (and here https://github.com/tiddlyweb/tiddlyweb/blob/master/test/httptest.yaml ) uses a list of string fragments as the expected output. That works for simple scenarious but has some limitations.\n\nI think, for the most part, that constraints in testing tools can be useful because it leads to tests which are more explicit and thus more readable. For assertions that can\u0027t be expressed relatively simply (in a kind of call and response way) this is probably not the tool for the job. That\u0027s good! We don\u0027t want job that does all things with mediocrity. We want several tools, each of which does a smaller number of jobs well.\n\nIn this case the job is \"traverse the breadth of the API to confirm it is in rude health\" not \"validate the payload from a query that expects a large response\".\n\nI should probably make that last paragraph a more explicit part of the description in the intro. Will do so tomorrow.","commit_id":"cbe1d2abfa2f3be1d938d57a87f2ce07cdc215a1"},{"author":{"_account_id":3012,"name":"Dina Belova","email":"dbelova@mirantis.com","username":"dbelova"},"change_message_id":"b3a0fb39440cd0155fdd58e0dbd50f5b1522e19c","unresolved":false,"context_lines":[{"line_number":53,"context_line":"completeness) a new target for ``tox`` will be defined that runs a"},{"line_number":54,"context_line":"suite of HTTP tests which are expressed as HTTP requests with"},{"line_number":55,"context_line":"expected responses. The exact format should be determined from"},{"line_number":56,"context_line":"discussion but some examples are listed in the further discussion"},{"line_number":57,"context_line":"below."},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"This same suite of tests will also be registered as a check job to"}],"source_content_type":"text/x-rst","patch_set":5,"id":"9aa7fdbe_bf8027c1","line":56,"range":{"start_line":56,"start_character":55,"end_line":56,"end_character":65},"updated":"2014-10-23 08:55:02.000000000","message":"description probably?","commit_id":"2616367622ccaa768455de401a0a604721e7afea"},{"author":{"_account_id":3012,"name":"Dina Belova","email":"dbelova@mirantis.com","username":"dbelova"},"change_message_id":"b3a0fb39440cd0155fdd58e0dbd50f5b1522e19c","unresolved":false,"context_lines":[{"line_number":68,"context_line":"To ensure the tests are lightweight and fast no web server will be used"},{"line_number":69,"context_line":"and only one of whichever lightweight datastore (mongo, sqlite3) for"},{"line_number":70,"context_line":"limited persistence. These tests are for testing the HTTP API aspect of"},{"line_number":71,"context_line":"Ceilometer and as such are not scenario tests: We don\u0027t need to run"},{"line_number":72,"context_line":"these tests against every available datastore."},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"\"No web server\" will be achieved by using `wsgi-intercept`_ or"}],"source_content_type":"text/x-rst","patch_set":5,"id":"9aa7fdbe_1f56935c","line":71,"range":{"start_line":71,"start_character":47,"end_line":71,"end_character":49},"updated":"2014-10-23 08:55:02.000000000","message":"capital letter?","commit_id":"2616367622ccaa768455de401a0a604721e7afea"},{"author":{"_account_id":3012,"name":"Dina Belova","email":"dbelova@mirantis.com","username":"dbelova"},"change_message_id":"b3a0fb39440cd0155fdd58e0dbd50f5b1522e19c","unresolved":false,"context_lines":[{"line_number":113,"context_line":""},{"line_number":114,"context_line":"* What format for the tests?"},{"line_number":115,"context_line":"* How to handle authN and authZ in a minimal fashion? Ideally keystone"},{"line_number":116,"context_line":"  won\u0027t be running. If that\u0027s the case, what needs to be done."},{"line_number":117,"context_line":"* How to aggregate request and response groups into single tests?"},{"line_number":118,"context_line":"  One option is for each YAML file to be treated as a single test."},{"line_number":119,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"9aa7fdbe_9f97e3eb","line":116,"updated":"2014-10-23 08:55:02.000000000","message":"Actually at Mirantis we had kind of testing issue like this, and that time we found no actual opportunity to workaround the Keystone. It was for a while ago, so I\u0027ll be happy it\u0027ll be possible :)","commit_id":"2616367622ccaa768455de401a0a604721e7afea"},{"author":{"_account_id":3012,"name":"Dina Belova","email":"dbelova@mirantis.com","username":"dbelova"},"change_message_id":"b3a0fb39440cd0155fdd58e0dbd50f5b1522e19c","unresolved":false,"context_lines":[{"line_number":149,"context_line":"  complexity of the harness significantly. Slow tests discourage people"},{"line_number":150,"context_line":"  running them and complexity leads to breakage."},{"line_number":151,"context_line":""},{"line_number":152,"context_line":"* Do nothing. Using existing unit tests and tempests test. This is"},{"line_number":153,"context_line":"  not a good option as neither the existing unit tests nor Tempest tests"},{"line_number":154,"context_line":"  present good (and readable) coverage."},{"line_number":155,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"9aa7fdbe_3f88b7c4","line":152,"range":{"start_line":152,"start_character":2,"end_line":152,"end_character":12},"updated":"2014-10-23 08:55:02.000000000","message":"Cool :)","commit_id":"2616367622ccaa768455de401a0a604721e7afea"},{"author":{"_account_id":3012,"name":"Dina Belova","email":"dbelova@mirantis.com","username":"dbelova"},"change_message_id":"b3a0fb39440cd0155fdd58e0dbd50f5b1522e19c","unresolved":false,"context_lines":[{"line_number":206,"context_line":"Assignee(s)"},{"line_number":207,"context_line":"-----------"},{"line_number":208,"context_line":""},{"line_number":209,"context_line":"Who is leading the writing of the code? Or is this a blueprint where you\u0027re"},{"line_number":210,"context_line":"throwing it out there to see who picks it up?"},{"line_number":211,"context_line":""},{"line_number":212,"context_line":"If more than one person is working on the implementation, please designate the"}],"source_content_type":"text/x-rst","patch_set":5,"id":"9aa7fdbe_df6b2bef","line":209,"updated":"2014-10-23 08:55:02.000000000","message":"Remove lines 209-213","commit_id":"2616367622ccaa768455de401a0a604721e7afea"},{"author":{"_account_id":3012,"name":"Dina Belova","email":"dbelova@mirantis.com","username":"dbelova"},"change_message_id":"b3a0fb39440cd0155fdd58e0dbd50f5b1522e19c","unresolved":false,"context_lines":[{"line_number":216,"context_line":"  chdent"},{"line_number":217,"context_line":""},{"line_number":218,"context_line":"Other contributors:"},{"line_number":219,"context_line":"  Volunteers Welcome"},{"line_number":220,"context_line":""},{"line_number":221,"context_line":"Ongoing maintainer:"},{"line_number":222,"context_line":"  chdent"}],"source_content_type":"text/x-rst","patch_set":5,"id":"9aa7fdbe_7fa5df45","line":219,"updated":"2014-10-23 08:55:02.000000000","message":"Please add me here, I\u0027d love to participate","commit_id":"2616367622ccaa768455de401a0a604721e7afea"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":5,"context_line":" http://creativecommons.org/licenses/by/3.0/legalcode"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":8,"context_line":"Declarative HTTP API Tests"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/ceilometer/+spec/declarative-http-tets"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_28fb1fc2","line":8,"updated":"2014-10-23 11:29:24.000000000","message":"REST API as opposed HTTP?","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":5,"context_line":" http://creativecommons.org/licenses/by/3.0/legalcode"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":8,"context_line":"Declarative HTTP API Tests"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/ceilometer/+spec/declarative-http-tets"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_8e72530f","line":8,"in_reply_to":"9aa7fdbe_28fb1fc2","updated":"2014-10-23 12:37:44.000000000","message":"No, definitely not. The point here is to confirm (and make visible) the HTTP request: headers, urls, status codes, etc. The title is an important aspect of the goal here...\n\nNevermind the fact that vast majority of the OpenStack APIs are not REST, what\u0027s being tested here is not hypermedia, or content-negotiation, or idempotency, or custom media types of any of that stuff. Just requests and responses.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":1669,"name":"Julien Danjou","display_name":"jd","email":"julien@danjou.info","username":"jdanjou"},"change_message_id":"0be9de9b8b0e25c232d4b0a2745b57b2971a8e1b","unresolved":false,"context_lines":[{"line_number":5,"context_line":" http://creativecommons.org/licenses/by/3.0/legalcode"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":8,"context_line":"Declarative HTTP API Tests"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/ceilometer/+spec/declarative-http-tets"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_0b11f05c","line":8,"in_reply_to":"9aa7fdbe_3498d6bf","updated":"2014-10-23 14:36:50.000000000","message":"/me stands with Chris. :p","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"a7b73ece34526d5570c10295d6083757e3996707","unresolved":false,"context_lines":[{"line_number":5,"context_line":" http://creativecommons.org/licenses/by/3.0/legalcode"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":8,"context_line":"Declarative HTTP API Tests"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/ceilometer/+spec/declarative-http-tets"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_3498d6bf","line":8,"in_reply_to":"9aa7fdbe_8e72530f","updated":"2014-10-23 13:45:08.000000000","message":"Not intending to get into any kind of high-REST versus low-REST type discussion, just meaning that we tend to loosely refer to the OpenStack APIs as being \"REST\" (air-quotes optional) as opposed to HTTP. Not a biggie either way.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/ceilometer/+spec/declarative-http-tets"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"This spec proposes a suite of \"grey-box\" tests for the Ceilometer HTTP"},{"line_number":14,"context_line":"API that are driven by one or several human-readable text files that"},{"line_number":15,"context_line":"declare URL requests alongside expected responses. These tests will"},{"line_number":16,"context_line":"provide end to end tests of API endpoints that in addition to confirming"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_0813bb0d","line":13,"updated":"2014-10-23 11:29:24.000000000","message":"Given that these tests will be driven entirely through the public REST API, as opposed to leveraging knowledge of the internals, seems more \"black-box\" than \"grey-box\".","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/ceilometer/+spec/declarative-http-tets"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"This spec proposes a suite of \"grey-box\" tests for the Ceilometer HTTP"},{"line_number":14,"context_line":"API that are driven by one or several human-readable text files that"},{"line_number":15,"context_line":"declare URL requests alongside expected responses. These tests will"},{"line_number":16,"context_line":"provide end to end tests of API endpoints that in addition to confirming"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_0ebae36d","line":13,"in_reply_to":"9aa7fdbe_0813bb0d","updated":"2014-10-23 12:37:44.000000000","message":"I said grey because there will need to be at least some setup of things like persistence and I wanted to leave open the option (if people want it) for out-of-the-http-band verification.\n\nI\u0027m happy to call it black if that makes it easier to consume (the spec).","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"a7b73ece34526d5570c10295d6083757e3996707","unresolved":false,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/ceilometer/+spec/declarative-http-tets"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"This spec proposes a suite of \"grey-box\" tests for the Ceilometer HTTP"},{"line_number":14,"context_line":"API that are driven by one or several human-readable text files that"},{"line_number":15,"context_line":"declare URL requests alongside expected responses. These tests will"},{"line_number":16,"context_line":"provide end to end tests of API endpoints that in addition to confirming"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_54dc7a73","line":13,"in_reply_to":"9aa7fdbe_0ebae36d","updated":"2014-10-23 13:45:08.000000000","message":"Just wanted to be accurate, so if there is potentially some knowledge of the internals required in the setup mechanics, then \"grey-box\" is indeed more accurate. If the mechanism is purely driven by public APIs, then it\u0027s \"black-box\". Seems you want to leave the door open at least to the former, which is fine.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":13,"context_line":"This spec proposes a suite of \"grey-box\" tests for the Ceilometer HTTP"},{"line_number":14,"context_line":"API that are driven by one or several human-readable text files that"},{"line_number":15,"context_line":"declare URL requests alongside expected responses. These tests will"},{"line_number":16,"context_line":"provide end to end tests of API endpoints that in addition to confirming"},{"line_number":17,"context_line":"the functionality of the API will provide a cogent and readable overview"},{"line_number":18,"context_line":"of the HTTP API."},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_08e1fbfd","line":16,"updated":"2014-10-23 11:29:24.000000000","message":"So \"end to end\" in this context implies the API is running against a fully-fledged (non-mocked) storage driver, but only a subset of the supported drivers will be included?\n\n(e.g would it be sufficient to test the API layer by limiting the DB to say mongo only?)\n\nShould you be considering the capabilities API here at all? (i.e. where there isn\u0027t full feature parity across storage drivers)","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":13,"context_line":"This spec proposes a suite of \"grey-box\" tests for the Ceilometer HTTP"},{"line_number":14,"context_line":"API that are driven by one or several human-readable text files that"},{"line_number":15,"context_line":"declare URL requests alongside expected responses. These tests will"},{"line_number":16,"context_line":"provide end to end tests of API endpoints that in addition to confirming"},{"line_number":17,"context_line":"the functionality of the API will provide a cogent and readable overview"},{"line_number":18,"context_line":"of the HTTP API."},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_ae1ab77a","line":16,"in_reply_to":"9aa7fdbe_08e1fbfd","updated":"2014-10-23 12:37:44.000000000","message":"In my initial plan, yes, just one storage driver, for sake of simplicity. If it is good enough I\u0027d choose sqlite as no external service required and it would be nice to be able to run these things in a limited environment.\n\nHowever down in the alternatives I do say that multiple drivers could be an option if that\u0027s what people want. It\u0027s a strange artifact of this review process that on the one hand things are sort of supposed to land fairly fully formed while on the other hand it would be nice to be able to build these ideas consensually. Thus the rather ambiguous language, at least initially.\n\nI don\u0027t know, should we be considering the capabilities API?","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"a7b73ece34526d5570c10295d6083757e3996707","unresolved":false,"context_lines":[{"line_number":13,"context_line":"This spec proposes a suite of \"grey-box\" tests for the Ceilometer HTTP"},{"line_number":14,"context_line":"API that are driven by one or several human-readable text files that"},{"line_number":15,"context_line":"declare URL requests alongside expected responses. These tests will"},{"line_number":16,"context_line":"provide end to end tests of API endpoints that in addition to confirming"},{"line_number":17,"context_line":"the functionality of the API will provide a cogent and readable overview"},{"line_number":18,"context_line":"of the HTTP API."},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_d465cab5","line":16,"in_reply_to":"9aa7fdbe_ae1ab77a","updated":"2014-10-23 13:45:08.000000000","message":"\u003e should we be considering the capabilities API\n\nYes, eventually, once the mechanism is extended to run the tests are to run over multiple storage backends (in which case certain backends may not expose all features in the API, so selective skipping of tests may be required).","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":15,"context_line":"declare URL requests alongside expected responses. These tests will"},{"line_number":16,"context_line":"provide end to end tests of API endpoints that in addition to confirming"},{"line_number":17,"context_line":"the functionality of the API will provide a cogent and readable overview"},{"line_number":18,"context_line":"of the HTTP API."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"The suite of tests augments rather than replaces existing tests"},{"line_number":21,"context_line":"presuming that several small tools, each doing a focused job, are more"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_08b81be4","line":18,"updated":"2014-10-23 11:29:24.000000000","message":"So seems like a dual goal here:\n\n* ensure the API behaves as expected\n* document those expectations in a way that\u0027s consumable by developers only\n\n(i.e. we\u0027re not expecting users or operators to look at these tests for guidance on the API?)","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":15,"context_line":"declare URL requests alongside expected responses. These tests will"},{"line_number":16,"context_line":"provide end to end tests of API endpoints that in addition to confirming"},{"line_number":17,"context_line":"the functionality of the API will provide a cogent and readable overview"},{"line_number":18,"context_line":"of the HTTP API."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"The suite of tests augments rather than replaces existing tests"},{"line_number":21,"context_line":"presuming that several small tools, each doing a focused job, are more"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_b984bf5e","line":18,"in_reply_to":"9aa7fdbe_08b81be4","updated":"2014-10-23 12:37:44.000000000","message":"Yes, consumable by developers, not users or operators. At the moment there is no way to look at the ceilo code and, by that inspection alone, clearly figure out what URLs are available and from there trace and modify implementations. This is a clear (and very unfortunate) limitation in pecan.\n\nThese tests won\u0027t entirely solve that problem, but will at least go some distance to exposing some aspects and hopefully demonstrate the benefits of that exposure.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":22,"context_line":"effective than one job doing everything. In this case the balance of the"},{"line_number":23,"context_line":"focus is on traversing the breadth of the HTTP API to confirm it is in"},{"line_number":24,"context_line":"rude health and demonstrating expected behaviors, not validation of data"},{"line_number":25,"context_line":"provided by the API (unit tests ought to do that)."},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"Transparency over the API ought to lead to better testing and"},{"line_number":28,"context_line":"ultimately better APIs."}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_4894e342","line":25,"updated":"2014-10-23 11:29:24.000000000","message":"But that validation of those data would be done in any case by the expected response matching, or?","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":22,"context_line":"effective than one job doing everything. In this case the balance of the"},{"line_number":23,"context_line":"focus is on traversing the breadth of the HTTP API to confirm it is in"},{"line_number":24,"context_line":"rude health and demonstrating expected behaviors, not validation of data"},{"line_number":25,"context_line":"provided by the API (unit tests ought to do that)."},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"Transparency over the API ought to lead to better testing and"},{"line_number":28,"context_line":"ultimately better APIs."}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_59dcb373","line":25,"in_reply_to":"9aa7fdbe_4894e342","updated":"2014-10-23 12:37:44.000000000","message":"I would think the validation would be partial or at least imperfect. One way to make the tests simple is to have the response validation be simple string matches. This means that e.g. JSON object formats would not be fully deserialized, just matched for expected string contents.\n\nThis is okay as long as the JSON objects are validated in other tests.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":24,"context_line":"rude health and demonstrating expected behaviors, not validation of data"},{"line_number":25,"context_line":"provided by the API (unit tests ought to do that)."},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"Transparency over the API ought to lead to better testing and"},{"line_number":28,"context_line":"ultimately better APIs."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_e85677cd","line":27,"updated":"2014-10-23 11:29:24.000000000","message":"I\u0027m not entirely sure what \"Transparency over the API\" means in this context ... is it that the documentation provided by these tests will make the API easier for developers to understand?","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":24,"context_line":"rude health and demonstrating expected behaviors, not validation of data"},{"line_number":25,"context_line":"provided by the API (unit tests ought to do that)."},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"Transparency over the API ought to lead to better testing and"},{"line_number":28,"context_line":"ultimately better APIs."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_798ed77a","line":27,"in_reply_to":"9aa7fdbe_e85677cd","updated":"2014-10-23 12:37:44.000000000","message":"See my comment above unable to determine URLs by inspection.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":34,"context_line":"encumbered with Python code and traditional unit test infrastructure."},{"line_number":35,"context_line":"These amount to noise when the purpose of a test is both to test *and*"},{"line_number":36,"context_line":"to make visible the testing of the HTTP API. What should be most visible"},{"line_number":37,"context_line":"is HTTP itself."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"The integration testing harness, `Tempest`_, suffers from some of"},{"line_number":40,"context_line":"the same problems as above while also undergoing a refinement in purpose."}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_c896133f","line":37,"updated":"2014-10-23 11:29:24.000000000","message":"\"What should be most visible is *the API* itself\"?","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"a7b73ece34526d5570c10295d6083757e3996707","unresolved":false,"context_lines":[{"line_number":34,"context_line":"encumbered with Python code and traditional unit test infrastructure."},{"line_number":35,"context_line":"These amount to noise when the purpose of a test is both to test *and*"},{"line_number":36,"context_line":"to make visible the testing of the HTTP API. What should be most visible"},{"line_number":37,"context_line":"is HTTP itself."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"The integration testing harness, `Tempest`_, suffers from some of"},{"line_number":40,"context_line":"the same problems as above while also undergoing a refinement in purpose."}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_e854b200","line":37,"in_reply_to":"9aa7fdbe_39744f67","updated":"2014-10-23 13:45:08.000000000","message":"Maybe we\u0027re just spliting hairs here. Yes, the structure of HTTP is highly visible in the way these tests are expressed. But it\u0027s the shape of the API itself (in the sense of the endpoints, actions, query params, request/response JSON payload formats etc.) that would be front-and-centre in my eyes.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":34,"context_line":"encumbered with Python code and traditional unit test infrastructure."},{"line_number":35,"context_line":"These amount to noise when the purpose of a test is both to test *and*"},{"line_number":36,"context_line":"to make visible the testing of the HTTP API. What should be most visible"},{"line_number":37,"context_line":"is HTTP itself."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"The integration testing harness, `Tempest`_, suffers from some of"},{"line_number":40,"context_line":"the same problems as above while also undergoing a refinement in purpose."}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_39744f67","line":37,"in_reply_to":"9aa7fdbe_c896133f","updated":"2014-10-23 12:37:44.000000000","message":"No, the point is making HTTP visible.\n\nVisible HTTP leads to the HTTP being more consumable by random clients. If we wanted to test just the API we could just run the tests with ceilometer client.\n\nThe end game here is slowly but surely driving towards APIs that use HTTP so well that it becomes easy to write lots of different mini clients for the APIs.\n\nThat end is a _very_ long way away, but in the meantime we can gain some better testing.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":39,"context_line":"The integration testing harness, `Tempest`_, suffers from some of"},{"line_number":40,"context_line":"the same problems as above while also undergoing a refinement in purpose."},{"line_number":41,"context_line":"Rather than testing *all* the things, the future Tempest will evolve"},{"line_number":42,"context_line":"to efficiently test the integration of *most* things. In this future"},{"line_number":43,"context_line":"projects which wish to affirm their systems in detail will need to"},{"line_number":44,"context_line":"do that testing themselves."},{"line_number":45,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_88a88b78","line":42,"updated":"2014-10-23 11:29:24.000000000","message":"You might want to add some language here around Tempest focusing on pure \"black-box\" testing, and the issues introduced by branchless Tempest in terms of testing new features.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":39,"context_line":"The integration testing harness, `Tempest`_, suffers from some of"},{"line_number":40,"context_line":"the same problems as above while also undergoing a refinement in purpose."},{"line_number":41,"context_line":"Rather than testing *all* the things, the future Tempest will evolve"},{"line_number":42,"context_line":"to efficiently test the integration of *most* things. In this future"},{"line_number":43,"context_line":"projects which wish to affirm their systems in detail will need to"},{"line_number":44,"context_line":"do that testing themselves."},{"line_number":45,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_b9075fc9","line":42,"in_reply_to":"9aa7fdbe_88a88b78","updated":"2014-10-23 12:37:44.000000000","message":"I\u0027m not sure I understand those issue well enough to articulate them effectively. Can you provide something?","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"a7b73ece34526d5570c10295d6083757e3996707","unresolved":false,"context_lines":[{"line_number":39,"context_line":"The integration testing harness, `Tempest`_, suffers from some of"},{"line_number":40,"context_line":"the same problems as above while also undergoing a refinement in purpose."},{"line_number":41,"context_line":"Rather than testing *all* the things, the future Tempest will evolve"},{"line_number":42,"context_line":"to efficiently test the integration of *most* things. In this future"},{"line_number":43,"context_line":"projects which wish to affirm their systems in detail will need to"},{"line_number":44,"context_line":"do that testing themselves."},{"line_number":45,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_148752d8","line":42,"in_reply_to":"9aa7fdbe_b9075fc9","updated":"2014-10-23 13:45:08.000000000","message":"How about:\n\nTempest also more strictly requires that testcases have no knowledge of the internals, where this proposal would be free of such constraints, which may prove convenient for test setup mechanics.\n\nIn addition, Tempest is now \"branchless\", so that the same test branch is run against stable/{icehouse|juno} and master. This requires that tests for newly testable features either discover the availability of such features dynamically, or are explicitly configured to be skipped depending on the branch Tempest is being run against (neither option being ideal). Whereas this proposal would be free of such complications, as the tests would live in-tree and thus be branched in the same way as the rest of the codebase.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":49,"context_line":"Proposed change"},{"line_number":50,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"To address the shortcomings in HTTP testing (readability and"},{"line_number":53,"context_line":"completeness) a new target for ``tox`` will be defined that runs a"},{"line_number":54,"context_line":"suite of HTTP tests which are expressed as HTTP requests with"},{"line_number":55,"context_line":"expected responses. The exact format should be determined from"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_a8b8efc8","line":52,"updated":"2014-10-23 11:29:24.000000000","message":"s/HTTP testing/API testing/ ?","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":56,"context_line":"discussion as these sorts of things always cause a lot of"},{"line_number":57,"context_line":"contention. Best to get that out of the way early."},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"This same suite of tests will also be registered as a check job to"},{"line_number":60,"context_line":"be run against patches proposed to the code review process."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"In the ideal form the tests will run all the usual HTTP methods"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_c8f53358","line":59,"updated":"2014-10-23 11:29:24.000000000","message":"Both the check and the gate pipelines, surely?","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"a7b73ece34526d5570c10295d6083757e3996707","unresolved":false,"context_lines":[{"line_number":56,"context_line":"discussion as these sorts of things always cause a lot of"},{"line_number":57,"context_line":"contention. Best to get that out of the way early."},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"This same suite of tests will also be registered as a check job to"},{"line_number":60,"context_line":"be run against patches proposed to the code review process."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"In the ideal form the tests will run all the usual HTTP methods"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_94534238","line":59,"in_reply_to":"9aa7fdbe_192a2b2c","updated":"2014-10-23 13:45:08.000000000","message":"I don\u0027t think the idea is run fewer tests in the gate pipeline as compared to the check pipeline, as checks are run far more often (for each update to the changeset, of which there are generally many more than there are pre-merge verification attempts).\n\nRather IIUC the idea is to remove the unecessary co-gating between *all* projects from both pipelines.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":56,"context_line":"discussion as these sorts of things always cause a lot of"},{"line_number":57,"context_line":"contention. Best to get that out of the way early."},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"This same suite of tests will also be registered as a check job to"},{"line_number":60,"context_line":"be run against patches proposed to the code review process."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"In the ideal form the tests will run all the usual HTTP methods"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_192a2b2c","line":59,"in_reply_to":"9aa7fdbe_c8f53358","updated":"2014-10-23 12:37:44.000000000","message":"I\u0027m not clear on that. I\u0027ve gotten the impression (from I\u0027m not sure where) that limiting the amount of tests in gate is a goal. If that\u0027s not the case, then yes, check and gate.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":63,"context_line":"against every URL made available by all the active endpoints"},{"line_number":64,"context_line":"presented over HTTP by Ceilometer. This means that it should cover"},{"line_number":65,"context_line":"both the pending v3 API that will be presented by Gnocchi as well as"},{"line_number":66,"context_line":"the soon to deprecated older APIs."},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"To ensure the tests are lightweight and fast no web server will be used"},{"line_number":69,"context_line":"and only one of whichever lightweight datastore (mongo, sqlite3) for"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_e89d970b","line":66,"updated":"2014-10-23 11:29:24.000000000","message":"\"deprecated older APIs\" won\u0027t include the v1 API though for obvious reasons.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":84,"context_line":"  partial requests."},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"* A list of dictionaries that represent requests and expected"},{"line_number":87,"context_line":"  responses. This is easy to express in a common format like"},{"line_number":88,"context_line":"  YAML, can deal with partial information well, and handle a form of"},{"line_number":89,"context_line":"  inheritance. The author has some `good success`_ with this format"},{"line_number":90,"context_line":"  so tends to prefer it."}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_3397104a","line":87,"updated":"2014-10-23 11:29:24.000000000","message":"Presumably where the expected responses themselves contain dictionaries, the matching logic will be independent of key ordering?","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":84,"context_line":"  partial requests."},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"* A list of dictionaries that represent requests and expected"},{"line_number":87,"context_line":"  responses. This is easy to express in a common format like"},{"line_number":88,"context_line":"  YAML, can deal with partial information well, and handle a form of"},{"line_number":89,"context_line":"  inheritance. The author has some `good success`_ with this format"},{"line_number":90,"context_line":"  so tends to prefer it."}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_f9dba74c","line":87,"in_reply_to":"9aa7fdbe_3397104a","updated":"2014-10-23 12:37:44.000000000","message":"Yes, but as said above I\u0027d prefer not to deserialize the string responses if possible so ordering will not be an issue.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":93,"context_line":""},{"line_number":94,"context_line":"* A test runner that parses YAML files to generate tests which will"},{"line_number":95,"context_line":"  be captured by the testing harness. This runner, besides making"},{"line_number":96,"context_line":"  the HTML requests and processing responses, will be responsible"},{"line_number":97,"context_line":"  for setup: establishing the necessary services and other"},{"line_number":98,"context_line":"  prerequisites of the tests."},{"line_number":99,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_887c0bb7","line":96,"updated":"2014-10-23 11:29:24.000000000","message":"s/HTML requests/API requests/\n\nor\n\ns/HTML requests/HTTP requests/","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":93,"context_line":""},{"line_number":94,"context_line":"* A test runner that parses YAML files to generate tests which will"},{"line_number":95,"context_line":"  be captured by the testing harness. This runner, besides making"},{"line_number":96,"context_line":"  the HTML requests and processing responses, will be responsible"},{"line_number":97,"context_line":"  for setup: establishing the necessary services and other"},{"line_number":98,"context_line":"  prerequisites of the tests."},{"line_number":99,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_19e14bfe","line":96,"in_reply_to":"9aa7fdbe_887c0bb7","updated":"2014-10-23 12:37:44.000000000","message":"Yeah, typo. HTTP.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":94,"context_line":"* A test runner that parses YAML files to generate tests which will"},{"line_number":95,"context_line":"  be captured by the testing harness. This runner, besides making"},{"line_number":96,"context_line":"  the HTML requests and processing responses, will be responsible"},{"line_number":97,"context_line":"  for setup: establishing the necessary services and other"},{"line_number":98,"context_line":"  prerequisites of the tests."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"* A directory including multiple YAML files each representing a"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_28607f00","line":97,"range":{"start_line":97,"start_character":13,"end_line":97,"end_character":48},"updated":"2014-10-23 11:29:24.000000000","message":"\"establishing the necessary services\" \u003d\u003e we won\u0027t be spinning up and extra standalone services, amiright?","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":94,"context_line":"* A test runner that parses YAML files to generate tests which will"},{"line_number":95,"context_line":"  be captured by the testing harness. This runner, besides making"},{"line_number":96,"context_line":"  the HTML requests and processing responses, will be responsible"},{"line_number":97,"context_line":"  for setup: establishing the necessary services and other"},{"line_number":98,"context_line":"  prerequisites of the tests."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"* A directory including multiple YAML files each representing a"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_79efb7e9","line":97,"in_reply_to":"9aa7fdbe_28607f00","updated":"2014-10-23 12:37:44.000000000","message":"Depends on whether we need to spin up mongo or mysql or ____ or not.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"a7b73ece34526d5570c10295d6083757e3996707","unresolved":false,"context_lines":[{"line_number":94,"context_line":"* A test runner that parses YAML files to generate tests which will"},{"line_number":95,"context_line":"  be captured by the testing harness. This runner, besides making"},{"line_number":96,"context_line":"  the HTML requests and processing responses, will be responsible"},{"line_number":97,"context_line":"  for setup: establishing the necessary services and other"},{"line_number":98,"context_line":"  prerequisites of the tests."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"* A directory including multiple YAML files each representing a"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_08a49606","line":97,"in_reply_to":"9aa7fdbe_79efb7e9","updated":"2014-10-23 13:45:08.000000000","message":"A-ha, k ... I was thinking of services in the sense of ceilometer agents and other OpenStack services, as opposed to databases.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":113,"context_line":""},{"line_number":114,"context_line":"* What format for the tests?"},{"line_number":115,"context_line":"* How to handle authN and authZ in a minimal fashion? Ideally keystone"},{"line_number":116,"context_line":"  won\u0027t be running. If that\u0027s the case, what needs to be done?"},{"line_number":117,"context_line":"* How to aggregate request and response groups into single tests?"},{"line_number":118,"context_line":"  One option is for each YAML file to be treated as a single test."},{"line_number":119,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_33747067","line":116,"updated":"2014-10-23 11:29:24.000000000","message":"If wsgi_intercept is calling directly into the wsgi app, could the authZ be effectively faked by setting the X-Roles and X-Project-Id etc. headers directly?\n\n(e.g. to make assertions that the API maintains tenant segregation for non-admin users, but allows global visibility for admin)","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":113,"context_line":""},{"line_number":114,"context_line":"* What format for the tests?"},{"line_number":115,"context_line":"* How to handle authN and authZ in a minimal fashion? Ideally keystone"},{"line_number":116,"context_line":"  won\u0027t be running. If that\u0027s the case, what needs to be done?"},{"line_number":117,"context_line":"* How to aggregate request and response groups into single tests?"},{"line_number":118,"context_line":"  One option is for each YAML file to be treated as a single test."},{"line_number":119,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_99c37b56","line":116,"in_reply_to":"9aa7fdbe_33747067","updated":"2014-10-23 12:37:44.000000000","message":"It depends on where in the wsgi app stack we want to enter. But yes, we can set whatever headers we like.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"a7b73ece34526d5570c10295d6083757e3996707","unresolved":false,"context_lines":[{"line_number":113,"context_line":""},{"line_number":114,"context_line":"* What format for the tests?"},{"line_number":115,"context_line":"* How to handle authN and authZ in a minimal fashion? Ideally keystone"},{"line_number":116,"context_line":"  won\u0027t be running. If that\u0027s the case, what needs to be done?"},{"line_number":117,"context_line":"* How to aggregate request and response groups into single tests?"},{"line_number":118,"context_line":"  One option is for each YAML file to be treated as a single test."},{"line_number":119,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_e83e728b","line":116,"in_reply_to":"9aa7fdbe_99c37b56","updated":"2014-10-23 13:45:08.000000000","message":"Cool, that gives the capability to test the RBAC without involving keystone, even if it is a tad artificial (i.e. in reality X,Auth-Token as opposed to X-Roles/X-Project-Id etc. would not be set by the caller)","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":117,"context_line":"* How to aggregate request and response groups into single tests?"},{"line_number":118,"context_line":"  One option is for each YAML file to be treated as a single test."},{"line_number":119,"context_line":""},{"line_number":120,"context_line":".. _wsgi-intercept: https://pypi.python.org/pypi/wsgi_intercept"},{"line_number":121,"context_line":".. _good success: https://github.com/tiddlyweb/tiddlyweb/blob/master/test/httptest.yaml"},{"line_number":122,"context_line":""},{"line_number":123,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_d3c16455","line":120,"updated":"2014-10-23 11:29:24.000000000","message":"Am I understanding the purpose of wsgi_intercept corrctly, that it;s to effectively bypass \"real\" HTTP and instead call directly into the wsgi app?\n\nIf so, would this really be HTTP testing, or more like API testing?","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":117,"context_line":"* How to aggregate request and response groups into single tests?"},{"line_number":118,"context_line":"  One option is for each YAML file to be treated as a single test."},{"line_number":119,"context_line":""},{"line_number":120,"context_line":".. _wsgi-intercept: https://pypi.python.org/pypi/wsgi_intercept"},{"line_number":121,"context_line":".. _good success: https://github.com/tiddlyweb/tiddlyweb/blob/master/test/httptest.yaml"},{"line_number":122,"context_line":""},{"line_number":123,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_d996e33e","line":120,"in_reply_to":"9aa7fdbe_d3c16455","updated":"2014-10-23 12:37:44.000000000","message":"It depends on what you mean by \"real\".\n\nWhen you run a wsgi app with wsgi-intercept, you still use an HTTP library to make requests to a domain that is hosting the wsgi app. It\u0027s just that wsgi-intercept \"intercepts\" requests to that particular domain and sends the request directly to the app, managing a wsgi.input and a StringIO.\n\nSo you  still do `request.get(\u0027http://some.where/some/thing\u0027)`.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":198,"context_line":"----------------"},{"line_number":199,"context_line":""},{"line_number":200,"context_line":"If a developer adds to or changes the HTTP API those changes will need"},{"line_number":201,"context_line":"to reflected in this test suite."},{"line_number":202,"context_line":""},{"line_number":203,"context_line":"Implementation"},{"line_number":204,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_132a0c2c","line":201,"updated":"2014-10-23 11:29:24.000000000","message":"Would you envisage this declarative approach becoming the primary testing vector for new APIs?\n\n(or would we continue to add new programmatic API tests also for more detailed test scenarios needing more expressive power?)","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":198,"context_line":"----------------"},{"line_number":199,"context_line":""},{"line_number":200,"context_line":"If a developer adds to or changes the HTTP API those changes will need"},{"line_number":201,"context_line":"to reflected in this test suite."},{"line_number":202,"context_line":""},{"line_number":203,"context_line":"Implementation"},{"line_number":204,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_d9bd03b9","line":201,"in_reply_to":"9aa7fdbe_132a0c2c","updated":"2014-10-23 12:37:44.000000000","message":"I think of it as augmenting the existing (so-called) unit tests, not replacing, but I\u0027m not certain we predict how these things fit in until we see what happens.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"a7b73ece34526d5570c10295d6083757e3996707","unresolved":false,"context_lines":[{"line_number":198,"context_line":"----------------"},{"line_number":199,"context_line":""},{"line_number":200,"context_line":"If a developer adds to or changes the HTTP API those changes will need"},{"line_number":201,"context_line":"to reflected in this test suite."},{"line_number":202,"context_line":""},{"line_number":203,"context_line":"Implementation"},{"line_number":204,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_08f2d605","line":201,"in_reply_to":"9aa7fdbe_d9bd03b9","updated":"2014-10-23 13:45:08.000000000","message":"OK, so when a new API is added, we will continue to expect programmatic tests for the more involved test cases, while recommending that the declarative approach is also used for the more direct tests.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"7170ae7e4377d47e740d1aab9ff7454b496e30e4","unresolved":false,"context_lines":[{"line_number":235,"context_line":"time that should be done by the implementor of the feature but there"},{"line_number":236,"context_line":"will be times that larger cleanups are required."},{"line_number":237,"context_line":""},{"line_number":238,"context_line":"Similarly if storage handling becomes a part of the test suite, then"},{"line_number":239,"context_line":"as new storage systems are implemented (or removed), they will need"},{"line_number":240,"context_line":"to be handled."},{"line_number":241,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_d3da444b","line":238,"updated":"2014-10-23 11:29:24.000000000","message":"Storage handling becoming a part of the test suite meaning that the tests are runnable against a selection of, or even all, supported storage drivers?","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"b72942bc3a42cc32b5caa4e6b1d925295bf28e0f","unresolved":false,"context_lines":[{"line_number":235,"context_line":"time that should be done by the implementor of the feature but there"},{"line_number":236,"context_line":"will be times that larger cleanups are required."},{"line_number":237,"context_line":""},{"line_number":238,"context_line":"Similarly if storage handling becomes a part of the test suite, then"},{"line_number":239,"context_line":"as new storage systems are implemented (or removed), they will need"},{"line_number":240,"context_line":"to be handled."},{"line_number":241,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9aa7fdbe_d9e423e3","line":238,"in_reply_to":"9aa7fdbe_d3da444b","updated":"2014-10-23 12:37:44.000000000","message":"Yeah, if the 4th alternative above is chosen.","commit_id":"119447540308b7d897ee20dc45c0e3708ca80a0b"},{"author":{"_account_id":13273,"name":"Lena Novokshonova","email":"enovokshonova@mirantis.com","username":"enovokshonova"},"change_message_id":"14746d7bef23349cbf15e445a61dcf01a91463d9","unresolved":false,"context_lines":[{"line_number":8,"context_line":"Declarative HTTP API Tests"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/ceilometer/+spec/declarative-http-tets"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"This spec proposes a suite of \"grey-box\" tests for the Ceilometer HTTP"},{"line_number":14,"context_line":"API that are driven by one or several human-readable text files that"}],"source_content_type":"text/x-rst","patch_set":8,"id":"9aa7fdbe_6da4f54a","line":11,"range":{"start_line":11,"start_character":0,"end_line":11,"end_character":71},"updated":"2014-10-28 09:21:10.000000000","message":"Is not there a misprint? https://blueprints.launchpad.net/ceilometer/+spec/declarative-http-tets\nor https://blueprints.launchpad.net/ceilometer/+spec/declarative-http-tests?","commit_id":"b6a89343ef63b39d1e20da3170ea098fcbcda627"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"305d043b467a09f59e42663bf0fae89dbd06dbe9","unresolved":false,"context_lines":[{"line_number":8,"context_line":"Declarative HTTP API Tests"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/ceilometer/+spec/declarative-http-tets"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"This spec proposes a suite of \"grey-box\" tests for the Ceilometer HTTP"},{"line_number":14,"context_line":"API that are driven by one or several human-readable text files that"}],"source_content_type":"text/x-rst","patch_set":8,"id":"9aa7fdbe_1ff0eac7","line":11,"in_reply_to":"9aa7fdbe_6da4f54a","updated":"2014-10-28 10:35:59.000000000","message":"Thanks, fixed in next set.","commit_id":"b6a89343ef63b39d1e20da3170ea098fcbcda627"}]}