)]}'
{"specs/juno/fully-async-api.rst":[{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"1727b46c0575c5cd961fa2a9636e9140880b7c20","unresolved":false,"context_lines":[{"line_number":16,"context_line":"conductor and drivers."},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"Problem description"},{"line_number":19,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Today, all of Ironic\u0027s REST API requests block on the RPC layer, because all"},{"line_number":22,"context_line":"RPC methods are invoked as \"call\" rather than \"cast\". Methods currently"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_e6d70fbe","line":19,"updated":"2014-05-27 08:43:08.000000000","message":"To me this section does not actually describe the problem itself, it only describes what we have now. In a previous section you at least write about scalability, but I\u0027m not sure how blocking on API level actually affects scalability, provided that: 1. we\u0027re using asynchronous framework; 2. afaik we can have as many API instances as we want. Could you please expand this section to actually state the very problem(s) with blocking?","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":16,"context_line":"conductor and drivers."},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"Problem description"},{"line_number":19,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Today, all of Ironic\u0027s REST API requests block on the RPC layer, because all"},{"line_number":22,"context_line":"RPC methods are invoked as \"call\" rather than \"cast\". Methods currently"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_60f8ff49","line":19,"in_reply_to":"3ae8d1ca_e6d70fbe","updated":"2014-05-28 21:29:38.000000000","message":"\"provided that we\u0027re using asynchronous framework\"\n\nExactly. We\u0027re *not*.\n\nThe current API is synchronous. Many client requests block until the operation is finished (or at least guaranteed to be started). This results in timeouts, etc... the very opposite of async programming.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"429a7b0701633a77f10c280f39e6d06d22613b09","unresolved":false,"context_lines":[{"line_number":16,"context_line":"conductor and drivers."},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"Problem description"},{"line_number":19,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Today, all of Ironic\u0027s REST API requests block on the RPC layer, because all"},{"line_number":22,"context_line":"RPC methods are invoked as \"call\" rather than \"cast\". Methods currently"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_f4fdc03c","line":19,"in_reply_to":"3ae8d1ca_e6d70fbe","updated":"2014-05-28 15:00:21.000000000","message":"+1\n\nI guess it\u0027s more related to the behaviour of the single API instance, which is blocking after some number of long-running queries is received (basically once all the api greenthreads became busy).\n\nI think it will be clearer to mention that we have too much potential long-running work inside the API greenthreads, which may cause API to hang.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":16,"context_line":"conductor and drivers."},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"Problem description"},{"line_number":19,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Today, all of Ironic\u0027s REST API requests block on the RPC layer, because all"},{"line_number":22,"context_line":"RPC methods are invoked as \"call\" rather than \"cast\". Methods currently"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_208517ad","line":19,"in_reply_to":"3ae8d1ca_f4fdc03c","updated":"2014-05-28 21:29:38.000000000","message":"This actually has nothing to do with the number of API threads.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"429a7b0701633a77f10c280f39e6d06d22613b09","unresolved":false,"context_lines":[{"line_number":23,"context_line":"block on RPC calls for the following reasons:"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"* blocking simply so that the conductor can lock the node prior to updating"},{"line_number":26,"context_line":"  the database;"},{"line_number":27,"context_line":"* blocking on confirmation that a long-lived action (eg, deploy) will be"},{"line_number":28,"context_line":"  started;"},{"line_number":29,"context_line":"* blocking on data transfer, eg. because the API must fetch information held"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_778c5236","line":26,"updated":"2014-05-28 15:00:21.000000000","message":"Sorry, did you mean race condition between conductors? Aren\u0027t we handle this synchronously and throw an appropriate error back to the user?","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"429a7b0701633a77f10c280f39e6d06d22613b09","unresolved":false,"context_lines":[{"line_number":25,"context_line":"* blocking simply so that the conductor can lock the node prior to updating"},{"line_number":26,"context_line":"  the database;"},{"line_number":27,"context_line":"* blocking on confirmation that a long-lived action (eg, deploy) will be"},{"line_number":28,"context_line":"  started;"},{"line_number":29,"context_line":"* blocking on data transfer, eg. because the API must fetch information held"},{"line_number":30,"context_line":"  in the memory state of a conductor, or which the conductor must fetch"},{"line_number":31,"context_line":"  from the physical hardware;"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_97b1dee3","line":28,"updated":"2014-05-28 15:00:21.000000000","message":"e.g. synchronous validation which hang?","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":25,"context_line":"* blocking simply so that the conductor can lock the node prior to updating"},{"line_number":26,"context_line":"  the database;"},{"line_number":27,"context_line":"* blocking on confirmation that a long-lived action (eg, deploy) will be"},{"line_number":28,"context_line":"  started;"},{"line_number":29,"context_line":"* blocking on data transfer, eg. because the API must fetch information held"},{"line_number":30,"context_line":"  in the memory state of a conductor, or which the conductor must fetch"},{"line_number":31,"context_line":"  from the physical hardware;"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_a0c047e1","line":28,"in_reply_to":"3ae8d1ca_97b1dee3","updated":"2014-05-28 21:29:38.000000000","message":"all the code which takes a TaskManager lock prior to starting the work","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"429a7b0701633a77f10c280f39e6d06d22613b09","unresolved":false,"context_lines":[{"line_number":29,"context_line":"* blocking on data transfer, eg. because the API must fetch information held"},{"line_number":30,"context_line":"  in the memory state of a conductor, or which the conductor must fetch"},{"line_number":31,"context_line":"  from the physical hardware;"},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"Proposed change"},{"line_number":34,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":35,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_f7d6a2c2","line":32,"updated":"2014-05-28 15:00:21.000000000","message":"Maybe to add that potential AMQP lags will also reflect on the API responsiveness?","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":29,"context_line":"* blocking on data transfer, eg. because the API must fetch information held"},{"line_number":30,"context_line":"  in the memory state of a conductor, or which the conductor must fetch"},{"line_number":31,"context_line":"  from the physical hardware;"},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"Proposed change"},{"line_number":34,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":35,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_e0488f89","line":32,"in_reply_to":"3ae8d1ca_f7d6a2c2","updated":"2014-05-28 21:29:38.000000000","message":"That is somewhat unavoidable -- the API has to get information from somewhere -- whether another service over AMQP or from the DB. Either way, network latency will affect API responsiveness.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"f4ab07b44a41dde678808dc394826c1a5e1bb2eb","unresolved":false,"context_lines":[{"line_number":33,"context_line":"Proposed change"},{"line_number":34,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"Calls that merely touch the database will ..."},{"line_number":37,"context_line":"XXX: How does nova handle instance-update if n-cond is offline?"},{"line_number":38,"context_line":"This includes the following methods:"},{"line_number":39,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"5adbc538_8adb931a","line":36,"updated":"2014-05-26 22:11:24.000000000","message":"?","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"f4ab07b44a41dde678808dc394826c1a5e1bb2eb","unresolved":false,"context_lines":[{"line_number":34,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"Calls that merely touch the database will ..."},{"line_number":37,"context_line":"XXX: How does nova handle instance-update if n-cond is offline?"},{"line_number":38,"context_line":"This includes the following methods:"},{"line_number":39,"context_line":""},{"line_number":40,"context_line":"* update_node"}],"source_content_type":"text/x-rst","patch_set":5,"id":"5adbc538_6ae4870d","line":37,"updated":"2014-05-26 22:11:24.000000000","message":"Breaks. n-conductor is not like ir-cond.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"f4ab07b44a41dde678808dc394826c1a5e1bb2eb","unresolved":false,"context_lines":[{"line_number":45,"context_line":"Calls that merely wait for confirmation that a conductor received the request"},{"line_number":46,"context_line":"and is capable of starting the requested action will be converted to \"cast\"."},{"line_number":47,"context_line":"Any preconditions which can be checked in the API will be done within the API,"},{"line_number":48,"context_line":"rather than the conductor. The conductor will merely log a NOTICE for"},{"line_number":49,"context_line":"unservicable requests. The conductor will not reject requests due to"},{"line_number":50,"context_line":"NodeLocked, but will be queued up locally and processed in the order received."},{"line_number":51,"context_line":"This includes the following methods:"}],"source_content_type":"text/x-rst","patch_set":5,"id":"5adbc538_8a1ab315","line":48,"updated":"2014-05-26 22:11:24.000000000","message":"Doing it in the API *rather than* the conductor means the conductor may violation preconditions since there is an arbitrary time period between cast and action. I suggest doing it in both places.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":45,"context_line":"Calls that merely wait for confirmation that a conductor received the request"},{"line_number":46,"context_line":"and is capable of starting the requested action will be converted to \"cast\"."},{"line_number":47,"context_line":"Any preconditions which can be checked in the API will be done within the API,"},{"line_number":48,"context_line":"rather than the conductor. The conductor will merely log a NOTICE for"},{"line_number":49,"context_line":"unservicable requests. The conductor will not reject requests due to"},{"line_number":50,"context_line":"NodeLocked, but will be queued up locally and processed in the order received."},{"line_number":51,"context_line":"This includes the following methods:"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_80326b0e","line":48,"in_reply_to":"5adbc538_8a1ab315","updated":"2014-05-28 21:29:38.000000000","message":"good point.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"f4ab07b44a41dde678808dc394826c1a5e1bb2eb","unresolved":false,"context_lines":[{"line_number":46,"context_line":"and is capable of starting the requested action will be converted to \"cast\"."},{"line_number":47,"context_line":"Any preconditions which can be checked in the API will be done within the API,"},{"line_number":48,"context_line":"rather than the conductor. The conductor will merely log a NOTICE for"},{"line_number":49,"context_line":"unservicable requests. The conductor will not reject requests due to"},{"line_number":50,"context_line":"NodeLocked, but will be queued up locally and processed in the order received."},{"line_number":51,"context_line":"This includes the following methods:"},{"line_number":52,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"5adbc538_cad3db36","line":49,"updated":"2014-05-26 22:11:24.000000000","message":"How will users determine that the request was unservicable (and how to correct it?)","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":46,"context_line":"and is capable of starting the requested action will be converted to \"cast\"."},{"line_number":47,"context_line":"Any preconditions which can be checked in the API will be done within the API,"},{"line_number":48,"context_line":"rather than the conductor. The conductor will merely log a NOTICE for"},{"line_number":49,"context_line":"unservicable requests. The conductor will not reject requests due to"},{"line_number":50,"context_line":"NodeLocked, but will be queued up locally and processed in the order received."},{"line_number":51,"context_line":"This includes the following methods:"},{"line_number":52,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_8980d747","line":49,"in_reply_to":"3ae8d1ca_3a9b9173","updated":"2014-05-28 21:29:38.000000000","message":"that\u0027s even more than a scheduler -- a task management system","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":46,"context_line":"and is capable of starting the requested action will be converted to \"cast\"."},{"line_number":47,"context_line":"Any preconditions which can be checked in the API will be done within the API,"},{"line_number":48,"context_line":"rather than the conductor. The conductor will merely log a NOTICE for"},{"line_number":49,"context_line":"unservicable requests. The conductor will not reject requests due to"},{"line_number":50,"context_line":"NodeLocked, but will be queued up locally and processed in the order received."},{"line_number":51,"context_line":"This includes the following methods:"},{"line_number":52,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_49624fae","line":49,"in_reply_to":"5adbc538_cad3db36","updated":"2014-05-28 21:29:38.000000000","message":"If the validation in the API passed, then the request was servicable at that time.\n\nIf subsequently, it failed the same validation after the conductor received it, it\u0027s because some other state changed (eg, node was deleted; or a duplicate request was processed).\n\nShould the user re-issue the request, they will get an error from the API server (eg, \"no such node\" or \"node already deploying\") without having to wait on an RPC call.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"429a7b0701633a77f10c280f39e6d06d22613b09","unresolved":false,"context_lines":[{"line_number":46,"context_line":"and is capable of starting the requested action will be converted to \"cast\"."},{"line_number":47,"context_line":"Any preconditions which can be checked in the API will be done within the API,"},{"line_number":48,"context_line":"rather than the conductor. The conductor will merely log a NOTICE for"},{"line_number":49,"context_line":"unservicable requests. The conductor will not reject requests due to"},{"line_number":50,"context_line":"NodeLocked, but will be queued up locally and processed in the order received."},{"line_number":51,"context_line":"This includes the following methods:"},{"line_number":52,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_3a9b9173","line":49,"in_reply_to":"5adbc538_cad3db36","updated":"2014-05-28 15:00:21.000000000","message":"Probably we will need some REST API resource which can be poked (task?), so all the errors will be stored in a dedicated place instead of single last_error field.\n\nWe\u0027re coming back to the old discussion regarding the third Ironic service - scheduler :)","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"f4ab07b44a41dde678808dc394826c1a5e1bb2eb","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Any preconditions which can be checked in the API will be done within the API,"},{"line_number":48,"context_line":"rather than the conductor. The conductor will merely log a NOTICE for"},{"line_number":49,"context_line":"unservicable requests. The conductor will not reject requests due to"},{"line_number":50,"context_line":"NodeLocked, but will be queued up locally and processed in the order received."},{"line_number":51,"context_line":"This includes the following methods:"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"* change_node_power_state"}],"source_content_type":"text/x-rst","patch_set":5,"id":"5adbc538_0adea33f","line":50,"updated":"2014-05-26 22:11:24.000000000","message":"This means there could be thousands or millions of such queued requests, which seems likely to lead to hard to diagnose (or reproduce) situations. I\u0027m very worried about this.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Any preconditions which can be checked in the API will be done within the API,"},{"line_number":48,"context_line":"rather than the conductor. The conductor will merely log a NOTICE for"},{"line_number":49,"context_line":"unservicable requests. The conductor will not reject requests due to"},{"line_number":50,"context_line":"NodeLocked, but will be queued up locally and processed in the order received."},{"line_number":51,"context_line":"This includes the following methods:"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"* change_node_power_state"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_c921fffd","line":50,"in_reply_to":"3ae8d1ca_3a1c71c1","updated":"2014-05-28 21:29:38.000000000","message":"AMQP already implements / enforces a TTL policy on messages. It feels ilke we\u0027d be reimplementing that ...","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"429a7b0701633a77f10c280f39e6d06d22613b09","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Any preconditions which can be checked in the API will be done within the API,"},{"line_number":48,"context_line":"rather than the conductor. The conductor will merely log a NOTICE for"},{"line_number":49,"context_line":"unservicable requests. The conductor will not reject requests due to"},{"line_number":50,"context_line":"NodeLocked, but will be queued up locally and processed in the order received."},{"line_number":51,"context_line":"This includes the following methods:"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"* change_node_power_state"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_3a1c71c1","line":50,"in_reply_to":"5adbc538_0adea33f","updated":"2014-05-28 15:00:21.000000000","message":"How about to limit the size of the queue? Under normal conditions, the queue of limited size is more convenient than just rejecting all the requests which are not fit into the conductor bandwidth. In this way we will just spread peak loads over the time, but still prevent DDoS (I mean DDoS is still possible, but it won\u0027t crash the Ironic, just will make it busy for some time)","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Any preconditions which can be checked in the API will be done within the API,"},{"line_number":48,"context_line":"rather than the conductor. The conductor will merely log a NOTICE for"},{"line_number":49,"context_line":"unservicable requests. The conductor will not reject requests due to"},{"line_number":50,"context_line":"NodeLocked, but will be queued up locally and processed in the order received."},{"line_number":51,"context_line":"This includes the following methods:"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"* change_node_power_state"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_09a9277b","line":50,"in_reply_to":"5adbc538_0adea33f","updated":"2014-05-28 21:29:38.000000000","message":"adding alternative to spec","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"1727b46c0575c5cd961fa2a9636e9140880b7c20","unresolved":false,"context_lines":[{"line_number":56,"context_line":"* set_console_mode"},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"Calls that require fetching information from a conductor will instead fetch it"},{"line_number":59,"context_line":"from the database. This will require data model changes, and may require new"},{"line_number":60,"context_line":"periodic task(s) to keep the data model in sync, such that the API service can"},{"line_number":61,"context_line":"reliably return the requested information without an RPC call.  This includes"},{"line_number":62,"context_line":"the following methods:"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_06931bfb","line":59,"updated":"2014-05-27 08:43:08.000000000","message":"Thus you\u0027re gonna block on waiting for the response from database, which, depending on the situation, can be just as slow","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":6623,"name":"Roman Prykhodchenko","email":"me@romcheg.me","username":"prykhodchenko"},"change_message_id":"4f3f470656aa8dda822297e8cb0313a6dca27d39","unresolved":false,"context_lines":[{"line_number":56,"context_line":"* set_console_mode"},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"Calls that require fetching information from a conductor will instead fetch it"},{"line_number":59,"context_line":"from the database. This will require data model changes, and may require new"},{"line_number":60,"context_line":"periodic task(s) to keep the data model in sync, such that the API service can"},{"line_number":61,"context_line":"reliably return the requested information without an RPC call.  This includes"},{"line_number":62,"context_line":"the following methods:"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_652bb194","line":59,"in_reply_to":"3ae8d1ca_06931bfb","updated":"2014-05-28 20:27:19.000000000","message":"But that doesn\u0027t lock the node itself, so the other requests like that may be processed simultaneously.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":56,"context_line":"* set_console_mode"},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"Calls that require fetching information from a conductor will instead fetch it"},{"line_number":59,"context_line":"from the database. This will require data model changes, and may require new"},{"line_number":60,"context_line":"periodic task(s) to keep the data model in sync, such that the API service can"},{"line_number":61,"context_line":"reliably return the requested information without an RPC call.  This includes"},{"line_number":62,"context_line":"the following methods:"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_297a63f1","line":59,"in_reply_to":"3ae8d1ca_06931bfb","updated":"2014-05-28 21:29:38.000000000","message":"There\u0027s nothing we can do about that. The data has to be somewhere!","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"429a7b0701633a77f10c280f39e6d06d22613b09","unresolved":false,"context_lines":[{"line_number":59,"context_line":"from the database. This will require data model changes, and may require new"},{"line_number":60,"context_line":"periodic task(s) to keep the data model in sync, such that the API service can"},{"line_number":61,"context_line":"reliably return the requested information without an RPC call.  This includes"},{"line_number":62,"context_line":"the following methods:"},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"* validate_driver_interfaces"},{"line_number":65,"context_line":"* get_console_information"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_9a3ca5e4","line":62,"updated":"2014-05-28 15:00:21.000000000","message":"This may introduce the new field for race conditions. Currently we\u0027re lock the node, validate it and start a background job. There is no room between validation and the job, node is locked all the time.\n\nIn the way proposed here the information in the DB may be stale, especially under heavy load, so all the actions done on the API side may become useless.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":59,"context_line":"from the database. This will require data model changes, and may require new"},{"line_number":60,"context_line":"periodic task(s) to keep the data model in sync, such that the API service can"},{"line_number":61,"context_line":"reliably return the requested information without an RPC call.  This includes"},{"line_number":62,"context_line":"the following methods:"},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"* validate_driver_interfaces"},{"line_number":65,"context_line":"* get_console_information"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_895e1775","line":62,"in_reply_to":"3ae8d1ca_9a3ca5e4","updated":"2014-05-28 21:29:38.000000000","message":"\"all actions on the API side become useless\" -- no, they just queue up and take longer, rather than the HTTP service failing (whcih is what happens today)\n\nAs far as the race condition, we\u0027ll still do the validation on the conductor side. We\u0027ll just \"cast\" the request instead of \"calling\" so the API won\u0027t wait on the lock acquisition and the RPC response","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":6623,"name":"Roman Prykhodchenko","email":"me@romcheg.me","username":"prykhodchenko"},"change_message_id":"4f3f470656aa8dda822297e8cb0313a6dca27d39","unresolved":false,"context_lines":[{"line_number":59,"context_line":"from the database. This will require data model changes, and may require new"},{"line_number":60,"context_line":"periodic task(s) to keep the data model in sync, such that the API service can"},{"line_number":61,"context_line":"reliably return the requested information without an RPC call.  This includes"},{"line_number":62,"context_line":"the following methods:"},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"* validate_driver_interfaces"},{"line_number":65,"context_line":"* get_console_information"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_4056a3d3","line":62,"in_reply_to":"3ae8d1ca_9a3ca5e4","updated":"2014-05-28 20:27:19.000000000","message":"I think it\u0027s possible to perform different kinds of checking: e.g., if someone/something asks the state of a node, Ironic can return a value stored in the DB.\n\nHowever, if Ironic has to have the most accurate information about the node, e.g., when doing a deployment, it is possible to lock the node and ask the conductor to check the state.\n\nThat does not sound like a great design, however, it\u0027ll allow to avoid several scenarios when there are several clients that pull information about nodes periodically but have to wait each other because nodes get locked continuously.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"429a7b0701633a77f10c280f39e6d06d22613b09","unresolved":false,"context_lines":[{"line_number":66,"context_line":""},{"line_number":67,"context_line":"Calls that require the conductor to perform some action in order to determine"},{"line_number":68,"context_line":"the requested information will need a periodic task to keep the data model in"},{"line_number":69,"context_line":"sync. This includes the following methods:"},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"* validation of driver ManagementInterfaces"},{"line_number":72,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_7d78b38f","line":69,"updated":"2014-05-28 15:00:21.000000000","message":"Looks similar to the previous. Maybe to join these parts?","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":66,"context_line":""},{"line_number":67,"context_line":"Calls that require the conductor to perform some action in order to determine"},{"line_number":68,"context_line":"the requested information will need a periodic task to keep the data model in"},{"line_number":69,"context_line":"sync. This includes the following methods:"},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"* validation of driver ManagementInterfaces"},{"line_number":72,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_094a07ac","line":69,"in_reply_to":"3ae8d1ca_7d78b38f","updated":"2014-05-28 21:29:38.000000000","message":"yep, reworded a lot","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"f4ab07b44a41dde678808dc394826c1a5e1bb2eb","unresolved":false,"context_lines":[{"line_number":69,"context_line":"sync. This includes the following methods:"},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"* validation of driver ManagementInterfaces"},{"line_number":72,"context_line":""},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"Alternatives"},{"line_number":75,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"5adbc538_4a3fcb25","line":72,"updated":"2014-05-26 22:11:24.000000000","message":"I found this way of describing things a bit hard to follow. Perhaps it would be easier if the new rules / principles were described, then corner cases, and finally enumerating which calls fall into which rules and corners.\n\ne.g.\n\n* Anything can read from the database without locking at any time.\n* Only the conductor is allowed to write to the database.\n* Conductors must write progress information to the database in a timely manner (e.g. before any potentially blocking step).\n* API requests may only answer queries from the database.\n\nThe silent discard thing -really- worries me- when nova does that it is one of the most frustrating experiences, and you should talk to the nodepool authors about how shitty writing robust code against the nova API is.\n\nI believe the key thing that we need is a clear model for users about when to resubmit a request vs being patient. E.g. \u0027all requests are reliable and you never need to resubmit\u0027 would be one such model. Another is \u0027any request may be dropped at any time and if the request has not been processed within 5 seconds you should resubmit it\u0027 is another. The one this proposal seems to have is \u0027any request may be dropped, or may be serviced an arbitrarily large amount of time later, so you have to assume all requests are still going to be serviced but also resubmit them to be sure.\u0027","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":69,"context_line":"sync. This includes the following methods:"},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"* validation of driver ManagementInterfaces"},{"line_number":72,"context_line":""},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"Alternatives"},{"line_number":75,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_69e9cb7f","line":72,"in_reply_to":"5adbc538_4a3fcb25","updated":"2014-05-28 21:29:38.000000000","message":"at least nova doesn\u0027t fall over if you spam \"nova reboot NNN\" requests :)\n\nI\u0027ll see what I can do about the structure of this section in the rev after next (it\u0027s almost done and I need to go)","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"1727b46c0575c5cd961fa2a9636e9140880b7c20","unresolved":false,"context_lines":[{"line_number":101,"context_line":"Security impact"},{"line_number":102,"context_line":"---------------"},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"There is currently a DoS vulnerability whereby any authenticated user can"},{"line_number":105,"context_line":"issue a sufficient number of requests to:"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"  GET /v1/nodes/NNN/validate"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_59042415","line":104,"updated":"2014-05-27 08:43:08.000000000","message":"I think the proper way of fixing DoS is request throttling.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":101,"context_line":"Security impact"},{"line_number":102,"context_line":"---------------"},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"There is currently a DoS vulnerability whereby any authenticated user can"},{"line_number":105,"context_line":"issue a sufficient number of requests to:"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"  GET /v1/nodes/NNN/validate"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_09f84748","line":104,"in_reply_to":"3ae8d1ca_59042415","updated":"2014-05-28 21:29:38.000000000","message":"The bar to this DoS is *incredibly* low. Like, one per minute could still hit it under certain conditions.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"f4ab07b44a41dde678808dc394826c1a5e1bb2eb","unresolved":false,"context_lines":[{"line_number":118,"context_line":""},{"line_number":119,"context_line":"Both of these are possible even with a relatively low rate of requests."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"This change will resolve this DoS vulnerability."},{"line_number":122,"context_line":""},{"line_number":123,"context_line":"Other end user impact"},{"line_number":124,"context_line":"---------------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"5adbc538_8a9f135f","line":121,"updated":"2014-05-26 22:11:24.000000000","message":"It will add a new DOS where any authenticated user can saturate the AMQP queue. (whereas at the moment they saturate the API listening socket where external load-balancers can detect this, the rabbit queue is not surfaced externally anywhere).","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":118,"context_line":""},{"line_number":119,"context_line":"Both of these are possible even with a relatively low rate of requests."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"This change will resolve this DoS vulnerability."},{"line_number":122,"context_line":""},{"line_number":123,"context_line":"Other end user impact"},{"line_number":124,"context_line":"---------------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_89a3774e","line":121,"in_reply_to":"5adbc538_8a9f135f","updated":"2014-05-28 21:29:38.000000000","message":"That\u0027s much easier to address. AIUI, other services handle this with API rate limiting and a reasonable TTL on AMQP messages.","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"429a7b0701633a77f10c280f39e6d06d22613b09","unresolved":false,"context_lines":[{"line_number":190,"context_line":"References"},{"line_number":191,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":192,"context_line":""},{"line_number":193,"context_line":"XXX: Add references to nova/compute/rpcapi.py and when cast vs call are used."}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_976f1ec5","line":193,"updated":"2014-05-28 15:00:21.000000000","message":"Is there an already existent doc describing this, or you plan to gather this info by reading the code?\n\nIf there is no existent one, we could just provide some statistics, f.e. \"only 5 methods from nova rpcapi use _call_, which is a 3% of the total API methods\"","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"6b268c3d3706cf37f732e6f3abd10c215161c3af","unresolved":false,"context_lines":[{"line_number":190,"context_line":"References"},{"line_number":191,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":192,"context_line":""},{"line_number":193,"context_line":"XXX: Add references to nova/compute/rpcapi.py and when cast vs call are used."}],"source_content_type":"text/x-rst","patch_set":5,"id":"3ae8d1ca_098f67cd","line":193,"in_reply_to":"3ae8d1ca_976f1ec5","updated":"2014-05-28 21:29:38.000000000","message":"no doc AFAIK. I can do some grep\u0027ing...","commit_id":"e55316c8c43ab30a00f8454c2347a61c92f357b7"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"af977df0d7c528e0ef89a45bbd5b2c9e2cdf418d","unresolved":false,"context_lines":[{"line_number":25,"context_line":"Such operations require an RPC response from the assigned conductor service"},{"line_number":26,"context_line":"prior to returning an HTTP response to the client, and the conductor must"},{"line_number":27,"context_line":"acquire a lock on the node before sending an AMQP response back to the API"},{"line_number":28,"context_line":"service.  Therefor API responsiveness is, at best, the sum time it takes for"},{"line_number":29,"context_line":"two AMQP messages and several database calls. This is far from ideal."},{"line_number":30,"context_line":""},{"line_number":31,"context_line":"However, this is frequently resulting in service interruptions in several"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_24ee588c","line":28,"updated":"2014-05-29 23:16:47.000000000","message":"nit: Therefore","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"af977df0d7c528e0ef89a45bbd5b2c9e2cdf418d","unresolved":false,"context_lines":[{"line_number":32,"context_line":"common situations, eg. because the resource in question is already being acted"},{"line_number":33,"context_line":"upon, a conductor takes a long time to respond, a conductor\u0027s worker pool is"},{"line_number":34,"context_line":"full, or a conductor\u0027s RPC listener pool has no available threads and the AMQP"},{"line_number":35,"context_line":"request times out in the queue."},{"line_number":36,"context_line":""},{"line_number":37,"context_line":""},{"line_number":38,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_c4acd438","line":35,"updated":"2014-05-29 23:16:47.000000000","message":"BTW looks like the conductor worker pool can be removed once this lands.  IMHO there is no purpose for it with a fully async API.","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":10626,"name":"Russell Haering","email":"russellhaering@gmail.com","username":"russell_h"},"change_message_id":"057ce4b3326e7feaf5d4c012ba6f6c36987f9511","unresolved":false,"context_lines":[{"line_number":52,"context_line":"Calls that wait for confirmation that a conductor received the request and is"},{"line_number":53,"context_line":"capable of starting the requested action will be converted to \"cast\".  Any"},{"line_number":54,"context_line":"preconditions must be checked in the API service, and checked again in the"},{"line_number":55,"context_line":"conductor after acquiring a lock on the node. The conductor will merely log a"},{"line_number":56,"context_line":"NOTICE for unservicable requests (since they will be the result of requests"},{"line_number":57,"context_line":"which were once valid but are no longer). Should the user re-issue a request"},{"line_number":58,"context_line":"which was unservicable, they should receive an error from the API service"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_2282ac4f","line":55,"updated":"2014-05-30 16:15:24.000000000","message":"My concern here is that it becomes very difficult to introspect the state of the system. If two conflicting requests are made for a single node, one will be silently dropped. API consumers will have to poll for completion of their request, but have no reasonable way of detecting that their request has been dropped.\n\nWe\u0027ve talked about having the API lock the node as a workaround, but avoiding race conditions gets tough.\n\nI propose that instead we use database transactions to disallow conflicting requests. In short, whenever a request comes in that would modify a Node, validate that every {power,provision}_state matches its corresponding target_{power,provision}_state as part of the UPDATE query, and fail the API request if there is a mismatch.\n\nOnce the database has been updated, a cast will be sent to the conductor either telling it specifically what to do as we do today (ie, provision a node), or we could switch to just sending a single type of request which says \"converge node\", and the conductor figures out what to do by looking for any target state that doesn\u0027t match the actual state.\n\nIf we ever want to add support for an eventually-consistent/non-transactional database, this can be done in the dbapi layer using an external locking mechanism.","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"44c0947104212505e2033558c756ef411a249d18","unresolved":false,"context_lines":[{"line_number":53,"context_line":"capable of starting the requested action will be converted to \"cast\".  Any"},{"line_number":54,"context_line":"preconditions must be checked in the API service, and checked again in the"},{"line_number":55,"context_line":"conductor after acquiring a lock on the node. The conductor will merely log a"},{"line_number":56,"context_line":"NOTICE for unservicable requests (since they will be the result of requests"},{"line_number":57,"context_line":"which were once valid but are no longer). Should the user re-issue a request"},{"line_number":58,"context_line":"which was unservicable, they should receive an error from the API service"},{"line_number":59,"context_line":"because it would fail the same validation check that the conductor performed."}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_96303058","line":56,"updated":"2014-05-30 10:38:28.000000000","message":"I rather don\u0027t agree with NOTICE. WSME logs client errors with WARN level, and with NOTICE it maybe hard to debug on a large production system.","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"af977df0d7c528e0ef89a45bbd5b2c9e2cdf418d","unresolved":false,"context_lines":[{"line_number":54,"context_line":"preconditions must be checked in the API service, and checked again in the"},{"line_number":55,"context_line":"conductor after acquiring a lock on the node. The conductor will merely log a"},{"line_number":56,"context_line":"NOTICE for unservicable requests (since they will be the result of requests"},{"line_number":57,"context_line":"which were once valid but are no longer). Should the user re-issue a request"},{"line_number":58,"context_line":"which was unservicable, they should receive an error from the API service"},{"line_number":59,"context_line":"because it would fail the same validation check that the conductor performed."},{"line_number":60,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_cf0f8d2d","line":57,"updated":"2014-05-29 23:16:47.000000000","message":"nit: If the user...?","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"44c0947104212505e2033558c756ef411a249d18","unresolved":false,"context_lines":[{"line_number":56,"context_line":"NOTICE for unservicable requests (since they will be the result of requests"},{"line_number":57,"context_line":"which were once valid but are no longer). Should the user re-issue a request"},{"line_number":58,"context_line":"which was unservicable, they should receive an error from the API service"},{"line_number":59,"context_line":"because it would fail the same validation check that the conductor performed."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"The conductor will no longer fail requests due to NodeLocked; these will be"},{"line_number":62,"context_line":"queued up locally and processed in the order received. Additionally, a"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_d688f8e9","line":59,"updated":"2014-05-30 10:38:28.000000000","message":"I agree with @Max +if won\u0027t work this way if we have a long-lasting action (e.g. deploy) and user issues more actions, that may be incompatible with the ongoing action (e.g. they expect node in POWER ON state, but deploy fails or vice-versa). Any ideas what to do here?","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"af977df0d7c528e0ef89a45bbd5b2c9e2cdf418d","unresolved":false,"context_lines":[{"line_number":56,"context_line":"NOTICE for unservicable requests (since they will be the result of requests"},{"line_number":57,"context_line":"which were once valid but are no longer). Should the user re-issue a request"},{"line_number":58,"context_line":"which was unservicable, they should receive an error from the API service"},{"line_number":59,"context_line":"because it would fail the same validation check that the conductor performed."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"The conductor will no longer fail requests due to NodeLocked; these will be"},{"line_number":62,"context_line":"queued up locally and processed in the order received. Additionally, a"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_6f6e79cb","line":59,"updated":"2014-05-29 23:16:47.000000000","message":"What if validation will fail because of some particular check possible only inside conductor? E.g. API will pass all the time, but conductor, since it has more capabilities, will perform extra checks and fail. Driver validation?\n\nOr the plan is to port all the possible checks to the API + put all the data needed for this checks to DB","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":6773,"name":"Lucas Alvares Gomes","email":"lucasagomes@gmail.com","username":"lucasagomes"},"change_message_id":"5c7b6fe51d40a98e22e68689b7ab14a344cd14f7","unresolved":false,"context_lines":[{"line_number":59,"context_line":"because it would fail the same validation check that the conductor performed."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"The conductor will no longer fail requests due to NodeLocked; these will be"},{"line_number":62,"context_line":"queued up locally and processed in the order received. Additionally, a"},{"line_number":63,"context_line":"conductor should not unlock a node between requests if there are pending"},{"line_number":64,"context_line":"requests already in its local queue for that node. This prevents situations"},{"line_number":65,"context_line":"where the conductor ring rebalances and messages could be processed out of"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_3547b2bd","line":62,"updated":"2014-06-04 15:55:21.000000000","message":"1- Queue size should have a limit I supposed; If the queue is full what\u0027s going to be returned to the client when he request another operation on that node?\n\n2- Will this queue be visible to the client somehow? How the client will know when the operation he requested started and when it\u0027s finished?","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"af977df0d7c528e0ef89a45bbd5b2c9e2cdf418d","unresolved":false,"context_lines":[{"line_number":63,"context_line":"conductor should not unlock a node between requests if there are pending"},{"line_number":64,"context_line":"requests already in its local queue for that node. This prevents situations"},{"line_number":65,"context_line":"where the conductor ring rebalances and messages could be processed out of"},{"line_number":66,"context_line":"order."},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"This includes the following methods:"},{"line_number":69,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_4f8d3d31","line":66,"updated":"2014-05-29 23:16:47.000000000","message":"Not related to this spec, but rebalance will be complicated. E.g. during rebalance will need to ensure that there are no active locks on a node if the node is being moved to another conductor","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"44c0947104212505e2033558c756ef411a249d18","unresolved":false,"context_lines":[{"line_number":82,"context_line":""},{"line_number":83,"context_line":"All HTTP GET requests should be serviceable from the database, and should not"},{"line_number":84,"context_line":"require RPC calls to the conductor. This will require data model changes, and"},{"line_number":85,"context_line":"may require new periodic task(s) to keep the data model in sync, such that the"},{"line_number":86,"context_line":"API service can reliably return the requested information without an RPC call."},{"line_number":87,"context_line":"This includes the following methods:"},{"line_number":88,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_16d8a0ee","line":85,"updated":"2014-05-30 10:38:28.000000000","message":"Could you expand this \"keeping data model in sync\" part? It may become source of problems, when we get to actual code. What exactly are we going to sync?","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":6773,"name":"Lucas Alvares Gomes","email":"lucasagomes@gmail.com","username":"lucasagomes"},"change_message_id":"5c7b6fe51d40a98e22e68689b7ab14a344cd14f7","unresolved":false,"context_lines":[{"line_number":82,"context_line":""},{"line_number":83,"context_line":"All HTTP GET requests should be serviceable from the database, and should not"},{"line_number":84,"context_line":"require RPC calls to the conductor. This will require data model changes, and"},{"line_number":85,"context_line":"may require new periodic task(s) to keep the data model in sync, such that the"},{"line_number":86,"context_line":"API service can reliably return the requested information without an RPC call."},{"line_number":87,"context_line":"This includes the following methods:"},{"line_number":88,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_95516638","line":85,"updated":"2014-06-04 15:55:21.000000000","message":"I\u0027m a bit worried about periodic tasks since they all run in a single greenthread, we might need some work here to make periodic tasks to run in parallel (at least in a dedicated greenthread, but multi process may be the best)","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":6773,"name":"Lucas Alvares Gomes","email":"lucasagomes@gmail.com","username":"lucasagomes"},"change_message_id":"5c7b6fe51d40a98e22e68689b7ab14a344cd14f7","unresolved":false,"context_lines":[{"line_number":88,"context_line":""},{"line_number":89,"context_line":"* validate_driver_interfaces"},{"line_number":90,"context_line":"* get_console_information"},{"line_number":91,"context_line":""},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"Alternatives"},{"line_number":94,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_306a6081","line":91,"updated":"2014-06-04 15:55:21.000000000","message":"maybe not related to this blueprint, but including a callback mechanism to notify other services after the operation request is complete would be great.\n\nWe could offer both, pooling and callback... Let\u0027s say another service want Ironic to notify it when the operation is done so it could pass a ?callback\u003d\u003cURL\u003e flag to the API and the conductor would POST {\u0027request_id\u0027: \u003cid\u003e, ...} to that URL after the operation is complete.","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"44c0947104212505e2033558c756ef411a249d18","unresolved":false,"context_lines":[{"line_number":107,"context_line":"There is an ongoing discussion about using the taskflow library to expose"},{"line_number":108,"context_line":"scheduling of complex state changes. While interesting, and something that has"},{"line_number":109,"context_line":"long been discussed in Ironic, this would require a new major version of the"},{"line_number":110,"context_line":"API, and the creation of new resource types and API endpoints.  This proposal"},{"line_number":111,"context_line":"aims to address the pressing issues without the need for an overhaul of the"},{"line_number":112,"context_line":"REST API."},{"line_number":113,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_36fcbc3e","line":110,"updated":"2014-05-30 10:38:28.000000000","message":"Probably not right now, but I don\u0027t think that introducing request ID\u0027s will require new major version of API. Just one more endpoint for checking request status by ID.","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"8eee286966c20f99724a3a3e6daddf2a160877d2","unresolved":false,"context_lines":[{"line_number":108,"context_line":"scheduling of complex state changes. While interesting, and something that has"},{"line_number":109,"context_line":"long been discussed in Ironic, this would require a new major version of the"},{"line_number":110,"context_line":"API, and the creation of new resource types and API endpoints.  This proposal"},{"line_number":111,"context_line":"aims to address the pressing issues without the need for an overhaul of the"},{"line_number":112,"context_line":"REST API."},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"An alternative to locally queuing requests per node would be to selectively"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_01f5bc82","line":111,"updated":"2014-05-30 10:59:58.000000000","message":"Are these issues that pressing? I mean, we did workarounds for NodeLocked, we can do more... Doesn\u0027t it make sense to think of a proper solution right now, when we\u0027re rarely used in production and not graduated yet?\n\nI am quite doubtful, because from what I learned when working as backend engineer at Skype is that in high-load systems half-baked solution maybe worse than no solution or set of simple workarounds. We also did come to a complex \"taskflow\" internal process after having a few generations of partial solutions.","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":8968,"name":"Max Lobur","email":"max_lobur@outlook.com","username":"max_lobur"},"change_message_id":"af977df0d7c528e0ef89a45bbd5b2c9e2cdf418d","unresolved":false,"context_lines":[{"line_number":117,"context_line":"to which messages are sent for nodes that are mapped to that conductor."},{"line_number":118,"context_line":"It may be possible to create unique consumer for each node, thus keeping"},{"line_number":119,"context_line":"the messages queued in AMQP (rather than locally in the conductor)."},{"line_number":120,"context_line":"This is likely to require changes in the hash ring and/or RPC API."},{"line_number":121,"context_line":""},{"line_number":122,"context_line":""},{"line_number":123,"context_line":"Data model impact"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_0a3163ec","line":120,"updated":"2014-05-29 23:16:47.000000000","message":"Nice Idea. It simplify rebalancing\n\nLet\u0027s say we have a topic per node. Each conductor process will need to know the list of nodes it responsible for (basically a list of topics to consume). The message will be removed from queue once conductor complete the task. \nIf conductor hang before completion -\u003e the ring gets rebalanced -\u003e all the remaining conductors get the new lists of nodes they responsible for -\u003e one of them picks the unfinished task and starts it from again. Looks like a lightweight task manager","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"44c0947104212505e2033558c756ef411a249d18","unresolved":false,"context_lines":[{"line_number":140,"context_line":"Nova driver impact"},{"line_number":141,"context_line":"------------------"},{"line_number":142,"context_line":""},{"line_number":143,"context_line":"This change should have no impact on the Nova driver API."},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"Security impact"},{"line_number":146,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_969af012","line":143,"updated":"2014-05-30 10:38:28.000000000","message":"Ah, remembered one more concern. What will Nova driver do, if we find it\u0027s request unserviceable and throw it away? Not it returns error to Nova, are we going to do timeouts? Or retries?","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"44c0947104212505e2033558c756ef411a249d18","unresolved":false,"context_lines":[{"line_number":162,"context_line":""},{"line_number":163,"context_line":"Both of these are possible even with a relatively low rate of requests."},{"line_number":164,"context_line":""},{"line_number":165,"context_line":"This change will resolve this DoS vulnerability."},{"line_number":166,"context_line":""},{"line_number":167,"context_line":"Other end user impact"},{"line_number":168,"context_line":"---------------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_966830f7","line":165,"updated":"2014-05-30 10:38:28.000000000","message":"Please consider adding here concern about number of requests a user can schedule. With blocking the problem is the same but at least will be directly visible. With async approach new requests will just come and fill the queue, without operators noticing it for some time,","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"44c0947104212505e2033558c756ef411a249d18","unresolved":false,"context_lines":[{"line_number":228,"context_line":"Documentation Impact"},{"line_number":229,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":230,"context_line":""},{"line_number":231,"context_line":"None."},{"line_number":232,"context_line":""},{"line_number":233,"context_line":""},{"line_number":234,"context_line":"References"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_764ac484","line":231,"updated":"2014-05-30 10:38:28.000000000","message":"I think we should clearly explain in user docs how this model works, what to expect in case of failures etc. Actually copy some parts of this spec to the docs.","commit_id":"030673a3e86c36276cae9f4894af4b5173d30d94"},{"author":{"_account_id":7419,"name":"Imre Farkas","email":"farkas.imre@gmail.com","username":"ifarkas"},"change_message_id":"9a047d23e798f943677627179c9f575aef6577eb","unresolved":false,"context_lines":[{"line_number":53,"context_line":"views of the system nodes through database tables. The database model will be"},{"line_number":54,"context_line":"modified to include tables that represent nodes in one of two states: an"},{"line_number":55,"context_line":"*observed* state that represents current state of nodes, and a *requested*"},{"line_number":56,"context_line":"state that represents a change from the current state."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"To eliminate blocking on RPC calls, the API service will be modified to"},{"line_number":59,"context_line":"request a state change on a node without waiting for the conductor to do any"}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_328d6bd4","line":56,"updated":"2014-07-17 13:14:59.000000000","message":"There\u0027s an orthogonal set of hw requests which doesn\u0027t fit nicely to this observed vs requested states model. Here\u0027s a use case as an example:\n\nThe WS-Management API of the Dell DRAC cards supports the following call: DCIM_RAIDService.GetRAIDLevels() - it takes a RAID controller and a set of physical disks as its input and its output is a list the possible of RAID levels for the setup given as the input. That doesn\u0027t involve any state change but it would be useful for the Ironic user to be able to use such feature to determine the supported RAID levels for a given setup, prior creating it.","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":10626,"name":"Russell Haering","email":"russellhaering@gmail.com","username":"russell_h"},"change_message_id":"11e4502e21ee73661d82e65f8aa3bf6cfd6ab144","unresolved":false,"context_lines":[{"line_number":60,"context_line":"actual work on the request. It performs this request by updating values within"},{"line_number":61,"context_line":"the node requested state table, and then making a **cast** call to the"},{"line_number":62,"context_line":"conductor that will trigger a converge of the state tables. This eliminates"},{"line_number":63,"context_line":"the block on long-lived actions."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"The conductor will be modified to have a period task that will update the"},{"line_number":66,"context_line":"*observed* state table. This table can be used to satisfy API GET requests"}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_c1989178","line":63,"updated":"2014-08-01 19:52:05.000000000","message":"Will the API permit changes to the requested state when there are already changes pending?\n\nFor example: if a consumer kicks off a build by setting target_provision_state to \"active\", can the target_provision_state be set to \"deleted\" before the build completes?\n\nIf so, this complicates cases where an API consumer polls for completion of a specific task. For example, if nova-compute is polling, waiting for the node to go active, it must also be modified to recognize a situation where the node will _never_ go active.\n\nIn either case I like this approach.","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"afa4b7fafe862cef891eab60afcfa4e9e3b9d7c9","unresolved":false,"context_lines":[{"line_number":62,"context_line":"conductor that will trigger a converge of the state tables. This eliminates"},{"line_number":63,"context_line":"the block on long-lived actions."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"The conductor will be modified to have a period task that will update the"},{"line_number":66,"context_line":"*observed* state table. This table can be used to satisfy API GET requests"},{"line_number":67,"context_line":"for node detail, and thus eliminate the blocking on data transfer."},{"line_number":68,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_f56db233","line":65,"updated":"2014-07-02 05:48:19.000000000","message":"nit: periodic","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":7419,"name":"Imre Farkas","email":"farkas.imre@gmail.com","username":"ifarkas"},"change_message_id":"9a047d23e798f943677627179c9f575aef6577eb","unresolved":false,"context_lines":[{"line_number":63,"context_line":"the block on long-lived actions."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"The conductor will be modified to have a period task that will update the"},{"line_number":66,"context_line":"*observed* state table. This table can be used to satisfy API GET requests"},{"line_number":67,"context_line":"for node detail, and thus eliminate the blocking on data transfer."},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"The conductor will also perform node state convergence of the new tables."}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_6f84605d","line":66,"updated":"2014-07-17 13:14:59.000000000","message":"It would be good to point out whether the API will report back requested state changes along with the observed state, or only the latter without any indication of a requested state.","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"afa4b7fafe862cef891eab60afcfa4e9e3b9d7c9","unresolved":false,"context_lines":[{"line_number":103,"context_line":"long been discussed in Ironic, this would require a new major version of the"},{"line_number":104,"context_line":"API, and the creation of new resource types and API endpoints.  This proposal"},{"line_number":105,"context_line":"aims to address the pressing issues without the need for an overhaul of the"},{"line_number":106,"context_line":"REST API."},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"An alternative to locally queuing requests per node would be to selectively"},{"line_number":109,"context_line":"remove AMQP requests from the queue when there is no work in progress"}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_15694625","line":106,"updated":"2014-07-02 05:48:19.000000000","message":"this paragraph does not seem accurate any longer, as the two-table-representation model is itself a new API","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"afa4b7fafe862cef891eab60afcfa4e9e3b9d7c9","unresolved":false,"context_lines":[{"line_number":111,"context_line":"to which messages are sent for nodes that are mapped to that conductor."},{"line_number":112,"context_line":"It may be possible to create unique consumer for each node, thus keeping"},{"line_number":113,"context_line":"the messages queued in AMQP (rather than locally in the conductor)."},{"line_number":114,"context_line":"This is likely to require changes in the hash ring and/or RPC API."},{"line_number":115,"context_line":""},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"Data model impact"}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_35780af2","line":114,"updated":"2014-07-02 05:48:19.000000000","message":"I believe this would be fairly easy to implement in code, but would have\nscaling implications as it would drastically increase the number of concurrent\nrabbit queues. Today, a deployment has only as many queues as conductors.\nIt\u0027s worth calling out that implication.","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"5c84d7f031b19986a2ebc16e2da7b7815f923527","unresolved":false,"context_lines":[{"line_number":112,"context_line":"It may be possible to create unique consumer for each node, thus keeping"},{"line_number":113,"context_line":"the messages queued in AMQP (rather than locally in the conductor)."},{"line_number":114,"context_line":"This is likely to require changes in the hash ring and/or RPC API."},{"line_number":115,"context_line":""},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"Data model impact"},{"line_number":118,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_4ecb0953","line":115,"updated":"2014-07-02 23:58:54.000000000","message":"Its actually super hard to remove requests from a queue. Its not a DB.","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":10626,"name":"Russell Haering","email":"russellhaering@gmail.com","username":"russell_h"},"change_message_id":"11e4502e21ee73661d82e65f8aa3bf6cfd6ab144","unresolved":false,"context_lines":[{"line_number":127,"context_line":"table. This table will be used by API service to request operations on the"},{"line_number":128,"context_line":"node as state changes. When states in this table differ from the *observed*"},{"line_number":129,"context_line":"states, work will need to be started to converge the states. Only the API"},{"line_number":130,"context_line":"service can write to this table."},{"line_number":131,"context_line":""},{"line_number":132,"context_line":""},{"line_number":133,"context_line":"REST API impact"}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_e1fcb50f","line":130,"updated":"2014-08-01 19:52:05.000000000","message":"It shouldn\u0027t really be necessary to have two tables for this, especially since there should be a 1:1 relationship between rows in these tables.\n\nThe nodes table already contains:\n\n  1. Observed provision and power states\n  2. Requested provision and power states\n  3. Additional information that can largely be viewed as requested provisioning parameters.\n\nThis already seems mostly sufficient for this proposal. The flow would look something like:\n\n  1. API consumer sets some instance_info which gets written into the node table\n  2. API consumer sets target_provision_state to ACTIVE, which gets written into the node table\n  3. A converge is triggered via a cast (or periodic task if the message is lost), and the conductor does its thing. When done, target_provision_state is unset, and provision_state is updated.\n\nAnd so on, with little to no data model change required.\n\nThoughts?","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"afa4b7fafe862cef891eab60afcfa4e9e3b9d7c9","unresolved":false,"context_lines":[{"line_number":138,"context_line":"currently return 201 (Created) and 204 (No content, but request fulfilled)."},{"line_number":139,"context_line":"These will need to be changed to return 202 (Accepted, not fulfilled)."},{"line_number":140,"context_line":"Because of this, clients will likely need to use additional requests to"},{"line_number":141,"context_line":"verify that their initial request was completed (or failed)."},{"line_number":142,"context_line":""},{"line_number":143,"context_line":"The RPC API will also need a new version to change the call()\u0027s to cast()\u0027s."},{"line_number":144,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_5573be14","line":141,"updated":"2014-07-02 05:48:19.000000000","message":"This could use a more detailed description of the proposed REST API. For instance...\n\nWhat does a state change request look like? What does a metadata change request look like?\nAt what point does the conductor validate changed metadata (eg, changed IPMI creds)? \nWhat if they\u0027re not valid? How is that relayed back to the client?\nMore generally, how will a client identify when their request has been completed?\nHow are errors exposed to the client? What about different types of errors (power fail, deploy fail, etc)?\nDoes the conductor retry convergence in certain cases and not others? If so, how does it indicate that\nit hit an error but is trying again, vs. hit an error and is giving up?","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":10662,"name":"Gopi Krishna S","email":"saripurigopi@gmail.com","username":"saripurigopi"},"change_message_id":"142afb5e18fb686e4b4ff14afa5792003d4cf91a","unresolved":false,"context_lines":[{"line_number":141,"context_line":"verify that their initial request was completed (or failed)."},{"line_number":142,"context_line":""},{"line_number":143,"context_line":"The RPC API will also need a new version to change the call()\u0027s to cast()\u0027s."},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"Driver API impact"},{"line_number":146,"context_line":"-----------------"},{"line_number":147,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_d480624b","line":144,"updated":"2014-08-01 10:12:27.000000000","message":"I\u0027ve submitted blueprint for adding GET support for vendor_passthru APIs keeping sync call behaviour in mind. The review link is - https://review.openstack.org/#/c/108646/4/specs/juno/get-vendor-passthru.rst . This helps external clients/applications with monitoring the baremetal-nodes.\n \nCould you share an example for GET. In case of full-async-api, how can any external client gets response for GET request after converting all call()\u0027s to cast()\u0027s.","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"afa4b7fafe862cef891eab60afcfa4e9e3b9d7c9","unresolved":false,"context_lines":[{"line_number":145,"context_line":"Driver API impact"},{"line_number":146,"context_line":"-----------------"},{"line_number":147,"context_line":""},{"line_number":148,"context_line":"Unsure of driver API impact at this time."},{"line_number":149,"context_line":""},{"line_number":150,"context_line":"Nova driver impact"},{"line_number":151,"context_line":"------------------"}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_7561a238","line":148,"updated":"2014-07-02 05:48:19.000000000","message":"My current suspicion is this won\u0027t have any impact on the driver API.\nAlso, since drivers don\u0027t manipulate the database, they should not be\naffected by the DB schema changes.","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"afa4b7fafe862cef891eab60afcfa4e9e3b9d7c9","unresolved":false,"context_lines":[{"line_number":157,"context_line":"Security impact"},{"line_number":158,"context_line":"---------------"},{"line_number":159,"context_line":""},{"line_number":160,"context_line":"There is currently a DoS vulnerability whereby any authenticated user can"},{"line_number":161,"context_line":"issue a sufficient number of requests to:"},{"line_number":162,"context_line":""},{"line_number":163,"context_line":"  GET /v1/nodes/NNN/validate"}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_9564764a","line":160,"updated":"2014-07-02 05:48:19.000000000","message":"This was fixed in patch 6318ee1d","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"afa4b7fafe862cef891eab60afcfa4e9e3b9d7c9","unresolved":false,"context_lines":[{"line_number":166,"context_line":"node uses the IPMIToolPowerDriver, tickling the BMC. This can have two"},{"line_number":167,"context_line":"negative effects:"},{"line_number":168,"context_line":""},{"line_number":169,"context_line":"* it can starve the ironic-conductor processes out of RPC workers, as the"},{"line_number":170,"context_line":"  workers are waiting for IPMI to complete, causing further REST API requests"},{"line_number":171,"context_line":"  to timeout or fail due to having no available workers on the other side of"},{"line_number":172,"context_line":"  the RPC bus."}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_b55b3a86","line":169,"updated":"2014-07-02 05:48:19.000000000","message":"This issue persists at scale, if there are enough nodes that all the\ngreenthreads are able to be occupied by other work (such as changing power state)","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"afa4b7fafe862cef891eab60afcfa4e9e3b9d7c9","unresolved":false,"context_lines":[{"line_number":170,"context_line":"  workers are waiting for IPMI to complete, causing further REST API requests"},{"line_number":171,"context_line":"  to timeout or fail due to having no available workers on the other side of"},{"line_number":172,"context_line":"  the RPC bus."},{"line_number":173,"context_line":"* it can actually DoS the BMC, causing it to crash."},{"line_number":174,"context_line":""},{"line_number":175,"context_line":"Both of these are possible even with a relatively low rate of requests."},{"line_number":176,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_d55eee75","line":173,"updated":"2014-07-02 05:48:19.000000000","message":"This was fixed with 6318ee1d","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"afa4b7fafe862cef891eab60afcfa4e9e3b9d7c9","unresolved":false,"context_lines":[{"line_number":201,"context_line":"Other deployer impact"},{"line_number":202,"context_line":"---------------------"},{"line_number":203,"context_line":""},{"line_number":204,"context_line":"None."},{"line_number":205,"context_line":""},{"line_number":206,"context_line":"Developer impact"},{"line_number":207,"context_line":"----------------"}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_f554d255","line":204,"updated":"2014-07-02 05:48:19.000000000","message":"Worth calling out the upgrade requirements between versions","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"afa4b7fafe862cef891eab60afcfa4e9e3b9d7c9","unresolved":false,"context_lines":[{"line_number":202,"context_line":"---------------------"},{"line_number":203,"context_line":""},{"line_number":204,"context_line":"None."},{"line_number":205,"context_line":""},{"line_number":206,"context_line":"Developer impact"},{"line_number":207,"context_line":"----------------"},{"line_number":208,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_1550665d","line":205,"updated":"2014-07-02 05:48:19.000000000","message":"and worth calling out the need for python-ironicclient to support the new version\nas well","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"afa4b7fafe862cef891eab60afcfa4e9e3b9d7c9","unresolved":false,"context_lines":[{"line_number":206,"context_line":"Developer impact"},{"line_number":207,"context_line":"----------------"},{"line_number":208,"context_line":""},{"line_number":209,"context_line":"None."},{"line_number":210,"context_line":""},{"line_number":211,"context_line":""},{"line_number":212,"context_line":"Implementation"}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_354f2abf","line":209,"updated":"2014-07-02 05:48:19.000000000","message":"Worth calling out the requirements to maintain two versions of the API\nuntil such time as we can deprecate the /v1/ API (minimum one full cycle)","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"afa4b7fafe862cef891eab60afcfa4e9e3b9d7c9","unresolved":false,"context_lines":[{"line_number":226,"context_line":"Dependencies"},{"line_number":227,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":228,"context_line":""},{"line_number":229,"context_line":"XXX: Is there a dependent relationship between this change and properly"},{"line_number":230,"context_line":"versioning RPC calls? Should we do one before the other?"},{"line_number":231,"context_line":""},{"line_number":232,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_554adeac","line":229,"updated":"2014-07-02 05:48:19.000000000","message":"RPC versioning of the current calls has been completed\n\nI\u0027m not aware of other deps","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"afa4b7fafe862cef891eab60afcfa4e9e3b9d7c9","unresolved":false,"context_lines":[{"line_number":233,"context_line":"Testing"},{"line_number":234,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":235,"context_line":""},{"line_number":236,"context_line":"Only minor changes to unit tests will be necessary, and these are also"},{"line_number":237,"context_line":"sufficient."},{"line_number":238,"context_line":""},{"line_number":239,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"baada198_75c8c222","line":236,"updated":"2014-07-02 05:48:19.000000000","message":"This para is now incorrect. We\u0027ll need to write all new tempest tests","commit_id":"8bf1e0316156c209938221f4ba8fc73c41b1d7b3"}]}