)]}'
{"specs/kilo/cinder-objects.rst":[{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"77d292372e4e83ce2999110e46f5fa20c246b10b","unresolved":false,"context_lines":[{"line_number":13,"context_line":"The goal of this spec is to introduce objects into cinder.  An object is used"},{"line_number":14,"context_line":"to bundle data with methods that can operate on them.  By using objects,"},{"line_number":15,"context_line":"we have a standardized interface to the database, as well as standardize the"},{"line_number":16,"context_line":"data passed between cinder services over RPC.  This separates the code from"},{"line_number":17,"context_line":"the actual database implementation, making it easier for rolling upgrades."},{"line_number":18,"context_line":""},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_3c62b3ad","line":16,"updated":"2014-10-21 23:03:55.000000000","message":"I think its great to standardize the data passed over RPC; I am not sure though why something simpler/existing isn\u0027t satisfactory for that (or why something like protocol buffers wouldn\u0027t work just fine, it has a defined schema, strict enforcement, easily readable format files, optimized code generation from the format files...).","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"2fca0c84f65c34353db4d9f60be7b00b42a4613d","unresolved":false,"context_lines":[{"line_number":13,"context_line":"The goal of this spec is to introduce objects into cinder.  An object is used"},{"line_number":14,"context_line":"to bundle data with methods that can operate on them.  By using objects,"},{"line_number":15,"context_line":"we have a standardized interface to the database, as well as standardize the"},{"line_number":16,"context_line":"data passed between cinder services over RPC.  This separates the code from"},{"line_number":17,"context_line":"the actual database implementation, making it easier for rolling upgrades."},{"line_number":18,"context_line":""},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_3c99d31f","line":16,"in_reply_to":"9aa7fdbe_3c62b3ad","updated":"2014-10-21 23:08:07.000000000","message":"https://developers.google.com/protocol-buffers/docs/pythontutorial","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"4c19f00bc5f3919d090e97f5a4be86fa2cae93e2","unresolved":false,"context_lines":[{"line_number":13,"context_line":"The goal of this spec is to introduce objects into cinder.  An object is used"},{"line_number":14,"context_line":"to bundle data with methods that can operate on them.  By using objects,"},{"line_number":15,"context_line":"we have a standardized interface to the database, as well as standardize the"},{"line_number":16,"context_line":"data passed between cinder services over RPC.  This separates the code from"},{"line_number":17,"context_line":"the actual database implementation, making it easier for rolling upgrades."},{"line_number":18,"context_line":""},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_82109aa9","line":16,"in_reply_to":"9aa7fdbe_3c99d31f","updated":"2014-10-22 01:01:09.000000000","message":"It could be a possibility.  It is not to say that cinder \"objects\" can turn into something that uses protocol-buffers.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"4e6d315479fac9b2f534eb8dacdad15e2098b7ff","unresolved":false,"context_lines":[{"line_number":13,"context_line":"The goal of this spec is to introduce objects into cinder.  An object is used"},{"line_number":14,"context_line":"to bundle data with methods that can operate on them.  By using objects,"},{"line_number":15,"context_line":"we have a standardized interface to the database, as well as standardize the"},{"line_number":16,"context_line":"data passed between cinder services over RPC.  This separates the code from"},{"line_number":17,"context_line":"the actual database implementation, making it easier for rolling upgrades."},{"line_number":18,"context_line":""},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_b616a5ef","line":16,"in_reply_to":"9aa7fdbe_82109aa9","updated":"2014-10-22 17:27:19.000000000","message":"You could use protbuf for the serialization part if you want. Your object serializer implementation can really do that however it likes. I don\u0027t really see any benefit to doing that over JSON for our purposes, though.\n\nI am not a protbuf expert, but I don\u0027t think it is a peer to objects, but rather a peer to JSON.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"9935d0767983b35d3b01ae8936ba9b1f3f948935","unresolved":false,"context_lines":[{"line_number":13,"context_line":"The goal of this spec is to introduce objects into cinder.  An object is used"},{"line_number":14,"context_line":"to bundle data with methods that can operate on them.  By using objects,"},{"line_number":15,"context_line":"we have a standardized interface to the database, as well as standardize the"},{"line_number":16,"context_line":"data passed between cinder services over RPC.  This separates the code from"},{"line_number":17,"context_line":"the actual database implementation, making it easier for rolling upgrades."},{"line_number":18,"context_line":""},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_f1991720","line":16,"in_reply_to":"9aa7fdbe_b616a5ef","updated":"2014-10-22 17:57:12.000000000","message":"Fair enough, I was just mentioning that if all that is wanted is a \u0027standardized interface between services over RPC\u0027 then there exists solutions that are purposelessly made for these kinds of things (and don\u0027t require a full object remoting capability) and have have been battle tested by large companies for a extended period(s) of time.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"d131c8f07afb14d7e4366db0d592635049b0f0f1","unresolved":false,"context_lines":[{"line_number":13,"context_line":"The goal of this spec is to introduce objects into cinder.  An object is used"},{"line_number":14,"context_line":"to bundle data with methods that can operate on them.  By using objects,"},{"line_number":15,"context_line":"we have a standardized interface to the database, as well as standardize the"},{"line_number":16,"context_line":"data passed between cinder services over RPC.  This separates the code from"},{"line_number":17,"context_line":"the actual database implementation, making it easier for rolling upgrades."},{"line_number":18,"context_line":""},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_515f4341","line":16,"in_reply_to":"9aa7fdbe_f1991720","updated":"2014-10-22 18:00:23.000000000","message":"Yep, and if that is all that cinder wants, then that\u0027s cool. I want cinder to get to nova\u0027s (increasing) level of piecewise-upgrade capability, and I think this (or something equivalent) is what is needed.\n\nIf cinder wants to use protbuf for object serialization instead of JSON, I\u0027m excited to see how that plays out.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"3fddf00717ca8d08ea4f92b3a6130e9c2252feea","unresolved":false,"context_lines":[{"line_number":20,"context_line":"Problem description"},{"line_number":21,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"There are a few problems that exists today in cinder:"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"* Cinder converts everything to a primitive before sending it over RPC.  These"},{"line_number":26,"context_line":"  objects are the sqlalchemy objects that come from the database API.  If"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_53368253","line":23,"updated":"2014-10-22 02:41:11.000000000","message":"Another reason for objects is we do not want direct database calls inside driver code or anywhere outside the api.  By using objects, we can use the object methods instead of direct db calls.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"89b8d1a54e3a4c81c8697159e3bcb6867aa6b403","unresolved":false,"context_lines":[{"line_number":20,"context_line":"Problem description"},{"line_number":21,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"There are a few problems that exists today in cinder:"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"* Cinder converts everything to a primitive before sending it over RPC.  These"},{"line_number":26,"context_line":"  objects are the sqlalchemy objects that come from the database API.  If"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_e22161fd","line":23,"in_reply_to":"9aa7fdbe_53368253","updated":"2014-10-22 05:07:53.000000000","message":"Sure, although this starts to feel like an API contract that is trying to be enforced at the object layer instead of the API layer. If drivers are accessing cinder\u0027s database that seems more like a violation of the API contract and should be stomped out via other means. Trying to enforce a API contract at the object layer is more like \u0027late-enforcement\u0027 of the expected API, which seems to slightly be the wrong place for that enforcement to occur (instead I would expect a stricter contract at the API level instead).","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"4e6d315479fac9b2f534eb8dacdad15e2098b7ff","unresolved":false,"context_lines":[{"line_number":20,"context_line":"Problem description"},{"line_number":21,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"There are a few problems that exists today in cinder:"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"* Cinder converts everything to a primitive before sending it over RPC.  These"},{"line_number":26,"context_line":"  objects are the sqlalchemy objects that come from the database API.  If"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_36b9d5e9","line":23,"in_reply_to":"9aa7fdbe_8cba7a21","updated":"2014-10-22 17:27:19.000000000","message":"Whether cinder decides not to allow drivers access to the object model (and thus the database) is clearly a project decision that need not be the same as what others do.\n\nIn Nova, our drivers use the object model and I don\u0027t think I\u0027d see it realistically working any other way.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"36cd500f7ba506177de2fd765639ff3fea1c72e0","unresolved":false,"context_lines":[{"line_number":20,"context_line":"Problem description"},{"line_number":21,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"There are a few problems that exists today in cinder:"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"* Cinder converts everything to a primitive before sending it over RPC.  These"},{"line_number":26,"context_line":"  objects are the sqlalchemy objects that come from the database API.  If"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_8cba7a21","line":23,"in_reply_to":"9aa7fdbe_e22161fd","updated":"2014-10-22 14:24:19.000000000","message":"Sometimes there are exceptions made.  I do not believe you can always enforce it.  \n\nhttps://github.com/openstack/cinder/blob/stable/juno/cinder/volume/drivers/glusterfs.py#L328.  \n\nhttps://github.com/openstack/cinder/blob/stable/juno/cinder/volume/drivers/fujitsu_eternus_dx_iscsi.py#L90\n\nhttps://github.com/openstack/cinder/blob/stable/juno/cinder/volume/drivers/nexenta/nfs.py#L420","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"96f9e75ae04c7dc3ac7b94da0bfe0185722188f6","unresolved":false,"context_lines":[{"line_number":22,"context_line":""},{"line_number":23,"context_line":"There are a few problems that exists today in cinder:"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"* Cinder converts everything to a primitive before sending it over RPC.  These"},{"line_number":26,"context_line":"  objects are the sqlalchemy objects that come from the database API.  If"},{"line_number":27,"context_line":"  the database schema changes, it could break the existing code that uses"},{"line_number":28,"context_line":"  these objects."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_5c421711","line":25,"updated":"2014-10-21 22:50:21.000000000","message":"There is no conductor in Cinder. So hm hm wtf?","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"4c19f00bc5f3919d090e97f5a4be86fa2cae93e2","unresolved":false,"context_lines":[{"line_number":22,"context_line":""},{"line_number":23,"context_line":"There are a few problems that exists today in cinder:"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"* Cinder converts everything to a primitive before sending it over RPC.  These"},{"line_number":26,"context_line":"  objects are the sqlalchemy objects that come from the database API.  If"},{"line_number":27,"context_line":"  the database schema changes, it could break the existing code that uses"},{"line_number":28,"context_line":"  these objects."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_42beb2d5","line":25,"in_reply_to":"9aa7fdbe_5c421711","updated":"2014-10-22 01:01:09.000000000","message":"No conductor in cinder.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"edc6d96a1ed48e21a70bd60f3c325bc7f5772292","unresolved":false,"context_lines":[{"line_number":22,"context_line":""},{"line_number":23,"context_line":"There are a few problems that exists today in cinder:"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"* Cinder converts everything to a primitive before sending it over RPC.  These"},{"line_number":26,"context_line":"  objects are the sqlalchemy objects that come from the database API.  If"},{"line_number":27,"context_line":"  the database schema changes, it could break the existing code that uses"},{"line_number":28,"context_line":"  these objects."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_d59f9eb2","line":25,"in_reply_to":"9aa7fdbe_5c421711","updated":"2014-10-22 18:19:23.000000000","message":"There is no conductor in Cinder, but that doesn\u0027t mean there is no RPC API communication between cinder-api, cinder-scheduler, and cinder-volume endpoints.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"4e6d315479fac9b2f534eb8dacdad15e2098b7ff","unresolved":false,"context_lines":[{"line_number":22,"context_line":""},{"line_number":23,"context_line":"There are a few problems that exists today in cinder:"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"* Cinder converts everything to a primitive before sending it over RPC.  These"},{"line_number":26,"context_line":"  objects are the sqlalchemy objects that come from the database API.  If"},{"line_number":27,"context_line":"  the database schema changes, it could break the existing code that uses"},{"line_number":28,"context_line":"  these objects."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_5696594d","line":25,"in_reply_to":"9aa7fdbe_5c421711","updated":"2014-10-22 17:27:19.000000000","message":"What does conductor have to do with this statement?","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"96f9e75ae04c7dc3ac7b94da0bfe0185722188f6","unresolved":false,"context_lines":[{"line_number":27,"context_line":"  the database schema changes, it could break the existing code that uses"},{"line_number":28,"context_line":"  these objects."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"* sqlalchemy objects do not have a version associated with it.  If the"},{"line_number":31,"context_line":"  database schema changes, the code could be broken since it was expecting"},{"line_number":32,"context_line":"  certain attributes to be present that may no longer be present in the new"},{"line_number":33,"context_line":"  schema."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_bc46a31f","line":30,"updated":"2014-10-21 22:50:21.000000000","message":"It\u0027s the issue of how we are developing code (like hard depending logic to data) which is actually wrong. \n\nVersioning of objects is hiding not solving the issue.\n\n+ If you would like to get versioning why not just adding new column \n\nversion \u003d \u003cIntegagre\u003e\n\nTo sqlalchemy models? \n\nWhy we need ORM on Facade that is on ORM that is on DB, seems like very wired approach","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1207,"name":"Duncan Thomas","email":"duncan.thomas@gmail.com","username":"duncan-thomas"},"change_message_id":"35065d3b3d527d9117ad10086941ee9fe55bea92","unresolved":false,"context_lines":[{"line_number":27,"context_line":"  the database schema changes, it could break the existing code that uses"},{"line_number":28,"context_line":"  these objects."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"* sqlalchemy objects do not have a version associated with it.  If the"},{"line_number":31,"context_line":"  database schema changes, the code could be broken since it was expecting"},{"line_number":32,"context_line":"  certain attributes to be present that may no longer be present in the new"},{"line_number":33,"context_line":"  schema."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_fb3bb699","line":30,"in_reply_to":"9aa7fdbe_bc46a31f","updated":"2014-10-24 19:56:16.000000000","message":"Adding a version field solves nothing - you still need code that can work with the N different versions. This facade gives you a single place to put that code","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"4c19f00bc5f3919d090e97f5a4be86fa2cae93e2","unresolved":false,"context_lines":[{"line_number":27,"context_line":"  the database schema changes, it could break the existing code that uses"},{"line_number":28,"context_line":"  these objects."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"* sqlalchemy objects do not have a version associated with it.  If the"},{"line_number":31,"context_line":"  database schema changes, the code could be broken since it was expecting"},{"line_number":32,"context_line":"  certain attributes to be present that may no longer be present in the new"},{"line_number":33,"context_line":"  schema."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_c2cc6264","line":30,"in_reply_to":"9aa7fdbe_bc46a31f","updated":"2014-10-22 01:01:09.000000000","message":"It is more than just a facade for the db.  It allows us to send data over RPC efficiently.  The object model is self-serializing.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"4e6d315479fac9b2f534eb8dacdad15e2098b7ff","unresolved":false,"context_lines":[{"line_number":27,"context_line":"  the database schema changes, it could break the existing code that uses"},{"line_number":28,"context_line":"  these objects."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"* sqlalchemy objects do not have a version associated with it.  If the"},{"line_number":31,"context_line":"  database schema changes, the code could be broken since it was expecting"},{"line_number":32,"context_line":"  certain attributes to be present that may no longer be present in the new"},{"line_number":33,"context_line":"  schema."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_96c5213e","line":30,"in_reply_to":"9aa7fdbe_c2cc6264","updated":"2014-10-22 17:27:19.000000000","message":"Regardless, unless you make your DB schema purely additive and forward-compatible, you\u0027ll eventually need to _change_ something. Using an object model other than SQLAlchemy (which is necessarily tied to the schema of the database) helps to allow the underpinnings to move without needing to expose that all the way up the stack (or vice versa). Further, when things do need to change and the stack needs to know, attaching a version helps to know when you\u0027re dealing with an older client so you can emulate the older behavior (or fail gracefully).","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"4b7b1c94b7bb746f06af9128f3bdd18aae53f4a8","unresolved":false,"context_lines":[{"line_number":27,"context_line":"  the database schema changes, it could break the existing code that uses"},{"line_number":28,"context_line":"  these objects."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"* sqlalchemy objects do not have a version associated with it.  If the"},{"line_number":31,"context_line":"  database schema changes, the code could be broken since it was expecting"},{"line_number":32,"context_line":"  certain attributes to be present that may no longer be present in the new"},{"line_number":33,"context_line":"  schema."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_565374e1","line":30,"in_reply_to":"9aa7fdbe_fb3bb699","updated":"2014-10-24 19:58:41.000000000","message":"Correct.  Everything depends on how you implement it.  I have to re-write this spec to reflect all this.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"77d292372e4e83ce2999110e46f5fa20c246b10b","unresolved":false,"context_lines":[{"line_number":32,"context_line":"  certain attributes to be present that may no longer be present in the new"},{"line_number":33,"context_line":"  schema."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* sqlalchemy objects attributes do not enforce the type of data that they"},{"line_number":36,"context_line":"  should contain.  For example, a string can be assigned instead of an integer"},{"line_number":37,"context_line":"  from the API layer and it makes it to the database, where mysql"},{"line_number":38,"context_line":"  automatically converts it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_1cd40f1f","line":35,"updated":"2014-10-21 23:03:55.000000000","message":"This just seems like badly done input validation, I\u0027m not sure why objects is needed to get this. Something like https://pypi.python.org/pypi/voluptuous would seem just as applicable and more specific to doing just doing validation (do one thing and only one thing well...).","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":6172,"name":"Boris Pavlovic","email":"boris@pavlovic.me","username":"boris-42"},"change_message_id":"96f9e75ae04c7dc3ac7b94da0bfe0185722188f6","unresolved":false,"context_lines":[{"line_number":32,"context_line":"  certain attributes to be present that may no longer be present in the new"},{"line_number":33,"context_line":"  schema."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* sqlalchemy objects attributes do not enforce the type of data that they"},{"line_number":36,"context_line":"  should contain.  For example, a string can be assigned instead of an integer"},{"line_number":37,"context_line":"  from the API layer and it makes it to the database, where mysql"},{"line_number":38,"context_line":"  automatically converts it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_bc6dc39b","line":35,"updated":"2014-10-21 22:50:21.000000000","message":"Use strict types in mysql and it won\u0027t convert it. \n\nPlus imho I would like to see solution on top of sqlalchmy models I bet it\u0027s quite simple to implement.\n\nLike we can go through models class and get types of fileds, and perform this validation before storing models.. It is quite simple..","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"4c19f00bc5f3919d090e97f5a4be86fa2cae93e2","unresolved":false,"context_lines":[{"line_number":32,"context_line":"  certain attributes to be present that may no longer be present in the new"},{"line_number":33,"context_line":"  schema."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* sqlalchemy objects attributes do not enforce the type of data that they"},{"line_number":36,"context_line":"  should contain.  For example, a string can be assigned instead of an integer"},{"line_number":37,"context_line":"  from the API layer and it makes it to the database, where mysql"},{"line_number":38,"context_line":"  automatically converts it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_825eda74","line":35,"in_reply_to":"9aa7fdbe_1cd40f1f","updated":"2014-10-22 01:01:09.000000000","message":"This needs to work for other databases that could be used, in addition to mysql, e.g. postgreSQL.  Having a layer that is independent of the database backend would help.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"8917975dd84c8f505b75359935de8dd27d6d20b8","unresolved":false,"context_lines":[{"line_number":35,"context_line":"* sqlalchemy objects attributes do not enforce the type of data that they"},{"line_number":36,"context_line":"  should contain.  For example, a string can be assigned instead of an integer"},{"line_number":37,"context_line":"  from the API layer and it makes it to the database, where mysql"},{"line_number":38,"context_line":"  automatically converts it."},{"line_number":39,"context_line":""},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_7c86fb14","line":38,"updated":"2014-10-21 22:57:58.000000000","message":"that\u0027s a mysql bug, wont happen on other databases, and this should be validated for with a custom type.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"77d292372e4e83ce2999110e46f5fa20c246b10b","unresolved":false,"context_lines":[{"line_number":42,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Most of the work on objects has been done by nova and can be reused in cinder."},{"line_number":45,"context_line":"The plan is to copy over the the base object code (i.e. nova/objects/base.py)"},{"line_number":46,"context_line":"and modify it to fit into cinder.  The base object code in nova will give us:"},{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_7ce19bfd","line":45,"updated":"2014-10-21 23:03:55.000000000","message":"Eck, lets not ;)\n\nAt least use https://review.openstack.org/#/c/127532/ when its ready.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"943c2e9fd910abdf3de9b04bfb7397a1d5b221d4","unresolved":false,"context_lines":[{"line_number":42,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Most of the work on objects has been done by nova and can be reused in cinder."},{"line_number":45,"context_line":"The plan is to copy over the the base object code (i.e. nova/objects/base.py)"},{"line_number":46,"context_line":"and modify it to fit into cinder.  The base object code in nova will give us:"},{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_f12f57f4","line":45,"in_reply_to":"9aa7fdbe_22bd2e69","updated":"2014-10-22 17:48:53.000000000","message":"I was just hoping that instead of \u0027copying\u0027 you can just use the new library that will be created. Copying just makes tech debt that needs to be cleaned up by someone at a later time (which seems not so nice to  impose on someone if a library exists in the first place).","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"edc6d96a1ed48e21a70bd60f3c325bc7f5772292","unresolved":false,"context_lines":[{"line_number":42,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Most of the work on objects has been done by nova and can be reused in cinder."},{"line_number":45,"context_line":"The plan is to copy over the the base object code (i.e. nova/objects/base.py)"},{"line_number":46,"context_line":"and modify it to fit into cinder.  The base object code in nova will give us:"},{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_b583fa54","line":45,"in_reply_to":"9aa7fdbe_7ce19bfd","updated":"2014-10-22 18:19:23.000000000","message":"@Josh: ++","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"4c19f00bc5f3919d090e97f5a4be86fa2cae93e2","unresolved":false,"context_lines":[{"line_number":42,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Most of the work on objects has been done by nova and can be reused in cinder."},{"line_number":45,"context_line":"The plan is to copy over the the base object code (i.e. nova/objects/base.py)"},{"line_number":46,"context_line":"and modify it to fit into cinder.  The base object code in nova will give us:"},{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_22bd2e69","line":45,"in_reply_to":"9aa7fdbe_7ce19bfd","updated":"2014-10-22 01:01:09.000000000","message":"It is just a base I can start from.  We can modify it as we evolve it to cinder\u0027s needs.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"93a6df905b985e230a3ac9ef85ae9489fa26b5fe","unresolved":false,"context_lines":[{"line_number":44,"context_line":"Most of the work on objects has been done by nova and can be reused in cinder."},{"line_number":45,"context_line":"The plan is to copy over the the base object code (i.e. nova/objects/base.py)"},{"line_number":46,"context_line":"and modify it to fit into cinder.  The base object code in nova will give us:"},{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_a55e7110","line":47,"updated":"2014-10-22 15:29:29.000000000","message":"sqlalchemy already does attribute validation and tracking of attribute changes.   in the case of attribute changes, this is so far the aspect of nova.objects that disturbs me, in that we are doing this same fairly expensive process twice, except that SQLAlchemy\u0027s is very mature.   where nova.objects goes above and beyond an API serialization layer is where it makes me concerned.  nova.objects could better be tapping into SQLAlchemy\u0027s existing systems for this rather than reinventing it.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"4e6d315479fac9b2f534eb8dacdad15e2098b7ff","unresolved":false,"context_lines":[{"line_number":44,"context_line":"Most of the work on objects has been done by nova and can be reused in cinder."},{"line_number":45,"context_line":"The plan is to copy over the the base object code (i.e. nova/objects/base.py)"},{"line_number":46,"context_line":"and modify it to fit into cinder.  The base object code in nova will give us:"},{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_76699d24","line":47,"in_reply_to":"9aa7fdbe_a55e7110","updated":"2014-10-22 17:27:19.000000000","message":"At least in nova, most of the deployment doesn\u0027t have access to the SQL schema, or even need sqlalchemy installed. Those nodes still need attribute tracking and attribute validation. Since we (of course) wouldn\u0027t want to consult a node over RPC that *does* have such access each time we need to check an attribute, we need a model that can exist everywhere, regardless of access to the database, schema, or even the sqlalchemy library.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"77d292372e4e83ce2999110e46f5fa20c246b10b","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_9c955f42","line":50,"updated":"2014-10-21 23:03:55.000000000","message":"How though does it give you transactional consistency when you are mutating a set of objects that need to be saved in one transaction, is that needed in cinder? That is part of my worry, but de-normalizing the current code into this, transactions become harder to perform (since where now is that transaction, when the object changes? when multiple objects need to change?) Seems like it actually makes transactional consistency harder, maybe that\u0027s not cared about, but it is an issue.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"9355ae3af3a16cdf12c6a2a52253c5d003ba0acf","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_de4edb95","line":50,"in_reply_to":"9aa7fdbe_150e8634","updated":"2014-10-22 20:37:14.000000000","message":"Except cinder-api, cinder-scheduler, cinder-volume are all talking to the database directly right? So if the idea is use objects to allow for live-upgrades of the database schema without breaking running applications then it seems relevant that not all the payloads in cinder come over RPC (but most come from the database directly), in fact if most payloads come from the database then it seems even more relevant than worrying about payloads coming over RPC. If most of cinder was comminicating via RPC then I\u0027d be aligned with the \u0027talking to each other angle\u0027 ;)","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"e8564e2c665df964da9ed4cacb9ff86ae91da33f","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_159e8656","line":50,"in_reply_to":"9aa7fdbe_150e8634","updated":"2014-10-22 18:30:59.000000000","message":"Ok, if that\u0027s one of the major points then don\u0027t things like protocol buffers have documentation and patterns for handling this? For example, some of the nice docs @ https://developers.google.com/protocol-buffers/docs/proto#updating\n\nI am just brainstorming that maybe solutions exist outside of openstack in other opensource projects if all that\u0027s desired is  objects that can handle alterations of payload schema changes. Maybe I\u0027m still confused though (highly possible, hahaha); overall I just don\u0027t want us to fall into the pit of NIH (which we already do enough of) if we can avoid it.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"5e7b2c4f7a123c8d60d6effa653e88a174ec7251","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_15bbe672","line":50,"in_reply_to":"9aa7fdbe_159e8656","updated":"2014-10-22 18:37:30.000000000","message":"I\u0027m actually very familiar with GPB, as I used it to implement the original Drizzle replication stream format:\n\nhttp://www.joinfu.com/2009/10/drizzle-replication-changes-in-api-to-support-group-commit/\n\nThat said, the link you give above simply lists the rules you need to abide by to make backwards-compatible changes to a schema. There are similar rules in nova/objects regarding how to make compatible changes to a nova object.\n\nCould GPB (or Thrift) be used as a replacement for the serialization format in nova/objects and *some* of the boilerplate ob_make_compatible() code that exists in nova/objects? Yes, absolutely. I think experimenting with that in an oslo-versionedobjects lib would be a slick idea. I might even do that myself, since I\u0027m familiar with GPB. ;) But, then again, I did complain to Dan this morning that I feel like I\u0027m drowning and Dan says I commit to too much stuff, and he\u0027s right... :)","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"2e264b0d1188e6a248ef05ee758e6626909054e5","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_98ff0baa","line":50,"in_reply_to":"9aa7fdbe_15bbe672","updated":"2014-10-22 18:58:06.000000000","message":"Don\u0027t drown!","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"e18073bba5166004676cdb9b556c50cdd00eb32d","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_85a08da0","line":50,"in_reply_to":"9aa7fdbe_4ce892e2","updated":"2014-10-22 15:31:40.000000000","message":"I currently don\u0027t have anything in mind to guarantee it :)\n\nI\u0027m just more of noting that when u granularize things down to an object level (for ex, update attribute X of object Y to \u0027z\u0027) that you now make it that much harder to have transactions at a level of operations (operation could be \u0027update_quota\u0027 for example).","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"4e6d315479fac9b2f534eb8dacdad15e2098b7ff","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_96e601b3","line":50,"in_reply_to":"9aa7fdbe_80c07b7d","updated":"2014-10-22 17:27:19.000000000","message":"Have you guys actually looked at any of the nova code? When it makes sense to save a single attribute, we save it. When it makes sense to save multiple attributes in one go, we do it. When it makes sense to make something a synchronous call back to the database without having looked up the object first, we do it.\n\nPerhaps you\u0027re concerned about saving batches of objects in a single transaction? You put your remotable decorator wherever you want the call to be handled by a node with direct access to the database. The whole list or complex object goes over RPC to a node that can act on the database in a single transaction. The implementation can decide if it needs to only update changed fields, all the fields, check what is in the database first, etc.\n\nIn nova, we have coordinated instance updates that affect the instances table, the instance_info_cache table, etc. When we call save() on the instance, the remote client is not the one doing the save of the instance, then the save of the info_cache, then the save of the next thing. It passes the group of things to the node that does the DB work, which does the proper transactionally-safe update.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"0c9944daecfac36ec6a817fe90056096ffe2f2bc","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_80c07b7d","line":50,"in_reply_to":"9aa7fdbe_85a08da0","updated":"2014-10-22 15:53:15.000000000","message":"+1","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"edc6d96a1ed48e21a70bd60f3c325bc7f5772292","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_150e8634","line":50,"in_reply_to":"9aa7fdbe_85a08da0","updated":"2014-10-22 18:19:23.000000000","message":"@Josh/Thang: The point of versioned objects is not to change the way that the database record is written during a transaction, nor is it to change the way that database schemas are migrated. The point of versioned objects is to allow cinder-api, cinder-scheduler and cinder-volume endpoints running different versions of the Cinder codebase to continue to communicate with each other and not break when the payload schema changes.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"e67ef7e0c614c5c0b335a264d8e5be746b3b0159","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_84ffa6a9","line":50,"in_reply_to":"9aa7fdbe_899b3f71","updated":"2014-10-22 21:43:48.000000000","message":"Fair enough, thanks for explaining (good to know).","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"61aac147cb2b142fd89ac82a5983f531df7df58b","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_c23ffd23","line":50,"in_reply_to":"9aa7fdbe_8dafebe2","updated":"2014-10-22 05:03:45.000000000","message":"Put a lock on what? What/how would that thing be locked?","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"0397718af0cd3006986474c32d1d00db614b4018","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_b18f8fcb","line":50,"in_reply_to":"9aa7fdbe_91f28bd0","updated":"2014-10-22 17:56:30.000000000","message":"Nope, that\u0027s not the case. Each object implementation has full control over what happens in each interaction with the database, making it as large or small as they want.\n\nI think you have exactly the same level (grain?) of control over this as you do with a SQLAlchemy model. The difference might be around SQLisms that SA maintains for you. Of course, if you use SA under the covers, that gets handled there, and if you\u0027re not using SQL as a backing store at all, then you aren\u0027t bound by those rules (which is a good thing).","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"943c2e9fd910abdf3de9b04bfb7397a1d5b221d4","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_91f28bd0","line":50,"in_reply_to":"9aa7fdbe_96e601b3","updated":"2014-10-22 17:48:53.000000000","message":"Not exactly what I am saying here,\n\nI was more noting that there is a difference (and that difference should be considered) between transactions around object changes (or many objects) and transactions around operations/function applications. \n\nObjects appear (from my basic understanding, which I\u0027m happy to learn more about if there are docs somewhere) to granularize it so that no longer can you have operational level transactions (for example, create a entry in a table for an instance, update the users quota and do XYZ other mutations; or do none of it at all).","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"4c19f00bc5f3919d090e97f5a4be86fa2cae93e2","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_8dafebe2","line":50,"in_reply_to":"9aa7fdbe_9c955f42","updated":"2014-10-22 01:01:09.000000000","message":"I believe objects are capable of handling changes done within a transaction.  You could put a lock on it, so that no changes can be made as long as the volume is in-flight.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"2937ba3453e0a3360d3f736ab453c33f7b1c0ec5","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_d5515e0f","line":50,"in_reply_to":"9aa7fdbe_b18f8fcb","updated":"2014-10-22 18:06:16.000000000","message":"Interesting.\n\nHas any project used more than a \u0027SQL as a backing store\u0027 yet? I wasn\u0027t aware that this had happened (it *always* keeps on getting talked about though, ha). Is the intention for nova that this separation will help allow that to be possible (someday... in the future?).","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"cf3d03f9f0c9fe5b4e5cf6f864cf4dd3ced17c02","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_be2cb703","line":50,"in_reply_to":"9aa7fdbe_b18f8fcb","updated":"2014-10-22 20:29:38.000000000","message":"So then to gain transactional consistency at a level of \u0027operations\u0027 you are basically having to create a remoteable method that is the \u0027full\u0027 operation?\n\nThis then (at the target) runs the operation I assume with some data that it would have to fetch locally before doing writes? The need for only using local data (read at the start of the transaction with whatever consistency the operation needs) is to avoid writing stale data from other associated tables that were passed to the object (maybe becoming an attribute of that object I suppose) in some prior call (this is where the attributes that are passed along as part of objects to the RPC target concern me, in that what\u0027s to stop those attributes from being out of date, or not relevant anymore during the write part of the transaction? I assume this is handled?).","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"f8d0779b656eb505e0537dba6e4d8c922598d1dd","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_899b3f71","line":50,"in_reply_to":"9aa7fdbe_be2cb703","updated":"2014-10-22 20:59:58.000000000","message":"Just like any operation that talks to the database.. If you pull from the DB, then make some changes and write data back, you have to be careful not to write stale data. Objects doesn\u0027t propose some magic bullet for making this easy, nor does it even try to help. You have a copy of the object (which is probably stale) and you have a list of changed fields. You can write just the changed fields to the database, you can pull the current contents of the database and diff them (and try to avoid the same race again), or whatever you want. That\u0027s not a thing that it tries to solve, so it\u0027s not relevant to the discussion here.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"36cd500f7ba506177de2fd765639ff3fea1c72e0","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_4ce892e2","line":50,"in_reply_to":"9aa7fdbe_c23ffd23","updated":"2014-10-22 14:24:19.000000000","message":"You can put a lock on the volume, maybe with a locked column in the volumes table, or snapshot with a locked column in the snapshots table, etc.  It depends on what the transaction is operating on.\n\nWhat do you have in mind to guarantee transactional consistency?","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"e1bfed8cce788b5cbd9deb1026c9b34064946ffb","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_b59cdac1","line":50,"in_reply_to":"9aa7fdbe_d5515e0f","updated":"2014-10-22 18:08:47.000000000","message":"Ceilometer recommends mongodb - http://docs.openstack.org/developer/ceilometer/install/manual.html","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2a5c59720303ad4971f4f5d77f1051066125ac33","unresolved":false,"context_lines":[{"line_number":47,"context_line":"built-in serialization, attribute validation, implementation to track"},{"line_number":48,"context_line":"attribute changes, backwards-compatible support for dict-based"},{"line_number":49,"context_line":"objects, and methods (that should be extended) to query and modify the"},{"line_number":50,"context_line":"database."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_95e35632","line":50,"in_reply_to":"9aa7fdbe_d5515e0f","updated":"2014-10-22 18:12:46.000000000","message":"Certainly not across nova yet (that we know of). There are a couple of API places where we don\u0027t use the SA ORM for the query, and instead get back raw row-based results and convert those into objects. The RPC client doesn\u0027t know that we\u0027re bypassing that. I didn\u0027t write that code, so I\u0027m not sure what the reasoning was, but I think it was related to having more control over the query to reduce the result set for something specific (it\u0027s one of the nova-network queries if you go hunting).\n\nRAX has also expressed a desire to subclass Instance so that they can reimplement a very few operations in a very mysql-specific wait to achieve certain gains.\n\nHowever, the point is, if someone wanted to come along and implement a NoSQL backend, they should be able to just reimplement a subset of class methods and make that happen (assuming our abstractions aren\u0027t too leaky).","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"edc6d96a1ed48e21a70bd60f3c325bc7f5772292","unresolved":false,"context_lines":[{"line_number":52,"context_line":"By using objects, we can reduce the risk made by database schema changes."},{"line_number":53,"context_line":"Each object will have a major and minor version number associated with it."},{"line_number":54,"context_line":"This makes it easier to identify methods and attributes that should be"},{"line_number":55,"context_line":"associated with it, and reduces the risk of old code breaking new database"},{"line_number":56,"context_line":"schemas and vice versa.  Furthermore, by defining the attributes and data"},{"line_number":57,"context_line":"types an object can have, we can enforce what the object will look like and"},{"line_number":58,"context_line":"validate the attributes being assigned to it, reducing TypeError exceptions"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_550f2e2e","line":55,"updated":"2014-10-22 18:19:23.000000000","message":"technically, it reduces the risk of new code breaking old code, not the other way around :)","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"e5f3feae8480b926848b19513fffbebd8b194773","unresolved":false,"context_lines":[{"line_number":56,"context_line":"schemas and vice versa.  Furthermore, by defining the attributes and data"},{"line_number":57,"context_line":"types an object can have, we can enforce what the object will look like and"},{"line_number":58,"context_line":"validate the attributes being assigned to it, reducing TypeError exceptions"},{"line_number":59,"context_line":"from occurring."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"By using objects, we have a standardized interface to the database, as well as"},{"line_number":62,"context_line":"standardize the data passed between cinder services over RPC.  CRUD (create,"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_bfabf51c","line":59,"updated":"2014-10-21 23:53:04.000000000","message":"Why are we using python again? If you really want this it might just be better to switch to a statically typed language... Feels slightly out of place in python IMHO.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"edc6d96a1ed48e21a70bd60f3c325bc7f5772292","unresolved":false,"context_lines":[{"line_number":56,"context_line":"schemas and vice versa.  Furthermore, by defining the attributes and data"},{"line_number":57,"context_line":"types an object can have, we can enforce what the object will look like and"},{"line_number":58,"context_line":"validate the attributes being assigned to it, reducing TypeError exceptions"},{"line_number":59,"context_line":"from occurring."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"By using objects, we have a standardized interface to the database, as well as"},{"line_number":62,"context_line":"standardize the data passed between cinder services over RPC.  CRUD (create,"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_f56f42c3","line":59,"in_reply_to":"9aa7fdbe_bfabf51c","updated":"2014-10-22 18:19:23.000000000","message":"Yeah, I agree with Josh here. Type validation is not really the point of versioned objects. Instead, the main reason for objects is to allow having endpoints communicate with each successfully even though one side of the communication link may be on an older version of the Cinder code base than the other. This ability allows for less (or no) downtime during upgrades of the Cinder software. It is this -- rolling upgradeability -- that is the primary motivator for nova.objects.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"77d292372e4e83ce2999110e46f5fa20c246b10b","unresolved":false,"context_lines":[{"line_number":58,"context_line":"validate the attributes being assigned to it, reducing TypeError exceptions"},{"line_number":59,"context_line":"from occurring."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"By using objects, we have a standardized interface to the database, as well as"},{"line_number":62,"context_line":"standardize the data passed between cinder services over RPC.  CRUD (create,"},{"line_number":63,"context_line":"read, update, and delete) operations that were normally kicked off by a call"},{"line_number":64,"context_line":"to cinder/db/api.py can now be kicked off by directly by a call to the object"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_1c896f26","line":61,"updated":"2014-10-21 23:03:55.000000000","message":"Sounds an awfully lot like a ORM that is now not the sqlalchemy ORM ;)\n\nThis also confuses me, if this is just for the database layer, why not improve the sqlalchemy ORM? I can understand the niceness for using it with RPC, but if its primarily just a new ORM over the database then it seems like we already have one of those...","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"4c19f00bc5f3919d090e97f5a4be86fa2cae93e2","unresolved":false,"context_lines":[{"line_number":58,"context_line":"validate the attributes being assigned to it, reducing TypeError exceptions"},{"line_number":59,"context_line":"from occurring."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"By using objects, we have a standardized interface to the database, as well as"},{"line_number":62,"context_line":"standardize the data passed between cinder services over RPC.  CRUD (create,"},{"line_number":63,"context_line":"read, update, and delete) operations that were normally kicked off by a call"},{"line_number":64,"context_line":"to cinder/db/api.py can now be kicked off by directly by a call to the object"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_2d30bf9d","line":61,"in_reply_to":"9aa7fdbe_1c896f26","updated":"2014-10-22 01:01:09.000000000","message":"Maybe the wording can be changed, but objects are meant to group together related data, e.g. volumes, metadata, and consistency group, and provide rich functionality for those data.\n\nThis is done nice in nova IMHO, and I believe it could be done nicely in cinder.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"4e6d315479fac9b2f534eb8dacdad15e2098b7ff","unresolved":false,"context_lines":[{"line_number":58,"context_line":"validate the attributes being assigned to it, reducing TypeError exceptions"},{"line_number":59,"context_line":"from occurring."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"By using objects, we have a standardized interface to the database, as well as"},{"line_number":62,"context_line":"standardize the data passed between cinder services over RPC.  CRUD (create,"},{"line_number":63,"context_line":"read, update, and delete) operations that were normally kicked off by a call"},{"line_number":64,"context_line":"to cinder/db/api.py can now be kicked off by directly by a call to the object"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_b6f8654a","line":61,"in_reply_to":"9aa7fdbe_2d30bf9d","updated":"2014-10-22 17:27:19.000000000","message":"In nova, most of the deployment doesn\u0027t use SQLalchemy, have access to the database, etc.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"943c2e9fd910abdf3de9b04bfb7397a1d5b221d4","unresolved":false,"context_lines":[{"line_number":58,"context_line":"validate the attributes being assigned to it, reducing TypeError exceptions"},{"line_number":59,"context_line":"from occurring."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"By using objects, we have a standardized interface to the database, as well as"},{"line_number":62,"context_line":"standardize the data passed between cinder services over RPC.  CRUD (create,"},{"line_number":63,"context_line":"read, update, and delete) operations that were normally kicked off by a call"},{"line_number":64,"context_line":"to cinder/db/api.py can now be kicked off by directly by a call to the object"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_71c6074b","line":61,"in_reply_to":"9aa7fdbe_b6f8654a","updated":"2014-10-22 17:48:53.000000000","message":"Cinder though afaik doesn\u0027t have this same restriction that nova has, needing isolation away from hypervisors due to security \u0027issues\u0027 if a hypervisor ever gets owned, so the deployment and reasoning for having that isolation I suspect is different in the cinder case.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"93a6df905b985e230a3ac9ef85ae9489fa26b5fe","unresolved":false,"context_lines":[{"line_number":74,"context_line":"The following is the skeleton of the base object class, taken from"},{"line_number":75,"context_line":"nova/objects/base.py.  More details can be found under"},{"line_number":76,"context_line":"https://github.com/openstack/nova/blob/master/nova/objects/base.py ::"},{"line_number":77,"context_line":""},{"line_number":78,"context_line":"    class CinderObject(object):"},{"line_number":79,"context_line":"        \"\"\"Base class and object factory.\"\"\""},{"line_number":80,"context_line":"        VERSION \u003d \u00271.0\u0027"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_05281d33","line":77,"updated":"2014-10-22 15:29:29.000000000","message":"this is the core of nova.object\u0027s issue in that it leads to inefficient database patterns.  the \"object\" wraps itself over API methods which means to deal with multiple objects simultaneously means you have to deal with multiple requests, connections and database transactions simultaneously.    it leads me to suspect that the API of nova.objects is too fine-grained and may be a major source of nova\u0027s inefficiencies.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"edc6d96a1ed48e21a70bd60f3c325bc7f5772292","unresolved":false,"context_lines":[{"line_number":74,"context_line":"The following is the skeleton of the base object class, taken from"},{"line_number":75,"context_line":"nova/objects/base.py.  More details can be found under"},{"line_number":76,"context_line":"https://github.com/openstack/nova/blob/master/nova/objects/base.py ::"},{"line_number":77,"context_line":""},{"line_number":78,"context_line":"    class CinderObject(object):"},{"line_number":79,"context_line":"        \"\"\"Base class and object factory.\"\"\""},{"line_number":80,"context_line":"        VERSION \u003d \u00271.0\u0027"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_b5d13a59","line":77,"in_reply_to":"9aa7fdbe_05281d33","updated":"2014-10-22 18:19:23.000000000","message":"@Michael: Well, while I agree with you that the DB session implementation in Nova is *far* from ideal -- and I support your \"EngineFacade is not a Facade\" efforts completely -- I also think that is a separate concern here.\n\nI actually don\u0027t recommend taking nova.objects as-is -- see my above support for Josh\u0027s idea of using oslo-versioned-objects and getting that library clean and usable.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"4e6d315479fac9b2f534eb8dacdad15e2098b7ff","unresolved":false,"context_lines":[{"line_number":74,"context_line":"The following is the skeleton of the base object class, taken from"},{"line_number":75,"context_line":"nova/objects/base.py.  More details can be found under"},{"line_number":76,"context_line":"https://github.com/openstack/nova/blob/master/nova/objects/base.py ::"},{"line_number":77,"context_line":""},{"line_number":78,"context_line":"    class CinderObject(object):"},{"line_number":79,"context_line":"        \"\"\"Base class and object factory.\"\"\""},{"line_number":80,"context_line":"        VERSION \u003d \u00271.0\u0027"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_56b1f98d","line":77,"in_reply_to":"9aa7fdbe_05281d33","updated":"2014-10-22 17:27:19.000000000","message":"Poor development practices lead to inefficient database patterns. There\u0027s nothing about the objects API that means you have to do things inefficiently. Since the objects API is strikingly similar to the SQLAlchemy ORM\u0027s API, I don\u0027t see how this can be the case :)","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"379c76f32c6b5d8fa7d9292a598e8e3ac636ece8","unresolved":false,"context_lines":[{"line_number":74,"context_line":"The following is the skeleton of the base object class, taken from"},{"line_number":75,"context_line":"nova/objects/base.py.  More details can be found under"},{"line_number":76,"context_line":"https://github.com/openstack/nova/blob/master/nova/objects/base.py ::"},{"line_number":77,"context_line":""},{"line_number":78,"context_line":"    class CinderObject(object):"},{"line_number":79,"context_line":"        \"\"\"Base class and object factory.\"\"\""},{"line_number":80,"context_line":"        VERSION \u003d \u00271.0\u0027"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_c84fee69","line":77,"in_reply_to":"9aa7fdbe_29dc13e4","updated":"2014-10-23 13:37:36.000000000","message":"@Mike, almost 100% of what you see in nova.db.sqlalchemy.api existed before nova.objects was a thing. The objects work has done very little to it, other that drive some change to get consistency, and things like allowing us to specify only the things we want to avoid joining tables we don\u0027t need each time.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"b852a1b843283900bc7113b2b814c7186142ff00","unresolved":false,"context_lines":[{"line_number":74,"context_line":"The following is the skeleton of the base object class, taken from"},{"line_number":75,"context_line":"nova/objects/base.py.  More details can be found under"},{"line_number":76,"context_line":"https://github.com/openstack/nova/blob/master/nova/objects/base.py ::"},{"line_number":77,"context_line":""},{"line_number":78,"context_line":"    class CinderObject(object):"},{"line_number":79,"context_line":"        \"\"\"Base class and object factory.\"\"\""},{"line_number":80,"context_line":"        VERSION \u003d \u00271.0\u0027"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_d106d3cd","line":77,"in_reply_to":"9aa7fdbe_31aabfd4","updated":"2014-10-22 17:35:32.000000000","message":"I dunno exactly what you mean by \"everything\" here. But, the only things that are remoted over RPC are calls specifically marked as remotable. That means I can query an instance object, make a bunch of changes to it, do some other things, pass it to another service over rpc, make more changes, and then call save() on that object. The save() is the *only* thing that goes back to the database node, which includes a copy of the instance (and all contents), and the list of changed fields. The specific implementation of save() (as defined in that object) in this case can do whatever it wants in terms of making sure the things from that list of changes get saved to the database in a single transaction, or whatever else.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"4827f591631f9ea6c202ff5d9946f05fd70be4d1","unresolved":false,"context_lines":[{"line_number":74,"context_line":"The following is the skeleton of the base object class, taken from"},{"line_number":75,"context_line":"nova/objects/base.py.  More details can be found under"},{"line_number":76,"context_line":"https://github.com/openstack/nova/blob/master/nova/objects/base.py ::"},{"line_number":77,"context_line":""},{"line_number":78,"context_line":"    class CinderObject(object):"},{"line_number":79,"context_line":"        \"\"\"Base class and object factory.\"\"\""},{"line_number":80,"context_line":"        VERSION \u003d \u00271.0\u0027"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_31aabfd4","line":77,"in_reply_to":"9aa7fdbe_56b1f98d","updated":"2014-10-22 17:30:39.000000000","message":"well that\u0027s not an apt comparison, the ORM is talking to a live database transaction.   From what I can tell, a single nova object is proxying everything over the API layer into an independent database transaction.     I was careful however to say \"I suspect\" since I dont know nova.objects well enough to know how exactly I\u0027d take multiple nova.objects and operate on them at once within a single database transaction.  If it handles that, then that\u0027s better.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"b45435ddd851cbe34c6aefa7eb2fd5f890246feb","unresolved":false,"context_lines":[{"line_number":74,"context_line":"The following is the skeleton of the base object class, taken from"},{"line_number":75,"context_line":"nova/objects/base.py.  More details can be found under"},{"line_number":76,"context_line":"https://github.com/openstack/nova/blob/master/nova/objects/base.py ::"},{"line_number":77,"context_line":""},{"line_number":78,"context_line":"    class CinderObject(object):"},{"line_number":79,"context_line":"        \"\"\"Base class and object factory.\"\"\""},{"line_number":80,"context_line":"        VERSION \u003d \u00271.0\u0027"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_29dc13e4","line":77,"in_reply_to":"9aa7fdbe_b5d13a59","updated":"2014-10-22 21:16:13.000000000","message":"@Jay, see you seem to be striking a middle ground here, \"nova objects is fine, but we really need to clean it up first\".   I can\u0027t speak to either since I really haven\u0027t begun the process of getting into it.  I\u0027m still sitting at this vague level where in nova//sqlalchemy/api.py, I see lots of functions that do very small things within their own transaction and are public facing.  As yet I haven\u0027t taken the time to understand the bigger picture for why this is, but I keep getting this perception from others, \"it\u0027s nova.objects, it turns what should be one simple operation into 50\".  Hence here you see me a little nervous about it, but please note I am not -1/+1ing this at all, I haven\u0027t done the work to evaluate nova.objects.  This is intentional because at the moment it would mentally distract me from the immediate DB issues I need to fix in openstack and I\u0027m trying to stay focused.   Though I really think waiting for an oslo.objects library would be very appropriate here, copying libraries between projects never helps anyone.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"77d292372e4e83ce2999110e46f5fa20c246b10b","unresolved":false,"context_lines":[{"line_number":787,"context_line":"However, there will continue to be direct database calls, and data passed"},{"line_number":788,"context_line":"between cinder service continue to be primitives.  We would still have to be"},{"line_number":789,"context_line":"careful to validate data types and accepted values that can go into the"},{"line_number":790,"context_line":"database."},{"line_number":791,"context_line":""},{"line_number":792,"context_line":"Data model impact"},{"line_number":793,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_3c2f13f8","line":790,"updated":"2014-10-21 23:03:55.000000000","message":"I would go farther, there is another option, potentially simpler.\n\nProvide a stevedore loadable module/s that can provide the database functions that are used. This module will have somewhat stable set of functions that it will expose. Those functions can then internally know how to convert from a newer version of the database objects to an older one.\n\nFor example the API it could expose could be the following:\n\n  def get_db_api(desired_version):\n     converter \u003d get_converter(desired_version, my_version)\n     return api_implementation(converter)\n\nThen this converter object is responsible for knowing how to translate to the desired version (which could be release N - 1) and the current release (N). Since this exposed API would be a stevedore loadable module a newer module could be installed before upgrading the database, the running application would then start using it (it would have to ask stevedore for the \u0027latest\u0027 compatible version) and then the database would be upgraded, then the application can be upgraded as desired (rolled out or other). Now the only code that is affected is this tiny little stevedore findable module/s, and not a full RPC solution...\n\nIsn\u0027t the above simpler (?) and doesn\u0027t create a new ORM model that now has to be maintained on-top of the sqlalchemy ORM model that already exists...","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"edc6d96a1ed48e21a70bd60f3c325bc7f5772292","unresolved":false,"context_lines":[{"line_number":787,"context_line":"However, there will continue to be direct database calls, and data passed"},{"line_number":788,"context_line":"between cinder service continue to be primitives.  We would still have to be"},{"line_number":789,"context_line":"careful to validate data types and accepted values that can go into the"},{"line_number":790,"context_line":"database."},{"line_number":791,"context_line":""},{"line_number":792,"context_line":"Data model impact"},{"line_number":793,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_f5274262","line":790,"in_reply_to":"9aa7fdbe_3c2f13f8","updated":"2014-10-22 18:19:23.000000000","message":"@Josh: I think you are missing the point of the nova.objects work. It is not at all about migrating the database schema. It is all about the RPC API, and allowing rolling upgrades of distributed endpoints while keeping the system up and running.","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"e8564e2c665df964da9ed4cacb9ff86ae91da33f","unresolved":false,"context_lines":[{"line_number":787,"context_line":"However, there will continue to be direct database calls, and data passed"},{"line_number":788,"context_line":"between cinder service continue to be primitives.  We would still have to be"},{"line_number":789,"context_line":"careful to validate data types and accepted values that can go into the"},{"line_number":790,"context_line":"database."},{"line_number":791,"context_line":""},{"line_number":792,"context_line":"Data model impact"},{"line_number":793,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9aa7fdbe_155f6673","line":790,"in_reply_to":"9aa7fdbe_f5274262","updated":"2014-10-22 18:30:59.000000000","message":"Quite possibly missing the point :)\n\nThanks for helping clarify :)","commit_id":"854c2b6c0e5a721dff465e3a483a55572bd7f486"},{"author":{"_account_id":1207,"name":"Duncan Thomas","email":"duncan.thomas@gmail.com","username":"duncan-thomas"},"change_message_id":"34121dc3c4d252df40b15c9c5ec5c17d66698d45","unresolved":false,"context_lines":[{"line_number":27,"context_line":"* There are multiple places where database calls are made in cinder, and this"},{"line_number":28,"context_line":"  could result in race conditions."},{"line_number":29,"context_line":"* The database implementation is directly tied to SQLAlchemy.  This limits"},{"line_number":30,"context_line":"  support for NoSQL databases."},{"line_number":31,"context_line":""},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a961159_c7414464","line":30,"updated":"2015-01-08 15:50:04.000000000","message":"The tight binding of db schema to code is far far the biggest problem problem being solved here. I\u0027m not seeing anything below that helps with races.","commit_id":"1e113d0334bd0db1c3a5f61b5fbc18491ecee7a9"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"a4e4bc84c6960399f35203d59b9820bb7b2a9406","unresolved":false,"context_lines":[{"line_number":27,"context_line":"* There are multiple places where database calls are made in cinder, and this"},{"line_number":28,"context_line":"  could result in race conditions."},{"line_number":29,"context_line":"* The database implementation is directly tied to SQLAlchemy.  This limits"},{"line_number":30,"context_line":"  support for NoSQL databases."},{"line_number":31,"context_line":""},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a961159_b23554f8","line":30,"in_reply_to":"3a961159_c7414464","updated":"2015-01-08 16:17:02.000000000","message":"I will update the spec to emphasize schema independence more.  As far as races are concerned, by implementing \"expected_state\" in the save method of each object, the state of the object can be checked that it is in the correct state before it is saved, or gracefully fail.  It might help with race conditions, but it may not totally eliminate it.  It really depends on how it is written and implemented, but I will note it.","commit_id":"1e113d0334bd0db1c3a5f61b5fbc18491ecee7a9"},{"author":{"_account_id":1207,"name":"Duncan Thomas","email":"duncan.thomas@gmail.com","username":"duncan-thomas"},"change_message_id":"34121dc3c4d252df40b15c9c5ec5c17d66698d45","unresolved":false,"context_lines":[{"line_number":60,"context_line":"  additional arguments (e.g. expected_volume_state and expected_task_state)"},{"line_number":61,"context_line":"  would be included in the object save method, which would check against the"},{"line_number":62,"context_line":"  in-database copy of the volume, snapshot, backup, consistency group, or"},{"line_number":63,"context_line":"  quota before updates are made and fail gracefully or appropriateky sync up"},{"line_number":64,"context_line":"  the data."},{"line_number":65,"context_line":""},{"line_number":66,"context_line":"The work will be done incrementally in stages, as follows."}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a961159_c73864f0","line":63,"updated":"2015-01-08 15:50:04.000000000","message":"typo","commit_id":"1e113d0334bd0db1c3a5f61b5fbc18491ecee7a9"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"a4e4bc84c6960399f35203d59b9820bb7b2a9406","unresolved":false,"context_lines":[{"line_number":60,"context_line":"  additional arguments (e.g. expected_volume_state and expected_task_state)"},{"line_number":61,"context_line":"  would be included in the object save method, which would check against the"},{"line_number":62,"context_line":"  in-database copy of the volume, snapshot, backup, consistency group, or"},{"line_number":63,"context_line":"  quota before updates are made and fail gracefully or appropriateky sync up"},{"line_number":64,"context_line":"  the data."},{"line_number":65,"context_line":""},{"line_number":66,"context_line":"The work will be done incrementally in stages, as follows."}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a961159_d23820f0","line":63,"in_reply_to":"3a961159_c73864f0","updated":"2015-01-08 16:17:02.000000000","message":"Done","commit_id":"1e113d0334bd0db1c3a5f61b5fbc18491ecee7a9"},{"author":{"_account_id":1207,"name":"Duncan Thomas","email":"duncan.thomas@gmail.com","username":"duncan-thomas"},"change_message_id":"34121dc3c4d252df40b15c9c5ec5c17d66698d45","unresolved":false,"context_lines":[{"line_number":82,"context_line":"would help with rolling upgrades and improve the stability of cinder."},{"line_number":83,"context_line":"However, you would still take an additional database hit when the volume,"},{"line_number":84,"context_line":"snapshot, backup, or consistency group ids are used to query the sqlalchemy"},{"line_number":85,"context_line":"objects.  Also, cinder would continue to be tightly tied to SQLAlchemy."},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"Data model impact"},{"line_number":88,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a961159_12146842","line":85,"updated":"2015-01-08 15:50:04.000000000","message":"Given the number of possible races, the db load is not likely to be significantly reduced, unless we move state out of the db into something more light weight (part of the proposal for the micro-state work)","commit_id":"1e113d0334bd0db1c3a5f61b5fbc18491ecee7a9"},{"author":{"_account_id":1207,"name":"Duncan Thomas","email":"duncan.thomas@gmail.com","username":"duncan-thomas"},"change_message_id":"34121dc3c4d252df40b15c9c5ec5c17d66698d45","unresolved":false,"context_lines":[{"line_number":87,"context_line":"Data model impact"},{"line_number":88,"context_line":"-----------------"},{"line_number":89,"context_line":""},{"line_number":90,"context_line":"None.  The objects being introduced are not stored in the database.  Instead,"},{"line_number":91,"context_line":"these objects are a replacement for sqlalchemy objects that is being used to"},{"line_number":92,"context_line":"represent service, volume, volume type, snapshot, quota, backup, consistency"},{"line_number":93,"context_line":"group and consistency group snapshot throughout cinder internals."}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a961159_b22ab47b","line":90,"updated":"2015-01-08 15:50:04.000000000","message":"not *directly* stored in the db. Their contents is still very much stored, there\u0027s just better abstraction.","commit_id":"1e113d0334bd0db1c3a5f61b5fbc18491ecee7a9"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"a4e4bc84c6960399f35203d59b9820bb7b2a9406","unresolved":false,"context_lines":[{"line_number":87,"context_line":"Data model impact"},{"line_number":88,"context_line":"-----------------"},{"line_number":89,"context_line":""},{"line_number":90,"context_line":"None.  The objects being introduced are not stored in the database.  Instead,"},{"line_number":91,"context_line":"these objects are a replacement for sqlalchemy objects that is being used to"},{"line_number":92,"context_line":"represent service, volume, volume type, snapshot, quota, backup, consistency"},{"line_number":93,"context_line":"group and consistency group snapshot throughout cinder internals."}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a961159_72276cc7","line":90,"in_reply_to":"3a961159_b22ab47b","updated":"2015-01-08 16:17:02.000000000","message":"I will update the spec to say it is an abstraction layer.","commit_id":"1e113d0334bd0db1c3a5f61b5fbc18491ecee7a9"},{"author":{"_account_id":1207,"name":"Duncan Thomas","email":"duncan.thomas@gmail.com","username":"duncan-thomas"},"change_message_id":"34121dc3c4d252df40b15c9c5ec5c17d66698d45","unresolved":false,"context_lines":[{"line_number":115,"context_line":"Performance Impact"},{"line_number":116,"context_line":"------------------"},{"line_number":117,"context_line":""},{"line_number":118,"context_line":"None"},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"Other deployer impact"},{"line_number":121,"context_line":"---------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a961159_b251d40f","line":118,"updated":"2015-01-08 15:50:04.000000000","message":"Unknown. The extra martialling is likely to be a performance hit compared to direct sqlalchemy, hopefully not a large one.","commit_id":"1e113d0334bd0db1c3a5f61b5fbc18491ecee7a9"},{"author":{"_account_id":8247,"name":"Thang Pham","email":"thang.g.pham@gmail.com","username":"thang.pham"},"change_message_id":"a4e4bc84c6960399f35203d59b9820bb7b2a9406","unresolved":false,"context_lines":[{"line_number":115,"context_line":"Performance Impact"},{"line_number":116,"context_line":"------------------"},{"line_number":117,"context_line":""},{"line_number":118,"context_line":"None"},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"Other deployer impact"},{"line_number":121,"context_line":"---------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a961159_3208842c","line":118,"in_reply_to":"3a961159_b251d40f","updated":"2015-01-08 16:17:02.000000000","message":"Probably, because of the transformation of the SQLAlchemy object into cinder object.","commit_id":"1e113d0334bd0db1c3a5f61b5fbc18491ecee7a9"}]}
