)]}'
{"specs/newton/approved/image-cache-verification.rst":[{"author":{"_account_id":9555,"name":"Matthew Booth","email":"mbooth@redhat.com","username":"MatthewBooth"},"change_message_id":"8bd4400d2c1b7e23f5b7d63fd18e8d551c494bfb","unresolved":false,"context_lines":[{"line_number":24,"context_line":"Currently, Nova\u0027s integrity verification for cached images is limited to the"},{"line_number":25,"context_line":"verification of a SHA-1 hash. Since the hash is stored in the same filesystem"},{"line_number":26,"context_line":"as the cached disk image by default, this integrity check provides little"},{"line_number":27,"context_line":"protection in the case where a malicious user has access to the image cache. A"},{"line_number":28,"context_line":"malicious user may overwrite a cached image with an image containing malware,"},{"line_number":29,"context_line":"and then circumvent Nova\u0027s integrity checking mechanism by overwriting the"},{"line_number":30,"context_line":"disk image info file so that it contains a SHA-1 hash matching the corrupted"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_89f5eacc","line":27,"updated":"2016-05-24 14:06:02.000000000","message":"If a malicious user has access to the image cache we\u0027ve got bigger problems. How did they get it?","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":15524,"name":"dane-fichter","email":"dane.fichter@jhuapl.edu","username":"dane-fichter"},"change_message_id":"3e3a98dfc02a49d20b4efb5b8ea8fb4beb28e0cf","unresolved":false,"context_lines":[{"line_number":24,"context_line":"Currently, Nova\u0027s integrity verification for cached images is limited to the"},{"line_number":25,"context_line":"verification of a SHA-1 hash. Since the hash is stored in the same filesystem"},{"line_number":26,"context_line":"as the cached disk image by default, this integrity check provides little"},{"line_number":27,"context_line":"protection in the case where a malicious user has access to the image cache. A"},{"line_number":28,"context_line":"malicious user may overwrite a cached image with an image containing malware,"},{"line_number":29,"context_line":"and then circumvent Nova\u0027s integrity checking mechanism by overwriting the"},{"line_number":30,"context_line":"disk image info file so that it contains a SHA-1 hash matching the corrupted"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_644b845a","line":27,"in_reply_to":"bab6814e_89f5eacc","updated":"2016-05-24 15:23:20.000000000","message":"This change isn\u0027t intended to close a specific bug or resolve a specific CVE, so we aren\u0027t hypothesizing specific zero days or vulnerabilities. This is a security engineering change. We begin by assuming that some bug in OpenStack or some compromise within the company hosting a deployment\u0027s compute service allows an adversary to access the image cache. Then we consider what damage they can do and how we can mitigate this damage and alert the deployers that a compromise has occurred. This feature is an attempt to do that by actually imposing integrity checks on images stored in the cache.","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":9555,"name":"Matthew Booth","email":"mbooth@redhat.com","username":"MatthewBooth"},"change_message_id":"8bd4400d2c1b7e23f5b7d63fd18e8d551c494bfb","unresolved":false,"context_lines":[{"line_number":25,"context_line":"verification of a SHA-1 hash. Since the hash is stored in the same filesystem"},{"line_number":26,"context_line":"as the cached disk image by default, this integrity check provides little"},{"line_number":27,"context_line":"protection in the case where a malicious user has access to the image cache. A"},{"line_number":28,"context_line":"malicious user may overwrite a cached image with an image containing malware,"},{"line_number":29,"context_line":"and then circumvent Nova\u0027s integrity checking mechanism by overwriting the"},{"line_number":30,"context_line":"disk image info file so that it contains a SHA-1 hash matching the corrupted"},{"line_number":31,"context_line":"image."}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_33626b17","line":28,"updated":"2016-05-24 14:06:02.000000000","message":"This sounds bad. How can they do this?","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":15524,"name":"dane-fichter","email":"dane.fichter@jhuapl.edu","username":"dane-fichter"},"change_message_id":"3e3a98dfc02a49d20b4efb5b8ea8fb4beb28e0cf","unresolved":false,"context_lines":[{"line_number":25,"context_line":"verification of a SHA-1 hash. Since the hash is stored in the same filesystem"},{"line_number":26,"context_line":"as the cached disk image by default, this integrity check provides little"},{"line_number":27,"context_line":"protection in the case where a malicious user has access to the image cache. A"},{"line_number":28,"context_line":"malicious user may overwrite a cached image with an image containing malware,"},{"line_number":29,"context_line":"and then circumvent Nova\u0027s integrity checking mechanism by overwriting the"},{"line_number":30,"context_line":"disk image info file so that it contains a SHA-1 hash matching the corrupted"},{"line_number":31,"context_line":"image."}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_411d92bc","line":28,"in_reply_to":"bab6814e_33626b17","updated":"2016-05-24 15:23:20.000000000","message":"I\u0027m confused by the question. If an attacker has access to the image cache, they can replace files as you would on any filesystem you have access to.","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":9555,"name":"Matthew Booth","email":"mbooth@redhat.com","username":"MatthewBooth"},"change_message_id":"9f4ec3f5f362e8d7273a6b22bf685ebaf87a79fc","unresolved":false,"context_lines":[{"line_number":25,"context_line":"verification of a SHA-1 hash. Since the hash is stored in the same filesystem"},{"line_number":26,"context_line":"as the cached disk image by default, this integrity check provides little"},{"line_number":27,"context_line":"protection in the case where a malicious user has access to the image cache. A"},{"line_number":28,"context_line":"malicious user may overwrite a cached image with an image containing malware,"},{"line_number":29,"context_line":"and then circumvent Nova\u0027s integrity checking mechanism by overwriting the"},{"line_number":30,"context_line":"disk image info file so that it contains a SHA-1 hash matching the corrupted"},{"line_number":31,"context_line":"image."}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_ad138f0c","line":28,"in_reply_to":"bab6814e_411d92bc","updated":"2016-05-24 15:50:43.000000000","message":"This does not offer any significant defence against an attacker who can replace arbitrary files on Nova\u0027s filesystem. An attacker who can do this does not need to corrupt the image cache, because they can directly corrupt the user\u0027s data and metadata. They can also read and potentially even rewrite nova\u0027s configuration, which means they can directly access rabbit and the db, which means that the entire openstack deployment is compromised.\n\nI\u0027m still not convinced by the use case here.","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":9555,"name":"Matthew Booth","email":"mbooth@redhat.com","username":"MatthewBooth"},"change_message_id":"8bd4400d2c1b7e23f5b7d63fd18e8d551c494bfb","unresolved":false,"context_lines":[{"line_number":27,"context_line":"protection in the case where a malicious user has access to the image cache. A"},{"line_number":28,"context_line":"malicious user may overwrite a cached image with an image containing malware,"},{"line_number":29,"context_line":"and then circumvent Nova\u0027s integrity checking mechanism by overwriting the"},{"line_number":30,"context_line":"disk image info file so that it contains a SHA-1 hash matching the corrupted"},{"line_number":31,"context_line":"image."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"Use Cases"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_533f2fd6","line":30,"updated":"2016-05-24 14:06:02.000000000","message":"Again, how is this possible?","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":15524,"name":"dane-fichter","email":"dane.fichter@jhuapl.edu","username":"dane-fichter"},"change_message_id":"3e3a98dfc02a49d20b4efb5b8ea8fb4beb28e0cf","unresolved":false,"context_lines":[{"line_number":27,"context_line":"protection in the case where a malicious user has access to the image cache. A"},{"line_number":28,"context_line":"malicious user may overwrite a cached image with an image containing malware,"},{"line_number":29,"context_line":"and then circumvent Nova\u0027s integrity checking mechanism by overwriting the"},{"line_number":30,"context_line":"disk image info file so that it contains a SHA-1 hash matching the corrupted"},{"line_number":31,"context_line":"image."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"Use Cases"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_24969c94","line":30,"in_reply_to":"bab6814e_533f2fd6","updated":"2016-05-24 15:23:20.000000000","message":"Again, if the attacker has access to the image cache, this is a simple matter of replacing a valid info file with an info file containing a SHA-1 hash matching the malicious image.","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":7764,"name":"Peter Hamilton","email":"peter.hamilton@jhuapl.edu","username":"Peter"},"change_message_id":"2141690f97ec65ce941f3e228ff3b106f1d35d74","unresolved":false,"context_lines":[{"line_number":28,"context_line":"malicious user may overwrite a cached image with an image containing malware,"},{"line_number":29,"context_line":"and then circumvent Nova\u0027s integrity checking mechanism by overwriting the"},{"line_number":30,"context_line":"disk image info file so that it contains a SHA-1 hash matching the corrupted"},{"line_number":31,"context_line":"image."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"Use Cases"},{"line_number":34,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_1154f8fc","line":31,"range":{"start_line":31,"start_character":5,"end_line":31,"end_character":6},"updated":"2016-05-20 18:54:42.000000000","message":"The image cache could also be hosted on remote storage that may be accessible by an attacker.","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":9555,"name":"Matthew Booth","email":"mbooth@redhat.com","username":"MatthewBooth"},"change_message_id":"8bd4400d2c1b7e23f5b7d63fd18e8d551c494bfb","unresolved":false,"context_lines":[{"line_number":28,"context_line":"malicious user may overwrite a cached image with an image containing malware,"},{"line_number":29,"context_line":"and then circumvent Nova\u0027s integrity checking mechanism by overwriting the"},{"line_number":30,"context_line":"disk image info file so that it contains a SHA-1 hash matching the corrupted"},{"line_number":31,"context_line":"image."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"Use Cases"},{"line_number":34,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_565b9d2e","line":31,"range":{"start_line":31,"start_character":5,"end_line":31,"end_character":6},"in_reply_to":"bab6814e_1154f8fc","updated":"2016-05-24 14:06:02.000000000","message":"This sounds like a genuine threat, but I\u0027m not convinced by the other 2. They would be bugs which we need to fix asap, but off the top of my head I don\u0027t know how a user could do either of these things. Of course, if I did I wouldn\u0027t tell you about it until the embargo was lifted on the CVE I filed for it.","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"c2575cb5628fbc8f6eac7f5e99afc946d3c8df3b","unresolved":false,"context_lines":[{"line_number":28,"context_line":"malicious user may overwrite a cached image with an image containing malware,"},{"line_number":29,"context_line":"and then circumvent Nova\u0027s integrity checking mechanism by overwriting the"},{"line_number":30,"context_line":"disk image info file so that it contains a SHA-1 hash matching the corrupted"},{"line_number":31,"context_line":"image."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"Use Cases"},{"line_number":34,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_92ed0691","line":31,"range":{"start_line":31,"start_character":5,"end_line":31,"end_character":6},"in_reply_to":"bab6814e_565b9d2e","updated":"2016-05-24 15:49:52.000000000","message":"I guess I\u0027m not sure why this is confusing. If I manage to get on the compute node, I can modify the base image (or likely the instance disk itself) in-place. For things that aren\u0027t likely to be written (i.e. all of /sbin) I could have arranged for already-booted instances to run a malicious sudo replacement the next time they do something that requires reading that block in.\n\nThe above is basically why I\u0027m not sure that any amount of checking the image at boot time is useful in any real way. I totally agree that checking when we download it from glance is useful to make sure we weren\u0027t MITM\u0027d during the fetch, but otherwise, I think anything further is theater. Expensive-for-the-cpu theater.","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":9555,"name":"Matthew Booth","email":"mbooth@redhat.com","username":"MatthewBooth"},"change_message_id":"8bd4400d2c1b7e23f5b7d63fd18e8d551c494bfb","unresolved":false,"context_lines":[{"line_number":35,"context_line":""},{"line_number":36,"context_line":"* A user wants a high degree of assurance that an instance booted from Nova\u0027s"},{"line_number":37,"context_line":"  image cache has not been corrupted or accidentally modified since Nova"},{"line_number":38,"context_line":"  initially stored it in its image cache."},{"line_number":39,"context_line":""},{"line_number":40,"context_line":"With this proposed change, Nova may be configured to perform image"},{"line_number":41,"context_line":"signature verification on the images stored in its image cache."}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_6c56d407","line":38,"updated":"2016-05-24 14:06:02.000000000","message":"The user has no way of knowing if the image they\u0027re booting is even in the image cache. The image cache is an internal optimisation whose details should be completely transparent to the user.\n\nI think the real use case is:\n\n* A user wants a high degree of assurance that the glance image used to boot their nova instance is exactly the one they requested.","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"c2575cb5628fbc8f6eac7f5e99afc946d3c8df3b","unresolved":false,"context_lines":[{"line_number":35,"context_line":""},{"line_number":36,"context_line":"* A user wants a high degree of assurance that an instance booted from Nova\u0027s"},{"line_number":37,"context_line":"  image cache has not been corrupted or accidentally modified since Nova"},{"line_number":38,"context_line":"  initially stored it in its image cache."},{"line_number":39,"context_line":""},{"line_number":40,"context_line":"With this proposed change, Nova may be configured to perform image"},{"line_number":41,"context_line":"signature verification on the images stored in its image cache."}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_52ad5e87","line":38,"in_reply_to":"bab6814e_6c56d407","updated":"2016-05-24 15:49:52.000000000","message":"Right, this is the user-facing goal I think. Dane\u0027s words above aren\u0027t really wrong, they\u0027re just not what the user is considering.","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":9555,"name":"Matthew Booth","email":"mbooth@redhat.com","username":"MatthewBooth"},"change_message_id":"8bd4400d2c1b7e23f5b7d63fd18e8d551c494bfb","unresolved":false,"context_lines":[{"line_number":43,"context_line":"Proposed change"},{"line_number":44,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"This change adds a \u0027verify_cache_signatures\u0027 config option to Nova. If this"},{"line_number":47,"context_line":"flag is set, Nova performs signature verification on cached images"},{"line_number":48,"context_line":"periodically, according to the period set by the existing"},{"line_number":49,"context_line":"\u0027checksum_interval_seconds\u0027 configuration option. To perform this"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_8c64d8e5","line":46,"updated":"2016-05-24 14:06:02.000000000","message":"Ditch the references to the cache. This should be something like \u0027verify_image_signatures\u0027. We can do this regardless of whether or not they\u0027re in the cache.","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":15524,"name":"dane-fichter","email":"dane.fichter@jhuapl.edu","username":"dane-fichter"},"change_message_id":"3e3a98dfc02a49d20b4efb5b8ea8fb4beb28e0cf","unresolved":false,"context_lines":[{"line_number":43,"context_line":"Proposed change"},{"line_number":44,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"This change adds a \u0027verify_cache_signatures\u0027 config option to Nova. If this"},{"line_number":47,"context_line":"flag is set, Nova performs signature verification on cached images"},{"line_number":48,"context_line":"periodically, according to the period set by the existing"},{"line_number":49,"context_line":"\u0027checksum_interval_seconds\u0027 configuration option. To perform this"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_e48134fd","line":46,"in_reply_to":"bab6814e_8c64d8e5","updated":"2016-05-24 15:23:20.000000000","message":"This already exists and is merged in Nova. If the \"verify_glance_signatures\" configuration flag is set, Nova validates the signatures associated with Glance images upon downloading these images. See:\n\nhttps://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/image-verification.html\n\nVerifying cache signatures is meant as a means of closing the last gap; images can be fetched either directly from Glance or from the image cache, but we in either case we want to verify that they match the signature which was valid when they were initially uploaded to Glance.","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":9555,"name":"Matthew Booth","email":"mbooth@redhat.com","username":"MatthewBooth"},"change_message_id":"8bd4400d2c1b7e23f5b7d63fd18e8d551c494bfb","unresolved":false,"context_lines":[{"line_number":46,"context_line":"This change adds a \u0027verify_cache_signatures\u0027 config option to Nova. If this"},{"line_number":47,"context_line":"flag is set, Nova performs signature verification on cached images"},{"line_number":48,"context_line":"periodically, according to the period set by the existing"},{"line_number":49,"context_line":"\u0027checksum_interval_seconds\u0027 configuration option. To perform this"},{"line_number":50,"context_line":"verification, Nova retrieves the image ID from the info file for each image,"},{"line_number":51,"context_line":"uses the Glance API to retrieve the signature metadata associated with each"},{"line_number":52,"context_line":"image, and verifies this metadata using the existing signature_utils module."}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_8cacd836","line":49,"updated":"2016-05-24 14:06:02.000000000","message":"The problem with doing this periodically is that you\u0027ve left open a window of attack. If your attack vector exists (not convinced) then your attacker can time their attack to fall within this check period, and we\u0027ve achieved nothing.\n\nI can think of 2 reasons to ever perform this check after initial download:\n\n1. Data corruption. This isn\u0027t the way to deal with this, though. We already have another defence against that, and I don\u0027t think that\u0027s required either. Filesystem/block device checksumming is the way to go here.\n\n2. Belt and braces against unforeseen bugs. This is the defence against a zero-day which *does* allow the attacker to manipulate the image cache. As mentioned above, a periodic check buys you nothing in practise in this case. If you want to defend against this you need to checksum the root disk immediately before first boot during spawn. This is going to slow down your boot, though, so this needs to be a config option or flavor metadata.","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"c2575cb5628fbc8f6eac7f5e99afc946d3c8df3b","unresolved":false,"context_lines":[{"line_number":46,"context_line":"This change adds a \u0027verify_cache_signatures\u0027 config option to Nova. If this"},{"line_number":47,"context_line":"flag is set, Nova performs signature verification on cached images"},{"line_number":48,"context_line":"periodically, according to the period set by the existing"},{"line_number":49,"context_line":"\u0027checksum_interval_seconds\u0027 configuration option. To perform this"},{"line_number":50,"context_line":"verification, Nova retrieves the image ID from the info file for each image,"},{"line_number":51,"context_line":"uses the Glance API to retrieve the signature metadata associated with each"},{"line_number":52,"context_line":"image, and verifies this metadata using the existing signature_utils module."}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_4d62e3d9","line":49,"in_reply_to":"bab6814e_47c5a259","updated":"2016-05-24 15:49:52.000000000","message":"Based on my argument above, I think periodic checking is actually more useful than checking at boot. If you check only at boot, then you never get an indication that an attacker has visited your system, until/unless that same image is used again on the same compute node and you happen to realize that it affects some of the other instances. On the other hand, if we\u0027re periodically validating the images, then we will are guaranteed to eventually be notified of the exploit.\n\nHowever, to my previous point of this being theater.. if I root the node, I think the first thing I\u0027m going to do is modify the nova code to disable this checking before I go to the hard trouble of modifying blocks in the images :)","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"},{"author":{"_account_id":15524,"name":"dane-fichter","email":"dane.fichter@jhuapl.edu","username":"dane-fichter"},"change_message_id":"3e3a98dfc02a49d20b4efb5b8ea8fb4beb28e0cf","unresolved":false,"context_lines":[{"line_number":46,"context_line":"This change adds a \u0027verify_cache_signatures\u0027 config option to Nova. If this"},{"line_number":47,"context_line":"flag is set, Nova performs signature verification on cached images"},{"line_number":48,"context_line":"periodically, according to the period set by the existing"},{"line_number":49,"context_line":"\u0027checksum_interval_seconds\u0027 configuration option. To perform this"},{"line_number":50,"context_line":"verification, Nova retrieves the image ID from the info file for each image,"},{"line_number":51,"context_line":"uses the Glance API to retrieve the signature metadata associated with each"},{"line_number":52,"context_line":"image, and verifies this metadata using the existing signature_utils module."}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_47c5a259","line":49,"in_reply_to":"bab6814e_8cacd836","updated":"2016-05-24 15:23:20.000000000","message":"Checking periodically was meant as a means of both fitting in the same scheme as the existing checksum verification mechanism,  but it also provides the deployer with a notification WHEN the images are compromised, which is valuable\n\nThe attacker would be able to use this \"window of attack\" if we were only talking about checksum verification, since it\u0027s only a matter of replacing the image file and the image info file, but the feasibility of this attack breaks down if Nova is verifying image signatures. The attacker would be able to swap in an image with a valid signature. If the deployment is still issuing certs and validating them correctly, then this essentially means that the attacker is limited to swapping one trusted image for another.","commit_id":"2505194ed34b7ac0426515d68111e1e855cf244e"}]}