)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"b30b68c4a7f156e158024e14d09ab8dcd6d5e480","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"763fb90a_3bf3ce03","updated":"2025-04-16 18:38:10.000000000","message":"Thanks for writing this up! One spelling issue about the option name and a question about the server side.","commit_id":"0c3cf8a4516e8d1ab5a22ef56e63f7a2caea55b9"},{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"0b056b15c6253e987dd272cba1288ae277a28099","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"94a3e4e9_cfee414d","updated":"2025-04-18 02:00:41.000000000","message":"Gah, sorry about removing the Topic ^. I was just going to rename it to `bp/bypass-chunked-transfer` and didn\u0027t know I don\u0027t have permissions to do that.","commit_id":"ead01b64d1973d87470dac570362d9ee12203dd4"},{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"b95b66cd3fdae7ff972658216ae96a4f6af273b5","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"0b58c7dd_c78ea3fa","updated":"2025-04-17 23:02:17.000000000","message":"I\u0027m not sure if it affects any automation but just a note that the spec file name could also be renamed to match \"bypass\" as one word.","commit_id":"ead01b64d1973d87470dac570362d9ee12203dd4"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"eda745a933c9840e28e39f93675056422e678348","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"36c838c9_87ba8439","updated":"2025-04-21 14:06:56.000000000","message":"Thank you Dan for inputs!","commit_id":"58feea65a4527162b27ff8d4438f55b657f408c9"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"afc06a85380476737837cb6b2d1a9f1e5f2ef2a7","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"4db8b56d_9f14b447","updated":"2025-04-21 14:44:11.000000000","message":"Adding rosmaita to see if there were any concerns for v2 to remove size calculation. May be I need to dig back to find out more about it if he is also not aware about the same.","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2af00ac1ef1823ad6fccd5e14b034aed9445cb78","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"5178f329_bf8a7f86","updated":"2025-04-21 14:39:26.000000000","message":"Melanie should be around in a couple hours, maybe we should also see what she thinks.","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"fd388101537bee17219764a1c85800143b650d69","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"a1b48078_a07d31e1","updated":"2025-04-21 15:26:52.000000000","message":"Okay, so I had a quick chat with rosmaita which I think helped clarify. The summary is:\n\nIn the before times, glance was *mostly* either backed by file on disk or swift. Chunked transfers were fine with either case because on file we don\u0027t need to pre-allocate and on swift we\u0027re storing the image in chunks anyway. So, knowing the size ahead of time was just not that important and Content-Length was a fine (and RESTful) way to do things. However, in the current times, swift is less common but ceph, cinder and similar backing technologies are the norm. These require pre-allocation and resizing is expensive.\n\nMy takeaway here, and I think it represents rosmaita\u0027s stance as well (but I\u0027ll let him confirm), is that we should provide glance as much info as possible as early as possible. That\u0027s fitting with the hash, the disk_format, and the size, all three of which would fail the upload if incorrect after we have the data. If the backend cares about size (ceph does, swift does not) then it can use that information to be as efficient as possible. If that information is not available, then doing the best effort (i.e. resize during upload) makes sense of course.\n\nFurther, adding this \"bypass chunked upload\" at the client layer just requires the caller to understand all these interactions in a way that shouldn\u0027t be required. We should not need to know what the backend is, whether or not it can resize efficiently, or that using chunked transfers means we don\u0027t communicate the size and thus require resizes on the backend. Keep transfer-related considerations separate from the backend considerations, and provide glance as much information as we have ahead of time to allow it to make the best decisions it can.\n\nTo me, the cleanest way to do this is to just go back to allowing us to set the size on the image at create time (like disk_format and the hashes). If it doesn\u0027t match after upload (or during if we want to check in the non-chunked mode) then we should reject, just like if the other two mismatch.","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"e7b54ca1b0a5c3629e202dbcd7d2f8a89893a986","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"5cae9a65_afc7250a","in_reply_to":"2283b8db_fcb037c0","updated":"2025-04-23 20:23:45.000000000","message":"I don\u0027t think it would be a hack? From what I could tell it\u0027s how Glance API currently supports specification of the image size for an upload [1]:\n\n```\n    def upload(self, request):\n        try:\n            request.get_content_type((\u0027application/octet-stream\u0027,))\n        except exception.InvalidContentType as e:\n            raise webob.exc.HTTPUnsupportedMediaType(explanation\u003de.msg)\n\n        if self.is_valid_encoding(request) and self.is_valid_method(request):\n            request.is_body_readable \u003d True\n\n        image_size \u003d request.content_length or None\n        return {\u0027size\u0027: image_size, \u0027data\u0027: request.body_file}\n```\n\n[1] https://github.com/openstack/glance/blob/d2c3e102fdff35c1dd15495c0445870e701d7e4b/glance/api/v2/image_data.py#L470-L480","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"addfa6a3ef72204f7f5b815920aa8bcca74a5f74","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"94f0e402_2ae6a34d","in_reply_to":"2411cb02_521fc14f","updated":"2025-04-23 02:25:58.000000000","message":"I don\u0027t see any problem with the idea of doing an API change to add image size to the v2 /images create API and using it when appropriate for efficiency but obviously doing that is a much larger change than adding something to the client to work with Glance\u0027s existing behavior which is to use the Content-Length header.\n\nThe concern about the caller knowing about \"bypass chunked upload\" details is a fair point. Personally I was viewing it as an option for the glanceclient Python bindings only, for more advanced users to be able to take advantage of faster uploads when appropriate.\n\nI was thinking there might be a nice compromise. The glanceclient Python bindings for upload() and stage() have a currently unused keyword argument \"image_size\" which in the docstring states it\u0027s a holdover from v1 that is not used. What we could do is reinstate the keyword argument and remove the docstring text about it being deprecated and unused, and if it is specified, bypass the chunked transfer encoding internally in glanceclient. That way the caller won\u0027t see implementation details like \"bypass chunked upload\" but will still be able to opt in to better performance with Glance\u0027s existing behavior. It would be backportable and work to add image size to the v2 create API could be done in parallel (note: I will not be able to help on the API change part and we\u0027ll need a Glance expert for that).","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"b6dfdd9f695962e79489df7b3aa42d1e2632259f","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"955a6616_cd511a42","in_reply_to":"26296f99_c0b5b65d","updated":"2025-04-24 13:49:28.000000000","message":"\u003e Sure, it\u0027s not ideal. But that\u0027s how Glance currently works, intentionally\n\nIt\u0027s not how it works \"intentionally\" - the resize-on-write path is only done in the exceptional case of not having image size (which the code clearly prefers to have). It\u0027s just that we don\u0027t get it when using the upload mechanism that glanceclient uses *intentionally* (and for good reason). The comments explain that the resizes are just necessary evil if we don\u0027t know. We\u0027ve had multiple revisions to the rbd resize code to make it suck as little as possible. It still sucks.\n\n\u003e Some customers have been stuck unable to upload an image *at all* because of the iterative chunked transfer encoding method which indirectly triggers a detach/attach of a Cinder volume on the Glance side each time the volume is resized larger and after a few iterations it fails (for them). From what I understand, the detach/attach is unavoidable on the Glance side because Cinder or os-brick does it internally. And IIRC it can\u0027t be avoided in Cinder/os-brick either.\n\nYep, understand. It definitely seems like it was an ill-considered and under-tested arrangement for sure. I\u0027d also make the counter argument that changing the entire mechanism that nova and glance (and everything else) use to transfer gigs of image data through multiple proxies and load balancers for a \"small\" bug like this might have unintended consequences that could also be under-tested until they hit the customer machines. Not saying that there\u0027s any real problem here, but it\u0027s a substantial difference in behavior for a bug fix.\n\n\u003e If all we have to do is set the already existing image_size keyword argument in glanceclient and have it tell Glance the size in the manner which Glance currently supports, then we can significantly improve quality of life for end users of Nova trying to snapshot their instances right now and can backport it for older versions too. Nova always knows the image size of a snapshot it is uploading from local storage to Glance.\n\nBut that\u0027s not \"all\" we have to do right? We have to undeprecate image_size, then make the presence of it disable chunked transfer (and in turn set the header) right? That means you\u0027re exchanging the concern of \"a bypass_chunked_transfer parameter is confusing\" and replacing it with bypassing chunked transfers implicitly when someone passes another argument that should have no bearing on how we transfer stuff. I think we *should* undeprecate `image_size` but use it to pre-set the image size before we do the transfer and not completely change _how_ we do that transfer. If you use it to disable chunked transfers and then we go back to doing the right thing, then that\u0027s two underlying changes in behavior in an otherwise stable client library.\n\nI get that it seems small and backportable and it\u0027s tempting to just do the quick thing that will make the pain stop. However, this is a client library\n\n\u003e If consensus is against it then that\u0027s fine, we can drop this proposal.\n\nSame, of course. Maybe someone will decide that since we\u0027re about to freeze glanceclient forever that it makes sense to change the long-standing transfer encoding behavior right before we do that and make the sdk do the right behavior, but it\u0027s definitely not my preference.","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"56595488cfef4d194d0590b60997e67bea009007","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"6128b3a8_6adc2237","in_reply_to":"47c9f669_f3c27d7f","updated":"2025-04-24 15:38:54.000000000","message":"Cool, thanks Abhishek. I will leave you to it then :)","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"3d5046408d035b2ce7db0f895892e62a80820622","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"621108fd_005766e8","in_reply_to":"5cae9a65_afc7250a","updated":"2025-04-23 21:20:54.000000000","message":"What I\u0027m saying is a hack is disabling chunked transfer encoding because we pass a size to the client call. Glance currently can\u0027t provide the image size to glance_store at the same time as chunked transfers are performed because it only allows the image size to be communicated via the content-length header. The header is used for the *chunk* size in chunked transfer, and thus can\u0027t be used for both the chunk size *and* the overall image size. My assertion is that this is broken on the glance (and glance store) side, and that hacking around it in the client (and to some degree nova) is wrong. To summarize:\n\n1. The goal is we need to provide the image size to glance so that it doesn\u0027t have to constantly resize during upload\n2. We can only provide the size via the content-length header (current limitation by glance)\n3. If we use chunked transfers (which makes the code on both side more efficient), the http library underneath us uses content-length (per the spec) for the chunk lengths and thus the overall length does not get communicated to the glance internals when it calls glance store.\n4. I think it\u0027s wrong to solve #1 by changing the client chunked-transfer behavior because of #2 and #3, which are glance internals, and also which only matter for a subset of the glance backends.\n\nI\u0027m asserting the glance server code (underneath the code you link to) is wrong and working/hacking around it in the client code (by pushing the need to know about this to the caller) just further buries the real problem and obfuscates the client interaction because of a detail on the store side which the caller literally should know nothing about.","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"5023f4adf6b0107d98f26389f62de6e4726c026e","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"cdb4f022_38f672f2","in_reply_to":"621108fd_005766e8","updated":"2025-04-23 22:05:20.000000000","message":"\u003e What I\u0027m saying is a hack is disabling chunked transfer encoding because we pass a size to the client call. Glance currently can\u0027t provide the image size to glance_store at the same time as chunked transfers are performed because it only allows the image size to be communicated via the content-length header. The header is used for the *chunk* size in chunked transfer, and thus can\u0027t be used for both the chunk size *and* the overall image size. My assertion is that this is broken on the glance (and glance store) side, and that hacking around it in the client (and to some degree nova) is wrong. To summarize:\n\nI think I see the confusion. The `Transfer-Encoding: chunked` and `Content-Length` headers are mutually exclusive [1]. When \"chunked transfer encoding\" is used, the Content-Length header should not be specified. The chunk size of a \"chunked transfer encoding\" is specified in a different way [2] (I think that shows a field in the message body?).\n\nI think that is why Glance communicates total image size by Content-Length because that\u0027s how the header is supposed to be used.\n\nThis whole issue has been confusing because of the word \"chunked\". The \"chunked transfer encoding\" is not the only way that data can be sent in smaller size chunks/blocks for efficiency. The requests library calls the Content-Length way a \"streaming upload\":\n\nhttps://requests.readthedocs.io/en/latest/user/advanced/#streaming-uploads\n\nwhereas \"chunked transfer encoding\" uses the Transfer-Encoding header without the Content-Length header:\n\nhttps://requests.readthedocs.io/en/latest/user/advanced/#chunk-encoded-requests\n\nThe latter \"chunked transfer encoding\" is typically used in situations where the data size is not known upfront, such as a live video stream or similar. And when the stream is finished the client is supposed to send a message with chunk-size 0 to signal the end of the data.\n\nBecause we know the image size ahead of time, we can use the Content-Length header to communicate the total expected data size to Glance and the requests library will send it in chunks automatically.\n\nUnless I\u0027m missing something, it seems this is working as intended.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.4\n[2] https://datatracker.ietf.org/doc/html/rfc2616#section-3.6.1","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"1c9848840cae5c3a3ded074bf151463e90b62538","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"26296f99_c0b5b65d","in_reply_to":"7c62f1f0_0f36db23","updated":"2025-04-23 22:57:13.000000000","message":"\u003e It\u0027s actually Content-Length in each of the chunks, IIRC, but yes.\n\nIt\u0027s not, according to the doc I linked earlier:\n\n```\n3.6.1 Chunked Transfer Coding\n\n   The chunked encoding modifies the body of a message in order to\n   transfer it as a series of chunks, each with its own size indicator,\n   followed by an OPTIONAL trailer containing entity-header fields. This\n   allows dynamically produced content to be transferred along with the\n   information necessary for the recipient to verify that it has\n   received the full message.\n\n       Chunked-Body   \u003d *chunk\n                        last-chunk\n                        trailer\n                        CRLF\n\n       chunk          \u003d chunk-size [ chunk-extension ] CRLF\n                        chunk-data CRLF\n       chunk-size     \u003d 1*HEX\n       last-chunk     \u003d 1*(\"0\") [ chunk-extension ] CRLF\n\n       chunk-extension\u003d *( \";\" chunk-ext-name [ \"\u003d\" chunk-ext-val ] )\n       chunk-ext-name \u003d token\n       chunk-ext-val  \u003d token | quoted-string\n       chunk-data     \u003d chunk-size(OCTET)\n       trailer        \u003d *(entity-header CRLF)\n\n   The chunk-size field is a string of hex digits indicating the size of\n   the chunk. The chunked encoding is ended by any chunk whose size is\n   zero, followed by the trailer, which is terminated by an empty line.\n   \n   The trailer allows the sender to include additional HTTP header\n   fields at the end of the message. The Trailer header field can be\n   used to indicate which header fields are included in a trailer (see\n   section 14.40).\n```\n\nhttps://datatracker.ietf.org/doc/html/rfc2616#section-3.6.1\n\n\u003e We definitely _can_ use non-chunked upload and use content-length to communicate the total image size to glance. I\u0027m not saying we can\u0027t. I am saying I do not think we *should* require that you choose either non-chunked uploads (and good write performance on ceph/cinder) or chunked uploads and poor performance from the client side with such an impact on how the backend works. To show the opposite example, chunk-based storage (like swift) can use the chunks (in an actually-chunked transfer) to keep the same blocks in the backend as they\u0027re currently stored/sent, as that\u0027s how swift works natively (it does not need to store them sequentially or do any resizing). If we require the callers to choose chunked or non-chunked because they know the interaction between the internals of glance and glance store (and requests) then we\u0027re pushing the need to know the backend semantics to the caller (i.e nova in this case).\n\u003e \n\u003e If we fix glance to not tie those two things together, then you can upload chunked or non-chunked, and glance-store can optimize for the backend in use in either case. If we don\u0027t then the caller needs to choose - a responsibility it can\u0027t really do well, nor should it even know about it.\n\nSure, it\u0027s not ideal. But that\u0027s how Glance currently works, intentionally, and if we have the ability to use it correctly in the universe it\u0027s in, I don\u0027t see why we shouldn\u0027t to alleviate end user pain.\n\nSome customers have been stuck unable to upload an image *at all* because of the iterative chunked transfer encoding method which indirectly triggers a detach/attach of a Cinder volume on the Glance side each time the volume is resized larger and after a few iterations it fails (for them). From what I understand, the detach/attach is unavoidable on the Glance side because Cinder or os-brick does it internally. And IIRC it can\u0027t be avoided in Cinder/os-brick either.\n\nIf all we have to do is set the already existing image_size keyword argument in glanceclient and have it tell Glance the size in the manner which Glance currently supports, then we can significantly improve quality of life for end users of Nova trying to snapshot their instances right now and can backport it for older versions too. Nova always knows the image size of a snapshot it is uploading from local storage to Glance.\n\nSeparately, it could be good to fix Glance too if you want Nova to not have to know about setting image_size\u003d\u003cimage size\u003e but I don\u0027t see why we should not do this minimal thing to fix the issue now from an end user perspective. Current callers of glanceclient would experience no change in behavior and if image_size is appropriate for them then they can opt in to using it.\n\nIf consensus is against it then that\u0027s fine, we can drop this proposal.","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"3e9389900a5e17ebf050398d1e109b29652af875","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"2283b8db_fcb037c0","in_reply_to":"94f0e402_2ae6a34d","updated":"2025-04-23 13:33:06.000000000","message":"I really don\u0027t like that, because someone using that will wonder why the performance (and behavior) of the transfer has changed when they provide the image size (or don\u0027t). I think any attempt to tie the chunking to the size just so the plumbing will do the right thing is a mistake, and just marries two internals together that do not need to be joined. If we want to do the hacky thing, then we should just pass another header called `Image-Size: XXX` or something to communicate the ultimate image size separate from the header used for the underlying transfer. But I really think that\u0027s just a better hack than using content-length.","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"9ad679154f77790d8198e651fef2d064e416dd1d","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"b7121034_14e36882","in_reply_to":"955a6616_cd511a42","updated":"2025-04-24 15:18:42.000000000","message":"\u003e \u003e Sure, it\u0027s not ideal. But that\u0027s how Glance currently works, intentionally\n\u003e \n\u003e It\u0027s not how it works \"intentionally\" - the resize-on-write path is only done in the exceptional case of not having image size (which the code clearly prefers to have). It\u0027s just that we don\u0027t get it when using the upload mechanism that glanceclient uses *intentionally* (and for good reason). The comments explain that the resizes are just necessary evil if we don\u0027t know. We\u0027ve had multiple revisions to the rbd resize code to make it suck as little as possible. It still sucks.\n\nI meant that Content-Length is intentionally how Glance currently supports specification of the image size for an upload. I was not talking about resize-before-write.\n\nAnyway, y\u0027all can let me know what you decide. If it\u0027s about glanceclient, I can help if desired. If not, feel free to abandon this spec.","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":5314,"name":"Brian Rosmaita","email":"rosmaita.fossdev@gmail.com","username":"brian-rosmaita"},"change_message_id":"3dea8268e6545c33fdb92c4d001a899e456e4479","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"2411cb02_521fc14f","in_reply_to":"a1b48078_a07d31e1","updated":"2025-04-21 15:42:19.000000000","message":"I agree with Dan\u0027s summary and also his general position on this.  I think it will be difficult to explain to end users what \"bypass chunked upload\" means and under what circumstances they would want to use it.  My feeling is that if the problem we are trying to solve is an excessive number of slow resize operations, then it\u0027s simpler to have the caller tell glance the image size in advance when it\u0027s known, and if we get a mismatch after the upload is complete, handle it consistently with disk_format and multihash.","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"d1fca8abf2c7b2eae1ea5c48346348dded4440fd","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"47c9f669_f3c27d7f","in_reply_to":"b7121034_14e36882","updated":"2025-04-24 15:34:15.000000000","message":"Hi Melwitt,\n\nThank you for your efforts on this, and sorry we could not came to conclusion earlier.\n\nToday we have discussed this in internal upstream glance upstream meeting and everyone agreed that we should not bypass the chunk transfer as we are on the verge of freezing the client development. What we agree on is;\n\n1. At client introduce new --size parameter for image-create and image-create-via-import command which will be used if --file option is provided or stdin is used to pass the data (image-upload and image-stage calls already have --size option) \n\n2. This option can be used by enduser to pass the image size in advance to the API\n\n3. At API side if size is provided and it is mismatched then we will reject the entire upload, if not specified then old process (resize on write) will be carried out.\n\n\nSO I will be moving this spec under glance side and there will be little work at client side to add this --size option to the API for upload and stage calls not as header but in request body.","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"8885e76f9918654428cff208906626d0e66770cd","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"7c62f1f0_0f36db23","in_reply_to":"cdb4f022_38f672f2","updated":"2025-04-23 22:27:31.000000000","message":"\u003e I think I see the confusion. The `Transfer-Encoding: chunked` and `Content-Length` headers are mutually exclusive [1]. When \"chunked transfer encoding\" is used, the Content-Length header should not be specified. The chunk size of a \"chunked transfer encoding\" is specified in a different way [2] (I think that shows a field in the message body?).\n\nIt\u0027s actually Content-Length in each of the chunks, IIRC, but yes.\n\n\u003e I think that is why Glance communicates total image size by Content-Length because that\u0027s how the header is supposed to be used.\n\nThat\u0027s how a non-chunked transfer is supposed to specify the size, if known, yes. However, the requests library doesn\u0027t send it if it\u0027s doing chunked transfers because you\u0027re not supposed to. Similarly, proxies along the way want to be able to read individual chunks and pass them along, and nowadays there are often multiples in each chain (haproxy, apache for mod_wsgi, etc).\n\n\u003e This whole issue has been confusing because of the word \"chunked\". The \"chunked transfer encoding\" is not the only way that data can be sent in smaller size chunks/blocks for efficiency. The requests library calls the Content-Length way a \"streaming upload\":\n\u003e \n\u003e https://requests.readthedocs.io/en/latest/user/advanced/#streaming-uploads\n\u003e \n\u003e whereas \"chunked transfer encoding\" uses the Transfer-Encoding header without the Content-Length header:\n\u003e \n\u003e https://requests.readthedocs.io/en/latest/user/advanced/#chunk-encoded-requests\n\u003e \n\u003e The latter \"chunked transfer encoding\" is typically used in situations where the data size is not known upfront, such as a live video stream or similar. And when the stream is finished the client is supposed to send a message with chunk-size 0 to signal the end of the data.\n\nIt\u0027s typically used for things like that, but not because the size isn\u0027t known (AFAIK). Things like MJPEG use it to chunk-send individual frames as units, even if the stream is of finite size. It does that to communicate the natural boundaries within the content to the receiver, without the receiver having to do its own buffering and interpretation on the fly. The same would happen if you were sending blocks or extents of a disk image from a source to a destination where there\u0027s a natural (or important) block size.\n\n\u003e Because we know the image size ahead of time, we can use the Content-Length header to communicate the total expected data size to Glance and the requests library will send it in chunks automatically.\n\u003e \n\u003e Unless I\u0027m missing something, it seems this is working as intended.\n\nWe definitely _can_ use non-chunked upload and use content-length to communicate the total image size to glance. I\u0027m not saying we can\u0027t. I am saying I do not think we *should* require that you choose either non-chunked uploads (and good write performance on ceph/cinder) or chunked uploads and poor performance from the client side with such an impact on how the backend works. To show the opposite example, chunk-based storage (like swift) can use the chunks (in an actually-chunked transfer) to keep the same blocks in the backend as they\u0027re currently stored/sent, as that\u0027s how swift works natively (it does not need to store them sequentially or do any resizing). If we require the callers to choose chunked or non-chunked because they know the interaction between the internals of glance and glance store (and requests) then we\u0027re pushing the need to know the backend semantics to the caller (i.e nova in this case).\n\nIf we fix glance to not tie those two things together, then you can upload chunked or non-chunked, and glance-store can optimize for the backend in use in either case. If we don\u0027t then the caller needs to choose - a responsibility it can\u0027t really do well, nor should it even know about it.","commit_id":"697e1760b3383c8b9e0ebebe104b5d291e1a836c"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"60217de218a6e12981321e8f42d1a18f4fcae3a5","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"040013a2_205cabd3","updated":"2025-04-25 15:16:48.000000000","message":"Thank you for review Dan, added few inline questions.","commit_id":"1e16b9242c0da02ba77d7ba155362c2fc53b1db5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"e515c905080fe11ac0aa911cb69b96352ea4fcb7","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":7,"id":"8067e393_dc00a947","updated":"2025-04-28 06:38:59.000000000","message":"Thank you Dan, made changes as per suggestions!!","commit_id":"b507d16f5ab274aad903244e7739f5edccdd414b"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"df217a479fc77e03ba87793dca808f723d6fd17c","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":8,"id":"0d06b203_2fe80c7e","updated":"2025-05-22 16:03:15.000000000","message":"One comment on the header change, not critical and we can just leave it to implementation if you want. But, I think the use of \"meta\" doesn\u0027t make much sense to me.\n\nOtherwise I think I\u0027m good.","commit_id":"50b79b6a02adda04dc053bd2614c50bc8fc48730"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"cc0484bb83698481583593a7301cf2650905e18b","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":8,"id":"7a08c9cb_2f418049","updated":"2025-05-22 17:19:06.000000000","message":"Thank you for review!!","commit_id":"50b79b6a02adda04dc053bd2614c50bc8fc48730"},{"author":{"_account_id":5314,"name":"Brian Rosmaita","email":"rosmaita.fossdev@gmail.com","username":"brian-rosmaita"},"change_message_id":"267d4bc8e5e5f63038f50c3776255e4c8bb5686d","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":9,"id":"0502c4f0_378dca3f","updated":"2025-05-28 15:33:42.000000000","message":"A few questions inline.  Also, two formatting issues:\n- when mentioning the new header, you need to use double-backticks to get a monospace font (single backtick is italics in RST)\n- when you mention ``--size``, you should enclose it in double-backticks also, because a plaintext ``--`` in RST gets turned into a single \"m-dash\" in HTML, which is kind of misleading (you can see this if you look at the HTML page [0]).\n\n[0] https://fe84552b8a9ac6c570c5-2c52d5c43a05db941913513785ce0245.ssl.cf2.rackcdn.com/openstack/d4e85c9d7372471192c993c07e5789d5/docs/specs/2025.2/approved/glance/set-size-on-upload.html","commit_id":"40b2ca52d1e7f056d3932f69d8defaf65b7f6197"},{"author":{"_account_id":5314,"name":"Brian Rosmaita","email":"rosmaita.fossdev@gmail.com","username":"brian-rosmaita"},"change_message_id":"20aca7ef78ab6c93efd1b7cc8edf7b4ea9ba5173","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":10,"id":"c055d6ef_a1e5a5e1","updated":"2025-05-29 12:57:09.000000000","message":"I forgot to mention that I\u0027m OK with not having the new header on the POST v2/images request; if it turns out that it would be useful, we can implement it later (though I\u0027d follow Dan\u0027s suggestion to use it as part of the request payload in that case).","commit_id":"e0eb7ac13429931622884f0add9548571988c1bf"},{"author":{"_account_id":5314,"name":"Brian Rosmaita","email":"rosmaita.fossdev@gmail.com","username":"brian-rosmaita"},"change_message_id":"8d7c973982d4162b5c49b3c1ccc83d9e1e2f5ebc","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":10,"id":"2b1b5965_63bc4eb8","updated":"2025-05-29 12:54:10.000000000","message":"My previous concerns have been addressed (and thanks for fixing the formatting! Much easier to read in HTML).\n\nI have a reservation about the \"Alternatives\" section, but that could be addressed as a followup.","commit_id":"e0eb7ac13429931622884f0add9548571988c1bf"}],"specs/2025.2/approved/glance/set-size-on-upload.rst":[{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"f42d4731f0068c32fd2818125ab7c8e3c653dd9e","unresolved":true,"context_lines":[{"line_number":56,"context_line":"is complete, we will compare the actual image size with the user-provided"},{"line_number":57,"context_line":"size; if they do not match, we will reject the image upload, delete the data"},{"line_number":58,"context_line":"from the backend, and return an appropriate error message to the user, along"},{"line_number":59,"context_line":"with a warning in the logs."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"If the image size is not included in the request body, we will have no option"},{"line_number":62,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":6,"id":"4d0c4fae_83f28c7f","line":59,"updated":"2025-04-25 15:01:14.000000000","message":"...mirroring the behavior of existing checks for image disk_format and hash...","commit_id":"1e16b9242c0da02ba77d7ba155362c2fc53b1db5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"60217de218a6e12981321e8f42d1a18f4fcae3a5","unresolved":false,"context_lines":[{"line_number":56,"context_line":"is complete, we will compare the actual image size with the user-provided"},{"line_number":57,"context_line":"size; if they do not match, we will reject the image upload, delete the data"},{"line_number":58,"context_line":"from the backend, and return an appropriate error message to the user, along"},{"line_number":59,"context_line":"with a warning in the logs."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"If the image size is not included in the request body, we will have no option"},{"line_number":62,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9ad24523_6e48ed3f","line":59,"in_reply_to":"4d0c4fae_83f28c7f","updated":"2025-04-25 15:16:48.000000000","message":"Acknowledged","commit_id":"1e16b9242c0da02ba77d7ba155362c2fc53b1db5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"f42d4731f0068c32fd2818125ab7c8e3c653dd9e","unresolved":true,"context_lines":[{"line_number":58,"context_line":"from the backend, and return an appropriate error message to the user, along"},{"line_number":59,"context_line":"with a warning in the logs."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"If the image size is not included in the request body, we will have no option"},{"line_number":62,"context_line":"but to allow the storage backends to perform resize-on-write operations or"},{"line_number":63,"context_line":"to utilize the single-part upload feature for the S3 backend."},{"line_number":64,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"0fb2b4fe_cbfe99e2","line":61,"range":{"start_line":61,"start_character":41,"end_line":61,"end_character":53},"updated":"2025-04-25 15:01:14.000000000","message":"Here and above, by \"request body\" you mean the create or update image JSON payload right? Since we\u0027re talking about upload here, I think \"request body\" is maybe a bit ambiguous, as the data stream is the actual upload \"request body\" to me.","commit_id":"1e16b9242c0da02ba77d7ba155362c2fc53b1db5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"60217de218a6e12981321e8f42d1a18f4fcae3a5","unresolved":true,"context_lines":[{"line_number":58,"context_line":"from the backend, and return an appropriate error message to the user, along"},{"line_number":59,"context_line":"with a warning in the logs."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"If the image size is not included in the request body, we will have no option"},{"line_number":62,"context_line":"but to allow the storage backends to perform resize-on-write operations or"},{"line_number":63,"context_line":"to utilize the single-part upload feature for the S3 backend."},{"line_number":64,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9925a408_4480bfa8","line":61,"range":{"start_line":61,"start_character":41,"end_line":61,"end_character":53},"in_reply_to":"0fb2b4fe_cbfe99e2","updated":"2025-04-25 15:16:48.000000000","message":"Ack, So are we going to introduce new header to pass the image size from client to API? OR we simply rely on --size passed to POST request and not PUT?","commit_id":"1e16b9242c0da02ba77d7ba155362c2fc53b1db5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"c6be4b6d7d9d2ce70092c5699b1a9024e24f756b","unresolved":false,"context_lines":[{"line_number":58,"context_line":"from the backend, and return an appropriate error message to the user, along"},{"line_number":59,"context_line":"with a warning in the logs."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"If the image size is not included in the request body, we will have no option"},{"line_number":62,"context_line":"but to allow the storage backends to perform resize-on-write operations or"},{"line_number":63,"context_line":"to utilize the single-part upload feature for the S3 backend."},{"line_number":64,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"689b4d78_859f7152","line":61,"range":{"start_line":61,"start_character":41,"end_line":61,"end_character":53},"in_reply_to":"3ae2a396_44fc9a92","updated":"2025-04-28 06:38:29.000000000","message":"Done","commit_id":"1e16b9242c0da02ba77d7ba155362c2fc53b1db5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"5930b0344b032787269847e5e7a2234de00cb236","unresolved":true,"context_lines":[{"line_number":58,"context_line":"from the backend, and return an appropriate error message to the user, along"},{"line_number":59,"context_line":"with a warning in the logs."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"If the image size is not included in the request body, we will have no option"},{"line_number":62,"context_line":"but to allow the storage backends to perform resize-on-write operations or"},{"line_number":63,"context_line":"to utilize the single-part upload feature for the S3 backend."},{"line_number":64,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"ce49dda4_940a2be7","line":61,"range":{"start_line":61,"start_character":41,"end_line":61,"end_character":53},"in_reply_to":"9925a408_4480bfa8","updated":"2025-04-25 15:25:14.000000000","message":"A header would be less work, but also more obscure (IMHO). It would require setting the size on the upload operation and not on the image itself (which is what makes more sense to me). We can do both of course as long as we enforce that they match.\n\nHowever, I didn\u0027t see either mentioned here, so which were you planning? \"Request body\" in the upload is the data payload itself. If you meant header, then I would say \"Request header\" (and specify what that\u0027s going to be).","commit_id":"1e16b9242c0da02ba77d7ba155362c2fc53b1db5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"e7165882d03448b9b3ce448f791440676125d625","unresolved":true,"context_lines":[{"line_number":58,"context_line":"from the backend, and return an appropriate error message to the user, along"},{"line_number":59,"context_line":"with a warning in the logs."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"If the image size is not included in the request body, we will have no option"},{"line_number":62,"context_line":"but to allow the storage backends to perform resize-on-write operations or"},{"line_number":63,"context_line":"to utilize the single-part upload feature for the S3 backend."},{"line_number":64,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"3ae2a396_44fc9a92","line":61,"range":{"start_line":61,"start_character":41,"end_line":61,"end_character":53},"in_reply_to":"ce49dda4_940a2be7","updated":"2025-04-25 15:48:04.000000000","message":"I will prefer a header to upload call. `x-openstack-image-size`","commit_id":"1e16b9242c0da02ba77d7ba155362c2fc53b1db5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"f42d4731f0068c32fd2818125ab7c8e3c653dd9e","unresolved":true,"context_lines":[{"line_number":140,"context_line":"* Calculate file size if file like object is provided"},{"line_number":141,"context_line":"* Make sure upload and stage commands passes size to API as request body"},{"line_number":142,"context_line":"* Make similar provision in openstack client and sdk"},{"line_number":143,"context_line":"* At Glance side compare provided size with actual size"},{"line_number":144,"context_line":"* Make sure image size is set before import operation starts"},{"line_number":145,"context_line":"* Required functional, unit and tempest tests"},{"line_number":146,"context_line":"* Document new behavior"}],"source_content_type":"text/x-rst","patch_set":6,"id":"62eb45b5_7fe10afa","line":143,"updated":"2025-04-25 15:01:14.000000000","message":"Don\u0027t we need to also make the API allow you to set size during create or on update?","commit_id":"1e16b9242c0da02ba77d7ba155362c2fc53b1db5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"c6be4b6d7d9d2ce70092c5699b1a9024e24f756b","unresolved":false,"context_lines":[{"line_number":140,"context_line":"* Calculate file size if file like object is provided"},{"line_number":141,"context_line":"* Make sure upload and stage commands passes size to API as request body"},{"line_number":142,"context_line":"* Make similar provision in openstack client and sdk"},{"line_number":143,"context_line":"* At Glance side compare provided size with actual size"},{"line_number":144,"context_line":"* Make sure image size is set before import operation starts"},{"line_number":145,"context_line":"* Required functional, unit and tempest tests"},{"line_number":146,"context_line":"* Document new behavior"}],"source_content_type":"text/x-rst","patch_set":6,"id":"d0188aa9_9fbd5392","line":143,"in_reply_to":"1b28a0a3_1ac2cc42","updated":"2025-04-28 06:38:29.000000000","message":"Done","commit_id":"1e16b9242c0da02ba77d7ba155362c2fc53b1db5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"5930b0344b032787269847e5e7a2234de00cb236","unresolved":true,"context_lines":[{"line_number":140,"context_line":"* Calculate file size if file like object is provided"},{"line_number":141,"context_line":"* Make sure upload and stage commands passes size to API as request body"},{"line_number":142,"context_line":"* Make similar provision in openstack client and sdk"},{"line_number":143,"context_line":"* At Glance side compare provided size with actual size"},{"line_number":144,"context_line":"* Make sure image size is set before import operation starts"},{"line_number":145,"context_line":"* Required functional, unit and tempest tests"},{"line_number":146,"context_line":"* Document new behavior"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7ca5c7d2_82ad81a4","line":143,"in_reply_to":"454f8880_2c803484","updated":"2025-04-25 15:25:14.000000000","message":"Depends on the answer to the above. Personally, I\u0027d like to end up with it as something you can set on the image ahead of time (i.e. in create, with update, separate from the upload). However, I know the separate-header-on-upload method is less work...","commit_id":"1e16b9242c0da02ba77d7ba155362c2fc53b1db5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"60217de218a6e12981321e8f42d1a18f4fcae3a5","unresolved":true,"context_lines":[{"line_number":140,"context_line":"* Calculate file size if file like object is provided"},{"line_number":141,"context_line":"* Make sure upload and stage commands passes size to API as request body"},{"line_number":142,"context_line":"* Make similar provision in openstack client and sdk"},{"line_number":143,"context_line":"* At Glance side compare provided size with actual size"},{"line_number":144,"context_line":"* Make sure image size is set before import operation starts"},{"line_number":145,"context_line":"* Required functional, unit and tempest tests"},{"line_number":146,"context_line":"* Document new behavior"}],"source_content_type":"text/x-rst","patch_set":6,"id":"454f8880_2c803484","line":143,"in_reply_to":"62eb45b5_7fe10afa","updated":"2025-04-25 15:16:48.000000000","message":"Ack, so in that case we also need to add --size option to image-create-via-import or no need for that?","commit_id":"1e16b9242c0da02ba77d7ba155362c2fc53b1db5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"e7165882d03448b9b3ce448f791440676125d625","unresolved":true,"context_lines":[{"line_number":140,"context_line":"* Calculate file size if file like object is provided"},{"line_number":141,"context_line":"* Make sure upload and stage commands passes size to API as request body"},{"line_number":142,"context_line":"* Make similar provision in openstack client and sdk"},{"line_number":143,"context_line":"* At Glance side compare provided size with actual size"},{"line_number":144,"context_line":"* Make sure image size is set before import operation starts"},{"line_number":145,"context_line":"* Required functional, unit and tempest tests"},{"line_number":146,"context_line":"* Document new behavior"}],"source_content_type":"text/x-rst","patch_set":6,"id":"b1bd3cd6_af53328a","line":143,"in_reply_to":"7ca5c7d2_82ad81a4","updated":"2025-04-25 15:48:04.000000000","message":"We should go with separate-header-on-upload. Also I guess updating image size is not allowed.","commit_id":"1e16b9242c0da02ba77d7ba155362c2fc53b1db5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"829b611ebbbf6ba336f1a5a2eb70c2e5e7fb94c5","unresolved":true,"context_lines":[{"line_number":140,"context_line":"* Calculate file size if file like object is provided"},{"line_number":141,"context_line":"* Make sure upload and stage commands passes size to API as request body"},{"line_number":142,"context_line":"* Make similar provision in openstack client and sdk"},{"line_number":143,"context_line":"* At Glance side compare provided size with actual size"},{"line_number":144,"context_line":"* Make sure image size is set before import operation starts"},{"line_number":145,"context_line":"* Required functional, unit and tempest tests"},{"line_number":146,"context_line":"* Document new behavior"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1b28a0a3_1ac2cc42","line":143,"in_reply_to":"b1bd3cd6_af53328a","updated":"2025-04-25 16:20:40.000000000","message":"It\u0027s not currently, but it could be allowed :) Like other things, letting it be changed until the image goes ACTIVE seems very much in line with our workflow. But, as you wish.","commit_id":"1e16b9242c0da02ba77d7ba155362c2fc53b1db5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"c24b29cbd2c93e3d94edf74ca3ffdbf7dba0af37","unresolved":false,"context_lines":[{"line_number":51,"context_line":"passed to API."},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"On the API side, we will read the size passed from glanceclient or any"},{"line_number":54,"context_line":"request source via header `x-openstack-image-size` and pass it to the"},{"line_number":55,"context_line":"designated storage backend. This approach will help prevent resize-on-write"},{"line_number":56,"context_line":"operations for Cinder and RBD backends and enable the use of the multipart"},{"line_number":57,"context_line":"upload feature for the S3 backend. After completing the image upload or"}],"source_content_type":"text/x-rst","patch_set":7,"id":"96a6c866_4f945e73","line":54,"range":{"start_line":54,"start_character":27,"end_line":54,"end_character":49},"updated":"2025-05-05 16:14:41.000000000","message":"If new revision is required then I will change this header name to `x-image-meta-size` to be consistent with other headers like, x-image-meta-store. Also x-image-meta-size was the header used in v1 api.","commit_id":"b507d16f5ab274aad903244e7739f5edccdd414b"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"6bd0d895c1cd44126f9d2a50054884fa971e5e9f","unresolved":true,"context_lines":[{"line_number":54,"context_line":"request source via header `x-openstack-image-size` and pass it to the"},{"line_number":55,"context_line":"designated storage backend. This approach will help prevent resize-on-write"},{"line_number":56,"context_line":"operations for Cinder and RBD backends and enable the use of the multipart"},{"line_number":57,"context_line":"upload feature for the S3 backend. After completing the image upload or"},{"line_number":58,"context_line":"staging process, we will compare the actual image size to the size provided"},{"line_number":59,"context_line":"by the user. If there is a discrepancy, we will reject the image upload,"},{"line_number":60,"context_line":"remove the data from the backend, and return an appropriate error message"},{"line_number":61,"context_line":"to the user. Additionally, we will log a warning that aligns with the"},{"line_number":62,"context_line":"existing checks for image disk format and hash."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"If the image size is not included in the request header, we will have no option"},{"line_number":65,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":7,"id":"b457b389_5c5198fc","line":62,"range":{"start_line":57,"start_character":35,"end_line":62,"end_character":47},"updated":"2025-05-07 08:13:13.000000000","message":"If we are going to validate the image size provided with actual image size then we need to make changes at glance_store side for each store to return actual size as it is now returning the size which we are passing as actual size and not the size of data written at the backend.\n\n[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/rbd.py#L664","commit_id":"b507d16f5ab274aad903244e7739f5edccdd414b"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"083167d99c4016fdaeb2a88df1814eb9df03642d","unresolved":true,"context_lines":[{"line_number":54,"context_line":"request source via header `x-openstack-image-size` and pass it to the"},{"line_number":55,"context_line":"designated storage backend. This approach will help prevent resize-on-write"},{"line_number":56,"context_line":"operations for Cinder and RBD backends and enable the use of the multipart"},{"line_number":57,"context_line":"upload feature for the S3 backend. After completing the image upload or"},{"line_number":58,"context_line":"staging process, we will compare the actual image size to the size provided"},{"line_number":59,"context_line":"by the user. If there is a discrepancy, we will reject the image upload,"},{"line_number":60,"context_line":"remove the data from the backend, and return an appropriate error message"},{"line_number":61,"context_line":"to the user. Additionally, we will log a warning that aligns with the"},{"line_number":62,"context_line":"existing checks for image disk format and hash."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"If the image size is not included in the request header, we will have no option"},{"line_number":65,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":7,"id":"9b1fa714_8fde27ef","line":62,"range":{"start_line":57,"start_character":35,"end_line":62,"end_character":47},"in_reply_to":"3c13f6c3_2ac85401","updated":"2025-05-07 15:26:26.000000000","message":"Ack, I will report a separate bug in glance_store for this and then fix store first before starting to work on actual feature. \n\nI also have one question here, Only RBD backend raises \"rbd.IncompleteWriteError: RBD incomplete write (Wrote only 8388608 out of 8394566 bytes)\" and aborts the operation and all other backends writes all the data and then set the claimed_size to the image and image becomes active. \n\nSo different behaviour will be there for rbd and rest of the backend, is it OK? OR I should catch this error at rbd side and retry the upload operation with setting claimed size to 0?","commit_id":"b507d16f5ab274aad903244e7739f5edccdd414b"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"afbf1e5032f6714a8637f6e475bc1d6f9fbc0bd2","unresolved":true,"context_lines":[{"line_number":54,"context_line":"request source via header `x-openstack-image-size` and pass it to the"},{"line_number":55,"context_line":"designated storage backend. This approach will help prevent resize-on-write"},{"line_number":56,"context_line":"operations for Cinder and RBD backends and enable the use of the multipart"},{"line_number":57,"context_line":"upload feature for the S3 backend. After completing the image upload or"},{"line_number":58,"context_line":"staging process, we will compare the actual image size to the size provided"},{"line_number":59,"context_line":"by the user. If there is a discrepancy, we will reject the image upload,"},{"line_number":60,"context_line":"remove the data from the backend, and return an appropriate error message"},{"line_number":61,"context_line":"to the user. Additionally, we will log a warning that aligns with the"},{"line_number":62,"context_line":"existing checks for image disk format and hash."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"If the image size is not included in the request header, we will have no option"},{"line_number":65,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":7,"id":"b8d24fd5_e9dfff80","line":62,"range":{"start_line":57,"start_character":35,"end_line":62,"end_character":47},"in_reply_to":"46b74e9b_42a9da7a","updated":"2025-05-07 14:48:08.000000000","message":"If you are talking about this [1] then we need to change store side to return actual size.\n\nhttps://github.com/openstack/glance/blob/master/glance/location.py#L514\n\nRegarding LimitingReader, are you suggesting we can calculate actual size there and compare it later with the size we passed to store?","commit_id":"b507d16f5ab274aad903244e7739f5edccdd414b"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"9378253c2d19ada648ceba35e1c7e75376913e03","unresolved":true,"context_lines":[{"line_number":54,"context_line":"request source via header `x-openstack-image-size` and pass it to the"},{"line_number":55,"context_line":"designated storage backend. This approach will help prevent resize-on-write"},{"line_number":56,"context_line":"operations for Cinder and RBD backends and enable the use of the multipart"},{"line_number":57,"context_line":"upload feature for the S3 backend. After completing the image upload or"},{"line_number":58,"context_line":"staging process, we will compare the actual image size to the size provided"},{"line_number":59,"context_line":"by the user. If there is a discrepancy, we will reject the image upload,"},{"line_number":60,"context_line":"remove the data from the backend, and return an appropriate error message"},{"line_number":61,"context_line":"to the user. Additionally, we will log a warning that aligns with the"},{"line_number":62,"context_line":"existing checks for image disk format and hash."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"If the image size is not included in the request header, we will have no option"},{"line_number":65,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":7,"id":"d36f09e5_08a9f56d","line":62,"range":{"start_line":57,"start_character":35,"end_line":62,"end_character":47},"in_reply_to":"7ffe1dfe_c01e1839","updated":"2025-05-07 15:00:31.000000000","message":"Yes, that is what we do!!!\nhttps://github.com/openstack/glance_store/blob/master/glance_store/_drivers/rbd.py#L664\n\nWe set actual bytes read as size only if we don\u0027t pass anything (no size) to store.","commit_id":"b507d16f5ab274aad903244e7739f5edccdd414b"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"64ebd2789b67a22cdf62e2fd24d7ac6cb0a1390a","unresolved":true,"context_lines":[{"line_number":54,"context_line":"request source via header `x-openstack-image-size` and pass it to the"},{"line_number":55,"context_line":"designated storage backend. This approach will help prevent resize-on-write"},{"line_number":56,"context_line":"operations for Cinder and RBD backends and enable the use of the multipart"},{"line_number":57,"context_line":"upload feature for the S3 backend. After completing the image upload or"},{"line_number":58,"context_line":"staging process, we will compare the actual image size to the size provided"},{"line_number":59,"context_line":"by the user. If there is a discrepancy, we will reject the image upload,"},{"line_number":60,"context_line":"remove the data from the backend, and return an appropriate error message"},{"line_number":61,"context_line":"to the user. Additionally, we will log a warning that aligns with the"},{"line_number":62,"context_line":"existing checks for image disk format and hash."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"If the image size is not included in the request header, we will have no option"},{"line_number":65,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":7,"id":"e20ea810_8f5e0515","line":62,"range":{"start_line":57,"start_character":35,"end_line":62,"end_character":47},"in_reply_to":"9b1fa714_8fde27ef","updated":"2025-05-07 15:39:07.000000000","message":"I dunno what the best behavior is, but I think it should be consistent across all the backends, of course. And I would expect there to be a test for each that validates the expected behavior...","commit_id":"b507d16f5ab274aad903244e7739f5edccdd414b"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"c2a2df9245e755f65ed8a430303a337ecd48069b","unresolved":true,"context_lines":[{"line_number":54,"context_line":"request source via header `x-openstack-image-size` and pass it to the"},{"line_number":55,"context_line":"designated storage backend. This approach will help prevent resize-on-write"},{"line_number":56,"context_line":"operations for Cinder and RBD backends and enable the use of the multipart"},{"line_number":57,"context_line":"upload feature for the S3 backend. After completing the image upload or"},{"line_number":58,"context_line":"staging process, we will compare the actual image size to the size provided"},{"line_number":59,"context_line":"by the user. If there is a discrepancy, we will reject the image upload,"},{"line_number":60,"context_line":"remove the data from the backend, and return an appropriate error message"},{"line_number":61,"context_line":"to the user. Additionally, we will log a warning that aligns with the"},{"line_number":62,"context_line":"existing checks for image disk format and hash."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"If the image size is not included in the request header, we will have no option"},{"line_number":65,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":7,"id":"46b74e9b_42a9da7a","line":62,"range":{"start_line":57,"start_character":35,"end_line":62,"end_character":47},"in_reply_to":"b457b389_5c5198fc","updated":"2025-05-07 13:34:29.000000000","message":"Or check it in the pipeline as we\u0027re feeding it to store, like the LimitingReader does.\n\nAlso, I was thinking there was code (perhaps leftover from glance v1) that checked that the size matches if it was pre-set on the image (my preference anyway). If that code is still there, perhaps that\u0027s the better option?","commit_id":"b507d16f5ab274aad903244e7739f5edccdd414b"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"08d94fd9e7f845fa50a7cba5372aaaaf596ed942","unresolved":true,"context_lines":[{"line_number":54,"context_line":"request source via header `x-openstack-image-size` and pass it to the"},{"line_number":55,"context_line":"designated storage backend. This approach will help prevent resize-on-write"},{"line_number":56,"context_line":"operations for Cinder and RBD backends and enable the use of the multipart"},{"line_number":57,"context_line":"upload feature for the S3 backend. After completing the image upload or"},{"line_number":58,"context_line":"staging process, we will compare the actual image size to the size provided"},{"line_number":59,"context_line":"by the user. If there is a discrepancy, we will reject the image upload,"},{"line_number":60,"context_line":"remove the data from the backend, and return an appropriate error message"},{"line_number":61,"context_line":"to the user. Additionally, we will log a warning that aligns with the"},{"line_number":62,"context_line":"existing checks for image disk format and hash."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"If the image size is not included in the request header, we will have no option"},{"line_number":65,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":7,"id":"7ffe1dfe_c01e1839","line":62,"range":{"start_line":57,"start_character":35,"end_line":62,"end_character":47},"in_reply_to":"b8d24fd5_e9dfff80","updated":"2025-05-07 14:54:02.000000000","message":"Wait, are you saying that we are currently setting the image size to the thing the user passed (if they did) and not the amount we read? What about when they don\u0027t pass it... is size not the amount read? Why would we be passing size to the store only to have it not verify it and return it back? If that\u0027s the case, then .. yeah, seems like that API is broken and needs fixing!","commit_id":"b507d16f5ab274aad903244e7739f5edccdd414b"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"cc0484bb83698481583593a7301cf2650905e18b","unresolved":false,"context_lines":[{"line_number":54,"context_line":"request source via header `x-openstack-image-size` and pass it to the"},{"line_number":55,"context_line":"designated storage backend. This approach will help prevent resize-on-write"},{"line_number":56,"context_line":"operations for Cinder and RBD backends and enable the use of the multipart"},{"line_number":57,"context_line":"upload feature for the S3 backend. After completing the image upload or"},{"line_number":58,"context_line":"staging process, we will compare the actual image size to the size provided"},{"line_number":59,"context_line":"by the user. If there is a discrepancy, we will reject the image upload,"},{"line_number":60,"context_line":"remove the data from the backend, and return an appropriate error message"},{"line_number":61,"context_line":"to the user. Additionally, we will log a warning that aligns with the"},{"line_number":62,"context_line":"existing checks for image disk format and hash."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"If the image size is not included in the request header, we will have no option"},{"line_number":65,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3919ea39_7cfb2d8c","line":62,"range":{"start_line":57,"start_character":35,"end_line":62,"end_character":47},"in_reply_to":"c9b8477b_4f080c0a","updated":"2025-05-22 17:19:06.000000000","message":"Done","commit_id":"b507d16f5ab274aad903244e7739f5edccdd414b"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"df217a479fc77e03ba87793dca808f723d6fd17c","unresolved":false,"context_lines":[{"line_number":54,"context_line":"request source via header `x-openstack-image-size` and pass it to the"},{"line_number":55,"context_line":"designated storage backend. This approach will help prevent resize-on-write"},{"line_number":56,"context_line":"operations for Cinder and RBD backends and enable the use of the multipart"},{"line_number":57,"context_line":"upload feature for the S3 backend. After completing the image upload or"},{"line_number":58,"context_line":"staging process, we will compare the actual image size to the size provided"},{"line_number":59,"context_line":"by the user. If there is a discrepancy, we will reject the image upload,"},{"line_number":60,"context_line":"remove the data from the backend, and return an appropriate error message"},{"line_number":61,"context_line":"to the user. Additionally, we will log a warning that aligns with the"},{"line_number":62,"context_line":"existing checks for image disk format and hash."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"If the image size is not included in the request header, we will have no option"},{"line_number":65,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":7,"id":"b77b4d4d_ed7870ad","line":62,"range":{"start_line":57,"start_character":35,"end_line":62,"end_character":47},"in_reply_to":"c9b8477b_4f080c0a","updated":"2025-05-22 16:03:15.000000000","message":"Done","commit_id":"b507d16f5ab274aad903244e7739f5edccdd414b"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"e7a9cd6edb3349deb4be3066a32acaf13239c52d","unresolved":true,"context_lines":[{"line_number":54,"context_line":"request source via header `x-openstack-image-size` and pass it to the"},{"line_number":55,"context_line":"designated storage backend. This approach will help prevent resize-on-write"},{"line_number":56,"context_line":"operations for Cinder and RBD backends and enable the use of the multipart"},{"line_number":57,"context_line":"upload feature for the S3 backend. After completing the image upload or"},{"line_number":58,"context_line":"staging process, we will compare the actual image size to the size provided"},{"line_number":59,"context_line":"by the user. If there is a discrepancy, we will reject the image upload,"},{"line_number":60,"context_line":"remove the data from the backend, and return an appropriate error message"},{"line_number":61,"context_line":"to the user. Additionally, we will log a warning that aligns with the"},{"line_number":62,"context_line":"existing checks for image disk format and hash."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"If the image size is not included in the request header, we will have no option"},{"line_number":65,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3c13f6c3_2ac85401","line":62,"range":{"start_line":57,"start_character":35,"end_line":62,"end_character":47},"in_reply_to":"d36f09e5_08a9f56d","updated":"2025-05-07 15:13:50.000000000","message":"That seems extremely broken to me.. So in that case, yeah, I\u0027m supportive of fixing the store drivers in the process here. Then you can just check `claimed_size \u003d\u003d returned_size` in that glance method and abort if wrong.","commit_id":"b507d16f5ab274aad903244e7739f5edccdd414b"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"f8508e735fc4244aeed2801c6f4d5b423d7f0ae0","unresolved":true,"context_lines":[{"line_number":54,"context_line":"request source via header `x-openstack-image-size` and pass it to the"},{"line_number":55,"context_line":"designated storage backend. This approach will help prevent resize-on-write"},{"line_number":56,"context_line":"operations for Cinder and RBD backends and enable the use of the multipart"},{"line_number":57,"context_line":"upload feature for the S3 backend. After completing the image upload or"},{"line_number":58,"context_line":"staging process, we will compare the actual image size to the size provided"},{"line_number":59,"context_line":"by the user. If there is a discrepancy, we will reject the image upload,"},{"line_number":60,"context_line":"remove the data from the backend, and return an appropriate error message"},{"line_number":61,"context_line":"to the user. Additionally, we will log a warning that aligns with the"},{"line_number":62,"context_line":"existing checks for image disk format and hash."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"If the image size is not included in the request header, we will have no option"},{"line_number":65,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":7,"id":"c9b8477b_4f080c0a","line":62,"range":{"start_line":57,"start_character":35,"end_line":62,"end_character":47},"in_reply_to":"e20ea810_8f5e0515","updated":"2025-05-08 05:59:55.000000000","message":"FYI, https://bugs.launchpad.net/glance-store/+bug/2110185","commit_id":"b507d16f5ab274aad903244e7739f5edccdd414b"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"df217a479fc77e03ba87793dca808f723d6fd17c","unresolved":true,"context_lines":[{"line_number":47,"context_line":"specify the image size via the --size command-line option. If the --size"},{"line_number":48,"context_line":"option is provided, we will not calculate the image size internally, and"},{"line_number":49,"context_line":"the user-specified size will be sent directly to the Glance API. We will"},{"line_number":50,"context_line":"introduce a new request header `x-image-meta-size` which will be"},{"line_number":51,"context_line":"passed to API."},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"On the API side, we will read the size passed from glanceclient or any"}],"source_content_type":"text/x-rst","patch_set":8,"id":"5da9bae8_7a0f33ec","line":50,"range":{"start_line":50,"start_character":40,"end_line":50,"end_character":45},"updated":"2025-05-22 16:03:15.000000000","message":"So, I don\u0027t like the use of \"meta\" here especially for an image that also has metadata.\n\nWhy was this changed from `x-openstack-image-size`?","commit_id":"50b79b6a02adda04dc053bd2614c50bc8fc48730"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"cc0484bb83698481583593a7301cf2650905e18b","unresolved":false,"context_lines":[{"line_number":47,"context_line":"specify the image size via the --size command-line option. If the --size"},{"line_number":48,"context_line":"option is provided, we will not calculate the image size internally, and"},{"line_number":49,"context_line":"the user-specified size will be sent directly to the Glance API. We will"},{"line_number":50,"context_line":"introduce a new request header `x-image-meta-size` which will be"},{"line_number":51,"context_line":"passed to API."},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"On the API side, we will read the size passed from glanceclient or any"}],"source_content_type":"text/x-rst","patch_set":8,"id":"53fe3384_3847039b","line":50,"range":{"start_line":50,"start_character":40,"end_line":50,"end_character":45},"in_reply_to":"5da9bae8_7a0f33ec","updated":"2025-05-22 17:19:06.000000000","message":"Acknowledged","commit_id":"50b79b6a02adda04dc053bd2614c50bc8fc48730"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"df217a479fc77e03ba87793dca808f723d6fd17c","unresolved":true,"context_lines":[{"line_number":59,"context_line":"by the user. If there is a discrepancy, we will reject the image upload,"},{"line_number":60,"context_line":"remove the data from the backend, and return an appropriate error message"},{"line_number":61,"context_line":"to the user. Additionally, we will log a warning that aligns with the"},{"line_number":62,"context_line":"existing checks for image disk format and hash."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"If the image size is not included in the request header, we will have no option"},{"line_number":65,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":8,"id":"afed59b2_e7f36fe9","line":62,"updated":"2025-05-22 16:03:15.000000000","message":"Might be worth a comment here that glance_store will check for running over the size limit during the data pipeline so it\u0027s clear that all we\u0027re doing is checking at the end for underruns.","commit_id":"50b79b6a02adda04dc053bd2614c50bc8fc48730"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"cc0484bb83698481583593a7301cf2650905e18b","unresolved":false,"context_lines":[{"line_number":59,"context_line":"by the user. If there is a discrepancy, we will reject the image upload,"},{"line_number":60,"context_line":"remove the data from the backend, and return an appropriate error message"},{"line_number":61,"context_line":"to the user. Additionally, we will log a warning that aligns with the"},{"line_number":62,"context_line":"existing checks for image disk format and hash."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"If the image size is not included in the request header, we will have no option"},{"line_number":65,"context_line":"but to allow the storage backends to perform resize-on-write operations or"}],"source_content_type":"text/x-rst","patch_set":8,"id":"034d0eac_bf5b2289","line":62,"in_reply_to":"afed59b2_e7f36fe9","updated":"2025-05-22 17:19:06.000000000","message":"Acknowledged","commit_id":"50b79b6a02adda04dc053bd2614c50bc8fc48730"},{"author":{"_account_id":5314,"name":"Brian Rosmaita","email":"rosmaita.fossdev@gmail.com","username":"brian-rosmaita"},"change_message_id":"267d4bc8e5e5f63038f50c3776255e4c8bb5686d","unresolved":true,"context_lines":[{"line_number":41,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"We propose to enable the use of the --size option for the image-upload and"},{"line_number":44,"context_line":"image-stage commands, as well as to add this option to the image-create"},{"line_number":45,"context_line":"command. The image size will be calculated automatically if a file-like object"},{"line_number":46,"context_line":"is detected during the upload or staging operations. Otherwise, users can"},{"line_number":47,"context_line":"specify the image size via the --size command-line option. If the --size"},{"line_number":48,"context_line":"option is provided, we will not calculate the image size internally, and"}],"source_content_type":"text/x-rst","patch_set":9,"id":"70952a90_db1ed70d","line":45,"range":{"start_line":44,"start_character":22,"end_line":45,"end_character":7},"updated":"2025-05-28 15:33:42.000000000","message":"There is some confusion here between the Images API and the python-glanceclient/python-openstackclient here.  I think a glance spec should deal in terms of the Images API, and then you can do a follow-up spec for the client changes.\n\nThis is important because the API image-create (POST /v2/images) only creates an image record.  Is the proposal that this call will recognize the x-openstack-image-size header?  If so, what happens when a different value for that header is passed on image-upload (PUT /v2/images/{image_id}/file)?  Also, if we are going to allow specifying the size on POST /v2/images, why not add it to the request body instead of using a header?\n\nI suspect, actually, that you don\u0027t mean to include the size specification for the POST /v2/images call.","commit_id":"40b2ca52d1e7f056d3932f69d8defaf65b7f6197"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"0d46e2546e345ab84eb63965289fe80314db8135","unresolved":true,"context_lines":[{"line_number":41,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"We propose to enable the use of the --size option for the image-upload and"},{"line_number":44,"context_line":"image-stage commands, as well as to add this option to the image-create"},{"line_number":45,"context_line":"command. The image size will be calculated automatically if a file-like object"},{"line_number":46,"context_line":"is detected during the upload or staging operations. Otherwise, users can"},{"line_number":47,"context_line":"specify the image size via the --size command-line option. If the --size"},{"line_number":48,"context_line":"option is provided, we will not calculate the image size internally, and"}],"source_content_type":"text/x-rst","patch_set":9,"id":"3f92a843_14f3114c","line":45,"range":{"start_line":44,"start_character":22,"end_line":45,"end_character":7},"in_reply_to":"1966cb43_cb4272e6","updated":"2025-05-28 15:52:45.000000000","message":"Well, to be clear: my preference is that we do provide the size in the _body_ during the POST, and not do the header thing. I\u0027m sure I said that in a previous comment (or perhaps online). It seems much better and more obvious to me than using the header approach, although I understand the latter is maybe easier.","commit_id":"40b2ca52d1e7f056d3932f69d8defaf65b7f6197"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"cba7e061fa8bb1a149cb92b459770a70e428e5aa","unresolved":true,"context_lines":[{"line_number":41,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"We propose to enable the use of the --size option for the image-upload and"},{"line_number":44,"context_line":"image-stage commands, as well as to add this option to the image-create"},{"line_number":45,"context_line":"command. The image size will be calculated automatically if a file-like object"},{"line_number":46,"context_line":"is detected during the upload or staging operations. Otherwise, users can"},{"line_number":47,"context_line":"specify the image size via the --size command-line option. If the --size"},{"line_number":48,"context_line":"option is provided, we will not calculate the image size internally, and"}],"source_content_type":"text/x-rst","patch_set":9,"id":"9d9189a5_b240fb0b","line":45,"range":{"start_line":44,"start_character":22,"end_line":45,"end_character":7},"in_reply_to":"31f11201_f46e7cd8","updated":"2025-05-28 16:16:29.000000000","message":"The declared size should always win over the actual data length, yes. It\u0027s a safety check. If we allow it in the body *and* a header, then I\u0027d say the first one wins, which would mean if you declared 10 in the body, then upload with header set to 5, we abort immediately, just like we do if you claimed 10 and write 11 or 8.\n\nHowever, to be clear, Abhi is not proposing adding it in the body, only the header. I was just stating my preference.","commit_id":"40b2ca52d1e7f056d3932f69d8defaf65b7f6197"},{"author":{"_account_id":5314,"name":"Brian Rosmaita","email":"rosmaita.fossdev@gmail.com","username":"brian-rosmaita"},"change_message_id":"fb8d62cbe2c1892e225143f2e2a35bb7561289b8","unresolved":true,"context_lines":[{"line_number":41,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"We propose to enable the use of the --size option for the image-upload and"},{"line_number":44,"context_line":"image-stage commands, as well as to add this option to the image-create"},{"line_number":45,"context_line":"command. The image size will be calculated automatically if a file-like object"},{"line_number":46,"context_line":"is detected during the upload or staging operations. Otherwise, users can"},{"line_number":47,"context_line":"specify the image size via the --size command-line option. If the --size"},{"line_number":48,"context_line":"option is provided, we will not calculate the image size internally, and"}],"source_content_type":"text/x-rst","patch_set":9,"id":"31f11201_f46e7cd8","line":45,"range":{"start_line":44,"start_character":22,"end_line":45,"end_character":7},"in_reply_to":"3f92a843_14f3114c","updated":"2025-05-28 16:08:05.000000000","message":"I guess in that case, the body size wins (in case of a conflict with the PUT /file)?","commit_id":"40b2ca52d1e7f056d3932f69d8defaf65b7f6197"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"9a6a59a3cf547ba3cf8e137bd5fccda480a10041","unresolved":true,"context_lines":[{"line_number":41,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"We propose to enable the use of the --size option for the image-upload and"},{"line_number":44,"context_line":"image-stage commands, as well as to add this option to the image-create"},{"line_number":45,"context_line":"command. The image size will be calculated automatically if a file-like object"},{"line_number":46,"context_line":"is detected during the upload or staging operations. Otherwise, users can"},{"line_number":47,"context_line":"specify the image size via the --size command-line option. If the --size"},{"line_number":48,"context_line":"option is provided, we will not calculate the image size internally, and"}],"source_content_type":"text/x-rst","patch_set":9,"id":"1966cb43_cb4272e6","line":45,"range":{"start_line":44,"start_character":22,"end_line":45,"end_character":7},"in_reply_to":"70952a90_db1ed70d","updated":"2025-05-28 15:37:25.000000000","message":"We don\u0027t want to include size for the POST","commit_id":"40b2ca52d1e7f056d3932f69d8defaf65b7f6197"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"22d1946da2ec48f9538a499ddfce76d6519a9120","unresolved":false,"context_lines":[{"line_number":41,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"We propose to enable the use of the --size option for the image-upload and"},{"line_number":44,"context_line":"image-stage commands, as well as to add this option to the image-create"},{"line_number":45,"context_line":"command. The image size will be calculated automatically if a file-like object"},{"line_number":46,"context_line":"is detected during the upload or staging operations. Otherwise, users can"},{"line_number":47,"context_line":"specify the image size via the --size command-line option. If the --size"},{"line_number":48,"context_line":"option is provided, we will not calculate the image size internally, and"}],"source_content_type":"text/x-rst","patch_set":9,"id":"72deabac_f01e55f8","line":45,"range":{"start_line":44,"start_character":22,"end_line":45,"end_character":7},"in_reply_to":"9d9189a5_b240fb0b","updated":"2025-05-29 05:04:47.000000000","message":"Agree, My preference is to have it as a header and avoid any changes to POST /v2/images to keep it simple and able to solve our use case.","commit_id":"40b2ca52d1e7f056d3932f69d8defaf65b7f6197"},{"author":{"_account_id":5314,"name":"Brian Rosmaita","email":"rosmaita.fossdev@gmail.com","username":"brian-rosmaita"},"change_message_id":"267d4bc8e5e5f63038f50c3776255e4c8bb5686d","unresolved":true,"context_lines":[{"line_number":92,"context_line":""},{"line_number":93,"context_line":"REST API impact"},{"line_number":94,"context_line":"---------------"},{"line_number":95,"context_line":""},{"line_number":96,"context_line":"HTTP 400 resonse will be returned if user provided size does not matches"},{"line_number":97,"context_line":"with actual image size."},{"line_number":98,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"36b5c52e_d4dc3438","line":95,"updated":"2025-05-28 15:33:42.000000000","message":"The POST /v2/images , PUT /v2/images/{image_id}/file , and PUT /v2/images/{image_id}/stage calls will be modified to recognize the `x-openstack-image-size` header.  (Although see my comment above about POST /v2/images)","commit_id":"40b2ca52d1e7f056d3932f69d8defaf65b7f6197"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"9a6a59a3cf547ba3cf8e137bd5fccda480a10041","unresolved":true,"context_lines":[{"line_number":92,"context_line":""},{"line_number":93,"context_line":"REST API impact"},{"line_number":94,"context_line":"---------------"},{"line_number":95,"context_line":""},{"line_number":96,"context_line":"HTTP 400 resonse will be returned if user provided size does not matches"},{"line_number":97,"context_line":"with actual image size."},{"line_number":98,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"9a3a5189_fe09ca41","line":95,"in_reply_to":"36b5c52e_d4dc3438","updated":"2025-05-28 15:37:25.000000000","message":"We will not change POST just PUT","commit_id":"40b2ca52d1e7f056d3932f69d8defaf65b7f6197"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"22d1946da2ec48f9538a499ddfce76d6519a9120","unresolved":false,"context_lines":[{"line_number":92,"context_line":""},{"line_number":93,"context_line":"REST API impact"},{"line_number":94,"context_line":"---------------"},{"line_number":95,"context_line":""},{"line_number":96,"context_line":"HTTP 400 resonse will be returned if user provided size does not matches"},{"line_number":97,"context_line":"with actual image size."},{"line_number":98,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"837ddb79_2f4de2e7","line":95,"in_reply_to":"9a3a5189_fe09ca41","updated":"2025-05-29 05:04:47.000000000","message":"Done","commit_id":"40b2ca52d1e7f056d3932f69d8defaf65b7f6197"},{"author":{"_account_id":5314,"name":"Brian Rosmaita","email":"rosmaita.fossdev@gmail.com","username":"brian-rosmaita"},"change_message_id":"267d4bc8e5e5f63038f50c3776255e4c8bb5686d","unresolved":true,"context_lines":[{"line_number":95,"context_line":""},{"line_number":96,"context_line":"HTTP 400 resonse will be returned if user provided size does not matches"},{"line_number":97,"context_line":"with actual image size."},{"line_number":98,"context_line":""},{"line_number":99,"context_line":"Security impact"},{"line_number":100,"context_line":"---------------"},{"line_number":101,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"82a7f714_b5348401","line":98,"updated":"2025-05-28 15:33:42.000000000","message":"Is there any impact on the Locations API?  What I mean is, do we want to allow setting the image size when setting the location?","commit_id":"40b2ca52d1e7f056d3932f69d8defaf65b7f6197"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"22d1946da2ec48f9538a499ddfce76d6519a9120","unresolved":false,"context_lines":[{"line_number":95,"context_line":""},{"line_number":96,"context_line":"HTTP 400 resonse will be returned if user provided size does not matches"},{"line_number":97,"context_line":"with actual image size."},{"line_number":98,"context_line":""},{"line_number":99,"context_line":"Security impact"},{"line_number":100,"context_line":"---------------"},{"line_number":101,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"7ec48951_6b730a3f","line":98,"in_reply_to":"4bb7643f_5847f5a7","updated":"2025-05-29 05:04:47.000000000","message":"Done","commit_id":"40b2ca52d1e7f056d3932f69d8defaf65b7f6197"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"9a6a59a3cf547ba3cf8e137bd5fccda480a10041","unresolved":true,"context_lines":[{"line_number":95,"context_line":""},{"line_number":96,"context_line":"HTTP 400 resonse will be returned if user provided size does not matches"},{"line_number":97,"context_line":"with actual image size."},{"line_number":98,"context_line":""},{"line_number":99,"context_line":"Security impact"},{"line_number":100,"context_line":"---------------"},{"line_number":101,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"4bb7643f_5847f5a7","line":98,"in_reply_to":"82a7f714_b5348401","updated":"2025-05-28 15:37:25.000000000","message":"No","commit_id":"40b2ca52d1e7f056d3932f69d8defaf65b7f6197"},{"author":{"_account_id":5314,"name":"Brian Rosmaita","email":"rosmaita.fossdev@gmail.com","username":"brian-rosmaita"},"change_message_id":"8d7c973982d4162b5c49b3c1ccc83d9e1e2f5ebc","unresolved":true,"context_lines":[{"line_number":79,"context_line":""},{"line_number":80,"context_line":"The Python Requests library also facilitates streaming uploads by sending"},{"line_number":81,"context_line":"data in chunks with a \"Content-Length: \u003cfile size\u003e\" header, rather than"},{"line_number":82,"context_line":"using \"Transfer-Encoding: chunked.\" This approach avoids chunked transfer,"},{"line_number":83,"context_line":"which we should steer clear of, as it is a more stable method for long"},{"line_number":84,"context_line":"transfers. Since the development of the glanceclient library is currently"},{"line_number":85,"context_line":"at a standstill, we should avoid introducing changes that could lead to"},{"line_number":86,"context_line":"instability."},{"line_number":87,"context_line":""}],"source_content_type":"text/x-rst","patch_set":10,"id":"8a1e5e16_889b38d9","line":84,"range":{"start_line":82,"start_character":36,"end_line":84,"end_character":10},"updated":"2025-05-29 12:54:10.000000000","message":"I\u0027m not going to hold up the spec over this because it\u0027s just the Alternative, but I find this sentence very confusing.  What does the \"This\" at the beginning of the sentence refer to, the Requests streaming approach, or the proposal discussed in this spec?  What do we want to \"steer clear of\" (I guess using streaming? seems counterintuitive and non-modern) and what is the \"more stable method\" (I guess chunked transfer, because it\u0027s more easily resumable, and what we currently have implemented?).  Also, I\u0027m confused about the \"Content-Length\" header ... my understanding from the discussion on PS4 and Melanie\u0027s summary of the RFC is that it\u0027s supposed to be the full content length and that the individual chunk sizes are encoded as prefix to the data load for each chunk, but Glance is currently using the Content-Length to indicate the size of each chunk as it\u0027s uploaded, with the payload consisting of pure data (no size prefix).  It doesn\u0027t really matter unless someone is going to write their own upload client, in which case we should document this somewhere for when osc/sdk get refactored.","commit_id":"e0eb7ac13429931622884f0add9548571988c1bf"}],"specs/2025.2/approved/python-glanceclient/by-pass-chunked-transfer.rst":[{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"b30b68c4a7f156e158024e14d09ab8dcd6d5e480","unresolved":true,"context_lines":[{"line_number":48,"context_line":"multipart-upload feature for s3 backend."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"To enable this, we are proposing to add new command line option"},{"line_number":51,"context_line":"`--by-pass-chunked-transfer` defautl False to `image-create`, `image-upload`, `image-stage`,"},{"line_number":52,"context_line":"`image-create-via-import` commands. If this option is True then we can stream the data"},{"line_number":53,"context_line":"without reading them into memory. "},{"line_number":54,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"31fb1fef_3f3a6691","line":51,"range":{"start_line":51,"start_character":3,"end_line":51,"end_character":10},"updated":"2025-04-16 18:38:10.000000000","message":"In English `bypass` is one word so I would make this `--bypass-chunked-transfer`.","commit_id":"0c3cf8a4516e8d1ab5a22ef56e63f7a2caea55b9"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"ed0e2efc272976b38471a83a966be26cd5f16435","unresolved":false,"context_lines":[{"line_number":48,"context_line":"multipart-upload feature for s3 backend."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"To enable this, we are proposing to add new command line option"},{"line_number":51,"context_line":"`--by-pass-chunked-transfer` defautl False to `image-create`, `image-upload`, `image-stage`,"},{"line_number":52,"context_line":"`image-create-via-import` commands. If this option is True then we can stream the data"},{"line_number":53,"context_line":"without reading them into memory. "},{"line_number":54,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"c090f493_94c047a8","line":51,"range":{"start_line":51,"start_character":3,"end_line":51,"end_character":10},"in_reply_to":"31fb1fef_3f3a6691","updated":"2025-04-17 06:09:59.000000000","message":"Acknowledged","commit_id":"0c3cf8a4516e8d1ab5a22ef56e63f7a2caea55b9"},{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"b30b68c4a7f156e158024e14d09ab8dcd6d5e480","unresolved":true,"context_lines":[{"line_number":50,"context_line":"To enable this, we are proposing to add new command line option"},{"line_number":51,"context_line":"`--by-pass-chunked-transfer` defautl False to `image-create`, `image-upload`, `image-stage`,"},{"line_number":52,"context_line":"`image-create-via-import` commands. If this option is True then we can stream the data"},{"line_number":53,"context_line":"without reading them into memory. "},{"line_number":54,"context_line":""},{"line_number":55,"context_line":"Alternatives"},{"line_number":56,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"8ddea2e4_69a91485","line":53,"updated":"2025-04-16 18:38:10.000000000","message":"@akekane@redhat.com a question was raised about the Glance server side of bypassing chunked transfer and whether or not the server will avoid loading the entire image into memory when it writes data to the Cinder volume if it knows the size upfront? I had thought not but wanted to ask you to make sure. The concern was about whether or not we will also need any work on the Glance server side as part of this spec.","commit_id":"0c3cf8a4516e8d1ab5a22ef56e63f7a2caea55b9"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"eda745a933c9840e28e39f93675056422e678348","unresolved":false,"context_lines":[{"line_number":50,"context_line":"To enable this, we are proposing to add new command line option"},{"line_number":51,"context_line":"`--by-pass-chunked-transfer` defautl False to `image-create`, `image-upload`, `image-stage`,"},{"line_number":52,"context_line":"`image-create-via-import` commands. If this option is True then we can stream the data"},{"line_number":53,"context_line":"without reading them into memory. "},{"line_number":54,"context_line":""},{"line_number":55,"context_line":"Alternatives"},{"line_number":56,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"f15735cd_275c3487","line":53,"in_reply_to":"4a6c3254_a5fa83ff","updated":"2025-04-21 14:06:56.000000000","message":"Acknowledged","commit_id":"0c3cf8a4516e8d1ab5a22ef56e63f7a2caea55b9"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"ed0e2efc272976b38471a83a966be26cd5f16435","unresolved":true,"context_lines":[{"line_number":50,"context_line":"To enable this, we are proposing to add new command line option"},{"line_number":51,"context_line":"`--by-pass-chunked-transfer` defautl False to `image-create`, `image-upload`, `image-stage`,"},{"line_number":52,"context_line":"`image-create-via-import` commands. If this option is True then we can stream the data"},{"line_number":53,"context_line":"without reading them into memory. "},{"line_number":54,"context_line":""},{"line_number":55,"context_line":"Alternatives"},{"line_number":56,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"d1cbc3d8_c442e9d3","line":53,"in_reply_to":"8ddea2e4_69a91485","updated":"2025-04-17 06:09:59.000000000","message":"I have tested this with rbd driver [1] where it was also resizing the volume if image size is not known in advance, now without making any changes in glance it does avoid resizing the image and creates the volume with actual size in advance.\n\nSample log:\nApr 17 05:31:04 akekane-uwsgi devstack@g-api.service[2184323]: DEBUG glance_store._drivers.rbd [None req-6a725911-f548-4168-a51e-10d269299fe4 admin admin] creating image 18fafcb8-b5f1-4841-82e0-2fe35d16215a with order 23 and size 2147483648 {{(pid\u003d2184323) add /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/_drivers/rbd.py:586}}\n\n[1] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/rbd.py#L586\n\n\nAlso it does transfer data in chunks so it does not load the entire data into memory.","commit_id":"0c3cf8a4516e8d1ab5a22ef56e63f7a2caea55b9"},{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"b95b66cd3fdae7ff972658216ae96a4f6af273b5","unresolved":true,"context_lines":[{"line_number":50,"context_line":"To enable this, we are proposing to add new command line option"},{"line_number":51,"context_line":"`--by-pass-chunked-transfer` defautl False to `image-create`, `image-upload`, `image-stage`,"},{"line_number":52,"context_line":"`image-create-via-import` commands. If this option is True then we can stream the data"},{"line_number":53,"context_line":"without reading them into memory. "},{"line_number":54,"context_line":""},{"line_number":55,"context_line":"Alternatives"},{"line_number":56,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"4a6c3254_a5fa83ff","line":53,"in_reply_to":"d1cbc3d8_c442e9d3","updated":"2025-04-17 23:02:17.000000000","message":"OK that\u0027s great, thanks for confirming 🙂","commit_id":"0c3cf8a4516e8d1ab5a22ef56e63f7a2caea55b9"},{"author":{"_account_id":4690,"name":"melanie witt","display_name":"melwitt","email":"melwittt@gmail.com","username":"melwitt"},"change_message_id":"b95b66cd3fdae7ff972658216ae96a4f6af273b5","unresolved":true,"context_lines":[{"line_number":8,"context_line":"Make provision to bypass chunked transfer encoding"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/python-glanceclient/+spec/bypass-chunked-transfer"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Historically, python-glanceclient uploads image data with"},{"line_number":14,"context_line":"\"Transer-Encoding: chunked\" with no choice to stream image data without the HTTP chunked"}],"source_content_type":"text/x-rst","patch_set":2,"id":"0e498dcc_62bf9272","line":11,"updated":"2025-04-17 23:02:17.000000000","message":"Reminder to rename the blueprint to match \"bypass\" as one word, if needed.","commit_id":"ead01b64d1973d87470dac570362d9ee12203dd4"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"eda745a933c9840e28e39f93675056422e678348","unresolved":false,"context_lines":[{"line_number":8,"context_line":"Make provision to bypass chunked transfer encoding"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/python-glanceclient/+spec/bypass-chunked-transfer"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Historically, python-glanceclient uploads image data with"},{"line_number":14,"context_line":"\"Transer-Encoding: chunked\" with no choice to stream image data without the HTTP chunked"}],"source_content_type":"text/x-rst","patch_set":2,"id":"91d1e76c_a228a455","line":11,"in_reply_to":"0e498dcc_62bf9272","updated":"2025-04-21 14:06:56.000000000","message":"Acknowledged","commit_id":"ead01b64d1973d87470dac570362d9ee12203dd4"},{"author":{"_account_id":25402,"name":"Francesco Pantano","email":"fpantano@redhat.com","username":"fmount"},"change_message_id":"7f94f62b008e8d991e52fc29b3f71eea0ea943f9","unresolved":true,"context_lines":[{"line_number":48,"context_line":"multipart-upload feature for s3 backend."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"To enable this, we are proposing to add new command line option"},{"line_number":51,"context_line":"`--bypass-chunked-transfer` defautl False to `image-create`, `image-upload`, `image-stage`,"},{"line_number":52,"context_line":"`image-create-via-import` commands. If this option is True then we can stream the data"},{"line_number":53,"context_line":"without reading them into memory. "},{"line_number":54,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"e815a738_596f0c4b","line":51,"updated":"2025-04-17 08:00:56.000000000","message":"typo: s/defautl/default","commit_id":"ead01b64d1973d87470dac570362d9ee12203dd4"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"eda745a933c9840e28e39f93675056422e678348","unresolved":false,"context_lines":[{"line_number":48,"context_line":"multipart-upload feature for s3 backend."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"To enable this, we are proposing to add new command line option"},{"line_number":51,"context_line":"`--bypass-chunked-transfer` defautl False to `image-create`, `image-upload`, `image-stage`,"},{"line_number":52,"context_line":"`image-create-via-import` commands. If this option is True then we can stream the data"},{"line_number":53,"context_line":"without reading them into memory. "},{"line_number":54,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"82e04c66_72ffc45d","line":51,"in_reply_to":"e815a738_596f0c4b","updated":"2025-04-21 14:06:56.000000000","message":"Acknowledged","commit_id":"ead01b64d1973d87470dac570362d9ee12203dd4"}],"specs/2025.2/approved/python-glanceclient/bypass-chunked-transfer.rst":[{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"6606d66fd339697879dac928afdd9d07114afc35","unresolved":true,"context_lines":[{"line_number":11,"context_line":"https://blueprints.launchpad.net/python-glanceclient/+spec/bypass-chunked-transfer"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Historically, python-glanceclient uploads image data with"},{"line_number":14,"context_line":"\"Transer-Encoding: chunked\" with no choice to stream image data without the HTTP chunked"},{"line_number":15,"context_line":"transfer mechanism. The Python requests library knows to do this based on whether it"},{"line_number":16,"context_line":"receives a Generator type as the upload data."},{"line_number":17,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"d4bdcb45_e95ea1e9","line":14,"updated":"2025-04-21 13:40:40.000000000","message":"Can you wrap this at 80 cols? It\u0027s very hard to review in place like this.","commit_id":"58feea65a4527162b27ff8d4438f55b657f408c9"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"eda745a933c9840e28e39f93675056422e678348","unresolved":false,"context_lines":[{"line_number":11,"context_line":"https://blueprints.launchpad.net/python-glanceclient/+spec/bypass-chunked-transfer"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Historically, python-glanceclient uploads image data with"},{"line_number":14,"context_line":"\"Transer-Encoding: chunked\" with no choice to stream image data without the HTTP chunked"},{"line_number":15,"context_line":"transfer mechanism. The Python requests library knows to do this based on whether it"},{"line_number":16,"context_line":"receives a Generator type as the upload data."},{"line_number":17,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"2763f61c_fd11d38e","line":14,"in_reply_to":"d4bdcb45_e95ea1e9","updated":"2025-04-21 14:06:56.000000000","message":"Acknowledged","commit_id":"58feea65a4527162b27ff8d4438f55b657f408c9"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"6606d66fd339697879dac928afdd9d07114afc35","unresolved":true,"context_lines":[{"line_number":55,"context_line":"Alternatives"},{"line_number":56,"context_line":"------------"},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"None"},{"line_number":59,"context_line":""},{"line_number":60,"context_line":"Data model impact"},{"line_number":61,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"7f86553c_a8203a4b","line":58,"updated":"2025-04-21 13:40:40.000000000","message":"What about just allowing glance to be told the content-length through a different mechanism? When I looked at this briefly I was surprised to find that we *only* declare the size through that header. What about just letting you set size on the image, but only if it\u0027s in QUEUED state, and when you upload, we refuse to go to ACTIVE if the size doesn\u0027t match what you declared? That seems like it would be really simple, and would enable you to use whatever transfer mechanism/encoding you want for the actual data movement.\n\nAlternately we could use something else like os-glance-anticipated-size (or some better name) but I think that just doing the set-before-upload pattern would be the most straightforward.","commit_id":"58feea65a4527162b27ff8d4438f55b657f408c9"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"0e1ffa098ab5b8f614e55c5fc944125ebaba5030","unresolved":true,"context_lines":[{"line_number":55,"context_line":"Alternatives"},{"line_number":56,"context_line":"------------"},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"None"},{"line_number":59,"context_line":""},{"line_number":60,"context_line":"Data model impact"},{"line_number":61,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"c01c0784_ab96d4ad","line":58,"in_reply_to":"0068ab93_d7f6df4c","updated":"2025-04-21 14:12:26.000000000","message":"Additional overhead for which user? For a shell user passing the filename to the client, it should just be automatic. For Nova, we already know the size of the thing we want to upload. When I was first looking at this, I was trying to figure out _how_ to provide glance with the size other than just from the file object. So it may be more \"overhead\" for nova, but I think it\u0027s what we _want_ to do...","commit_id":"58feea65a4527162b27ff8d4438f55b657f408c9"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2af00ac1ef1823ad6fccd5e14b034aed9445cb78","unresolved":true,"context_lines":[{"line_number":55,"context_line":"Alternatives"},{"line_number":56,"context_line":"------------"},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"None"},{"line_number":59,"context_line":""},{"line_number":60,"context_line":"Data model impact"},{"line_number":61,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"fe505e1f_5eb6d01c","line":58,"in_reply_to":"2836b956_4055d533","updated":"2025-04-21 14:39:26.000000000","message":"\u003e I am little confused here, by your latest comment it sounds that client will calculate the size and pass it to the glance and then after upload is complete and size mismatch we will reject the image? \n\nYes. This is how hashing works right? We declare, upload or import, then we check and fail if it doesn\u0027t match?\n\nWhat I\u0027m saying is the *caller* of the glanceclient functionality would declare the length first (however that needs to happen, but setting it on the image first seems correct to me) and then upload the data (which must match that size). The *user* (i.e. the human) would never see any of this, nor need to do anything differently. If they\u0027re uploading from the shell client, the shell client could/would do that if necessary based on the image file the human passed. Nova would always just set the size ahead of time. This eliminates the need for the caller (either human via the shell or nova via the API) to know the details of when, how, or why we should be using one type of transfer encoding or not, nor what kind of backend glance is using that may have that effect. If I\u0027m fully external, I don\u0027t know if glance is backed by ceph, cinder, or file. I shouldn\u0027t *have* to know that in order to select the right type of transfer mechanism. I think the proposal as it is here just \"fixes\" the specific problem of a backend not being compatible with chunked transfer, without addressing the larger problem of why the caller (human or nova) should need to care about those interactions.\n\n\u003e I think for v1 we were doing that (calculate size at client side) but it was dropped during v2.\n\nRight, I know v1 could do this, because we have remnants of code around that allowed it. I\u0027m not sure why v2 removed it, other than perhaps it seeming to be cleaner and more RESTful to take that header from HTTP instead. Were cinder and ceph present in the tree at the time? Did they support resize during upload and if so did we realize how slow that was? If you have some record of a strong reason to eliminate the \"declare size before upload\" behavior then that might be good reference material. But from where I\u0027m sitting, it seems like it\u0027s the right thing to do.","commit_id":"58feea65a4527162b27ff8d4438f55b657f408c9"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"eda745a933c9840e28e39f93675056422e678348","unresolved":true,"context_lines":[{"line_number":55,"context_line":"Alternatives"},{"line_number":56,"context_line":"------------"},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"None"},{"line_number":59,"context_line":""},{"line_number":60,"context_line":"Data model impact"},{"line_number":61,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"0068ab93_d7f6df4c","line":58,"in_reply_to":"7f86553c_a8203a4b","updated":"2025-04-21 14:06:56.000000000","message":"I think that will be additional overhead for the user, but as an additional way to above proposed change, I think we can do that as separate effort.","commit_id":"58feea65a4527162b27ff8d4438f55b657f408c9"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"9f2c91c3744095dde5417ea16fbf33c67071fd76","unresolved":true,"context_lines":[{"line_number":55,"context_line":"Alternatives"},{"line_number":56,"context_line":"------------"},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"None"},{"line_number":59,"context_line":""},{"line_number":60,"context_line":"Data model impact"},{"line_number":61,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"2836b956_4055d533","line":58,"in_reply_to":"c01c0784_ab96d4ad","updated":"2025-04-21 14:24:39.000000000","message":"I am little confused here, by your latest comment it sounds that client will calculate the size and pass it to the glance and then after upload is complete and size mismatch we will reject the image? By your first comment I thought that end user who will upload the image will set the size in advance and then we will reject it if it not matches.\n\nI think for v1 we were doing that (calculate size at client side) but it was dropped during v2.","commit_id":"58feea65a4527162b27ff8d4438f55b657f408c9"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"e515c905080fe11ac0aa911cb69b96352ea4fcb7","unresolved":false,"context_lines":[{"line_number":55,"context_line":"Alternatives"},{"line_number":56,"context_line":"------------"},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"None"},{"line_number":59,"context_line":""},{"line_number":60,"context_line":"Data model impact"},{"line_number":61,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"f0d64558_c0a6d963","line":58,"in_reply_to":"fe505e1f_5eb6d01c","updated":"2025-04-28 06:38:59.000000000","message":"Done","commit_id":"58feea65a4527162b27ff8d4438f55b657f408c9"}]}