)]}'
{"swift/common/utils.py":[{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"27325e6a5cf166f7ad21d9c89e5f938277f10e19","unresolved":false,"context_lines":[{"line_number":3426,"context_line":"            try:"},{"line_number":3427,"context_line":"                line \u003d line.decode(\u0027utf-8\u0027)"},{"line_number":3428,"context_line":"            except UnicodeDecodeError:"},{"line_number":3429,"context_line":"                line \u003d line.decode(\u0027latin1\u0027)"},{"line_number":3430,"context_line":"        headers.append(line)"},{"line_number":3431,"context_line":"        if done:"},{"line_number":3432,"context_line":"            break"}],"source_content_type":"text/x-python","patch_set":3,"id":"3a57f1b5_a3f05492","line":3429,"updated":"2016-02-11 18:15:20.000000000","message":"In general, making assumptions about header encodings seems dangerous. Suppose I want to set a tempurl key; there are two good ways to choose the key:\n\n* I randomly generate the key, with something like\n\n     \u0027\u0027.join(chr(random.randrange(255)) for _ in xrange(16))\n\n* I select my own key. Being reasonably clever and having noticed that my keyboard makes it easy to include letters like é, I set it to something like \u0027séçrët\u0027. I do it through curl and my shell uses utf-8, so the raw header comes through as b\u0027s\\xc3\\xa9\\xc3\\xa7r\\xc3\\xabt\u0027.\n\nEither works fine currently (in py2) because everything is bytes; we just pass the key straight into our hmac. However, if my cluster operator upgrades to py3, my key is maybe treated as utf-8, maybe treated as latin-1. The tempurl middleware just gets unicode; since it can\u0027t know what the original form was, it has to just *guess* the appropriate encoding.\n\nIf we assume randomly-generated keys are fairly common, latin-1 is the safer bet. (There\u0027s still the chance that my random string would actually decode as utf-8, but the odds of it *also* being able to re-encode as latin-1 should be wuite low.) But the manually-chosen key is out of luck; it comes through as b\u0027s\\xe9\\xe7r\\xebt\u0027 instead of b\u0027s\\xc3\\xa9\\xc3\\xa7r\\xc3\\xabt\u0027. And when I suddenly find myself unable to access my data, I\u0027m going to have a very hard time understanding why.\n\nThat said, this function only seems to be used in multi-part documents, where we expect few headers and the chance of non-ASCII data seems low. I mention it because this is definitely one of the headaches we need to consider as we try to add py3 support.","commit_id":"d9b22ac51cca579e97ec7bd5a6df3ad7bf56c3bf"},{"author":{"_account_id":9107,"name":"Victor Stinner","email":"vstinner@redhat.com","username":"haypo"},"change_message_id":"76a6874e308e07954d728e428ecfd864db648788","unresolved":false,"context_lines":[{"line_number":3426,"context_line":"            try:"},{"line_number":3427,"context_line":"                line \u003d line.decode(\u0027utf-8\u0027)"},{"line_number":3428,"context_line":"            except UnicodeDecodeError:"},{"line_number":3429,"context_line":"                line \u003d line.decode(\u0027latin1\u0027)"},{"line_number":3430,"context_line":"        headers.append(line)"},{"line_number":3431,"context_line":"        if done:"},{"line_number":3432,"context_line":"            break"}],"source_content_type":"text/x-python","patch_set":3,"id":"3a57f1b5_ff055c10","line":3429,"in_reply_to":"3a57f1b5_a3f05492","updated":"2016-02-12 10:59:30.000000000","message":"The HTTP protocol is very well defined. HTTP headers must be encoded to Latin1. If you want to use characters not encodable to Latin1, you must escape it using MIME encoding format like \u003d?UTF-8?Q?...?\u003d.\n\nOk, this is the theory. Now the practice: Swift doesn\u0027t respect the HTTP RFCs and encode headers to UTF-8. So I had to implement this heuristic to try UTF-8 or fallback to Latin1.\n\nThe correct fix is to modify Swift to respect the HTTP RFCs. See for example the Python client:\nhttps://github.com/openstack/python-swiftclient/blob/master/tests/unit/test_swiftclient.py#L118\n\nI don\u0027t know how other Swift clients encode HTTP headers :-/\n\nBut I suggest to fix the root cause later. I\u0027m sorry, but I\u0027m unable to fix all issues at once. I\u0027m working by writing increment changes to port unit tests one by one. Right now, it\u0027s just not possible to use these functions on Python 3, tests fail.\n\nThe idea is to reach a first milestone where unit tests pass, and later discuss how to enhance the code to handle \"correctly\" encodings. There is a similar discussion on my hmac patch:\nhttps://review.openstack.org/#/c/236998/","commit_id":"d9b22ac51cca579e97ec7bd5a6df3ad7bf56c3bf"}]}