)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"24a5e94243916779494fa1aa392b60aaf7abadef","unresolved":false,"context_lines":[{"line_number":18,"context_line":"setting the locale and thus we need to filter/modify the stream such"},{"line_number":19,"context_line":"that it doesn\u0027t crash."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"[1] Traceback:"},{"line_number":22,"context_line":"2018-03-01 09:58:00.515 | Traceback (most recent call last):"},{"line_number":23,"context_line":"2018-03-01 09:58:00.515 |   File \"/usr/local/bin/package-installs-v2\", line 137, in \u003cmodule\u003e"},{"line_number":24,"context_line":"2018-03-01 09:58:00.515 |     main()"},{"line_number":25,"context_line":"2018-03-01 09:58:00.515 |   File \"/usr/local/bin/package-installs-v2\", line 130, in main"},{"line_number":26,"context_line":"2018-03-01 09:58:00.515 |     process_output(install_args, follow\u003dTrue)"},{"line_number":27,"context_line":"2018-03-01 09:58:00.515 |     for line in iter(proc.stdout.readline, \u0027\u0027):"},{"line_number":28,"context_line":"2018-03-01 09:58:00.515 |   File \"/usr/lib/python3.5/encodings/ascii.py\", line 26, in decode"},{"line_number":29,"context_line":"2018-03-01 09:58:00.515 |     return codecs.ascii_decode(input, self.errors)[0]"},{"line_number":30,"context_line":"2018-03-01 09:58:00.515 | UnicodeDecodeError: \u0027ascii\u0027 codec can\u0027t decode byte 0xc3 in position 34: ordinal not in range(128)"},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"Change-Id: Ie4af9b4523459a630cfb98d09093bfe9ef7aa61e"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":4,"id":"9f6a8fd7_e10855da","line":30,"range":{"start_line":21,"start_character":0,"end_line":30,"end_character":124},"updated":"2018-04-18 09:20:07.000000000","message":"hi i hit this issue while trying to install java in a ubuntu 16.04 on a vm created form the default centos base image\n\nit appears this can be triggered if either the image or host local cannot parse the output.\n\ni have created a simple way to reproduce this here.\nhttps://github.com/SeanMooney/dib-elements/tree/locale-bug\n\ninstructions in readme.","commit_id":"b62ed1823c98a13790cba844a1dad9463e8e7fe6"}],"diskimage_builder/elements/package-installs/bin/package-installs-v2":[{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"183e96c711779d9da53e364c5ed9fb09ca23b498","unresolved":false,"context_lines":[{"line_number":30,"context_line":"def process_output(cmdline, follow\u003dFalse):"},{"line_number":31,"context_line":"    proc \u003d subprocess.Popen(cmdline, stdout\u003dsubprocess.PIPE,"},{"line_number":32,"context_line":"                            stderr\u003dsubprocess.STDOUT,"},{"line_number":33,"context_line":"                            universal_newlines\u003dTrue)"},{"line_number":34,"context_line":"    if follow:"},{"line_number":35,"context_line":"        print(\"Running command: %s\" % cmdline)"},{"line_number":36,"context_line":"        out \u003d \"\""}],"source_content_type":"application/octet-stream","patch_set":4,"id":"9f6a8fd7_cfdb9cea","side":"PARENT","line":33,"updated":"2018-04-23 06:07:23.000000000","message":"So I have a fairly small replicator for this : \n\n  http://paste.openstack.org/show/719707/\n\nSo universal_newlines says \"open stdout in text mode with default encoding from locale.getpreferredencoding(False)\"\n\nWhen this comes back as ANSI_X3.4-1968 (i.e. ascii) it assumes that anything coming from stdout will be ascii.  This is probably fair because getpreferredencoding() told it so, based on LC_ALL \n\nThe problem is, I guess, as the \"test.sh\" example shows, there\u0027s no guarantee that the program being run only puts out ascii characters.  \n\n LC_ALL\u003dC echo \"ƀ Ɓ Ƃ ƃ Ƅ ƅ\"\n\ndoesn\u0027t mean that suddenly those characters are escaped when echoed.  I guess they go into the pipe, come out the other side and the ascii codec explodes.\n\nThere\u0027s an additional complication that we set LC_ALL\u003dC in disk-image-create [1] unconditionally.  \n\nLC_ALL should override LANG?  That doesn\u0027t seem to be what is reported in https://github.com/SeanMooney/dib-elements/tree/locale-bug.\n\nAnd it raises the question of why we don\u0027t see this constantly? \n\nI think I may be more confused that when I first looked :/\n\n[1] https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/lib/disk-image-create#n25","commit_id":"3fb0950808b7d2929b15822941b9aaf77975b667"},{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"09086c4030d3782f3ec1fe30286edde883038e95","unresolved":false,"context_lines":[{"line_number":35,"context_line":"        print(\"Running command: %s\" % cmdline)"},{"line_number":36,"context_line":"        out \u003d \"\""},{"line_number":37,"context_line":"        with proc.stdout:"},{"line_number":38,"context_line":"            for line in iter(proc.stdout.readline, b\u0027\u0027):"},{"line_number":39,"context_line":"                line \u003d line.decode(encoding\u003dlocale.getpreferredencoding(False),"},{"line_number":40,"context_line":"                                   errors\u003d\u0027backslashreplace\u0027)"},{"line_number":41,"context_line":"                out +\u003d line"}],"source_content_type":"application/octet-stream","patch_set":4,"id":"df7087c5_750d0c3d","line":38,"updated":"2018-03-08 02:58:16.000000000","message":"Sorry but I just can\u0027t wrap my head around why this is incorrect.  AFAIK, opening iwth universal_newlines should mean that proc.stdout.readline returns a \"string\"?\n\nThis traceback seems to be about the same as [1] which doesn\u0027t have an answer either.\n\n[1] https://stackoverflow.com/questions/28994507/understanding-why-opening-a-subprocess-with-universal-newlines-later-causes-a-un","commit_id":"b62ed1823c98a13790cba844a1dad9463e8e7fe6"},{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"4820425ac5882a66c70709ab4f42f36ec6809fdc","unresolved":false,"context_lines":[{"line_number":36,"context_line":"        out \u003d \"\""},{"line_number":37,"context_line":"        with proc.stdout:"},{"line_number":38,"context_line":"            for line in iter(proc.stdout.readline, b\u0027\u0027):"},{"line_number":39,"context_line":"                line \u003d line.decode(encoding\u003dlocale.getpreferredencoding(False),"},{"line_number":40,"context_line":"                                   errors\u003d\u0027backslashreplace\u0027)"},{"line_number":41,"context_line":"                out +\u003d line"},{"line_number":42,"context_line":"                print(\"\u003e %s\" % line, end\u003d\"\")"}],"source_content_type":"application/octet-stream","patch_set":4,"id":"df7087c5_9a5b2e2d","line":39,"range":{"start_line":39,"start_character":35,"end_line":39,"end_character":78},"updated":"2018-03-07 23:19:09.000000000","message":"is this something we should do on every line of output?  it\u0027s probably not that expensive, but still","commit_id":"b62ed1823c98a13790cba844a1dad9463e8e7fe6"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"ddee21152684ca77731a17e9297f918255736820","unresolved":false,"context_lines":[{"line_number":36,"context_line":"        out \u003d \"\""},{"line_number":37,"context_line":"        with proc.stdout:"},{"line_number":38,"context_line":"            for line in iter(proc.stdout.readline, b\u0027\u0027):"},{"line_number":39,"context_line":"                line \u003d line.decode(encoding\u003dlocale.getpreferredencoding(False),"},{"line_number":40,"context_line":"                                   errors\u003d\u0027backslashreplace\u0027)"},{"line_number":41,"context_line":"                out +\u003d line"},{"line_number":42,"context_line":"                print(\"\u003e %s\" % line, end\u003d\"\")"}],"source_content_type":"application/octet-stream","patch_set":4,"id":"df7087c5_cf7a2a23","line":39,"range":{"start_line":39,"start_character":35,"end_line":39,"end_character":78},"in_reply_to":"df7087c5_9a5b2e2d","updated":"2018-03-08 22:32:04.000000000","message":"Using universal_newlines runs local.getpreferredencoding(False) at least once. Then it decodes the lines with strict error checking. The difference here is that we are saying its ok to have errors just backslashreplace them because we are doing our best.\n\nI think I am ok with that, but there appears to be one other thing going on here. The False argument to getpreferredencoding() means don\u0027t call locale.setlocale() (https://docs.python.org/3/library/locale.html#locale.setlocale) which you may actually need to call ahead of time to properly have getpreferredencoding(False) do what you expect.\n\nAt the very least I think we should probably call setlocale() in the main function early on so that the preferred encoding is reliable. I expect that this may fix it for the majority of cases with universal_newlines. However I think we should also consider dropping universal newlines so that we can do this errors\u003d\u0027backslashreplace\u0027 so that we are doing our best if the package manager emits unicode even when we are properly ascii or something else entirely like latin1. This would be a defensive measure against poorly behaving programs.","commit_id":"b62ed1823c98a13790cba844a1dad9463e8e7fe6"}]}