)]}'
{"swift/common/utils/checksum.py":[{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"0788353f4e847ea468954b105e6dccbf9bfaaed7","unresolved":true,"context_lines":[{"line_number":150,"context_line":"    # Intel 11850H   |  ~7GB/s   | ~2.6GB/s"},{"line_number":151,"context_line":"    # AMD 3900XT     | ~20GB/s   | ~5GB/s"},{"line_number":152,"context_line":"    #"},{"line_number":153,"context_line":"    # i.e., ISA-L is consistently 3-5x faster than kernel sockets"},{"line_number":154,"context_line":"    return crc32c_isal or crc32c_kern or crc32c_anycrc"},{"line_number":155,"context_line":""},{"line_number":156,"context_line":""}],"source_content_type":"text/x-python","patch_set":2,"id":"26de4d3f_c45f503e","side":"PARENT","line":153,"updated":"2026-10-15 22:31:07.000000000","message":"below are results of CRC32C benchmarking with ``anycrc 1.3.3`` and kernel on an INTEL XEON(R) GOLD 6548Y+ node, 4MB/64MB/1GB, kernel is ``2X`` faster than anycrc on the 64KB streaming mode.\n```\ncalculate anycrc-one-shot       algorithm\u003dCRC32C part_mib\u003d   4 calls\u003d    1 median_ms\u003d1.933704 min_ms\u003d1.932537 max_ms\u003d1.935624 gib_per_s\u003d2.020 result\u003d5b7ff2a2\ncalculate kernel-one-shot       algorithm\u003dCRC32C part_mib\u003d   4 calls\u003d    1 median_ms\u003d0.256690 min_ms\u003d0.256533 max_ms\u003d0.259260 gib_per_s\u003d15.218 result\u003d5b7ff2a2\ncalculate anycrc-stream-64KiB   algorithm\u003dCRC32C part_mib\u003d   4 calls\u003d   64 median_ms\u003d1.941390 min_ms\u003d1.940860 max_ms\u003d1.941688 gib_per_s\u003d2.012 result\u003d5b7ff2a2\ncalculate kernel-stream-64KiB   algorithm\u003dCRC32C part_mib\u003d   4 calls\u003d   64 median_ms\u003d0.879587 min_ms\u003d0.877981 max_ms\u003d0.911901 gib_per_s\u003d4.441 result\u003d5b7ff2a2\ncalculate anycrc-one-shot       algorithm\u003dCRC32C part_mib\u003d  64 calls\u003d    1 median_ms\u003d30.881732 min_ms\u003d30.878491 max_ms\u003d30.943439 gib_per_s\u003d2.024 result\u003dc3020233\ncalculate kernel-one-shot       algorithm\u003dCRC32C part_mib\u003d  64 calls\u003d    1 median_ms\u003d6.538491 min_ms\u003d6.501747 max_ms\u003d6.616599 gib_per_s\u003d9.559 result\u003dc3020233\ncalculate anycrc-stream-64KiB   algorithm\u003dCRC32C part_mib\u003d  64 calls\u003d 1024 median_ms\u003d31.051789 min_ms\u003d31.046226 max_ms\u003d31.062077 gib_per_s\u003d2.013 result\u003dc3020233\ncalculate kernel-stream-64KiB   algorithm\u003dCRC32C part_mib\u003d  64 calls\u003d 1024 median_ms\u003d14.059181 min_ms\u003d14.046107 max_ms\u003d14.579629 gib_per_s\u003d4.445 result\u003dc3020233\ncalculate anycrc-one-shot       algorithm\u003dCRC32C part_mib\u003d1024 calls\u003d    1 median_ms\u003d496.205120 min_ms\u003d496.160722 max_ms\u003d496.352374 gib_per_s\u003d2.015 result\u003dd54c4b99\ncalculate kernel-one-shot       algorithm\u003dCRC32C part_mib\u003d1024 calls\u003d    1 median_ms\u003d159.708964 min_ms\u003d159.644859 max_ms\u003d159.854345 gib_per_s\u003d6.261 result\u003dd54c4b99\ncalculate anycrc-stream-64KiB   algorithm\u003dCRC32C part_mib\u003d1024 calls\u003d16384 median_ms\u003d496.860666 min_ms\u003d496.844473 max_ms\u003d497.178319 gib_per_s\u003d2.013 result\u003dd54c4b99\ncalculate kernel-stream-64KiB   algorithm\u003dCRC32C part_mib\u003d1024 calls\u003d16384 median_ms\u003d228.111529 min_ms\u003d226.370660 max_ms\u003d228.557051 gib_per_s\u003d4.384 result\u003dd54c4b99\n```\n\nhowever, with new version ``2.0.0``, anycrc is much faster than kernel! for 64KB streaming with 4/64/1024 MB, it’s always ``5X`` of kernel throughputs.\n```\ncalculate anycrc-one-shot       algorithm\u003dCRC32C part_mib\u003d   4 calls\u003d    1 median_ms\u003d0.328198 min_ms\u003d0.165526 max_ms\u003d0.346559 gib_per_s\u003d11.902 result\u003d1eda0144\ncalculate kernel-one-shot       algorithm\u003dCRC32C part_mib\u003d   4 calls\u003d    1 median_ms\u003d0.250112 min_ms\u003d0.249737 max_ms\u003d0.250997 gib_per_s\u003d15.618 result\u003d1eda0144\ncalculate anycrc-stream-64KiB   algorithm\u003dCRC32C part_mib\u003d   4 calls\u003d   64 median_ms\u003d0.183976 min_ms\u003d0.183819 max_ms\u003d0.184193 gib_per_s\u003d21.232 result\u003d1eda0144\ncalculate kernel-stream-64KiB   algorithm\u003dCRC32C part_mib\u003d   4 calls\u003d   64 median_ms\u003d0.872053 min_ms\u003d0.869642 max_ms\u003d0.902442 gib_per_s\u003d4.479 result\u003d1eda0144\ncalculate anycrc-one-shot       algorithm\u003dCRC32C part_mib\u003d  64 calls\u003d    1 median_ms\u003d4.348642 min_ms\u003d4.328879 max_ms\u003d4.592256 gib_per_s\u003d14.372 result\u003d56e711cf\ncalculate kernel-one-shot       algorithm\u003dCRC32C part_mib\u003d  64 calls\u003d    1 median_ms\u003d6.424981 min_ms\u003d6.216489 max_ms\u003d6.687476 gib_per_s\u003d9.728 result\u003d56e711cf\ncalculate anycrc-stream-64KiB   algorithm\u003dCRC32C part_mib\u003d  64 calls\u003d 1024 median_ms\u003d2.939986 min_ms\u003d2.936836 max_ms\u003d2.941305 gib_per_s\u003d21.259 result\u003d56e711cf\ncalculate kernel-stream-64KiB   algorithm\u003dCRC32C part_mib\u003d  64 calls\u003d 1024 median_ms\u003d13.913362 min_ms\u003d13.894860 max_ms\u003d14.421297 gib_per_s\u003d4.492 result\u003d56e711cf\ncalculate anycrc-one-shot       algorithm\u003dCRC32C part_mib\u003d1024 calls\u003d    1 median_ms\u003d118.571047 min_ms\u003d118.487202 max_ms\u003d119.556805 gib_per_s\u003d8.434 result\u003dc998e3ca\ncalculate kernel-one-shot       algorithm\u003dCRC32C part_mib\u003d1024 calls\u003d    1 median_ms\u003d142.650820 min_ms\u003d142.355609 max_ms\u003d142.777662 gib_per_s\u003d7.010 result\u003dc998e3ca\ncalculate anycrc-stream-64KiB   algorithm\u003dCRC32C part_mib\u003d1024 calls\u003d16384 median_ms\u003d46.997393 min_ms\u003d46.981267 max_ms\u003d47.010564 gib_per_s\u003d21.278 result\u003dc998e3ca\ncalculate kernel-stream-64KiB   algorithm\u003dCRC32C part_mib\u003d1024 calls\u003d16384 median_ms\u003d222.281172 min_ms\u003d222.173216 max_ms\u003d222.465099 gib_per_s\u003d4.499 result\u003dc998e3ca\n```\n\nsince swift still supports python 3.7 which uses anycrc 1.3.4, we can\u0027t merge this patch yet until swift moves away from py37.","commit_id":"ad136f5b5656f30bdf0716b9ac7966c475b23780"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"6400d224a74e9e18a07801a2a2b2203c6da8da65","unresolved":true,"context_lines":[{"line_number":150,"context_line":"    # Intel 11850H   |  ~7GB/s   | ~2.6GB/s"},{"line_number":151,"context_line":"    # AMD 3900XT     | ~20GB/s   | ~5GB/s"},{"line_number":152,"context_line":"    #"},{"line_number":153,"context_line":"    # i.e., ISA-L is consistently 3-5x faster than kernel sockets"},{"line_number":154,"context_line":"    return crc32c_isal or crc32c_kern or crc32c_anycrc"},{"line_number":155,"context_line":""},{"line_number":156,"context_line":""}],"source_content_type":"text/x-python","patch_set":2,"id":"c997fa02_2fe5303d","side":"PARENT","line":153,"in_reply_to":"26de4d3f_c45f503e","updated":"2026-11-06 20:10:49.000000000","message":"\u003e it’s always 5X of kernel throughputs\n\nWell, that first `one-shot` is an outlier (anycrc @ 11.9GB/s vs kernel @ 15.6GB/s). Not sure why.\n\n\u003e since swift still supports python 3.7 which uses anycrc 1.3.4, we can\u0027t merge this patch yet until swift moves away from py37.\n\nIDK that *that\u0027s* necessarily true: I think it\u0027s less about \"we must use the fastest backend possible for all supported versions\" and more of a balancing act between making sure there\u0027s something that\u0027s \"fast enough\" vs on-going maintenance burden of having the extra backend (which [hasn\u0027t been zero](https://bugs.launchpad.net/swift/+bug/2163006)). In which case the question is \"is 2GB/s fast enough?\" and I\u0027m not sure I have a good sense of the answer.\n\nBut all this is to say, we *could* just view this `anycrc` speed-up as one of the carrots for deployers to upgrade their pythons. 😉\n\nAlso, this rather complicates the story of what precedence `_select_crc32c_impl` *should* have...","commit_id":"ad136f5b5656f30bdf0716b9ac7966c475b23780"}]}