Python 3.7 on Amazon Linux 2 lacks `sha512_224` in hashlib

[willieyz]

• During CI: Port ec2_compatibilitytests #665 , we found that the following Amazon Linux 2 containers failed ACVP tests due to missing sha512_224 support:
  • amazonlinux-2-aarch:base
  • amazonlinux-2-aarch:gcc-7x
  • amazonlinux-2-aarch:clang-7x

Error is reported during make quickcheck:

  File "/usr/lib64/python3.7/hashlib.py", line 150, in __hash_new
    return _hashlib.new(name, data)
ValueError: unsupported hash type

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./test/acvp_client.py", line 469, in <module>
    test(args.prompt, args.expected, args.output, args.version)
  File "./test/acvp_client.py", line 437, in test
    runTest(data, output)
  File "./test/acvp_client.py", line 417, in runTest
    runTestSingle(promptName, prompt, expectedResultName, expectedResult, output)
  File "./test/acvp_client.py", line 379, in runTestSingle
    result = run_sigGen_test(tg, tc)
  File "./test/acvp_client.py", line 206, in run_sigGen_test
    ph = compute_hash(tc["message"], tc["hashAlg"])
  File "./test/acvp_client.py", line 161, in compute_hash
    return hashlib.new("sha512_224", msg_bytes).hexdigest()
  File "/usr/lib64/python3.7/hashlib.py", line 156, in __hash_new
    return __get_builtin_constructor(name)(data)
  File "/usr/lib64/python3.7/hashlib.py", line 113, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha512_224

This failure is unrelated to test correctness; it is solely due to missing hash support required for computing the verification value.

• This issue is created to track and record the problem for a future fix.

[mkannwischer]

This is quite annoying. Ininitially I thought this is just truncation and can be implemented in Python on top of SHA512, but that is not the case: It is SHA512 with different initialization vectors + truncation.
I don't see an easy way to make this work when hashlib does not support SHA512/224 or SHA512/256.

Even worse, this does not seem to be only a Python 3.7 problem. Python 3.14 also does not guarantee support for those
truncated variants: https://docs.python.org/3.14/library/hashlib.htm

In fact, there is a algorithms_guaranteed property giving a list of algorithms that are guaranteed to be supported on all platforms: https://docs.python.org/3.14/library/hashlib.html#hashlib.algorithms_guaranteed
Querying that on the recent 3.14.0 gives

Python 3.14.0 (main, Jan  1 1980, 00:00:00) [Clang 19.1.7 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.algorithms_guaranteed
{'sha384', 'shake_128', 'sha256', 'sha224', 'sha3_224', 'blake2s', 'md5', 'sha3_256', 'sha1', 'sha3_384', 'blake2b', 'shake_256', 'sha512', 'sha3_512'}
>>> hashlib.algorithms_available
{'sha3_256', 'sha3_384', 'shake_256', 'sha512_224', 'sha512_256', 'md5-sha1', 'sm3', 'sha512', 'ripemd160', 'sha384', 'shake_128', 'sha256', 'blake2s', 'sha3_224', 'sha224', 'md5', 'sha1', 'blake2b', 'sha3_512'}

That suggests we really should not be relying on SHA512/224 or SHA512/256 to be supported by hashlib.
The best solution I can think of is skipping those specific ACVP testvectors if the hash functions are not available. But that requires changing the flow of the ACVP client back to what it was before #564.

Any better ideas @hanno-becker?

[hanno-becker]

One could run a preprocessing step where based on supported algorithms, the input and output JSONs are filtered before being passed to the main processing routine. This way, we would not revert on #564.

The filtering of input/output JSONs could even be a separate binary.

[mkannwischer]

One could run a preprocessing step where based on supported algorithms, the input and output JSONs are filtered before being passed to the main processing routine. This way, we would not revert on #564.

The filtering of input/output JSONs could even be a separate binary.

Pre-processing the files shoulds like a good idea. I suggest we do not run this by default, but only when a flag, e.g., --skip-unsupported is passed. This way we don't risk that a bug in the pre-processing disables our ACVP tests.
Then, when we encounter an unsupported sha512_256 or sha512_224, we ask the user to re-run with --skip-unsupported.