Closed
Description
Bug report
Bug description:
#99108 tracks the addition of a native HACL implementation to CPython. #119316 added an implementation of Blake2 to hashlib.
This compiles fine on single architecture macOS builds (as verified by CI); but universal2 builds running on an ARM64 laptop generate a compilation error:
To reproduce the problem: on a macOS machine, configure the build with:
$ configure --enable-universalsdk="`xcrun --show-sdk-path`" --with-universal-archs=universal2
$ make
This will eventually yield the compilation error:
gcc -c -I../../../Modules/_hacl -I../../../Modules/_hacl/include -D_BSD_SOURCE -D_DEFAULT_SOURCE -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -arch arm64 -arch x86_64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -fstack-protector-strong -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I../../../Include/internal -I../../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../../Include -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -mavx2 -DHACL_CAN_COMPILE_VEC256 -o Modules/_hacl/Hacl_Hash_Blake2b_Simd256.o ../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c
In file included from ../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:26:
In file included from ../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:40:
../../../Modules/_hacl/internal/../Hacl_Hash_Blake2b_Simd256.h:56:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'
Lib_IntVector_Intrinsics_vec256 *fst;
^
../../../Modules/_hacl/internal/../Hacl_Hash_Blake2b_Simd256.h:57:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'
Lib_IntVector_Intrinsics_vec256 *snd;
^
In file included from ../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:26:
../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:44:32: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'
Hacl_Hash_Blake2b_Simd256_init(Lib_IntVector_Intrinsics_vec256 *hash, uint32_t kk, uint32_t nn);
^
../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:49:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'
Lib_IntVector_Intrinsics_vec256 *wv,
^
../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:50:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'
Lib_IntVector_Intrinsics_vec256 *hash,
^
../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:59:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'
Lib_IntVector_Intrinsics_vec256 *wv,
^
../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:60:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'
Lib_IntVector_Intrinsics_vec256 *hash,
^
../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:71:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'
Lib_IntVector_Intrinsics_vec256 *hash
^
../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:76:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'
Lib_IntVector_Intrinsics_vec256 *st,
^
../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:83:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'
Lib_IntVector_Intrinsics_vec256 *st
^
../../../Modules/_hacl/internal/Hacl_Hash_Blake2b_Simd256.h:86:1: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'
Lib_IntVector_Intrinsics_vec256 *Hacl_Hash_Blake2b_Simd256_malloc_with_key(void);
^
../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:34:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'
Lib_IntVector_Intrinsics_vec256 *wv,
^
../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:35:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec256'
Lib_IntVector_Intrinsics_vec256 *hash,
^
../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:53:3: error: use of undeclared identifier 'Lib_IntVector_Intrinsics_vec256'
Lib_IntVector_Intrinsics_vec256 mask = Lib_IntVector_Intrinsics_vec256_zero;
^
../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:72:3: error: use of undeclared identifier 'mask'
mask =
^
../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:73:5: error: call to undeclared function 'Lib_IntVector_Intrinsics_vec256_load64s'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
Lib_IntVector_Intrinsics_vec256_load64s(FStar_UInt128_uint128_to_uint64(totlen),
^
../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:77:33: error: use of undeclared identifier 'Lib_IntVector_Intrinsics_vec256'; did you mean 'Lib_IntVector_Intrinsics_vec256_load64s'?
memcpy(wv, hash, 4U * sizeof (Lib_IntVector_Intrinsics_vec256));
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Lib_IntVector_Intrinsics_vec256_load64s
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/secure/_string.h:63:33: note: expanded from macro 'memcpy'
__builtin___memcpy_chk (dest, __VA_ARGS__, __darwin_obsz0 (dest))
^
../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:73:5: note: 'Lib_IntVector_Intrinsics_vec256_load64s' declared here
Lib_IntVector_Intrinsics_vec256_load64s(FStar_UInt128_uint128_to_uint64(totlen),
^
../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:78:3: error: use of undeclared identifier 'Lib_IntVector_Intrinsics_vec256'; did you mean 'Lib_IntVector_Intrinsics_vec256_load64s'?
Lib_IntVector_Intrinsics_vec256 *wv3 = wv + 3U;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Lib_IntVector_Intrinsics_vec256_load64s
../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:73:5: note: 'Lib_IntVector_Intrinsics_vec256_load64s' declared here
Lib_IntVector_Intrinsics_vec256_load64s(FStar_UInt128_uint128_to_uint64(totlen),
^
../../../Modules/_hacl/Hacl_Hash_Blake2b_Simd256.c:78:36: error: use of undeclared identifier 'wv3'
Lib_IntVector_Intrinsics_vec256 *wv3 = wv + 3U;
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [Modules/_hacl/Hacl_Hash_Blake2b_Simd256.o] Error 1
From what I can make out, the error comes from the detection of -mavx2 support. On a bare configure on an ARM64 machine, -mavx2 support is apparently unsupported:
configure:30537: checking whether C compiler accepts -mavx2
configure:30557: gcc -c -Werror -mavx2 conftest.c >&5
clang: error: argument unused during compilation: '-mavx2' [-Werror,-Wunused-command-line-argument]
and as a result, the Hacl_Hash_Blake2b_Simd256.c module isn't compiled. However, when universal support is enabled, -mavx2 is supported:
configure:30537: checking whether C compiler accepts -mavx2
configure:30557: gcc -c -arch arm64 -arch x86_64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -Werror -mavx2 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk conftest.c >&5
configure:30557: $? = 0
configure:30566: result: yes
and the module is included. Based on recent configure logs for x86_64 macOS builds, it appears that -mavx2 is supported on x86_64.
I'm not sufficiently familiar with the subject matter to comment on whether the fix here is to fix the autoconf detection to disable the problematic module on universal builds, or to correct the implementation so that it can compile for universal builds.
Tagging @msprotz @R1kM as the authors of the recent HACL* changes.
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Linked PRs
Metadata
Metadata
Assignees
Labels
bugs and security fixesbugs and security fixesThe build process and cross-buildThe build process and cross-buildAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error