From: Viktor Szakats Date: Sat, 18 Jan 2025 01:11:37 +0000 (+0100) Subject: GHA: add iOS jobs with LibreSSL, enable dependencies for Android via vcpkg X-Git-Tag: curl-8_12_0~76 X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=12a6de2f660dd692cce93cb65ce6e3ec126bb531;p=thirdparty%2Fcurl.git GHA: add iOS jobs with LibreSSL, enable dependencies for Android via vcpkg iOS: - add jobs with autotools, CMake, CMake Xcode generator. The Xcode generator is >10x slower than Unix Makefiles. Keep it because it's the one recommended by CMake and for having its own quirks we may want to know about. - build, cache and use LibreSSL for these jobs. With workaround for an iOS build issue fixed in master. - make Xcode generator work by explicitly disabling code signing. - make tests and examples build with the Xcode generator by setting `-DMACOSX_BUNDLE_GUI_IDENTIFIER=se.curl`, to avoid "Bundle identifier is missing" errors. - cmake: disable `CURL_USE_PKGCONFIG` by default for Apple device. - cmake: add `stdc++` library for BoringSSL and AWS-LC, with `OPENSSL_USE_STATIC_LIBS=ON` set. - cmake: add workaround for Xcode generator issue, where it cannot handle two targets depending on one custom command. A better fix may be dropping `tool_hugehelp.c` and `tool_ca_embed.c` from curltool library. For a future PR. Android: - add vcpkg to Android jobs, enable dependencies. Assisted-by: Tal Regev via #16045 - make vcpkg work with autotools. - pass `--with-brotli` to autotools to detect the vcpkg-supplied brotli. - enable BoringSSL for Android and add a job with it. - silence 457 CMake configure warnings about the Android NDK CMake scripts targeting freshly deprecated CMake versions. These were much more involved than imagined. Basically nothing works out of the box, and when combined, everything becomes a unique edge case. autotools builds were a much easier to make work than CMake ones. Also: - GHA/non-native: re-sync names to be shorter and more aligned with other workflows. - GHA: add `persist-credentials: false` where missing. Unresolved issues: - `OPENSSL_ROOT_DIR` ignored/mis-used when pointing it to LibreSSL. CMake seems to prepend the sysroot to the passed absolute directory. Found no workaround. - CMake when combined with Android, both the Google-recommended method and the built-in CMake method fail to provide a way to avoid `pkg-config` packages at system directories. Failed to find a knob that can remove `/usr/include` from the search path. The workaround is to disable zstd. (I enabled it by default in this release, maybe premature?: f2adb3b6d73cad0c28ec8a32f5fa969d0f6378a0 #15431) Disabling `pkg-config` doesn't work because vcpkg dependencies do not link without it. - CMake's Xcode generator is slow because each `try_compile()` feature check springs a new CMake + Xcode project taking a long time to run, just to compile single-liner C files. A known issue, with no solution. `-DCMAKE_MACOSX_BUNDLE=OFF` did not help, limiting build types to a single one (e.g. `Debug`) also had no effect. make | Xcode | GHA run :---- | :---- | :-------------------------------------------------------------------- 16s | 2m57s | https://github.com/curl/curl/actions/runs/12866334102/job/35868712426 23s | 4m13s | https://github.com/curl/curl/actions/runs/12868128013/job/35874212461 16s | 3m39s | https://github.com/curl/curl/actions/runs/12859073531/job/35849041880 14s | 2m23s | https://github.com/curl/curl/actions/runs/12858298423/job/35847201313 15s | 2m36s | https://github.com/curl/curl/actions/runs/12858058492/job/35846669761 19s | 3m19s | https://github.com/curl/curl/actions/runs/12868919430/job/35876601168 Closes #16043 --- diff --git a/.github/workflows/non-native.yml b/.github/workflows/non-native.yml index dfcdd24655..c1d8bf542b 100644 --- a/.github/workflows/non-native.yml +++ b/.github/workflows/non-native.yml @@ -37,7 +37,7 @@ permissions: {} jobs: netbsd: - name: 'NetBSD (CM, openssl, clang, ${{ matrix.arch }})' + name: 'NetBSD, CM clang openssl ${{ matrix.arch }}' runs-on: ubuntu-latest timeout-minutes: 10 strategy: @@ -77,7 +77,7 @@ jobs: echo '::endgroup::' openbsd: - name: 'OpenBSD (CM, libressl, clang, ${{ matrix.arch }})' + name: 'OpenBSD, CM clang libressl ${{ matrix.arch }}' runs-on: ubuntu-latest timeout-minutes: 10 strategy: @@ -117,7 +117,7 @@ jobs: echo '::endgroup::' freebsd: - name: "FreeBSD (${{ matrix.build == 'cmake' && 'CM' || 'AM' }}, openssl, ${{ matrix.compiler }}, ${{ matrix.arch }})" + name: "FreeBSD, ${{ matrix.build == 'cmake' && 'CM' || 'AM' }} ${{ matrix.compiler }} openssl ${{ matrix.arch }}" runs-on: ubuntu-latest timeout-minutes: 20 strategy: @@ -193,7 +193,7 @@ jobs: echo '::endgroup::' omnios: - name: 'OmniOS (AM, openssl, gcc, amd64)' + name: 'OmniOS, AM gcc openssl amd64' runs-on: ubuntu-latest timeout-minutes: 15 steps: @@ -224,39 +224,241 @@ jobs: time gmake -j3 examples echo '::endgroup::' + ios: + name: "iOS, ${{ (matrix.build.generator && format('CM-{0}', matrix.build.generator)) || (matrix.build.generate && 'CM' || 'AM' )}} ${{ matrix.build.name }} arm64" + runs-on: 'macos-latest' + timeout-minutes: 10 + env: + DEVELOPER_DIR: "/Applications/Xcode${{ matrix.build.xcode && format('_{0}', matrix.build.xcode) || '' }}.app/Contents/Developer" + CC: ${{ matrix.build.compiler || 'clang' }} + MAKEFLAGS: -j 4 + # renovate: datasource=github-tags depName=libressl-portable/portable versioning=semver registryUrl=https://github.com + libressl-version: 4.0.0 + strategy: + fail-fast: false + matrix: + build: + - name: 'libressl' + install_steps: libressl + configure: --with-openssl="$HOME/libressl" --without-libpsl + + - name: 'libressl' + install_steps: libressl + # FIXME: Could not make OPENSSL_ROOT_DIR work. CMake seems to prepend sysroot to it. + generate: >- + -DOPENSSL_INCLUDE_DIR="$HOME/libressl/include" + -DOPENSSL_SSL_LIBRARY="$HOME/libressl/lib/libssl.a" + -DOPENSSL_CRYPTO_LIBRARY="$HOME/libressl/lib/libcrypto.a" + -DCURL_USE_LIBPSL=OFF + + - name: 'libressl' + install_steps: libressl + generator: Xcode + generate: >- + -DCMAKE_XCODE_ATTRIBUTE_CODE_SIGNING_ALLOWED=OFF + -DMACOSX_BUNDLE_GUI_IDENTIFIER=se.curl + -DOPENSSL_INCLUDE_DIR="$HOME/libressl/include" + -DOPENSSL_SSL_LIBRARY="$HOME/libressl/lib/libssl.a" + -DOPENSSL_CRYPTO_LIBRARY="$HOME/libressl/lib/libcrypto.a" + -DCURL_USE_LIBPSL=OFF + + steps: + - name: 'brew install' + if: ${{ matrix.build.configure }} + run: | + echo automake libtool | xargs -Ix -n1 echo brew '"x"' > /tmp/Brewfile + while [[ $? == 0 ]]; do for i in 1 2 3; do brew update && brew bundle install --no-lock --file /tmp/Brewfile && break 2 || { echo Error: wait to try again; sleep 10; } done; false Too many retries; done + + - name: 'toolchain versions' + run: | + which "${CC}"; "${CC}" --version || true + xcodebuild -version || true + xcodebuild -sdk -version | grep '^Path:' || true + xcrun --sdk iphoneos --show-sdk-path 2>/dev/null || true + xcrun --sdk iphoneos --show-sdk-version || true + echo '::group::macros predefined'; "${CC}" -dM -E - < /dev/null | sort || true; echo '::endgroup::' + echo '::group::brew packages installed'; ls -l "$(brew --prefix)/opt"; echo '::endgroup::' + + - name: 'cache libressl' + if: contains(matrix.build.install_steps, 'libressl') + uses: actions/cache@1bd1e32a3bdc45362d1e726936510720a7c30a57 # v4 + id: cache-libressl + env: + cache-name: cache-libressl + with: + path: ~/libressl + key: iOS-${{ env.cache-name }}-${{ env.libressl-version }} + + - name: 'build libressl' + if: contains(matrix.build.install_steps, 'libressl') && steps.cache-libressl.outputs.cache-hit != 'true' + run: | + curl -LsSf --retry 6 --retry-connrefused --max-time 999 \ + https://github.com/libressl/portable/releases/download/v${{ env.libressl-version }}/libressl-${{ env.libressl-version }}.tar.gz | tar -x + cd libressl-${{ env.libressl-version }} + # FIXME: on the 4.0.1 release, delete '-DHAVE_ENDIAN_H=0' + cmake . \ + -DHAVE_ENDIAN_H=0 \ + -DCMAKE_INSTALL_PREFIX="$HOME/libressl" \ + -DCMAKE_SYSTEM_NAME=iOS \ + -DCMAKE_SYSTEM_PROCESSOR=aarch64 \ + -DBUILD_SHARED_LIBS=OFF \ + -DLIBRESSL_APPS=OFF \ + -DLIBRESSL_TESTS=OFF + cmake --build . + cmake --install . --verbose + + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4 + with: + persist-credentials: false + + - name: 'autoreconf' + if: ${{ matrix.build.configure }} + run: autoreconf -fi + + - name: 'configure' + run: | + if [ -n '${{ matrix.build.configure }}' ]; then + mkdir bld && cd bld && ../configure --enable-unity --enable-test-bundles --enable-warnings --enable-werror \ + --disable-dependency-tracking \ + CFLAGS="-isysroot $(xcrun --sdk iphoneos --show-sdk-path 2>/dev/null)" \ + --host=aarch64-apple-darwin \ + --with-apple-idn \ + ${{ matrix.build.configure }} + else + export OPENSSL_DIR=/ + # https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html#cross-compiling-for-ios-tvos-visionos-or-watchos + [ -n '${{ matrix.build.generator }}' ] && options='-G ${{ matrix.build.generator }}' + cmake -B bld -DCMAKE_UNITY_BUILD=ON -DCURL_TEST_BUNDLES=ON -DCURL_WERROR=ON \ + -DCMAKE_SYSTEM_NAME=iOS \ + -DUSE_APPLE_IDN=ON \ + ${{ matrix.build.generate }} ${options} + fi + + - name: 'configure log' + if: ${{ !cancelled() }} + run: cat bld/config.log bld/CMakeFiles/CMakeConfigureLog.yaml 2>/dev/null || true + + - name: 'curl_config.h' + run: | + echo '::group::raw'; cat bld/lib/curl_config.h || true; echo '::endgroup::' + grep -F '#define' bld/lib/curl_config.h | sort || true + + - name: 'build' + run: | + if [ -n '${{ matrix.build.configure }}' ]; then + make -C bld V=1 + else + cmake --build bld --verbose + fi + + - name: 'curl info' + run: find . -type f \( -name curl -o -name '*.dylib' -o -name '*.a' \) -exec file '{}' \; + + - name: 'build tests' + run: | + if [ -n '${{ matrix.build.configure }}' ]; then + make -C bld V=1 -C tests + else + cmake --build bld --target testdeps --verbose + fi + + - name: 'build examples' + run: | + if [ -n '${{ matrix.build.configure }}' ]; then + make -C bld examples V=1 + else + cmake --build bld --target curl-examples --verbose + fi + android: - name: "Android ${{ matrix.platform }} (${{ matrix.build == 'cmake' && 'CM' || 'AM' }}, arm64)" + name: "Android ${{ matrix.platform }}, ${{ matrix.build == 'cmake' && 'CM' || 'AM' }} ${{ matrix.name }} arm64" runs-on: 'ubuntu-latest' - timeout-minutes: 5 + timeout-minutes: 25 + env: + VCPKG_BINARY_SOURCES: 'clear;x-gha,readwrite' + VCPKG_DISABLE_METRICS: '1' + MAKEFLAGS: -j 5 strategy: matrix: - platform: ['21', '35'] - build: [autotools, cmake] + include: + - { build: 'autotools', platform: '21', name: "libressl" , install: 'brotli zstd libpsl nghttp2 openssl libssh2', + options: '--with-openssl --with-brotli' } + + - { build: 'cmake' , platform: '21', name: "libressl" , install: 'brotli zstd libpsl nghttp2 openssl libssh2', + options: '-DCURL_USE_OPENSSL=ON' } + + - { build: 'autotools', platform: '35', name: "libressl" , install: 'brotli zstd libpsl nghttp2 openssl libssh2', + options: '--with-openssl --with-brotli' } + + - { build: 'cmake' , platform: '35', name: "libressl" , install: 'brotli zstd libpsl nghttp2 openssl libssh2', + options: '-DCURL_USE_OPENSSL=ON' } + + # FIXME: Must disable zstd explicitly, otherwise cmake/pkg-config finds it in /usr/include + # and the build fails. I had found no option to disable this behavior. Other default + # dependencies not offered via vcpkg may also need this. + - { build: 'cmake' , platform: '35', name: "boringssl !zstd", install: ' libpsl boringssl', + options: '-DCURL_USE_OPENSSL=ON -DOPENSSL_USE_STATIC_LIBS=ON -DCURL_ZSTD=OFF' } + fail-fast: false steps: + - name: 'vcpkg cache setup' + if: ${{ matrix.install }} + uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7 + with: + script: | + core.exportVariable('ACTIONS_CACHE_URL', process.env.ACTIONS_CACHE_URL || ''); + core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || ''); + + - name: 'vcpkg versions' + if: ${{ matrix.install }} + timeout-minutes: 1 + run: | + git -C "$VCPKG_INSTALLATION_ROOT" show --no-patch --format='%H %ai' + vcpkg version + + - name: 'install prereqs for vcpkg' + if: contains(matrix.install, 'boringssl') + timeout-minutes: 5 + run: | + sudo rm -f /etc/apt/sources.list.d/microsoft-prod.list + sudo apt-get update -y + sudo apt-get install -y --no-install-suggests --no-install-recommends nasm + + - name: 'vcpkg build' + if: ${{ matrix.install }} + timeout-minutes: 20 + run: vcpkg x-set-installed ${{ matrix.install }} '--triplet=arm64-android' + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4 + with: + persist-credentials: false + + - name: 'autoreconf' + if: ${{ matrix.build == 'autotools' }} + run: autoreconf -fi - name: 'configure' run: | - if [ '${{ matrix.build }}' = 'cmake' ]; then + export PKG_CONFIG_PATH="$VCPKG_INSTALLATION_ROOT/installed/arm64-android/lib/pkgconfig" + if [ '${{ matrix.build }}' = 'cmake' ]; then # https://developer.android.com/ndk/guides/cmake cmake -B bld \ -DANDROID_ABI=arm64-v8a \ -DANDROID_PLATFORM='android-${{ matrix.platform }}' \ - -DCMAKE_TOOLCHAIN_FILE="${ANDROID_NDK_HOME}/build/cmake/android.toolchain.cmake" -DCMAKE_WARN_DEPRECATED=OFF \ + -DCMAKE_TOOLCHAIN_FILE="$VCPKG_INSTALLATION_ROOT/scripts/buildsystems/vcpkg.cmake" \ + -DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=${ANDROID_NDK_HOME}/build/cmake/android.toolchain.cmake -DCMAKE_WARN_DEPRECATED=OFF \ + -DVCPKG_INSTALLED_DIR="$VCPKG_INSTALLATION_ROOT/installed" \ + -DVCPKG_TARGET_TRIPLET=arm64-android \ -DCMAKE_UNITY_BUILD=ON -DCURL_TEST_BUNDLES=ON \ -DCURL_WERROR=ON \ - -DCURL_ENABLE_SSL=OFF \ - -DCURL_USE_LIBPSL=OFF + ${{ matrix.options }} else - autoreconf -fi TOOLCHAIN="${ANDROID_NDK_HOME}/toolchains/llvm/prebuilt/linux-x86_64" mkdir bld && cd bld && ../configure --disable-dependency-tracking --enable-unity --enable-test-bundles --enable-warnings --enable-werror \ CC="$TOOLCHAIN/bin/aarch64-linux-android${{ matrix.platform }}-clang" \ AR="$TOOLCHAIN/bin/llvm-ar" \ RANLIB="$TOOLCHAIN/bin/llvm-ranlib" \ - --host='aarch64-linux-android${{ matrix.platform }}' \ - --without-ssl \ - --without-libpsl + --host=aarch64-linux-android${{ matrix.platform }} \ + ${{ matrix.options }} fi - name: 'configure log' @@ -271,16 +473,15 @@ jobs: - name: 'build' run: | if [ '${{ matrix.build }}' = 'cmake' ]; then - cmake --build bld + cmake --build bld --verbose else - make -j5 -C bld + make -j5 -C bld V=1 fi - name: 'curl info' run: find . -type f \( -name curl -o -name '*.so' -o -name '*.a' \) -exec file '{}' \; - name: 'build tests' - if: ${{ matrix.build == 'cmake' }} # skip for autotools to save time run: | if [ '${{ matrix.build }}' = 'cmake' ]; then cmake --build bld --target testdeps @@ -289,7 +490,6 @@ jobs: fi - name: 'build examples' - if: ${{ matrix.build == 'cmake' }} # skip for autotools to save time run: | if [ '${{ matrix.build }}' = 'cmake' ]; then cmake --build bld --target curl-examples @@ -298,7 +498,7 @@ jobs: fi amiga: - name: "AmigaOS (${{ matrix.build == 'cmake' && 'CM' || 'AM' }}, AmiSSL, gcc, m68k)" + name: "AmigaOS, ${{ matrix.build == 'cmake' && 'CM' || 'AM' }} gcc AmiSSL m68k" runs-on: 'ubuntu-latest' timeout-minutes: 5 env: @@ -322,6 +522,8 @@ jobs: mv "$HOME/opt/appveyor" /opt - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4 + with: + persist-credentials: false - name: 'configure' run: | @@ -394,7 +596,7 @@ jobs: fi msdos: - name: "MS-DOS (${{ matrix.build == 'cmake' && 'CM' || 'AM' }}, openssl, djgpp, i586)" + name: "MS-DOS, ${{ matrix.build == 'cmake' && 'CM' || 'AM' }} djgpp openssl i586" runs-on: 'ubuntu-latest' timeout-minutes: 5 env: @@ -429,6 +631,8 @@ jobs: done - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4 + with: + persist-credentials: false - name: 'configure' run: | diff --git a/CMake/curl-config.cmake.in b/CMake/curl-config.cmake.in index 88db0f4934..edb5b1d34a 100644 --- a/CMake/curl-config.cmake.in +++ b/CMake/curl-config.cmake.in @@ -24,7 +24,9 @@ @PACKAGE_INIT@ # Keep condition in sync with CMake/curl-config.cmake.in -if((UNIX AND NOT ANDROID) OR VCPKG_TOOLCHAIN OR (MINGW AND NOT CMAKE_CROSSCOMPILING)) +if((UNIX AND NOT ANDROID AND (NOT APPLE OR CMAKE_SYSTEM_NAME MATCHES "Darwin")) OR + VCPKG_TOOLCHAIN OR + (MINGW AND NOT CMAKE_CROSSCOMPILING)) set(_curl_use_pkgconfig_default ON) else() set(_curl_use_pkgconfig_default OFF) diff --git a/CMakeLists.txt b/CMakeLists.txt index 1ffc0035cb..b3d13da982 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -323,7 +323,9 @@ endif() # Override to force-disable or force-enable the use of pkg-config. # Keep condition in sync with CMake/curl-config.cmake.in -if((UNIX AND NOT ANDROID) OR VCPKG_TOOLCHAIN OR (MINGW AND NOT CMAKE_CROSSCOMPILING)) +if((UNIX AND NOT ANDROID AND (NOT APPLE OR CMAKE_SYSTEM_NAME MATCHES "Darwin")) OR + VCPKG_TOOLCHAIN OR + (MINGW AND NOT CMAKE_CROSSCOMPILING)) set(_curl_use_pkgconfig_default ON) else() set(_curl_use_pkgconfig_default OFF) @@ -741,6 +743,13 @@ if(CURL_USE_OPENSSL) endif() cmake_pop_check_state() + if(HAVE_BORINGSSL OR HAVE_AWSLC) + if(OPENSSL_USE_STATIC_LIBS AND CMAKE_C_COMPILER_ID MATCHES "Clang") + list(APPEND CURL_LIBS "stdc++") + list(APPEND CMAKE_REQUIRED_LIBRARIES "stdc++") + endif() + endif() + if(HAVE_BORINGSSL) set(_openssl "BoringSSL") elseif(HAVE_AWSLC) diff --git a/docs/INSTALL-CMAKE.md b/docs/INSTALL-CMAKE.md index 8b6ca622f0..b74edc5216 100644 --- a/docs/INSTALL-CMAKE.md +++ b/docs/INSTALL-CMAKE.md @@ -280,7 +280,7 @@ Details via CMake - `CURL_USE_LIBUV`: Use libuv for event-based tests. Default: `OFF` - `CURL_USE_MBEDTLS`: Enable mbedTLS for SSL/TLS. Default: `OFF` - `CURL_USE_OPENSSL`: Enable OpenSSL for SSL/TLS. Default: `ON` if no other TLS backend was enabled. -- `CURL_USE_PKGCONFIG`: Enable `pkg-config` to detect dependencies. Default: `ON` for Unix (except Android), vcpkg, MinGW if not cross-compiling. +- `CURL_USE_PKGCONFIG`: Enable `pkg-config` to detect dependencies. Default: `ON` for Unix (except Android, Apple devices), vcpkg, MinGW if not cross-compiling. - `CURL_USE_RUSTLS`: Enable Rustls for SSL/TLS. Default: `OFF` - `CURL_USE_SCHANNEL`: Enable Windows native SSL/TLS (Schannel). Default: `OFF` - `CURL_USE_SECTRANSP`: Enable Apple OS native SSL/TLS (Secure Transport). Default: `OFF` @@ -302,6 +302,7 @@ Details via CMake ## Dependency options (via CMake) - `OPENSSL_ROOT_DIR`: Set this variable to the root installation of OpenSSL (and forks). +- `OPENSSL_USE_STATIC_LIBS`: Look for static OpenSSL libraries. - `ZLIB_INCLUDE_DIR`: The zlib include directory. - `ZLIB_LIBRARY`: Path to `zlib` library. diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index f9292249b2..70e3f5eff2 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -94,11 +94,16 @@ add_executable( ALIAS ${EXE_NAME} ) +set(_curl_files "${CURL_CFILES}" "${CURL_HFILES}") +if(CMAKE_GENERATOR STREQUAL "Xcode") + # Workaround for 'The custom command generating tool_hugehelp.c is attached to multiple targets [...] curl curltool' + list(REMOVE_ITEM _curl_files "tool_hugehelp.c" "tool_hugehelp.h" "tool_ca_embed.c") +endif() add_library( curltool # special libcurltool library just for unittests STATIC EXCLUDE_FROM_ALL - ${CURL_CFILES} ${CURLTOOL_LIBCURL_CFILES} ${CURL_HFILES} + ${_curl_files} ${CURLTOOL_LIBCURL_CFILES} ) target_compile_definitions(curltool PUBLIC "UNITTESTS" "CURL_STATICLIB") target_link_libraries(curltool PRIVATE ${CURL_LIBS})