Skip to content

Conversation

@jiridanek
Copy link
Member

@jiridanek jiridanek commented Nov 13, 2025

https://issues.redhat.com/browse/RHOAIENG-32541
https://issues.redhat.com/browse/RHAIENG-948

Konflux forum question https://redhat-internal.slack.com/archives/C04PZ7H0VA8/p1763064690715129

Does konflux have any limitations on build log length? I see weirdly truncated build log output


add libcurl-devel to build dependencies for ppc64le and s390x in pyarrow and onnx builds

https://konflux-ui.apps.stone-prod-p02.hjvn.p1.openshiftapps.com/ns/rhoai-tenant/applications/automation/pipelineruns/odh-workbench-jupyter-datascience-cpu-py312-on-pull-reques8wq2r

  -- Building AWS C++ SDK from source
  CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
    Could NOT find CURL (missing: CURL_LIBRARY CURL_INCLUDE_DIR)

Description

https://redhat-internal.slack.com/archives/C07SBP17R7Z/p1763017704164619?thread_ts=1761667034.429529&cid=C07SBP17R7Z

related to

How Has This Been Tested?

Self checklist (all need to be checked):

  • Ensure that you have run make test (gmake on macOS) before asking for review
  • Changes to everything except Dockerfile.konflux files should be done in odh/notebooks and automatically synced to rhds/notebooks. For Konflux-specific changes, modify Dockerfile.konflux files directly in rhds/notebooks as these require special attention in the downstream repository and flow to the upcoming RHOAI release.

Merge criteria:

  • The commits are squashed in a cohesive manner and have meaningful messages.
  • Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
  • The developer has manually tested the changes and verified that the changes work

@jiridanek
Copy link
Member Author

/build-konflux

@jiridanek jiridanek changed the title RHAIENG-948: fix(ppc): build failure due to missing curl RHAIENG-948: fix(ppc): build failure due to missing curl and openssl -devel packages Nov 13, 2025
@jiridanek
Copy link
Member Author

/build-konflux

4 similar comments
@jiridanek
Copy link
Member Author

/build-konflux

@jiridanek
Copy link
Member Author

/build-konflux

@jiridanek
Copy link
Member Author

/build-konflux

@jiridanek
Copy link
Member Author

/build-konflux

@jiridanek jiridanek changed the title RHAIENG-948: fix(ppc): build failure due to missing curl and openssl -devel packages RHAIENG-948: fix(ppc): build failure due to AIPCC base (missing curl and openssl -devel packages) and AIPCC wheels (some prereqs are missing) Nov 13, 2025
@jiridanek jiridanek changed the title RHAIENG-948: fix(ppc): build failure due to AIPCC base (missing curl and openssl -devel packages) and AIPCC wheels (some prereqs are missing) RHAIENG-948: fix(jupyter datascience on IBM/Power): build failure due to AIPCC base (missing curl and openssl -devel packages) and AIPCC wheels (some prereqs are missing) Nov 13, 2025
@jiridanek
Copy link
Member Author

/build-konflux

1 similar comment
@jiridanek
Copy link
Member Author

/build-konflux

@openshift-ci
Copy link

openshift-ci bot commented Nov 13, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ysok
Once this PR has been reviewed and has the lgtm label, please assign jstourac for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jiridanek
Copy link
Member Author

@openshift-ci openshift-ci bot removed the lgtm label Nov 13, 2025
@openshift-ci
Copy link

openshift-ci bot commented Nov 13, 2025

New changes are detected. LGTM label has been removed.

@jiridanek
Copy link
Member Author

/build-konflux

1 similar comment
@jiridanek
Copy link
Member Author

/build-konflux

@jiridanek
Copy link
Member Author

/build-konflux

2 similar comments
@jiridanek
Copy link
Member Author

/build-konflux

@jiridanek
Copy link
Member Author

/build-konflux

@jiridanek
Copy link
Member Author

jiridanek commented Nov 14, 2025

flaky fail

  [9/9] STEP 21/28: RUN /bin/bash <<'EOF' (set -Eeuxo pipefail...)
  + '[' ppc64le = ppc64le ']'
  + packages=(zlib-devel libjpeg-turbo-devel)
  + dnf install -y zlib-devel libjpeg-turbo-devel
  Updating Subscription Management repositories.
  Red Hat Enterprise Linux AI (3.0) for RHEL 9 pp 0.0  B/s |   0  B     00:00    
  Errors during downloading metadata for repository 'rhelai-3.0-for-rhel-9-ppc64le-rpms':
    - Curl error (91): SSL server certificate status verification FAILED for https://cdn.redhat.com/content/dist/layered/rhel9/ppc64le/rhelai/3.0/os/repodata/f3d9d976e9b6a95063f276350336b8f8ccdf794696d264b800402d2d3448a0d1-primary.xml.gz [No OCSP response received]
    - Curl error (91): SSL server certificate status verification FAILED for https://cdn.redhat.com/content/dist/layered/rhel9/ppc64le/rhelai/3.0/os/repodata/34719f8f9a58477d9fcfba62dbe8f7a5ac6fcd00f45c4a7923d32b108835d1d4-filelists.xml.gz [No OCSP response received]
    - Curl error (91): SSL server certificate status verification FAILED for https://cdn.redhat.com/content/dist/layered/rhel9/ppc64le/rhelai/3.0/os/repodata/a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml [No OCSP response received]
  Error: Failed to download metadata for repo 'rhelai-3.0-for-rhel-9-ppc64le-rpms': Yum repo downloading error: Downloading error(s): repodata/f3d9d976e9b6a95063f276350336b8f8ccdf794696d264b800402d2d3448a0d1-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/34719f8f9a58477d9fcfba62dbe8f7a5ac6fcd00f45c4a7923d32b108835d1d4-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml - Cannot download, all mirrors were already tried without success; repodata/736c3e31f78189f266e634c7b595660ae4643f985e60f1e8b0e7c9c4632045de-updateinfo.xml.gz - Cannot download, all mirrors were already tried without success
  subprocess exited with status 1

https://konflux-ui.apps.stone-prod-p02.hjvn.p1.openshiftapps.com/ns/rhoai-tenant/applications/automation/pipelineruns/odh-workbench-jupyter-datascience-cpu-py312-on-pull-requesxwp9t

@jiridanek
Copy link
Member Author

/build-konflux

2 similar comments
@jiridanek
Copy link
Member Author

/build-konflux

@jiridanek
Copy link
Member Author

/build-konflux

add `libcurl-devel` to build dependencies for `ppc64le` and `s390x` in pyarrow and onnx builds

```
  -- Building AWS C++ SDK from source
  CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
    Could NOT find CURL (missing: CURL_LIBRARY CURL_INCLUDE_DIR)
```

# Conflicts:
#	jupyter/datascience/ubi9-python-3.12/Dockerfile.cpu
add `libcurl-devel` to build dependencies for `ppc64le` and `s390x` in pyarrow and onnx builds

```
  -- Configuring done (1.8s)
  -- Generating done (0.1s)
  CMake Error at cmake_modules/ThirdpartyToolchain.cmake:5282 (set_property):
    The link interface of target "AWS::aws-c-cal" contains:

      OpenSSL::Crypto

    but the target was not found.  Possible reasons include:

      * There is a typo in the target name.
      * A find_package call is missing for an IMPORTED target.
      * An ALIAS target is missing.
```
…heel index

```
+ pip install --no-cache-dir -r requirements-build.txt

Looking in indexes: https://console.redhat.com/api/pypi/public-rhai/rhoai/3.0/cpu-ubi9/simple/

Ignoring oldest-supported-numpy: markers 'python_version < "3.9"' don't match your environment

ERROR: Could not find a version that satisfies the requirement cython>=0.29.31 (from versions: none)

ERROR: No matching distribution found for cython>=0.29.31
```
…eel index

```
+ pip install --no-cache-dir -r requirements.txt
  Looking in indexes: https://console.redhat.com/api/pypi/public-rhai/rhoai/3.0/cpu-ubi9/simple/
  ERROR: Could not find a version that satisfies the requirement numpy>=1.22 (from versions: none)
  ERROR: No matching distribution found for numpy>=1.22
  subprocess exited with status 1
```
…in aipcc wheel index

```
+ pip wheel . -w /root/onnx_wheel
...
        ERROR: Could not find a version that satisfies the requirement protobuf>=4.25.1 (from versions: none)
        ERROR: No matching distribution found for protobuf>=4.25.1
        [end of output]
```
…aipcc wheel index

```
+ pip install --no-cache-dir /tmp/wheels/pyarrow-17.0.0-cp312-cp312-linux_ppc64le.whl
Looking in indexes: https://console.redhat.com/api/pypi/public-rhai/rhoai/3.0/cpu-ubi9/simple/
Processing /tmp/wheels/pyarrow-17.0.0-cp312-cp312-linux_ppc64le.whl
INFO: pip is looking at multiple versions of pyarrow to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement numpy>=1.16.6 (from pyarrow) (from versions: none)
ERROR: No matching distribution found for numpy>=1.16.6
```
…d dependencies in aipcc wheel index

```
  × Failed to download and build `cryptography==43.0.3`
  ├─▶ Failed to resolve requirements from `build-system.requires`
  ├─▶ No solution found when resolving: `maturin>=1, <2`, `cffi>=1.12
  │   ; platform_python_implementation != 'PyPy'`, `setuptools!=74.0.0,
  │   !=74.1.0, !=74.1.1, !=74.1.2, !=74.1.3, !=75.0.0, !=75.1.0, !=75.2.0`
  ╰─▶ Because only maturin==1.9.6 is available and maturin==1.9.6 has no
      wheels with a matching platform tag (e.g., `manylinux_2_34_ppc64le`), we
      can conclude that maturin==1.9.6 cannot be used.
      And because you require maturin==1.9.6, we can conclude that your
      requirements are unsatisfiable.

      hint: Wheels are available for `maturin` (v1.9.6) on the following
      platforms: `linux_aarch64`, `linux_x86_64`
```
…ld pillow

```

      line 20, in run_setup
          return super().run_setup(setup_script)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File
      "/tmp/.tmpoAXecA/builds-v0/.tmpeN0QF3/lib64/python3.12/site-packages/setuptools/build_meta.py",
      line 317, in run_setup
          exec(code, locals())
        File "<string>", line 1108, in <module>
      RequiredDependencyException:
      The headers or library files could not be found for zlib,
      a required dependency when compiling Pillow from source.
```
…el to build pillow

```
        The headers or library files could not be found for jpeg,
        a required dependency when compiling Pillow from source.

        Please see the install instructions at:

        https://pillow.readthedocs.io/en/latest/installation/basic-installation.html
```
…build maturin

```
        Could not find openssl via pkg-config:
        pkg-config exited with status code 1
        > PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
      PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 pkg-config --libs --cflags openssl
        The system library `openssl` required by crate `openssl-sys` was not
      found.
        The file `openssl.pc` needs to be installed and the PKG_CONFIG_PATH
      environment variable must contain its parent directory.
        PKG_CONFIG_PATH contains the following:
            - /usr/local/lib/pkgconfig/
        HINT: you may need to install a package such as openssl, openssl-dev
      or openssl-devel.
...
        Could not find directory of OpenSSL installation, and this `-sys`
      crate cannot
        proceed without this knowledge. If OpenSSL is installed and this crate
      had
        trouble finding it,  you can set the `OPENSSL_DIR` environment
      variable for the
        compilation process.
        Make sure you also have the development packages of openssl installed.
        For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.
...
      warning: build failed, waiting for other jobs to finish...
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status:
```
…scikit-learn

```
  × Failed to download and build `scikit-learn==1.7.2`
  ├─▶ Failed to install requirements from `build-system.requires`
  ├─▶ Failed to build `scipy==1.16.3`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `mesonpy.build_wheel` failed (exit status: 1)
      [stdout]
      + meson setup
...
      ../meson.build:88:0: ERROR: Unknown compiler(s): [['gfortran'],
      ['flang-new'], ['flang'], ['nvfortran'], ['pgfortran'], ['ifort'],
      ['ifx'], ['g95']]
      The following exception(s) were encountered:
      Running `gfortran --help` gave "[Errno 2] No such file or directory:
      'gfortran'"
```
@jiridanek
Copy link
Member Author

/build-konflux

@jiridanek jiridanek added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Nov 14, 2025
####################
FROM ${BASE_IMAGE} AS cpu-base
USER 0
RUN subscription-manager register --org 18631088 --activationkey thisisunsafe
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll remove this in a subsequent PR. For now, I don't want to incur the 45 min delay needed to rebuild and learn if things work without it nor not.

@github-actions
Copy link

There is a problem with the Gemini CLI PR review. Please check the action logs for details.

@openshift-ci
Copy link

openshift-ci bot commented Nov 14, 2025

@jiridanek: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/images 2eed737 link true /test images

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@jiridanek jiridanek merged commit 9ecacbc into red-hat-data-services:main Nov 14, 2025
24 of 48 checks passed
@jiridanek jiridanek deleted the jd_curl_devel branch November 14, 2025 12:48
@jiridanek
Copy link
Member Author

@coderabbitai create github issue to update our base-images Dockerfiles to better match the AIPCC images so that we don't have these discrepancies between opendatahub-io/notebooks and red-hat-data-services/notebooks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-requested tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants