Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NEW: Add libnvjpeg2k and libnvtiff #28142

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

carterbox
Copy link
Member

@carterbox carterbox commented Nov 7, 2024

Checklist

  • Title of this PR is meaningful: e.g. "Adding my_nifty_package", not "updated meta.yaml".
  • License file is packaged (see here for an example).
  • Source is from official source.
  • Package does not vendor other packages. (If a package uses the source of another package, they should be separate packages or the licenses of all packages need to be packaged).
  • If static libraries are linked in, the license of the static library is packaged.
  • Package does not ship static libraries. If static libraries are needed, follow CFEP-18.
  • Build number is 0.
  • A tarball (url) rather than a repo (e.g. git_url) is used in your recipe (see here for more details).
  • GitHub users listed in the maintainer section have posted a comment confirming they are willing to be listed there.
  • When in trouble, please check our knowledge base documentation before pinging a team.

Closes #28216
Closes #28217

Copy link

github-actions bot commented Nov 7, 2024

Hi! This is the staged-recipes linter and your PR looks excellent! 🚀

@conda-forge-admin
Copy link
Contributor

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/nvjpeg2k/meta.yaml) and found it was in an excellent condition.

recipes/nvjpeg2k/meta.yaml Outdated Show resolved Hide resolved
ln -s ${PREFIX}/${targetsDir}/$j ${PREFIX}/$j

if [[ $j =~ \.so\. ]]; then
patchelf --set-rpath '$ORIGIN' --force-rpath ${PREFIX}/${targetsDir}/$j
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@carterbox
Copy link
Member Author

Packaging libnvjpeg2k0-0.8.0.38-hbc370b7_0
number of files: 4
Skipping binary relocation logic
Making absolute symlink relative (lib/libnvjpeg2k.so.0.8.0.38 -> /home/conda/staged-recipes/build_artifacts/libnvjpeg2k-split_1731453510442/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/targets/x86_64-linux/lib/libnvjpeg2k.so.0.8.0.38 :-> ../targets/x86_64-linux/lib/libnvjpeg2k.so.0.8.0.38)
Making absolute symlink relative (lib/libnvjpeg2k.so.0 -> /home/conda/staged-recipes/build_artifacts/libnvjpeg2k-split_1731453510442/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/targets/x86_64-linux/lib/libnvjpeg2k.so.0 :-> ../targets/x86_64-linux/lib/libnvjpeg2k.so.0.8.0.38)
   INFO: sysroot: '/home/conda/staged-recipes/build_artifacts/libnvjpeg2k-split_1731453510442/_build_env/x86_64-conda-linux-gnu/sysroot/' files: '['usr/share/zoneinfo', 'usr/share/locale/zh_TW/LC_MESSAGES/libc.mo', 'usr/share/locale/zh_CN/LC_MESSAGES/libc.mo', 'usr/share/locale/vi/LC_MESSAGES/libc.mo']'
   INFO (libnvjpeg2k0,targets/x86_64-linux/lib/libnvjpeg2k.so.0.8.0.38): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/librt.so.1 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h4a8ded7_18
   INFO (libnvjpeg2k0,targets/x86_64-linux/lib/libnvjpeg2k.so.0.8.0.38): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libpthread.so.0 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h4a8ded7_18
   INFO (libnvjpeg2k0,targets/x86_64-linux/lib/libnvjpeg2k.so.0.8.0.38): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libdl.so.2 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h4a8ded7_18
   INFO (libnvjpeg2k0,targets/x86_64-linux/lib/libnvjpeg2k.so.0.8.0.38): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libm.so.6 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h4a8ded7_18
  ERROR (libnvjpeg2k0,targets/x86_64-linux/lib/libnvjpeg2k.so.0.8.0.38): $RPATH/libgcc_s.so.1 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
   INFO (libnvjpeg2k0,targets/x86_64-linux/lib/libnvjpeg2k.so.0.8.0.38): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libc.so.6 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h4a8ded7_18
   INFO (libnvjpeg2k0,targets/x86_64-linux/lib/libnvjpeg2k.so.0.8.0.38): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/ld-linux-x86-64.so.2 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h4a8ded7_18
WARNING (libnvjpeg2k0): dso library package conda-forge/linux-64::libstdcxx==14.2.0=hc0a3c3a_1 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)

@jakirkham
Copy link
Member

Thanks Daniel! 🙏

Likely we need the same RPATH fixes that we have added for the CTK: conda-forge/cuda-feedstock#10

Perhaps the changes from this PR ( conda-forge/libnvjpeg-feedstock#8 ) would be a good starting point for fixing the issues in libraries

If we have executables that need RPATH fixes, we can discuss those as well. Sometimes these differ depending on the layout

@carterbox
Copy link
Member Author

I used the nvjpeg-feedstock as a template to create this recipe. Those fixes had already been merged before I started this recipe, so they are already incorporated.

The same overlinking errors occur and are suppressed in the libnjpeg feedstock. This is why I have added the conda-forge.yml to this recipe which supressed that error. Unfortunately, staged-recipes is not a perfect replica of the end feedstock, so it is ignored for now.

Packaging libnvjpeg
Packaging libnvjpeg-12.3.3.54-h97fd463_0
number of files: 4
Skipping binary relocation logic
Making absolute symlink relative (lib/libnvjpeg.so.12 -> /home/conda/feedstock_root/build_artifacts/libnvjpeg_1731689908939/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/targets/x86_64-linux/lib/libnvjpeg.so.12 :-> ../targets/x86_64-linux/lib/libnvjpeg.so.12.3.3.54)
Making absolute symlink relative (lib/libnvjpeg.so.12.3.3.54 -> /home/conda/feedstock_root/build_artifacts/libnvjpeg_1731689908939/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/targets/x86_64-linux/lib/libnvjpeg.so.12.3.3.54 :-> ../targets/x86_64-linux/lib/libnvjpeg.so.12.3.3.54)
   INFO: sysroot: '/home/conda/feedstock_root/build_artifacts/libnvjpeg_1731689908939/_build_env/x86_64-conda-linux-gnu/sysroot/' files: '['usr/share/zoneinfo', 'usr/share/locale/zh_TW/LC_MESSAGES/libc.mo', 'usr/share/locale/zh_CN/LC_MESSAGES/libc.mo', 'usr/share/locale/vi/LC_MESSAGES/libc.mo']'
   INFO (libnvjpeg,targets/x86_64-linux/lib/libnvjpeg.so.12.3.3.54): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/librt.so.1 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h4a8ded7_18
   INFO (libnvjpeg,targets/x86_64-linux/lib/libnvjpeg.so.12.3.3.54): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libpthread.so.0 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h4a8ded7_18
   INFO (libnvjpeg,targets/x86_64-linux/lib/libnvjpeg.so.12.3.3.54): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libdl.so.2 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h4a8ded7_18
   INFO (libnvjpeg,targets/x86_64-linux/lib/libnvjpeg.so.12.3.3.54): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libm.so.6 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h4a8ded7_18
  ERROR (libnvjpeg,targets/x86_64-linux/lib/libnvjpeg.so.12.3.3.54): $RPATH/libgcc_s.so.1 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
   INFO (libnvjpeg,targets/x86_64-linux/lib/libnvjpeg.so.12.3.3.54): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/libc.so.6 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h4a8ded7_18
   INFO (libnvjpeg,targets/x86_64-linux/lib/libnvjpeg.so.12.3.3.54): Needed DSO x86_64-conda-linux-gnu/sysroot/lib64/ld-linux-x86-64.so.2 found in CDT/compiler package conda-forge/noarch::sysroot_linux-64==2.17=h4a8ded7_18
WARNING (libnvjpeg): dso library package conda-forge/linux-64::libstdcxx==14.2.0=hc0a3c3a_1 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)

@conda-forge-admin
Copy link
Contributor

conda-forge-admin commented Nov 15, 2024

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/nvjpeg2k/meta.yaml, recipes/nvtiff/meta.yaml) and found some lint.

Here's what I've got...

For recipes/nvjpeg2k/meta.yaml:

  • ℹ️ The recipe is not parsable by parser conda-souschef (grayskull). Your recipe may not receive automatic updates and/or may not be compatible with conda-forge's infrastructure. Please check the logs for more information and ensure your recipe can be parsed.

For recipes/nvtiff/meta.yaml:

  • ❌ When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).

For recipes/nvtiff/meta.yaml:

  • ℹ️ The recipe is not parsable by parser conda-forge-tick (the bot). Your recipe may not receive automatic updates and/or may not be compatible with conda-forge's infrastructure. Please check the logs for more information and ensure your recipe can be parsed.
  • ℹ️ The recipe is not parsable by parser conda-souschef (grayskull). Your recipe may not receive automatic updates and/or may not be compatible with conda-forge's infrastructure. Please check the logs for more information and ensure your recipe can be parsed.

@carterbox carterbox changed the title NEW: Add libnvjpeg2k NEW: Add libnvjpeg2k and libnvtiff Nov 15, 2024
@carterbox
Copy link
Member Author

@adibbley, please review

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is using the targetsDir layout necessary for these packages? This is generally only used by packages included in cuda. The rpms/debs for these libraries install to system standard locations, such as:

$ rpm -qpl libnvjpeg2k0-cuda-12-0.8.0.38-1.x86_64.rpm
warning: libnvjpeg2k0-cuda-12-0.8.0.38-1.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d42d0685: NOKEY
/usr/lib64/libnvjpeg2k
/usr/lib64/libnvjpeg2k/12
/usr/lib64/libnvjpeg2k/12/libnvjpeg2k.so.0
/usr/lib64/libnvjpeg2k/12/libnvjpeg2k.so.0.8.0.38
/usr/share/licenses/libnvjpeg2k0-cuda-12
/usr/share/licenses/libnvjpeg2k0-cuda-12/LICENSE

Note: libnvjpeg2k/12 subdir is used as those formats support side by side installs of the cuda variants, which is not needed here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume by "cuda" you mean the "cuda toolkit"?

Is using the targetsDir layout necessary for these packages?

You tell me! I believe the goal is to be consistent with layout on other platforms, so this is the feedback I'm looking for.

@jakirkham
Copy link
Member

jakirkham commented Nov 20, 2024

When you have a moment, could you please add a conda-forge.yml in each recipe directory like so?

os_version:
  linux_64: cos7
  linux_aarch64: cos7

It looks like conda-smithy includes this during feedstock generation. So this should get pulled into the final feedstocks at the end

This will help up track which OS version these binaries support and help us update to new OS versions/images when ready

Copy link

github-actions bot commented Nov 20, 2024

Hi! This is the staged-recipes linter and I found some lint.

It looks like some changes were made outside the recipes/ directory. To ensure everything runs smoothly, please make sure that recipes are only added to the recipes/ directory and no other files are changed.

If these changes are intentional (and you aren't submitting a recipe), please add a maintenance label to the PR.

File-specific lints and/or hints:

  • .ci_support/linux64_cuda120.yaml:

    • lints:
      • Do not edit files outside of the recipes/ directory.
  • .azure-pipelines/azure-pipelines-linux.yml:

    • lints:
      • Do not edit files outside of the recipes/ directory.

@carterbox carterbox force-pushed the nvjpeg2k branch 2 times, most recently from 649517b to c1f444c Compare November 20, 2024 20:06
@carterbox
Copy link
Member Author

The builds will not succeed because nvtiff requires a later version of glibc than the containers in staged recipes provide.

Copy link

Hi! This is the staged-recipes linter and your PR looks excellent! 🚀

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonder if you can really use alma8 at this point ? My understanding is conda-forge/conda-forge.github.io#1941 and conda-forge/conda-forge-pinning-feedstock#6626 need to be merged first

Copy link
Member Author

@carterbox carterbox Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This config parameter doesn't work yet which is why the docker_image key is being used in the conda_build_config.yml. However, John wants it here so that when the new pinnings go live, there is no accidental building against to alma9.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

Package request: nvtiff Package request: libnvjpeg2k
5 participants