Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PythonPackage fix python shebangs by default #3499

Open
wants to merge 2 commits into
base: 5.0.x
Choose a base branch
from

Conversation

VRehnberg
Copy link
Contributor

@VRehnberg VRehnberg commented Nov 6, 2024

Fixes #3498

Make fix_python_shebang_for = ['bin/*'] default in PythonPackage
easyblock. As discussed in:
easybuilders#3498
@VRehnberg VRehnberg changed the title PythonPackage fix python shebangs by default, fixes #3498 PythonPackage fix python shebangs by default Nov 6, 2024
@bartoldeman bartoldeman added the EasyBuild-5.0 EasyBuild 5.0 label Nov 20, 2024
@Micket Micket added this to the 5.0 milestone Nov 24, 2024
@VRehnberg
Copy link
Contributor Author

VRehnberg commented Nov 28, 2024

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS accelerate-0.33.0-foss-2023a-CUDA-12.1.1.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-30 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/3486c47db361886fde40ddac959ef5e9 for a full test report.


Manual check of all shebangs. OK

@Micket
Copy link
Contributor

Micket commented Nov 28, 2024

Can you please run a mixed selection of easyconfigs? I'd grab something that uses CMakePythonPackage, PythonPackage, CargoPythonPackage, PythonBundle, just to get a selection of tests and ensure nothing unexpected happens.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS deepdiff-6.7.1-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-30 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/345b36451e1bf392c30a74ab6ec3fcd0 for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS BeautifulSoup-4.12.2-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-14 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/240db6f13957da66e8adff8c90ba3fd2 for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS tensorboardX-2.6.2.2-foss-2023a.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-33 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/618640bf0e7d87c7a516f13896af3e64 for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS zlib-1.2.12.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-14 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/92f2bced1b90c36e8fbae1e65151a26c for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS aiohttp-3.8.5-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-14 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/bff7cb237b578950dd70871d1fae3f65 for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS binutils-2.39.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-14 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/3a0360ff8cd3589d058e30797e00a820 for a full test report.

@VRehnberg
Copy link
Contributor Author

Can you please run a mixed selection of easyconfigs? I'd grab something that uses CMakePythonPackage, PythonPackage, CargoPythonPackage, PythonBundle, just to get a selection of tests and ensure nothing unexpected happens.

I submitted a few, but was a bit careless so got some more dependencies than I wanted, so will probably spam this PR for a bit. If you have something else you want you can probably build from your own framework_dev with --include-easyblocks-from-pr=3499 -u.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS PyTorch-Lightning-2.2.1-foss-2023a-CUDA-12.1.1.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-14 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/e265364fea77c9375aa23d2284cabf8d for a full test report.

@VRehnberg
Copy link
Contributor Author

VRehnberg commented Nov 28, 2024

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
alvis3-25 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/ecf3da60db23639180952f6094b1d1bb for a full test report.


ERROR EasyBuild encountered an error (at easybuild/framework/easybuild/base/exceptions.py:126 in __init__): Use of unknown easyconfig parameter 'build_type' when getting parameter value (at easybuild/framework/easybuild/framework/easyconfig/easyconfig.py:1763 in __getitem__)

Probably not related to this pr

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS Pillow-9.4.0-GCCcore-12.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-38 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/ff7c44f6874af0203b945d39a2dafbf3 for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS hypothesis-6.68.2-GCCcore-12.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis3-38 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/73cacab1fd399d10f0f67b18c40a1cc0 for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS cppy-1.2.1-GCCcore-12.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-37 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/730b2586a1a2454caa098853cceb1321 for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS pybind11-2.10.3-GCCcore-12.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-37 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/20e4510b70784b2ebffda3cef6283ba4 for a full test report.

@VRehnberg

This comment was marked as off-topic.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS Tk-8.6.12-GCCcore-12.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-33 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/2ac2e4d96d234c8b44f1377f95b00a94 for a full test report.

@VRehnberg
Copy link
Contributor Author

VRehnberg commented Nov 28, 2024

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
alvis4-37 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/1411573f48f71c09da8d26b66647b0e7 for a full test report.


Same as SciPy-bundle in another toolchain. buildtype not recognized as an easyconfig parameter. Probably unrelated.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS Tkinter-3.10.8-GCCcore-12.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-33 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/46cec6710dbad393525f4e8a27829e84 for a full test report.

@VRehnberg
Copy link
Contributor Author

VRehnberg commented Nov 28, 2024

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
alvis4-33 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/78893316d4e0514675c6a892bf6b6970 for a full test report.


This build shouldn't even have started as SciPy-bundle failed.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS t-SNE-CUDA-3.0.1-foss-2023a-CUDA-12.1.1.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis4-36 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/059331c5461d60c402ea2c932789ab0c for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS tokenizers-0.15.2-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis3-38 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/421b95c7b82f9a3c49198f788c1b9aef for a full test report.

@VRehnberg
Copy link
Contributor Author

Test report by @VRehnberg

Overview of tested easyconfigs (in order)

  • SUCCESS cryptography-41.0.1-GCCcore-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis3-38 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/7d8e584c3f6d17fd5d8b4562df331204 for a full test report.

@VRehnberg
Copy link
Contributor Author

That should have covered most if not all combinations with some connection to PythonPackage.

There were some consistent failures with scipy in SciPy-bundle about unrecognized easyconfig parameter build_type. I can't see how it could be related, but I can also not see why it appears.

Another unrelated thing is that -x/--extended-dry-run does not seem to work with 5.0.x

== Temporary log file in case of crash /dev/shm/eb-tm46gh6z/easybuild-1b8mye27.log
file /dev/shm/eb-tm46gh6z/fake_vsc_xfdmngdn removed
ERROR: pattern '.*(lmod|LMOD_CMD)' (Lmod) not found in defined 'module' function.
Specify the correct modules tool to avoid weird problems due to this mismatch, see the --modules-tool and --avail-modules-tools command line options.
Or alternatively, use --allow-modules-tool-mismatch to stop treating this as an error. Obtained definition of 'module' function: 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Changed default
Development

Successfully merging this pull request may close these issues.

3 participants