Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Test lowest versions of all required and optional dependencies #3639

Draft
wants to merge 14 commits into
base: main
Choose a base branch
from
Draft

WIP: Test lowest versions of all required and optional dependencies #3639

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
3a5d6da
Use uv package manager to test lowest versions in ci_tests_legacy.yaml
weiji14 Nov 21, 2024
f303964
Temporarily enable ci_tests_legacy.yaml
weiji14 Nov 21, 2024
3f6909d
Pin to netCDF>=1.7
weiji14 Nov 21, 2024
788fc46
Pin contextily>=1, IPython>=8, pyarrow>=14
weiji14 Nov 21, 2024
4139f4e
Pin geopandas>=0.14, rioxarray>=0.14
weiji14 Nov 21, 2024
1263075
Add --color=yes to test_no_images addopts
weiji14 Nov 21, 2024
d1fa38d
Pin packaging>=22.0
weiji14 Nov 21, 2024
1c69406
Test with pytest-mpl and pytest-doctestplus
weiji14 Nov 21, 2024
3b7b777
Install into uv venv
weiji14 Nov 21, 2024
6120bbe
Pin contextily>=1.2
weiji14 Nov 21, 2024
3299229
Pin pyarrow>=16
weiji14 Nov 21, 2024
0584459
Merge branch 'main' into uv/resolution-lowest
weiji14 Nov 22, 2024
7d542a9
Update wording on ci_tests_legacy.yaml header comment
weiji14 Nov 22, 2024
4733bda
Pin pyarrow>=13 and use pytest.mark.skipif on string_view type test
weiji14 Nov 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 19 additions & 24 deletions .github/workflows/ci_tests_legacy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Test PyGMT with GMT legacy versions on Linux/macOS/Windows
# Test PyGMT with GMT legacy versions and old Python dependencies on Linux/macOS/Windows
#
# This workflow runs regular PyGMT tests with GMT legacy versions. Due to the minor
# baseline image changes between GMT versions, the workflow only runs the tests but
# doesn't do image comparisons.
# This workflow runs regular PyGMT tests with GMT legacy versions and old versions of
# all optional and required Python dependencies. Due to minor baseline image changes
# between GMT versions, the workflow runs the tests but doesn't do image comparisons.
#
# It is scheduled to run every Tuesday on the main branch.
#
Expand All @@ -12,7 +12,7 @@ on:
# push:
# branches: [ main ]
# Uncomment the 'pull_request' line below to trigger the workflow in PR
# pull_request:
pull_request:
# types: [ready_for_review]
# paths:
# - 'pygmt/**'
Expand Down Expand Up @@ -62,23 +62,6 @@ jobs:
python=3.10
gmt=${{ matrix.gmt_version }}
ghostscript<10
numpy<2
pandas
xarray
netCDF4
packaging
contextily
geopandas
ipython
pyarrow
rioxarray
sphinx-gallery
make
pip
python-build
pytest
pytest-doctestplus
pytest-mpl

# Download cached remote files (artifacts) from GitHub
- name: Download remote data from GitHub
Expand All @@ -95,10 +78,22 @@ jobs:
env:
GH_TOKEN: ${{ github.token }}

# Install uv package manager
- name: Install uv
uses: astral-sh/setup-uv@v3

# Install the package that we want to test
- name: Install the package
run: make install
run: |
uv venv
source .venv/bin/activate
uv run --with pip==23 --resolution lowest-direct --all-extras --dev make install
uv pip list
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using --resolution lowest-direct for now, because --resolution lowest grabs many other transitive dependencies that may be hard to install (e.g. missing wheels for newer Python versions, so requires compilation from source). We could switch to --resolution lowest once the Python ecosystem does lower bound pins a bit more thoroughly (which may take years).


# Run the tests but skip images
- name: Run tests
run: make test_no_images PYTEST_EXTRA="-r P"
run: |
source .venv/bin/activate
uv run --with pytest==8,pytest-mpl==0.17,pytest-doctestplus==1.2 --resolution lowest-direct --all-extras --dev make test_no_images PYTEST_EXTRA="-r P"
env:
GMT_LIBRARY_PATH: $CONDA_PREFIX/lib
9 changes: 8 additions & 1 deletion pygmt/tests/test_clib_to_numpy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import pandas as pd
import pytest
from packaging.version import Version
from pygmt._show_versions import _get_module_version
from pygmt.clib.conversion import _to_numpy
from pygmt.helpers.testing import skip_if_no

Expand Down Expand Up @@ -323,7 +324,13 @@ def test_to_numpy_pyarrow_array_pyarrow_dtypes_numeric_with_na(dtype, expected_d
"utf8", # alias for string
"large_string",
"large_utf8", # alias for large_string
"string_view",
pytest.param(
"string_view",
marks=pytest.mark.skipif(
Version(_get_module_version("pyarrow")) < Version("16"),
reason="string_view type was added since pyarrow 16",
),
),
],
)
def test_to_numpy_pyarrow_array_pyarrow_dtypes_string(dtype):
Expand Down
14 changes: 7 additions & 7 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,18 +33,18 @@ dependencies = [
"numpy>=1.24",
"pandas>=2.0",
"xarray>=2023.04",
"netCDF4",
"packaging",
"netCDF4>=1.7",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We discussed making NetCDF an optional dependency before in #429. Maybe we could revisit that discussion?

Copy link
Member

@seisman seisman Nov 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point. NetCDF4 was needed by xarray when reading and writing netCDF files. As mentioned in #239 (comment), previously, our load_xxx functions (e.g., load_earth_relief) called xarray.load_dataarray to load local netCDF grids into an xarray.DataArray object, making netCDF a highly "required" dependency.

After we refactor our load_xxx function in #3120, we no longer call xarray.load_dataarray, so netCDF should be no longer highly needed by PyGMT.

I've opened PR #3643 to see how the PyGMT relies on the netCDF package.

Edit: There are only 14 failures in the Python 3.11 + Ubuntu CI job after removing netCDF entirely (see https://github.com/GenericMappingTools/pygmt/actions/runs/11968416461/job/33367210857?pr=3643) and most of them are caused by load_static_earth_relief.

"packaging>=22.0",
]
dynamic = ["version"]

[project.optional-dependencies]
all = [
"contextily",
"geopandas",
"IPython", # 'ipython' is not the correct module name.
"pyarrow",
"rioxarray",
"contextily>=1.2",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to pin to contextily>=1.2, otherwise this code block won't work as expected (there is a chance that an ImportError will be raised if contextily=1.1 is installed, but xyzservices is not installed, breaking the _HAS_CONTEXTILY logic):

pygmt/pygmt/datasets/tile_map.py

Lines 12 to 19 in 97185e8

try:
import contextily
from xyzservices import TileProvider
_HAS_CONTEXTILY = True
except ImportError:
TileProvider = None
_HAS_CONTEXTILY = False

Xref geopandas/contextily#183, where xyzservices was included as a requirement of contextily (in v1.2.0)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to know that we require contexily>=1.2

"geopandas>=0.14",
"IPython>=8", # 'ipython' is not the correct module name.
"pyarrow>=13",
"rioxarray>=0.14",
]

[project.urls]
Expand Down
Loading