Sparse checkout for git pulls

[tetracionist]

Added support for sparse-checkout when cloning a GitHub repo (#15185).
This is ideal for teams that have larger repos who don't want to clone all the folders in their repo.

When directories are specified in the Prefect.yaml or GitHubRepository storage class, we will use sparse checkout to get the directories and any file in the root directory (cone-mode). Also supports without cone-mode, although this is not recommended in the official git docs (see https://git-scm.com/docs/git-sparse-checkout)

Example usage in a Prefect.yaml

pull:
- prefect.deployments.steps.git_clone:
    repository: https://github.com/tetracionist/prefect.git
    branch: sparse-checkout-for-git-pulls
    access_token:
    directories: [src/integrations/prefect-azure, src/integrations/prefect-dask]
    cone_mode: True # set to true by default, only need to include if False

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• If this pull request adds new functionality, it includes unit tests that cover the changes
• If this pull request removes docs files, it includes redirect settings in mint.json.
• If this pull request adds functions or classes, it includes helpful docstrings.

[codspeed-hq]

CodSpeed Performance Report

Merging #15824 will not alter performance

_{Comparing tetracionist:sparse-checkout-for-git-pulls (cb5d049) with main (567138f)}

Summary

✅ 3 untouched benchmarks

[desertaxle]

Thanks for the PR @tetracionist! This is looking pretty good, but I have a couple of questions about this implementation after a first pass.

[desertaxle]

It looks like the init subcommand has been deprecated, and the set subcommand is the preferred approach based on the git docs. It seems like we would update this method to only call git sparse-checkout set with the provided directories, but let me know if that would cause issues.

[tetracionist]

I think this should be okay, I will try using git sparse-checkout set

[desertaxle]

It looks like if sparse checkout mode is used then you won't be able to include submodules in the checkout. Is is possible to have both?

[tetracionist]

I put this in the else block as I was unsure what submodules did, but are they something to do with adding code from another repo into the cloned repo?

I think this would be easy with cone mode but might be trickier without.
Would it be useful to also sparse-checkout the submodule?

[desertaxle]

Yeah, submodules allow you to reference another git repository from a parent repository. I'm not sure how sparse checkout would interact with submodules, but I think it'd be useful to apply sparse checkout to submodules, also.

[github-actions]

This pull request is stale because it has been open 14 days with no activity. To keep this pull request open remove stale label or comment.