Add catalog-server endpoint to update packages #1392
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP
This PR adds a
/update-packages
endpoint to the catalog server.The endpoint updates packages that have not been updated since a certain interval (default 6 hours), in batches (default 100), until a certain amount of request processing time has passed (default 5 minutes).
The idea is that a cron-like job will call this endpoint regularly, synchronized with the update interval.
There is a
force
query parameter that disables the update intervals so that we can update packages that we just importedA few open questions:
/update-packages
takes longer than the request timeout (requests on Cloud Run can be configured to timeout at up to 60min, so this might not be an issue in practice), how should we continue? We could wait for the next update, or post an update task (https://cloud.google.com/tasks)lastUpdate
ascending (oldest to newest) so that we make progress on least-recently updated packages first?force
query parameter in production?force
is a blunt instrument. In past projects we would inject a clock into all services and use a test clock in tests. I'm not sure if we can do that with the emulators. We could have options to override the package import time, or have a test-only process that mutates the package times between import and update steps.In particular, we need to test:
notUpdatedSince
are not updated again