Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

performance + refactoring + fix: delegated roles update (add/remove) metadata #627

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

kairoaraujo
Copy link
Member

@kairoaraujo kairoaraujo commented Oct 14, 2024

Improve bootstrap performance for hash bins (large number such as 16384 PyPI/RubyGems)

Before : Added 2048 hash bins in 119.36732602119446 seconds
Current: Added 2048 hash bins in 1.8565280437469482 seconds

PyPI (PEP 458) and RubyGems size creation:
Added 16384 hash bins in 15.13554310798645 seconds

Fix query to bump empty target files

the actual crud query fails if there is not target files (artifacts) in
the database (TargetFiles) and do not return the the Target Role.

This commmit fixes this query using isouter

https://docs.sqlalchemy.org/en/20/orm/queryguide/query.html#sqlalchemy.orm.Query.join.params.isouter

Refactor: update_snapshot.

Split update_snapshot in a functional way for:

  • updating targets roles for hashbin delegated roles
  • updating targets roles for custom delegated roles

TODO:

Improve the performance and use the celery multi-threading

   repository-service-tuf-worker-1  | [2024-10-14 06:00:00,132: DEBUG/ForkPoolWorker-9] PRE_RUN: bump_online_roles
   repository-service-tuf-worker-1  | [2024-10-14 06:00:00,417: DEBUG/ForkPoolWorker-9] Checking for expired hash bin roles (16384 bins)
   repository-service-tuf-worker-1  | [2024-10-14 06:01:26,330: DEBUG/ForkPoolWorker-9] Total 16384 expired roles in 85.92996311187744s
   repository-service-tuf-worker-1  | [2024-10-14 06:04:07,217: INFO/ForkPoolWorker-9] Task app.repository_service_tuf_worker[bump_online_roles] succeeded in 247.09811807099322s: True
  • is_expired: ~90s
  • _update_targetfiles_hashbin_delegated_roles: ~250s
  • _update_targetfiles_custom_delegated_roles (Nice to Have)

kairoaraujo and others added 6 commits October 11, 2024 12:10
Before : Added 2048 hash bins in 119.36732602119446 seconds
Current: Added 2048 hash bins in 1.8565280437469482 seconds

PyPI (PEP 458) and RubyGems size creation:

Added 16384 hash bins in 15.13554310798645 seconds

Signed-off-by: Kairo Araujo <[email protected]>
the actual crud query fails if there is not target files (artifacts) in
the database (TargetFiles) and do not return the the Target Role.

This commmit fixes this query using isouter

https://docs.sqlalchemy.org/en/20/orm/queryguide/query.html#sqlalchemy.orm.Query.join.params.isouter

Signed-off-by: Kairo Araujo <[email protected]>
Refactor the `update_snapshot`.

Split `update_snapshot` in a functional way for:
  - updating targets roles for hashbin delegated roles
  - updating targets roles for custom delegated roles

TODO:
Improve the performance and use the celery multi-threading
  - `is_expired`
  - `_update_targetfiles_hashbin_delegated_roles`
  - `_update_targetfiles_custom_delegated_roles` (Nice to Have)

Signed-off-by: Kairo Araujo <[email protected]>
Code works, but requires refactoring and structure

Signed-off-by: Kairo Araujo <[email protected]>
Signed-off-by: Kairo Araujo <[email protected]>
Signed-off-by: Kairo de Araujo <[email protected]>
@kairoaraujo kairoaraujo force-pushed the perform-use-multithread-writing-files branch from 18658b9 to 11e49c6 Compare October 20, 2024 09:11
Signed-off-by: Kairo de Araujo <[email protected]>
@kairoaraujo kairoaraujo force-pushed the perform-use-multithread-writing-files branch from 11e49c6 to 78b9cb6 Compare October 20, 2024 09:21
Signed-off-by: Kairo de Araujo <[email protected]>
@kairoaraujo kairoaraujo force-pushed the perform-use-multithread-writing-files branch from d913b14 to 3059daf Compare October 23, 2024 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant