Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[catalog-server] Manifests have a max size of 1MB #1358

Open
justinfagnani opened this issue Oct 8, 2022 · 0 comments
Open

[catalog-server] Manifests have a max size of 1MB #1358

justinfagnani opened this issue Oct 8, 2022 · 0 comments
Labels
New Site Issues for the rewrite of the site and catalog.

Comments

@justinfagnani
Copy link
Collaborator

Firestore has a limitation for the maximum field size of 1MB. Some manifests in the wild already exceed this size.

Options:

  1. Always use unpkg or similar CDN for manifests. This would add latency and network ingress for every page request, but unpkg is cached by Cloudflare and very fast.
  2. Store manifests in Cloud Storage. Cheap, fast, the preferred architecture for GCP. Requires making a blob storage interface for the Catalog class to use.
  3. Destructure the manifest into actual Firestore fields. JSON maps to Firestore's document model very well, and the manifest format is relatively shallow - each package contains a flat list of modules. Modules would exceed 1MB exceedingly rarely. This would allow for some more structured searching of manifest data, but we don't have use-cases for that yet.
  4. Compress the mainfest. Easy to do with current Firestore schema. We can serve the gzipped data directly, saving CPU.
@justinfagnani justinfagnani added the New Site Issues for the rewrite of the site and catalog. label Oct 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
New Site Issues for the rewrite of the site and catalog.
Projects
None yet
Development

No branches or pull requests

1 participant