Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add modular parquet de-/encryption #19858

Open
brainslush opened this issue Nov 18, 2024 · 2 comments
Open

Add modular parquet de-/encryption #19858

brainslush opened this issue Nov 18, 2024 · 2 comments
Labels
enhancement New feature or an improvement of an existing feature

Comments

@brainslush
Copy link

Description

Currently there are efforts to implement parquet de-/encryption into arrow-rs:
apache/arrow-rs#6637

I suggest to extend the existing scan,read,sink and write interfaces to handle encrypted parquet.

I offer to implement it.

@brainslush brainslush added the enhancement New feature or an improvement of an existing feature label Nov 18, 2024
@coastalwhite
Copy link
Collaborator

I quite agree with most of what is being said in the arrow-r/s thread:

  • We are interested but unless someone steps up to drive this effort resource or time-wise, I am not sure we have the bandwidth at the moment.
  • Any implementation of this should use one of the pure Rut cryptography efforts (ring or RustCrypto).
  • I am not 100% aware of the standard surrounding this, but I think this should be a relatively transparent step, maybe in PageReader or just above.

If you want to take this up, I suggest you first made a PR with a rough draft.

@rok
Copy link

rok commented Nov 21, 2024

Glad to see this!
apache/arrow-rs#6637 will most likely be ring based. I'd be happy to help with reviews etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or an improvement of an existing feature
Projects
None yet
Development

No branches or pull requests

3 participants