[FEATURE REQUEST #32] On-Premises S3 / S3 Compatible... #389

lefebsy · 2024-10-21T20:13:14Z

Description:

This is a proposition of Polaris core storage implementation, copy of the aws, without the required arnRole parameter, replaced by endpoint, path style, and credentials.
You can choose a strategy for the client calling catalog :

Default option - "vending token" with STS assume role like aws. It is working with MinIO, and should works also with Dell ECS, NetApp StorageGRID, etc...
"vending keys" :
It will disabling Polaris credential "SubScoping", but if you do not care a lot about security
- the catalog can send his own keys to the client
- you can also store in catalog keys dedicated to clients. You can then revoke them externally without breaking the catalog.

There is also possiblity during creation of the catalog (cli, curl api) to choose between 2 strategies to communicate the S3 access and secret keys :

Default option - by environment name variable.
Polaris need these variables available inside is running context. A kubernetes production deployment should be suited for this configuration (envFromSecrets deployed to Polaris pods)
by direct value ( /!\ can be printed in debug logs or included inside JSON getCatalog API response.

Let me know your opinion about this design proposal.
Thank you

Included Changes:

New type of storage "S3_COMPATIBLE".
Tested against MinIO with self-signed certificate
regtests/run_spark_sql_s3compatible.sh

Type of change:

Bug fix (non-breaking change which fixes an issue)
Documentation update
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist:

Please delete options that are not relevant.

I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
If adding new functionality, I have discussed my implementation with the community using the linked GitHub issue

...s-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java

eric-maynard · 2024-10-22T18:05:37Z

spec/polaris-management-service.yml

@@ -901,6 +903,58 @@ components:
      required:
        - roleArn

+    S3StorageConfigInfo:


Is this specific to s3compat, or is it also meant to be used for s3 itself?

This implementation focuses on OnPrem S3 because there is already the AWS class..

However, in a next step I can try to let it work seamlessly with AWS too :

I think that roleArn is not mandatory for AWS S3, so let it to the existing implementation for this scenario

Using access and secret key should work with AWS S3 too

I have overrided the AWS STS endpoint with S3 endpoint. I could add a modification, maybe with a STS endpoint property... something like "if property is empty" -> "STSclient call AWS default STS endpoint" else -> "STS client call the endpoint setted" or a boolean with a clear and explicit description

Region, (maybe little more reflexion is needed to avoid conflict)

Add region property

I have removed the cross region tweak of the AWS FileIOClientFactory, it can be kept to assure a full compatibility

If we can unify these, I think that would be ideal. But I don't know enough about how S3 vs S3Compatible are similar/different to say how possible that is.

I think it will not be too hard to unify, in a next step. I miss AWS access to do tests, but for what I know or seen :

STS endpoint is a AWS specific endpoint, in S3 compatible solutions, when it is available, it is merged with S3 endpoint.

Region is available in S3 compatible solutions, but not used a lot, or mostly implemented to be compliant with aws sdk clients.

about the cross region, it is more related to Polaris and Iceberg with the Iceberg overloaded S3fileIO, but I seen that [AWS] S3FileIO - Add Cross-Region Bucket Access iceberg#11259 have been merged 3 weeks ago --> so next Iceberg version will be ok soon.

MinIO claim that their product API is 100% compatible with AWS S3 API. Almost the same for many alternatives...

The S3 Compatible implementation could easily propose an optional parameter "arnRole" like the mandatory one in the existing aws class, with less regexp patern to allow more flexibility for some implementation where "aws" inside the string is replaced by the product name (exemple "ecs" for DELL ECS)... It could help for a smooth transition

spec/polaris-management-service.yml

eric-maynard · 2024-10-22T18:07:09Z

spec/polaris-management-service.yml

+          enum:
+            - TOKEN_WITH_ASSUME_ROLE
+            - KEYS_SAME_AS_CATALOG
+            - KEYS_DEDICATED_TO_CLIENT


How does this work? What identifies a client?

Here client is anything trying to obtain keys (or security token service) from this catalog (spark, trino,...). There is no particular distinction of identity.

This is not the right term to use in the context of Polaris?

I think the term is correct, I was just stuck trying to understand how the service will track which keys are dedicated to which client.

Ok.

It's simply one key for catalog itself, then another unique key for any clients whoever they are. I Let client distinction to the principal/role/privilege level. I think it is hard at the class storage/credential level to stick a pair of keys to each different clients.

It is a basic way, when SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION is True and there is not a temporary token, to not divulge internal catalog key and serve a key that can be deactivated or rotated for security concerns without breaking catalog itself.

After discussing with MonkeyCanCode here
Prod Deployment credentials the advantage in this proposal is that you have not to rely on the main credentials provided at the global Polaris service level.

Today if you revoke the Polaris service credentials for AWS, all catalogs with AWS storages are instantly KO.

In this implementation each catalog is independent. It is the same idea about clients keys, to not breaking catalog when clients keys are revoked or rotated fo security reasons.

you have not to rely on the main credentials provided at the global Polaris service level.

I think this is the key point here. I agree that the experience you describe is bad, but I'm not sure that fixing it should be a blocker for s3compat support (or that this is the right fix).

Would you be okay saving this for later, or carving it out into a different PR? In my view relying on the global credentials in production is universally a bad idea, regardless of what STORAGE_TYPE you're using.

eric-maynard · 2024-11-05T08:47:39Z

polaris-core/src/main/java/org/apache/polaris/core/storage/PolarisCredentialProperty.java

@@ -23,6 +23,8 @@ public enum PolarisCredentialProperty {
  AWS_KEY_ID(String.class, "s3.access-key-id", "the aws access key id"),
  AWS_SECRET_KEY(String.class, "s3.secret-access-key", "the aws access key secret"),
  AWS_TOKEN(String.class, "s3.session-token", "the aws scoped access token"),
+  AWS_ENDPOINT(String.class, "s3.endpoint", "the aws s3 endpoint"),
+  AWS_PATH_STYLE_ACCESS(Boolean.class, "s3.path-style-access", "the aws s3 path style access"),


whether or not to use path-style access

Many S3COMPATIBLE solutions are deployed without network devices or configurations in front of them allowing support of dynamic hosts names including buckets.
TLS certificate with private AC could also be a challenge for dynamic host name. "*. domain" can also be forbidden by some enterprise security policy

Path style is useful in many cases. In ideal world I agree it should stay deprecated...

.../org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java

.../java/org/apache/polaris/core/storage/s3compatible/S3CompatibleStorageConfigurationInfo.java

regtests/minio/miniodata/Readme.md

collado-mike

I guess I don't understand what isn't supported today with catalog-properties. E.g., in https://github.com/apache/polaris/blob/main/polaris-service/src/test/java/org/apache/polaris/service/catalog/PolarisSparkIntegrationTest.java , we use S3MockContainer as an S3 endpoint, which requires the same path-style access and custom enpdoint configuration as what's included here. Can we not follow the same pattern for minio?

As a rule, I think vending static credentials is not a good idea. Some customization for how the STS client is instantiated, possibly with support for custom profiles for different catalogs could make sense. But I think, ultimately, the credentials returned should always be a temporary session token. Even if we just call GetSessionToken without requiring an IAM role, it would vastly more secure than sending raw credentials.

collado-mike · 2024-11-05T19:14:10Z

.../org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java

+    propertiesMap.put(PolarisCredentialProperty.AWS_ENDPOINT, storageConfig.getS3Endpoint());
+    propertiesMap.put(
+        PolarisCredentialProperty.AWS_PATH_STYLE_ACCESS,
+        storageConfig.getS3PathStyleAccess().toString());


These are catalog properties, not credential-vending properties. These should be set at the catalog-level when it is created. Those properties would then be passed into the FileIO when it is constructed.

To be refactor for satisfying change to requested : boolean "SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION".
I will try to find a way to move it in the catalog properties. But catalog properties are not forwarded to "S3CompatibleCredentialsStorageIntegration.java", only storage properties by default.

collado-mike · 2024-11-05T19:17:43Z

.../org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java

+  public void createStsClient(S3CompatibleStorageConfigurationInfo s3storageConfig) {
+
+    LOGGER.debug("S3Compatible - createStsClient()");
+    StsClientBuilder stsBuilder = software.amazon.awssdk.services.sts.StsClient.builder();


Why is this constructed here rather than being passed in as a constructor parameter?

Like AWS class ?
Find it weird to put something related to a storage type outside the class and provided by the constructor. Seems Azure class is keeping it inside the class too.
No ?

collado-mike · 2024-11-05T19:18:21Z

.../org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java

+import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
+import software.amazon.awssdk.services.sts.model.AssumeRoleResponse;
+
+/** Credential vendor that supports generating */


comment seems to just ...

collado-mike · 2024-11-05T19:23:37Z

.../org/apache/polaris/core/storage/s3compatible/S3CompatibleCredentialsStorageIntegration.java

+          String cli = System.getenv(storageConfig.getS3CredentialsClientAccessKeyId());
+          String cls = System.getenv(storageConfig.getS3CredentialsClientSecretAccessKey());


Seems you could rely on the DefaultCredentialsProvider and maybe allow profiles to be specified? This would allow for env variables, but also file configuration or other means of retrieving credentials.

Do not find how to conciliate this with createCatalog() REST API... And not happy with the idea to let this credentials catalog's fully and exclusively at Polaris service level.
Bad compromise ?

lefebsy · 2024-11-06T00:55:41Z

I guess I don't understand what isn't supported today with catalog-properties. E.g., in https://github.com/apache/polaris/blob/main/polaris-service/src/test/java/org/apache/polaris/service/catalog/PolarisSparkIntegrationTest.java , we use S3MockContainer as an S3 endpoint, which requires the same path-style access and custom enpdoint configuration as what's included here. Can we not follow the same pattern for minio?

Not found how endpoint or other properties can be set from REST API create or update catalog. Only arnRole is accepted as mandatory parameter today in AWS storage type.

As a rule, I think vending static credentials is not a good idea. Some customization for how the STS client is instantiated, possibly with support for custom profiles for different catalogs could make sense. But I think, ultimately, the credentials returned should always be a temporary session token. Even if we just call GetSessionToken without requiring an IAM role, it would vastly more secure than sending raw credentials.

I agree, by default now it is STS "AssumeRole" without any "role". Raw credentials are poor fallback scenario when STS are not available. There is enterprise context where STS, assumeRole etc are not allowed. Only pair of keys are available. By example, Dell ECS require additional policy to enable STS AssumeRole. It's not activated out of the box. I tried to be explicit about this degraded security pattern.

"GetSessionToken" is not part of S3 API, it is IAM API. It is not available in MinIO Unsupported action GetSessionToken (Service: Sts, Status Code: 400...), and not sure about other products. For the moment I've only find "AssumeRole" as IAM STS API method often implemented.

lefebsy requested review from jbonofre, ashvina, RussellSpitzer, snazy, vvcephei, takidau, jackye1995, flyrain, eric-maynard, collado-mike and ebyhr as code owners October 21, 2024 20:13

lefebsy changed the title ~~add s3 compatible storage - first commit~~ [FEATURE REQUEST] On-Premise S3... #32 Oct 21, 2024

lefebsy changed the title ~~[FEATURE REQUEST] On-Premise S3... #32~~ [FEATURE REQUEST #32] On-Premise S3... Oct 21, 2024

lefebsy changed the title ~~[FEATURE REQUEST #32] On-Premise S3...~~ [FEATURE REQUEST #32] On-Premises S3... Oct 21, 2024

This comment was marked as outdated.

Sign in to view

lefebsy force-pushed the s3compatible branch from ee0a8ef to 138c1f7 Compare October 22, 2024 17:48

eric-maynard reviewed Oct 22, 2024

View reviewed changes

...s-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java Outdated Show resolved Hide resolved

eric-maynard reviewed Oct 22, 2024

View reviewed changes

...s-core/src/main/java/org/apache/polaris/core/storage/s3/S3CredentialsStorageIntegration.java Outdated Show resolved Hide resolved

eric-maynard reviewed Oct 22, 2024

View reviewed changes

spec/polaris-management-service.yml Outdated Show resolved Hide resolved

eric-maynard reviewed Oct 22, 2024

View reviewed changes

spec/polaris-management-service.yml Show resolved Hide resolved

eric-maynard reviewed Oct 22, 2024

View reviewed changes

lefebsy force-pushed the s3compatible branch 2 times, most recently from dd8d860 to e2c296b Compare October 28, 2024 15:56

lefebsy requested a review from eric-maynard October 28, 2024 16:12

sungwy mentioned this pull request Oct 29, 2024

WIP: Testing out using Polaris Docker image in Integration Test Suite apache/iceberg-python#1252

Closed

lefebsy and others added 4 commits October 31, 2024 18:36

add s3 compatible storage - first commit

d61eaa9

Create Readme.md

d64b000

typo and cleaning

c1c9f62

renaming

f9e41eb

lefebsy added 3 commits October 31, 2024 18:36

format, typo, refactoring

c0e87ae

typo

c536273

better descriptions

fb42d0e

lefebsy force-pushed the s3compatible branch from cb39f08 to fb42d0e Compare October 31, 2024 17:39