Bring back GCS ops.

[michaelbanfield]

This was original deleted because

https://pypi.org/project/tensorflow-gcs-config/

exists, however the source code of this is actually built from tensorflow/io . The reason we have the dedicated package is to have a smaller .so for disk space constrained environments.

We would like to leave these ops in tensorflow/io also.

[google-cla]

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

[google-cla]

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

[vnghia]

I would like to ask if it has some overlaps with gcs filesystem. In addition, we now use google-cloud-cpp as a dependency, I think you could use that library instead of writing your own function 😄

[kvignesh1420]

@michaelbanfield Currently, the file system plugins are being migrated from tensorflow to tensorflow-io and the progress is being tracked here #1183. The goal is to have the file system extensions use a common modular approach and have a similar python API usage for all the schemes.

[yongtang]

My understanding is that the GCS ops is to be able to make adjustment (e.g., max_cache_size, block_size, service account, refresh token, etc), in real time within the same graph session. This is useful in situation where tensorflow has already been initialized when user get access.

[vnghia]

I didn't expect there are many dependencies related to gcs. We will have to think about the design of gcs right now. There are some problems:

• As I am using google-cloud-cpp, we lost control to some parameters ( auth, token, ... ).
• The inheritance ( like RetryingGcsFileSystem* ) will not be possible. I am thinking about introducing a new env to control it. add test for filesystem plugins #1221 (comment) )

Apologize for all the back and forth.

[yongtang]

@vnvo2409 I think the issue with env variable is that you have to pass the env at the initialization time. This can post some limitations where tensorflow might have already been imported/initialized.

An alternative approach is to hold a global variable in C/C++ for all the gcs related configurations. This will allow having an extra ops to modify the gcs configurations in runtime within the same graph session. This is more or less the same way gcs ops works in this PR.

[michaelbanfield]

Thanks for the comments, moving gcs filesystem to use google-cloud-cpp makes a lot of sense.

As discussed above these ops allow clients to modify gcs auth while tensorflow is already initialized. There are a few use cases that depend on this so we would want to retain this functionality when splitting to the modular filesystem.

Is there any concerns merging in these ops for the current filesystem? They used to live within tensorflow but were moved out to IO. Once the modular GCS filesystem supports configuration after initialization we can move to that.

[yongtang]

@michaelbanfield That should be fine. Though the test is failing now. Can you apply the following patch:

diff --git a/tensorflow_io/core/BUILD b/tensorflow_io/core/BUILD
index 3563b04..e661149 100644
--- a/tensorflow_io/core/BUILD
+++ b/tensorflow_io/core/BUILD
@@ -724,6 +724,7 @@ cc_binary(
         "//tensorflow_io/core:text_ops",
         "//tensorflow_io/core:ignite_ops",
         "//tensorflow_io/core:mongodb_ops",
+        "//tensorflow_io/gcs:gcs_config_ops",
         "@local_config_tf//:libtensorflow_framework",
         "@local_config_tf//:tf_header_lib",
     ] + select({
diff --git a/tests/test_gcs_config_ops.py b/tests/test_gcs_config_ops.py
index 7d01409..291986c 100644
--- a/tests/test_gcs_config_ops.py
+++ b/tests/test_gcs_config_ops.py
@@ -29,7 +29,7 @@ tf_v1 = tf.version.VERSION.startswith("1")
 class GcsConfigOpsTest(test.TestCase):
     """GCS Config OPS test"""
 
-    @pytest.mark.skipif(sys.platform == "darwin", reason=None)
+    @pytest.mark.skipif(sys.platform == "win32", reason="Windows not working yet")
     def test_set_block_cache(self):
         """test_set_block_cache"""
         cfg = gcs.BlockCacheParams(max_bytes=1024 * 1024 * 1024)

Once the patch is applied I think all tests will pass.

[kvignesh1420]

@michaelbanfield I think the dynamic_cast<RetryingGcsFileSystem*>(filesystem) is returning a null pointer which in turn is raising the error. I think this is because the gs file system plugin has already been registered and there is a type mismatch. Can we remove this typecast operation? Even the tests are failing for linux and macos: https://github.com/tensorflow/io/pull/1229/checks?check_run_id=2221952977

[google-cla]

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.