fix: ignore invalid/corrupt staging files #990
Merged
+22
−7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
issue: if one arrow file is present in staging directory of an ingestor, the server panics in the sync flow and does not sync the other files to the backend store this is a two fold problem - 1. find the reason behind corruption of an arrow file 2. discard the invalid/corrupt arrow file and let server sync the other files by notifying the name or the path of the corrupt file for further analysis This PR solves problem 2 where server notifies the path of the corrupt file name and ignores the file so that other files do not get stuck in the staging directory
de-sh
reviewed
Nov 27, 2024
de-sh
reviewed
Nov 27, 2024
de-sh
reviewed
Nov 27, 2024
Signed-off-by: Devdutt Shenoi <[email protected]>
de-sh
reviewed
Nov 27, 2024
Signed-off-by: Devdutt Shenoi <[email protected]>
de-sh
previously approved these changes
Nov 27, 2024
de-sh
reviewed
Nov 27, 2024
Signed-off-by: Devdutt Shenoi <[email protected]>
de-sh
previously approved these changes
Nov 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
issue: if one arrow file is present in staging directory of an ingestor, the server panics in the sync flow and does not sync the other files to the backend store
this is a two fold problem -
This PR solves problem 2 where server notifies the path of the corrupt file name and ignores the file so that other files do not get stuck in the staging directory