Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Inconsistency & Unexpected Structure Monitoring #62

Open
GalvinGao opened this issue May 2, 2022 · 0 comments
Open

Data Inconsistency & Unexpected Structure Monitoring #62

GalvinGao opened this issue May 2, 2022 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@GalvinGao
Copy link
Member

GalvinGao commented May 2, 2022

The latest activity in CN, opened last day (2022-05-01 16PM GMT+8) have empty DropInfos served on API for a whole 5 hours until a member @FlandiaYingman found such issue and informed us last night (2022-05-01 21PM GMT+8) and fixed in several minutes by manually purging the server-side cache. This has caused reports only starting from after 21PM GMT+8 and have caused some of the reports being flagged due to DropInfo mismatch.

In the future we should've been able to mitigate this by properly purging the cache after an update via Admin API have been performed (#58), but also we could add an additional end-to-end check via the real API on the soracli, so that consistency of API response is guaranteed. Further, this logic could've been abstracted and added to the server side too, creating autonomous monitoring tasks to continuously guard the service from serving dirty caches anymore in the future.

@GalvinGao GalvinGao added the enhancement New feature or request label May 2, 2022
@GalvinGao GalvinGao self-assigned this May 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant