Transparency Log of Report Invalidations

[GalvinGao]

Due to various external or internal reasons, sometimes we need to do report invalidations or changes based on the issue lays on the dataset, for a more easy-to-interpret dataset there has. However, those changes are currently not transparent to the external users and even sometimes causing problems for ourselves to keep track of exact records in which have been edited. Therefore there's a proposed transparency log of changes made to the dataset, in an effort of keeping the track and to the community.

The log consists of several features:

• A dashboard of on-going continuous automated invalidations. Currently, if the report is not meeting with the DropInfo defined in a specific stage, the report will be automatically flagged as non-reliable, effectively invalidating the record. Other than that, there's no other on-going continuous invalidations.
• A list of instantaneous invalidations. Those typically consists of service abusing from a particular IP address or Account, as a huge number of their reports are statistically extremely unlikely to happen, in which we consider an abuse to the service and will flag those IP/Accounts manually. Such cases are currently relatively unlikely to happen, but we do have to intervene several times before to avoid huge deviation on the dataset.
• A dataset snapshots of before & after invalidations are generated. This could be somewhat trivial to implement and its importance might not be so apparent. Open to discussion on this idea.