New policy for excluding locations from our Mapbox and API exports

[simonw]

Follow-on from #702 in which we learned that our logic that excludes locations from our API export and website if their last recorded non-skip report used certain availability tags ("May be a vaccination site in the future" etc).

We should reconsider this policy now that we rely much more heavily on data scraped and imported from other sources.

[simonw]

Here's the current code:

vial/vaccinate/api/search.py

Lines 347 to 361 in 3aca574

	def filter_for_export(qs):
	# Filter down to locations that we think should be exported
	# to the public map on www.vaccinatethestates.com
	return qs.exclude(
	dn_latest_non_skip_report__planned_closure__lt=datetime.date.today()
	).exclude(
	dn_latest_non_skip_report__availability_tags__slug__in=(
	"incorrect_contact_information",
	"location_permanently_closed",
	"may_be_a_vaccination_site_in_the_future",
	"not_open_to_the_public",
	"will_never_be_a_vaccination_site",
	"only_staff",
	)
	)

[simonw]

Could we do this based on the date of that report?

select location.public_id, location.name, location.full_address, report.created_at, json_agg(availability_tag.name)
from location join report on location.dn_latest_non_skip_report_id = report.id
join call_report_availability_tag on call_report_availability_tag.report_id = report.id
join availability_tag on call_report_availability_tag.availabilitytag_id = availability_tag.id
where location. soft_deleted = false and location.id in (select location_id from concordance_location where concordanceidentifier_id in (select id from concordance_identifier where authority = '_issue' and identifier = '701')) group by location.public_id, location.name, location.full_address, report.created_at order by report.created_at desc

https://vial.calltheshots.us/dashboard/?sql=select+location.public_id%2C+location.name%2C+location.full_address%2C+report.created_at%2C+json_agg%28availability_tag.name%29%0D%0Afrom+location+join+report+on+location.dn_latest_non_skip_report_id+%3D+report.id%0D%0Ajoin+call_report_availability_tag+on+call_report_availability_tag.report_id+%3D+report.id%0D%0Ajoin+availability_tag+on+call_report_availability_tag.availabilitytag_id+%3D+availability_tag.id%0D%0Awhere+location.+soft_deleted+%3D+false+and+location.id+in+%28select+location_id+from+concordance_location+where+concordanceidentifier_id+in+%28select+id+from+concordance_identifier+where+authority+%3D+%27_issue%27+and+identifier+%3D+%27701%27%29%29+group+by+location.public_id%2C+location.name%2C+location.full_address%2C+report.created_at+order+by+report.created_at+desc%3ALdkKfys_atKjhtWiSqijkWvPIErh7ZTDLwjOqwc5cJA

That shows the dates on the reports, for the locations we investigated in #702 - sorted by most recent at the top. 2021-06-19 16:23 is the most recent one, the oldest is 2021-03-30 and it looks like about a third of them are June, almost all of the rest are May, a few are April and March.

[simonw]

The issue here is actually a pretty big one: at what point do we consider a location "closed"? Do we stop showing it on our mapbox-powered vaccinatethestates.com site? Do we also exclude it from our API export?

[simonw]

A plan for this is forming in #712.