Data Quality¶
Every row Aeolus returns carries three data-quality columns, derived from what the upstream network actually published:
| Column | What it holds |
|---|---|
qa_code |
The upstream's own quality token, verbatim — verified, Ratified, "2", Channel Disagreement, P, … — or null where the upstream said nothing for that row. Aeolus never invents one. |
qa_tier |
A cross-network comparability tier derived from qa_code through the network's vocabulary: reference_full_qc, reference_provisional, lcs_calibrated, lcs_factory_only, flagged or unknown. |
ratification_stage |
Where the row is in a regulatory ratification cycle: unratified, ratified, supplied, not_applicable, or null where the network has a ratification concept but no per-row status is available. |
Two more columns say where the row came from: network (who produced it) and backend (which of Aeolus's fetchers served it — RDATA, SOS, ERG_REST, …).
The pre-0.5.0 ratification column is still present as a deprecated mirror derived from the three columns above (Ratified, Provisional, Indicative, Validated, Invalid, Unvalidated, None). It is removed in 1.0; set AEOLUS_LEGACY_COLUMNS=0 to drop it now and check your code no longer reads it. Filter on qa_code or qa_tier instead.
What each network publishes¶
| Network | qa_code |
qa_tier |
|---|---|---|
| AURN | verified / unverified — from the site's per-pollutant ratified_to date in the openair metadata |
reference_full_qc / reference_provisional |
| SAQN, WAQN, NI, AQE | Ratified / Provisional, the same way |
reference_full_qc / reference_provisional |
| AURN-SOS and the other near-real-time sources | always the network's unratified token | reference_provisional |
| EEA | the EIONET Verification code: "1" verified, "2" preliminary verified, "3" not verified, "0" Airbase (status not recorded) |
reference_full_qc for "1", reference_provisional for "2"/"3", unknown for "0" |
| PurpleAir | the channel-agreement label: Validated, Single Channel (A)/(B), Below Detection Limit, Channel Disagreement, Sensor Saturation, Invalid, Unvalidated |
lcs_factory_only (validated, single channel, below detection limit), flagged (disagreement, saturation, invalid), unknown (unvalidated) |
| Breathe London | RatificationStatus (P), null where absent |
lcs_calibrated, else unknown |
| AirNow | Provisional — the AirNow feed is never ratified; certified data is in EPA's AQS |
reference_provisional |
| LAQN, LMAM, Sensor.Community, Sonitus, OpenAQ, AirQo | null — these feeds publish no per-row flag | unknown |
The vocabularies live in src/aeolus/data/qa_vocabularies/<NETWORK>.yaml and are shared with Argus.
Filtering by quality¶
import aeolus
from datetime import datetime
data = aeolus.download("AURN", ["MY1"], datetime(2023, 1, 1), datetime(2023, 12, 31))
# Only ratified rows
ratified = data[data["ratification_stage"] == "ratified"]
# Anything that has passed full QA/QC, whatever the network
reference = data[data["qa_tier"] == "reference_full_qc"]
# Drop rows the upstream flagged (PurpleAir channel disagreement etc.)
clean = data[data["qa_tier"] != "flagged"]
# How much of a download is ratified?
print(data.groupby(["network", "ratification_stage"], dropna=False).size())
For PurpleAir, aeolus.sources.purpleair.fetch_purpleair_data(..., include_flagged=False) keeps only rows whose qa_code is Validated (aeolus.download has no such option; filter the frame instead).
Recommendations¶
- For regulatory reporting, use
ratification_stage == "ratified"; provisional data can change on ratification. - Compare networks on
qa_tier, not on the rawqa_code— the codes are each network's own words. - A null
qa_codemeans the upstream said nothing; it is not a bad reading. Check the network's documentation page for what that implies. - Recent AURN-family data is provisional for roughly three to six months; the metadata's
ratified_todate is what Aeolus uses to tell.
Coming from the 0.4 ratification column? See Migrating to 0.5.