The client chosen our solution, the FLUENT Data Observability Platform, because it provides the ability to monitor the health of operations through the addition of metadata to all ingestion pipelines.
The platform can track important metrics such as the duration of GTFS daily updates, delays, retries, and ETL execution status, as well as GTFS-RT system connectivity. It can also monitor dataset availability and schema changes for GTFS and GTFS-RT, and track the number of records for GTFS daily updates and GTFS-RT message updates.
Fluent is equipped with data profiling and anomaly detection capabilities, which enable it to detect abnormal schedule changes, such as bus stop arriving time changes and bus line changes for bus stops, as well as spikes in estimated passenger numbers. Additionally, the platform implements data validation by enforcing business rules, such as GTFS-RT validation based on GTFS trip unique identifiers and epoch time validation for next bus arrival.