Success Story: Traffic Events and Alerts: Data Integration and Analytics to Support Managerial Decisions
NCC-Bulgaria is founded by the Institute of Information and Communication Technologies at the Bulgarian Academy of Sciences, the Sofia University “St. Kliment Ohridski” and the University of National and World Economy.
NCC-Bulgaria is focused on:
- Creating a roadmap for successful work in the field of high performance computing, big data analysis and artificial intelligence.
- Analyzing the existing competencies and facilitating the use of HPC/HPDA/AI in Bulgaria
- Raising awareness and promoting HPC/HPDA/AI use in companies and the public sector.
Industrial Organisations Involved:
Companies in the transport sector
Companies in the Logistics and Supply Chain Management sector
The considerable technical challenge is to accurately identify, extract, transfer and integrate reliable and fast changing data from traffic events and alerts to be followed by sophisticated and exhaustive analysis (descriptive and predictive). On one hand, the large volumes of data being transferred, the enormous diversity of traffic sensors, events, and alerts to be encompassed, and the proliferation of the transmission protocols is a very difficult technical and scientific task. On the other hand, consistent assurance and maintenance of collected data quality while selecting the right tools and algorithms for data analytics and presentation, is not a trivial task.
To solve those challenges, a referential IoT architecture is proposed, consisting of the following components: data extraction and cleaning module NiFi, data distribution module Kafka, data storage module Hadoop HDFS, data analytics modules Hive, Impala and Hue, and data presentation module PowerBI. Building such an architecture enables flexibility and speed in data loading and cleaning up from a multitude of traffic information platforms, different payload formats (CSV, JSON, XML, etc.) and transmission protocols (HTTPS, FTPS, SFTP, etc.), reliable ingestion, dispatch, consumption and storage of the extracted traffic events, powerful data analysis engines for both streaming and batch processing, and, finally, convenient visual presentation of the results ready to be used by the business stakeholders.
The big data, generated in the Transport and Logistics & SCM fields, is not adequately collected, organized and used for the optimization of business processes, activities and services.
The companies, operating in those sectors, could benefit from the real-time analytics and predictive instruments, for:
- Optimizing transportation routes
- Decreasing transportation and logistics expenses
- Optimizing staff management
- Warnings and alerts for traffic incidents, natural disasters and extraordinary events
- Real-time management of autonomous vehicles
- Flexible and reliable, multi-protocol and multi-network data extraction
- Distributed high availability file system and event dispatching system
- Powerful data analytics including streaming and batch processing
- Cost effective Open-Source components
SUCCESS STORY # HIGHLIGHTS:
- Keywords: IoT, Big Data, Traffic events, Traffic alerts, Analytics, Apache Ecosystem, Reference Architecture
- Industry sector: Transport, Logistics & Supply Chain Management
- Technology: IoT, Apache Ecosystem and HPC/HPDA Integration
Referential IoT Architecture for transferring and computational analysis of traffic events data
Traffic Events ETL
Traffic Events Report
University of National and World Economy team (firstname.lastname@example.org)
This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 951732. The JU receives support from the European Union’s Horizon 2020 research and innovation program and Germany, Bulgaria, Austria, Croatia, Cyprus, the Czech Republic, Denmark, Estonia, Finland, Greece, Hungary, Ireland, Italy, Lithuania, Latvia, Poland, Portugal, Romania, Slovenia, Spain, Sweden, the United Kingdom, France, the Netherlands, Belgium, Luxembourg, Slovakia, Norway, Switzerland, Turkey, Republic of North Macedonia, Iceland, Montenegro