Success Story: Traffic Events and Alerts: Data Integration and Analytics to Support Managerial Decisions

NCC-BULGARIA

NCC-Bulgaria is founded by the Institute of Information and Communication Technologies at the Bulgarian Academy of Sciences, the Sofia University “St. Kliment Ohridski” and the University of National and World Economy.

NCC-Bulgaria is focused on:

• Creating a roadmap for successful work in the field of high performance computing, big data analysis and artificial intelligence.
• Analyzing the existing competencies and facilitating the use of HPC/HPDA/AI in Bulgaria
• Raising awareness and promoting HPC/HPDA/AI use in companies and the public sector.

Industrial Organisations Involved:

Companies in the transport sector

Companies in the Logistics and Supply Chain Management sector

Technical/scientific Challenge:

The considerable technical challenge is to accurately identify, extract, transfer and integrate reliable and fast changing data from traffic events and alerts to be followed by sophisticated and exhaustive analysis (descriptive and predictive). On one hand, the large volumes of data being transferred, the enormous diversity of traffic sensors, events, and alerts to be encompassed, and the proliferation of the transmission protocols is a very difficult technical and scientific task. On the other hand, consistent assurance and maintenance of collected data quality while selecting the right tools and algorithms for data analytics and presentation, is not a trivial task.

Solution:

To solve those challenges, a referential IoT architecture is proposed, consisting of the following components: data extraction and cleaning module NiFi, data distribution module Kafka, data storage module Hadoop HDFS, data analytics modules Hive, Impala and Hue, and data presentation module PowerBI. Building such an architecture enables flexibility and speed in data loading and cleaning up from a multitude of traffic information platforms, different payload formats (CSV, JSON, XML, etc.) and transmission protocols (HTTPS, FTPS, SFTP, etc.), reliable ingestion, dispatch, consumption and storage of the extracted traffic events, powerful data analysis engines for both streaming and batch processing, and, finally, convenient visual presentation of the results ready to be used by the business stakeholders.

Business impact:

The big data, generated in the Transport and Logistics & SCM fields, is not adequately collected, organized and used for the optimization of business processes, activities and services.

The companies, operating in those sectors, could benefit from the real-time analytics and predictive instruments, for:

• Optimizing transportation routes
• Decreasing transportation and logistics expenses
• Optimizing staff management
• Warnings and alerts for traffic incidents, natural disasters and extraordinary events
• Real-time management of autonomous vehicles

Benefits:

• Flexible and reliable, multi-protocol and multi-network data extraction
• Distributed high availability file system and event dispatching system
• Powerful data analytics including streaming and batch processing
• Cost effective Open-Source components

Referential IoT Architecture for transferring and computational analysis of traffic events data

Traffic Events ETL

Traffic Events Report