Use cases
The construction sector is currently one of the less automated sectors. The applications of recent technology innovations can benefit it in many ways. The construction processes infrastructure and health and safety conditions can improve. The speed and efficiency of construction activities will maximize. Risks of structural and safety failures will scale down, reducing the long-term costs. Digitalisation of the equipment will prolong the useful life of structures.
First Solution
The solution focuses on monitoring and detecting building hazards. Drones (Unmanned Aerial Vehicles (UAVs)) will conduct the piloted visual inspections in near-real-time. This solution will deploy a computing edge infrastructure based in micro-services.
Conventional inspections of structures are nowadays based on visual investigation methods. It is hard to access the large structures for a detailed inspection. For instance, bridges, chimneys, towers, dams, industrial power plants, historical buildings and monuments. These structures usually need a complex technical inspection, especially, of the critical hard-to-reach components. At present it is possible only with special equipment (e.g., under-bridge units, elevating platforms) and personnel.
The rapid development of UAVs suggests they can soon be able to do this job. They will collect images from the construction sites and structures. While the AI model will then analyze them to extract detailed information.
Schematic representation of the VER-UC1.
Second solution
This solutions will deploy sensors in workers, machinery and the construction sites. Sensors will detect dangerous situations during the construction process. This will help to avoid workers run-over by machinery and other potential risks.
In the pilot, machinery and workers will have sensors on them. The Wireless Sensor Network (WSN) will inform their position and status in real-time. The raw data from the construction site will go to the FRACTAL node through WSN. Then key insights will be sent to the IoT platform, that will manage the process further. It will register possible dangers, establish proximity alarms and a protocol for emergency.
Inspection model VER-UC1.
Deep learning methods and the U-Net and ResNet architectures will serve as the basis for the AI. The system will be first trained off-line with a dataset at an early stage of the project. To improve the accuracy and to adapt, the system will be able to learn from new environmental data later on. The target information of the model will be mechanical characteristics of the structure.
End-to-End solution VER-UC1.
This use case will demonstrate:
- Application of data-driven models to improve energy efficiency and reduction of environmental pollutants.
- AI-based self-learning and self-adaptation of the data-driven models. Enactment of the control strategies enabled by these features of the models.
- Identification of potential cyber-security breaches through anomaly detection.
The goal is to create an intelligent control system, that will reduce emissions. It will react to the changes in the traffic, traffic light pre-emption and many other. The system will base on the electronic control units and other similar devices. While the intelligent system requires increased computational effort, the hardware is resource constrained. This is the challenge that the FRACTAL is going to tackle.
The air-path is a multifaceted system consisting of components of various complexity levels. The system will be able to self-learn and respond to the situational context. This ability will help it to correct its components and shortcomings. This also gives it scalability, allowing for further commercial exploitation at multi-vehicle level.
The systems´ potential to become fog-based can provide information on driving patterns and traffic infrastructure in the future. Data-driven model can lead to higher precision in the control and diagnosis of the automotive air-path. Simplified development with lower computational effort over existing controllers will enable these enhancements.
The environment of Cyber Physical Systems of Systems (CPSoS) is prone to changes. They need an appropriate response from the control system. To achieve that, it should have the ability to self-learn and to apply improvements. The application of collective learnings to the CPS system will improve its resilience and energy-awareness, replacing present model-based control of the Learning Engine. The hierarchical structure of the system has certain benefits. It can expand its situation cognition and proactive adaptation at the air path level. This can be also combined with air quality monitoring and traffic management systems.
This use case strives to balance a trade-off between fog-based and edge-based computation. It also investigates crucial issues around massive computational power inside the system.
Smart metering is a hot topic and one of the top use cases for the internet of things. The goal is to read the meters from a distance by connecting them to the internet. The utility providers will be able to receive information from them over the network, without the need to visit customers´ site.
One of the steps to smart metering is the electrification of the equipment, which is not yet in place. It should cover both meters and the infrastructure around them. The challenge is that meters often work with pure mechanical principles. They lack power supply and an electronic interface for accessing their stand. It is especially the case of legacy utility meters such as gas, and water meters. But to electrify infrastructure and replace them with a smart device is a big investment.
Smart meter diagram.
The alternative would be a low-cost non-invasive battery-operated device with a camera. The camera will be taking pictures and do the pattern recognition of the meter stand. The system will then send the extracted values over the cellular network. Such a device needs to be at the extreme edge, so its size should be in the range of a 3-5 cm2 to fit on a meter. It should consume as little power as possible, so it can stay in the field for many years. It also needs to be efficient and reliable in reading the meter stand, no matter the lighting conditions. Last but not least, it has to send the data over a wireless channel, even if there is a limited connectivity at the location (e.g., in a basement).
The project proposes a solution of a single chip based on the RISC-V platform. It will be installed on the ACP MtCe501 modem, which supports connectivity over NB-IoT. The digital camera will complete the demonstrator device. FPGA platform will command the camera to take a picture of the meter. It will also run a pattern recognition algorithm on the RISC- V platform (PULP) to extract the meter stand. Then it will use the MtCe501 modem to send the extracted data to the cloud.
Technological pillars used:
(1) Deep learning, edge computing for meter reading;
(2) IT connectivity to send data over NB-IoT;
(3) Security, safety to make the SoC secure and reliable.
A
This use case will focus on building a sensor fusion for edge computing. Securing the bounds of execution time is critical for functional safety systems. The advanced hardware acceleration will guarantee execution time for object detection safe systems. A widely used object detection algorithm (e.g. YOLOv3, Tiny YOLOv3) will be implemented as a FRACTAL building block. that guarantees execution time with advanced HW acceleration for vision-based object detection safe systems, guaranteeing bounds of execution time are critical for functional safety systems to build a sensor fusion for edge computing use cases.
This building block will enable users to easily extend the modular FRACTAL platform with AI/ML for real-time object detection support.
As shown in the figure below, this advanced object detection capability can also incorporate the communication features. The FRACTAL platform will provide these features. This will enable remote monitoring and/or local control of critical industrial systems.
Industry 4.0 use case.
The image processing feature could improve with AI/ML algorithms integration. This will enable industry oriented applications, like predictive maintenance or autonomous robot guidance. Such applications should advance the digitalization and boost the industry 4.0 paradigm.
- stopping precision
- odometer view
- rolling stock coupling operation
- person and obstacle detection
CAF Istanbul’s fully automated metro.
CV&AI enhanced system will have optical sensor video output and train TCMS (Train Management Control System) data as inputs. These inputs will collect the required information for cognitive information processing (deep learning and computer vision algorithms) and the system will command actions for actuators (train TBL – Traction Brake Lever) as an output.
The AI-enhanced algorithms for (driverless) autonomous train operation will need a further substantial effort to increase the TRL before bringing it to the market. AI-enhanced technology will fulfil with strict standards and safety regulation in order to be certified. In addition, regarding the certification process of railway systems and according to EN-5012x standards, AI-enhanced techniques are not currently recommended, so the adoption of this kind of solutions in such a domain is still a challenge.
The goal of this use case is to apply the FRACTAL approach to develop an AI-based smart mobile totem, for advertisement and customer support inside shopping malls. These totems could have a disruptive impact on retail and shopping mall business providing personalized advertisements and product recommendations and driving customers towards their selected destination/product (wayfinding service). The platform will evolve into anthropomorphic robots with more advanced capabilities creating an even more immersive user experience. and enable their adoption not only in the retail sector but also in a smart city, providing service related to mobility, safety and security, logistics and goods delivery. Considering the processing algorithms (based on advanced AI approaches) which will be developed and used to extract meaningful proximity information, edge computing is needed to elaborate data collected using heterogeneous technologies. In fact, such context-aware mobile totem will be equipped with heterogeneous sensors (like cameras, microphones, proximity sensors, etc.) and therefore are able to collect a huge amount of data that can be processed to better understand their surroundings. Same sensors enable such autonomous objects to execute different tasks like patrolling and security monitoring during the night when the mall is closed.
The mobile totem will be equipped with different sensors that provide information to be elaborated using its AI-based processing unit. The output of this decision will select the most appropriated multimedia advertisements and will pilot customer movements (e “local” mode). Obtained information from cooperative totems can be shared with other elements in the same area (. “cooperative” mode). Mobile totems should be able to communicate and cooperate to improve the execution of their tasks. They will coordinate their movements, to optimize at the same time the service provided to customers and their energy consumption (reducing, for example, useless movements). Totems exchange locally detected information, including also user feedback and content selected to improve the effectiveness of the advertisements they are providing. In this way, similar contents can be shown to the same group of people, in different locations, using different mobile totems.
In such a context, our aim is to make these devices more accessible and faster to use and understood, using Artificial Intelligence. The inference of such is provided by a neural network and also ruled based approaches, optimized for running on the embedded device installed in the intelligent totem. Different embedded technologies will be investigated and compared by following the FRACTAL approach and platform (i.e. VERSAL). 
Schematic detailed representation of the VAL-UC7.
The “Smart Physical Demonstration and Evaluation Robot” (SPIDER) is an autonomous robot prototype. Within this use case, VIF will integrate the Cognitive Edge Node developed in FRACTAL in their autonomous robot SPIDER and evaluate its applicability for performing computational intensive relevant vehicle functions of variable complexity at the edge of the network (near the source of the data) while still being able to guarantee extra-functional properties (dependability, timeliness) for preserving safety- and security operational behaviours.
The (user-task dependent) computationally intensive relevant vehicle functions might be task dependent, like, for instance, enhanced AI-based computer vision and AI based decision making techniques, sensor fusion, the creation of an occupancy grid, all of which are applicable for the demanding requirements of the automotive market. By performing the computationally intensive data processing at the edge of the network, so that the SPIDER robot only sends aggregated data to the cloud, reduces communication bandwidth requirements and thus fosters node autonomy by reducing the cloud functionality to management and control.
SPIDER autonomous robot use case.
Two SPIDER functions shall be deployed on the Cognitive Edge Node platform developed in FRACTAL maintaining safety and security by providing high availability at the same time.
The hardware platform to be used will be the medium performance node (RISC-V). VIF will run (user-task dependent) computationally intensive applications (like enhanced AI-based computer vision techniques, sensor fusion or creation of an occupancy grid), on the FRACTAL Cognitive Edge Node’s FPSoC platform to demonstrate its applicability for the automotive market, where the applicability will be verified by the execution of predefined demanding tests, designed to stress the component. The separated implementation of the functions, which suggests a clear separation on the Cognitive Edge Node’s FPSoC hardware design level, ensures that neither security issues nor erroneous decisions made by (uncertified) AI algorithms can impact the functional safety.
The use case employs the FRACTAL technology for a warehouse with intelligent autonomous shuttles based on cognitive computing for swarm intelligence, thereby improving availability, throughput and safety.
The goal is to improve the warehouse throughput, considering that delays in warehouse operation is critically undesirable, since it has a domino effect on the supply chain. The handling, storage and retrieval of warehouse goods by automated shuttles are optimized using Artificial intelligence techniques. AI will optimally organize and analyses the masses of generated data, in order to improve the warehouse throughput.
The automated shuttle systems shall operate as agents of SWARM intelligent system to improve its reliability. To eliminate the need for a central coordinator in which communication failures could de-stabilise the system. The shuttles to gain better computational capability to host AI functions shall use the FRACTAL nodes. Real- time Information (e.g diagnostics, battery health, task) hosted on the shuttle operation are registered in the AI database (Big data). Therefore, the FRACTAL node will be suitable to satisfy the computational requirement at low energy.
Depiction of automated shuttle system in a warehouse.
The following benefits are expected from the SWARM capabilities:
The SWARM of shuttles shall cooperate autonomously to perform a common goal, which is to provide high and reliable warehouse throughput for transported goods. The FRACTAL –based shuttle will adapt to upcoming jobs and failures such that tasks will be completed even if resources fail (e.g., failure of a shuttle, failure of a lift, failure of a track).
Obstacle avoidance and resolution: the SWARM will adapt to avoid obstacles by adjusting paths and collaborate to clear them if possible.
Improved availability: Routing algorithms based on SWARM intelligence will be introduced to ensure fast delivery of goods in the overall warehouse in the presence of faulty and degraded resources (e.g., faulty track, faulty lift, faulty shuttle).
Safety: The SWARM will co-operate to support safety-critical scenarios, e.g. avoiding collision between shuttles and human engineers during online-maintenance, which is also strongly desired.
The need for the FRACTAL node is justified by the high computational requirements demanded by the SWARM intelligence algorithms based on AI. These tasks consume resources, especially power and storage. The proposed FRACTAL nodes offer safe and low-energy with the ability to host cognitive functions, making it a suitable candidate for exploitation. The following inputs from the project will relevantly serve or the demonstration of this use case: (1) Specification and methodology, (2) FRACTAL node cognitive computing abilities (For SWARM intelligence), (3) Safety, Security & Low Power Techniques Reliability and real-time.