JARVIS

The JARVIS project aims to develop a reusable set of tools that enable AI-driven multimodal means of interactions.

The perception of the workplace plays a pivotal role in the automation of industrial processes. It not only enhances efficiency but also significantly increases the safety of human-robot collaboration.

The EU-funded project JARVIS (Intersubjective AI-driven multimodal interaction for advanced user-centric human robot collaborative applications) is running from 2024 to 2027. The tools developed as part of the project include:

Interfaces for both physical and remote information exchange, robot control and programming.
Provision of social skills to a variety of robots to enable seamless user-centric interaction that extends human ability for complex tasks.
Demonstration of application scalability and the ability to achieve economies at scale.

Within this project, KUKA is leading the work package 6, which focuses on the implementation and integration of cognitive and intelligent mechatronics for advanced human-robot interaction. Collaborating with Tecnalia (Spain), LMS and TFCC (Greece), CEA (France), SINTEF (Norway), Collins (Ireland), CEA (France) and others, KUKA is concentrating on scene reconstruction, understanding, object recognition, and visual inspection withing this work package.

Jarvis_environment_reconstruction — Scene reconstruction (point cloud)

During the scene reconstruction and understanding phase, raw data is transformed into meaningful information using sensor data fusion and AI-based methods for object detection and classification. The generated 3D model of a scene should contain not only geometrical but also semantic information. The latter aids in detecting discrepancies or misplacement of objects/parts. All these techniques contribute to executing a safe robotic process, such as avoiding collisions with humans and recognizing unexpected events.

Jarvis_object_recognition — Object recognition

The developed technologies will be integrated and validated in three industrial use cases:

In the automotive sector for the assembly of plug-in battery at a factory in Turkey.
In aeronautics for the production of aircraft seats at a factory in Northern Ireland.
In the energy sector for visual inspection of offshore setups in Norway.

Learn more about the JARVIS project.