Fraunhofer Institute for Production Systems and Design Technology
Fraunhofer Institute for Production Systems and Design Technology

Interfaces

From wearable robotics to virtual reality, people and machines are collaborating in more and more ways. In the process, the real and digital worlds are merging.

© Fraunhofer IPK
Human-robot collaboration in action: In addition to a suction pad and a force sensor, the robot is also equipped with a safety skin that enables direct collaboration in contact with humans. This allows humans to move 35 kg packages of solar panels effortlessly through space – with robotic colleagues.
© Fraunhofer IPK
  1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
  2. A robot must obey orders given it by human beings except where such orders would conflict with the First Law.
  3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

The three laws of robotics according to Isaac Asimov may come from a fictional story – but in times when humans and machines cooperate more closely than ever before, they seem more relevant than ever. Industrial companies must guarantee the safety of their employees and at the same time the best possible operation of their plants. For both, it is essential that communication across system boundaries – that is between people and machines – is as smooth as possible.

While Asimov still centered his science fiction narratives around the interaction with robots as physical entities, today many more aspects of this type of interaction have to be considered. For it no longer takes place only within the material space, but also in the virtual space, also called cyberspace, in which digital data form the basis of communication.

Of course, these two systems are not strictly demarcated from one another. The »real« and the »digital« worlds are not dual opposites, but are in many cases connected by interfaces: the dashboard of a machine tool, which a skilled worker uses to initiate a machining process. The sensor that transmits real-world stimuli as an electronic signal, which then becomes the data point of a digital twin and thus provides information about the status of a system. Or the VR goggles, which visualize digital models, making the virtual world perceptible to the human senses. These are all examples of border crossings in which people enter into direct exchange with machines. The space in which this exchange is made possible by the interaction of virtual, augmented and physical reality is now also known as the »metaverse.«

Inflexible and ruthless – the old image of the industrial robot

For a long time, industrial robots were known primarily as steel motion machines in the automotive industry. But advances in sensor technology and algorithms are now increasingly enabling new and more flexible applications. This is opening up new processes and even entire domains for robotics. Contrary to many prejudices, however, the focus is often not on replacing, but on collaborating with humans.

Conventional industrial robots are characterized by their tireless diligence and consistent quality of work. At the same time, they are essentially rigid systems that continuously work through a predefined list of instructions. Accordingly, a carefully structured environment is necessary for robots to develop their capabilities. In the process, humans not only introduce unrest into a carefully balanced system, they often have to stay out of it for their own safety. After all, robots are basically blind to begin with and have neither the ability to notice the presence of humans nor the cognitive capacity to maneuver safely around them.

Stronger together

New sensor systems and intelligent approaches to data processing provide the basis for addressing these sensory and cognitive deficits of robots. New developments in force control and safety technology also permit them to leave their protective fences without posing a threat to humans. The interaction of intelligent robots and modern safety technology enables entirely new types of division of labor, which allows for parallel activities of humans and robots in closer spatial and temporal proximity.

Previously manual activities can now be partially automated without valuable space being taken up by an additional safety cell. The greatest advantage of such approaches is that humans and robots can merge their different strengths. While robots continue to contribute significant power and endurance, humans can compensate for the weaknesses of conventional automation systems with their problem-solving skills and fine motor skills. Workplace ergonomics can also often be significantly improved by converting to human-robot collaboration.

Fraunhofer IAO came to a similar conclusion in its 2018 »Homo Digitalis« study. 70 percent of respondents said they would leave physically demanding tasks primarily to robots. Conversely, however, only a few trusted robots with the necessary decision- making and problem-solving skills to act independently. However, one in two could well imagine joint interaction between humans and robots.

© Fraunhofer IPK / Larissa Klassen
PowerGrasp uses compressed air to strengthen and support movements and thus relieve the strain on workers, for example in assembly.

Robotic assistance systems

One way to leverage this potential is through direct physical interaction between humans and robots. At Fraunhofer IPK, new generations of robotic systems are being developed that can safely transport loads such as 35 kg packages of solar panels in a human-robot collaboration. Communication between the partners takes place through direct contact. Sensors determine the force exerted by a human on the package, and intelligent algorithms use this information to calculate the intended movement. Deep integration of the system into the robot's control system means that the processes take place within a control cycle, giving the human operator the feeling that the package is attached to a guiding rail along which it can be moved effortlessly. In the SHERLOCK project, researchers are also developing processes that use similar methods to allow simple interaction with robots in order to align components in an ergonomically favorable way.

Robotic support can also be much more discreet: Using a cordless screwdriver to screw components together overhead can fatigue the arm muscles in the long run. The active exosuit PowerGrasp uses compressed air to strengthen and support movements and thus relieve the strain on workers in assembly, as an example for a use case. The system is designed as a textile vest and does not restrict freedom of movement. Thanks to artificial intelligence methods, PowerGrasp recognizes both the type of movement and the degree of fatigue and can provide targeted support.

 

Artificial Intelligence for more flexibility

Human-robot collaboration also brings new challenges. For example, robots must learn to deal with changes in their environment that are initiated by humans and are thus unpredictable for them. They must therefore be able to detect deviations from a target state and subsequently react to them dynamically. Humans and robots must also be able to assess the current (movement) intentions of their counterparts.

To do this, the machines must become intelligent: Image processing algorithms, for example, make it possible to identify an inaccurately positioned screw and successfully carry on despite a deviation from the planned state. AI cannot only process images and videos, but also movements. For example, the PowerGrasp vest described above contains accelerometer and gyrometer sensors that measure and classify movements. To train the exosuit's artificial intelligence, the researchers first collect a dataset of typical hand movements and activities from the field of mechanical engineering in six-dimensional motion data. Neural networks with time components can not only recognize the activity in this, but also evaluate fatigue states.

 

Better together

AI plays an important role in enabling technical systems to follow humans. In industrial production, numerous and sometimes very complex tasks can be automated. Not every scenario and every human movement can be anticipated and appropriate actions programmed. Instead, robots must learn to look, listen, feel and thus integrate themselves into the working world of humans. The more real-time information about their working environment is available in the process, the better. To get the most comprehensive picture possible of their environment, robots need »sensory organs« and a »brain«: suitable sensor technology and advanced data and information management.

Vice versa, humans also need support in communicating with machines. Digital assistance systems bridge the boundary between physical and digital worlds and lead to the metaverse advancing ever deeper into production-specific processes. This revolution in collaborative working, which focuses on people and their strengths – their almost unlimited creativity, but also certain fine motor skills – is helping to meet the challenges of today. For example, demographic change is causing an increasing shortage of skilled workers, which is particularly glaring in the technical sector. Aging workers and their invaluable expertise will soon be missing from companies. Virtual assistance systems have the potential to help both qualify new employees and reduce human effort overall.

From the smallest unit of individual humanmachine interaction to large cyber-physical systems – at Fraunhofer IPK these topics are being advanced in many research and development projects. Our researchers are shedding light on important aspects that have often been neglected in everyday industrial life. For them, classic automation expertise goes hand in hand with software development, data management, factory planning and other relevant disciplines. This allows our scientists to transcend the »block thinking« between physical and virtual worlds that is still widespread in industry and thus develop the best possible solutions in the area of assistance systems, for example. In this way, research and development ensures that people have sustainable and useful helpers in robots and other machines in the long term.

You might also be interested in: