Augmented Reality

Digitisation in the working world has become a matter of course for many companies. It is already well advanced in the planning and engineering of logistics systems. The demands on cycle and response times as well as the use of sophisticated technologies have intensified in recent years. Augmented reality is primarily used in warehouses for inventories and picking. It is as yet less commonly used for the efficient set-up and operational maintenance of logistics systems. If, for example, the installation progress is delayed, this can have serious consequences for the project lead time; a system standstill and idle times lead to lost turnover and extra costs. The manufacturer has to intervene as quickly as possible. Specialist know-how is needed in many cases. Unfortunately, the specialists are often far away, so that valuable time is lost. This is a significant cost factor with globally installed systems and may tie up valuable resources for several days. If we also bear in mind the fact that the problem can often be solved by a specialist in next to no time, we quickly realise just how big the influence of travelling time is on the downtime or cycle time.

The question thus arises as to how fitters, service technicians, commissioning teams and customers can be given fast and competent support on site using the know-how of specialists. How can unnecessary service calls and empty trips be avoided? The use of augmented reality (AR) appears to show great promise here. This prompted Gilgen Logistics to launch a pilot project.

Gilgen Logistics found valuable partners for the development in the Institute for Information Systems of the University of Applied Sciences and Arts Northwestern Switzerland and the Computer Perception & Virtual Reality Lab of the Institute for Human Centred Engineering of the Bern University of Applied Sciences. The project was also supported by Innosuisse, the Swiss Innovation Agency.

Since the basic functions of a mixed-reality headset are already in general use, the team focussed on marking and animating components of the elements. The supporter hereby marks those components of the virtual 3D model that are to be highlighted in colour. The operator can then be shown animations about the assembly or installation sequence, position or setting/adjustment.

The operator is on site and experiencing problems with the system that cannot be solved without the help of a specialist. They therefore put on an AR headset, connecting them to a supporter via the microphone and Voice over IP. The supporter can see the live image from the AR headset’s camera on their own monitor. The headset can identify the type of element by means of a marker and downloads the corresponding 3D data from the Gilgen Logistics data server. The element is then augmented, i.e. the virtual 3D model is superimposed over the real element. The supporter sees the same 3D model on their PC and can now mark, show and hide individual faulty components for the operator and guide them through the repair, installation or troubleshooting process. All of this time, the operator has both their hands free for work.

A number of relatively new and complex IT technologies had to be combined for this pilot project:

Several network servers were needed to exchange the data. On the one hand, this guarantees a continuous flow of data for the 3D synchronisation. Because this exchange has to work with a very low latency, a software was used that can also be found in multiplayer online games. On the other hand, the two clients (HoloLens and browser app) had to be linked.

A technology that is familiar from video conference tools was used for real-time communication. This allows the web browser and the HoloLens to exchange video and audio streams directly. The HoloLens is hereby connected to the Internet via WLAN.

Gilgen uses the HoloLens headset from Microsoft for the application. This AR headset shows the wearer 3D objects that are not really there in the room. This takes place stereoscopically (separately for both eyes) and in real-time. The most important aspect here is that the headset can determine the wearer’s position and orientation, and thus their line of vision, very quickly and precisely. The HoloLens detects the room with a 3D depth camera and infra-red sensors. The Unity game engine is used to generate the 3D graphics. Gilgen machines and their exact positions are identified with the HoloLens by means of a QR code.

The 3D objects that are shown in the HoloLens also appear in the supporter’s browser. The supporter can also rotate the 3D model in any direction. The most important innovation in this project was that the supporter can now select and highlight a part that is causing problems in their 3D image of the machine. The selected part is transmitted to the HoloLens and the supporter can give the operator precise instructions, just as if they were standing next to them.

“This project has shown us that augmented reality in intralogistics has a great potential for improving efficiency,” concludes Daniel Fricker, Head of Developments. “The use of augmented reality improves the safety and trust of customers in using the system. This pilot project has laid the foundations for further projects, such as developing a network link between the system and the Gilgen support department, bringing us closer to our goal of the ‘efficient processing of realisation projects and the best possible customer support’.”

Our Head of Engineering - Daniel Fricker - tests the brand new Hololens 2 from Microsoft. It will soon be in live use by the service technicians.