You’re driving on a gloomy day but the road is clear. As you round the corner, you suddenly see a large animal out in front and slam hard on the brakes, narrowly missing it. Again. As you pause for breath, happy that unlike the first few times you avoided a collision, the startled beast flickers and is gone. Road test complete, you take your headset off and get out of the car, checking for other traffic on the test route.
Even the most technologically sceptical of us out there are probably plugged in, connected and attached to our phones, tablets or PCs in ways that were unimaginable barely a few years ago. Last month, T&FME visited – appropriately it was a virtual tour – Volvo Cars’ research hub to see how tests like the one at the top of the story fuse cutting-edge technology like augmented reality (AR) and virtual reality (VR), computing horsepower and engineering used in our day-to-day lives to help create its world-leading safety development programmes.
“Look, we’re not promising that your next Volvo will come with a CGI co-pilot or something,” says our guide Robyn Nel, “but what we are learning as much as we can about human behaviour through the use of technology – and all of that learning is going to go towards helping us create safer cars.”
Much of this work is conducted at Volvo’s Gothenburg hub and within the campus’ Open Innovation Arena. Through a maze of screens, carved open car interiors and cabling, its teams are working with an array of technology to take simulated trips into an alternative reality.
Combining the hallucinatory with the holistic, they are learning with every trip how to fix situations that would be impossible or pretty unsafe to look at in the real world, says Timmy Ghiurau, Innovation Leader, VR and AR Expert.
Crucially, it means that they can test new safety features far earlier in development, slashing time and money.
“The ability to simulate every possible scenario as early as possible in the process helps us to be much more efficient which means we can spend much more time to improve the technology to be better which means safer, ” he tells T&FME. “We can also have higher accuracy to understand user behaviour in the context of autonomous driving and interaction with safety systems.
“When we’re developing self-driving cars and technologies around the collision avoidance for instance, it is really important that we cover all possible scenarios that could happen on the road. And it’s really important that we try to see the human behaviour in this aspect of though because some of these scenarios are either too dangerous to test in real life or they’re too rare – which would mean that you will need to drive forever to actually for this scenario to occur.”
You may assume that could mean endless replays of pedestrians suddenly stepping in front of your vehicle, but in [ahem] reality, Ghiurau and the team are looking at a huge array of potential situations.
“For instance, we can try and see how the car will react to a scenario – and how you will react – when a car in front of you has a mattress tied to the top and that mattress goes loose on the highway just a few meters in front of you,” he says slightly sinisterly.
“Not only that, but we can actually simulate that scenario and can play it out over and over again with slightly different variations and different parameters to make sure that we cover those things. We can try bringing more users in the loop and we simulate it.”
He adds: “We not only focus on the car sensors and how they detect it but also how you as a user respond to it and by doing that we’re trying to cover when the impossible might come up.”
Unsurprisingly, Volvo has gone straight to the technology that powers games consoles and PCs for inspiration.
“We are really inspired by what the gaming community is doing,” says Ghiurau. “They’re always pushing the boundaries of technology of rendering VR and AI. We are also trying to challenge them with real-world use cases such as what we do in automotive– and it’s a good way of doing it since we don’t have to reinvent the wheel over and over.”
Volvo realised early that it would need to look for other firms to partner with.
“We started to identify the gaps within our processes and all the parallel initiatives within simulations and virtual experiences,” he explains. “After our needs were identified, we started to collaborate with our partners, listened for internal and external requests and we started to evaluate these technologies.”
Volvo turned to one of the world’s leading game engine developers Unity (games using its engine tech include the popular Ori and the Will of the Wisps from Microsoft and Nintendo’s Mario Kart Tour) to create the 3D virtual world it needed.
“We can learn from what has been done in games for tools that we’re using today,” continues Ghiurau. “Volvo Cars has been using simulator technologies for many years, however for the past five years we have been collaborating with Unity and evaluating their simulation solutions in our toolchain. The game engine from Unity is a real-time 3D Tool and here at the Innovation Hub we can tie all these things together and create virtual environments. We bring our cars; their sensors. And then we have VR headsets where we can actually put the user into the simulation and track their eyes and interactions with our systems.”
If that wasn’t enough sensory information, the team are using another state-of-the-art gaming tool, Tesla suits, which allow users to touch and feel inside a VR world. Ghiurau says the suits haptic technology lets you track how reacts physiologically when placed into often hazardous real-life scenarios.
“The possibilities are endless especially for training situations where we need to perform the same movement,” he says. “We can actually record and simulate it over and over again. With the biosensing the suit is equipped with we can see the user’s stress levels or heartbeat and if we combine that with eye tracking it gives us a good impression of how the users are feeling in a certain traffic scenario.”
Test subjects wearing the pulsating and tight-fitting suits may look like they are about to embark on a trip to outer space, but Ghiurau explains the sartorial inelegance is accelerating development of an array of safety technology: “We get the advantage of trying the latest things and incorporating it in our tool chain. It is pretty similar with a game, as I said, so we can take advantage of millions of bright brains. They’re part of a community that developed environments, props, streets and other scripts that we can reuse so we don’t need to reinvent the wheel which improves the efficiency process.
“The main advantage is the extra immersion added through high resolution VR headsets as well as haptic feedback. By using the same tools from an early design stage – through the research and development process, user testing, and storytelling trough consumer facing applications – we are able to reuse assets through experiences and different simulation rigs. This means we can be faster and more efficient because we don’t have to wait for a real car in order to test our technology. Instead we can focus more on quality. ”
Pointing to a screen, Ghiurau, shows us a digital car park with a Volvo (virtually) sitting in one of its bays: “We are trying a plug-in from Luminar’s LIDAR sensor and we can actually look at how we place the sensor on the car model and then how accurate it will get…we can add more elements in this parking scenario like a pedestrian in front of the car and our machine learning software can actually teach the car how to park itself.
“We can test numerous of scenarios like this. We can try every possible occurrence, but we can also put it on a driving rig where we actually do it in real time with the user in the middle.”
Switching screens now, he shows two identical pieces of footage taken by cameras on a car.
The only difference being that only one of the clips is real.
“If you think about it, the car sees the world through a set of cameras and sensors, right? On the left here we can see the output from our cameras on a test track and we see the lane detection and object detection and on the right. The active safety team replicates an environment using Unity where the road is digital and the car is ‘natural’ so we can actually see if our perception and detection algorithms work.”
The cameras don’t have to be strapped onto a vehicle either. One of the most impressive abilities shown by Ghiurau is the use of everyday phones and tablets to test the LIDAR system.
“The latest phones and tablets include such a sensor in the device. With the latest iPad we can actually scan the environment around us and we stream the point cloud (think an image made of dots) into Unity directly. We’re trying to capture the environment around us and use it for a really accurate augmented reality and overlay virtual objects anchored in the virtual space. This helps us to understand how the LIDAR sensors react to the environment in real time and prototype and do quick testing.”
VR is becoming familiar to a growing number of households but Volvo is using headsets that have resolutions that match the human eye to sidestep the challenges of the Coronavirus pandemic by allowing teams to share and collaborate on new car and feature designs in 3D across the globe.
“Adopting more virtual tools and methods is key to collaborate around research and development of such features, and being able to simulate virtually in a cost efficient and fast way,” enthuses Ghiurau.
“We can connect with our office in US and can look at the same car; have workshops meetings… I can even see my own hands and see the environment around me at super low latency that enables us to actually drive a car. We can do design reviews; look at a car concept. I’ve got multiple people who can do this at the same time. No matter where they are.”
One of the main purposes of the Open Innovation Arena is to test how drivers will interact inside an actual car interior without having to do the testing on the road. Standing besides a mock-up of a Volvo Car, Alexander Eriksson, senior design engineer – Human Factors, puts a fully (Tesla) suited and VR-goggled driver into a scenario where they are sharing control of a car with an automated system. The user experience expert explains that this is an easier way to test how humans will react with new technology, years before it’s ready to be used on a highway for real.
Eriksson and his team are concerned with every minor detail of how people interact in a car environment from pulling on the seat belt or how they may react to vibrations from the vehicle. The point, he says, is to recreate an experience as close to driving, so that a response to a new feature, like the switching off an autonomous driving mode to manual, is as realistic as possible.
“We can test how the users interact with the systems with the screens and also evaluate our car designs with the user experience,” he says.
Ghiurau adds: “We have a lot of learning about human behaviour and user attention. It allows us to understand how the car can communicate with the driver in the safest possible way. We learn where the warning signs are most effective and what senses needs to be triggered by using sound, signs or vibrations). We also learn how different user’s with different age, gender and cultural background react to certain traffic scenarios.”
If and when Volvo is ready to put autonomous vehicles on the road, it will have been Eriksson and colleagues’ research that will make it reassuringly safe for drivers.
He explains: “With that (switch of modes) it’s important to find the right balance between getting a fast response but at the same time it cannot stress them out. Should the driver remain unresponsive, then the vehicle can take measures to take control. (What we get) is exactly how your brain works when you want to perform an action… then we can actually track movement and calibrate.”
Arguably it is the work of Casper Wickman, Senior Technical Specialist – Human Centric Methods and Tools and team that is at the very sharpest edge of the technology being used in Gothenburg. One of his latest achievements was the invention and development of the Augmented Reality Car (ARCAR); enabling driving with an Varjo Mixed Reality Headset which overlays virtual content inside and outside the car. Volvo Cars is the first car maker to make this set-up possible and claims the seamless matching of high resolution cameras and computer generated imaging provides an almost limitless palette of pitfalls to be presented to a driver.
“Since the VR headset allows us to mix virtual reality with reality we can simulate situations from accident reports and other Volvo real-world data but in a safe way,” explains Ghiurau.
“The high resolution and quality of the assets adds to the immersion,and response times. We also have the ability to blend between realities in a seamless way.”
Rather than be content in racking up thousands of virtual miles in the safety of the Open Innovation Arena, Casper Wickman and team are taking the headsets and Tesla suits out onto the road in the ARCAR.
“We can increase the severity of the driving situation. And with that, I mean that everything becomes much more realistic in terms of your responsibilities as a driver. In a simulator, you can just start over again and take it from the beginning but when you’re actually driving….that’s where the behavior of the driver becomes really realistic.”
The ARCAR works with a driver being fed scenarios, such as a virtual deer running out onto the road ahead of them, by a tester and a laptop sitting on the rear seats of the vehicle: “A person can sit and stop the applications and start sequences that we want to try out. Due to the high resolution of the XL1 headset, it’s possible for the driver to read all the information in the car and even use the mirrors and the real-world camera when operating it. With this method we can do this over and over again and tune the safety system into a safe system. There are other manufacturers using (simulator) technology but this is unique. Volvo is the only one combining technology like this.”
While he can’t share the exact numbers, Ghiurau tells T&FME that Volvo Cars has seen significant savings, “especially in cutting lead time in increase efficiency. Some of these technologies already replaced super expensive traditional simulation systems.”
The technology and approach being utilised could yet also pave the way for an array of other applications.
“There are a lot of opportunities with this new technology to improve the mobility ecosystem overall, like city planning, more interactive mobility services, as well as traffic control, education, driving tests, etc. And, also, how the future of showrooms and dealerships could look like,” says Ghiurau. “We are looking into how these technologies can shape the future of dealerships, retail and the way our users will interact with our products.”
