“Whenever new images get downlinked from the rover, we go through and build a full 3D scene automatically in the cloud, and that streams to the scientists’ devices,” Menzies explained. “So, they are able to put on the HoloLens and walk around on the surface of Mars shortly after the downlink is completed.”

Establishing this kind of immersive visual experience of Mars gives researchers the ability to plan missions and make scientific observations on a much more intuitive level. Rather than scrutinizing the wide-angle pictures taken by the rover’s stereo cameras, OnSight constructs full scenes mined both from the rover’s perspective, as well as orbital data collected from Mars’s suite of observation satellites. The choice to go with an augmented reality device like HoloLens rather than a virtual reality headset like Oculus allows teams to keep one foot in the office, so that they can keep track of the data they are monitoring and analyzing.

“Our strength is that we combine all of those images and build one consistent 3D model of Mars,” Menzies said. “Then, we put it on somebody’s head and we render not through the rover camera model, but straight into the human eye, with a human perspective at a human scale.”

“You can actually walk up to four kilometers in OnSight right now in a single location,” he added. “We actually build locations for every place the rover has visited, like a series of different scenes.”

In this way, planetary scientists can wander through Gale Crater, where Curiosity is located, and study Martian surface features up close, using the same observational skill set that they’re trained to use in the field on Earth. On top of that, the Curiosity team can figure out where the rover should drive to next by using OnSight to scout out the best path to follow. Obstacles that might appear relatively benign in still images are sometimes revealed to be dicier when viewed through the HoloLens, while geological formations that would not otherwise have been especially noticeable “pop out” in the display, according to Menzies.

“What’s really powerful about this is when you stop to think, ‘this is all real,’” he told me. “These are all real images made from real photons bouncing off the Martian surface. This is what it is actually like to look around on another planet. Mars stops being this abstract blurry orb in the sky and it becomes a place that you can visit. You can hold meetings there. You can go on a date there if you really wanted to.”

“That’s what is interesting to me,” he continued. “Mars may not have trees, and deer, and oceans, but it’s got an immense amount of different geological structures that are exciting to explore and in many ways much more extreme. The canyons are deeper, the mountains are taller, the hills are steeper, and so you get really interesting stuff.”

.

Buzz Aldrin playing Martian tour guide in Destination: Mars attraction. Credit: NASA/JPL-Caltech/Microsoft

OnSight is in active mission use now at JPL, but NASA has plans to bring it to the public this summer with a new interactive exhibit at the Kennedy Space Center called “Destination: Mars.” Visitors will don their own HoloLens headsets and be guided through some of the scenes constructed through OnSight by a holographic Buzz Aldrin.

Ultimately, though, this fruitful partnership between mixed reality interfaces and space exploration has the potential to fundamentally shift the way that humans interact with the wider cosmos, and the robotic spacecraft we’ve sent to explore it.

“We are just barely beginning to scratch the surface of what technologies like augmented reality and virtual reality can achieve,” Menzies said. “It is no wonder that we have to wipe tears off our displays because people have experienced Mars for the first time.”

“We are entering a new era of space exploration, one in which we all get to go.”

Quelle: MOTHERBOARD