Microsoft HoloLens. How do they work?

Without any doubt Microsoft's keynote on jan-21 got an epic "one more thing". Epic. Microsoft introduced its HoloLens and everyone were impressed. In this article I'm going to try to explain how this HoloLens works using what I listen in the show, what is shown in the promotional videos and what I know about previous products from Microsoft.

HoloLens are not like Google Glass

This is the first thing I must clarify. No, they have nothing to to with them. The Google Glass works putting a tiny screen in the visual way of your eye. In front of only one eye. Then your brain mix what you see with your left eye (the one with the visor) and your right eye (without the visor) to make you think you see a screen in front of you.

With Google Glass that virtual screen you'll see in front of you will move while you move your head. Let's say you move your head over your right shoulder, the virtual screen will remain in the center of your vision. The screen moves with you.

It can not be otherwise, since Google Glass does not have enough power to do otherwise.

How is the effect? Take your mobile phone and put it in front of your nose; Now move your head sideways while moving the mobile to keep it always in front of your nose. That is the effect with Google Glass.

The HoloLens doesn't work the same. The HoloLens shows that virtual screen as a hologram. Want to see the effect? Simple. Look at the monitor where you are reading this. Try to keep it centered on your view. Now turn your head to your right trying to keep on reading. See? The monitor now appears on the left side of your view.

If you keep on turning your head would come a moment when the monitor no longer would be seen, but it's still there, if you turn your head the other way you'll see it taking up space in your view gradually until being at the center of your view again.

How Do They Do It?

As I mention before the HoloLens are not the same as Google Glass. The HoloLens contains a Kinetic but focusing outwards. The Kinetic of XBox looks to you in your room and creates a 3D model positioning yourself as volume inside the room. The Kinetic in HoloLens see what you see, the room, and it generates a three-dimensional model of it.

Over this 3D model, with the program inside the HoloLens, the user chooses a surface, such as a wall, where he wants to place the virtual screen. The Kinetic in the HoloLens records that position in a 3D space and the program maintains that "hologram" in that position while you move your head.

The HoloLens integrates, apart of the "kind of Kinetic", also many other sensors that report the position of your head, movements and orientation determing the 3D position of your hear. That gives the necessary information such as the environment and where we are regarding it.

It's like setting a painting in a room. You choose a wall and hangs the painting there. The painting will always stay there and you'll see it from different positions while you move, but of course the painting is not going to follow you.

In HoloLens this virtual screen is like the painting on the wall. Once placed, while moving, the software in HoloLens maintains the position that painting / virtual screen in the same place, showing it in the glasses where they calculate that the painting should be.

That with a virtual screen, but HoloLens go further. Suppose you're a designer of motorbikes. You're sitting at your PC where you have an Autocad-like program where you have your wire-frame model of the bike. Well, with the HoloLens you can make a 3D model of the bike to appear as a 3D hologram to the left of your desktop. As you make changes with the program on the PC model of bike this updates your 3D hologram model in the left of your desktop. If you look at the 3D hologram model you will see it there. Your hand will go to it... and through it, of course, is just a hologram.

If you get up from your desk and look at the 3D model you will see that your glasses treat the object as if it were a real model and you can move the head around it and the software in the glasses will calculate what angle you're looking at and will adjust the 3D hologram position without your knowing. The effect is that you are viewing the hologram from another point of view and your brain believe that the object is there and it is real.

Now the Google Glass look extremely primitive, right?

In HoloLens if you need it, you can have a floating screen, not generated in 3D space but to follow you as you like with Google Glass, for example to watch a video while walking or videoconferencing. But the power of HoloLens is another, defining the 3D objects in space and keep them there and adapt its vision to the position you occupy in the room.

Do they need a PC?

No. HoloLens have three CPUs inside, memory and storage. The HoloLens are a PC. They are the "next-PC". They do not need a smartphone, no PC or tablet. Computing power are in those three CPUs.

The three CPUs are: a traditional CPU that handles the rough calculation, a traditional GPU which is the graphics coprocessor that all PCs or smartphones have, and HPU that handles all calculations of 3D holograms.

How do you interface with them?

HoloLens can be controlled in two ways. Those are the ways in which you control an XBOX with Kinetic. The voice and gestures.

The voice is simple, the HoloLens hear your surroundings and listen to your commands as does my XBOX Kinetic when I say "Shutdown XBOX YES" and XBOX turns off.

The other method is with hand gestures. Imagine that we define a hologram of a vase of flowers on the table of our living room. If we make the gesture of grabbing the object the program in HoloLens assumes you realy grabbed the vase and will move it following the moving of your hand until you let it go over another surface.

Or for example make the gesture of clicking on an virtual screen object (like the painting in the above example) and a menu with options will pop-up.

O zoom gesture with two fingers to enlarge a hologram.

All these gestures with your hands are seen by the cameras of the Kinetic fitted in the HoloLens and the program will interpret what you want to do.

In the XBOX Kinetic I show my open hand and move it in front the console like a PC mouse. This movement is reproduced on the screen as the mouse pointer. When I close my hand into a fist, XBOX see it through the cameras of the Kinetic and assumes that this gesture is like a click and translates it into a click on the screen by opening the game you have selected.

This works just the same with the "eyes" of Kinetic in the HoloLens which sees your hand from the same prespective your real eyes, but the principle is the same.

Uses of the gadget

Hundreds. I'll show you a few examples.

Place a "painting" on the wall which is actually a screen where you play videos from Netflix. You could see it from different angles as the real TV in your living room.

Working with 3D models that you are designing and being able to watch them in real time. You can build them with your hands, rotate them, paint them, place them in a specific area of your room and leave them there. Off-course, leave them there... in the virtual reality.

Stick notes on the refrigerator. Virtual notes will remain stuck to the refrigerator door.

Games that use the furniture in your room like game arena. You'll see the character jumping on your couch, or jumping from the couch to the floor or to the table. This is possible because your HoloLens use the 3D model of the room with furniture for characters to interact with.

Lego stlye construction blocks which merges with your furniture.

Remote assisted works so the program in HoloLens shows the parts to use and what you must do with them. For example if HoloLens were connected to the computer in your car it would show the pressures and measurement in the probes in the engine and the position the figures over the real probes.

Posibilities are hundreds such as seen in these two short videos filmed by the brand.

Conclusions

It was bang on the table from Microsoft to display his power of invention, innovation and especially in a moment when Google is recalling its Google Glass and before Facebook and its Oculus Riff hit the market.

The world of games is changing and the way we interact with them too.

And this is not vaporware. Microsoft did a terrific live demonstration yesterday.

You can see it at this website, in the "Webcast on Demand" section get the video played and use the bottom bar to play from 1:53:00 to 1:58:02. It's only five minutes, but of a great live demonstration of this amazing technology.

I love this new Microsoft ... thanks Mr. Nadella for returning the illusion.

PS: The club of the presbyopia, myopia and other eye diseases ask ... Can be used with corrective lenses? WE DON'T KNOW.

This article was originaly published in spanish at WINTABLET.INFO and translated by the author himself