the below video has been floating around, like the images do inside them, but what makes it tick and and where does the technology come from?
Microsoft has several research labs in the world and Cambridge, UK is an important one. Several recent examples of their work: the functional programming language F#; the body part recognition technology in Kinect for Xbox 360; and the advert prediction algorithm used in Bing. The lab has over 100 researchers from all over Europe.
Their latest pet project ´Vermeer´ was introduced on CES, the huge trade fair, and is a novel interactive 360° viewable display.It enables simultaneous users 360° viewing of the 3D object. On top of that by smart usage of the Kinect technology you can interact with the hologram.
Let´s focus on a key element: the 360° viewable image. The news reports we have seen are vague and just copy the text of the youtube movie. Let´s dig slightly deeper without becoming too technical.
The first thing you notice is that you don´t need to wear glasses at all to see a 3D image floating. The principle behind it is already old: it´s called auto-stereoscopy. I have such a monitor since 1999 on my desktop so I know very well how great it is to work without these obnoxious glasses, but the limitations are also clear. Auto-stereoscopy has a smart optic layer over the screen which acts as lenses to split the image to both eyes. Net result is a perfect 3D image, but…..
The viewing angle is very, very limited. In case you recently saw an Nintendo 3DS you will have seen auto-stereoscopy at it´s worst at work. Move the little toy a bit and all you get is headache. There are solutions to get around this problem by tracking your pupil and adjusting the image based on the position of your eyes.
They work perfectly in my experience but they only allow for one user. Researchers all over the world are busy with multi-user implementations. So far the fruit of this labor has made it possible to view a 3D television without glasses with multiple people on a sofa. Not exactly with the price tag to conquer the living room, but a great achievement. However it´s still far from a hologram.
The Cambridge researchers use a quickly rotating diffuser to split the image not to one very narrow sweet spot but to 360°. This 3D auto-stereoscopic display is based on volumetric lightning, in the hardware world known as light field displays. So suddenly you can move around the object and still have that convincing depth image.
To create this effect the ´Vermeer´ creates 192 different views per second which are split in a full circle. So from your specific point of view standing still you only get a limited number of them, but good enough to freely move your head.
But what you see is not what someone else sees standing at the other end. So we need to render more viewpoints from all angles we project the light to. The more viewpoints you can render the more fluid the image will be so Microsoft Cambridge had to render large numbers of frames per seconds. As you can see in the video this creates problems hence the flicker you will notice from the fixed camera position.
To understand why just take the normal gaming experience where you see 60 frames per second. More is possible thanks to very fast graphics cards but it´s usually fixed at 60fps as this is the speed the majority of current monitors uses. Even fantastic graphic cards will have trouble rendering 120 frames per second.
Now imagine having to serve 192 monitors which all look at the scene from a slightly different angle to create a hologram. If you want perfect fluid motion defined as 60 fps you would need to render 60×192 frames per second. A staggering 11520 fps! This is currently unthinkable so the Cambridge specialists have settled for the minimum acceptable speed of 15 frames per second which the eye will interpret as almost fluid if there is no motion inside the view.
If you want to view a tennis match this way you would have interesting shots of two exercising players but would never notice a ball. Exactly what happened with the first generation of of LCD tv´s as you may remember. That´s why the emphasis is on a static image. For fast moving objects you need a much higher fps.
The UK based Cambridge researchers goal was to be able to project a static image so they researched technology to get to the minimum level of 15 fps. They found the suitable technology in the new GPU based computing. By offloading relatively simple tasks to a modern graphic card you can realize a speed up of 10-50 times over the fastest available general CPU.
The CPU/GPU processing became hugely popular the last few years and large companies like Zeta, responsible for all the graphic animations in films like Lord of the Rings have replaced their old servers with stacks and stacks of special workstation class graphic cards.
It´s usually based on Nvidia video cards as their Cuda framework is the most advanced but unfortunately closed source. Thanks to the relentless support by the highly capable NVidia engineers they have been most successful. They help large companies get to convert their huge clusters of PC´s or their super computers by stacks of graphics cards which function as computing cores is instrumental. An open source alternative is OpenCL. Both approaches use the many cores of a video card (up to 512 per card!) to compute in parallel the needed information.
The net result: a staggering 2880 images per second are rendered to create the hologram. And yes that number is exactly the 15 fps for 192 viewpoints.
We won´t dig deeper in the actual image trickery to produce a hologram as it would make this post way too long. For the same reason we will skip the way the Cambridge lab uses a Kinect to track the fingers interacting with the hologram.
For further reading I recommend the ACM publications if you happen to have access to them. Vermeer has been published in the October issue. An new publication about the holodesk is planned for an ACM publication in April. This ´behind the scenes´ article is just based on my personal knowledge of the technologies involved so all errors are mine (_Max Huijgen_)
Feel free to amend the article in the comments or ask question