We desperately need to have amazing content for virtual reality to go mainstream. And you can create it today. There are several ways how to do it. One way of doing this is by using computer software (Unity, Unreal or Cryengine) to create it. Created content is imaginary and is usually used to build VR games. But what about if we want to capture the real world? In this blog post, we will look at some of the VR capture methods that will allow you to bring the real world into virtual reality.
360 video and photo capture
360 videos are created by filming all 360 degrees of a scene at the same time. Users can view the video from any angle. Turn and move the device and the 360-degree video will follow, creating an immersive “virtual reality” type experience. 360-degree video is typically recorded using either with a special rig of cameras or using a dedicated camera that contains multiple camera lenses embedded into the device. The resulting footage is then stitched to form a single video. This process is done either by the camera itself or using specialized video editing software that can analyze common visuals and audio to synchronize and link the different camera feeds together.
Captured content can be separated into either monoscopic or stereoscopic 360 content. Stereoscopic 360 content allows you to have a 360° overview of the environment and creates the 3D effect with objects close by. Monoscopic 360 content lack the 3D effect but is much easier to produce and easier to distribute.
Light field capture
Unlike standard 360º video, light field video captured with Lytro Immerge allows for six degrees of motion freedom within the camera’s volume, which is about one meter. This means that you could conceivably move around within the volume of the sphere and lean in toward objects. In addition to adding a degree of positionally tracked volume to the scene, a true light field recording like Lytro’s creates both horizontal and vertical parallax, giving the scene true depth and perspective regardless of your viewing angle.
Light fields are a relatively simple concept and are not that new, but their actual implementation is extremely hard to pull off. A ‘light field’ (a.k.a. ‘plenoptic function’) is really just all the light that passes through an area or volume. Physicists have been talking about light fields since at least 1846. Lytro really popularized the idea by developing the world’s first consumer light field camera back in 2012. A Light Field can be captured using an array of multiple cameras as well as a plenoptic device like the Lytro ILLUM, with an array of microlenses placed across its sensor. The core principle in both cases is that the Light Field capture system needs to be able to record the path of light rays from multiple viewpoints.
With Lytro Immerge Light Field camera, the light rays’ path is captured via a densely packed spherical array of proprietary camera hardware and computational technology. In its spherical configuration, a sufficient set of Light Field data is captured from light rays that intersect the camera’s surface. With that captured Light Field data, the Lytro Immerge system mathematically reconstructs a spherical Light Field Volume, which is roughly the same physical dimension as the camera.
Volumetric 3D capture
Volumetric VR gives you the a feeling of reality. Users see scenes with three-dimensional humans — who actually look like real people — but they can also physically walk around these “characters” and watch them from any angle. Unlike film, there are no “takes” or “shots” in VR that are edited in post-production — it’s much more fluid as the viewer is the one framing the scene and choosing their own perspective. In that sense, the viewer takes that role from the director, which opens up entirely new possibilities for storytelling and acting. One of the companies working in this area is 8i.
While traditional approaches to VR content turn cameras outward, 8i turns the cameras inward. 8i uses off-the-shelf high-definition cameras to record video of a real person from various viewpoints. Then it uses its own software to capture, analyze, compress, and recreate in real-time all the viewpoints of a fully volumetric 3D human. You can see an example of such capture below.
The captured volumetric video can also be used for AR. In the clip below you can see how Microsoft captured holograms that can then be viewed through their Hololens.
The fundamental principle used by photogrammetry is triangulation. By taking photographs from at least two different locations, so-called “lines of sight” can be developed from each camera to points on the object. These lines of sight (sometimes called rays owing to their optical nature) are mathematically intersected to produce the 3-dimensional coordinates of the points of interest. A somewhat similar application is the scanning of objects to automatically make 3D models of them. Some programs like Photoscan, or 123D Catch have been made to allow people to quickly make 3D models using this photogrammetry method. It should be noted though that the produced model often still contains gaps, so additional cleanup with software is often still necessary. Microsoft is also making their own play in this area.
Photogrammetry is already used in different fields, such as topographic mapping, architecture, engineering, manufacturing, quality control, police investigation, and geology, as well as by archaeologists to quickly produce plans of large or complex sites and by meteorologists as a way to determine the actual wind speed of a tornado where objective weather data cannot be obtained. It is also used to combine live action with computer-generated imagery in movies post-production. Photogrammetry was used extensively to create photorealistic environmental assets for video games.
What to do with the captured material?
Let’s assume you’re interested more in 360 video and photogrammetry as these technologies are already available to the public today and don’t require a steep learning curve. Once you recorded and post-edited a 360 video you can then publish it on an online video platform like YouTube or you can build an interactive VR story on a platform like Viar360. Interactive stories give the viewers ability to move from one 360 video to another and this way control how the story unfolds. If you have used photogrammetry to capture a 3D model with one of the 3D scanning apps, then you can take that model and place it into a CGI environment with game engines like Unity or Unreal. In the second half of 2017, you’ll be able to merge your videos recorded with your 360 camera and 3D models captured with your 3D scanning app in Viar360. But lets leave this for another time.