From the stage of F8, Joaquin Quinonero, Facebook’s Director of Applied Machine Learning, described a new technique the company is using to improve

 the watching experience for 360 videos. The format is challenging to deliver because of its size, but Facebook is using machine learning to reduce the number of pixels that have to be rendered at any one time. By predicting where a viewer will look next, rendering priority can be given to that location  — particularly helpful for users with lower quality internet access.

The status quo for 360 videos is reactive rather than proactive rendering. Mike Coward, engineering director for Facebook’s VR video team echoed the frustration of users to me when he described the unpleasantness of turning your head in VR only to see a blurry scene.

One partial fix is to optimize compression. But teams at the company are already using machine learning to select across the thousand-plus compression techniques for individual snippets of video. The other way to reduce the streaming load is to just cut down on what you’re rendering. And rather than reduce quality across the board, Facebook’s approach improves resolution for exactly what you’re most likely to look at next.

Step one was to use the resources of the company to monitor where people actually do look when watching 360 videos. Facebook’s VR video team created a heat-map that highlighted the most popular spots that users looked at within videos. From there, Facebook built a generative saliency map using a deep neural network. This model makes it possible to perform predictions on new videos that haven’t previously been watched or studied.

If a human were to be given the task of predicting where someone might look, they might study their natural environment and look for anomalies that could catch one’s interest — think birds or a car driving by.

Abstracting away to the neural net, the physical cars and birds cease to matter. Facebook’s model was trained on a massive corpus of videos to identify interesting subsets of a video frame. Coward told me that the model, when faced with a surfer in the ocean, is capable of picking selecting the surfer as most interesting, despite the fact that both are fast moving entities.

After implementing the prediction model, Facebook was able to increase resolution by 39 percent on VR devices. Aside from improving resolution and making 360 videos accessible to people without great network connections, the technology could some day make it possible to offer preemptive suggestions to creators on how to make videos more engaging.

In the last 12 months, the 360° video movement has grown rapidly. New cameras, headsets, delivery systems and content seem to be emerging daily. This quick post and video tutorial will cover some of the inherent challenges involved with editing and finishing 360 video.

What is 360 Video?

If you are on Facebook or read the NY Times online, you have most likely seen the video posts.. click on a video and pan around the 360 video. You may own an attachable headset for your mobile phone (Samsung Gear, Google Cardboard, Daydream) or even experienced “immersive VR” on headset wired to a computer (Oculus Rift, HTC Vive).

While VR is a general term for content viewable with a headset, we will focus here specifically on “cinematic VR” or 360 video. 360 video is generally shot with an array of synced cameras than stitched into an equirectangular video format (sometimes called lat/long)

Editing and finishing professional quality 360 videos generally involves a fair amount of post work. The equirectangular format introduces pixel distortion (think of a sphere unwrapped to a flat image) and simplest tasks like adding titles or isolated color correction need consideration.

Some of the most typical tasks include:

• Reorienting footage
• Removing Objects
• Masking for color correction of fixes
• Inserting 2D graphics into 360
• Stabilizing and smoothing camera motion

Click here to read the full article on Avid Blogs

A Tutorial on 360 Video Just for You!

Teaching people how to capture 360° video is fundamental to the success of virtual reality. That’s why we are thrilled to partner with Jeremy Sciarappa, to create an end-to-end comprehensive tutorial on capturing 360° video.  

Since 2014 Jeremy has been a full-time YouTuber, creating tutorials and reviews for cameras. Now he is focused on 360° capture and post-production.  

The video titled “360-Degree Video Workflow: 360Heros Mount and CamMan” covers the full end-to-end 360 video production process: 

• Shooting using our Pro6 six-camera rig
• Media Management using 360CamMan
• Stitching footage filmed with different mounting positions 
  
  “I’ve used the rig to shoot several different projects, and the experience was great.” –
  Jeremy Sciarappa on working with the 360Heros Pro6