Stereo VFX with 360° video


In earlier posts I’ve talked about my first VR experience and how it really resonated with me. One of the elements that really contributed to the impact the experience had on me was that it was CG. CG characters and worlds can be so fantastic; pulling imagery from our nightmares, creating environments and landscapes that amaze us, presenting us with characters and situations that stay with us.

My background is in computer animation. I’m in love with the notion that I can design, model and bring a character to life through animation – that it can engage with people and pull some kind of emotional experience from them. When I got the opportunity to work in 360° video production I knew it would also be a great opportunity to involve these CG elements. The major piece of the puzzle was being able to render from Maya with an equirectangular output. Added to that was being able to do this in stereo, as I believe that for real presence and immersion in 360° video, it should be stereoscopic. After plenty of research I discovered some of the tools that could make this happen.

I discovered this excellent plugin from Andrew Hazelden, the Domemaster3D Stereoscopic Shader.

Domemaster Stereo Cameras in Maya

There’s a great blog explaining how to use the plugin to render for stereoscopic 360° video here. Essentially, the plugin creates a stereo camera rig in Maya, that can render equirectangular images. There are also some key settings that are able to be changed, depending on the needs of your scene. Achieving a good stereoscopic pair isn’t just a matter of creating a stereo camera rig and pressing render – if you want some great reading on how to create strong stereoscopic imagery then I really recommend 3D Storytelling: How Stereoscopic 3D Works and How to Use It by Bruce Block and Phil McNally.

Some of the key Domemaster settings

If you’re rendering for 360° video, you’ll typically leave the Field of View (Horizontal and Vertical) at the default. The main settings to get started with in the Domemaster plugin are the Cameras Separation and the Zero Parallax Distance. Cameras Separation determines the distance between our left and right cameras. Since the left and right cameras represent our eyes, you’ll likely want this to remain at the default setting of 6 (2.5 inches).

That said, the Cameras Separation setting can be used to manipulate the depth of a 3D scene. The closer these two cameras are together, the flatter an object will look – the further apart the cameras are, your objects will appear to have more volume.

The Zero Parallax Distance is where the cameras converge. Objects here don’t have any sense of depth; objects in front of the Zero Parallax Distance will feel closer to you and appear to have a nice sense of depth, while objects after the Zero Parallax Distance will feel further away. In using a stereo camera rig, we’re adding a Z-axis to what’s essentially flat imagery. Objects either side (in-front or behind) the Zero Parallax Distance are travelling along this axis.

Integrating 3D elements from Maya

We shot a comedy short about two hipster types chatting about how cool their night was. The dialogue was the main driver of the piece for me, the two characters used many phrases and words that really lent themselves well to being represented as motion graphics – that being the case I thought that it would be a good piece to use to experiment with 3D geometry.

Here is my workflow, I found my own way through trial and error – it’s a workflow that should be iterated and improved upon but good enough to get the results I need until a better way reveals itself.

  • Take a still from the footage of the section you want to use for your VFX / graphic (remember to take one for each eye). This will serve as a reference.
  • Create a scene in Maya with your LatLong Stereo camera – match the scene frames per second to that of your footage.
  • Import the 360° still for each eye into Maya and apply each one to a sphere. It’s a good idea to have these spheres on different layers you can turn on and off. Use one of the stills for the Mental Ray IBL information
  • Use some simple geometry to place into the scene so that you can get your positioning and depth right (it’s a good idea to know in advance the kind of VFX you want – you should take measurements of your environment; things like height of the camera rig, distance to the sofa, distance to the ceiling etc. Integrating these measurements with Maya’s units of measurement will help you to place the objects at the right depth in your Maya scene and gives you more of a lead as to how something will look).
  • Do a test render of one still from each eye to check how it looks through the headset – remember to tweak the settings in the camera as described above if you need to
  • Once you get the positioning right, start to build your geometry and animate if needed.
  • Render out and sort the left eye and right eye renders into their own folders.

Once the renders are done it’s time to take them into After Effects (or whatever your preference would be). In the picture below you can see that I’ve split the footage into left-eye / right-eye. When you come to import your animation renders, remember to apply the same motion / transform / scale attributes to them as you have your footage – this means they’ll match up. Here you can carry out some colour corrections or shadow effects – there’s a great plugin for After Effects here to carry out post-production work with 360° equirectangular footage.

Stereo pairs of the live plate and VFX ©Retinize

Creating VFX and motion graphics for 3D 360°  video provides new opportunities for artists to connect with an audience – due to the immersive nature of VR, these graphics really get to leave their impact on an audience in a way that no other medium allows them to.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s