SpookVerb

Posted on 04/04/2017 Under Blog

Using the FB360 Spatial Workstation as a frontend for IRCAM’s SPAT to create 3rd order Ambisonic audio for Cinematic VR

SpookVerb is a Max patch that utilises IRCAM’s SPAT software suite for Max as a realtime multichannel panning and reverberation engine. It has 16 mono source inputs (input 1-16), 4 reverberation engines, and outputs a 3rd order Ambisonics audio stream in AmbiX format (output 1-16). It also passes 6 channels of audio unaltered (inputs and outputs 17-22).

A basic user interface lets you monitor levels and DSP settings. You can open the Spat.Oper window to set Source and Reverb parameters and view the placement of the spatialised audio sources. Settings in the Spat.Oper window can be saved as presets by [shift+] clicking one of the four preset buttons and recalled by clicking them. These four presets can be saved and loaded as files with the ‘Save Preset’ and ‘load Preset’ buttons. There is a View section where you can select to view all sources or only sources that are actually sounding. The ‘View Hold’ and ‘View Threshold’ parameters are to optimise this view.

SpookVerb can be used as a ‘stand alone’ reverb unit, or in combination with the FB360 Spatial Workstation and the Reaper Template provided. When used in combination with Reaper and the FB360 Spatial Workstation template, the 16 mono input sources receive audio from tracks 1 to 16 in Reaper, and the Azimuth, Elevation, Distance and Aperture parameters of sources in SPAT can be controlled by the Azimuth, Elevation, Distance and Spread settings of the FB360 Spatialiser plugin on their respective tracks in Reaper. The reverb routing of the sources can be controlled by the SpookVerb JSFX plugin on those tracks in Reaper.

Since SpookVerbVR is build in Max and the original patch is provided, you can alter the input and output channels and format, reverb engines, etc. This Setup is very basic but useful. You can make it your own: think of monitoring with custom HRTF’s or speaker setups, video overlay on the interface with jitter or extended OSC control with ToscA. Load your own logo in the top part of the patch to impress.

Software

Configuring Audio Routing

  • Setup JackRouter with 22 Virtual Input Channels and 24 virtual output channels.
  • Launch JackPilot and click Start. Then click Routing and notice that Max and Reaper should be available as Send and Receive Ports. If not, (re-)launch the respective applications and select JackRouter as their audio device for input and output: In SpookVerb: Click “under the hood” and select “Core Audio” as audio driver and JackRouter as input and output. In Reaper: Go to Audio Device Settings and select “JackRouter” as Audio Device
  • The first time you will have to set up routing manually but you can save this routing as a “Studio Setup” in the File Menu of JackPilot:
  • REAPER Send Ports 1-22 to MAX Receive Ports 1-22 (for the 16 mono sound sources and the 6 feedthrough channels)
  • REAPER Send Ports 23-24 to SYSTEM Receive Ports 1-2 (for Monitoring)
  • MAX Send Ports 1-22 to REAPER Receive Ports 1-22 (for the 3rd order ambisonics signal and the 6 feedthrough channels)

Audio Channels

REAPER TO MAX

Ch 1 – 16 = Source channels 1 to 16

Ch 17 – 20 = (1st order Ambisonic) feedthrough channels

Ch 21 – 22 = (stereo) feedthrough channels

 

MAX TO REAPER

Ch 1 – 16 = 3rd order Ambisonics channels (AmbiX format)

Ch 17 – 20 = (1st order Ambisonic) feedthrough channels

Ch 21 – 22 = (stereo) feedthrough channels

Setting up OSC communication in Reaper

  • Copy “SpookVerb_v03.ReaperOSC” to the OSC configuration file directory of Reaper […/Users/“your name”/Library/Application Support/REAPER/OSC]
  • Create a new OSC control surface: Go to Reaper/Preferences/“Control/OSC/Web”. Click “Add”, select “OSC”, name it SpookVerb, Select the “SpookVerb_v03” Pattern Config file and Send to port 7000 on the local host (Device IP=127.0.0.1)

Plugin Installation

  • Install the FB360 Spatial Workstation
  • Copy “SpookVerbVR_JSFX” to the Effects directory of Reaper […/Users/”your name”/Library/Application Support/REAPER/Effects]

The SpookVerb Reaper Template

SOURCE TRACKS

  • Tracks 1 – 16 are tracks for MONO audio sources that can be spatialised. These first 16 tracks are fixed and their first (mono) output need to be routed to channels 1 to 16 respectively. They always need to be track 1 to 16 in your Reaper project. Do not move them around! The first two plugins on these tracks are fixed. Plugin 1 should always be the spatial workstation and Plugin 2 should always be the SpookVerb JSFX. You can insert as many plugins after thats as needed for mono signal processing of the sound sources. These tracks are for MONO audio playback only.
  • The orange track called [1stO AmbiX] is a 4 channel AmbiX feedthrough channel trough Max and will come back in your Reaper project unchanged. The 4 channels of this track need to be routed to channels 17 – 20. You can duplicate this track as many times as needed.
  • The orange track called [STEREO] is a 2 channel stereo feedthrough channel through Max and will come back in your Reaper project unchanged. The 2 channels of this track need to be routed to channels 21 – 22. You can duplicate this track as many times as needed.
  • Video Reference track: a handy track to keep a reference of the 360 video in your project for exact timing reference.

MASTER SECTION

  • MASTER 3rdO receives a 3rd order AmbiX audio feed from MAX 3rdO and a 1st order AmbiX feed from MAX AmbiX thru. The output is routed to FB360_Spatial for monitoring and export in the FB360 format.
  • MAX 3rdO receives a 3rd order AmbiX audio stream from Max. The input is set to channels 1 – 16 and the output is routed to MASTER 3rdO.
  • MAX AmbiX thru receives the 4 channel ambisonics feedthrough from Max. The input is set to channels 17 – 20 and the output is routed to MASTER 3rdO.
  • MASTER st receives a stereo audio feed from MAX ST thru. The output is routed to FB360_Stereo for monitoring and export.
  • MAX ST thru receives the 2 channel stereo feedthrough from Max. The input is set to channels 21 – 22 and the output is routed to MASTER st.

MONITOR

  • The Purple Tracks contain some control and metering plugins from the FB360 spatial workstation.
  • Track HP_FB360 receives audio from the FB360_Spatial and and FB360_HLST tracks and is routed to channel 23-24 that feed the system output.

Setting up the FB360 Spatial Workstation Spatialiser Plugin and SpookVerb Plugin

  • The FB360 spatialiser plugin should always be inserted as the first plugin on tracks 1 to 16. It should be setup so that the audio in its first input channel will exit the plugin unaltered. This will be the case when Attenuation, Doppler and Room effects are turned off. Now this plugin can be used to send out OSC messages of its Azimuth, Elevation, Distance and Spread values without altering the (mono) audio that passes through. The Azimuth, Elevation, Distance and Spread paramaters control the Azimuth, Elevation, Distance and Aperture of the linked audio source in SPAT and can of course be automated.
  • The SpookVerb JSFX plugin should always be inserted as the second plugin on tracks 1 to 16. With this plugin you can automate to which reverb this track will be send

Exporting

Since the SpookVerb patch is not a plugin, exporting needs to be done in realtime: record-arm the tracks you would like to export and set the desired record source of the tracks. In most cases Record: output (multichannel, latency compensated) will do the trick. Make a time selection of the part you would like to export, now your ‘Video Reference’ track can come in handy. Then under [options] select Record mode: time selection auto punch and press record. Sit back and enjoy your mix.

Tips and Tricks

  • [Ctrl+] clicking in the Spat.oper window can show you a front view of the sound sources (to see the elevation)
  • You can zoom in and out in the Spat.oper window by scrolling up and down.
  • Remember that the Spatialiser and SpookVerb plugin should always be the first two plugin on the 16 spatial audio tracks in Reaper. Afterwards you can insert any processing plugin that you’d like. This processing of the audio will take place before the signal is being sent to Max.
  • If you prefer to control all the audio levels in Reaper in stead of using the level drop by distance, set the drop of the audio sources to 0dB in the Spat.oper window. Remember to make a preset of this if you’d like to use this setting more often.

Known Bugs / Limitations

  • Tracks 1 to 16 in Reaper cannot be moved around. Unfortunately Reaper does not work with fixed assignable track numbers, so when you move these tracks, the channel numbers in the OSC messages also change.
  • Using too much automation can cause data overflow and OSC massages will be delayed or dropped. Optimising OSC packed size and buffer can improve performance.
  • Maybe because of a defect in the dump output data of Spat, the ‘Room Size’ parameter of the reverb engines in spat is currently not saved in the presets. Hopefully a future update of Spat will resolve this issue. Or it could be a mistake in the patch.
  • Preset files are Spat version specific, so beware of this when updating Spat.