One of the most powerful, exciting aspects of modular video synthesis is the freedom to connect audio and video signals together in a fluid, direct manner.
In a modular system, signals are patched from outputs to inputs. The inputs and outputs aren’t defined by a file type or signal protocol like a software environments — the standard is universal: voltage fluctuating over time. This is true whether or not the signal is a square wave from an oscillator, a control voltage from a sequencer, or the image from a video camera.
We are often asked how existing audio synthesizers and modules can be used as part of a video synthesis system. The key is an understanding of the frequency (or speed) of the signals in play. The metaphor we like most is that audio is 2D and video is 3D.
The 2 Frequency Domains of Audio Synthesis
With audio and sound, things come in twos. You have two ears, a stereo recording has two channels, and in audio synthesis we are concerned with two frequency domains. Subsonic frequencies are below the limits of human hearing, and signals in this frequency range animate and change sound across time. Frequencies within the range of human hearing are in the audio domain and create audible sound.
In an audio synthesizer, signals from the subsonic frequency range are used to modify signals in the audio frequency range to synthesize specific timbres or amplitude shapes and to compose music through sequencing.
Subsonic
Signals below 20 Hz are in the subsonic frequency domain. Example subsonic frequency generators include LFOs, envelope generators, sequencers, and random voltage generators.
Audio
Signals between 20 Hz and 20 KHz are in the audio frequency domain. Example audio frequency generators include VCOs and samplers.
The 3 Frequency Domains of Video Synthesis
With video and images, things come in threes. You have three types of photoreceptor cells in your retinas, RGB image processing happens in three channels, and in video synthesis we are concerned with three frequency domains.
In a video synthesizer, signals from the vertical frequency range (~30 Hz to ~15KHz) are used to modify signals in the horizontal frequency range (~15 KHz to ~6 MHz) to create patterns and shapes. Signals in the animation frequency range (less than the video frame rate, ~30 Hz) modify signals in the upper two frequency ranges to create motion. In order to create a a two dimensional pattern or shape on the screen, you need both a Horizontal signal generator and a Vertical signal generator in your patch. To put this shape into motion, changing or moving across multiple frames of video, you also need an Animation signal generator.
If you’re perceptive, you’ve already made these observations:
The Subsonic domain of audio synthesis overlaps the Animation domain of video synthesis.
The Audio domain of audio synthesis overlaps the Vertical domain of video synthesis.
Signals generated by audio synthesizers just barely overlap the Horizontal domain of video synthesis.
What does this mean for you? If you have audio synthesis modules, you probably have the ability to generate 2 out of the 3 ingredients required for 2D pattern and shape generation already! The missing element is a signal generator capable of high frequency signals within the Horizontal domain.
Animation
Signals below frame rate (~30 Hz) are in the animation frequency domain. This domain overlaps the Subsonic domain of audio synthesis, so we can use the same subsonic signal generators like LFOs and envelope generators to create movement and motion in a video synthesizer.
Vertical
Signals between frame rate (~30 Hz) and the horizontal sync rate (~15KHz) are in the vertical frequency domain. This domain overlaps the Audio domain of audio synthesis, so we can use the same audio signal generators like VCOs and samplers to create the vertical elements of a pattern or shape in a video synthesizer.
Horizontal
Signals above the horizontal sync rate (~15 KHz) and video bandwidth (~6MHz) are in the horizontal frequency domain. Audio synthesizers don’t create signals in this domain, so we need a signal generator designed specifically for video synthesis, like the Prismatic Ray VCO.
Expanding Your Audio Synthesizer Into The Video Dimension
A modular synthesizer with both audio and video components is the ultimate audiovisualizer. In traditional audiovisualizers, control elements are extracted from music after its composition, but in a modular environment control elements and other signals can create sound and image simultaneously, in real time. The same envelope generator you’re using to taper the decay time of your synthesized kick drum can also skew the dimensions of a colorful sphere on the screen.
Our getting started page provides an introduction to our video module format. On our system examples page, we outline some starting options for users expanding an existing audio synth. As you expand, the important thing to remember is the rules of two and three we’ve outlined on this page. After adding a video synthesis core module to your system, focus your purchases at first on the video synthesizer modules that do what your system doesn’t do already: generate or process signals in the Horizontal range.