Let's Discuss Visual Storytelling + Video Synthesis: Patching Event Based Narratives

This thread has been a joy to read, thank you everyone, such a diverse range of ideas.

My route into video synthesis was during my second year of university in 2015 (electronic engineering with music technology), Paracosm’s Lumen provided a more affordable path into the field. I used Lumen to do visuals for a regular techno party I was a part of, sequencing additional parameters and audio reactivity in Pure Data. In terms of narrative I felt the XY pad in Lumen being able to interpolate between 4 different parameter presets was really powerful, it allowed me to navigate between known points, islands of stability, into more chaotic behaviour and back out to familiar scenes.

At some (most!) points of the night I’d want to dance rather than knob twiddle so I also programmed some rudimentary random control patches in PD. I started to explore the idea of using stochastic matrices to create a framework for probability based control but ironically this got side lined by building my Cadet synth. I’d used them in my first year to create a C program that could ‘improvise’ jazz, which given my ropey understanding of both programming and jazz at the time, went surprisingly well. I figured if it worked for music it would work for video. I love the idea of a tool that could be seeded with preset scenes/control parameters and then move between them using an algorithm which could also be influenced in real time by the user.

In terms of what a hardware tool would look like, @Rik_bS is pretty much on the money

I dream of a “macro controller” and event sequencer, that many signals can be routed through and manipulated, with set “scenes” that can be programmed then either switched through or played.

The closest example I can draw from is lighting controllers - where the scenes are programmed ahead of time and played live with the music or sequenced as part of show automation.

Maybe something that can add in some of functionality of Sensory Translator, Escher Sketch, a VCA matrix with some additional computational controls.

7 Likes

old school lighting boards were a blast to operate/perform

nichole and I spent many years up the spiral staircase of an old art building running shows from a tiny black box


this kind of vibe

I remember getting a new set of young techies and playing the spanish version of troll 2 over the headset at one point during a rehearsal

6 Likes

“Scene level control” is a must. But also tricky in a patchable context. We can either treat each individual connection as a node in the scene (in which case you end up needing a crossfader with a number of channels equal to the number of connections to morph between settings). But this quickly leads us back to the superiority of a digital system in this regard, where the cost of adding another parameter group is negligible.

So that’s a puzzle. I love giant modular systems, but I don’t think you should need a giant modular system to introduce higher order sequencing and scene control. Whatever we come up with for animation needs to be at least as accessible as the other components of the system. A powerful scene transition should be possible with only a couple of modules.

Pushing all of this into a purely digital control environment could potentially diminish the attractiveness of using a real time analog computing environment with a universal voltage standard for creating images, and get us thinking more about the results (frames), than about the individual components of shape and color which compose them (that is, to me, video synthesis!) There’s a place for it all, of course.

But there are two ways we’re trying to combat this design challenge. One is the concept of “embedding a transformation into the image itself.” If we can embed the spatial and scaling context of a transformation into XY vector images (aka HV ramps), and use those as the source to generate a pattern (using a separate module!), then we are able to transfer an entire group of parameter settings (x, y, width, height, lens distortion, etc) using only two cables! Now it is much easier to do a scene transformation – since to change scenes we can just crossfade between HV source ramps representing two separate compositions of spatial coordinates.

The second way is to make every generator also a processor for other generators. For example, if one module is creating a “space” then that space itself can be patched into another module and further modified, in a hierarchy of vector modifiers. So now if you can crossfade between the frontend parameters for the “parent module” you end up creating a cascading effect through every derivative branch of a modulation tree.

So this is exciting stuff for me; but describes more the structure of a patched transformation space, and describes what some of our new modules (ART3 especially) are designed for, and explains the lineage of thinking behind modules like Navigator and Shapechanger. I think there is room for a “spatial generator/processor” that is digital in nature, allowing a large variety of user saved presets, and fluid interpolation across their settings. TBC2’s internal generators will be an interesting place to start experimenting with this.

To add movement to that, we haven’t gone very far with it yet. Navigator is a ramps rotator and Diver is a ramps positioner – but there’s nothing yet for “the scene”. Coordinated motion across many objects on a singular timeline.

But I think we approach it as a hierarchy transformation as well. We need a master transport control of some kind, to start – a “low frequency ramp source” with the voltage of that ramp representing percentage of timeline length. And then that single low frequency ramp/envelope is the timeline for any number of subramps or envelopes created through waveshaping (easing curves! ideally an analog bezier implementation) and subdivisions through wavefolding. These modifications of an overall “timeline ramp” cascade to any subdivisions. We can stretch or modify the time scale of the entire parented animation simply by changing the frequency of our “low frequency transport ramp.” If we want to change the easing curve of the entire animation, we just need to bend the waveshape of that envelope.

To state it a bit simpler: we start with one envelope generator. You can Play/Pause/Stop that envelope with buttons. You can scrub it. You can change the overall speed of it and FM that speed. It shows you a frame count. It feels like a sequencer. It’s output though, is just an analogue linear ramp. It is a voltage constant for the timeline position itself. Next, perhaps we patch this ramp into a “3-act waveshaper.” This is like a very slow colorizer. Instead of color bands though, you have time segments as subdivisions of the transport ramp.

Perhaps “Act 1” is patched to slide a shape across the screen. “Act 2” is then patched to a module that makes the shape jump in scale, and “Act 3” fades the whole thing to black. Now we have a 3 part animation. The master transport ramp can start/stop/pause it, speed it up, slow it down, warp its timing, etc without touching the “3-act” controls at all.
How does it go from there? Another 3-act waveshaper. But instead of the transport ramp as it’s source, we use the Act 1 out from our first module. Now Act 1 has 3 more acts, and those subdivisions go to details of the animation. Maybe as the shape slides from left to right, it now wiggles up and down, then slows slightly, then bounces.

You get the idea.

So, I guess my current thoughts for an “animator” module lie in this area. “Clockless” timeline and subsegments through waveshaping techniques, inspired by the Scanimate. Perhaps an “event trigger” module designed more for singular events/envelopes that happen at time points (comparator thresholds) along the timeline. Needless to say, I think we’ll be doing at least 2 modules that fit together in this kind of workflow for Gen3 at some point. But this thread wasn’t really started to workshop product designs so much as just sit down and hear all your thoughts on the subject. It’ll be a minute before we can concentrate on these designs, so I want to listen and get the pulse of the community on the subject, so I have some time to think on it from different perspectives.

7 Likes

ahh damn, I’ll save my pitch for later…
(parks the idea of a digitally controlled 16 in, 16 out VCA/mixing matrix with a hands-on interface, internal modulation sources and per-event control of each parameter)

4 Likes

ahh damn, I’ll save my pitch for later…

Well I mean, I want to hear all of your ideas! Whether that’s theory concepts or concrete user interface imaginings. I just meant I don’t want our own R&D projects to dominate the discussion, there’s plenty of that in other threads. :slight_smile: I don’t have all this figured out yet. But I’ve been ruminating on all this for years.

I think there’s huge potential for the concept of the “Matrix VCA router” in general. The EAB Videolab had one:
image

6 Likes

So cool to see people thinking about lots of the stuff that occupies much of my brain, and in such thoughtful ways. Been savoring everyone’s posts.

I’m super interested in the intersection of music, video, and (for lack of a better term) videogames.

First, music stuff: The modular and outboard tools available in the audio world (or a good software environment with the right MIDI controllers) allows for a certain kind of abstract storytelling that I really enjoy. Being able to play with “character” via melody/harmony/percussion/etc and “space” via reverb/delay/etc and “movement” (modulation) — and then find emotional resonance in the way these things are tied together — feels like it somehow charts a path for what could be done with “narrative” in video synthesis. Creating and then destroying walls of sound + light. Playing with things like tension and release (peek-a-boo really hit home as I’m a nü-dad), chaos and order, perspective, speed, and scale in a timeline being creating is realtime is what I strive toward in my AV setups. Being able to go from slowly traversing seas of feedback to percussive physical drama (ie realtime Autechre Ganz Graf) and anywhere in between over the course of a set.

Game stuff: I feel like there’s so much to talk about here, but I dream of being able to synthesize and interact with characters and environments (both terms used verrrry loosely), using hands-on controls. Ming Mecca had some really cool ideas (or, more accessible, this software sequencer called New Path), and games (and game controllers) in general are of course really good at letting players explore and interact in physically/expressively responsive ways. Ways to generate simple AI-like attractors and repellers and build them into a system that feels like a garden/starfield/turbulent ocean sea/kids’ playground of cross-pollination and procedural interactions. Systems of objects and spaces that relate to each other in highly controllable ways.

This is going down yet another path, but I’ve been messing with some neural net text generators and they are getting crazy impressive. I only bring this up because it allows places and people and dialogue to be conjured from a very limited input, very quickly.

Anyway, I guess I’ll summarize by saying, WTB: synth but also Earthbound but also a delay pedal but also F-Zero but also a Mood Organ but also a Euclidean sequencer but also a Memory Palace? IDK.

4 Likes

Definitely. I’ve been obsessed since writing the Fortress firmware, with the idea of “self playing game AI” as a visualizer. Nothing complex, just very simple parameters; one agent draws a maze, one agent solves it – voltage control over the maze drawing algorithm’s complexity and the solving agent’s pathfinding algorithm. “Ant farm simulators”, is maybe a good way to put this. Just the idea of the actor/agent as a visual entity is something we can borrow from games directly when composing patches.

More event based states will help with this. For example, you can patch the collision between two shapes (like the pong paddle and ball) with AND logic and keyers on a video synthesizer, but to change the ball’s direction requires a flip flop and a counter. Ming Mecca definitely had some amazing ideas on how to make all of this patchable.

7 Likes

Random Poetry Generator would be nice.

3 Likes

@creatorlars , thank you for starting this conversation. Most of my video synthesis work involves silent, long captures of animated images generated from my video synth. Here is my current creative conundrum with regards to my desired audio-visual workflow.

I discovered video synthesis though music. As is such, my creative process is built around songwriting. Typically, when I record audio, I will start with what is essentially a loop of drums and sequenced audio synths and record a take. From there, I’ll edit out the drums to map out a structure. After that, I layer on some overdubs. Then the recording is ready for mixdown. This is a pretty standard process.

I would love to be able to mirror this process with my video synthesizer whereas, I would be able to make a video capture for the foundation, and then capture a different patch that is reactive to each successive overdub. Once I have recorded and captured all of the audio-visual elements of the song, I would like to mix those captures together. I feel that with a TBC2, or more, I might be able to partially achieve. The piece of hardware that I feel is missing is a sync-able video pool or recorder that can capture and play back images in sync with my Audio DAW (with a basic cv or midi sync signal).

I’m not sure how this could be implemented. What are your thoughts?

1 Like

Yes! A sampler module/instrument is an inevitability – you’re right, we’re trying to build that off the TBC2 architecture and then implement something I imagine similar to a “video 4 track recorder.” Something with a 2X2 interface capable of inserts and overdubs through analog key input. I imagine stop-frame animation style transport control could be useful here as well as looping / drum machine style overdubs.

The next step for us in this particular direction will be with integrating hardware media decoding/encoding using the existing hardware platforms (TBC2/Memory Palace) as development environments.

But I’m curious about what any of you have put together with existing video gear. There are some video samplers out there, like the P10, and video software environments capable of this kind of workflow (most are not oriented to live streaming though.)

7 Likes

Love hearing about all of your amazing ideas, Lars.

I understand your motive for keeping things in the analog domain. You have pushed and continue to push the envelopes of the possible in the analog domain. However, I hope you will continue your development of digital tools. I think you know I want an HD version of Memory Palace. But I also want a miniature DVE.

4 Likes

Oh, I want to do it all, of course. If things pick up it would be nice to hire a full time software developer so we can make more progress. My main concern is with supporting new workflows or techniques that aren’t already possible with digital environments. But… I feel like most any hardware tool outside the industry boxes of “video mixers and converters” falls in that category, whether analogue or digital.

In the context of this discussion, a modular analogue animation system is something quite different from a digital event programmer / integrated digital processing / parameterization environment. They’re powerful in different ways. The digital animation stuff (parameter grids, modulation programs, etc) I see mostly as a modulation/animation subsystem for existing digital processors (like Memory Palace) rather than a separate module. For separate modules I naturally want to think in modular blocks – meaning not just a giant animation module/workstation, but building block modules that can support a small system, but also expand in a modular way to a massive system (level complexity / number of channels is user controllable, in other words.)

6 Likes

Yes, I’m with you completely. No need re-inventing the wheel. If DVEs already exist, there’s not much pressure to create one for Eurorack. However, I/O is still a problem. Without a DAC/ADC module that supports DVI/SDI, our options for integrating outboard gear are pretty limited. There’s no way to insert outboard gear into the LXZ signal path. I mean, I guess one could have a whole battery of TBC2s and ESG3s, but that seems like overkill, and an expensive solution.

But what would really open things up is a very simple DVE channel in the existing LZX standard. 1V RGB I/O, six degrees of 3D Cartesian freedom, CV control over those six parameters. A joystick and/or a few buttons/knobs, but no sequencing. That can be handled by external animation modules. MIDI would be nice, but not essential. Keep the menu-diving to an absolute minimum.

It may be entirely possible for you to do this all in the analog domain. Maybe you’ve already done it with Chromagnon, or some other module. Information is scarce, so I don’t really know its full capabilities. But the key for me is the six degrees of freedom in a Cartesian volume – or at least the illusion of such. Similarly to the HSL dilemma, I want to use my existing, deeply ingrained conceptualization of 3D space. I’m not at all opposed to re-wiring my brain to do things differently, but that would take time away from actually making art. And I’ve only got a few decades left before I’m too old to do any of this anymore.

6 Likes

I am so glad that I found this thread! It’s very interesting to read the many thoughts posted. So many different perspectives and ideas.

I’m planning on starting an LZX Gen3 system asap, I don’t currently have any video synth modules, but have a 12U eurorack. I want to be able to merge the music with video, and the demo videos have really drawn me in.

My main goal is to have fun being creative! The initial plan is to use Chromagnon and Keyer to merge recorded videos with interactive ‘knob-wiggling’. The possibilities seem endless. I’m interested in blending abstract video with recorded fun stuff to see where it leads me.

I enjoy computer graphics, have worked on-and-off in this area since 1980, when we had memory mapped frame buffers and we wrote one pixel at a time. It took forever to record frame at a time to film and eventually a large simple VCR. Today I’m experimenting with a NVIDIA Jetson Nano to see what kind of abstract stuff I can create. The hardware can model and animate at crazy rates in real time! I see using these kinds of videos with Keyer both as foreground and key.

I’m super excited to be starting this phase of creative expression!

5 Likes

for some reason I’m picturing an old Umatic editing desk when I think of a timeline function.

with big jogwheels to set the begin and end positions and a start button, with segmentated LED displays for the voltage positions.

I loved working with those editors. the sounds they made and the frame to frame directness.

So if I translate this to a timeline editor, it could be a multisegmented envelope (such as the envelopes used in the DX7) with a way to jog through the points set in the editor.
Bezier curves would be great , so maybe OLED displays would be better to see these variables.
If you would have several channels of this, then I could see so much cool possibilities!

7 Likes

Do you have a Doepfer a-151? I read somewhere that this guy can’t really handle video rate signals, ie that if softens the video that passes though it. Thanks for any experience you can share!

@giantmecha - it’s not that bad - I put an example on instagram - check out the #tag #lzxveurorack

AMAZING ideas in this thread WOW I’m just blown away !!! This topic is EXACTLY up my alley so I’m obligated to make a post and revive the thread haha !

To introduce myself: im a longtime lurker / lzx wishlister with a background in CG animation and touchdesigner.

Ive been trying to wrap my head around how to realize this sort of setup for the last four or five years. In my eyes this sort of storytelling is what this scene is missing out on when compared to other visual art disciplines. But this is also a result of being abstract art !!! :relieved:

To TRY and answer the question though (from a more traditional animation and video mixing stand-point), I feel it comes down to:
-multiple stock video / png viewers with alpha
-blending modes
-deep branching trees of upstream / downstream keyers
-more ways to alter buffers ie: scrubbing, stretching time, seamless looping of some sort
-oscillographics / rutt etra / scanimate style animation to move in faux 3d space
-preferably an “FX send” per layer to effect pieces of the composition
-master controller for positioning “precomped” shape or source animations around the final frame
-preset saving and precomping to some degree

But as I’ve read, MANY people have pitched these ideas with better context to an lzx system ! These ideas are just to my personal taste.

I guess in theory this setup would consist of a TON of keyers and media players, layers of memory palace / fairlight cvi style processors, scan processors and oscillographic sources, and a master composition control of some sort to position the pieces in a frame for a final shot / scene switch / save presets etc.

I could imagine an artist designing an improvised scene like this live and possibly even having an improv actor acting alongside them captured with a camera and composited into the shot (or even tracked into mocap live and control a 3d character to composite in the shot)…hmmmm this is definitely a touchdesigner project…

I also felt the need to have this dream setup revolve around stock images / footage because this last year I REALLY started appreciating the possibilities in editing and mangling them into what u might need in the moment ! That being said, this setup is really more of a video collage editor than an analog video synth lol but thats just what I would personally want :joy:

8 Likes

Musing on this point-

Scale of industry-wise I can use Elektron MIDI p-locks / CV out, a Cirklon, or any other existing eurorack solutions like the Flux temporal sequencer.

Are there any interesting sequencing options with a video or visual design focus?

I’d be curious if there are any elements unique to video synthesis (and signal sync) that would necessitate a different mindset, especially if these sequencers would be less stuck in a quantized time domain or on “the grid”.

4 Likes

That really nails the open question for me, too. I think we should experiment with every possible workflow to gain more data about that. On the visual side of things, we can look at differences between realtime music production / sequencing vs edit decision lists & animator’s worksheets, as a thought exercise. What’s missing in one but present in the other? What elements do they share? Can the ideas be combined in ways that make new kinds of sense?

3 Likes