Scriptable Video Module

Thanks for opening this discussion! Mainly I want to be a spectator at this point, and just get my finger on the pulse of what you all want or are imagining.

My thought is more of a devboard (like Raspberry Pi or Arduino), but with perhaps a module version, of what is currently the Memory Palace/TBC2 platform. There are several devboards out there with video in and out, but it is pretty much always HDMI only and the boards with more than a single input are few. What I think we could offer is the devboard with at least two external video input channels and an output that runs synchronous to the inputs. This means you have proper analog genlocked I/O, and the device can then be modular and expandable. We would put into place all the basic infrastructure in the FPGA for time base correction/frame sync, frame buffering, and all the drivers. So it would effectively be a user scriptable video mixer/effects processor. Add MIDI controller and code to make pretty much whatever dedicated machine you wanted.


TBH its fine ifs its a standalone, that means the board can be bigger if needed by design, and I can save some hp on my briefcase, and bring the board to gigs if I want.

The primary things I’m interested in knowing right now are: What are the use cases for you? What problems does this solve? What can you do with this that you can’t do with existing gear? How far is your imagination taking you when considering a scriptable environment?

We haven’t made the decision yet, but we may also open source the software portion of the Orion modules (the VHDL/FPGA portion would be fixed.) This would make each module in the series a reprogrammable video DSP focused on whatever video processing is going on in the FPGA core (in the case of Memory Palace, for example, there is a hardware accelerated texture-mapping unit, colorspace converter, chroma/luma keyer and alpha compositor.)

All of this stuff requires, at the least, clean code and documentation on our part. So we need to get through our first release cycles and let the firmware mature first. After that though, we may have a really great platform to springboard off of.

Check out the Pynq board… it’s a Python scriptable environment for Zynq (the architecture we’re using.) One option Ed’s brought up is to fork the Pynq platform and then implement device drivers for it for our various FPGA/DSP cores. That would provide a good basis for further extension.

Another selling point of a devboard is that it could be used as an OEM base for other people’s video synth/video mixer products – the same way we use the Raspberry Pi for our Andor 1 media player.


I’m reading on it and I’m already in love with the idea, seems like a very powerfull and dev-friendly platform.

The idea to embedd a jupyter notebook on the A9 cortex (linux) its very amazing.

Regarding what are the use cases I can think of:

  1. Machine Vision (Artificial Vision and Machine Learning) for tracking objects from the input video, and provide that information as analog signals to the modular, or to stick an tracking point or object into that position in the video, in real time!.

  2. Video Effects, to write filters like in ffmpeg (like this or this videos I made) or like, but again like in real time.

  3. Complex/Dynamic Key Generators, for video mappings but in real time?

So some of this can be done on computers but in batch process (non-real time). Most of this falls in the research areas of Computer Vision / Computer Science and its not very artsy. Or maybe it is! :slight_smile:

Here are some examples of video effects that are scripted thru filters in ffmpeg its very interesting


I work a lot with Touch Designer, Quartz Composer and Isadora so my dream module would be something that includes a visual programming language. That would make it much more accessible to people form a non programming background.

1 Like


My point of view
What i like about eurorack is their simplicity / ergonomy : one module / one function and patch cable.
The best scriptable module i know is a laptop. If i miss a module it s something like Expert sleepers ES8 ( cv《》usb) with video rate frequency ability, to interface lzx with say pure data or jitter…

1 Like

I agree with you partially, but there are stuff that its too complex to write for a GPU or it will work too slow on a CPU, so having a FPGA on board is a completely different thing. Because its basically reprogramable hardware, so you can change the hardware (make the physical logial gates change) based on your source code (known as VHL), this is super powerfull for highly concurrent and real time applications that, as I said before, can be solved in a Laptop but in a batch process, not in real time.

1 Like

right. i inderstand.
But what i m doing with jitter is enough for me , real time and an interfacing module would be great

as for me, I’d like a video effects unit that can do the things a Korg Entrancer can do, but then in full PAL format.

It would be great to be able to edit weird digital functions and then use them on analog video.
and automation come to mind.

(I’m an HPC / Scientific R&D person IRL.)

I’m happy to do anything up to and including writing code in Vivado (but I suppose that depends on the licensing…). I don’t care so much for visual programming for something like this, personally, but I understand the appeal as it is a natural analogue for patching. I would worry about the cycles eaten up by a framework that made things easier. I enjoy hybrid programming models - and I think the Zynq type parts are really exciting in that way.

The part of a video processing module that I would find the most daunting (and thus the greatest value add) is all of the integration and testing of the peripherals (jacks and thus ADCs, encoders, display, etc.) that are connected to the SoM.


That’s definitely the value add from our perspective too. There aren’t any existing Zynq or FPGA+MCU boards out there that provide extensive interfacing to analog video encoder/decoder peripherals, so that makes it hard to tinker.

While a visual programming language would be awesome, and it may even be possible to port existing ones to run on this board, we’re thinking more at the primitive level of just the platform with core drivers – so any visual programming language would probably be another layer on top of that, and on the user application side. Maybe there’s an OEM/devboard that’s more barebones and a visual scripting/reprogrammable interface module is something else in the Orion Series lineup. All the “Orion Class III” modules (currently TBC2 and Memory Palace) use the core architecture of Zynq SoM + loads of video related hardware peripherals.


I’d be pretty excited about any kind of devboard version of the MP/TBC2!

Pretty agnostic on language but I’d hope there would be both a high level and a low level way to engage with it. I have a C&G ETC and while I think they did a great job with that platform, PyGame doesn’t cover everything I want to do with a programmable system. I’d be interested in doing more demoscene-style low-level programming, ideally with a mix of CPU and GPU, 3D geometry, etc. I’ve also been daydreaming lately about a FPGA RNN/GAN implementation, so you could train models on a PC and then apply them with low latency in hardware.

My #1 request, though, would probably be a video module with a long sample buffer (whether an official Orion module or the devboard) that can participate in the monome i2c serial messaging system (teletype/ansible/mannequins/ER-301/Matrixarchate/etc.) I’d be most interested in some ways of scripting loop points on a sampler to recreate Raphael Montanez Ortiz style chopping & sliding buffer effects.


For me the most interesting point right now would be a Perlin/Simplex noise generator. It is an essential part of my digital work ( and would love to see how a staircase performs as an fbm. Lars asked me about this idea long ago and I’ll be happy to either take the challenge on developing it myself or helping others in the task.
A second point will be to gather in one module few of the effects present in devices like the roland entrancer. Flip-flop, slit-scan, pixelate, etc. In the direction of what Memory Palace would be able to do but in a smaller factor.


I’d like to see something that reacts to the video input and gives correlated CV/video outputs

so we could say get a static (if the picture is static) CV amount correlated to the amount of Green in the picture

building patches that react to the video output is something I’m interested in

I’m going to continue thinking of other things and re read this thread but I really enjoy the idea of programming some video synth tools


@wednesdayayay Great thoughts – I have a lot of ideas like this for image analysis across multiple frames (longer temporal constant.) Another constant CV value it would be nice to have is angle of movement, amount of motion across the frame, etc.


this in real-time :heart_eyes::heart_eyes:

a system like the ffmpeg filters that can be cascaded/chained/routed would be awesome


angle of movement sounds rad too

this is maybe a little off topic maybe not

paint by numbers

I was recently painting a backdrop for our production of winnie the pooh
the way we typically do this is project onto the flats trace then some shading then paint over it in whatever the desired style is

what I would like to do simply is
chroma key out several major colors (maybe just RGB since we tend to only buy RG&B plaint and mix ourselves anyway)
under each color would be an oscillator (static image is just fine)
maybe Red would be full horizontal Green would be between horizontal and vertical and Blue would be vertical

this seems doable in a system with 3 oscillators, 3 polar fringes, 3 doorways and a visual cortex

is this too ridiculous of a use case?

scrolling bang

with the additional scrolling options that diver and memory palace open up perhaps it would be interesting to derive triggers/gates/bangs/go messages for scripting from a settable threshold for NESW faces around the outside of the viewing screen and allowing these triggers etc. to be assigned wherever
this could really open up some self playing/wiping

picture in picture

picture in picture being a bit easier would be very nice but would require more than one RGB path on the module, multiple modules or a 3 channel grayscale PIP workflow which would still be fun and elevating I feel.

thinking back i seem to remember the sampler module being talked about as a video collage module so maybe some similar territory is covered

color pick and switch

something like the polar fringe chroma keying where in addition to picking the color you want to pluck out or around you also pick the color that goes in there instead
“typing this out loud” made me realize that polar fringe and color chords would cover this but it would be neat to have it as a simple programmable & reusable module so we can that could be stacked for a larger re-colorization patch


perhaps even something where you could assign quadrants of the screen ( I seem to remember someone mentioning this a while back and I’ve got some of my own experiments going that I haven’t written about yet in this territory) tasks
if we could write conditional statements based on the location (within the quadrant) of a color blob meeting a specific requirement for size and color that would be pretty neat
automatic and or guided drawing could be neat the more I look into this application the more I see spaces where memory palace should be able to function similarly I think
but having it be scriptable rather than more about the physical patching could be very interesting
I’m going to dive into this application and get some more concrete suggestions

piggy backing on @luix post above from that same website

some kind of CRTifying would be great

I believe something that allows for ASCIIing live video with a live keyboard input for swapping in and out letters (like muting and and unmuting that “letter channel”) would take the idea further into some new territory

I think there was an old-school Quicktime demo that did this; source code and all. I’d guess from the 2004 or so era, but maybe earlier.

1 Like

so I really enjoy watching the processing tutorial video from here

and he just recently released a video about a new application called runway that I think anyone interested in this machine learning area may want to take a look at

that being said it is helping me focus what I want to see from the video synth

spade coco

so one of the models (spade coco) allows for you do import or draw with certain colors that have been trained for so dark blue might be ocean and green might be grass
you draw (using a built in basic drawing app) or import an image and then it is translated into a picture you can see what I drew and the result
the idea that we could potentially have something drawing images soley based on the colors present in our LZX patches opens up some very interesting possibilities




I think the skeleton / face tracking would be very useful too if we could key either the resulting XY space from face detection input or the skeleton out of the input so that it could be composited in somewhere else

it would be a lot of fun to take the resulting “skeleton key” through doorway and get it nice and soft then mix that back in with the original through a visual cortex/marble index

spade landscapes

so there is another model (spade landscapes) which takes text input and outputs a picture

“the sun is on fire a bucket of fish and ten little boys”
obviously if you make an effort to work within the confines described models then you will get a more “pleasing” output
I’m not sure how to make this work with LZX just yet…
although there is another model that takes an input image and outputs text…
we could create the google translate of video synths
create an image to be described textually
then create an image from that text
rinse repeat
so I just tried the im2txt model on that same image I generated above
and it gave me
“a fire hydrant that is on a sidewalk .”
sending it back through the original model

then I got
a red fire hydrant sitting on the side of a road .

then it got stuck in a loop
so the tasty aspect here would be keying in video synth stuff with the images thus throwing things off a bit so that we don’t end up getting stuck in a loop :slight_smile:

dense depth

there is a great model (dense depth) which predictably gives a grayscale output that represents the depth of the input
I can only image patching in depth in this way combined with a dual cortex/projector setup
red/blue the outputs of cortex 1/2 slightly offset (projectors) in the X axis and we should be able to make some 3d video synth…


there is one called Photosketch

this is a wonderful effect

I have only played around in runway for an hour or so and it has been great fun

1 Like