Conversion to LZX RGB format

So, let’s break down all the complex signal flow steps that introduce latency.

For the purpose of illustration, we have a PC with a mouse, video playback software, an SDI video output card, an SDI to analog converter, an analog to 1V LZX TBC (TBC2), an analog keyer module (FKG3), an analog output encoder module (ESG3), and a consumer grade LCD video monitor with Analog/YPbPr input.

The latency we are measuring is the time it takes between “click the Play button” and “LCD monitor starts to show the frame.”

Each of the following steps introduces latency:

  1. The USB peripheral drivers poll the mouse and register a click.
  2. The software registers the click event and process it.
  3. The software starts buffering a video file into memory until it reaches a threshold at which it decides is safe to start streaming without hiccups.
  4. The video output card shows the first frame of video.
  5. The SDI to Analog converter processes the first frame of video and converts it into an analog signal.
  6. The TBC2 module captures the frame to memory and then waits for the next “house sync” vertical refresh pulse.
  7. The TBC2 module streams the frame to it’s 1V RGB outputs. Now we are inside the LZX 1V patch.
  8. The FKG3 keyer module processes the analog signal, transitioning for example, between the image and a synthesized pattern.
  9. The ESG3 analog encoder converts LZX 1V RGB back into viewable analog video signal.
  10. The analog video signal is captured by the LCD monitor and converted into a digital bitstream, and written to the monitor’s internal frame store.
  11. The LCD monitor refreshes, and you see the result.

All 11 steps introduce latency. Now, we can start estimating and measuring in context – the latency between two different steps, for example.

Steps #7 thru #9 are measured in nanoseconds. In this case, the latency is maybe 100ns? No more than a pixel or two. Each analog op-amp in the signal path adds a latency of around 10ns.

For the sake of proportional context, 100ns is about 10000 times faster than even Step #1 (registering a mouse click at 1ms polling speeds.)

And for the sake of argument, steps #1 thru #6 and #10 thru #11 have no effect on the output image you will see. There’s no difference if this was a latency of 1 frame or 100 frames, nothing is changing besides the time it takes for you to see it.

So in this example case, the only significant latency factor is the user interface element – UI response time between click and display. Whether that is relevant and how much it effects your workflow is inherently dependent on your process itself. For example, in a pre-rendered mixdown scenario for recapture, it doesn’t really matter at all. In a performance context where you are clicking the mouse in time with a live audio signal, it matters more. As mentioned above, it is the display or projector that is most likely to introduce an annoying amount of latency in context.

Things only get more complicated or start to effect the resulting image you will see when we are discussing recursion/feedback. And in that case, “low latency = always good” is not true at all. For example, you want at least one frame of latency in order to build up recursive images that stack on top of each other – if it’s less than that, you get more of just an optical smear – also good, but a different technique. Even delaying the recursion (variable programmed latency – frame delay memory) is an important technique.

The recursion latency and total latency are two different things entirely. Both are relative to different steps in the flow. For example, if we patch the FKG3 output back to one of it’s inputs, our total latency (click to display) remains constant. But the recursion latency of the FKG3 feedback path is much much faster (20ns-30ns ish.) You’d likely want to get a low pass filter into that feedback loop (i.e., intentionally increase the latency) or take the feedback path thru an additional TBC or frame delay to have that make a range of interesting pattern modulations.

For the sake of further discussion, let’s define some more terms:

  • Frontend software latency. Steps #1 thru #3. Anywhere between a few milliseconds to a few seconds. Want it to go faster? Buy a fancy gamer mouse, and optimize your PC (faster RAM, faster hard driver, better software.)

  • Frontend video latency. This is the latency between steps #4 and #5. A frame or two. Maybe more. A device that has analog output (rather than going SDI to Analog converter in step #5) might reduce it a little, but probably not by much. Once you’re streaming SDI, the digital devices can communicate with very low latency, and broadcast converters are generally very consistent. Cheap consumer grade converters from Amazon probably do more in software or have less memory or are MCU rather than FPGA based, so they will have more latency.

  • Input TBC latency. 1+ frames. In the case of TBC2, the frame latency is adjustable.

  • Analog processing latency. Almost instantaneous. Usually less than a microsecond even in worst case. If an object were to fly past your face at this speed, you’d never know it was there.

  • Backend / display latency. Could be anywhere from almost instantaneous (a CRT monitor) to several frames (a cheapo projector or a long video transmission line from the sound booth to front of house in a venue.) Want to decrease this, buy a display with faster refresh.

  • Software user interface latency The sum of all other latency increments. For example, the interval between clicking the mouse vs frame displayed on the monitor.

  • Analog/hardware user interface latency The sum of analog processing latency and backend/display latency. For example, the time between turning a knob on your video synth and seeing the result. The only latency here is going to come from the display you choose.

8 Likes

Part 2. Let’s talk about the OP suggestion – a direct analog input amp on your video synthesizer instead of an input TBC – relative to the example case described.

Here’s the conundrum.

  1. If you genlock your entire video synth to the incoming video stream, this saves you 1 frame of delay from the TBC. This does not make any difference at all on frontend software latency or frontend video latency or backend/display latency. So the reality is you are only going to reduce latency at a single step among many equivalent steps. All the other steps are going to still introduce latency. Disadvantages: Only one video signal can be input to your system at a time this way. And without a TBC, recursive processes between that video input and your video synth’s output will “Smear across” instead of “stack up in a feedback loop.” You are also limited by the stability of this single video source – if it’s not perfect, you have glitches that propagate throughout the whole system.

  2. The other case as it relates to the example would be to genlock the the video output itself, #4 so that it’s in time with your system. This requires a device that is capable of that in the first place. This doesn’t really do anything for you! You’re not escaping the use of a TBC – you’re just moving where it is in the signal path. The video output card is just going to delay itself to match the external timing – just like the TBC2 module would.

So the latency gains of an input preamp is incredibly contextual to what I would consider a marginal use case – and most of times even an undesirable use case, which has lots of “ifs” or “buts” about how you can use it. And even in the best case scenario – you’re reducing the entire signal path from a few frames down by one – not eliminating it entirely.

A list of things you can do to reduce latency in general

  • Optimize the PC that displays the initial video file.
  • Use broadcast grade converters and video interfaces, and not cheap consumer grade ones.
  • Use a broadcast grade display monitor or a fast refresh monitor designed for gamers or others who need a low latency monitoring solution. Or use a CRT.
  • Use a higher frame rate video format (60fps progressive vs 30fps progressive, for example)
  • Invent time travel and apply negative latency.

Key points…

  • Video processes will always involve some kind of latency, even analog systems
  • Low latency video systems only involve 3-6 frames of total latency. It’s fast enough to mash buttons to the beat in your video playback software, output and process with video synth, and display the results – and the brain doesn’t register something as too far “off time.”
  • High latency video systems would involve 7-60+ frames of latency and be annoying to use in a live context. This usually comes from a slow display/monitor/projector.
  • Recursion latency is not the same as total latency, and there are many ways to accomplish both ultra fast latency or delayed recursions within the same system – without effecting total latency.
  • Analog latency is not the same thing as total latency – and this is why we use analog systems for video synthesis! Each module only adds an incredibly tiny amount of delay – it’s effectively real time. Doesn’t matter how many modules you patch, the latency stackup is barely present. Whereas a string of digital video processors or mixers for example, are in most coases going to add at least a frame or two of delay at each step.
8 Likes

Thank you Lars for your very detailed rationalization of why it doesn’t matter that the TBC introduces ~30 ms latency. :stuck_out_tongue_winking_eye:

Anything we can do to reduce latency is a worthwhile enterprise for real-time performance. We can’t completely eliminate it, but we should do everything we can to minimize it. Just like science. We can’t possibly understand everything about the universe, but that is not a justification for making no attempt whatsoever.

1 Like

I should also say that I’ve been told I have a very “fast eye”. I definitely notice when audio and picture are out of sync by only one frame. Maybe other people aren’t as sensitive to it, but after 30+ years of creating visual music, it’s not possible for me to turn off that part of my brain. :grinning:

2 Likes

@dryodryo if you’re sure your eye can detect a single frame of latency then might be worth taking a look at what your reaction time actually is: Human Benchmark

I have here a link for you and a quote worth reading for your endeavors:

“They were better at detecting the audiovisual asynchrony if the sound preceded the video rather than if the video preceded the sound (131 vs. 258 ms thresholds for speech, and 75 vs. 188 ms thresholds for the hammer, respectively)”

4 Likes

I’ll also have to second this point here. @dryodryo one thing you’re forgetting as well is that most people who will watch your performances are not as versed in video as you. Nobody is going to watch your performance and think to themselves “This X frame delay makes it absolutely unbearable.” The context of a live performance is especially important, in that you are likely not playing to a room full of broadcast technology literate critics.

The exact workflow you’re talking about is something I’ve been working with for months. Midi triggering clips through LZX as a performance instrument. Even with 5 frames of delay, nobody can tell, especially because nobody is going to be specifically looking for that. They’ll be watching and appreciating your performance as it is, if it is moving and interesting.

4 Likes

I’m certainly not trying to say that a frame of latency doesn’t matter. I design latency free analogue video equipment for a living! :slight_smile: I’m trying to say that where that frame of latency occurs very much does. I’m sure you can detect the difference between 1 frame of delay and 2 frames of delay – but can you detect the difference between 6 frames of delay and 7 frames of delay? Maybe. Your perception happens on an exponential scale. 1 vs 2 frames is huge. 4 vs 5 frames is much smaller.

“Latency” as a concept, in isolation, is not specific enough to give us the big picture. Maybe with audio it is. Maybe with USB polling of a gamer mouse it is. Maybe with LCD monitor refresh times it is. But in our complex signal processing context, it’s like saying “fruit” as if “apples, oranges, and bananas” had negligible differences. I don’t really have a subjective opinion about this at all – I’m trying to illustrate that there are always objective physical constants involved, and where they may matter in practical terms may be elusive. In the example given, you have much more to gain from optimizing your playback PC, using high quality converters and display, than from an analog input amp.

I also deeply want to understand whether it is recursion latency or user interface latency optimization you are after, and if the interface in question is coming from software or is it the controls on the modules themselves?

If it is the controls on the modules themselves, just use a CRT for preview and you will have effectively zero latency. Done! Frontend buffering latency isn’t an issue. If you want us to design an effectively latency free video processing workflow – it’s already there. The latency of the TBC itself only matters in relation to the interface of external gear on the frontend of your system. So if you want to avoid frontend latency… don’t use a frontend video playback interface as part of your performance – that’s where the interface latency would propagate. If you need instantaneous playback of a video clip after you press a button, Memory Palace already provides that kind of functionality by preloading frames into memory.

Want optical camera feedback without frame delay at all? Awesome! But you’re now in experimental territory – we don’t have an out of the box solution for that. But in my experiments, I’ve got along just fine by using analog YPbPr or RGsB directly into the module minijacks, with RCA to 3.5mm converters, being fed from a camera with those outputs and a genlock input. SMX3 would be an ideal module for this purpose, since you can adjust colorspace and gain freely. You need a tube camera to really make that significant though – otherwise the CCD itself, video encoder, and processing going on inside the camera are going to introduce frame delay too! So if this is what you want, find a tube camera and start experimenting. If those experiments go well and you are finding that you need a formal DC restoration interface to improve on them, talk to us – let’s see what camera you’re using, and whether others have the same workflows, and we could justify releasing a module. Heck, even drop by the lab with your tube cam after your Chromagnon arrives and we can take some real measurements of what is actually going on.

Even if I were to bypass the frame memory in TBC2 firmware, you’re not going to get latency free performance like an analog signal path. There’s going to be a measurable processing delay through the process of ADC (video decoder IC, like ADV7181C) and then through the FPGA fabric, and then on to a DAC (video encoder IC, like ADV7393).

5 Likes

I’d also like to point out that when I say things like “TBC2 introduces 1+ frames of delay”, I’m not describing the performance of the module. I’m describing how time base correction / frame synchronization / genlock inputs work. They always introduce 1+ frames of delay. That’s what they’re supposed to do! That’s what we want them to do. That’s how a TBC synchronizes video sources.

And when I say “frontend user input latency doesn’t matter to your analog patch”, I mean that literally – not subjectively or dismissively. I’m trying to explain that if you turn the knob in your analog patch, you will see the results instantly. Even if your source is coming in through TBC2, that frontend latency will absolutely zero effect on the latency between the action of turning the knob and what you see on the output.

The intention behind my detailed reply is to educate, not rationalize. There’s nothing to rationalize. We’re discussing how time measurements factor into creative video processing workflows – “latency = bad” and “minimizing latency is something we should always do” sound like marketing brochures for broadcast or audio gear. I don’t want any disinformation spread here, I don’t want anyone to think they need to buy something that they don’t, and I want to make sure not just yourself – but others reading this user forum have a comprehensive and contextual view of the measurements we’re discussing and when they are – or are not – relevant.

6 Likes

True. But is that relevant to the Syntonie vu003? It appears the LM1881 is just there for blanking. Though it uses the backporch signal. I wouldn’t rule it out without testing. It could work.

1 Like

I had some success with getting a RCA rev Prismatic Ray that uses LM1881 to sync to a 1080P60 sync signal from Brian McKenna’s VS010. I breadboarded a H/V → CSync circuit to take the H/V pulses out of the VS010 into a CSync signal on a RCA plug and it seemed pretty solid to me.

I’ll have to try futzing with my VU003 and some other gear to see if we can find any answers on that! I’m pretty sure the colorspace will not look correct though with an HD source and would need some resistor tweaks.

3 Likes

Hey Lars I was just teasing you. I’ve got nothing but admiration and gratitude for what you and LZX are doing.

I’ve already got high performance PCs & peripherals, just trying to shave as much delay off the system as possible. Some may think I’m obsessive about this, but I assure you all it’s a big problem when trying to trigger visual events from an external device. The real time performance implications of 1+ frames of delay are such that it becomes difficult or impossible to play musically. At least for me.

As far as recursive delay goes, I want to control it. The possibility of no delay at all is intriguing, and I will start looking into camera options. TBC2’s delay control will come in very handy, I assure you, but I want that effect when I want it, and not all the time.

Regarding tube cameras, I’m not that obsessive. Dealing with those aging beasts doesn’t sound fun, and I may have to live with whatever delay is introduced by CCD or CMOS cameras.

Hey, it’s an exciting time to be alive. The creative tools I want are finally in my price range. There are a few missing pieces to the jigsaw puzzle, but I can still see the picture.

:grinning:

4 Likes

Thanks, and I’m sorry if I jumped down your throat there and made some assumptions. Obviously I am obsessive about the topic as well, and honestly I love these kinds of discussions. Even when they get heated. And I want to help you. It will help me a lot once I understand the baseline for your setup in a “case study” sort of way. Even if this is a purely theoretical shopping list of potential gear. Then I can really dig in and help you optimize it. And “what are you trying to do exactly” absolutely must be addressed in any discussion like this. I will be able to suggest options you may not be considering if we can figure that out. Again, even if it’s a theoretical process flow for now.

I think what you want for “latency free musical synchronicity” is going to be to keep things on the analog side. The kind of responsiveness you get between the input and the display there, is something you’ll get spoiled to really quick if you’re used to the latency stackup between digital converters and IO.

If you’re trying to play along with a video feed but don’t care if there’s some pre-buffering when you first click play, you can still get latency free behavior by monitoring at the point the feed is entering the analog signal path, too. I plan to have a second encoder in my system for using like this. This way I am monitoring the incoming video feed at the point it enters the analog path (after any TBC delay has been applied.)

4 Likes

I dug up some old diagrams of our Visionary series “Triple Video Interface” module – which is pretty much what you are asking for in the OP. These diagrams show two ways to use it: one with a component video source, and one with three monochrome cameras.

We could bring this back – but the thing that frustrated everyone is that using it requires that it’s either the only external source (genlocking the system to that source) or that your sources all have genlock capabilities natively (nothing modern does except for high end broadcast cams, so we’d be selling a product that only supports a treasure hunt for vintage gear.) So it seemed to just lead to user frustration or disappointment in the end – when users found out it was not a path to getting multiple simultaneous video feeds into the system.

So this would necessarily have to go in an implicitly “Legacy / Specialty Interfaces” category of modules, since how it’s used is dependent on special gear and sync flow puzzles as illustrated.

TBC2 on the other hand is exactly what the user community asked for when we asked what was needed: a dual channel interface, “anything to LZX, always works, some good effects and media features.” So if my bringing it up is repetitive, it’s for the folks searching for this topic in the future, and the likely best answer for them.

6 Likes

No worries Lars, I didn’t feel like you reacted harshly.

I get what you’re saying about how most users just want to get signals in without dealing with genlock. TBC2 meets or exceeds the needs of the vast majority of users.

For my part, I am coming from experience with both legacy systems (IP, EAB etc) and with pro video. In both cases, genlock to house sync is assumed.

The Triple Video Interface might not be the best solution to latency free input. It seems a tad baroque, and I can see why users might be frustrated or disappointed with it.

What I would love to see is a dead simple, dumb, single channel component video decoder with genlock input. I simply want to convert YUV or RGB broadcast video to the LZX standard. VGA input would also be great, but that is less important.

Regarding staying in the analog domain for musical synchronicity, that is not possible unless someone builds or modifies some analog control gear to work with LZX. There is no analog musical keyboard available. I am stuck with MIDI. And a big part of my anticipated workflow is sequencing. Yes, I want to perform live into the video system, but I also want to record and edit that performance in the PC. So you see why I’m so concerned about latency in the video rig, because I’ve already got a potentially problematic level of latency in the digital audio domain.

Thanks!

1 Like

That’s exactly what the TVI is. You mean Genlock/Sync output, correct? (The Genlock input would be on the camera.) If we put a Genlock input on it, then it is a TBC module and not an input amp (which is what you want to avoid.) An input amp by nature, is expecting a synchronous input source. I am just making sure you understand this.

If we brought it back, it would be revised to Gen3 standards to support HD/SD signals – and may receive extra features.

SVGA/XVGA/VGA standards can be received with the TBC2 module and scaled to one of the 15 different Gen3 “house timing” formats supported, but they are not native operational modes. So an input amp module would not be capable of supporting them.

If you are just talking about the VGA connector itself (as a way to receive SD/HD component video through a different interconnect) then that is possible.

So you see why I’m so concerned about latency in the video rig, because I’ve already got a potentially problematic level of latency in the digital audio domain.

So in this case, it is not minimal latency you want, it is being able to match the existing latency of your audio workflow, correct?

1 Like

If we put a Genlock input on it, then it is a TBC module and not an input amp (which is what you want to avoid.) An input amp by nature, is expecting a synchronous input source. I am just making sure you understand this.

Ah, I see, thank you for clearing that up. I guess the module wouldn’t need a genlock input in the use case I envision, all devices locked to house sync.

So in this case, it is not minimal latency you want, it is being able to match the existing latency of your audio workflow, correct?

Not really, I want minimal latency, especially in the recursive domain. The existing latency is something I do not like.

I gave some thought to the analog audio synthesizer issue. It’s not that CV controllers don’t exist, its that they are in the wrong voltage range. This brings back the need for a configurable voltage scaler with presets for different standards.

Which reminds me, I can’t wait to hook my Theremax up to Chromagnon. But the issue there again is the voltage, which is 0-10V linear, very oddball.

Thanks

2 Likes

A configurable voltage scaler is just an attenuator. And if you want presets, use a sequential (or addressable) switch in front of a mixer. Run your CV through the common input to the switch, each output of the switch goes to a different input on the mixer. Then each knob on the mixer is one of your presets for a scaling factor.

Or if you’re outputting your CV from a computer (which it sounds like you might be planning to do?), just do the scaling in the computer.

1 Like

You could probably modify a Cadet Scaler circuit to multiply by 10x to get the 0-1v LZX signals into the 0-10v range. I’m no expert, however there are many brilliant individuals on this forum who could probably figure out the component substitutions with ease. Maybe they could chime in.

To scale down to 0-1v, there are a few options available from Ommi Industries and BSO. The WMD SPO is a nice two channel scaler that I’ve used to scale 0-5v audio oscillators into the 0-1 range.

1 Like

No patch memory, but Visible Signals makes the Wrangler, which is a variable scaler.

1 Like

Thanks for the tips. Another thing that I’m concerned with is accuracy and repeatability. Part of my vision is running signals through patches in sequential passes, building up layers that are as close to being pixel accurately registered as possible. Think Scanimate on steroids. To keep the process as error-free as possible, I want to minimize the number of potentiometers involved. Even the new LZX regime of detente knob positions with tweaker calibration makes me nervous.

As for voltage scaling in the computer, that may or may not be an option or a convenient solution. I have many unanswered questions. I’m planning on getting an Expert Sleeper ED-9 once they are available again; I can ask the designer if it’s easy to configure for different voltage standards. I don’t want to be attenuating outputs in the DAW or virtual instrument; I’m concerned that would make monitoring levels difficult or impossible.

I was aware of the Visible Signals Wrangler, but again, I’m trying to avoid knobs.

I was aware of the Brown Shoes Only module. The big issue there is that some of the LZX gear is expecting a signed voltage -1 to +1, and the BSO 5:1 only outputs positive 0 to 1V. Thanks for the info about the WMD Scale Polarize Offset module. That looks good except for the potentiometer issue.

The Omii Industries Curtail module looks like it will handle signed or unsigned signals , and doesn’t have any pots to worry about. So that’s a good option as long as there’s no need to do any DC offset. But I can easily envision a situation in which that would be needed. E.g. 0 to -10V needs to be converted to -1 to +1 V.

This is why I am saying I want a configurable voltage scaler. There is a plethora of standards out there. Or should I say there really is no standard out there. When dealing with analog synths from the 70’s and 80’s it’s pretty much all 0 to 10 V, 1V/octave for pitch. But when you get to Eurorack audio, it’s all over the map.

So ideally I would like to see a voltage scaler with no knobs, but instead some switches or even a menu system. It should be capable of attenuation, amplification, DC offset, and inversion. That way it could convert to or from LZX voltage standards, or even among various random Eurorack modules.

The voltages it should support would be:

-10 to +10
0 to +10
-5 to +5
0 to +5
-1 to +1
0 to 1

1 Like