Do you have any other HD format sources that you can use to test this and can you link footage of the type of jitter you’re seeing?
I’m assuming you’re taking the YPbPr out of the Roland V-1600HD into the front panel inputs of VU003B and its rear Y output is routed directly to ESG’s rear Sync Input. Are you finding its different if you skip VU003B entirely and output the Y from V-1600HD into ESG’s rear Sync input? Since you can just bump ESG’s Brightness controls up to get a visible matte color, I don’t think it’s critical that you have the color bars from V-1600HD visible on the encoder to do this test. It might help eliminate some other routing issues if you have an HD distribution amplifier that can route the V-1600HD color bars separately to VU003B’s YPbPr input and ESG’s Sync input.
From what I’m reading in the schematics of VU003B, there is very minimal processing of the original sync sources outside of how it extracts them from a valid video signal. It uses the same sync separator chip as the Gen3 modules, LMH1980. I’d suspect there’s something about the compatibility of V-1600HD’s timing vs. ESG’s tolerance before VU003B but curious to see footage.