It looks like the resolution on inputs 1-3 is only 480p while the resolution for input4 for goes up to full HDi/p.
Even if the output resolution from your first mixer PinP composition is 1080p, it will be downscaled to 480p unless sent to input4.
This downscaled feed when sourced for the second PinP(which is also downscaling the image to produce the effect) is likely the source of the final fidelity loss you’re seeing