Main RF generation
If the Modulator is bypassed (transformer connected), the modulator input must be desabled. So you the 100ohm diff resistor shall be removed? (BB input ~7kohm). Layout must be carefully done to avoid stub on diff transmission line.
What is the low pass response of the DAc output filter? (amplitude and phase flatness)
An input matching circuit on the LO input of the LTC5598 may be required: zin=~41 ohm