Should I be more concerned with point gap or timing

On my 81 KZ305, I can't get them both right. To get timing correct on both cylinders, one point set must be gapped under spec., and the other over spec.

Did you have install new breaker points? Used points or one worn out breaker point can´t set to the correct gap or ignition timing on both cylinder.
Other like the big KZ 650/900/1000 breaker points and timing plate, you can´t set up the breaker points gap and timing indepandent from each other on your engine.