KZ 650 GSXR 750 front end swap

I am starting a 1978 KZ650 cafe racer project this winter. So far the biggest progress has been the front end swap, I chose a front end off a 2001 GSXR750. After trying to make the stock gixxer stem work I opted to swap the steering stems...first I grinded the weld off the bottom of the lower triple. The stem then tapped out easily. From there I took the pieces to a machinist where he pressed out the stem from the lower GSXR triple, then a type of sleeve was made that copied the GSXR stems stepped design then welded to the kz650 stem and pressed into the GSXR triple.

Then the opening in the GSXR upper triple had to be machined larger to accept the KZ stem. We opted to bore it a little larger than needed and need fabricate a metal press fit sleeve. Now the stem is secured in metal instead of aluminum. However, the only issue is that the kz stem does not line up flush with the upper triple. I thought about milling material off the bottom to close the gap, but there is not enough material there. Heres what I mean about the gap.

I am not sure if this is an issue or not...I am thinking it will be fine because the nut under the triple seems to do most of the securing and besides the bolt on top engages most of the threads anyhow. What do you guys think?
Here it is bolted up.