I figured it out. The bracket that holds the axle to the block is incorrect. I tried to use the bracket from my automatic trans. The axle is the same but the bracket that bolts to the carrier bearing is different.
The green one is the manual and the black one is the auto. You can see that the one on the right is slightly offset.
X04 317 printed on black , j30 3901 on the green.
Now how do I swap them? I guess I'm going to look up carrier bearins and try to swap to the other axle.
I'm glad I found this as it will likely make my drivetrain noise go away.