Yes, this is a big deal! You don't need to have any more cam end play than you do with the crank. Yes, the cam will knock back and forth and can break the cam plate if it is as excessive as your seeing and not corrected. You may need to pull it back apart and make sure you didn't install the spacer backwards. It will make a difference.
