You will not ruin the brake levers by removing the sissy bar. However you will need to adjust out the extra slack in the inner cable.
What you will do is take the bike further away from originality. For better or worse, that's what the bikes had back then. So if you ever imagine anyone might want to restore it, please keep all the parts aside.