For example, I dont want to allow ssh or rdp traffic between servers other than the primary interface.
You state that the primary interface is on its own VLAN, and the other interfaces are on different VLANs. Therefore some device on the network is handling inter-VLAN routing. I'd create the ACLs at that point.