With a VPS, the virtualization layer allows malicious admins to take a complete snapshot of the server (storage and RAM together) and inspect it at will; the nature of the snapshot is that you would not notice it. The equivalent in a dedicated server would entail one of the following:
- Switching off the server, take a complete copy of the disk, reboot.
- Install spying hardware directly within the machine (e.g. to make inspection of RAM).
- Add some spying software in the operating system (again from direct access to the disk on the rebooted machine).
All these imply at least a shutdown/reboot, that you may notice -- unless the spying hardware had been there since day 1, of course.
Disk encryption won't actually help you much. Either the machine can reboot alone -- in which case a snapshot of the disk would be perfectly able to boot in, say, a VM; or you have to type in a decryption password whenever your server boots, which is both inconvenient, and vulnerable to eavesdropping from the malicious admin.
All of this requires taking some explicit actions that may not go unnoticed. A motivated attacker won't be deterred, and will plan a phony "reboot for electrical maintenance" to get your machine down for some minutes without you panicking. What dedicated hardware protects you against, as opposed to a VPS, is casual indelicacies from subaltern admins / interns.
(One may argue that having your own hardware also protects you against side-channel attacks from other customers at the same hosting service. Cross-VM side-channel attacks have been demonstrated in lab conditions, so whether they could be pulled off in the wild is an open question.)