According to League of Legends wikia, both armor and magic resistance "damage multiplier" is calculated with the following formula (replace mr with armor the same):
if enemy MR is greater or equal to 0 (MR >= 0) calculation is like below;
100 / (MR+100)
which means;
for 100 magic resistance → × 0.5 incoming magic damage (50% reduction)
so for a single spell with magic damage of 800 will deal 400 damage (considering your have no magic penetration).
if enemy MR is less than 0 (MR < 0) calculation is like below;
2 − (100 / (100-MR))
which means;
for −25 magic resistance → × 1.2 incoming magic damage (20% increase)
so for a single spell with magic damage of 800 will deal 960 damage (considering your have no magic penetration).