Lets say my scope is mounted 32.5mm above the dovetails and I could get a setup that would allow me to mount it 29mm above the dovetails.
It would be 3.5mm difference.

Would that have a drasic change on my POI @ 100 yards?

No, everything else being equal, it will move your POI up by 3.5mm.

This may not seem like much at 100m, but it will slightly extend the range at which hold-over becomes necessary. The smaller your game, the more important this becomes.

Ummm, right theory wrong sign (he's lowering his scope, you figgured for a higher setup).
POA/POI will change approx 3.5mm DOWN, and the point blank range will be 2-5 meters shorter.

If he lowers the scope, and keeps the same point of aim, he is actually raising the rifle. This will raise the point of impact.

Raising the scope, and keeping the same point of aim, would lower the rifle and lower the point of impact.