ALL SIGHTS DRIFT OUT FROM LEFT TO RIGHT.
If you are driving it to the left then you are wedging it tighter into the slot.

-Spencer

Do you drift sights,blanks,etc from dovetails a set way like left to right or right to left ???
If you have a dial/digital vernier caliper, I'd be putting on either opening of the dovetail to see if one side is larger.Then use a brass drift punch to drive it out FROM the SMALLEST side.
I've had sucsess with this method previously.

I also agree with left to right. Definately use a good brass drift.

If I'm not mistaken, most dovetails are tapered from side to side making it only possible to drive them out one side or the other. If you are trying to drive it into the wedge it will be difficult.