My 25-06 hates heavy bullets. It does love Nosler 100 grain Partitions. IMR 4350 and RL-22 were favorite powders. I had them up near 3400 feet per second with the RL-22. For accuracy work it might be easier to see what your doing if you had a bit more magnification. If I ever get concerned about accuracy the first thing I di is put on one of my "diagnostic" 6.5-20 power scopes.

Try IMR-4831 or H-4831 these are old stand for the 25-06. In my 25-284 same case capacity I loaded 100gr Sierra's with 44.0gr of Varget and it shot them real well. So you might want to try this to.