I finally broke down and bought a lower recently. Now I want to build up a decent shooting full size rifle.
A2 or M4 butt and standard or flat top upper.

What is the best source(s) of lower parts and building up an upper?

Don't want a full target racegun, but I do not want a minute of barn door shooter either!

Any assistance will be appreciated!

There are three legs to the accuracy triangle with an AR:
1. Get a good trigger. If you like single stage get a JP trigger (available from JP Enterprises or Brownells). If you like two stage triggers get a RRA Match trigger that has been worked over by John Holliger at White Oak Precision/White Oak Armament in Carlock, IL.
2. Get a good barrel. Your best bang for the buck is far and way a White Oak Armament with a matched bolt head. Define your purpose first before you decide on what profile barrel to get. Get a 1:7" or 1:8" twist barrel so you can shoot the longer (77grain), more accurate bullets from time to time. I really like their SPR profiled barrel. Not too heavy, but able to shoot accurately out to 700 meters.
3. Free float the barrel. The best bang for your buck this time is the Badger Ordnance unit, also available from Brownells. That, plus a WOA barrel and a good trigger wil get you into the 1/2" - 3/4" range without any trouble (five shot groups).

Fulton Armory sells a good book on AR-15 assembly.

A good, reasonbly priced source of parts is RB Precision. They are good to deal with.

lawndart

I really like Rock River Arms...Bushmaster is also very good, but I think Rock River is better. Armalite makes some good stuff, but their prices reflect that......The internal parts are pretty much the same, but make sure you get a good barrel, upper receiver combination.

TC, lawndart gave some super advice but it is also a fairly high dollar option but also a tack driver.
I have the basic same upper except mine is a Kreiger match 24" barrel with a machined matching bolt and sitting on a Billet Diamond lower. It is one of the sage rat guns and shoots ragged one hole groups if I can do my part. The down side to all this is COST....

If you are on a somewhat budget and want the most bang for your buck, go with a SS ultra match heavy barreled upper from Olympic Arms. I have owned several and they have all been shooters.

Here's a picture of my custom AR

Like most things in life, you can spend as much on an AR as you like.

For a shooter grade, something to plink with, teach young shooters and keep the looters off your lawn, I like Bushmaster and Rock River.

I have had great luck with the standard chrome lined Bushy and Colt barrels. They all have shot better than I can with iron sights.

The M4, Magpul or CAR style collapsable stocks look mean, but if I wasn't getting in and out of a vehicle, I would stick with the A2 stock myself. Especially if I was using a 20 or even 18 inch barrel.

For small parts, I have had good luck with Bushmaster and Brownells. Brownells also sells a lighter weight trigger spring kit that I can recommend.

Also, check out AR15 dot com. There are some folks over there who eat, drink and sleep AR-15s. Some of their BS is actually useful!