There is nothing wrong with a 223 bolt gun. In the wind they are no 22-250 or a Swift. Lizard has a slew of 223 Varmint rifles. Most built up on XPs and Rem 700s. All thats left is the action and bolt. In fact, whenever I see a deal on a 700 action, I always point him at it and he snaggs them up. The caliber really doesent matter as the barrel goes in the trash.
You cant load a bolt action 223 a whole lot faster, but you can use slower powders getting you a bit more velocity. You do have the option of seating the longer bullets out more to make more room for powder and as the throat errodes, the can chase the lands. In an AR, the mags limit your OAL. Bolt guns are much nicer on brass than ARs are. Viper/Stiller actions is making a Remington 700 Action clone out of stainless. Without a doubt the smartest way to go about building up a good bolt gun on a budget.
Lizard built me a 22 BR on a Stolle Panda Action a couple years ago. Equals a 22-250 with less powder. I put a 8 twist barrel on it so I could shoot the heavies.
Keep in mind Remington puts the same barrels on the same actions on all their rifles, short of the stuff that comes out of the custom shop. In other words, a PSS has the same hammer forged barrel that a ADL has. A lot of the $$$ difference is the stock, stainless vs carbon, and CDI factor.