I believe your right and I don't think there would be any real harm. Just thinking in the interest of safety.... I have a similar situation with my 338~06, but just made my eldest son happy my giving him all my 3006 stuff.
The only 223 I have now is in the form of a Bushmaster m4, so it would be an easy barrel swap.