Caldwell was born November 13, 1947 in Spartanburg, South Carolina, to Mr. and Mrs. Toy Talmadge Caldwell Sr. He began playing guitar before his teen years with his younger brother Tommy Caldwell. He developed a unique style of playing, playing the electric guitar using his thumb rather than a pick. Toy played basketball and football in high school with friends George McCorkle, Jerry Eubanks, and Doug Gray. While very involved in sports, the boys eventually became interested in music including jazz and blues. By age sixteen, Toy was passionate about music, sports, and his other obsession, motorcycles. He also enjoyed hunting and fishing. Caldwell decided to serve his country and enlisted in the United States Marine Corps. In 1966, he reported for recruit training at Parris Island, South Carolina. After being wounded in Vietnam in September 1968, he was evacuated for 2 weeks, then returned for duty. Caldwell was discharged in 1969 and once again began playing music with his high school buddies. The Spartanburg chapter of the Marine Corps League is named the Hutchings-Caldwells Detachment in honor of Toy, his brother Tommy and another Marine.