Nathaniel Carlyle Adderley was born in Tampa, Florida, but moved to Tallahassee when his parents were hired to teach at Florida A&M University. His father played trumpet professionally in his younger years, and he passed down his trumpet to Cannonball. When Cannonball picked up the alto saxophone, he passed the trumpet to Nat, who began playing in 1946. He and Cannonball played with Ray Charles in the early 1940s in Tallahassee and in amateur gigs around the area.