Smith was born on April 7, 1932, in Gans, Oklahoma, and was raised in Oakland, California. He began his music career performing at the Remember Me Cafe in San Francisco at the age of 15, but he was not financially successful at first. Throughout the 1950s, he was not able to continue his music career, so he worked at various other jobs, including truck driving and bronco busting. He appeared on the California Hayride television show in the mid-1950s before serving two years in the military. After his discharge, he began playing in a band in the San Francisco Bay Area.
In 1961, country music legend Ernest Tubb heard the band play and, after an audition, hired Smith to play guitar for the Texas Troubadours. Thus, Smith is heard playing in most of Tubb's 1960s recordings. His first solo single was 'Tear Stained Pillow/Eleven Long Years on the local Plaid label. Smith's stage name began to catch on after he released his second solo single, "I'll Just Go Home," in 1966 for Kapp Records, and he first cracked the Billboard charts with his second single, "The Only Thing I Want."
Smith permanently parted ways with Tubb and the Texas Troubadours in 1969 and he released his first solo album, Drinking Champagne, in 1969. The album's title track had reached the Top 40 on the country charts the previous year.
In 1970, Smith signed with Decca Records, and his popularity quickly soared, starting off with his 1972 top 10 hit, "I've Found Someone of My Own." He began recording songs written by some of the biggest names in the industry; for instance, in March 1973, his rendition of Bill Anderson's "The Lord Knows I'm Drinking" became his first number-one country hit. When Decca became MCA Records in 1973, Cal enjoyed his biggest successes. In 1974, he recorded two of his greatest hits, "It's Time to Pay the Fiddler" and "Country Bumpkin," which received Song of the Year Awards from both the Academy of Country Music and the Country Music Association.