"For every show we've ever done, there is always hype, expectancy."

-John Paul Jones

Jones was born in Sidcup, Kent (now part of Greater London). He started playing piano at the age of six, learning his keyboard skills from his father, Joe Baldwin, a pianist and arranger for big bands in the 1940s and 1950s, notably with the Ambrose Orchestra. His mother was also in the music business which allowed the family to often perform together touring around England.

Jones joined his first band, The Deltas, at 15. He then played bass for jazz-rock London group, Jett Blacks, a collective that included guitarist John McLaughlin. Jones' big break came in 1962 when he was hired by Jet Harris and Tony Meehan of the successful British group The Shadows for a two-year stint. Shortly before hiring Jones, Harris and Meehan had just had a Number 1 hit with "Diamonds" (a track on which Jones' bandmate-to-be Jimmy Page had played.) Jones' collaboration with the Shadows nearly prevented the future formation of Led Zeppelin, when the parties engaged in talks about the possibility of Jones replacing their bassist Brian Locking, who left the band in October 1963.

During his time as a session player, Jones often crossed paths with guitarist Jimmy Page, a fellow session veteran. In June 1966, Page joined The Yardbirds, and in 1967 Jones contributed to that band's Little Games album. The following winter, during the sessions for Donovan's The Hurdy Gurdy Man, Jones expressed to Page a desire to be part of any projects the guitarist might be planning. Later that year, The Yardbirds disbanded, leaving Page and bassist Chris Dreja to complete some previously booked Yardbirds dates in Scandinavia. Before a new band could be assembled, Dreja left to take up photography. Jones, at the suggestion of his wife, inquired to Page about the vacant position, and the guitarist eagerly invited Jones to collaborate.