Graves was born Peter Duesler Aurness on March 18, 1926 in Minneapolis, Minnesota, a son of Methodist parents Rolf Cirkler Aurness (1894–1982), a businessman, and his wife Ruth (née Duesler, died 1986), a journalist.

Graves' ancestry was Norwegian and German. The family name originally was "Aursnes," but when Rolf's Norwegian father, Peter Aursnes, immigrated to New York City in 1887, he changed the spelling. Peter used the stage name "Graves", a maternal family name.

Peter Graves' Family Tree