When James Graham was born on 1 January 1784, in Beaver, Pennsylvania, United States, his father, Hugh Graham, was 30 and his mother, Hannah Catharine Martin, was 30. He married Mary "Polly" Rayle in 1828. They were the parents of at least 5 sons and 5 daughters. He died on 11 January 1864, in his hometown, at the age of 80, and was buried in Fairview, Ohioville, Beaver, Pennsylvania, United States.