When James Stuart was born on 8 April 1881, in Lancefield, Victoria, Australia, his father, Robert Stewart, was 42 and his mother, Elizabeth Luckhurst, was 40. He married Agnes Jessie Knight on 8 March 1916, in Rushworth, Victoria, Australia. They were the parents of at least 1 son and 2 daughters. He died on 16 July 1979, in Eltham, Victoria, Australia, at the age of 98, and was buried in Eltham Cemetery, Eltham, Victoria, Australia.