In 1964 James Holloway was recalled to active duty to serve as Governor of the United States Naval Home in Philadelphia, Pennsylvania, until 1966. Additionally, he received the honorary degrees of Doctor of Laws from Muhlenberg College in Allentown, Pennsylvania in 1944, and later from the University of Notre Dame, and Doctor of Humane Letters from Villanova University in Philadelphia, Pennsylvania in 1948. He died from an aortic aneurysm at the age of 85. The Admiral James L. Holloway Jr. Award is presented annually by the Navy League of the US to the outstanding Navy Reserve Officers Training Corps midshipman in the nation. The Rear Admiral James L. Holloway Jr. Trophy honors the NavalAcademy midshipman who has contributed the most to Varsity Offshore J/24 sailing through his leadership, dedication to the team, and sailing skills.