James Madison Downey (12 December 1809–28 March 1884), Speaker of the Restored House of Delegates and member of the Convention of 1864, was born in Franklin County, Pennsylvania, and was the son of William Downey, a farmer, and Maria Dorothy Hartman Downey. Nothing is known of his education. On 16 October 1834 he married Ann Eliza Funk. Six of their seven daughters and four of their five sons survived childhood. About 1847 Downey moved across the state line to Washington County, Maryland, where he farmed and cared for his aging father. On 15 March 1858 Downey paid $12,000 for Loudoun Mills, a sixty-five-acre tract in Loudoun County, and soon thereafter moved to Virginia. He sold the contents of his mill, distillery, store, and farm in September 1859. The following February he advertised the sale of the rest of his property, but he continued to reside there and in August 1860 employed four men at the mill. Census and tax records for that year indicate that he owned no slaves.

Civil War
Downey opposed secession in 1861, but because his property was in a contested area, during the Civil War armies and raiding parties on both sides seized grain, fodder, and many thousands of fence rails and bricks for campfires and shelters. Downey attempted to mitigate the harsh treatment of loyal Loudoun County residents by United States authorities who, among other things, cut off travel and trade to markets in Maryland and elsewhere in the United States. In 1864 he asked that the Union regional military commander send cavalry troops to Loudoun to protect loyal citizens and their crops. The shortages were made more severe by unscrupulous military officials and speculators, one of whom tried to recruit Downey. Three of Downey's daughters died during the Civil War. Confederates killed one of his sons while he was guiding Union troops, hanged one of his sons-in-law, and arrested Downey three times. A Confederate guerilla threatened his life in October 1863, after which Downey hid out in Maryland and Washington, D.C., until the end of the war.

On 28 May 1863 Downey was one of two Loudoun County men elected to the House of Delegates of the loyal Restored government, which met in Alexandria. The eleven delegates, who represented eight counties and the city of Norfolk, elected Downey as Speaker for the session that ran from 7 December 1863 to 8 February 1864. On 21 December 1863 he voted with the majority for a bill calling for a convention to revise the state constitution.

Convention of 1864
Downey was one of three men elected on 21 January 1864 to represent Loudoun County in the constitutional convention that met in Alexandria from 13 February through 11 April 1864. He voted with the majority to deny access to newspaper reporters who refused to take an oath of loyalty to the Restored government. At Downey's suggestion, the convention formed standing Committees on the Bill of Rights and Qualification of Voters, on Emancipation and Education, on the Executive Department and Judiciary, and on the Legislative Department. As chair of the Committee on the Legislative Department, he introduced on 23 February resolutions defining the membership of the General Assembly. Downey also submitted an ordinance defining treason and its punishment, and he introduced a resolution recommending that money from the sale of property confiscated from Confederates be used to relieve loyal families who had lost property and also to reduce the state debt, fund internal improvements, and support free schools. He proposed to establish a statewide system of free public schools for the poor, and he advocated promoting art and science curricula in colleges and seminaries.

Downey voted with the majority on 10 March to abolish slavery in the state. In proposing an amendment to disfranchise men who had given aid or comfort to the rebellion, he faced criticism that a convention with a mere seventeen delegates would appear ridiculous revoking the voting rights of thousands of men. Downey replied, "If we don't disfranchise traitors we shall render ourselves ridiculous in the eyes of the world." He also had to defend himself against a charge that he had given aid and comfort, even sold whiskey, to Rebel soldiers, and in spite of the treatment many of his neighbors had received from the Federal government during the war, Downey opposed having the convention appeal the matter to Abraham Lincoln for fear of alienating the president. On 31 March Downey voted with the majority to disfranchise and to prohibit from holding public office all men who served in the Confederate government and Congress, in rebellious state legislatures, and as high-ranking Confederate military officers. He chaired the committee to prepare the schedule for implementing the revised constitution. Downey voted on 4 April with the minority to submit it to the people for approval, rather than proclaiming it in effect. On 7 April he voted with the minority against the adoption of the final version, but he later signed the new constitution, which recognized the creation of West Virginia as a separate state, provided funding for primary and free schools, reduced to three the number of judges on the Supreme Court of Appeals, and required voting by paper ballot for state officers and members of the General Assembly.

House of Delegates
Downey won reelection to the House of Delegates on 26 May 1864 and on 6 December was reelected Speaker. He presided over the House until it adjourned on 7 March 1865. Following the collapse of the Confederacy and the relocation of the Restored government to Richmond, he called the House to order in extra session in the State Capitol on 19 June. Three days later the House approved and the Senate of Virginia subsequently passed a bill restoring the voting rights of many of the men the new state constitution had disfranchised. The House also passed a bill to legalize marriages of formerly enslaved African Americans, but the Senate declined to consider it because it had adopted a rule to accept no new proposals. Downey opposed granting the vote to freedmen, and before closing the proceedings he congratulated the House on having kept Virginia out of the hands of the abolitionists. He added that although radicals "may force negro suffrage on other States with provisional governments, they cannot pile it upon us."

Downey returned in May 1865 to his Loudoun County home. A decade later he received $3,222 in compensation for the property that Union forces had destroyed or appropriated. Downey resumed operating his mills, although his profits fell far below what he had earned in 1860. He also reopened the store and distillery and continued to farm. His wife died on 16 May 1881. James Madison Downey died at his son's residence near Taylorstown, in Loudoun County, on 28 March 1884, and was buried at the cemetery of New Jerusalem Lutheran Church, just outside the nearby town of Lovettsville.

Sources Consulted:
Birth, marriage, and death dates of both 24 and 28 Mar. 1884 in Downey family Bible records (compiled typescript, 1981), provided by Richard M. Cochran (2009), copy in Dictionary of Virginia Biography Files, Library of Virginia (LVA); Loudoun Co. Deed Book, 5-Q:169–170; Downey correspondence in United States War Department, The War of the Rebellion: A Compilation of the Official Records of the Union and Confederate Armies (1880–1901), 1st ser., 43, pt. 1, 776–777, in Francis Harrison Pierpont Executive Papers (1861–1865), Accession 36928, Record Group 3, LVA, in Records of the Office of the Secretary of War, Letters Received, Record Group 107, and in Records of Civil War Special Agencies of the Treasury Department, Entries 661, 666, Record Group 366, both in National Archives and Records Administration (NARA), Washington, D.C.; Alexandria Gazette, 1 June (p. 2), 9 Dec. 1863 (p. 3), 24 Mar. (first quotation on p. 1), 28 Mar. (p. 1), 31 Mar. 1864 (p. 1); New York Times, 2 July 1865 (second quotation); Journal of the House of Delegates of the State of Virginia (Alexandria), 1863–1864 sess., 3, 4, 38, 42, 1864–1865 sess., 4, 1865 extra sess. (Richmond), 3; Journal of the Constitutional Convention Which Convened at Alexandria on the 13th Day of February 1864 (1864), 3, 6–10, 16–18, 33, 41, 43–44, 47, 48; testimony of Downey, 29 May 1873, Claims 12,3461, 12,462, 12,463 (with self-reported birthplace), Southern Claims Commission Approved Claims, 1871–1880, Virginia, Records of the Accounting Officers of the Department of the Treasury, Record Group 217, NARA; Anna Pierpont Siviter, Recollections of War and Peace, 1861–1868 (1938), 172, 174; Taylor M. Chamberlin and James D. Peshek, Crossing the Line: Civilian Trade and Travel between Loudoun County, Virginia, and Maryland during the Civil War (2002), 22, 27–32, 34, 38, 45–46; Eugene M. Scheel, Loudoun Discovered: Communities, Corners, and Crossroads, vol. 5: Waterford, the German Settlement, and Between the Hills (2002), 76–79; estate records in Loudoun Co. Will Book, 3-I:357–358, 488, Thomas Balch Library, Leesburg, Va.; death notices in Leesburg Mirror, 3 Apr. 1884, and Hamilton Loudoun Telephone, 4 Apr. 1884 (both with death date of Friday, 28 Mar. 1884).

Written for the Dictionary of Virginia Biography by Donald W. Gunter.

How to cite this page:
Donald W. Gunter,"James Madison Downey (1809–1884)," Dictionary of Virginia Biography, Library of Virginia (1998– ), published 2016 (http://www.lva.virginia.gov/public/dvb/bio.asp?b=Downey_James_Madison, accessed [today's date]).

Return to the Dictionary of Virginia Biography Search page.