Richard Ewell's second division, under Jubal Early, swept down the Harrisburg Road, deployed in a battle line three brigades wide, almost a mile across (1,600 m) and almost half a mile (800 m) wider than the Union defensive line. Early started with a large-scale artillery bombardment. The Georgia brigade of Brigadier-General John B. Gordon was then directed for a frontal attack against Barlow's Knoll, pinning down the defenders, while the brigades of Brigadier-General Harry T. Hays and Colonel Isaac E. Avery swung around their exposed flank. At the same time the Georgians under Doles launched a synchronized assault with Gordon. The defenders of Barlow's Knoll targeted by Gordon were 900 men of von Gilsa's brigade; in May, two of his regiments had been the initial target of Thomas J. "Stonewall" Jackson's flanking attack at Chancellorsville. The men of the 54th and 68th New York held out as long as they could, but they were overwhelmed. Then the 153rd Pennsylvania succumbed. Barlow, attempting to rally his troops, was shot in the side and captured. Barlow's second brigade, under Ames, came under attack by Doles and Gordon. Both Union brigades conducted a disorderly retreat to the south.^[38]

The left flank of the XI Corps was held by Gen. Schimmelfennig's division. They were subjected to a deadly artillery crossfire from Rodes' and Early's batteries, and as they deployed they were attacked by Doles' infantry. Doles' and Early's troops were able to employ a flanking attack and roll up three brigade of the corps from the right, and they fell back in confusion toward the town. A desperate counterattack by the 157th New York from von Amsberg's brigade was surrounded on three sides, causing it to suffer 307 casualties (75%).^[39]

Gen. Howard, witnessing this disaster, sent forward an artillery battery and an infantry brigade from von Steinwehr's reserve force, under Col. Charles Coster. Coster's battle line just north of the town in Kuhn's brickyard was overwhelmed by Hays and Avery. He provided valuable cover for the retreating soldiers, but at a high price: of Coster's 800 men, 313 were captured, as were two of the four guns from the battery.^[40]

The collapse of the XI Corps was completed by 4 p.m., after a fight of less than an hour. They suffered 3,200 casualties (1,400 of them prisoners), about half the number sent forward from Cemetery Hill. The losses in Gordon's and Doles's brigades were under 750.^[41]