The attack finally began around 6 p.m. Two of Early's brigades (under Brig. Gens. Harry T. Hays and Robert F. Hoke) pushed back Sedgwick's left-center across the Plank Road, but Anderson's effort was a slight one and McLaws once again contributed nothing: the final Confederate attack was made and repulsed. Throughout the day on May 4, Hooker provided no assistance or useful guidance to Sedgwick, and Sedgwick thought about little else than protecting his line of retreat.^[21]

General Benham of the U.S. Engineering Corps had added a bridge at Scott's Dam helping communicate with General Hooker. When the retreat was planned, General Benham on May 4 added a second bridge and he and General Sedgwick agreed to cross at night to avoid losing a large portion of his corps. The Union 6th Corps began retreating to a pre-planned smaller line closer to the bridges, and began their retreat without losses.^[32]