On the morning of July 4, with Lee's army still present, Meade ordered his cavalry to get to the rear of Lee's army.^[122] In a heavy rain, the armies stared at one another across the bloody fields, on the same day that, some 900 miles (1,400 km) away, the Vicksburg garrison surrendered to Major General Ulysses S. Grant. Lee had reformed his lines into a defensive position on Seminary Ridge the night of July 3, evacuating the town of Gettysburg. The Confederates remained on the battlefield's west side, hoping that Meade would attack, but the cautious Union commander decided against the risk, a decision for which he would later be criticized. Both armies began to collect their remaining wounded and bury some of the dead. A proposal by Lee for a prisoner exchange was rejected by Meade.^[123]

Late in the rainy afternoon, Lee started moving the non-fighting portion of his army back to Virginia. Cavalry under Brigadier General John D. Imboden was entrusted to escort the seventeen-mile long wagon train of supplies and wounded men, using a long route through Cashtown and Greencastle to Williamsport, Maryland. After sunset, the fighting portion of Lee's army began its retreat to Virginia using a more direct (but more mountainous) route that began on the road to Fairfield.^[124] Although Lee knew exactly what he needed to do, Meade's situation was different. Meade needed to remain at Gettysburg until he was certain Lee was gone. If Meade left first, he could possibly leave an opening for Lee to get to Washington or Baltimore. In addition, the army that left the battlefield first was often considered the defeated army.^[125]