On the evening of May 3 and all day May 4, Hooker remained in his defenses north of Chancellorsville. Lee observed that Hooker was threatening no offensive action, so felt comfortable ordering Anderson's division to join the battle against Sedgwick. He sent orders to Early and McLaws to cooperate in a joint attack, but the orders reached his subordinates after dark, so the attack was planned for May 4.^[21]

By this time Sedgwick had placed his divisions into a strong defensive position with its flanks anchored on the Rappahannock, three sides of a rectangle extending south of the Plank Road. Early's plan was to drive the Union troops off Marye's Heights and the other high ground west of Fredericksburg. Lee ordered McLaws to engage from the west "to prevent [the enemy] concentrating on General Early."^[21]

At 7 a.m. on May 4, Early recaptured Marye's Heights, cutting Sedgwick off from the town. Early reoccupied Marye's Heights on the morning of May 4, cutting Sedgwick off from the town. However, McLaws was reluctant to take any action.