Around 5:30 p.m.,having completed his circuit around the enemy, Jackson turned to Robert Rodes and asked him "General, are you ready?" When Rodes nodded, Jackson replied, "You may go forward then." Most of the men of the XI Corps were encamped and sitting down for supper and had their rifles unloaded and stacked. Their first clue to the impending onslaught was the observation of numerous animals, such as rabbits and foxes, fleeing in their direction out of the western woods. This was followed by the crackle of musket fire, and then the unmistakable scream of the "Rebel Yell".

Two of von Gilsa's regiments, the 153rd Pennsylvania and 54th New York, had been placed up as a heavy skirmish line and the massive Confederate assault rolled completely over them. A few men managed to get off a shot or two before fleeing. The pair of artillery pieces at the very end of the XI Corps line were captured by the Confederates and promptly turned on their former owners. Devens's division collapsed in a matter of minutes, slammed on three sides by almost 30,000 Confederates. Col. Robert Reily and his 75th Ohio managed to resist for about ten minutes before the regiment disintegrated with 150 casualties, including Reily himself, and joined the rest of the fleeing mob.

Col. Lee would later write sarcastically, "A rifle pit is useless when the enemy is on the same side and in rear of your line." Some men tried to stand and resist, but they were knocked over by their fleeing comrades and a hail of Confederate bullets. Maj. Gen. Carl Schurz ordered his division to shift from an east-west alignment to north-south, which they did with amazing precision and speed. They resisted for about 20 minutes and "Leatherbreeches" Dilger managed to drive the Confederates off the turnpike for a bit with his guns, but the sheer weight of Jackson's assault overwhelmed them, too, and they soon had to flee.

The chaos unfurling on the Union right had gone unnoticed at Hooker's headquarters until at last the sound of gunfire could be heard in the distance, followed by a panic-stricken mob of men and horses pouring into the Chancellorsville clearing. A staff officer yelled "My God, here they come!" as the mob ran to and past the Chancellor mansion. Hooker jumped onto his horse and frantically tried to take action. He ordered Maj. Gen Hiram Berry's division of the III Corps, once his own division, forward, yelling "Receive them on your bayonets!" Artillerymen around the clearing began moving guns into position around Fairview Cemetery.^[11]

Meanwhile, down at Hazel Grove, the 8th Pennsylvania Cavalry were relaxing and awaiting orders to chase after Confederate wagon trains, also oblivious to the collapse of the XI Corps. The regiment's commander, Maj. Pennock Huey, received a notice that General Howard was requesting some cavalry. Huey saddled up his men and headed west along the turnpike, where they ran straight into Robert Rodes's division. After a confused fight, the 8th Pennsylvania Cavalry retreated to the safety of the Chancellorsville clearing with the loss of 30 men and three officers.^[11]