To the left of the 4th Grenadiers were the two squares of the 1st/ and 2nd/3rd Chasseurs who angled further to the west and had suffered more from artillery fire than the grenadiers. But as their advance mounted the ridge they found it apparently abandoned and covered with dead. Suddenly 1,500 British Foot Guards under Maitland who had been lying down to protect themselves from the French artillery rose and devastated them with point-blank volleys. The chasseurs deployed to answer the fire, but some 300 fell from the first volley, including Colonel Mallet and General Michel, and both battalion commanders. A bayonet charge by the Foot Guards then broke the leaderless squares, which fell back onto the following column. The 4th Chasseurs battalion, 800 strong, now came up onto the exposed battalions of British Foot Guards, who lost all cohesion and dashed back up the slope as a disorganized crowd with the chasseurs in pursuit. At the crest the chasseurs came upon the battery that had caused severe casualties on the 1st and 2nd/3rd Chasseurs. They opened fire and swept away the gunners. The left flank of their square now came under fire from a heavy formation of British skirmishers, which the chasseurs drove back. But the skirmishers were replaced by the 52nd Light Infantry (2nd Division), led by John Colborne, which wheeled in line onto the chasseurs' flank and poured a devastating fire into them. The chasseurs returned a very sharp fire which killed or wounded some 150 men of the 52nd. The 52nd then charged, and under this onslaught, the chasseurs broke.

The last of the Guard retreated headlong. A ripple of panic passed through the French lines as the astounding news spread: "La Garde recule. Sauve qui peut!" ("The Guard is retreating. Every man for himself!") Wellington now stood up in Copenhagen's stirrups and waved his hat in the air to signal a general advance. His army rushed forward from the lines and threw themselves upon the retreating French.

The surviving Imperial Guard rallied on their three reserve battalions (some sources say four) just south of La Haye Sainte for a last stand. A charge from Adam's Brigade and the Hanoverian Landwehr Osnabrück Battalion, plus Vivian's and Vandeleur's relatively fresh cavalry brigades to their right, threw them into confusion. Those left in semi-cohesive units retreated towards La Belle Alliance. It was during this retreat that some of the Guards were invited to surrender, eliciting the famous, if apocryphal,retort "La Garde meurt, elle ne se rend pas!" ("The Guard dies, it does not surrender!").