Leo's frustration at his military failures led him to believe, in the fashion of the time, that the Empire had lost divine favour. Already in 722 he had tried to force the conversion of the Empire's Jews, but soon he began to turn his attention to the veneration of icons, which some bishops had come to regard as idolatrous. Following the renewed eruption of Thera in 726, he published an edict condemning their use, and had the image of Christ removed from the Chalke Gate, the ceremonial entrance to the Great Palace of Constantinople. The Emperor showed himself increasingly critical of the iconophiles, and in a court council in 730 he formally banned depictions of religious figures.

Leo's espousal of iconoclasm caused reactions among both the populace and the Church. The soldiers that took down the image of Christ from the Chalke were lynched, and a thematic rebellion that broke out in Greece in 727, was at least in part motivated by iconophile fervour. The Patriarch Germanos I resigned, to be replaced by the more pliant Anastasios. The emperor's edict drew the condemnation of popes Gregory II and Gregory III, as well as John of Damascus. Generally however, the dispute remained limited, as Leo refrained from actively persecuting iconophiles.