Harald's favour at the imperial court quickly declined after the death of Michael IV in December 1041, which was followed by conflicts between the new emperor Michael V and the powerful empress Zoe.

During the turmoil, Harald was arrested and imprisoned, but the sources disagree on the grounds. The sources also disagree on how Harald got out of prison, but he may have been helped by someone outside to escape in the midst of the revolt that had begun against the new emperor.