Dark Night of the Soul
As Everlyn Underhill mentions in Mysticism, there may be several reasons for the dark night of the soul.
Disidentification with the absolute
The obvious one is to facilitate a disidentification with the absolute, to allow for a more fluidity between the relative and absolute. If we are stuck in the absolute, if we have the absolute as our identity - even subtly, we are still stuck.
And then there are two other reasons, or rather mechanisms through which this process unfolds.
Exhaustion
One is exhaustion. During the brilliant sun of enlightenment, there is often tremendous amounts of energies coming through this human self, and there is often a tremendous level of activity and engagement in the world. Although it seems that it will go on forever, there is also a certain exhaustion that comes up on the level of our human self. And this exhaustion builds up until there is a collapse.
Shadow
The other is an integration of a subtle shadow, in Jungian terms. When we have any identity, what is left out is in the shadow. And in this case, our identity was the absolute (beyond and including all polarities, the superhuman), which means that the shadow is the relative (the world of form, differentiation, the purely human). So we are thrown out of our primary identity, and into our shadow. We become utterly human with all our weaknesses, flaws and delusions.
From being superhuman - beyond the human, and living through human life with a tremendous clarity, insights, compassion and engagement, we become utterly human - with flaws, weaknesses, delusions.
These two, and probably other mechanisms as well, come together to create the dark night of the soul.