Ego in the context of yoga philosophy does not mean to be egotistic or arrogant. It also does not mean the psychological definition of ego as Freud intended. Here we mean the identification to the false self. As an analogy, think of an unhatched chick inside an egg. The chick may think the eggshell is part of who he is. It feels safe within that space. He may be attached to it. Similarly, we may think our ego is who we are, our personality, our relationships, our religion, our race etc. And while it may make us feel that we belong somewhere, it is not who we are.
When my mind wandered, it seemed that I have temporarily lost grip of this ego or false self. The space in between thoughts, the unexpected gap between attachments, the loosening of the concept of "who I am" can scare us because of our clinging towards our identity. And yet, it is the observation between these thoughts, the cracking the eggshell, that allows us to move away from who we think we are and closer to who we really are. It is when we allow the small self to get lost and the ego to wander that we experience freedom from a very limited definition of that "who I am".