when I was in the library, so I gestured to say I was on my way.

Oh? You didn't see him at all, but you guessed that he saw you, and so you gestured. Which means...