Maybe I'm asking for the impossible. Logic in a fable.
But if I was in the other group I would go up to Haken's group, if nothing else to see if any close friends of mine were in there, to see if their powers were working any better than mine, ask about any other survivors etc, and I would expect the Eight to do the same. Actually if they could see it was Haken and the others-who were something of a group of leaders amongst the Coneheads, it would be even more likely the others would go to them.
That's why I believe all the elves were a. either descended from a small group, or B. did eventually meet up and reunite, but agree to splinter off. You might have had one founding group get lost and lose contact for thousands of years, but not multiple ones.