Actually, to be technically correct, elves do sometimes have issues with inbreeding, or rather a small gene pool that keeps getting shallower.
The reason? They breed without Recognition. Recognition ensures that the child will be genetically superior, gifted, uniquely suited for survival, and have no defects to be carried down.
Without Recognition? You get the "never clever" Go-Backs.
Just adding my two cents.