Which hardly matters when people are taking the homes of people living there currently and killing them. What difference does it make if some of someone’s ancestors centuries ago lived in that general part of the world?

You could justify European colonialism in Africa under similar logic, on the grounds that since humans evolved there before spreading out to the rest of the world, all Europeans have ancestors that lived on that continent at some point in the past and would merely be “reclaiming” it.