I mean, he was also a Jedi, so separating kids from their parents is pretty par for the course. Maybe they were simply sent off to Empire training camp boarding school.