I think you’re underestimating the size of the castle complex; there are multiple buildings within the walls. Also, they’re expensive to build, maintain and use, so it’s not that weird to keep things compact.