You may just be connecting to a server that is much closer, there are also more smaller servers for a much smaller client based too. People who host these servers are usually in the IT community and probably hella overspecd the server vs userbase size too. Lemmy is also an open source project that has a lot of eyes to solve and fix issues