While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.
As I am told, this was the issue:
- There is an vulnerability which was exploited
- Several people had their JWT cookies leaked, including at least one admin
- Attackers started changing site settings and posting fake announcements etc
Our mitigations:
- We removed the vulnerability
- Deleted all comments and private messages that contained the exploit
- Rotated JWT secret which invalidated all existing cookies
The vulnerability will be fixed by the Lemmy devs.
Details of the vulnerability are here
Many thanks for all that helped, and sorry for any inconvenience caused!
Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been ‘stolen’ and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).
For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.
It’s… slightly more complicated than that, but it should be possible.
This was one of my early fears… that an instance might get offline, attacked, whatever, and that all your user data might be lost. It appears that a feature like this might be inevitable. Or at least a feature that would let a user to have at least 1 copy of his account on another instance.
There is also the DB and storage growth to consider if something like this is implemeneted. If all instances hold all data from every other instance, than that might turn into a shit show pretty soon. You can get away with a certain number of users on only one server, but after a certain threshold, you have to scale… and that costs a lot of money, which is a no go for smaller instances and they will probably shut down if they’re forced to hold data for every user account in the fediverse.
If I had to propose something, I’d suggest writing a spec and implementing an API for Lemmy servers that would allow one to submit signed actions, like posts, comments, likes, display name, etc. Then people will write clients that will allow generation of certificates to be used to sign those messages + add the ability to export/back up the key to be used by other apps too.
If storage space ever becomes an issue, you could potentially shard communities. I’m more worried about the network traffic. I already suggested a routing algorithm that would spread the load between all federated instances and would scale like
O(log(N)). There was some interest, but it would be a long term project. There’s still a lot of performance that can be gained by simple optimisations.Yeah, there are more pressing issues at hand, a lot of bugs, a lot of new features that need to be implemented ASAP. Like user following, Lemmy is still lacking that.