Background: 15 years of experience in software and apparently spoiled because it was already set up correctly.
Been practicing doing my own servers, published a test site and 24 hours later, root was compromised.
Rolled back to the backup before I made it public and now I have a security checklist.
I don’t think I’m ever opening up anything to the internet. It’s scary out there.
I don’t trust my competence, and if I did, I dont trust my attention to detail. That’s why I outsource my security: pihole+firebog for links, ISP for my firewall, and Tailscale for tunnels. I’m not claiming any of them are the best, but they’re all better than me.
I’m having the opposite problem right now. Tightend a VM down so hard that now I can’t get into it.
At least you had a backup
I’ve been quite stupid with this but never really had issues. Ever since I changed the open ssh port from 22 to something else, my server is basically ignored by botnets. These days I obviously also have some other tricks like fail2ban, but it was funny how effective that was.
We’re not really supposed to expose the ssh port to the internet at all. Better to hide it behind a vpn.
But it’s too damn convenient for so many use cases. Fuck it. Fail2Ban works fine.
You can also set up an ssh tarpit on port 22, which will tie up the bot’s resources and get them stuck in a loop for a while. But I didn’t think it was worth attracting extra attention from the bot admins to satisfy my pettiness.
Almost the same here. I also change some ssh settings: disable root login, disable password, allow only public key login. That’s about it. I never had any problems.
Fsil2ban piped to pfblocker works great. Plus snort
Weird. My last setup had a NAT with a few VMs hosting a few different services. For example, Jellyfin, a web server, and novnc/vm. That turned out perfectly fine and it was exposed to the web. You must have had a vulnerable version of whatever web host you were using, or maybe if you had SSH open without rate limits.
I’m confused. I never disable root user and never got hacked.
Is the issue that the app is coded in a shitty way maybe ?
You can’t really disable the root user. You can make it so they can’t login remotely, which is highly suggested.
sudo passwd -l rootThis disables the root user
There’s no real advantage to disable the root user, and I really don’t recommend it. You can disable SSH root login, and as long as you ensure root has a secure password that’s different than your own account your system is just as safe with the added advantage of having the root account incase something happens.
That wouldn’t be defense in depth. You want to limit anything that’s not necessary as it can become a source of attack. There is no reason root should be enabled.
I don’t understand. You will still need to do administrative tasks once in a while so it isn’t really unnecessary, and if root can’t be logged in, that will mean you will have to use sudo instead, which could be an attack vector just as su.
Why do like, houses have doors man. You gotta eliminate all points of egress for security, maaaan. /s
There’s no particular reason to disable root, and with a hardened system, it’s not even a problem you need to worry about…
Another thing you can do under certain circumstances which I’m sure someone on here will point out is depreciated is use TCP Wrappers. If you are only connecting to ssh from known IP addresses or IP address ranges then you can effectively block the rest of the world from accessing you. I used a combination of ipset list, fail2ban and tcp wrappers along with my firewall which like is also something old called iptables-persistent. I’ve also moved my ssh port up high and created several other fake ports that keep anyone port scanning my IP guessing.
These days I have all ports closed except for my wireguard port and access all of my hosted services through it.
You can’t really disable it anyway.
Hardening is mostly prevent root login from outside in case every other layer of authentication and access control broke, do not allow regular user to su/sudo into it for free, and have a tight grip on anything that’s executable and have a setuid bit set. I did not install a system from scratch in a long time but I believe this would be the default on most things that are not geared toward end-user devices, too.
This sounds like something everyone should go through at least once, to underscore the importance of hardening that can be easily taken for granted
I can’t even figure out how to expose my services to the internet, honestly it’s probably for the best Wireguard gets the job done in the end.
I’m interested, how do you expose your services (on your PC I assume) to the internet through wireguard? Is it theough some VPN?
Just a note, while wireguard makes its own VPN tunnel, it differs slightly in that it isn’t the typical VPN server with vpn connecting clients, it is more akin to a peer network. Each peer device gets it’s own “server” and “client” config section in the setup file. And you share the public keys between each peer before hand, and set the IPs to use.
Wireguard IS a VPN. He has somehow through his challenges of exposing services to the internet, exposed wireguard from his home to the internet for him to connect to. Then he can connect to his internal services from there.
It’s honestly the best option and how I operate as well. I only have a handful of items exposed and even those flow through a DMZ proxy before hitting their destination servers.
Oh, I thought it was a protocol for virtual networks, that merely VPNs used. The more you know!
Edit: spelled out VPN 😅
deleted by creator
VPN’s are neat, besides the fact they’re capable of masking your IP & DNS they’re also capable of providing resources to devices outside a network.
A good example is the server at my work is only accessible on my works network, to access the server remotely without exposing it directly to the internet would be to use a VPN tunnel.
I’ve gotta say this post made me appreciate switching to lemmy. This post is actually helpful for the poor sap that didn’t know better, instead of pure salt like another site I won’t mention.
I shared it because, out there, there is a junior engineer experiencing severe imposter syndrome. And here I am, someone who has successfully delivered applications with millions of users and advanced to leadership roles within the tech industry, who overlook basic security principles.
We all make mistakes!
There’s a 40 year I.T. veteran here that still suffers imposter syndrome. It’s a real thing I’ve never been able to shake off
Just look at who is in the White House, mate - and not just the president, but basically you can pick anyone he’s hand-picked for his staff.
Surely that’s an instant cure for any qualified person feeling imposter syndrome in their job.
Basic setup for me is scripted on a new system. In regards to ssh, I make sure:
- Root account is disabled, sudo only
- ssh only by keys
- sshd blocks all users but a few, via AllowUsers
- All ‘default usernames’ are removed, like ec2-user or ubuntu for AWS ec2 systems
- The default ssh port moved if ssh has to be exposed to the Internet. No, this doesn’t make it “more secure” but damn, it reduces the script denials in my system logs, fight me.
- Services are only allowed connections by an allow list of IPs or subnets. Internal, when possible.
My systems are not “unhackable” but not low-hanging fruit, either. I assume everything I have out there can be hacked by someone SUPER determined, and have a vector of protection to mitigate backwash in case they gain full access.
- The default ssh port moved if ssh has to be exposed to the Internet. No, this doesn’t make it “more secure” but damn, it reduces the script denials in my system logs, fight me.
Gosh I get unreasonably frustrated when someone says yeah but that’s just security through obscurity. Like yeah, we all know what nmap is, a persistent threat will just look at all 65535 and figure out where ssh is listening… But if you change your threat model and talk about bots? Logs are much cleaner and moving ports gets rid of a lot of traffic. Obviously so does enabling keys only.
Also does anyone still port knock these days?
Also does anyone still port knock these days?
Enter Masscan, probably a net negative for the internet, so use with care.
I didn’t see anything about port knocking there, it rather looks like it has the opposite focus - a quote from that page is “features that support widespread scanning of many machines are supported, while in-depth scanning of single machines aren’t.”
Sure yeah it’s a discovery tool OOTB, but I’ve used it to perform specific packet sequences as well.
I use port knock. Really helps against scans if you are the edge device.
Literally the only time I got somewhat hacked was when I left the default port of the service. Obscurity is reasonable, combined with other things like the ones mentioned here make you pretty much invulnerable to casuals. Somebody needs to target you to get anything.
Also does anyone still port knock these days?
If they did, would we know?
As a linux n00b who just recently took the plunge and set up a public site (tho really just for my own / selfhosting),
Can anyone recommend a good guide or starting place for how to harden the setup? Im running mint on my former gaming rig, site is set up LAMP
The other poster gave you a lot. If that’s too much at once, the really low hanging fruit you want to start with is:
-
Choose an active, secure distro. There’s a lot of flavors of Linux out there and they can be fun to try but if you’re putting something up publicly it should be running on one that’s well maintained and known for security. CentOS and Debian are excellent easy choices for example.
-
Similarly, pick well maintained software with a track record. Nginx and Apache have been around forever and have excellent track records, for example, both for being secure and fixing flaws quickly.
-
If you use Docker, once again keep an eye out for things that are actively maintained. If you decide to use Nginx, there will be five million containers to choose from. DockerHub gives you the tools to make this determination: Download number is a decent proxy for “how many people are using this” and the list of updates tells you how often and how recently it’s being updated.
-
Finally, definitely do look at the other poster’s notes about SSH. 5 seconds after you put up an SSH server, you’ll be getting hit with rogue login attempts.
-
Definitely get a password manager, and it’s not just one password per server but one password per service. Your login password to the computer is different from your login to any other things your server is running.
The rest requires research, but these steps will protect you from the most common threats pretty effectively. The world is full of bots poking at every service they can find, so keeping them out is crucial. You won’t be protected from a dedicated, knowledgeable attacker until you do the rest of what the other poster said, and then some, so try not to make too many enemies.
Thank you!
The TLDR is here : https://www.digitalocean.com/community/tutorials/recommended-security-measures-to-protect-your-servers
You won’t be protected from a dedicated, knowledgeable attacker until you do the rest of what the other poster said, and then some,
You’re right I didn’t even get to ACME and PKI or TOTP
https://letsencrypt.org/getting-started/
https://openbao.org/docs/secrets/pki/
https://openbao.org/docs/secrets/totp/
And for bonus points build your own certificate authority to sign it all.
https://smallstep.com/blog/build-a-tiny-ca-with-raspberry-pi-yubikey/
Thank you for this! I’ve got some homework to do!
Good luck on your journey.
I would suggest having two servers one to test and one to expose to the Internet. That way if you make a mistake hopefully you’ll find it before you expose it to the Internet.
-
Paranoid external security. I’m assuming you already have a domain name. I’m also assuming you have some ICANN anonymization setup.
This is your local reverse Proxy. You can manage all this with a container called nginx proxy manager, but it could benefit you to know it’s inner workings first. https://www.howtogeek.com/devops/what-is-a-reverse-proxy-and-how-does-it-work/
https://cloud9sc.com/nginx-proxy-manager-hardening/
https://github.com/NginxProxyManager/nginx-proxy-manager
Next you’ll want to proxy your IP address as you don’t want that pointing to your home address
https://developers.cloudflare.com/learning-paths/get-started-free/onboarding/proxy-dns-records/
Remote access is next. I would suggest setting up wireguard on a machine that’s not your webserver, but you can also set that up in a container as well. Either way you’ll need to punch another hole in your router to point to your wire guard bastion host on your local network. It has many clients for windows and linux and android and IOS
https://github.com/angristan/wireguard-install
https://www.wireguard.com/quickstart/
https://github.com/linuxserver/docker-wireguard
Now internally, I’m assuming you’re using Linux. In that case I’d suggest securing your ssh on all machines that you log into. On the machines you’re running you should also install fail2ban, UFW, git, and some monitoring if you have the overhead but the monitoring part is outside of the purview of this comment. If you’re using UFW your very first command should be
sudo ufw allow sshhttps://www.howtogeek.com/443156/the-best-ways-to-secure-your-ssh-server/
https://github.com/fail2ban/fail2ban
https://www.digitalocean.com/community/tutorials/ufw-essentials-common-firewall-rules-and-commands
Now for securing internal linux harden the kernel and remove root user. If you do this you should have a password manager setup. keepassx or bitwarden are ones I like. If those suck I’m sure someone will suggest something better. The password manager will have the root password for all of your Linux machines and they should be different passwords.
https://www.makeuseof.com/ways-improve-linux-user-account-security/
https://bitwarden.com/help/self-host-an-organization/
Finally you can harden the kernel
https://codezup.com/the-ultimate-guide-to-hardening-your-linux-system-with-kernel-parameters/
TLDR: it takes research but a good place to start is here
Correct, horse battery staple.
I couldn’t justify putting correct in my username on Lemmy. But I loved the reference too much not to use it, so here I am, a less secure truncated version of a better password.
Permitting inbound SSH attempts, but disallowing actual logins, is an effective strategy to identify compromised hosts in real-time.
The origin address of any login attempt is betraying it shouldn’t be trusted, and be fed into tarpits and block lists.
Endlessh and fail2ban are great to setup a ssh honeypot. There even is a Prometheus exporter version for some nice stats
Just expose endlessh on your public port 22 and if needed, configure your actual ssh on a different port. But generally: avoid exposing ssh if you don’t actually need it or at least disable root login and disable password authentication completely.
https://github.com/skeeto/endlessh https://github.com/shizunge/endlessh-go https://github.com/itskenny0/fail2ban-endlessh
If it is your single purpose to create a blocklist of suspect IP addresses, I guess this could be a honeypot strategy.
If it’s to secure your own servers, you’re only playing whack-a-mole using this method. For every IP you block, ten more will pop up.
Instead of blacklisting, it’s better to whitelist the IP addresses or ranges that have a legitimate reason to connect to your server, or alternatively use someting like geoip firewall rules to limit the scope of your exposure.
Since I’ve switched to using SSH keys for all auth Ive had no problems I’m aware of. Plus I don’t need to remember a bunch of passwords.
But then I’ve had no training in this area. What do I know
I’ve recently seen login attempts using keys, found it curious…
Probably still looking for hosts that have weak Debian SSH keys that users forgot to replace. https://www.hezmatt.org/~mpalmer/blog/2024/04/09/how-i-tripped-over-the-debian-weak-keys-vuln.html
I disabled ssh on IPv4 and that reduced hacking attempts by 99%.
It’s on IPv6 port 22 with a DNS pointing to it. I can log into it remotely by hostname. Easy.
sudo sudokuecho gecko
Do not allow username/password login for ssh. Force certificate authentication only!
Do not allow username/password login for ssh
This is disabled by default for the root user.
$ man sshd_config ... PermitRootLogin Specifies whether root can log in using ssh(1). The argument must be yes, prohibit-password, forced-commands-only, or no. The default is prohibit-password. ...If it’s public facing, how about dont turn on ssh to the public, open it to select ips or ranges. Use a non standard port, use a cert or even a radius with TOTP like privacyIdea. How about a port knocker to open the non standard port as well. Autoban to lock out source ips.
That’s just off the top of my head.
There’s a lot you can do to harden a host.
dont turn on ssh to the public, open it to select ips or ranges
What if you don’t have a static IP, do you ask your ISP in what range their public addresses fall?
Why though? If u have a strong password, it will take eternity to brute force
