You Need to Rethink that “Jump Server.”

If you’ve been following the news (oh look, Yahoo was hacked again), then you know that everyone’s a target for hacking. I’ve explained before that even companies in the “we don’t have anything anyone would want to hack” do, in fact, have something someone would want to hack – often information that can stepping-stone the attacker to what they really want.

I also know a lot of you out there rely on “jump servers.” You are asking for a cyber punch in the throat.

For those who don’t know the term, a “jump server” usually exists in environments that like to think of themselves as highly secure. The idea is that admins are completely firewalled off from the datacenter, and instead have to log into a “jump server,” which is itself in the datacenter, and is the portal by which admins access the rest of the datacenter.

This is how Yahoo got hacked the first time. They too had a jump server, and a clever attacker managed to get a piece of malware onto it, and used to to leak information over a long, long period of time.

So let’s be clear on a few points.

A jump server alone does not make your environment inherently more secure. Just because someone has to go through an extra step to get to servers does not make those servers more secure. Everyone has this idiotic concept that their firewall represents an impenetrable barrier, and that everything inside the firewall is rainbows and unicorns. It isn’t. As an attacker, I appreciate a jump server, because it’s a single point of failure that I need to hack.

A jump server can be useful when it’s a sort of privileged access proxy. That is, I connect to it as a normal user, and it does whatever I ask it to using elevated privileges that only last for the duration of my stay. This is what Microsoft’s JIT/JEA approach attempts to create, although it’s pretty early days on that, and we don’t have tooling to make managing it very easy. But if you’re logging into a jump server as an admin, meaning your privileged credential lives outside the datacenter, then you’re not adding a bit of security to anything.

If you have a jump server, you need to lock that sumbitch down, and hard. You don’t need anti-malware tools on it; you need an aggressive whitelist of what is allowed to run, and a system – like AppLocker on Windows – that will nuke anything not on the whitelist. And, when it does nuke something, will send alert emails to the entire company (which AppLocker won’t). Not kidding – the Yahoo kids knew something was up, but didn’t make loud enough noises. This jump server should be destroyed and rebuilt frequently – like, once a day – using automated tooling. Doing so makes it harder for a hacker to take hold. Access to the jump server must be via two-factor authentication (2FA), period, making it harder for a compromised account to insert code onto the server. Think defense in depth, here – imagine what might happen, put in a means of blocking that, and then imagine what would happen if the block failed, and deal with that also. For example, a jump server’s NICs should be prohibited from entering any kind of promiscuous mode where they could listen to all the traffic on their segment.

If you use this jump server approach, have more than one. Your datacenter should be internally segmented, by firewalls, in to secure zones. That helps to minimize an attacker’s reach should he penetrate any one zone. A jump server per zone – secured in the way I’ve described – can allow administrative access to each zone.

Human and business controls must exist for all access to privileged operations. A jump server, if that’s your approach, should log each and every access. It should prompt admins for a reason they’re there – such as a ticket number for whatever problem they’re working on. Processes should exist to reconcile tickets and access attempts, so that you can detect inappropriate accesses to the jump server.

Never discuss you jump server configuration or protocols with anyone, even inside the company. Obscurity is not security, but there’s no reason to give an attacker any clues, either. Things like user on boarding should be done in person or over the phone, not in electronic communications. Jump server documentation should exist in paper form, not in SharePoint. If you’re going to put so many eggs into one jump server basket, then you need to treat that server like a state secret.

15 thoughts on “You Need to Rethink that “Jump Server.”

  1. Orin Thomas (@orinthomas)

    Security layers are even better. Jump servers. Privileged Access workstations (device guard & cred guard). Privileged Access Management. Just Enough Administration 😉

    Like the auto-rebuild idea. Could work for PAWs as well.

  2. Jay Adams

    Definitely great concerns to think about. I’ve seen many a customer with a jump box and vendors all over it with full access, sharing credentials… ugh.

    Using an access proxy like System Frontier would greatly reduce the attack surface and add logging for everything that’s being done. (See “shameless plug”). Unlike JIT/JEA, it *is* easy to setup and is even easier for the delegated users.

  3. Joerg Jaspert

    And two things you are missing: Regularly let (trusted) outsiders test the server. That is, pentest, try to hack it. If they find something, listen and fix it. And run your system in read-only mode. If you cant install/copy anything on it, it gets way harder to get your hackers toolkit onto the machine.

    We have that at work for ours. They havent yet found anything (if you lock down users ability to copy things onto it and then also deny execution of binaries not pre-approved, it gets hard to break in), but they do it regularly. (This doesn’t even need secure boot things, Linux for examples has the grsecurity patchset).

  4. Marco Cancedda

    lock your firewall to your IP and explain how the hell this setup would be insecure…you’re another time waster, aren’t you 🙂

  5. Marcho

    It’s also worth mentioning that pen testing is not the end-all, be-all of security. While very effective for known vulnerabilities, 0-day exploits and anything undisclosed will remain undiscovered. That’s why I put a bit more weight on the other points you guys are making.

  6. Decrypt512

    Good article for the most part. Guess some of this is just 101 nowadays. You go through a RDS box out there to “jump” through you without a doubt need MFA, as well as IP whitelisting. Creating the automated system to dump user VHD/UHD and rinse and repeat every night is pretty quick as well.

  7. Oliver

    I didn’t know the term “jump server” but that’s what I use to connect to my different environments : a server with 2 open ports: ssh fort the initial set up/local admin, and openvpn to access to the internal network.

    I will definitely harden the vpn servers to prevent a social hack of a ssh/openvpn account, I’ll try with 2FA/yubikey ASAP. But it would be interesting to know how to work without any “jump server”:
    1) How administrators can use their configuration management tools without remote access.
    2) How can people and CI can deploy something on a remote environment without a VPN.
    3) How can we easily share and manage openvpn credentials if they are rebuild frequently ? I can build a vpn server with certs for everybody in 5 minutes with ansible, but should I recreate certs each time ? Should I build a system to distribute client certs everyday ? How to secure the VPN renewal/cert distribution ?

  8. Pingback: Newsletter: December 23, 2016 | Notes from MWhite

  9. Pingback: cron.weekly issue #60: Debian, Vim, Gitlab, Jenkins, Piwik, Nginx, MySQL & more!

  10. Christoph Wegener

    Great article. Very much to the point! Not sure where you got the info about AppLocker not sending E-mails though. That’s certainly incorrect.

  11. Sumit Khanna

    When I was at Defcon earlier this year, there was a talk about a new open source project called BloodHound. It’s meant for Windows environment based companies, but it essentially creates a privilege graph showing access to machines in a domain:

    The talk was really impressive. The trouble in most Active Directory based companies is that you have groups within groups within groups and you can usually assign people permissions they don’t need without even realizing it. You just need to compromise on of their machines to start jumping around. Their talk/powershell script shows just how easy it is for an unprivileged user to map out all those relations.

    In the talk they even found permissions two companies away across two trusted domains for one of their clients.

Comments are closed