Which VPN protocols aren't safe anymore?

Figuring out the difference between VPN types can be confusing – especially if you're new to the world of VPNs. For anyone unfamiliar with VPN protocols, the comparisons can seem like strings of random letters and numbers that mean nothing.

So, to help clear things up, I'll cut through all the algorithms and jargon to give you the real differences between the protocols that today's best VPNs use to keep your personal data secure – and those that just don't make the cut anymore.

What is a VPN Protocol?

A VPN protocol is a set of rules that determine how data is transmitted securely between your device and the VPN server. Protocols are necessary for the VPN server to understand the data you send through, when you connect to it, and subsequently understand how to process it and respond.

All VPN protocols create a secure connection (or try to) but they don't all do it the same way. It's the encryption methods, authentication processes, and data transmission techniques that differentiate protocols.

So, to explain how some protocols are unsecure, I'll need to discuss the difference between asymmetric encryption and symmetric encryption first.

Most of your data transfer is carried out using symmetric encryption. This method uses the same key for both encryption and decryption – and this makes it easy to use without significant processing overhead. However, the downside to symmetric encryption is that both parties need to know the same encryption key.

This presents a fundamental problem: how do you securely share the symmetric key with a party you've never met before without a third party intercepting it?

Well, this is where asymmetric encryption enters the picture. With asymmetric encryption, the key is split into an encryption ("public") and decryption ("private") key. You provide your public key to the party you want to have a secure transmission with. They use your public key to encrypt the information and then you decrypt it with your private key. To send data back, you encrypt with their public key, and so on.

This process is more complex and computationally expensive, so asymmetric encryption is typically only used to transmit the initial symmetric encryption key. Once the symmetric key has been securely transferred, symmetric encryption is used for the rest of the communication.

Battle-tested protocols like OpenVPN and WireGuard navigate this key-sharing process without obvious flaws, making it easy to set up new encrypted tunnels without third-party interception. However, at least one protocol does have fundamental flaws in its key transmission process, making it susceptible to hacking.

So, let's take a closer look at it – PPTP.

Is PPTP secure?

PPTP, which stands for Point-to-Point Tunneling Protocol, is Microsoft's in-house VPN protocol developed for Windows.

It has a lot of issues, though, and way too many to mention here in depth. I'll go over the main problem, however, which is that the asymmetric key used to negotiate the session isn't secure.

PPTP uses Microsoft's MS-CHAPv2 for key negotiation, which is based on Microsoft’s existing authentication standards. Essentially, if you've already got a network of Microsoft machines all talking to each other, you can plug in a PPTP VPN without making any changes to how they verify identity.

PPTP piggybacks on this existing infrastructure to generate a session key. Then, the key is used to create the encryption needed for streaming data using the RC4 encryption scheme between the VPN clients.

The asymmetric key used to negotiate PPTP sessions isn't secure.

However, there is a fundamental flaw in how MS-CHAPv2 works – it's easily cracked using brute force. In fact, depending on the computational resources you have at hand, you can break the session key within a day.

PPTP also doesn't provide Perfect Forward Secrecy by default, which means that once the session key is broken, it can be used to decrypt past sessions, too. There’s also an issue with how RC4 works. Basically, you can't verify if the data sent via RC4 has been intercepted and changed mid-transmission, so if you're using PPTP, a hacker could read everything you send over a VPN and change the data at will.

While there are slightly safer implementations of PPTP that ditch MS-CHAPv2 in favor of the public-key cryptography I mentioned earlier, there's no real reason to use PPTP unless you’re trying to support legacy systems.

As for commercial VPNs? I'd give any VPN provider that still offers PPTP a wide berth. Even Microsoft suggests that you ditch PPTP and use an alternate VPN solution instead.

Is IPSec secure?

The argument against PPTP is pretty solid – but the reasons for my doubts about IPSec and L2TP are harder to nail down.

To give you a little background, L2TP builds an encrypted tunnel and IPSec is the security suite that provides authentication and encryption for the content sent through the tunnel. When combined, they form a fully encrypted VPN protocol.

So, when we talk about VPNs, IPSec and L2TP are usually discussed together, even if we only refer to one or the other.

It's uncertain exactly how IPSec/L2TP is unsecure. However, leaks provided by Edward Snowden in 2014 revealed that the NSA and GCHQ have a vested interest in decrypting as much VPN traffic as possible to support their SIGINT programs. As part of this program, it's suggested that the NSA has to decrypt a significant portion of the world's internet traffic transported over IPSec/L2TP.

This could mean that the IPSec suite itself contains a vulnerability, or has been deliberately weakened to make it easier to decrypt, it could also mean that the NSA has compromised key internet infrastructure in other ways unrelated to the actual protocol that allows them to decrypt traffic.

Without actual technical implementations to correlate its claims, it's difficult to say exactly why IPSec is insecure.

Nevertheless, the leaked documents are enough to make me steer clear of IPSec too – at least as it relates to IKEv1. Thankfully, IKEv2 has replaced IKEv1 in most implementations built around IPSec, and it seems like a far more secure protocol.

Which VPN protocols are safe?

OpenVPN is considered the gold standard of VPN security. It’s open-source, trusted by hundreds of thousands of organizations the world over, and built on top of the OpenSSL library which provides encryption used by basically every website ever.

If you're on the hunt for a secure VPN and notice that a potential pick uses OpenVPN - it's a solid option. When configured properly, it's virtually bulletproof.

Then, there's WireGuard. It's almost as secure as OpenVPN but, by default, there's one noticeable issue. WireGuard stores your IP when doing Network Address Translation. So, if a hacker or employee broke into the VPN server, they'd be able to link your traffic to your IP address. Luckily, it can be fixed pretty easily by VPN providers.

OpenVPN is considered the gold standard of VPN security.

To address this issue, top-tier VPN providers that offer WireGuard use something called a "double-NAT" approach to anonymize your traffic while it's being assigned an address inside their servers. For example, NordVPN used this approach when building its proprietary WireGuard-based NordLynx protocol.

Speaking of, when it comes to provider-specific VPN protocols, it's hard to gauge them on a case-by-case basis. However, there are a few golden rules to keep in mind.

One of the most important, for example, is to check that the protocol has been subject to a third-party audit. A VPN provider that has built its own protocol is attempting to replicate the work of potentially hundreds of software, network, and cryptography engineers. In my opinion, any VPN worth its salt will be comfortable letting a third-party firm inspect its products, down to the nitty-gritty, and check for potential vulnerabilities.

Audits indicate transparent and privacy-focused services. So, where possible, look for providers that undergo these audits on a regular basis, like ExpressVPN and NordVPN.

VPN protocols and quantum computing

Quantum computing poses a potential threat to a lot of encryption methods – including those used by VPN protocols. Traditional encryption algorithms rely on mathematical problems that are hard for classical computers to solve but could be more easily tackled by quantum computers.

For example, quantum computers could theoretically break RSA encryption, one of the most common asymmetrical key exchange methods. The actual impact depends on the development of quantum computing technology and its practical capabilities.

For now, quantum computers are not yet advanced enough to break these encryption methods on a large scale.

Researchers have already developed several algorithms designed to be secure against quantum attacks in response to the potential threat. Post-quantum algorithms are now baked into most encryption libraries – and top-tier VPN providers like NordVPN and ExpressVPN are already integrating these into their VPN encryption protocols.