
IPsec: What Is IPsec? How IPsec VPNs Work, Tunnel vs Transport Mode
IPsec stands for Internet Protocol Security. It is a set of protocols that authenticates and encrypts IP packets, so data travelling across an untrusted network, such as the internet, stays private and tamper-proof. In plain terms, IPsec turns an open road into a sealed, armoured tunnel: anyone watching the traffic can see that something is moving, but they cannot read it or change it without being detected.
IPsec is the technology behind most site-to-site VPNs and many remote-access VPNs. This guide explains what IPsec is, how it works step by step, the difference between its two protocols AH and ESP, tunnel mode versus transport mode, and how IPsec compares with SSL/TLS VPNs, plus a practical setup checklist most guides leave out.
What Is IPsec?
IPsec is a framework defined by the Internet Engineering Task Force (IETF), not a single protocol. It operates at the network layer (Layer 3) of the OSI model, which means it protects all traffic from an IP address, whatever application produced it. Unlike a browser-based VPN that only protects web traffic, IPsec protects every packet leaving a device or network, including email, file sharing, voice and database traffic.
IPsec provides three core services. Confidentiality comes from encryption, so the contents of packets cannot be read. Integrity comes from hashing, so any change to a packet in transit is detected. Authentication proves that the sender is who it claims to be, and anti-replay protection stops attackers from capturing and resending old packets. Together, these services let two parties who have never met build a trusted connection over a public network.
How Does IPsec Work? Step by Step
Before any protected data flows, the two endpoints negotiate a security association (SA), an agreement about the algorithms, keys and parameters they will use. This negotiation is handled by the Internet Key Exchange (IKE) protocol, usually IKEv2 in modern deployments, and happens in two phases.
In phase 1, the endpoints authenticate each other, using pre-shared keys or digital certificates, and agree on an encrypted channel for further negotiation. In phase 2, they agree on the IPsec SAs for the actual data traffic, including which protocol to use and which encryption and hashing algorithms to apply. Once the SAs are established, each outgoing packet is encrypted and authenticated, sent across the network, and decrypted and verified by the receiver. When the SA expires, the endpoints renegotiate it, so keys are refreshed regularly.
Think of it like a phone call. Phase 1 is checking the caller’s identity and agreeing to speak in code. Phase 2 is agreeing on the code itself. The conversation that follows is protected, and if the call drops, a new one is set up the same way.
AH vs ESP: The Two IPsec Protocols
IPsec uses two main protocols, and the difference between them is one of the most tested topics in networking exams.
| Feature | AH (Authentication Header) | ESP (Encapsulating Security Payload) |
|---|---|---|
| Encrypts data | No | Yes |
| Authenticates data | Yes | Yes |
| Protects the IP header | Yes, most of it | No, only the payload |
| Protocol number | 51 | 50 |
| Typical use today | Rare | Almost always |
AH authenticates the whole packet, including most of the IP header, but does not encrypt anything, so anyone can still read the contents. ESP encrypts the payload and authenticates it, which is what most organisations actually need. Because AH does not work reliably through network address translation (NAT), which changes IP headers, ESP is used in nearly all modern IPsec deployments, often combined with NAT traversal (NAT-T).
Tunnel Mode vs Transport Mode
IPsec can operate in two modes, and choosing the wrong one is a common mistake.
| Feature | Transport mode | Tunnel mode |
|---|---|---|
| What is protected | Only the IP payload | The entire original IP packet |
| IP header | Original header stays visible | A new outer header is added |
| Typical use | Host to host, end to end | Gateway to gateway, site to site |
| Hides internal addresses | No | Yes |
| Overhead | Lower | Higher, due to the extra header |
In transport mode, two computers protect traffic directly between themselves. The original IP header remains visible, so routers can see where the packet is going, but the data inside is encrypted. In tunnel mode, the entire original packet is wrapped inside a new packet with a new header, creating a tunnel between two gateways, such as two office firewalls. The internal addresses are hidden, and any traffic between the two networks can pass through the tunnel. Most IPsec VPNs use tunnel mode with ESP, because that is the combination that connects whole networks securely.
IPsec VPNs: Site-to-Site and Remote Access
A site-to-site IPsec VPN connects two networks, for example a head office and a branch office, or an office and a cloud virtual network. Each side has a gateway that encrypts traffic leaving its network and decrypts traffic arriving. Users on either side work normally, unaware that their traffic crosses the public internet in encrypted form. SCADA systems at remote industrial sites are often connected this way, so control traffic never travels in the clear.
A remote-access IPsec VPN connects an individual device, such as a laptop or phone, to a corporate network. The user runs VPN client software, authenticates, and receives an internal IP address. From then on, their traffic is tunnelled to the corporate gateway. Modern remote-access deployments often use IKEv2 with certificate or multi-factor authentication, and many organisations are now combining IPsec with zero-trust controls, so that a VPN connection alone is not enough to reach sensitive systems.
IPsec vs SSL/TLS VPN: Which Should You Use?
IPsec is often compared with SSL/TLS VPNs, the browser-based VPNs many people use to reach web applications. The two solve different problems, and the table below is the comparison most guides skip.
| Feature | IPsec VPN | SSL/TLS VPN |
|---|---|---|
| Layer | Network layer (Layer 3) | Transport / application layer |
| Protects | All IP traffic | Usually web and app traffic |
| Client software | Usually required | Often just a browser |
| Best for | Site-to-site, always-on access | Remote users, web apps |
| Firewall traversal | Needs UDP 500/4500 open | Usually works over HTTPS (443) |
| Complexity | Higher to set up | Lower to deploy |
Choose IPsec when you need to connect whole networks, protect non-web traffic, or run an always-on connection between sites. Choose an SSL/TLS VPN when remote users mainly need web applications and you want the simplest client experience. Many organisations use both: IPsec between sites and cloud networks, and SSL/TLS VPN for individual remote workers.
Benefits and Limitations of IPsec
IPsec’s strengths are its maturity, its protection of all traffic, and its wide support in routers, firewalls and operating systems. It is a standard, so equipment from different vendors can usually work together, and it can be combined with strong authentication such as certificates and multi-factor authentication.
Its limitations are real, too. Configuration is complex, and a small mismatch in algorithms, keys or lifetimes between the two ends will stop the tunnel from forming. IPsec adds overhead, which can reduce throughput and increase latency, especially on small packets. It does not protect against malware, phishing or vulnerabilities on the endpoints themselves; it only protects traffic in transit. And because it operates at the network layer, it is harder to inspect than TLS traffic, which is why monitoring and logging at the gateways matter.
IPsec Setup Checklist: Ports, Protocols and Troubleshooting
A working IPsec deployment needs more than the right algorithms. Use this checklist when setting one up, and again when a tunnel fails.
- Allow the right traffic: UDP port 500 for IKE, UDP port 4500 for NAT-T, and IP protocol 50 for ESP and 51 for AH through firewalls.
- Use IKEv2: it is faster, more reliable and more secure than IKEv1, with built-in support for NAT-T and dead peer detection.
- Choose strong algorithms: AES-256 for encryption and SHA-256 or stronger for integrity; avoid DES, 3DES, MD5 and SHA-1.
- Match both ends: encryption, hashing, Diffie-Hellman group, lifetime and pre-shared key or certificate must agree on both sides.
- Plan for NAT: enable NAT-T where a device sits behind NAT, and confirm ESP traffic is not being blocked.
- Monitor the tunnel: log SA establishment and failure, alert on repeated failures, and check throughput and latency regularly.
- Map threats: use the MITRE ATT&CK framework to understand how attackers try to bypass network protections, and test your VPN against those techniques.
When a tunnel will not come up, check the basics first: connectivity between the gateways, matching phase 1 and phase 2 settings, correct pre-shared keys, time synchronisation for certificates, and firewall rules for ports 500 and 4500. Most IPsec failures are configuration mismatches, not product bugs.
Frequently Asked Questions
What is IPsec in simple terms?
IPsec is a set of protocols that encrypts and authenticates internet traffic at the network layer, so data sent over the public internet stays private and cannot be tampered with. It is the technology behind most site-to-site VPNs.
What is the difference between IPsec tunnel mode and transport mode?
Transport mode encrypts only the payload of a packet, leaving the original IP header visible, and is used host to host. Tunnel mode encrypts the entire original packet inside a new packet, hiding internal addresses, and is used gateway to gateway.
What is AH and ESP in IPsec?
AH (Authentication Header) authenticates packets but does not encrypt them. ESP (Encapsulating Security Payload) encrypts and authenticates the payload. ESP is used in almost all modern IPsec deployments.
Does IPsec use TCP or UDP?
IPsec uses UDP port 500 for IKE negotiation and UDP port 4500 for NAT traversal, plus IP protocol 50 for ESP and 51 for AH. It does not use TCP for the protected traffic itself.
Is IPsec better than SSL VPN?
Neither is universally better. IPsec is better for connecting whole networks and protecting all traffic, while SSL/TLS VPNs are simpler for remote users who mainly need web applications. Many organisations use both.
Conclusion
IPsec is the quiet workhorse of network security: a standard, mature framework that encrypts and authenticates traffic at the network layer, so whole networks can communicate safely over the public internet. Understand AH and ESP, tunnel and transport mode, and the IKE negotiation that sets it all up, and you will understand not just what IPsec is, but when to use it and how to keep it running. Choose strong algorithms, match both ends carefully, and monitor the tunnel, and IPsec will do its job without your users ever noticing it is there.



[…] your mail matters, because losing the device can mean losing the mail. Protect the connection too: IPsec tunnels and VPNs add a layer of protection when you collect mail over untrusted networks. Finally, […]
[…] guide to IPsec: what it is and how it works explains the protocol most site-to-site VPNs use, and our guide to site-to-site VPN covers the […]
[…] behind an address is similar to understanding the protocol behind a connection: our guide to IPsec explains how one common protocol protects traffic in […]
[…] over IP networks, and the same principles that protect other traffic apply here. For example, IPsec protects network traffic at the IP layer, and it is a useful layer of defence for devices that send […]