About IPSec VPN Negotiations

The devices at either end of an IPSec VPN tunnel are IPSec peers. To build the VPN tunnel, IPSec peers exchange a series of messages about encryption and authentication, and attempt to agree on many different parameters. This process is known as VPN negotiations. One device in the negotiation sequence is the initiator and the other device is the responder.

VPN negotiations happen in two distinct phases: Phase 1 and Phase 2.

Phase 1

The main purpose of Phase 1 is to set up a secure encrypted channel through which the two peers can negotiate Phase 2. When Phase 1 finishes successfully, the peers quickly move on to Phase 2 negotiations. If Phase 1 fails, the devices cannot begin Phase 2.

Phase 2

The purpose of Phase 2 negotiations is for the two peers to agree on a set of parameters that define what traffic can go through the VPN, and how to encrypt and authenticate the traffic. This agreement is called a Security Association.

The Phase 1 and Phase 2 configurations must match for the devices on either end of the tunnel.

Phase 1 Negotiations

Some of the features described in this section are only available to participants in the WatchGuard Beta program. If a feature described in this section is not available in your version of Fireware, it is a beta-only feature.

In Phase 1 negotiations, the two peers exchange credentials. The devices identify each other and negotiate to find a common set of Phase 1 settings to use. When Phase 1 negotiations are completed, the two peers have a Phase 1 Security Association (SA). This SA is valid for only a certain amount of time. After the Phase 1 SA expires, if the two peers must complete Phase 2 negotiations again, they must also negotiate Phase 1 again.

Phase 1 negotiations include these steps:

  1. The devices exchange credentials.

The credentials can be a certificate or a pre-shared key. Both gateway endpoints must use the same credential method. If one peer uses a pre-shared key, the other peer must also use a pre-shared key, and the keys must match. If one peer uses a certificate, the other peer must also use a certificate.

In Fireware v12.6 or higher, you can specify a hex-based pre-shared key for BOVPNs and BOVPN virtual interfaces. If one peer uses a hex-based pre-shared key, the other peer must use the same hex-based pre-shared key.

  1. The devices identify each other.

Each device provides a Phase 1 identifier, which can be an IP address, domain name, domain information, or an X500 name. The VPN configuration on each peer contains the Phase 1 identifier of the local and the remote device, and the configurations must match.

  1. For IKEv1, the peers decide whether to use Main Mode or Aggressive Mode.

For IKEv1, phase 1 negotiations can use one of two different modes: Main Mode or Aggressive Mode. The device that starts the IKE negotiations (the initiator) sends either a Main Mode proposal or an Aggressive Mode proposal. The responder can reject the proposal if it is not configured to use that mode. Aggressive Mode is less secure but faster than Main Mode.

When you use Aggressive mode, the number of exchanges between two endpoints is fewer than it would be if you used Main Mode, and the exchange relies mainly on the ID types used in the exchange by both appliances. Aggressive Mode does not ensure the identity of the peer. Main Mode ensures the identity of both peers, but can only be used if both sides have a static IP address.

  1. The peers agree on Phase 1 parameters.
    • Whether to use NAT traversal — this is always enabled for IKEv2
    • Whether to send IKE keep-alive messages (only for IKEv1 between Fireboxes)
    • Whether to use Dead Peer Detection (RFC 3706) — this is always enabled for IKEv2.
  2. The peers agree on Phase 1 Transform settings.

Transform settings include a set of authentication and encryption parameters, and the maximum amount of time for the Phase 1 SA. The settings in the Phase 1 transform must exactly match a Phase 1 transform on the IKE peer, or IKE negotiations fail.

The items you can set in the transform are:

  • Authentication — The type of authentication (SHA1, SHA2, or MD5). SHA2 is stronger than either SHA1 or MD5.
  • Encryption — The type of encryption algorithm (DES, 3DES, or AES). AES is the most secure. In Fireware v12.2 or higher, you can specify an AES-GCM option if you select IKEv2.
  • SA Life — The amount of time until the Phase 1 Security Association expires.
  • Key Group — The Diffie-Hellman key group.

For IKEv2, the Phase 1 transform settings are shared for all BOVPN gateways that have a remote gateway with a dynamic IP address.

SHA-2 is not supported on XTM 21, 22, 23, 505, 510, 520, 530, 515, 525, 535, 545, 810, 820, 830, 1050, and 2050 devices. The hardware cryptographic acceleration in those models does not support SHA-2. All other models support SHA-2.

Phase 2 Negotiations

After the two IPSec peers complete Phase 1 negotiations, Phase 2 negotiations begin. The purpose of Phase 2 negotiations is to establish the Phase 2 SA (sometimes called the IPSec SA). The IPSec SA is a set of traffic specifications that tell the device what traffic to send over the VPN, and how to encrypt and authenticate that traffic. In Phase 2 negotiations, the two peers agree on a set of communication parameters. When you configure the BOVPN tunnel in Policy Manager or in Fireware Web UI, you specify the Phase 2 parameters.

Because the peers use the Phase 1 SA to secure the Phase 2 negotiations, and you define the Phase 1 SA settings in the BOVPN Gateway settings, you must specify the gateway to use for each tunnel.

Phase 2 negotiations include these steps:

  1. The peers use the Phase 1 SA to secure Phase 2 negotiations.

Phase 2 negotiations can only begin after Phase 1 SA has been established.

  1. The peers exchange Phase 2 identifiers (IDs).

Phase 2 IDs are always sent as a pair in a Phase 2 proposal: one indicates which IP addresses behind the local device can send traffic over the VPN, and the other indicates which IP addresses behind the remote device can send traffic over the VPN. This is also known as a tunnel route. You can specify the Phase 2 IDs for the local and remote peer as a host IP address, a network IP address, or an IP address range.

  1. The peers agree on whether to use Perfect Forward Secrecy (PFS).

PFS specifies how Phase 2 keys are derived. When PFS is selected, both IKE peers must use PFS, or Phase 2 rekeys fail. PFS guarantees that if an encryption key used to protect the data transmission is compromised, an attacker can get access to only the data protected by that key, not additional keys. If the peers agree to use PFS, they must also agree on the Diffie-Hellman key group to use for PFS.

  1. The peers agree on a Phase 2 proposal.

The Phase 2 proposal includes the IP addresses that can send traffic over the tunnel, and a group of encryption and authentication parameters. Fireware XTM sends these parameters in a Phase 2 proposal. The proposal includes the algorithm to use to authenticate data, the algorithm to use to encrypt data, and how often to make new Phase 2 encryption keys.

The items you can set in a Phase 2 proposal include:


For a manual BOVPN, you can select the type of protocol to use: Authentication Header (AH) or Encapsulating Security Payload (ESP). ESP provides authentication and encryption of the data. AH provides authentication without encryption. We recommend you select ESP. Managed BOVPN and Mobile VPN with IPSec always use ESP.


Authentication makes sure that the information received is exactly the same as the information sent. You can use SHA1 or MD5 as the algorithm the peers use to authenticate IKE messages from each other. SHA1 is more secure.


Encryption keeps the data confidential. You can select DES, 3DES, or AES. AES is the most secure. In Fireware v12.2 or higher, you can specify an AES-GCM option.

Force Key Expiration

To make sure Phase 2 encryption keys change periodically, always enable key expiration. The longer a Phase 2 encryption key is in use, the more data an attacker can collect to use to mount an attack on the key.

See Also

How IPSec VPNs Work

Give Us Feedback  ●   Get Support  ●   All Product Documentation  ●   Technical Search