About IPSec Algorithms and Protocols

IPSec is a collection of cryptography-based services and security protocols that protect communication between devices that send traffic through an untrusted network. Because IPSec is built on a collection of widely known protocols and algorithms, you can create an IPSec VPN between your Firebox and many other devices or cloud-based endpoints that support these standard protocols.

Encryption Algorithms

Encryption algorithms protect the data so it cannot be read by a third-party while in transit. Fireware supports three encryption algorithms:

  • AES (Advanced Encryption Standard) — AES is the strongest encryption algorithm available. Fireware can use AES encryption keys of these lengths: 128, 192, or 256 bits. AES is faster than 3DES.
  • 3DES (Triple-DES) — An encryption algorithm based on DES that uses the DES cipher algorithm three times to encrypt the data. The encryption key is 168-bit. 3DES is slower than AES.
    The Sweet32 vulnerability affects 3DES.
  • DES (Data Encryption Standard) — Uses an encryption key that is 56 bits long. DES is the weakest of the three algorithms, and it is considered to be insecure.

Authentication Algorithms

Authentication algorithms verify the data integrity and authenticity of a message. Fireware supports three authentication algorithms:

HMAC-MD5 (Hash Message Authentication Code — Message Digest Algorithm 5)

MD5 produces a 128-bit (16 byte) message digest, which makes it faster than SHA1 or SHA2. This is the least secure algorithm.

HMAC-SHA1 (Hash Message Authentication Code — Secure Hash Algorithm 1)

SHA1 produces a 160-bit (20 byte) message digest. Although slower than MD5, this larger digest size makes it stronger against brute force attacks. SHA-1 is considered to be mostly insecure because of a vulnerability.

HMAC-SHA2 (Hash Message Authentication Code — Secure Hash Algorithm 2)

SHA2 is the most secure algorithm. Fireware v11.8 and higher supports three variants of SHA2 with different message digest lengths.

  • SHA2-256 — produces a 265-bit (32 byte) message digest
  • SHA2-384 — produces a 384-bit (48 byte) message digest
  • SHA2-512 — produces a 512-bit (64 byte) message digest

SHA2 is stronger than either SHA1 or MD5. We recommend that you specify a SHA2 variant.

SHA-2 is not supported on XTM 21, 22, 23, 505, 510, 520, 530, 515, 525, 535, 545, 810, 820, 830, 1050, and 2050 devices. The hardware cryptographic acceleration in those models does not support SHA-2. All other models support SHA-2.

Galois/Counter Mode (GCM)

GCM (Galois/Counter Mode) is an authenticated encryption algorithm known for its security, efficiency, and performance. Authentication and encryption occur simultaneously. If you specify AES-GCM in your BOVPN or BOVPN virtual interface configuration, you might see performance increases on Fireboxes without a hardware crypto chip. This includes Firebox T55 and T70 models.

Fireware v12.2 or higher supports AES-GCM for IPSec BOVPN and BOVPN virtual interfaces. You can specify these options:

  • AES-GCM (128-bit)
  • AES-GCM (192-bit)
  • AES-GCM (256-bit)

Phase 1

AES-GCM is supported as a Phase 1 transform for IKEv2. IKEv1 is not supported.

Phase 2

AES-GCM is supported as a Phase 2 proposal for ESP (Encapsulating Security Payload). AES-GCM is not supported for AH (Authentication Header).

AES-GCM uses an Integrity Check Value (ICV) to verify data integrity. Fireware supports a 16-byte Integrity Check Value (ICV). Other ICV lengths are not supported.

GCM is required by NSA Suite B, a cryptographic standard specified by the United States government.

For more information about AES-GCM in IPSec ESP, see RFC 4106.

AES-GCM is not supported for Mobile VPN with IPSec.

IKE Protocol

IKE (Internet Key Exchange) is a protocol used to set up security associations for IPSec. These security associations establish shared session secrets from which keys are derived for encryption of tunneled data. IKE is also used to authenticate the two IPSec peers. Fireware supports IKEv1 and IKEv2 in the BOVPN gateway or BOVPN Virtual Interface configuration.

  • IKEv1 is defined in RFC 2409.
  • IKEv2 is defined in RFC 7296.

IKEv2 requires Fireware v11.11.2 or higher.

Diffie-Hellman Key Exchange Algorithm

The Diffie-Hellman (DH) key exchange algorithm is a method used to make a shared encryption key available to two entities without an exchange of the key. The encryption key for the two devices is used as a symmetric key for encrypting data. Only the two parties involved in the DH key exchange can deduce the shared key, and the key is never sent over the wire.

A Diffie-Hellman key group is a group of integers used for the Diffie-Hellman key exchange. Fireware can use DH groups 1, 2, 5, 14, 15, 19, 20, and 21.

In Fireware v12.10 and higher, Fireware supports Diffie-Hellman Group 21.

For more information, see About Diffie-Hellman Groups.


Defined in RFC 2402, AH  (Authentication Header) is a protocol that you can use in manual BOVPN Phase 2 VPN negotiations. To provide security, AH adds authentication information to the IP datagram. Most VPN tunnels do not use AH because it does not provide encryption.


Defined in RFC 2406, ESP (Encapsulating Security Payload) provides authentication and encryption of data. ESP takes the original payload of a data packet and replaces it with encrypted data. It adds integrity checks to make sure that the data is not altered in transit, and that the data came from the proper source. We recommend that you use ESP in BOVPN Phase 2 negotiations because ESP is more secure than AH. Mobile VPN with IPSec always uses ESP.

Recommended Settings

The default BOVPN settings on the Firebox are meant for compatibility with older WatchGuard devices and third-party devices. If the peer endpoint device supports IKEv2 and stronger encryption and authentication settings, we recommend that you change default settings for greater security and performance.

For recommended settings, see Improve Branch Office VPN (BOVPN) Tunnel Availability.

See Also

How IPSec VPNs Work