Internet Key Exchange (IKE) is a protocol used to set up a IPSec Security Associations (SAs) security attributes like encryption key, encryption algorithm, and mode, between IPSec peers. Internet Key Exchange allows IPSec peers to dynamically exchange keys and negotiate IPSec Security Associations (SAs). Using Internet Key Exchange (IKE), IPSec Security Associations (SAs) can be dynamically established and removed at a negotiated time period.

Internet Key Exchange (IKE) is an IETF protocol and it has two versions, an old version IKEv1 (RFC 2409, RFC 4109) and a relatively new version, IKEv2 (RFC 5996, RFC 7296 and RFC 7427).

Internet Key Exchange is a hybrid protocol made from Oakley, SKEME (A Versatile Secure Key Exchange Mechanism for Internet) and ISAKMP (Internet Security Association and Key Management Protocol) protocols. ISAKMP protocol is a framework for exchanging encryption keys and security association payloads.

IKE uses UDP, Port Number 500.

Internet Key Exchange Version 1 (IKEv1)

The operation IKEv1 can be broken down into two phases. 1) Phase 1 (IKE SA Negotiation) and 2) Phase 2 (IPSec SA Negotiation). IKEv1 Phase 1 SA negotiation is for protecting IKE. IKEv1 Phase 2 SA negotiation is for protecting IPSec (real user traffic).

IKEv1 Phase 1 negotiation can happen in two modes, either using Main Mode or using Aggressive Mode. IKEv1 Phase 1 Main mode has three pairs of messages (total six messages) between IPSec peers. IKE Phase 1 Aggressive Mode has only three message exchanges. The purpose of IKEv1 Phase 1 is to establish IKE SA.

IKEv1 Phase 2 (Quick Mode) has only three messages. The purpose of IKEv1 Phase 2 is to establish IPSec SA.

Phase 1 is used to negotiate the parameters and key material required to establish IKE Security Association (SA) between two IPSec peers. The Security Associations (SAs) negotiated in Phase 1 is then used to protect future IKE communication.

Following explanation is based on the assumption that the peers are using Pre-Shared Key for authentication.

IKEv1 Phase 1 Main Mode

IKEv1 Phase 1 Main Mode - Message 1: IKEv1 Main mode first message pair consists of the IKEv1 Security Association proposals. The Initiator (device which initiates IPSec) proposes policies by sending one or more Security Association proposals. IKEv1 Main Mode Message 1 contains IKE header, SA payload, Proposal payload, and Transform payload. IKE use different types of "Payloads" to share information about common Security Associations and Keys. Payload has a header and other information which is useful to DOI. IKE can be DOI stands for Domain of Interpretation, in this case, IPSec.

SA payload is used to specify that this particular ISAKMP exchange is for IPSec negotiation. IKE/ISAKMP is a generic protocol which can be used to negotiate different protocols. Therefore, SA payload contains a Domain of Interpretation (DOI), which is used to mention this IKE/ISAKMP negotiation is for IPSec.

Proposal payload contains a proposal number, Protocol ID, SPI size, number of transforms and SPI.

Transform payload contains transform number, transform ID, and IKE SA attributes like Authentication (Pre-shared keys or Digital Certificates), Hashing Algorithms (MD5 or SHA1), Encryption (DES, 3DES or AES), Tunnel lifetime unit (Secs), Tunnel lifetime in seconds, Diffie-Hellman Groups.

Initiator and Responder must calculate a value, called as cookie. Cookie value is used to protect IPSec peers against DoS and Re-Play attacks. RFC suggests a method for creating the cookie by performing a fast hash (Example MD5) over the IP Source and Destination Address, the UDP Source and Destination Ports and a locally generated secret random value. The field "Initiator SPI", marked in below capture is the Cookie value generated by Initiator.Responder Cookie value is kept as empty, becuase this is the very first message.

IKEv1 Phase 1 Main Mode - Message 2: IKEv1 Main Mode Message 2 is the response from the Responder to the packet sent from the initiator. The purpose of Message 2 is to inform Initiator the SA attributes agreed upon. Most of the fields are the same as in the packet sent by the initiator. Only one proposal payload and transform payload is there in Message 2, which is the agreed proposal and transform payload. Also note that both the cookie values are filled.

IKEv1 Phase 1 Main Mode - Message 3: The direction of third message is from the Initiator to the Responder. This message contains Diffie-Hellman Key Exchange Payload and Nonce payload, from Initiator.

IKEv1 Phase 1 Main Mode - Message 4: The direction of fourth message is from the Responder to the Initiator. This message contains Diffie-Hellman Key Exchange Payload and Nonce payload, from Responder. A Nonce is a very large random number used in IKE. IKE Nounce random number is also used to calculate keying material.

Please note that the first four IKEv1 Phase 1 Main mode messages are not encrypted.

IKEv1 Phase 1 Main Mode - Message 5: Both IPSec peers calculate Diffie-Hellman shared secret. Three keys are generated by both peers for authentication and encryption. 5th message contains Identification payload and Hash Payload, from Initiator. Identification payload and Hash Payload are used for identitification and authentication. Identification payload and Hash Payload payloads are sent encrypted, in IKEv1 Phase 1 Main Mode.

IKEv1 Phase 1 Main Mode - Message 6: 6th message contains Identification payload and Hash Payload. Identification payload and Hash Payload are used for identitification and authentication from Responder. Identification payload and Hash Payload payloads are sent encrypted in IKEv1 Phase 1 Main Mode.

IKEv1 Phase 1 Aggressive Mode

IKEv1 Phase 1, when in Aggressive mode, uses only three messages to establish IKE SA. The first two messages in Aggressive mode exchange include almost everything required to form IKE SA. IKEv1 Phase1 Aggressive Mode is quicker than Main Mode, but endpoint identities are exchanged in Clear-Text. When comparing Main Mode and Aggressive Mode, Main mode is considered more secure than Aggressive Mode, because the Identification payload is encrypted in Main Mode.

IKEv1 Phase 1 Aggressive Mode - Message 1: In IKEv1 Phase1 Aggressive Mode, all the necessary information required to generate the Diffie-Hellman shared secret is exchanged in the first two messages between peers. The first message sent from the Initiator includes SA payload, Proposal payload, and Transform payload, similar to Main Mode. The main difference in Aggressive Mode is that the first message includes Diffie-Hellman Key Exchange payload and the Nonce payload. Identification payload is also added in the first message. Note that the Identification payload is sent as Clear-Text, not encrypted.

IKEv1 Phase 1 Aggressive Mode - Message 2: Now the Responder can generate the Diffie-Hellman shared secret. The Responder generates the Diffie-Hellman shared secret. Responder generates the Hash also for Authentication purposes. Similar to Message 2 in IKEv1 Phase1 Main Mode, Message 2 in IKEv1 Phase1 Aggressive Mode also contains the agreed proposal and transform pair from the Responder. It also contains Key Exchange Payload, Nounce Payload, Identity Payload and an Authentication Hash from the Responder.

IKEv1 Phase 1 Aggressive Mode - Message 3: Now the Initiator can generate the Diffie-Hellman shared secret. The Initiator generates the Diffie-Hellman shared secret. Initiator generates the Hash also for Authentication purposes. The Hash payload is sent as encrypted.

Phase 1 can be negotiated using Main Mode (6 messages) or Aggressive Mode (3 messages). Once the Phase 1 messages are exchanged succesfully, the IKE SA is established. Once an IKE SA (using Main Mode or Aggressive Mode) is established, it can be used to generate IPSec SAs, in Phase 2 negotiation. Now the peers will proceed to exchange Phase 2 (Quick Mode) messages for negotiating IPSec SA.

IKEv1 Phase 2 - Quick Mode

IKEv2 Phase 2 negotiation is done in only one mode, that is Quick Mode. IKEv1 Phase 2 (Quick Mode) consists of 3 message exchanges. Ofcourse, the message exchanges in Phase 2 (Quick Mode) are protected by encryption and authentication, using the keys derived in the Phase 1.

If Perfect Forward Secrecy (PFS) is enabled, new shared keys should be generated for use. The Diffie-Hellman Key generation is carried out again using new Nonces exchanged between peers.

Since there is no meaning in showing encrypted capture screen shots, I am not attaching any Wireshark capture screen shots for Quick Mode.

In Quick Mode, following parameters are negotiated for the IPSec SA.

• Encryption algorithm (DES, 3DES, AES)
• Hashing algorithm (MD5, SHA-1, SHA-2)
• Encapsulation protocol (AH or ESP)
• SA lifetime (time in seconds or data transfer in kilobytes)
• Mode (Tunnel, Transport)

Once IKEv1 Phase 2 (Quick Mode) negotiation is complete, a unidirectional SA is generated by each peer. Each peer will generate at least two SAs. One in inbound direction and in outbound direction.