VPN Basics: Encryption and Security Protocols in a VPN

Encryption is the process of encoding data so that only a computer with the right decoder will be able to read and use it.The most common forms of encryption are symmetric-key encryption or public-key encryption:

In symmetric-key encryption, all computers (or users) share the same key used to both encrypt and decrypt a message.

In public-key encryption, each computer (or user) has a public-private key pair. One computer uses its private key to encrypt a message, and another computer uses the corresponding public key to decrypt that message.

In a VPN, the computers at each end of the tunnel encrypt the data entering the tunnel and decrypt it at the other end.  A site-to-site VPN could use either Internet protocol security protocol (IPSec) or generic routing encapsulation (GRE).

Generic Routing Encapsulation (GRE) is a tunneling protocol developed by Cisco Systems that can encapsulate a wide variety of network layer protocols inside virtual point-to-point links over an Internet Protocol network.

IPSec  (Internet Protocol Security) is a protocol suite for securing Internet Protocol (IP) communications by authenticating and encrypting each IP packet of a communication session. IPSec can encrypt data between various devices, including router to router, firewall to router, desktop to router, and desktop to server.

IPSec consists of two sub-protocols which provide the instructions a VPN needs to secure its packets:

Encapsulated Security Payload (ESP) encrypts the packet's payload (the data it's transporting) with a symmetric key.

Authentication Header (AH) uses a hashing operation on the packet header to help hide certain packet information (like the sender's identity) until it gets to its destination.

Networked devices can use IPSec in one of two encryption modes. In transport mode, devices encrypt the data traveling between them. In tunnel mode, the devices build a virtual tunnel between two networks