Skip to content

TLS Internals

This document goes deeper into TLS internals than the TLS fundamentals document, covering the record Layer architecture, detailed handshake message formats for TLS 1.3, cipher suite construction, key Exchange mechanisms, and common implementation pitfalls. This is the material you need to understand When debugging TLS connections, configuring servers, or evaluating cryptographic strength.

TLS is structured as a layered protocol with four sub-protocols operating over a reliable transport (TCP):

+-----------------------------------+
| Application Data |
+-----------------------------------+
| Handshake Protocol |
+-----------------------------------+
| Change Cipher Spec |
+-----------------------------------+
| Alert Protocol |
+-----------------------------------+
| Record Layer |
+-----------------------------------+
| TCP |
+-----------------------------------+

The TLS record layer fragments application data (and handshake messages) into records. Each record Has:

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Content Type (8) | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Version (16) |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Length (16) |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Fragment (variable) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Content types:

TypeValueDescription
CHANGE_CIPHER_SPEC20Deprecated in TLS 1.3 (replaced by KeyUpdate)
ALERT21Error or warning notifications
HANDSHAKE22Handshake protocol messages
APPLICATION_DATA23Encrypted application data

The handshake protocol is responsible for authentication, key exchange, and negotiation of Cryptographic parameters. Handshake messages are carried inside TLS records with content type 22.

Alert messages convey errors and state changes:

LevelDescriptionCommon Alerts
Warning (1)Non-fatal; connection continuesclose_notify, no_certificate, bad_certificate
Fatal (2)Connection must be terminatedhandshake_failure, decode_error, illegal_parameter

In TLS 1.2, this protocol signals the transition to encrypted communication. In TLS 1.3, it is Deprecated. Key changes are signaled within the handshake protocol itself.

FeatureTLS 1.2TLS 1.3Reason
RenegotiationSupportedRemovedComplex, caused RC4 injection attacks
CompressionSupportedRemovedCRIME attack (compression oracle)
Static RSA key exchangeSupportedRemovedNo forward secrecy
Non-AEAD ciphersSupportedRemovedCBC ciphers vulnerable to padding oracles
Custom DHE groupsSupportedRemovedWeak groups (e.g., export-grade)
SHA-1 in signaturesSupportedRemovedSHA-1 is cryptographically weak
MD5 in signaturesSupportedRemovedMD5 is broken
FeatureDescription
0-RTT dataSend application data in the first flight (repeat connections)
Signature algorithmsExplicit negotiation of hash+signature pairs (RFC 8446 Section 4.2.3)
Key scheduleDerived key hierarchy using HKDF (RFC 5869)
Post-handshake authServer can request client certificate after the handshake
Encrypted Server HelloServer Hello is encrypted (hides server identity from observers)
KeyUpdateIn-band key rotation without renegotiation

TLS 1.2 cipher suites are complex strings like TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256. TLS 1.3 Cipher suites only specify the AEAD algorithm:

TLS 1.3 Cipher SuiteAEAD AlgorithmHash (HKDF)
TLS_AES_128_GCM_SHA256AES-128-GCMSHA-256
TLS_AES_256_GCM_SHA384AES-256-GCMSHA-384
TLS_CHACHA20_POLY1305_SHA256ChaCha20-Poly1305SHA-256
TLS_AES_128_CCM_SHA256AES-128-CCMSHA-256
TLS_AES_128_CCM_8_SHA256AES-128-CCM-8SHA-256

The key exchange algorithm is no longer part of the cipher suite. It is negotiated separately via The supported_groups extension.

Client Server
| |
|--- ClientHello -------------------------------->| Flight 1
| supported_versions, supported_groups, |
| signature_algorithms, key_share |
| |
|<-- ServerHello ---------------------------------| Flight 2
| selected_version, selected_group, |
| key_share |
|<-- EncryptedExtensions ------------------------|
|<-- Certificate ---------------------------------|
|<-- CertificateVerify ---------------------------|
|<-- Finished ------------------------------------|
| |
|--- Finished ----------------------------------->| Flight 3
| |
|==== Application Data =========================|

The ClientHello carries the client”s capabilities and parameters:

FieldDescription
legacy_version0x0303 (TLS 1.2) for compatibility with middleboxes
random32 bytes of random (used in key derivation)
legacy_session_idSession ID for compatibility (TLS 1.3 uses PSK)
cipher_suitesList of supported TLS 1.3 cipher suites
legacy_compression_methods[0x00] (no compression)
extensionssupported_versions, supported_groups, key_share,
signature_algorithms, psk_key_exchange_modes,
server_name (SNI), etc.
FieldDescription
legacy_version0x0303 (always, even for TLS 1.3)
random32 bytes of random
legacy_session_id_echoEcho of client’s session_id
cipher_suiteSelected cipher suite
legacy_compression_method0x00
extensionssupported_version (TLS 1.3), key_share,
pre_shared_key (if PSK selected)

After the ServerHello, all subsequent handshake messages are encrypted. EncryptedExtensions carries Server-side configuration that does not affect the cryptographic parameters:

  • server_name indication (whether SNI was used)
  • max_fragment_length (negotiate smaller records)
  • application_layer_protocol_negotiation (ALPN)
  • early_data (whether 0-RTT is accepted)

The server sends its certificate chain. In TLS 1.3, the Certificate message is sent encrypted. The Certificate chain includes:

  1. Leaf certificate: The server’s end-entity certificate
  2. Intermediate certificates: One or more intermediate CA certificates
  3. Root certificate: NOT included (the client must already trust it)

This message proves that the server holds the private key corresponding to the certificate’s public Key. It contains a digital signature over a transcript hash of all handshake messages so far.

Signature = Sign(private_key, Hash("TLS 1.3, server CertificateVerify" || 0x20...0x20 || transcript_hash))

The 0x20...0x20 is 64 bytes of spaces (0x20), a context string that binds the signature to TLS 1.3 Specifically.

Both sides send a Finished message, which contains a verify_data value derived from the handshake Transcript:

verify_data = HMAC(finished_key, Hash(transcript))

The Finished message is the first message encrypted with the newly derived traffic keys. If the Verify_data does not match, the handshake has been tampered with and the connection is terminated.

ECDHE (Elliptic Curve Diffie-Hellman Ephemeral)

Section titled “ECDHE (Elliptic Curve Diffie-Hellman Ephemeral)”

The most widely used key exchange in TLS 1.3. Both sides generate an ephemeral (temporary) key pair On an elliptic curve, exchange public keys, and derive a shared secret.

Client generates: (priv_c, pub_c)
Server generates: (priv_s, pub_s)
Shared secret = ECDH(priv_c, pub_s) = ECDH(priv_s, pub_c) = x-coordinate of (priv_c * pub_s)

Supported curves (RFC 8446):

CurveKey SizeSecurity Level
X25519256 bits128 bits
secp256r1 (P-256)256 bits128 bits
secp384r1 (P-384)384 bits192 bits
secp521r1 (P-521)521 bits256 bits

X25519 is the recommended default. It is faster than NIST curves, has simpler implementation (fewer Edge cases), and uses a constant-time algorithm that is resistant to timing attacks.

DHE (Finite Field Diffie-Hellman Ephemeral)

Section titled “DHE (Finite Field Diffie-Hellman Ephemeral)”

Traditional Diffie-Hellman over a finite field. Slower than ECDHE for equivalent security levels. Supported groups:

Group (ffdhe)Prime SizeSecurity Level
ffdhe20482048 bits112 bits
ffdhe30723072 bits128 bits
ffdhe40964096 bits150 bits
ffdhe61446144 bits175 bits
ffdhe81928192 bits200+ bits

TLS 1.3 supports PSK-based key exchange, which can be used alone or combined with (EC)DHE (called “PSK with (EC)DHE” or “psk_dhe_ke”).

PSK ModeForward SecrecyUse Case
PSK onlyNoIoT devices, resumption tickets
PSK + (EC)DHEYesRecommended for resumption (security)

PSKs are established either externally (configured on both sides) or via a previous TLS handshake (session resumption via NewSessionTicket).

The client indicates which PSK modes it supports in the psk_key_exchange_modes extension:

  • psk_ke: PSK-only key establishment (no forward secrecy)
  • psk_dhe_ke: PSK combined with (EC)DHE (forward secrecy maintained)

TLS 1.3 implementations should prefer psk_dhe_ke for session resumption. This provides forward Secrecy even for resumed sessions. If the PSK is compromised, past traffic remains secure because The (EC)DHE exchange was ephemeral.

AEAD (Authenticated Encryption with Associated Data) provides both confidentiality and integrity in A single operation. TLS 1.3 requires AEAD ciphers exclusively.

AES-GCM (Galois/Counter Mode):

  • AES-128-GCM: 128-bit key, 96-bit nonce, 128-bit authentication tag
  • AES-256-GCM: 256-bit key, 96-bit nonce, 128-bit authentication tag
  • Hardware-accelerated on most modern CPUs (AES-NI instruction set)
  • The most widely deployed AEAD cipher

ChaCha20-Poly1305:

  • 256-bit key, 96-bit nonce, 128-bit authentication tag
  • Software-optimized design (no special hardware needed)
  • Preferred on mobile devices and ARM platforms without AES-NI
  • Designed by Daniel J. Bernstein

AES-CCM:

  • AES-128-CCM: 128-bit key, CBC-MAC mode
  • AES-128-CCM-8: Same but with truncated 64-bit tag (faster but weaker)
  • Required for compatibility with constrained IoT devices (RFC 8446 mandates support for at least one CCM cipher)
  • Slower than GCM

TLS 1.3 uses HKDF (HMAC-based Extract-and-Expand Key Derivation Function, RFC 5869) for all key Derivation. The key schedule is:

0-RTT
|
Early Secret = HKDF-Extract(PSK or 0)
|
Handshake Secret = HKDF-Extract(DHE shared secret, Early Secret)
|
Master Secret = HKDF-Extract(DHE shared secret, Handshake Secret)
|
Application Traffic Secret 0
Application Traffic Secret 1 (after KeyUpdate)
...

Each secret is expanded into multiple keys:

Traffic Keys = {
client_write_key,
server_write_key,
client_write_iv,
server_write_iv
}

The per-record nonce is derived from the IV and a sequence number:

nonce = IV XOR (sequence_number << 64)

The sequence number is a 64-bit counter that increments for each record. Since the sequence number Is included in the nonce, every record has a unique nonce, even with the same IV.

When the server presents a certificate chain, the client builds a verification path:

  1. Parse the leaf certificate and extract the issuer’s distinguished name
  2. Find the issuer in the chain (or in the client’s trust store)
  3. Repeat until a trusted root certificate is reached
  4. Verify signatures at each step (each certificate is signed by its issuer)

For each certificate in the chain:

  1. Verify the signature algorithm is acceptable (not MD5, not SHA-1)
  2. Verify the certificate is within its validity period (not_before to not_after)
  3. Verify the certificate has not been revoked (OCSP or CRL)
  4. Verify the certificate’s intended purpose (serverAuth for TLS)
  5. Verify the certificate’s constraints (name constraints, path length)

OCSP (Online Certificate Status Protocol, RFC 6960):

The client sends a query to the CA’s OCSP responder asking whether a specific certificate is Revoked. The responder returns “good”, “revoked”, or “unknown”.

Terminal window
# Check certificate revocation with openssl
openssl ocsp -issuer intermediate.pem -cert server.pem \
-url http://ocsp.example.com/ -resp_text

OCSP Stapling (RFC 6066):

The server periodically obtains an OCSP response from the CA and “staples” it to the TLS handshake. The client does not need to contact the CA directly, improving performance and privacy.

Terminal window
# Test OCSP stapling with openssl
openssl s_client -connect example.com:443 -status -servername example.com

CRL (Certificate Revocation List):

The CA publishes a list of revoked certificate serial numbers. The client downloads and checks the List. CRLs can be large and are not updated frequently, making them less practical for real-time Revocation checking.

CAA (Certification Authority Authorization, RFC 6844)

Section titled “CAA (Certification Authority Authorization, RFC 6844)”

A DNS record that specifies which CAs are authorized to issue certificates for a domain:

example.com. IN CAA 0 issue "letsencrypt.org"
example.com. IN CAA 0 issuewild "*.example.com" "letsencrypt.org"

TLS records have a maximum size (configurable, default 16KB). Application data larger than this is Fragmented into multiple records. The receiver reassembles the fragments.

In TLS 1.3, the maximum record size is negotiated via the max_fragment_length extension:

ValueMax Record Size
12^9 (512 bytes)
22^10 (1024 bytes)
32^11 (2048 bytes)
42^12 (4096 bytes)

Smaller records reduce latency (the receiver can process data sooner) but increase overhead (more Records = more TLS record headers and MACs).

TLS 1.2 with CBC cipher suites uses MAC-then-Encrypt (MAC the plaintext, then encrypt both). This is Vulnerable to padding oracle attacks (Lucky13, POODLE).

TLS 1.3 uses AEAD ciphers exclusively, which combine encryption and authentication in a single Operation (effectively encrypt-then-MAC). This eliminates padding oracle attacks entirely.

The client sends the server hostname in the ClientHello’s server_name extension. This allows a Single IP address to host multiple TLS-enabled websites (virtual hosting).

Terminal window
# Test SNI
openssl s_client -connect 93.184.216.1:443 -servername example.com

ALPN (Application-Layer Protocol Negotiation, RFC 7301)

Section titled “ALPN (Application-Layer Protocol Negotiation, RFC 7301)”

Negotiates the application protocol (HTTP/1.1, HTTP/2, h2c, etc.) during the TLS handshake:

Terminal window
# Test ALPN
openssl s_client -connect example.com:443 -alpn h2,http/1.1

The server encrypts the session state into a ticket and sends it to the client. On a subsequent Connection, the client presents the ticket, and the server decrypts it to resume the session without Storing state.

In TLS 1.3, session tickets are sent via the NewSessionTicket post-handshake message.

The client lists supported TLS versions in this extension. The server responds with the selected Version in the ServerHello. This extension allows TLS 1.3 to be negotiated without changing the Legacy_version field.

Contains the PSK identity and the binder value (an HMAC over the transcript up to this point, using The PSK). The binder prevents a man-in-the-middle from substituting a different PSK.

Forward secrecy (also called perfect forward secrecy, PFS) ensures that compromising the server’s Private key does not compromise past session keys. Each session uses an ephemeral key exchange, so Recording encrypted traffic and later obtaining the server’s private key does not allow decryption Of past sessions.

Without forward secrecy (static RSA key exchange), the session key is encrypted with the server’s Static RSA private key. If an attacker records the handshake and later obtains the private key (through theft, court order, or cryptanalysis), they can decrypt all past sessions.

Which Cipher Suites Provide Forward Secrecy

Section titled “Which Cipher Suites Provide Forward Secrecy”
Key ExchangeForward Secrecy
Static RSANo
ECDHEYes
DHEYes
PSK onlyNo
PSK + (EC)DHEYes