The curious case of curl, SSL SNI and the HTTP Host header

Written by Claudio Kuenzler - 0 comments

Published on February 13th 2017 - last updated on May 29th 2019 - Listed in Internet Linux SSL TLS

---

As the digial age gets older, encryption of http traffic is becoming a new standard. Encrypted http connections will one day even be a requirement for serving web pages in Google's Chrome browser:

Eventually, we plan to label all HTTP pages as non-secure, and change the HTTP security indicator to the red triangle that we use for broken HTTPS.

Back in the dark ages of the Internet (= the 90's) every SSL listener required a dedicated IP address. But luckily nowadays there is Server Name Indication (SNI) support. SNI allows to run multiple SSL/TLS certificates on the same IP address. Much like it's been possible for a long time to serve multiple domains on the same IP address (virtual hosts). Since 2006 SNI was broadly accepted in operating systems and browsers. Before that: Your own bad luck if you still use that (it's 2017, get over it).

However not all tools are yet aware of SNI. I had to come across this myself when I wanted to check the response headers of a domain with a SNI certificate with the "curl" command:

$ curl -H "Host: testsite.example.net" https://lb.example.com -v -I
* Rebuilt URL to: https://lb.example.com/
* Hostname was NOT found in DNS cache
*   Trying 192.168.14.100...
* Connected to lb.example.com (192.168.14.100) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
*      subject: OU=Domain Control Validated; OU=Gandi Standard Wildcard SSL; CN=*.example.ch
*      start date: 2016-10-05 00:00:00 GMT
*      expire date: 2019-10-05 23:59:59 GMT
*      subjectAltName does not match lb.example.com
* SSL: no alternative certificate subject name matches target host name 'lb.example.com'
* Closing connection 0
* SSLv3, TLS alert, Client hello (1):
curl: (51) SSL: no alternative certificate subject name matches target host name 'lb.example.com'

The returned server certificate is a wildcard certificate for *.example.ch which has nothing to do with...

a) the requested HTTP Host Header testsite.example.net

b) the requested server name lb.example.com 

Of course curl then exits with a warning, that no certificate could be found for the host name. And there's actually a (more or less) simple explanation (found on http://superuser.com/questions/793600/curl-and-sni-enabled-server):

"SNI sends the hostname inside the TLS handshake (ClientHello). The server then chooses the correct certificate based on this information. Only after the TLS connection is successfully established it will send the HTTP-Request, which contains the Host header you specified."

Another but better and more technical and understandable explanation was found on the Apache wiki (https://wiki.apache.org/httpd/NameBasedSSLVHosts and https://wiki.apache.org/httpd/NameBasedSSLVHostsWithSNI).:

"Apache needs to know the name of the host in order to choose the correct certificate to setup the encryption layer. But the name of the host being requested is contained only in the HTTP request headers, which are part of the encrypted content. It is therefore not available until after the encryption is already negotiated. This means that the correct certificate cannot be selected"

"The solution is an extension to the SSL protocol called Server Name Indication (RFC 4366), which allows the client to include the requested hostname in the first message of its SSL handshake (connection setup). This allows the server to determine the correct named virtual host for the request and set the connection up accordingly from the start."

Of course this explanation goes for all web server, not only Apache.

So using the "HTTP Host Header" is not a correct way to get to the SNI certificate. One would need a curl parameter to define the "SNI Hostname". Something like this, a parameter called --sni-hostname, was actually requested in issue #607 on curl's Github repository. The issue itself was closed and as a workaround the --resolve parameter was mentioned. And this indeed does the job:

$ curl --resolve testsite.example.net:443:lb.example.com https://testsite.example.net -v -I
* Resolve testsite.example.net:443:lb.example.com found illegal!
* Rebuilt URL to: https://testsite.example.net/
* Hostname was NOT found in DNS cache
*   Trying 194.40.217.90...
* Connected to testsite.example.net (194.40.217.90) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
*      subject: OU=Domain Control Validated; OU=Gandi Standard Wildcard SSL; CN=*.example.net
*      start date: 2016-10-05 00:00:00 GMT
*      expire date: 2019-10-05 23:59:59 GMT
*      subjectAltName: testsite.example.net matched
*      issuer: C=FR; ST=Paris; L=Paris; O=Gandi; CN=Gandi Standard SSL CA 2
*      SSL certificate verify ok.
> HEAD / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: testsite.example.net
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
[...]

This tells curl to force a DNS resolving of the requested "testsite.example.net" to lb.example.com. The correct certificate is now used.

The curl command becomes rather complicated, agreed. But there's also another issue #614 which takes the SNI problem one step further by adding a new feature called the "--connect-to" parameter. The new parameter was officially added in curl version 7.49.0.

Fixed in 7.49.0 - May 18 2016

Changes:

    schannel: Add ALPN support
    SSH: support CURLINFO_FILETIME
    SSH: new CURLOPT_QUOTE command "statvfs"
    wolfssl: Add ALPN support
    http2: added --http2-prior-knowledge
    http2: added CURL_HTTP_VERSION_2_PRIOR_KNOWLEDGE
libcurl: added CURLOPT_CONNECT_TO
    curl: added --connect-to
    libcurl: added CURLOPT_TCP_FASTOPEN
    curl: added --tcp-fastopen
    curl: remove support for --ftpport, -http-request and --socks

However my curl version is still 7.35.0 and I was not in the mood to recompile it manually.

Using openssl to check the SNI certificate

Besides the workaround using curl with the --resolve parameter (and in the future using the --connect-to parameter), there's also the way to check the certificate using ... *drumrolls* ... openssl!
Yes, openssl itself can of course also be used to check the SNI certificate by using the -servername parameter:

$ openssl s_client -connect lb.example.com:443 -servername testsite.example.net
CONNECTED(00000003)
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/OU=Domain Control Validated/OU=Gandi Standard Wildcard SSL/CN=*.example.net
   i:/C=FR/ST=Paris/L=Paris/O=Gandi/CN=Gandi Standard SSL CA 2
 1 s:/C=FR/ST=Paris/L=Paris/O=Gandi/CN=Gandi Standard SSL CA 2
   i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
 2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
[...]

As you can see, the correct wildcard certificate *.example.net was returned in the certificate chain. 

Reminder to myself: Don't let the curl output fool you.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.