13 min read

Your employer runs SSL MiTM attacks on you

TL;DR companies are buying appliances that run SSL MiTM (Man in The Middle) attacks against their users, decrypting sessions on the fly without the user’s knowledge. You should find out if this is happening to you.

As a self described privacy advocate, I consider myself pretty cognizant of when I might be under some sort of network surveillance; I know what to look for, and enjoy understanding ways to avoid it (often by not visiting certain sites from certain networks), but one day I hit something that surprised me. It was enough to make me take notes so that others could see if they were under a similar setup. One day, while working for $company, after a scheduled ‘network upgrade’ I was unable to interact with github.com, a site that I had been using daily up to that point. The error I’d get was…

✗ git pull
fatal: unable to access 'https://github.com/xxxxxxxx/xxxxxxxx.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

This made me suspicious, so I followed the trail to learn what was happening. Obviously we know that Github’s cert is valid (tech news sites would be blowing up about it if it weren’t), so were we being blocked from accessing Github, or was it something with the SSL certificate that was throwing an error?


Digging into into it more, I asked curl to tell me what it knew about the certificate:

✗ curl -v -k https://github.com/xxxxxxxx/xxxxxxxx.git/
* Hostname was NOT found in DNS cache
*   Trying
* Connected to github.com ( port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-GCM-SHA384
* Server certificate:
*    subject: businessCategory=Private Organization;;; serialNumber=5157550; street=548 4th Street; postalCode=94107; C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=github.com
*    start date: 2014-04-08 00:00:00 GMT
*    expire date: 2016-04-12 12:00:00 GMT
*    issuer: C=xx; ST=xxxxxx; L=xxxxxxxx; O=xxxxxx; CN=WSG_SubCA; emailAddress=xxxxxx@xxxxxxx
*    SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET /xxxxxxxx/xxxxxxxx.git/ HTTP/1.1
> User-Agent: curl/7.35.0
> Host: github.com
> Accept: */*
< HTTP/1.1 301 Moved Permanently
* Server GitHub.com is not blacklisted
< Server: GitHub.com
< Date: Wed, 25 Jun 2014 15:10:14 GMT
< Content-Type: text/html
< Location: https://github.com/xxxxxxxx/xxxxxxxx/
< Vary: Accept-Encoding
< X-Served-By: bc4c952d089501afbfc8f7ff525da31c
< Content-Length: 178
< Age: 0
< Via: HTTPS/1.1 localhost.localdomain
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
* Connection #0 to host github.com left intact

So it looks like SSL is being intercepted by the gateway, did this mean something (or someone?) was doing SSL decryption on outgoing traffic from our network? That would be… interesting, and possibly criminal if it was a malicious attacker, so I continued digging. Using Chrome to browse Github told me more:

github.com has asked Chrome to block any certificates with errors, but the certificate that Chrome received during this connection attempt has an error.

 Error type: HSTS failure
 Subject: github.com
 Issuer: WSG_SubCA
 Public key hashes:

So HSTS was the issue, it was not allowing this certificate. I have HSTS set on my webservers and, not surprisingly, Github is smart enough to do that too. OWASP defines HSTS as:

HTTP Strict Transport Security (HSTS) is an opt-in security enhancement that is specified by a web application through the use of a special response header. Once a supported browser receives this header that browser will prevent any communications from being sent over HTTP to the specified domain and will instead send all communications over HTTPS. It also prevents HTTPS click through prompts on browsers. The specification has been released and published end of 2012 as RFC 6797 (HTTP Strict Transport Security (HSTS)) by the IETF.

So what happens is that you after you hit an SSL protected site, it sends a response header to verify the SSL has not been intercepted or otherwise messed with during the round trip. This was the issue blocking my requests to Github. Meanwhile the HSTS error was pretty revealing, you see it above and in this screenshot:

Verified by WSG_SubCA huh?
The issuer of the SSL certificate isn't something that we'd expect (GitHub, Inc. or github.com), it's __WSG_SubCA__. What the hell is that? So with some quick sleuthing on [DuckDuckGo](https://duckduckgo.com) I found that we were dealing with a [Websense Content Gateway](https://www.websense.com/content/web-security-gateway-features.aspx). With impressive (sounding) whitepapers talking about, _"Next Generation Secure Gateways"_ with, _"Advanced Classification Engine"_ it was clear we were dealing with an _Enterprise Solution(tm)_, so why was it stripping the right SSL and giving me a Websense CA instead? Reading up on the Websense Content Gateway (WCG), I found the helpful [release notes](https://www.websense.com/content/support/library/web/v78/wcg_release_notes/78rn_intro.aspx#689076) that spelled out how it worked its magic, under _"New in Websense Content Gateway v7.8.1"_ specifically:
  • Support for SSL (HTTPS) decryption, analysis, and re-encryption has been re-engineered for version 7.8.1” and inserting our own generated SSL and injecting that into the session. SSL support is now part of the core Content Gateway proxy.

Other interesting features of the product include:

  • Pre-installs the trusted Root CA tree used by Mozilla Firefox
  • Can be used with explicit and transparent proxy deployments
  • Can be configured to accept HTTPS traffic from multiple inbound ports (client to proxy)
  • Provides certificate handling and management features
  • Includes configurable certificate validation features
  • Includes robust decryption bypass features to simplify the process of maintaining compliance with government and corporate laws and policies, and to whitelist trusted sites and blacklist untrusted sites
  • Allows for customizable messages to users when there is a certificate verification failure or connection error

How does this work?

So now we know what’s happening, but how is it happening? Open source security community OWASP has a great image showing what’s going on:

What is a MiTM attack? (OWASP)
A simplified example, just ignore that the interceptor here is labed as an Attacker
So the SSL, encrypted traffic is intercepted, it is decrypted so the network appliance can scan it and then it is passed along to the target website to get the requested information for the user. I first learned that this was possible in 2009 at [DEF CON](https://defcon.org) 19 when Moxie Marlinspike released his software [SSLStrip](http://www.thoughtcrime.org/software/sslstrip/) to take advantage of this, but it always seemed something a bad hacker would use, not something shopped to enterprise companies as a security tool. But if this was happening for all users on the network, they would be complaining about the warning and eventally discover the bogus __WSG_SubCA__, why aren't they getting tipped off? Simple, the network admins control what happens when a system logs into the network via [Active Directory](https://en.wikipedia.org/wiki/Active_Directory), the Microsoft directory service. It's likely that they drop the __WSG_SubCA__ on to the user's system, and tell the browser that it's "trusted" so it won't give an error when it uses it. See in the following image where it says, "Root Certificates Installed in Browser":
Illustration from iboss.com

EDIT in the comments, Steven has corrected me on this, ”…in the deploying a trusted root certificate to PCs is not a function of Active Directory Certificate Services. That is done through Group Policy, ADCS is for operating an enterprise certificate authority." So while I do mention the ADCS below, I thought it was a function of AD, and not it’s own thing. Thanks for the correction, my Windows admin knowledge is lacking, so I appreciate the comment.

I found an article that talks about how Active Directory Certificate Services works, and it can distribute any SSL certificate to a user’s systems. WSG even has a simple web GUI to help you create your own Internal Root CA. Problem solved on your end! But if this is what’s happening it has terrible privacy and security implications, if all user’s SSL traffic was decrypted and captured things like banking, heath information and other PII (Personally Identifiable Information) that must remain secret. But when I hit sites like Paypal and Mastercard I noticed, I was getting the correct SSL certificates, so there was obviously a whitelist in play here. This was also verified by reviewing the release notes it talks about how the “SSL decryption bypass” works:

Web Security Gateway: SSL decryption bypass To support organizations using SSL Manager to manage encrypted traf fic, and who do not want to decrypt HTTPS sessions that users establish with sens itive sites, such as person al banking or health provider sites, administrators can now specify cate gories of sites that will bypass SSL decryption. A list of hostnames or IP addresses for which SSL decryption is not performed can also be maintained. These capabilities are configured on the Scanning > SSL Decryption Bypass page in TRITON - Web Security (Web Security Manager).

One of their Webinar from early 2010 they state that their SCIP (Secure Content Inspection Proxy) was setup by a 3rd party:

  • Websense has contracted with Microdasys to provide the HTTPS termination process used by WCG.

and that once the HTTPS/SSL is terminated:

  • The data is decrypted, forwarded to WCG.
  • WCG applies inspection rules (ie. WSE, Analytics)
  • encrypted and sent to its destination.
  • Applies security policies to all encrypted inbound and outbound Internet traffic.
  • Data can be decrypted and hence inspected for malware.

As you can see, they give positive reasons for doing this (malware is bad). Later, in a 2013 presentation they describe more the software can do:

  • An extended.log example without SSL decryption: - [12/Jan/2013:15:43:51 -0000] "CONNECT www.cia.gov:443/ HTTP/1.0" 200 127 200 0 0 0 383 127 542 76 0
  • An extended.log example with SSL decryption: - [12/Jan/2013:15:43:51 -0000] "CONNECT www.cia.gov:443/ HTTP/1.0" 200 127 200 0 0 0
383 127 542 76 0 - - [12/Jan/2013:15:43:52 -0000] "GET http://www.cia.gov/javascript/register_function-
cachekey1018.js HTTP/1.1" 200 52663 200 52663 0 0 840 297 829 287 0 - - [12/Jan/2013:15:43:53 -0000] "GET http://www.cia.gov/css/IEFixes.css HTTP/1.1" 200
3642 200 3642 0 0 810 279 799 269 0 - - [12/Jan/2013:15:43:53 -0000] "GET http://www.cia.gov/css/ciatheme-index.css
HTTP/1.1" 200 10657 200 10657 0 0 818 281 807 271 0 - - [12/Jan/2013:15:43:53 -0000] "GET http://www.cia.gov/css/base-cachekey6837.css
HTTP/1.1" 200 59571 200 59571 0 0 821 281 810 271 0 - - [12/Jan/2013:15:43:53 -0000] "GET http://www.cia.gov/javascript/javascript.js HTTP/1.1" 200 6092 200 6092 0 0 820 296 809 286 0

I found it intersting they used the URL for the CIA as an example. Later more justification that no one in security could say no to:

  • SSL decryption improves adherence to organisational policies
  • Access control
  • Monitoring
  • Reporting

But then they seem to be reaching a bit with…

  • Improves organisational and user security
  • Reduced risk of interception

Whatever, marketing can sell anything, but my favorite discovery from Websense was that they specically stated on page 5 of the presentation, that “SSL decryption uses the Man In The Middle (MitM) method”

SSL decryption uses the Man In The Middle (MitM) method
One picture is worth 1000s words, eh?
So even in their (clearly early, 2013) promotion materials, they describe this functionality for what it is: a Man in The Middle attack -or- MiTM (I prefer to have the T capitalized, call me old fashioned).

Why didn’t I see this?

So why didn’t I see this behavior? Why did I get an error when everyone else thought it was smooth sailing? I’m one of those geeky-weirdos that has had a long history of not wanting to work on Windows machines, in general. Usually at a new gig I’ll get the client’s permission to either dual-boot Windows and Linux, just wipe the machine and run Linux, or use my own personal laptop running Linux. There are always, “but you need windows to xxxxxx” reasons why this can’t work, but I’m always able to demonstrate work arounds to cover these issues. These days the easiest way to accomplish this now is to run a virtual desktop infrastructure (VDI) that mimics, or directly logs into a Windows server via remote desktop protocol. Sometimes this is launched via a ‘plugin’ in the browser so just visiting a URL will work, other times running commandline apps like rdesktop or similar gets you a window right on the RDP terminal. Point is, since I’m not running Windows I’m not getting the system administering setups, so I didn’t recieve their fake Root CA to validate sites for me blindly. I know I’m an edge case here, but sometimes that’s what you need to learn about what’s really going on.

What is Websense?

Ironically I knew about Websense from “back in the day”, in another life, when I actually administrated a Websense box, but at that time we used it to tally pageviews and provide other website analytics (which is a different kind of privacy violation you should know my stance on). Looking online I found that over the years that Websense has done quite well for themselves, “Texas-based Websense was founded in 1994 and has over 1,500 employees located around the world” when, earlier this year it was bought by “U.S. weapons and electronic equipment manufacturer Raytheon announced Monday it will buy the cybersecurity company Websense for $1.9 billion”. By the way, when I go to DEF CON each year, the ‘cybersecurity’ folks I’m meet don’t look like the Raytheon Cyber executives:

Raytheon Cyber Programs Vice President Jeffrey Snyder (right) speaks at the Reuters Cybersecurity Summit in Washington May 13, 2013. Reuters/Hyungwon Kang
These are not the cybersecurity experts you're looking for!

… but I digress.

End state

Look, I know what I’m going to hear, and I totally get it; this is the $company network and resources, they are ultimately responsible for defending themselves from malicious software entering the network and for misuse that their users might, knowingly or unknowingly, do. The point is that the users don’t know that their privacy and security is being thrown out out the window in the guise of security. Yes they might have seen a EULA or other notice stating that the company owns everything, but for years we have trained users that if they use HTTPS and they see that ‘Lock’ that they have end-to-end encryption, they have privacy, but they don’t know that this can be completely false if the company is using an appliance that circumvents that functionality and silently installs a hand-made, bogus Root CA to lie to them. Note that the action of blocking or censoring web content on a national level is considered evil, you have to admit that doing the same (maybe more so) and not implictly informing users how invasive their monitioring is, is a bit evil as well.

So, does your employer run SSL MiTM attacks on you? Take a closer look at the SSL certs you get when you load an HTTPS page and see what you can find.

Further reading…