TMG - HTTPS filtering vs. HTTPS Inspection

The following seems to create some confusion when administrators start of with TMG.

URL filtering and HTTPS Inspection (HTTPSi) are mutually exclusive features that can be run alone or together.  The one does not rely on the other.

URL Filtering
This feature that was introduced with TMG, for many of us it was a very welcome alternative to Websense.  It works by creating a Deny rule for the  URL categories that you specify.  You can have multiple rules like this to apply different levels of restriction for various user groups.

This filters sites based on their URL.  This will filter or block HTTP as well as HTTPS.

You can verify this by doing a "Query for URL category"  looking up a URL on HTTP and HTTPS return the same URL category and the rule therefore applies to both.

A feature added with SP1 was that you now also have the option of blocking sites for non-primary  categorizations.

You do not need certificates to get web filtering to works for you.  It works on simply denying or granting access to a URL, it does not attempt to open and reseal SSL tunnels.


HTTPS Inspection
To understand what this does you first need to get an idea of how TMG handles HTTP traffic.  TMG is a application layer firewall, so it is able to manipulate HTTP in many different ways.  As a proxy, the traffic is passed through the Web proxy Application Filter.  This mean TMG "looks into" the data and can apply it's restriction etc to it.

With HTTPS you have an SSL tunnel that starts at the web server and goes all the way through to the client.  Effectively TMG can now not manipulate the traffic since it does not pass through the Web Proxy Application Filter.  This allows any traffic to pass through the tunnel.

You have two choices.  You can have TMG simple verify the validity of the certificates on behalf of the clients.  This prevents users form ignoring the certificate warning the browsers pops up.

You still don't need certificates at this point. Since it is just verifying certificates before allowing or blocking access.

Secondly you have the option to verify and inspect traffic.

This enables the Web proxy application filter for HTTPS.  This means you have the check and control as you do for HTTP traffic.  TMG will now establish the primary tunnel from the web server to TMG, it will then inspect the traffic, then create a second tunnel between TMG and the client.  Because TMG needs to create the secondary tunnel it needs a certificate to sign it with.

TMG will generate this certificate for you and publish it to Active Directory.  Clients on the domain will then have the TMG as a trusted Root Certificate Authority.

Conclusion
The many different web access features in TMG allows you to have various levels of restriction , control, visibility and compression.  These features are independently applied and can be used in various combinations depending on the needs.

 

RDS Load Balanced Farm with RDCB using Windows NLB on physical and Hyper-V

In the article  Building a RDS Load Balanced Farm with a RD Connection Broker I cover how to build and configure an environment using DNS round robin to achieve the load balancing.

Round robin is an effective way to share the load across nodes.  It has one problem though, it is not fault tolerant.  If one of the nodes fail, request may still be directed to a node that is not responsive.  Also adding and removing nodes can only be done at DNS replication speed.   To get away from this limitation you can implement Network Load Balancing.  It has the advantages of detecting if a node has diesdand then only serves from the live nodes.  Machines can also be added and removed form the NLB without needing DNS changes. 

In this article I will step through configuring the Windows Network Load Balancing (NLB) for Hyper-V and physical environments.

This replaces the round robin section of TASK 4 from Building a RDS Load Balanced Farm with a RD Connection Broker

Prepare Hyper-V for NLB
This only needs to be done if you are using a virtualised environement.
Your RDS host machines will be bound in a NLB, this forces the MAC address to be spoofed.  This by default is not enable on Hyper-V

From the Hyper-V Manager

• Shut down and Power down the RDS host machines.
• Right Click the RDS VM and select Settings
• Select the Network adapter
• Check Enable spoofing of MAC addresses

Prepare Windows Network Load balancing
This needs to be done on every member server that will participate in the NLB

Add additional IPs
If you configure the host with a Unicast NLB you need to add an additional network adapter and assign it another "Management" IP address.  All the NLB IPs must be on the same subnet

You can also use this to manage your RDS since a load balanced farm address will get bounced around.

Install Network Load balancing

From the server manager

• Click Features
• Add Feature
• Add the Network Load Balancing Feature
• Next to install
• Close to Finish

Configure NLB Cluster
Once all the nodes have been configured you can create and add to the cluster

Create the cluster

• Open Administrative Tools
• Click Network Load Balancing Manager
• From the File Menu click Cluster
• New
• Specify the local host name
• Connect
• Select the NLB adapter
• The dedicated IP address should be the NLB adapter's IP
• Next
• Add
• Specify the IP you want to use for NLB (The virtual IP)
• Next
• Specify the FQDN for the NLB IP you created
• Select Unicast
• Next
• Finish

Add Nodes to the cluster
Once the cluster has been created you can now add the additional nodes that you prepared in the steps above
From the Network Load Balancing Manager

• Expand the Cluster you have created
• Select the Cluster name
• Right Click Add Host To Cluster
• Specify the name of the additional node
• Connect
• Select the NLB Interface
• Accept the dedicated IP address
• Next
• Finish

The new node will now be added and once the status is "Converged" the NLB is up and running.

Add DNS record for the NLB IP
Unlike round robin you will now only create a single DNS entry pointing to your NLB virtual IP

To finish up continue with  Building a RDS Load Balanced Farm with a RD Connection Broker.

Fixing Account Lockout - finding the Domain controller

In the articles below I covered some info on how to trace back the offending login machine.  But in an environment with multiple domain controllers you don't always know against which domain controller the invalid login attempts are happening.

http://fixmyitsystem.com/2011/04/scripts-to-see-where-your-account-is.html
http://fixmyitsystem.com/2011/02/how-to-find-out-why-your-account-keeps.html

One of out technical guy showed me the Account Lockout Status resource kit tool.  It is old but still works in a Win 2008 R2 domain.  http://www.microsoft.com/downloads/en/details.aspx?FamilyID=D1A5ED1D-CD55-4829-A189-99515B0E90F7

It works very simply.  Run the tool as a domain admin or specify a domain admin when specified a target.  From the File menu click "Select Target" specify the username that keeps getting locked out.

It will now query all the reachable domain controllers and check for the account status.  For me the best column to look at is Last Bad Pwd, this will show where the incorrect authentication request occurred.

Once you have the domain controller you can run the scripts or check the event logs as described in the above articles.

In practice this is the same as enumerating the current log on server for the machine which you would do from a command prompt with:

echo %logonserver%

Proxy testing and troubleshooting using Firefox and Firebug

It is often difficult to figure out what exactly is happening to your web traffic especially if you are trying to troubleshoot your proxy or reverse proxy servers.

I have been using HTTP Watch Pro for doing for a few years now.  It is however a paid for product, http://www.httpwatch.com

There is an alternative to this that works nearly as well for the stuff I check and called it is Firebug, and it is a add-on for Firefox. https://addons.mozilla.org/en-US/firefox/addon/firebug/.  The product can be described as an HTTP sniffer, that sits inside the browser.

So install the add-on and then turn on the Net section

Select the All tab and load up a page.  You should see loads of page elements loading.  If you expend one of these you can have a look at the Headers.  There is loads of information in here and they vary depending on the content / MIME type.

For checking that the traffic is flowing though your proxy look at the Via field, this should be the name of your proxy server.  If you are using web chaining you will see both proxies listed. (Very cool)

Spending some time in working out how this works and what the information means can be a big benefit and it is a skill sets that comes in surprisingly handy.

TMG HTTPS Inspection Part IV - Troubleshooting

VPN web traffic unexpectedly gets HTTPSi
When VPN client access internal web applications they will be going through the HTTPSi process.  This is only true for traffic that does not require a proxy as per the WPAD.  Traffic here will then be treated the same, being inspected and being validated.  The same restriction will apply.

Since you are far more likely to see self signed certificates on the internal network, this can cause a problem.  To resolve this issue I put in an exlusion in for the VPN address pool range.

The indicator to this problem was and error message in Firefox.

"SSL received a record that exceeded the maximum permissible length"

This does not affect HTTPSi when outbound HTTPS sessions are established through a TMG proxy




FireFox - This Connection is Untrusted
Firefox does not use the system certificate store, so certificates being published in Active directory may be installed on the client machine but, Firefox will still not trust it.  To resolve the issue the TMG root signing certificate need to be installed into the Firefox certificate store.

Step 1 Export the TMG Signing certificate.

• From the MMC add the Certificates Plug In
• Expand the Trusted Root Certification Authorities
• Expand Certificates
• Locate your TMG signing certificate
• Right Click  - All Tasks - Export
• Follow the Wizard and save the file as a Base-64 encoded .cer file

Step 2 Import TMG certificate into the Firefox certificate store

• Navigate to the FireFox options page
• Click on the Advanced Option
• Select the Encryption tab
• Click view certificates
• Select the Authorities tab
• Click Import
• Select the exported certificate from step 1

Step 3 Verify
Navigate to HTTPS site and validate that the certificate is signed by your TMG certificate authority  and that it is trusted.

Enterprise PKI
If you are Finding that your TMG signing certificate is not trusted on your environement you need to check to see that it is a Trusted Caertificate Authority in your AD Sites and Services.  The easiest way of checking this is the Run the Enterprise PKI snap-in from the mmc

• Right Click the Enterprise PKI select Manage AD containers
• Select the Certificates Authorities Container
• You should now see the Trusted certificate Authorities for your environement
• If the TMG certificate is listed there, open it and check the properties
• If there are displayed issues such as yellow exclamations and red crosses it probably means AD needs to replicate properly throughout. This can take a few hours depending on your environment.

If the certificate is not even lists you will have to start the Certificate issue process again from within TMG.  Ensure that you do this as an Enterprise Admin.



Additional issues / problems
I will be adding to this as I find or become aware of additional issues.  Add a comment or drop me a mail if you have any issues we can look at.

TMG HTTPS Inspection Part III - Enabling and Testing

Sequence
There are a few things to test,  and you would want to make sue they are all working before you proceed

It is slightly tricky to test this since there is no simple way to limit your testing to a user or machine since it works only on exclusion.  I would also suggest testing against new / different sites every time to avoid certificate caching getting in the way.  Also, do not test again any of the default Destination Exceptions sites.


Certificate Validation
This setting forces TMG to verify the certificates that it encounter to make sure that they are valid and trusted.

• From the HTTPS Outbound Inspection Page under the General tab
• Check Enable HTTPS Inspection
• Select the option "Do not inspect traffic but validate site certificates. Block HTTPS traffic if certificate is not valid"
• Switch to the Certificate Validation tab
• "Block Expired Certificates" and "Block server certificates that are not yet valid" are simple and is executed by TMG.  
• "Check for server certificate revocation" involves a lookup to the certificate CA.

Test first with the top two checked, if all is working fine then enable the revocation check and test again.  .

If everything is working correctly you can proceed with the next step. Un-check all three validation options and proceed to checking inspection

HTTPS Inspection
To test each bit individually make sure that all three validation option are disabled.  What we will be testing now is to check that TMG can successfully inspect the HTTPS traffic, and secondly that generating and signing certificates succeed.

• From the HTTPS Outbound Inspection Page under the General tab
• Check Enable HTTPS Inspection
• Select Inspect traffic and validate certificates 
• From the Client Notification tab check "Notify users that their HTTPS traffic is being inspected."     

I would suggest testing from a client machine with the Forefront client installed.  This will pop up messages from the TMG client when inspection is occurring.

Browse to a HTTPS testing site.
The TMG client should generate a popup displaying that inspection is occurring

The site should successfully open
If you now check the certificate details for the site you should see the following:






The certificate should be valid (no red cross on the icon)

The Issued to field should be the name of the site you are visiting (Indicated  in green)

The Issued by field should be the name of your TMG CA (Indicated in RED)



If this test is also successfully you should now be able to re-enable the certificate validation checks.

Test this again to ensure everything is working properly.   There are a few thing to be aware of. Check the "Things to consider when planning" section from : http://fixmyitsystem.com/2011/06/tmg-https-inspection-part-i-planning.html

TMG HTTPS Inspection Part II - Preparing and publishing TMG signing certificate

Why a certificate is needed
HTTPS means certificates and trusted Certificate Auhtorities.  When enabling HTTPS inspection (HTTPSi) you will essentially ask TMG to break the SSL tunnel, inspect the traffic, and then create another tunnel to the client. That second tunnel will be signed by the TMG CA.  To be able to do this without certificate warnings for every single HTTPS site, you will need to configure your TMG to be a Trusted Root Certificate Authority

How to generate the new signing certificate

You can use either PKI generated certificate or use a TMG generated certificate for this.  I would suggest using the TMG cert option.

From the TMG management console, Web Access Policy

• Click HTTPS Inspection
• Choose "Use FTMG to generate a certificate"
• Click on Generate
• Specify a "friendly" issuer name
• Set the expiration or leave a Never
• Optionally specify a Issuer statement
• Click Generate Certificate Now

You will now be able to view the generated certificate


You will note that the issued by name and the issued to fields are the same,  hence the term self signed certificate.

All subsequent certificates will have the same issued by information but the issued to site will be the site being accessed.

To make sure that the certificates are from a trusted certificate authority we will now need to publish this certificate to Active Directory so that it can be replicated out to the client machines withing the corporate network




Publish the certificate to Active Directory
From the HTTPS Outbound Inspection  option under the General tab click on the large "HTTPS Inspection Trusted Root CA Certificate Options" button

• Select Automatically through Active Directory
• Click Domain Administrative Credentials
• Specify Credential of an enterprise administrator
• OK
• OK

The certificate will now be sent through the Active Directory be published and trusted.

Give this some time to replicate and then check to see that it is installed correctly.

Verify the TMG as a Trusted Root Certificate Authority

• From the MMC console 
• Add the Certificates snap-in
• Select Computer account (You have to be logged in as and administrator for this to be an option)
• Expand the Trusted Root Certificate Authorities section
• Click on Certificates

In the list you should see the self signed certificate you generated earlier.  If you view this certificate you should see that the status has changed.  The Certificate Information section should now state the intended purposes, as opposed to the CA root is not trusted.  The Icon would also have changed from having a red cross on it to not having a red cross.

It is important that this step has completed successfully before you move on to the next step.

TMG HTTPS Inspection Part I - Planning

Why is it needed

By default outbound HTTPS traffic is not inspected by TMG.  This is because of the SSL tunnel that encapsulates all the traffic.  This is great for keeping sessions private between the client and the server.  The problem now exists that you have a bi -direction data stream in and out of your corporate network that you are no longer inspecting for malware etc.  It also allows users to potentially bypass corporate policies by tunneling traffic through the SSL channel, such as peer to peer or proxy avoidance sites.

To enable TMG to inspect the HTTPS traffic you need to configure HTTPS inspection under the Web Access Policy.

How it works

TMG has to become a "man in the middle" between the client and the server.  This is not different from what it is for normal HTTP traffic.  The HTTPS / SSL differences, in sequence, are:

1. TMG etablishes the SSL tunnel with the server and validates the site and certificate.
2. TMG then copies the certificate details and issues it's own corresponding certificate for the client.
3. TMG then uses this certificate and establishes a secondary secure channel to the client.

Essentially you now have two SSL tunnels as opposed to just one that goes all the way through.

Certificate Validation
As part of enhancing the SSL security TMG can also be configured to do additional Certificate Validation.  This can be configured to check for some or all of the following criteria:

• Block Expired Certificates after n days
• Block server certificates that are not yet valid
• Check for server certificate revocation (crl)

These are things that can normally be overridden by client during browsing.  These checks prevent those from getting to the client and having to make the decision.

Things to consider when planning
Whenever additional security is added you run the risk of additional impact on your users.  There are a few drawback that can be resolved, but one should be aware of them.

• TMG has to becomes a trusted certificate authority for all the client machines
• Extended Validation (EV) is not supported as EV, you would have to exclude these sites to get the green bar
• External SSTP (Secure Socket Tunneling Protocol VPN) is not supported unless they are manually exluded
• Clients must belong to the same domain as TMG
• Self signed certificates are not supported, unless excluded
• Not all browses use the machine certificate store EG Firefox certificate store needs to be configured separate to the machine

Privacy

Unless a user has the TMG client installed and the HTTPS inspection is set to "Notify users that their HTTPS traffic is being inspected" a user will not know that this is the case.  Unless they check every certificate to see who the CA is.  Because users expect SSL session to be private and secure you might want to or have to notify your users of this.  

Exclusions
Almost all of the potential problem above can be mitigated or worked around by buildiong proper exclusion lists.   Exclusion can be base done based on the Source and /or  Destination.  You can for example exclude all HR computers, or you can exlude all sites in the "Financial" url categories.

Conclusion
Depending on circumstances HTTPS inspection may or may not be a priority for your environement.  If you have to implement it you need to follow the steps to make it a smooth transition.  If not, all HTTPS traffic will be compromised...