29 October 2010

Webspy Vantage FTMG W3C import reverse or switch bytes in and out

When setting FTMG or even ISA 2006 to export the logs to w3c text files something strange happens...  The bytes in and bytes out values get switched around.  The result of this is that you will see what looks like a large amount of data going out towards a website as opposed to it coming in.

You can either edit the MSDEtoText.vbs files to switch the values for you.  (You can ask me for a copy of this script if you want one.) Or you can use import your logs directly form the MSDE database.

I raised this issue with the Webspy developers as a feature request and they confirmed that they now include an easy way for you to switch these values around.  You may want to change from the default if you are importing directly from the MSDE or if you have a script that allready fixes the issue. Note this is only for FTMG not for ISA 2006

When creating a storage you have to select the loader you will be using.

  • When you get to the loader section select Microsoft FTMG.
  • Click on properties.  
  • Change the format from Automatic detection to Forefront TMG (W3C).  
  • A new check box option will appear. Either check or uncheck "Reverse bytes sent and received to compensate for a bug in TMG's logging" depending or your scenario.






If you do this per input location after the storage has been created and contains data you will have to clear the storage and re-import all the logs.


25 October 2010

TMG Version numbers and what they mean

Trying to find out what version of TMG you currently have installed can be tricky or cryptic. It is important to ensure that all your array members on on the same version.  You may discover that the versions are different but no know what to install to update it.  Fortunately there is and easy way to figure it out.

Version numbers:

7.0.7734.100  -  Release version
7.0.8108.200  -  SP1
7.0.9027.400  -  SP1 Software Update
7.0.9027.410  -  SP1 Software Update 2 (KB2433623)
7.0.9027.441  -  SP1 Software Update 1 Rollup 3 (KB2498770)
7.0.9193.500  -  SP2 (KB2555840)
7.0.9193.515  -  SP2 Rollup 2 (KB2689195)


Where to find them

You can see these from the TMG console - System - Servers


You can also check on these in the Programs section in the control panel


Click on "View installed updates"


The Update Center is where your Malware and NIS definition versions are shown.  You generally would just need to scroll to the right a bit.


Generally you would not have to worry about these unless you have red icons somewhere on this page...

Conclusion
Keeping your environment in sync does not have to be tricky, as long as you know where to look and know what to install to bring all hosts up to date.

22 October 2010

Adding a static route using netsh and route commands

It is recommended by MS that from Windows 2008 onward you use the netsh command shell rather than route to manage routes on your servers.

The Network Shell - netsh
"is a command-line utility that allows you to configure and display the status of various network communications server roles and components after they are installed on computers running Windows Server® 2008 R2 and Windows Server® 2008"

True, you can do tons of things in netsh if you are willing to jump in there and have a look.  I suggest you take a manual with you http://technet.microsoft.com/en-us/library/cc754516(WS.10).aspx

For this article I am going to cover static routes.  This is something I always used to do with the route command.  Check out Static Routing 101

We will be configuring a machines with dual nics, one called Internal 10.0.0.5 and the other External 196.0.0.5 the external will be configured with the default gateway in the TCP/IP properties.  We want to add a static persistent route for all traffic to the 10.x.x.x range to be routed to the internal networks as opposed to the default gateway.

Using the route command

route add 10.0.0.0 MASK 255.0.0.0 10.0.0.5 -p

To check that it was completed successfully you can review the routes.

route print


To delete the route


Route delete 10.0.0.0

Using netsh

netsh
interface 
ipv4
show interfaces (make note of the correct interface name)
add route 10.0.0.0/8 "Internal" 10.0.0.5

To check that it was completed successfully you can review the routes.

netsh interface ipv4 show route

To delete the route

netsh interface ipv4 delete route 10.0.0.0/8 "Internal" 10.0.0.5


Since they both configure the same thing you can also use the route to check the netsh setting and vice versa.
Just for reference here is the add route command details from technet.

add route

Adds a route for a specified prefix. Time values can be expressed in days (d), hours (h), minutes (m), and seconds (s). For example, 2d represents two days.

Syntax

add route [prefix=]IP4Address/Integer [[interface=]String] [[nexthop=]IPv4Address] [[siteprefixlength=]Integer] [[metric=]Integer] [[validlifetime=]{Integer | infinite}] [[preferredlifetime=]{Integer | infinite}] [[store=]{active | persistent}]

Parameters


[ prefix=] IPv6Address/Integer
Required. Specifies the prefix for which to add a route. Integer specifies the prefix length.
[[ interface=] String]
Specifies an interface name or index.
[[ nexthop=] IPv6Address]
Specifies the gateway address, if the prefix is not on-link.
[[ siteprefixlength=] Integer]
Specifies the prefix length for the entire site, if the prefix is not on-link.
[[ metric=] Integer]
Specifies the route metric.
[[ validlifetime=]{ Integer| infinite}]
Specifies the lifetime over which the route is valid. The default value is infinite.
[[ preferredlifetime=]{ Integer| infinite}]
Specifies the lifetime over which the route is preferred. The default value is infinite.
[[ store=]{ active| persistent}]
Specifies whether the change lasts only until the next boot (active) or is persistent (persistent). The default selection is persistent."

Conclusion
It appears that the route command is, or has been deprecated and will not be around in the next version of Windows.  Netsh is a extremely powerful tool if you take the time to get to know it.  In just a few minutes of scratching around I found some of useful stuff that is not available anywhere else.  Don't be intimidated but be careful. By using netsh interface tcp reset  you will clear every singe network setting ......

20 October 2010

TMG Auto Proxy Configuration Part IV - Configuring the clients

In the previous parts of this article we explored how to setup your environment so that your clients can make use of the auto detection.  To understand how to build in some fault tolerance you need to understand which setting take preference.


When "Automatically detect setting" is check the following order will be used to determine the proxy.


  • DHCP 
  • DNS  


If both DHCP and DNS detection fails and the "Use automatic configuration script" is checked then it will retrieve the script directly from the specified address.

If "Automatically detect" is not checked but "Use automatic configuration script" is checked it will not auto detect and just retrieve the script directly.

Either of the automatically detect setting will take precedence over a manually specified proxy server.

Group Policy
The recommended method for configuring the clients is via a group policy.  Without going into a lot of detail, the setting is normally a user setting. and there are options that correspond with the internet explorer LAN connection options.


I want to also highlight another policy.  Disable caching of Auto-Proxy scripts. This policy is under User Configuration > Administrative templates > Windows Components > Internet Explorer

The idea here is simple - unless the hostname of the auto detect changes, use the cache script.  This is great in an environment where you are not making any changes.  If however you have more of a dynamic configuration you may want to enable this policy

As an example - you may add another exclusion in your auto configuration script, but if the browser is using a cached version the change will not be applied.  To make things worse.  If you log onto the same machine and test it will work, if you have never logged on and retrieved a script before...

Registry
Since a lot of the proxy setting are stored in the registry you can manipulate the registry and affect the required proxy changes.  Below is an image of a machine that is configured with all four check boxes checked.  You can see that it is a lot of entries.  Fortunately making the change in IE reflect immediately in the registry (if you remember to refresh )  This makes it fairly easy to build  your desired registry setting that can be exported as a .reg file.


Internet Explorer
Users can also set their proxy setting in their browser.  This may be handy for troubleshooting but it is generally not a scalable solution in any shape or form.

TMG Client
If the user is using the TMG client (previously the ISA firewall client) this can also be configured to auto configure the browser proxy.  The client can also be configured using Active Directory (http://technet.microsoft.com/en-us/library/ee658145.aspx)

Conclusion
There are many different way to configure the client to make use of the automatic proxy configuration.  The client can also be configured to have a fail over configuration by specifying multiple settings.  All these are great if things are working but any of these could be a potential problem area when trying to troubleshoot.

Pick one method to configure your client and make that the standard and prevent users from being allowed to change their settings, or at least reinforce your setting with a group policy.

19 October 2010

TMG Auto Proxy Configuration Part III - Using DNS

In part one of this post I highlighted why it is important to use proxy auto configuration.  There are various way to get your clients to use your auto configuration. In this part I will go through using DNS and a WPAD entry.

This is probably the simplest way of deploying your auto configuration but there are a few things to be aware of.

If you want to use DNS for auto configuration the configuration MUST be available on port 80 

Step 1 - Verify

  • Verify that your TMG server is publishing the auto configuration script.
  • Using your browser connect to http://proxyaddress/wpad.dat  and http://proxyIP/wpad.dat
  • If you are promtped to download the file you know it is working.
  • Test this by opening the Internet options, click lan setting from the connections tab
  • Make sure only the "Use Automatic configuration script" is checked and then enter the tested url from above.
  • Test for internet connectivity etc.
Step 2 - Configure DNS

  • From the DNS server's DNS mmc console
  • Right click the forward lookup zone of your domain
  • You can now either create an Alias (CNAME) or Host (A record)
  • Specify the FQDN or the IP of the proxy 

Step 3 - Windows 2008 DNS and 2003 DNS after MS09-008
If you are not using these as your DNS server have a look at http://support.microsoft.com/kb/934864

To reduce vulnerabilities associated with Dynamid DNS update a new feature has been introduced call the the  DNS Block list.  Basically it just blocks request for the DNS names that are blocked.  By default two entries are blocked ISATAP and WPAD. ISATAP  provides a transition between networks that are based on Internet Protocol version 4 (IPv4) and networks that are based solely on the newer IP version 6 (IPv6)


Server 2008
The following need to be performed form the command prompt on every authoritative DNS server
  • dnscmd /info /globalqueryblocklist - To show which entries are in the block list
  • dnscmd /config /globalqueryblocklist isatap - This updates the list to only block isatap

Server 2003 post MS09-008
The following need to be performed on every authoritative DNS server
  • Edit the following registry entry
  • HKLM\SYSTEM\CurrentControlSet\Services\DNS\Parameters\GlobalQueryBlockList
  • Remove the WPAD entry

To see if this entry is now being allowed you can do an nslookup wpad.yourdomain.com if you get an error despite there being an WPAD entry you know it is still being blocked.

Step 4 - Test

  • Log onto a client machine that is not defined in a DHCP scope with a WPAD setting
  • Open Internet explorer and make sure that only the "Automatically detect settings" check box is checked.
  • Restart IE
  • Test for internet connectivity.


Notes: 

  • A DHCP WPAD entry will take precedence if there is also a WPAD DNS entry.
  • In IE - Specifying "Automatically detect settings" will take precedence over  "Use Automatic configuration script"
  • In IE - The automatic automatic configurations will take precedence over manual configuration

TMG Auto Proxy Configuration Part II - Using DHCP

In part one of this post I highlighted why it is important to use proxy auto configuration.  There are various way to get your clients to use your auto configuration. In this part I will go through using DHCP and the WPAD option

Step 1 - Verify

  • Verify that your TMG server is publishing the auto configuration script.
  • Using your browser connect to http://proxyaddress/wpad.dat or http://proxyaddress:8080/wpad.dat
  • If you are promtped to download the file you know it is working.
  • Test this by opening the Internet options, click lan setting from the connections tab
  • Make sure only the "Use Automatic configuration script" is checked and then enter the tested url from above.
  • Test for internet connectivity etc.


Step 2 - Configure DHCP

  • From the DHCP server's DHCP mmc console
  • Expand IPv4
  • Right click IPv4 and click "Set Predefined Options"
  • Click a Add
  • In the Option type specify the following
  • Name - WPAD
  • Code - 252
  • Data - String
  • String - The tested url from step 1 - make sure wpad.dat is in lowercase


Step 3 - Apply to Server or Scope
Test your setting on a scope before implementing a server option

  • Expand to the scope where you want to implement
  • Right click Scope options and click Configure options
  • Click advanced and then Vendor Class, standard options
  • Form the Available options select 252 WPAD and click OK


Step 4 - Test a client

  • Log onto a client machine that is in the defined DHCP scope
  • Open Internet explorer and make sure that only the "Automatically detect settings" check box is checked.
  • Restart IE
  • Test for internet connectivity.


For more reference you can check out http://technet.microsoft.com/en-us/library/cc940962(WS.10).aspx

Notes: 

  • A DHCP WPAD entry will take precedence if there is also a WPAD DNS entry.
  • In IE - Specifying "Automatically detect settings" will take precedence over  "Use Automatic configuration script"
  • In IE - The automatic automatic configurations will take precedence over manual configuration

18 October 2010

TMG Auto Proxy Configuration Part I - Manual or Auto ?

The ability to configure your environment so that machines can automatically detect their proxy settings is an extremely useful one.  However there is a lot of confusion when it comes to understanding what that magic "Automatically Detects Setting" check box does and how it works.

To really understand the  importance of the auto configuration let us compare the manual proxy configuration compared to an automatic configuration script.

When you manually specify a proxy server you get to specify the server name and the port number.  You can specify the same proxy to be used for all protocols.  Up to this point the only real problem comes is when the host name of your proxy server changes.

When specifying the exceptions the biggest limitation becomes obvious.  Some believe that checking the "Bypass proxy server for local addresses" is all that is required.  Sadly this is not the case.  Only a URL that do not have a domain specified is considered a local address.  Therefore if your intranet address is http://myinranet/ it is considered local.

If you access access the same site via a http://myintranet.mycompany.com FQDN then it is considered an Internet site. This means that request will then be routed through the proxy.


You can start to manage this by specifying exceptions but now you have to redo these whenever the environment changes.  And since this setting is per user you have a problem.


When you configure the proxy client settings in TMG you have a number of options available to you to specify when and when not to use a proxy.  You can also specify a backup proxy should the primary not be available.

You can make this make this configuration available to clients by checking the "Publish automatic discovery information for this network" check box.

The automatic proxy configuration script contains a lot of information.  It is then published as the WPAD.DAT file.  This file is normally hosted on your TMG servers but it can also be hosted on any other web server.

To check it out all you have to do is point your browser to the autodetect url and that is  http://proxyserver/wpad.dat or http://proxyserver:8080/wpad.dat  and for the TMG firewall client  it is http://proxyserver:8080/array.dll?Get.Routing.Script.  They all deliver the same configuration file that will be used by IE to determine how web resources are accessed.

If you take the time to examine the file you will see that there are many different setting that are specified.  To try and do this manually would simply not be practical, also if you take into consideration that that this file is generated for you it is free of "human" typo etc that would be very difficult to trace.

Effectively, if you are a big enough deployment to use a proxy, you need to be using auto configuration.  There a various ways to setup your clients to retrieve and use your auto configuration, and I will go into those in further posts on the subject.

14 October 2010

TMG update center fails to update malware and NIS with a webchain

The TMG update center allows you to update the Malware and NIS signatures or definitions.  You can specify if you want to use your own internal WSUS server or point directly to Windows Update (recommended).

There are system policies that allows access to the Windows update server. so these don't need to be changed.

If it all works it is great, if it doesn't it's a pain to troubleshoot.  My scenario involves a TMG array placed behind the external TMG array, so no direct internet access is available.

This means that the only way for my update center to update is to use my external TMG as it's proxy.

Windows Update and indeed the update center make use of the Windows update service. So troubleshooting issues there will fix your update center.


So the problem symptoms:

Red x Icons in the Update Center


Checking for updates takes forever to fail.
Alerts in the dashboard and event log stating:


Log Name:      Application
Source:        Microsoft Forefront TMG Update Agent
Date:          2010/10/13 05:54:41 PM
Event ID:      23450
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Description:
An error occurred during an attempt to check for, download, or install definition updates on the server . 





Log Name:      Application
Source:        Microsoft Forefront TMG Update Agent
Date:          2010/10/13 05:54:41 PM
Event ID:      23481
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A

Description:
The last 60 attempts to check for updates for the Malware Inspection protection mechanism on the server  failed.


When checking out the c:\windows\windowsupdate.log file you also see the following errors.



2010-10-14 11:58:26:749 924 a90 Agent ** START **  Agent: Finding updates [CallerId = Forefront TMG]
2010-10-14 11:58:26:749 2600 aa0 COMAPI <<-- SUBMITTED -- COMAPI: Search [ClientId = Forefront TMG]
2010-10-14 11:58:26:749 924 a90 Agent *********
2010-10-14 11:58:26:749 924 a90 Agent  * Online = Yes; Ignore download priority = No
2010-10-14 11:58:26:749 924 a90 Agent  * Criteria = "(IsInstalled = 0 and IsHidden = 0 and CategoryIDs contains '84a54ea9-e574-457a-a750-17164c1d1679' and CategoryIDs contains 'e0789628-ce08-4437-be74-2495b842f43b') or (IsInstalled = 0 and IsHidden = 0 and CategoryIDs contains 'ae4483f4-f3ce-4956-ae80-93c18d8886a6' and CategoryIDs contains 'e0789628-ce08-4437-be74-2495b842f43b')"
2010-10-14 11:58:26:749 924 a90 Agent  * ServiceID = {7971F918-A847-4430-9279-4A52D1EFE18D} Third party service
2010-10-14 11:58:26:749 924 a90 Agent  * Search Scope = {Machine}
2010-10-14 11:58:26:753 924 a90 Misc Validating signature for C:\Windows\SoftwareDistribution\WuRedir\9482F4B4-E343-43B6-B170-9A65BC822C77\muv4wuredir.cab:
2010-10-14 11:58:26:757 924 a90 Misc Microsoft signed: Yes
2010-10-14 12:00:33:767 924 a90 Misc WARNING: WinHttp: SendRequestToServerForFileInformation failed with 0x801901f8
2010-10-14 12:00:33:768 924 a90 Misc WARNING: WinHttp: ShouldFileBeDownloaded failed with 0x801901f8
2010-10-14 12:02:40:782 924 a90 Misc WARNING: WinHttp: SendRequestToServerForFileInformation failed with 0x801901f8
2010-10-14 12:02:40:782 924 a90 Misc WARNING: WinHttp: ShouldFileBeDownloaded failed with 0x801901f8
2010-10-14 12:04:47:794 924 a90 Misc WARNING: WinHttp: SendRequestToServerForFileInformation failed with 0x801901f8



These all indicate that there is an error connecting to the Windows Update sites.  First step I did was to check if I could access the sites from IE on the TMG server.  This worked so since windows update uses WinHTTP  I imported my IE proxy settings


Open a command prompt as administrator



  • NetSH 
  • WinHTTP 
  • import Proxy ie
This then ensures that the proxy is set so you don't have to worry about it auto detecting.
If you want to know a bit more about WinHTTP check out this link



But still the problem persisted,  checking the Firewall logs on both TMG arrays also did not help because I simply did not see any request during the windows update process.   The only requests I saw were for traffic coming through this array in a Webchain.


Web chaining is a way to link proxy server together so that the one forwards it's request to another upstream proxy.


My web chain rule restricted access to only one specified URL set.  The I changed the web chain rule and included the Microsoft Update Sites from he Domain Name Sets. 




The I tried the updates again and this time it succeeded. 







NOTE:
If you are using WSUS on any port other than 80 and 443 you need to create an additional protocol and create an allow rule from the TMG servers to the WSUS server.




Conclusion
It is highly unlikely that you will have the same problem as I did, but going through the logs and setting will helpfully assist you with finding your own problem area.  The only place where you will get any useful information is the c:\windows\windowsupdate.log

11 October 2010

SCVMM Error (12700) Failed to start virtual machine


One of the oldest and best ways to keep track of the states of your virtual machines is to keep copies of your virtual disks.  This means multiple copies of your .vhd files.  This is NOT snapshots or checkpoints.

SCVMM gives you the option of migrating machines to your library.  This is handy but sometime you just want to grab an existing disk and replace the current one.  I recently tried that and ran into this error:


"Error (12700)
VMM cannot complete the Hyper-V operation on the hyperv01 server because of the error: 'servername' failed to start. (Virtual machine ID C4A9BFD2-848F-4F23-ACB7-608AD634AD2D)

'servername' Microsoft Emulated IDE Controller (Instance ID {83F8638B-8DCA-4152-9EDA-2CA8B33039B4}): Failed to Power on with Error 'General access denied error' (0x80070005). (Virtual machine ID C4A9BFD2-848F-4F23-ACB7-608AD634AD2D)

'servername': IDE/ATAPI Account does not have sufficient privilege to open attachment 'C:\ProgramData\Microsoft\Windows\Hyper-V\servername\Windows 2008 x64 Standard Template_disk_1.vhd'. Error: 'General access denied error' (0x80070005). (Virtual machine ID C4A9BFD2-848F-4F23-ACB7-608AD634AD2D)

'servername':  Account does not have sufficient privilege to open attachment 'C:\ProgramData\Microsoft\Windows\Hyper-V\servername\Windows 2008 x64 Standard Template_disk_1.vhd'. Error: 'General access denied error' (0x80070005). (Virtual machine ID C4A9BFD2-848F-4F23-ACB7-608AD634AD2D) 
 (Unknown error (0x8000)) 

Recommended Action
Resolve the issue in Hyper-V and then try the operation again."

I found this rather odd since I was the creator / owner of the file and I was logged in as an administrator.  Looking at the .vhd file I noticed a padlock over the file icon.

I have not been able to track down exactly what it means but changing the NTFS file permissions got rid of the padlock.  One that was done then I could start the virtual machine again.

The problem comes in when a file is copied to or created inside the virtual machine file structure.  So when you copy or or create a file in the following directory you get the padlock.

C:\ProgramData\Microsoft\Windows\Hyper-V\servername\

This is strange since you should inherit the permissions from the parent folder when you create or copy a file.  Looking at the parent folder advance permissions shows that there are permissions that apply on to the parent folder and subfolders but not the files too.

I change the advanced permission for "Virtual Machines" to apply to  "This folder, subfolders and files" and apply it to object in the container.  Then the padlock goes away.




07 October 2010

TMG non-primary URL filtering categorization Walkthrough

As part of

One of the new features that has been added is:

Including non-primary URL filtering categorizations. Forefront TMG uses an algorithm to select a URL’s “primary” category from among up to four categorizations provided by Microsoft Reputation Services (MRS). In Update 1 you can control access to sites that match any of the non-primary categorizations provided by MRS. For example, a URL with a primary categorization of News can now match a rule by any of its non-primary categorizations (such as Web Mail).
Documentation on this has been a little thin so hopefully this will help someone out there.

If we look at the Pre-software update environment and lookup a URL category we only get one match for www.yahoo.com

If you look at the Deny rule that is added in the "web access policy" we can deny access to the various categories.


So what do we get from Software Update 1 ?

When we do a category lookup we now see that multiple categories are listed.

Now there need to be a mechanism to turn non primary category filtering on or off.  When looking at the Deny Rule for the "web access policy categorizations" 

So my test is now going to be to access www.yahoo.com with non primary filtering turned on and off.  I am only filtering portal sites which was listed as a non primary category for that URL.

With it un checked we can see the Yahoo page



But when we check it then it blocks the request by the non primary url category

If you want to disply the category information on the Access Denied page you need to set it opn the deny rule.

Click advanced and check the "Add denied request catagory to notification. This option is only available when URL filtering is enabled" check box.


Conclusion
The ability to block sites by up to four category matches is a very powerful feature.  It is great that it has been added to TMG.  Since the implementation of this is on a rule level, you can selectively apply this to certain users only if you wish to do so.