31 October 2011

Building a corporate Torrent tracker server and clients

Private torrent and trackers are a great way to deal with transferring files over WAN links.  in http://fixmyitsystem.com/2011/10/using-private-torrent-to-transfer-large.html I covered the basics of configuring uTorrent as a private tracker.  In this article I will go through building a dedicated corporate torrent server. I will also cover how to configure client for remote administration and use.  It is a little strange referring to torrents in a client / server  manner but in this case this is the desired relationship.

The base server
I am building this server on Windows Server 2008 R2 Standard.
I Install the Web server role.
I Add the following additional role services over and above the default ones.
HTTP Redirection

The reason for installing IIS is that this will be the primary method for distributing the actual .tor torrent files.

Create four folders
  • C:\Active Downloads
  • C:\Completed downloads
  • C:\Torrent Seeds
  • C:\Torrent Files


Grant full NTFS access permissions on these folder to the service account that will be used for torrents.

Part 1 Install and configure uTorrent as a tracker
If you are going to be using this as a always on server i suggest you follow the guide on how to install uTorrent as a service.

Key to how this works is the uTorrent client and server components.  You will need to download the uTorrent client from http://www.utorrent.com/downloads/complete I would also suggest getting the web interface components from http://www.utorrent.com/help/guides/webui

Make sure you are logged in as the service accoutn when cofiguring the following.
The basic config is covered in http://fixmyitsystem.com/2011/10/using-private-torrent-to-transfer-large.html

A few setting you will need to double check when following this guide is
  • During installation make sure "Start uTorrent when Windows starts up" is NOT checked
  • Port Used for Incoming connections is fixed on 443
Once the installation is completed change the default directories.

  • Open the uTorrent Client
  • Select Options - Preferences
  • Directories
  • Check Put new downloads in... C:\Active Downloads
  • Check Move completed downloads... C:\Completed Downloads

Part 2  Installing the Web UI
The web UI is a great way to remotely control and manage both the torrent "server" and clients.  This process should be followed on all the client machines where you will be installing uTorrent as well.

  • Open the uTorrent Client
  • Select Options - Preferences
  • Advanced
  • Web UI
  • Enable Web UI




There are some other options here worth noting  Since we are already fixing the connection port we do not have to specify an alternative.  For the client however you might want to consider setting this.

We now have to deploy the downloaded WebUI files for the uTorrent client to use.
Rename the downloaded file to webui.zip
Copy the file to %appdata%\utorrent  normally C:\Users\username\AppData\Roaming\uTorrent

You should now be able to connect to the web UI by connecting to http://localhost:443/gui or http://servername.domain:443/gui

Part 3 Configure IIS
IIS allows us to make the torrent files available by a URL.  The web client can be instructed to obtain a torrent via a URL so this is an ideal scenario.

I add an additional IP address to the server and bind that IP to the default web site.

  • Open IIS Manager
  • Select the Default Site
  • Click Binding from the Right Hand Actions Pane
  • Change the IP address from "all unassigned" to the secondary IP

Change the physical path of the site

  • Click Basic setting from the actions pane
  • Change physical path to C:\Torrent Files

Enable Directory browsing

  • Double Click Directory browsing and Enable it from the actions pane

Allow the .torrent MIME type

  • Double Click MIME types
  • Click Add from the action pane
  • File name extention = .torrent
  • MIME type = application/torrent

.
Part 4 Getting it all to work together
The point of this is to get a solution working where you can centrally manage and distribute files.  For this you have the torrent tracker hosted centrally, and you are using the web UI on the remote machines to control what they download and also how much resource they can consume etc.

On the local central torrent server

  • Create torrent file or folder in C:\Torrent Seeds
  • Save .torrent file to C:\Torrent Files
  • The torrent should start seeding
  • Open browse to http://secondaryip/
  • The created torrent file should be listed - copy this link





To download this torrent remotely

  • Connect to web UI of the "client machine"
  • Log In
  • Click the add torrent from URL Icon
  • Add the torrent URL file from the URL link copied above

The torrent should now start downloading.


Conclusion
Following this guide you can build infrastructure that allows a single person to publish and distribute files over slow unreliable links.  The fact that both the torrent server and remote client can be managed remotely adds a huge advantage in case one needs to "jump in" and stop a constrain a torrent from seeding or downloading.
Since standard ports are used through the process it also simplifies firewall and Access Control Lists and NATs etc.



More on using torrents
http://fixmyitsystem.com/2011/10/building-corporate-torrent-tracker.html
http://fixmyitsystem.com/2011/10/using-private-torrent-to-transfer-large.html
http://fixmyitsystem.com/2011/11/run-any-executable-as-service.html





27 October 2011

Windows DNS unable to resolve some external DNS names

I ran into this issue recently and it only happened for a few public DNS names.  Overall things were working pretty normally.

DNS Basics
Typically a corporate domain joined machine is configured with a local DNS server for that domain.  All queries are sent to that DNS server.  If it cannot resolve the name it will pass it onto the DNS Forwarder servers.  If no forwarders are available the Root hint servers will be used.

In a larger organisation you might have loads of internal servers that forward to one another.  For server not able to do DNS queries externally you should disable "Use root hints if no forwarders are available."  This chain can continue until one or more servers are allowed to query out to the internet.  If no external forwarder servers are specified it would then use the root hints servers.

My recommendation here is to use one or more external forwarders on these servers.  Typically these would by the ones supplied by your ISP.  Very common now is to also use the Google public DNS service operating on 8.8.8.8 and 8.8.4.4

Do not specify the Google Public DNS service servers as a root hint. Because they are not authorative for any domain as they do not operate as Top Level Domain (TLD) name service

Root hint serves are authorative servers operated by various bodies.  For instance the l.root-servers.net. is operated by ICANN and j.root-servers.net is operated by Verisign.  These are hosted on multiple redundant servers published using anycast routing.  These are distributed world wide.




If you look at the nslookup result you will see the difference. Using nslookup hostname will resolve according to your specified IP / DHCP settings.  Using nslookup name DNSserver will for the use of the specified DNS server

Here I am resolving against the Google public DNS service



Here I am resolving against a root server




Troubleshooting
First I had to track down the servers that were allowed to query out to the internet.  The test for this was to do an nslookup form that server to a public DNS server.  If the connection failed like the image below you can deduce that no public DNS look up is allowed.


Internal servers like this one should all eventually point to the internal server(s) that are allowed to resolve externally.  Once all the internal configuration is correct we can start troubleshooting the outside.

When I checked the configuration of the DNS servers I found the following. The Root Hints servers for
l.root-servers.net & m.root-servers.net had unknown IP addresses.


Resolving these completed the list.
l.root-servers.net. is 199.7.83.42
m.root-servers.net. is 202.12.27.33

For more info check out http://root-servers.org/

I also discovered that one of the last internal DNS forwarder servers had no external forwarders specified.
This would in theory mean that this server would not be able to resolve any domain name that is hosted by the ICANN or WIDE Project organisations.

Conclusion
Trouble shooting public DNS name resolution starts at home.  Follow the chain of DNS server from the DCHP scopes down to the root hint servers as the problem could reside anywhere along the chain.


More Info
This useful table comes from http://en.wikipedia.org/wiki/Root_name_server there is some more interesting bids of info there too.



Letter
IPv4 address
IPv6 address
Old name
Operator
Location
#sites (global/local)
[4]
Software
A
198.41.0.4
2001:503:ba3e::2:30
ns.internic.net
Distributed using anycast
6/0
B
192.228.79.201 (since January 2004; originally was 128.9.0.107)[5]
2001:478:65::53 (not in root zone yet)
ns1.isi.edu
C
192.33.4.12
2001:500:2::c (not in root zone yet)
c.psi.net
Distributed using anycast
6/0
D
128.8.10.90
2001:500:2d::d
terp.umd.edu
E
192.203.230.10
N/A
ns.nasa.gov
F
192.5.5.241
2001:500:2f::f
ns.isc.org
Distributed using anycast
2/47
G
192.112.36.4
N/A
ns.nic.ddn.mil
Distributed using anycast
6/0
H
128.63.2.53
2001:500:1::803f:235
aos.arl.army.mil
I
192.36.148.17
2001:7fe::53
nic.nordu.net
Distributed using anycast
36
J
192.58.128.30 (since November 2002; originally was 198.41.0.10)
2001:503:c27::2:30
Distributed using anycast
63/7
K
193.0.14.129
2001:7fd::1
Distributed using anycast
5/13
L
199.7.83.42 (since November 2007; originally was 198.32.64.12)[8]
2001:500:3::42
Distributed using anycast
37/1
M
202.12.27.33
2001:dc3::35
distributed using anycast
5/1






How to configure your ADSL for Telkom VPI VCI and Encapsulation

Some DSL modems are pre-configured to work with Telkom.  Other need to be set up manually.

Here are the recommended settings:

  • Encapsulation required = PPPoE
  • Multiplexing = LLC based
  • VCI = 35
  • VPI = 8


To verify if your Telkom line is ADSL enabled check here:
https://secureapp.telkom.co.za/ADSLVERIFY_NEW/

Do do a speed test to see that you are getting the right kind of throughput run the test from here:
http://speedtest.net/


If you want to know what the settings are and what they mean read on...

Encapsulation
The Point-to-Point Protocol over Ethernet (PPPoE) is a network protocol for encapsulating Point-to-Point Protocol (PPP) frames inside Ethernet frames. It is used mainly with DSL services where individual users connect to the DSL modem over Ethernet.


Multiplexing
Logical Link Control (LLC) is the upper sublayer of the OSI data link layer. The LLC is the same for the various physical media (such as Ethernet, token ring, and WLAN).
The LLC sublayer is primarily concerned with:
Multiplexing protocols transmitted over the MAC layer (when transmitting) and demultiplexing them (when receiving).
Optionally providing flow control and detection and retransmission of dropped packets, if requested.


ADSL Options

VCI Stands for "Virtual Channel Identifier." The VCI, used in conjunction with the VPI (virtual path indicator), indicates where an ATM cell is to travel over a network. ATM, or asynchronous transfer mode, is a method that many ISPs (Internet Service Providers) use to transfer data to client computers. Because ATM sends packets over fixed channels, the data is easier to track than information sent over the standard TCP/IP protocol.

The VCI within each ATM cell defines the fixed channel on which the packet of information should be sent. It is a 16-bit field, compared to the VPI, which is only 8 bits. Since this numerical tag specifies the virtual channel that each packet belongs to, it prevents interference with other data being sent across the network.



The information was sourced from various sources on the internet so credit to them and all the other users that have contributed to the definitions above.

25 October 2011

Accommodate Tablets and Smart Phones in your company using TMG as a Wifi gateway

The Problem
There has been a steady increases of more and more tablet / slate devices and more and more user bringing their own un-manged devices to work.  With this increase there is a higher than ever demand for IT departments to provide the services need for these devices to fully function in the workplace.  I have been trying to figure a good way to accommodate their need on a corporate network without compromising security.

My simple answer was to just create a public hotspot for them and give them the same access anyone from the internet would have.

There was just one big catch though.  These devices can be anything from an I-Phone to an Android tablet to an unmanaged Windows XP laptop.  I needed a generic one size fits all approach to authenticate users who should be and should not be on the Wifi Hotspot.  Using a proxy would have been perfect except not all devices or apps support proxies, and you have a different "setup procedure" for each device. The only answer was to use a NAT.  The limitation with TMG though, is there is no native way to authenticate users for using a NAT.

The Answer
Collective Software has a product called Captivate that allows you do do just that.  In my example I needed users to provide a username and password into a web form before they were granted access.

http://www.collectivesoftware.com/Products/Captivate

Captivate allows you to use TMG forms based authentication to authenticate a user.  This has the advantage of using a "native authentication method, so you can use any of the TMG supported authentication validation methods:

  • Windows Active Directory
  • LDAP
  • RADIUS
  • RADIUS OTP
  • RSA SecureID
You can therefore have guest accounts residing in a separate LDAP store without ever having to create a domain user for them.  You can also use customised forms and provide the functionality of notify users about passwords that are expiring, and provide them a way to change it.  I customised the Exchange template for this (check http://fixmyitsystem.com/2010/11/customise-tmg-exchange-forms.html )


Now this all seems very basic, but image a users who never ever uses a PC, only their own private I-Pad...

Building the Solution
You will need to download the Captivate Evaluation package.
Download the Captivate Documentation. - The doc is really well put together and all you have to do to get it working is to follow the doc.

TMG server
I built a TMG server with three networks.


  • Corporate Internal
  • Wifi Internal
  • External



  • The relationship between Corporate and External is a NAT
  • The relation ship between Wifi and External is a NAT
  • The relation ship between Wifi and Corporate is a NAT *


I configured rules so that the Wifi network never communicates to the corporate network.  Instead the connections flow out to the Internet where it can be routed back to my internet published applications.  This gives me the advantage of have a single point of exposure to un-trusted networks such as the Internet and this Wifi Guest network.  TMG only communicates with the corporate network for the authentication and management.


It is of course also possible to configure rules to allow access to the corporate network, you can even allow this without having to authenticate the user with the Captivate plug in.

Captivate Configuration
Before you configure anything for this - read the PDF from http://www.collectivesoftware.com/Products/Captivate you will need to follow this guide for configuring the Luna scripts.

Essentially it installed as a web filter.  This then becomes an option to enable on the rules where you want this to apply.



As a minimum you will need to create a publishing rule to host the authentication form.  Secondly you need to configure your access rule for Internet access.

Changing the "Trigger Captivate process" essentially give the authenticated session a lifespan - something like a DHCP lease.

Enabling the Tracking options enable you to keep track of the user / machine relationship.  This is important if you would ever need to forensically trace a user to a session.  If however you simply want to  secure the access it is not required.

Internet Access and other rules
The only drawback you have is that the user is authenticated but it does not pass on the user credentials to NAT requests.  You can therefore not set user level access rules.  On the up side you can still apply URL filtering rules to prevent access to certain site categories. You are also able to limit access to IPs ports and protocols you want to allow.  You can do anything you would normally do with a secure NAT client.

Conclusion
Un-manged devices are un-trusted devices, they should not ever have unrestricted access to your corporate network.  Device limitations mean that the traditional health check / UAG kind of solution simply will not work for this.  It the past it was easy to simply say no to these requests but realistically you will have to accommodate them in the not too distant future.


Check the follow up article  - http://fixmyitsystem.com/2011/11/accommodate-tablets-and-smart-phones-in.html

21 October 2011

TMG SP2 customise URL filtering error pages

The old pre-SP2 error pages
ISA and TMG error pages have always been ugly.  Brown and ugly.  But thankfully with SP2 they have added improved error pages that are easier to customise.


When introducing URL filtering users would often get a screen that you wanted them to know was not an error but a purposeful denial.  To do this I customised the relevant error pages, after some to and fro this was the beast I could come up with with a "minimal effort"



Enable New Style Error Pages
After installing SP2 the default will still be to use the old style error messages.  To enable the new style you need to do the following:

  • From the TMG console select the Array
  • Right Click - Properties
  • Select the Error Pages tab
  • Select the "Use the new version available from ... SP2 onward"

You should now get the new style error pages


Already a big improvement.
If you had customised your error pages you need to also customise the new ones.  The exception to this is if you have specified a custom admin message.  This is retained.

Customise The error pages
To simply stick your company logo on the new error pages you need to just edit and replace the "C:\Program Files\Microsoft Forefront Threat Management Gateway\Templates\WebObjectsTemplates\ISA\HTML\logo.png" file.  This is a transparent png file so keep it that way or it will introduce a "big white block"

Editing this image will apply to all error pages.

For most - this is a far as you would normally go.

URL Filtering pages
The Web access policies allows you to specify a few options when defining the Block rule for the various URL categories you block.  You have a few options to choose from that render different levels of information on the error pages.  These actually change the error .htm file being used.

To set this you need to change the following

  • From the TMG management Console
  • Select the Web Acess Policy
  • The select the Deny rule that block the categories
  • Select the Actions Tab
  • Click the Advanced button


The options translate as follows:


12231.htm  - This page is used when the deny rule is set to display the URL category, but not the custom message; [URLCATEGORY] will be replaced with the category name.

12232.htm  - This page is used when the deny rule is set to display the custom message but not the URL category; [ADMINMESSAGE] will be replaced with the custom message.

12233.htm  - This page is used when the deny rule is set to display both the custom message and the URL category; [URLCATEGORY] will be replaced with the category name and [ADMINMESSAGE] will be replaced with the custom message.


In My case I chose to edit the 12231.htm file directly as opposed to using the "Admin Message"  This just gave me more control to do what I wanted to do.  Also to conform to corporate color all images were changed to grey scale

The net effect is the following:


Copy all your edited files to all the nodes in the array and remember - For the new pages to take effect you need to restart the firewall service on the TMG server.

Conclusion
The Improved error pages in SP2 is a welcome change.  If however you had customisasion on your old error pages you would have to transfer them to the new style ones.  Without too much effort you can stick a corporate look and feel, and relevant information on your pages.



Public DNS look up for corporate users

Problem
Companies that have the same internal domain name as the external public domain name would run into this problem.  To resolve an ip address from a DNS name you can simple do an nslookup.  But what happens if you want the public IP?  Attempting to resolve the IP internally would only return the internal IP address.



Solution
Using an internet based look up tool such as http://www.ipchecking.com allows for this.  


Nice features of this site are:


  • Returns the public IP of the queried hostname
  • Resolve your public IP address
  • Identifies if you are using a proxy
  • Provides extended WHOIS information


13 October 2011

Using a private torrent to transfer large files over unreliable WAN links

Torrent files are almost always associated with pirated file transfer, but the technology is actually rather remarkable.  Torrent are extremely robust and handle disconnects and reconnects and slow connection really well. If you are, for instance, attempting to transfer a 1GB file into deep dark Africa over slow unreliable links this is a good option to use.

Required Software
My preferred torrent client has always been ĀµTorrent (meaning Micro Torrent) or as it more commonly referred to uTorrent.  It support everything you want it to do, including a web interface for managing your torrents.  Bandwidth throttling, schedules and thresholds.  Most importantly for this article, it has it own internal ability to be a torrent tracker.  This remove the requirement to have a public web tracker.

Download this great free tool from http://www.utorrent.com/

Configure uTtorrent as a tracker

Configure incoming listening port

From the file menu
  • Options
  • Preferences
  • Connection
  • Make a note of the port number that you specify. 
  • As a suggestion use port 443 as this would enable access through most firewalls
  • DO NOT check Randomize port for each start (you will see later why not)
  • Restart uTorrent (Also exit from the system tray icon)




Enable the Tracking ability
From the file menu
  • Options
  • Preferences
  • Select Advance
  • find bt.enable_tracker
  • Change the value from False to True
  • You now need to restart uTorrent - exit it from the system tray to ensure it closes completely




Additional Optional Settings
One of the best things about uTorrent and most other torrent clients is the ability to set limits.  Check out the options under Bandwidth, Transfer Cap, Queuing and Scheduler.

Establish what your external IP address is
You will need to know your external internet IP address to host or track the torrent file.  The quick and easy way is to visit http://whatismyip.org or http://whatismyip.com.  If you have a fixed external IP this is all you will need.

Most ADSL accounts however have a dynamic IP assigned by the ISP at each reconnect.  To work around this problem you can have a dynamic DNS entry registered by DynDNS.  The provide a free service that I have used in the past and have been very impressed by.  Check it out at http://dyn.com/dns/dyndns-free/

Creating a torrent file
From the file menu
  • Create new torrent
  • Add your source directory or file
  • Under trackers - delete the defaults - add your details you collected earlier in the format http://your_ip_address:your_listening_port/announce
  • Check Start seeding to make it available right away
  • Check Private torrent to make it private
  • Create and save as..
This will now produce the "very small" torrent file


Distribute and download the torrent
The torrent file created in the step above can now be distributed in any number of ways including emailing it.

You need to have uTorrent install on the client side with the relevant firewall access allowed.
  • Open the torrent file
  • Specify where to save it 
  • Check the Start Torrent Box
  • OK
The torrent should now be downloading.  once complete it should switch status from Downloading to Seeding.

At this point you can even stop seeing from the original tracking host and all subsequent download will be from peer machines that are seeding.  The more peers you have the greater the availability will be and therefore the more robust it would be against any single peer becoming unavailable, this could be because of any number of reasons, including scheduled or transfer caps.

Troubleshooting
If you create the torrent and seeding it should start downloading and seeding it right away.  If it does not there is a problem.

  • You must set the incoming port before you enable bt.enable_tracker.  If you need to change the listening port, set it back to false.  Change the listening port then re-enable it.  Restarting after each step.
  • If there is a problem with the listening port you will see a red warning in the bottom right corner of the main screen.
  • The file needs to be seeded from a location with correct NTFS security try adding the file from the c:\ drive as a test.
  • Check firewall access on both the seeing and downloading machines to ensure ports are open.

Conclusion
Torrents might be considered "the naughty file type", but it can be used to overcome certain transfer problems.  It is simple, free and robust.  Since torrent clients are available for almost any OS this is also a great way to transfer between disparate OS types.

And now there is one more legitimate torrent file, other than downloading yet another Linux distribution...



More on using torrents
http://fixmyitsystem.com/2011/10/building-corporate-torrent-tracker.html
http://fixmyitsystem.com/2011/10/using-private-torrent-to-transfer-large.html
http://fixmyitsystem.com/2011/11/run-any-executable-as-service.html