29 November 2011

TMG log file change prevents Webspy import

Webspy Vantage is on of my favourite add on tools for TMG.  I use it for generating usage reports and for forensic tracing of user activity.
The TMG logs can be imported into webspy in one of  three ways.


  • Connect directly to the Logging Database on the TMG server if the logging is set to "SQL Server Express"
  • Import the log text files if the logging is set to "File" 
  • Use a MSDEtoText vbscript to log to SQL server express and generate a text file
TMG native reporting only works if logging to the local SQL Server express.  The advantage of using the last option is that you can keep native reporting and you are able to import files "offline" by simply importing the text files.  This reduces the time the TMG is losing resource while the logs are imported by Webspy.

This worked great until recently when there appear to have been a change to TMG logging.  The symptoms you would see in Webspy are the following:
  • Log files do not import any hits
  • Summaries cannot be generated as no storages are available to select (even thought they are there)
  • Reports contain no data




The reason for this is explained here by Stefanie from Webspy Support:

The problem is occurring because at the end of every log file is an additional TAB, which causes Vantage to think there is another field expected, and when it finds none it does not import the line. We have recently seen this issue with a few other clients

Existing files can also be fixed up by manually adding the word NULL to the end of the #Fields line, or removing the additional tab from the end of every line.

They have also updated the MSDEtoText Script to fix this issue available from:



Existing storage will have to be cleared or purged of all data for everything to start working properly again.


2 comments:

khizar khan said...

We are also facing the same issue, can you please recommend 3rd party log file editor. Because notepad is taking so much time to read the log file.
Secondly Is there any other way to fix this issue/ Because we have more than 53 log files, It is so difficult task to edit manually each log file.

Etienne Liebetrau said...

Khizar

You can check out this utility

http://www.fastvue.co/blog/free-windows-grep-for-w3c-log-files

it allows for filtering your logfiles and reduce their size. Alternatively you can maiil support@fastvue.co for help with the import.

Post a Comment