28 August 2014

How to Fix Remote Desktop Gateway WMI failure Unable to Update Local Resource Group

When attempting to add another machine to a local computer group you receiver the following error:

    WMI failure Unable to Update Local Resource Group

This only happen when you click Apply or OK.  You can specify the computer name and click add and that works fine, it adds it to the list, but it wont apply the change.

Adding the same machine to a new group or another group works fine.

You may have notices that there is some validation that happens.  If you attempt to add a a computer that does not have a corresponding DNS entry it will generate the following error:

   The IP address lookup failed for <servername> Ensure that you have typed the name or IP    address of the computer correctly.

When attempting to add a new machiene the new entry is verified and the error is generated, but whne you click on Apply or OK, a validation process runs through all the other computer names in that group.  If a server listed there no longer exist and therefore the DNS check fails you will see the WMI error.

So the fix is simple.  Remove the computers that no longer have a corresponding DNS entry.  Once the list is now valid it will allow you to add the new machines.

If you have a big local computer group, it might be easier just to work through the list of computers by looking at the exported XML configuration.

Do do this simple select the Gateway in the MMC - Right Click - Export policy and configuration settings.

This generates an XML file you can easily work with. BUT - DO NOT edit the file - it will fail an import validation check and probably break the Resource Authorisation Policies.

13 August 2014

Everything you should know about Hyper-V CSV Cache setup, monitoring and performance

Cluster Shared Volume Cache or CSV cache is as the name implies a cache for CSVs. Very simply put, it allocates a portion of the host's RAM to act as a block level read cache for the VHDs that reside on the CSV.  This translates into significant improvement in read I/O performance, which in turn leads to overall performance improvement.

The cache to cluster to host to VM to CSV relationship
To understand the actual working of this relationship I had to spend a fair amount of time with a fairly large cluster running multiple virtual machines on a number of  CSV volumes.

Cache to cluster
The CSV cache setting is a cluster wide setting and is integrated into the failover clustering feature.  What this means is that you only need to set the cache value on a single node and it will apply across all of the members.

Cache to CSV
The cache setting is applied per CSV volume.  So if you have a 1GB cache allocated that 1GB will potentially be multiplied by the number of CSVs.

Host to VM
Cache will only be provisioned by a host for a CSV on which it is the current owner of a VM running on that CSV

So by now you would have figured out that this gets a little complicated.

Here are a few examples to help explain the concept.

In a hypothetical example imagine a three node cluster with three CSVs and 9 running VMs. and the CSV cache is set to 1GB

In the first scenario imagine that the environment is perfectly balanced where each node runs three VM each VM is on a different CSV.

On a host level this means 1GB can be allocated to each CSV = 3GB per host = 9GB for the cluster - simple

In the next example imagine that each node only has VMs running on one CSV.  Keep in mind that RAM is only allocated to cache for an active CSV

On the host only 1 CSV is active so it is 1GB RAM x 1 = 1GB per host = 3GB for the cluster

In the third example, imagine that 6 VMs are not equally distributed on single host across three active CSVs.

So the host allocates 1GB per Active CSV = 3GB

  • On CSV 1 only 1 VM is running so it has 1GB of cache available to it.
  • On CSV 2 two VMs are running so it has 1GB of cache available so it is split in two so only 512MB per VM
  • On CSV 3 three VMs are running and the 1GB of cache is split 3 ways so only 341MB per VM

In reality cache is dynamically allocated and moved between the VMs based on their activity but this should illustrate the concept.

Monitoring performance on the host
When deciding how much RAM to allocate to cache you need to consider the "worst case scenario" where that amount is multiplied for each CSV.  Setting that amount too high could potentially starve the hosts and therefore VMs of RAM.  Setting it unnecessarily low would limit the potential performance benefit.

To see what is actually going you need to have a look at Performance Monitor

The counters are under Cluster CSV Volume Cache

  • Cache state will tell you if CSV is enable for that volume
  • The Cache size counters shows the maximum configured and the actual used.
  • Cache IO counters refers to  IO that is satisfied from cache
  • Disk IO counters refer to IO that has to be satisfied form disk and not cache

When we look at this across an entire cluster the picture becomes more clear.

  • If you look at HYPERV03 (Green) you see that only volume5 is active - that is because I only have 1 VM on that hosts and that vm is running on volume5

  • If you look at HYPERV04 Volume2 (Amber) you see that all the IO Read/Sec is satisfied from cache

  • If you look at HYPERV04 Volume4 (Blue) you see that all the IO Read/Sec is satisfied from disk

  • If you look at HYPERV05 volume 4 (Purple) you can see that IO Reads are split between cache and disk

Measuring performance in the VM
For this test I use a simple utility called ATTO Disk Benchmark http://www.attotech.com/disk-benchmark/ It provides a quick non invasive way to check your disk performance.  When looking at the images pay attention to the Transfer Rate in MB / Sec number at the bottom of the image.

In test scenario 1 the test VM is on HYPERV03 volume 5 so it has all the case available to it. So here we would see the absolute best case scenario.

In test scenario 2 the same test VM is moved to HYPERV05 where there is a fair amount for contention so this would be a real world typical example

In test scenario 3 the same test VM is moved back to HYPERV03 so it has the best possible resource available but this time I basically turned the CSV cache off by setting it ridiculously low to "16MB".

What we can see from this is that with CSV cache enabled you get a performance boot on read even on a busy CSV.  Even on a busy CSV it outperforms a dedicated non cached CSV.

How Much cache should be allocated
This is a sticky question because the answer is a very non specific answer of "it depends."  By default CSV cache is enable with 512MB of RAM allocated.  I have run the tests above a number of time with various amounts of cache configured.  What the results show is that if the cache gets exhausted the performance drops off. but while it is able to satisfy the IO requirement the same level of performance is realised regardless of the amount allocated.

What it comes down to is how well you know your environment.  If you have very few CSV in a cluster you can allocate a larger amount.  When you have more CSVs you need to keep in mind the multiplying effect of this and accordingly reduce the amount of cache.

Recommendation from Microsoft is a minimum of 512MB and no more than 64GB

Other factors to consider
The CSV cache can you you a very nice read IO performance increase but it is far from being the only performance factor.  The shared storage type, speed and contention also plays a significant role.

Write heavy VM will not benefit much from cache but read  heave ones would.

VM storage placement and host allocation can have a significant impact.

Enabling and adjusting the CSV cache.
This first became available in 2008R2 but was not enabled by default. By 2012 R2 it is and has a default allocation of 512MB.

Your CSV volumes MUST be formatted with NTFS.  If you use ReFS you will not be able to use CSV Cahe.

To adjust the amount of cache allocated you use the following PowerShell command.  Remembering it only needs to be done once on a single node it the cluster.

(Get-Cluster). BlockCacheSize = 1024

The number being the amount of MB

You can change this number on the fly and no reboot or anything like that is required.

CSV Cache is a great technique for improving performance of Hyper-V VMs.  Understanding how it works and why it works allows you to tune this feature to your environment to get the best benefit.