Citrix PVS: Write Cache filling up quickly

 

Citrix PVS Write Cache filling up Abnormally and Quickly

  When we are dealing with Citrix PVS image streaming, one of the most common issues most of the Citrix Admins faces is high "Write Cache Usage". In ideal cases write cache will consume from few MBs to couple of GBs of space at fresh state and it will grow as per the user activity. But consumption is all depends up on the image, apps and other technologies using in the environment. but still if the write cache is filling up more quickly at fresh boot itself and growing abnormally, definitely the Citrix admin should look at to this.

Below are the few tips to reduce the Citrix PVS Write Cache usage

1. Disable automatic updates for OS and all possible applications. Some of the application we cannot disable auto updates directly, but we can achieve this using GPOs, example : Chrome. Some of the apps like "Adobe reader DC" auto update can be disable by disable update task from "Task Scheduler". 

2. Verify "Task Scheduler" and disable all unwanted tasks, example: Adobe reader updates, One Drive Updates etc. 

3. Go to "Task manager" and Go to "Start up" tab. Disable all start up applications items which is not really need to initialize during startup. example: Java update scheduler, MS Teams etc.

4. If Write Cache is getting full more than 20% during fresh boot itself, it means huge changes are happening in machine during/after boot. One solution for this behavior is,

Put the image in private mode or make a PVS version of the image. Boot the image in maintenance mode and login with local account, Keep it login for couple of hours and then shutdown the machine. So all changes will be written on the vDisk. Then stream the devices with updated image. Observe now still write cache is getting full during the fresh boot itself or not.

5. If the vDisk image is made with Applayer, before streaming to the devices, boot the image once in private mode, login with local account, keep it login for couple of hours, then power off and convert to standard image mode and stream to the devices to reduce the write cache usage.

VMware VM Power On Error: Device Bootstrap is not available

  

Problem:

Unable to Power on VMware Virtual Machine, Giving error "Device Bootstrap is not available"

In VM event tab, it will show event similar to below:

Error message from MYESXIX001.MYCOM.MYDOMAIN.COM: Cannot open the 

configuration file /vmfs/volumes/5f043cc5-8e039f06-05eb-0025b501a000/DCPSQLFDB23/MYVM001.vmx

Reason:

The VMX or VMX~ is locked by one of the esxi in the cluster. Even if we move the VM in the particular Host, VM will fail to power on.

Solution:

Identify the ESXI host which is currently have lock on the VMX or VMX~ with below command. 

[root@I-MY-ESXI-01: vmfsfilelockinfo -p /vmfs/volumes/152525252-26252526122-ED3455555555/MYVM001/MYVM001.vmx -v 172.11.13.3 -u administrator@vsphere.local

OR

[root@I-MY-ESXI-01: vmfsfilelockinfo -p /vmfs/volumes/152525252-26252526122-ED3455555555/MYVM001/MYVM001.vmx~ -v 172.11.13.3 -u administrator@vsphere.local

### -v  <vcenter IP>

### -u  <vCenter credential>

Output:

vmfsfilelockinfo Version 1.0

Looking for lock owners on "MYVM001.VMX"

"MYVM001.VMX" is locked in Exclusive mode by host having mac address ['00:20:b2:01:10:01']

-----------------------------------------------------------------

Found 2 ESX hosts from Virtual Center Server.

------------------------------------------------------------------

Searching on Host MYESXI001.MYCOM.MYDOMAIN.COM

Searching on Host MYESXI002.MYCOM.MYDOMAIN.COM

    MAC Address : 00:20:b2:01:10:01

Host owning the lock on the VMX is MYESXI002.MYCOM.MYDOMAIN.COM, lockMode : Exclusive

Total time taken : 0.38 seconds.

   Above output saying, disk is locked by MYESXI002.MYCOM.MYDOMAIN.COM with MAC Address 00:20:b2:01:10:01.

Steps to Clear ESXi Lock:

1. Put the host in maintenance.

2. reboot the host.

Lock will be cleared during ESXi reboot and we will be able to power on or consolidate snapshot of VM.

Find an installed application/driver versions on multiple Windows Servers Via PowerShell Script.

 Purpose:

This article explaining the steps to collect installed driver/application version from multiple Windows servers via PowerShell.

Steps:

1. Create a folder as "c:\scripts" in any of the server in network. 

2. Create a txt file called "c:\scripts\serverlist.txt" and list all server hostname FQDN as below

        

3. Create a PowerShell script file "c:\scripts\Appversioncheck.ps1" and copy below texts.

########################################################

$serverlist = @(get-content -Path "c:\scripts\serverlist.txt") 

foreach ($server in $serverlist)

{

$virtioinfo = Get-WMIObject -computername $server -Query "SELECT * FROM Win32_Product Where name Like '%Google Chrome%'" | select Name, Version 

$virtioversion = $virtioinfo.version 

write-host "$server,$virtioversion"

} 

##########################################################

Note: Edit the application name as per your need.

4. Run this PowerShell script, it will give output with hostname and its application version.

Adobe Reader DC: An error Occurred that may be fixed by installing the latest version of the Extended Language Support Package.

 Problem:

Adobe reader showing error "An error Occurred that may be fixed by installing the latest version of the Extended Language Support Package" while opening documents which have Arabic or other Asian languages.

Reason:

This error is occurring due to the unsupported fonts contained in the PDF document.

Solution:

Download and install latest Adobe DC Extended Language Support Package from below URL.

https://helpx.adobe.com/acrobat/kb/windows-font-packs-32-bit-reader.html

VMware PowerCLI: Modify ESXi Hosts Primary and Secondary DNS Servers using Command line.

VMware ESXi DNS Server Configuration/Modification Using PowerCLI

We can easily change Primary and Secondary DNS Servers of all ESXi servers in a vCenter via below simple steps.

Steps:

1. Download and install VMware PowerCLI on any of the Windows machine in network. If there is a firewall between PowerCLI installed machine and vCenter, Port 443 should be allowed in firewall. In my case I installed PowerCLI in vCenter server itself.

PowerCLI Download URL

2. Launch PowerCLI by right click PowerCLI shortcut and choose "Run As administrator"

3. Execute below Command first to connect PowerCLI to vCenter Server.

Connect-VIserver 172.25.1.10 

        Note: Change vCenter IP with your actual vCenter IP

4. copy and paste below commands by editing DNS server IP with your actual DNS Server.

$dnsaddress = @()

$dnsaddress += "192.168.111.10"

$dnsaddress += "192.168.111.11"

Get-VMHost | Get-VMHostNetwork -ErrorAction SilentlyContinue | Set-VMHostNetwork -DnsAddress $dnsaddress

5. The command will print the output with the details of each ESXi hosts and its new DNS server.

vCenter 6.0: VCSA Appliance is not booting, Stuck at progress bar

 Problem:

1. vCenter 6.0: VCSA Appliance is not booting, Stuck at progress white bar for long time.

2. Unable to login with root password in VCSA appliance even after reset the root password, it saying wrong credentials.

3. /dev/sda3 is full while monitoring from rescue mode.

Reason:

This behavior could be because of VCSA file system is full due to audit.log. Most of the time /dev/sda3 will become full due to audit.log. This will affect VCSA booting and even root account new password may not be able to reset.

Solution:

Clear audit.log by login to rescue mode.

1. Open VCSA console and Reboot VCSA appliance VM.

2. Press "Space bar" on GRUB screen to avoid auto boot.

3. Press "p" to choose boot options, enter GRUB password, default password is "vmware"

4. Select "VMware vCenter Server appliance" and press "e"

5. Select the line start with "Kernel /vmlinuz...." and press "e"

6. Append "init=/bin/bash" at end of last line after putting a "space"

7. Press enter, then type "b" to boot.

8. Once it is booted, go to the file location of audit.log by executing command "cd /var/log/audit"

9. Execute command "truncate -s 0 audit.log" to clear the audit.log.

10. Verify the disk utilization by running command "df -h"

11. Reboot the VCSA appliance by running below commands.

mkfifo /dev/initctl

reboot -f

VMware Applvolume : Error From Manager (Error Code 401) Invalid Session Cookie: Session Key Does Not Match Any Active Sessions

 

Problem:

VMware Applvolume giving error "Error From Manager (Error Code 401) Invalid Session Cookie: Session Key Does Not Match Any Active Sessions". Users are failed to connect the AppStack.

Reason and Solution

Reason 1:

As per VMware knowledge base, the issue is happening because of additional authentication calls for Agent-Manager communications and solution is to disable "Agent Session Cookie feature"

Refer KBASE for more details.

Reason 2:

I faced this issue recently, after long troubleshooting I identified issue was with vCenter. vCenter was not accepting any users to login, Then I identified VCSA appliance was out of disk space and I fixed issue by clearing the audit.log.

Clearing Audit.log in VCSA Appliance

vSphere 6.7: VM Name is shows as /vmfs/volumes/xxxx, Unable to Power On VM, VM Files showing .vmx.lck and .vmx~

Problem:

Unbale to power on few virtual machines. VM names got changed suddenly with name similar to /vmfs/volumes/xxxx.

While browsing datastore, .vmx file icon is not showing as virtual machine file. Also the files .vmx.lck and .vmx~ are visible in virtual machine folder.

Reason:

This issue occurs whenever .vmx or vmdk is locked by any other esxi hosts. This locks preventing the files to lock by another esxi hosts.

Normally VM files are locked by host during power on and locks are released during power off. But some situations locks are not getting released even after VM is off and this issue occures.

Solution:

Identify which host is making this lock and clear the locks by killing the process or rebooting the esxi hosts. Some situations we cannot see any running process related to this locked file, in that situation only solution is to reboot the host.

Steps to Identify Lock:

1. Login to esxi host using SSH.

2. Execute command  

vmfsfilelockinfo -p <File path staring with /vmfs/volumes....> -v <vCenter IP> -u "administrator@vsphere.local"

Example:

vmfsfilelockinfo -p /vmfs/volumes/4444456-f3c44426-030f-3622222f000de/VM-0001/VM-0001.vmx -v 172.20.20.12 -u "administrator@vsphere.local"

3. This command will list the ESXi Host which have lock on this file. Identify the ESXi host and reboot it.