Thursday, September 12, 2019

Citrix XenServer: Create new Storage Repository from Newly Added Disk

Unable to find Newly added Local HDD on Citrix XenServer Management Console to create Storage repository.


Symptom:

Whenever we add new Hard drive to Xenserver Hypervisor Host to create new Storage repository, it will not be visible on XenCenter console to create the new repository.
It will give the message that, "No LUNs were found, Please verify your Hardware Configuration"




Cause:

XenServer will not recognize the new disk automatically from Add storage repository wizard on XenCenter Console,We have to add storage repository from command line.

Resolution:

1. Connect Host via SSH.
2. Enter "fdisk -l" command to identify new disk
3. Note the disk path for new disk, example: /dev/sdb
4. Execute command 
    xe sr-create name-label=<Storage ID> shared=false device-config:device=<Path of the Storage device> type=lvm content-type=user
Note: Give repository name instead of <Storage ID> and disk path instead of <Path of the Storage device>




5. Now newly added Storage repository will be visible on XenCenter Console.





Below is the Citrix Kbase Article for this behavior

https://support.citrix.com/article/CTX130897





Tuesday, September 3, 2019

ESXI Login Failure: Cannot Complete Login Due to an Incorrect User Name Or Password" message on ESXi login

"Cannot Complete Login Due to an Incorrect User Name Or Password" message on ESXi login, ESXi root has been locked due to multiple failed login attempts.


Symptoms:


VMware ESXi 6.0 and later versions showing message "Cannot Complete Login Due to an Incorrect User Name Or Password" while trying to login through vSphere Client and web client, even though we provide the correct credentials.

Same behaviour shows on ESXi 6.5 and ESXi 6.7 as well.

If you try to login Direct Console Interface (DCUI), ESXi will allow you to login with root account.


Login to DCUI, Press "F2", Go to  "View System Logs", choose "vodb" logs, Type "/locked" inside the log screen. It will display similar to below.



ESXI-ROOT-HAS-BEEN-LOCKED-FOR-900-SECONDS-VODB


Cause:


The ESXi "root" account is getting locked out due to multiple failed login attempts.

It may be because someone is trying to login to via ssh or vSphere client multiple times with wrong credentials.
Also there may be some application is integrated with ESXi with old or wrong credentials, Example: Backup Software or monitoring software.

Solution:


1) Login to ESXi Direct Console Interface (DCUI) with "root" account.

2) Press "F2", go to "Troubleshooting Options".
3) Enable "ESXi Shell" using "Enable ESXi Shell" Option and go back to main menu.
4) Press "CTRL+ALT+F1" to switch to "ESXi Shell"
5) Type Command "pam_tally2 --user root" to view total failed login attempt from root. 
from below example, There are 14 failed login attempt.


pam_tally2-user-root-esxi

6) Type Command "pam_tally2 --user root --reset" to reset the lock.



pam_tally2-user-root-reset-esxi

7) Now we will be able to login to ESXi via vSPhere Client or web client.


ESXi events showing the failed login attempts and "root" locked events.



ESXI-EVENTS-ROOT-LOCKED-FAILED-LOGIN-ATTEMPTS



Prevent The Account Lock Again

1. Login to ESXi via SSH (Enable SSH from ESXi settings if SSH not enabled)

2)  type Command "tail -100 /var/log/auth.log" to view last 100 events on auth.log
3) The result showing the source IP from where the failed login attempt is originated, In below example, the IP is "192.168.111.1"

2019-09-03T13:09:57Z sshd[2101425]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.111.1  user=root
2019-09-03T13:09:59Z sshd[2101419]: error: PAM: Authentication failure for root from 192.168.111.1
2019-09-03T13:09:59Z sshd[2101419]: Failed keyboard-interactive/pam for root from 192.168.111.1 port 64744 ssh2
2019-09-03T13:09:59Z sshd[2101426]: pam_tally2(sshd:auth): user root (0) tally 6, deny 5

4. Identify the source machine and resolve the reason of multiple login attempt. 
Example: 
If the source IP is for backup software, reconfigure backup with latest ESXi Credentials.

Monday, September 2, 2019

Shutdown Simplivity OmniStack Virtual Controller (OVC) procedure.

Shutdown or restart Simplivity OmniStack Virtual Controller (OVC) procedure.


Purpose:

The steps for grace full shutdown or reboot of Simplivity OmniStack Virtual Controller (OVC).

Steps:

1. Login to Simplivity vCenter Web Client.
2. From Home page, Navigate to " Simplivity Federation"

Simplivity-Federation-Vmware-Webclient
Simplivity Federation Menu on Vmware Web client

3. From, " Simplivity Federation" page, Click on "Hosts"


Simplivity-Federation-Vmware-Webclient-host-cvm-reboot

4. From the right Side pannel, right click the host on which OVC need shutdown, 
   go to " All Simplivity Actions", Click "Shutdown Virtual Controller"


Simplivity-Federation-Vmware-Webclient-host-cvm-shutdown

5. Once OVC is shutdown completely, Go to Virtual Machine Inventory and Power on the OmniStack Virtual Controller VM manually.

Simplivity Cluster Showing Datastore inaccessible or inactive in VMware Vsphere Client.

Simplivity Cluster showing Virtual Machine (VM) and Datastore in inaccessible state in VMware Vsphere Client.


Symptoms:

In Simplivity Cluster, one more more datastores status is showing inaccessible or inactive for one ore more ESXi Hosts.One more more VMs in the Simplivity cluster showing status "inaccessible".

Also ESXi showing events similar to below.

cpu22:3XXXXX)WARNING: NFS: 3XX: Lost connection to the server omni.cube.io mount point
/mnt/svtfs/0/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX, mounted as XXXXXXXXXXXXXXXX-0000-000000000000 ("SUN_L000")
cpu8:XXXXXX)StorageApdHandler: 265: APD Timer started for ident [25fdcdf9-d882ff30]
cpu8:XXXXXX)StorageApdHandler: 414: Device or filesystem with identifier
[25fdcdf9-d882ff30] has entered the All Paths Down state.

Work Around:


1. First trace the affected ESXi host by verifying each ESXi Datastore and VM status where it showing the status "inaccessible".
2. Restart the OmniStack Virtual Controller (OVC) on the ESXi host that cannot access the datastore.

Note: If the Datacenter have 2 or more OmniStack hosts, migrate all VMs to another Host to perform activity without disruption to VM. If Datacenter have only one OmniStack host, VM should be shutdown before performing any activity and there will be a disruption to the VM.

Permanent Solution:

For OmniStack Software running with v3.0.10, Upgrade the OmniStack Software Version to v3.0.10 Update 2.

For OmniStack Software running with v3.5.1, Upgrade the OmniStack Software Version to v3.5.2 or later.


Grace Restart of the OmniStack Virtual Controller OVC Procedure