Sunday, March 15, 2020

VMware vCenter 6.0 : 503 Service Unavailable: Failed to connect to endpoint


Problem

VMware vCenter 6.0 Web Client Showing error:

"503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f3d084a60c0] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)"


503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f3d084a60c0] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)"
Symptoms:

1) VMware vCenter Web Client login page is not loading. It is throwing error,
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f3d084a60c0] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)”
2) VMware Vcenter Inventory Service is not starting. 
While trying to start service manually with command “service-control --start --all”, its throwing error, 
Stdout = Starting VMware Inventory Service...
Waiting for VMware Inventory Service...............
WARNING: VMware Inventory Service may have failed to start.

3) /var/log/vmware/vmdird/vmdird-syslog.log shows the log similar to below


2018-04-21T14:47:41.0241248+00:00 err vmdird k@999123451949876: Bind Request Failed ([17] 172.17.xx.xx:389<-172.17.xx.xx:54753) error 49: Protocol version: 3, Bind DN: "cn=VCENTERSERVER.vmware.local,ou=Computers,dc=vsphere,dc=local", Method: 163


Cause:


The reason of this issue due to password mismatch in vmdird for the account specified in vmdird-syslog.log. This occurs when you restore the vCenter server from old backup or snapshot.


Resolution:


To resolve this issue, we need to reset the password of account specified in the vmdird-syslog.log.

On the above example, the account name is 
VCENTERSERVER.vmware.local@vsphere.local

Steps:


1. Capture snapshot for vCenter Server appliance VM, create snapshot for platform service Controller if PSC is external.

2. Login to Appliance via SSH with root (Login to PSC if PSC is external)
3. Run command “shell.set –enabled true” to enable Bash shell.
4. Type “Shell and press enter.
5. Type “/usr/lib/vmware-vmdir/bin/vdcadmintool” to open vdcadmintool tool.
6. You will get below options 
================================
Please select:
0. exit
1. Test LDAP connectivity
2. Force start replication cycle
3. Reset account password
4. Set log level and mask
5. Set vmdir state
================================
Select Option "3".
7. Enter the account name you want to reset. In our example, the account is "VCENTERSERVER.vmware.local@vsphere.local"
8. Copy and note the auto generated password. And press "0" to exit this mode.
Note: This tool will not show the proper password if some of special characters included in password and it will see an blank space on password output. If you observe space on auto generated password, repeat Step 7 till you get a valid password.

9. Connect to Vcenter server via SSH with root credential if PSC is external. If embedded, skip this and go to “step 12”.

10. Type “shell.set –enabled true” Press enter
11. Type “Shell” Press enter.
12. Run below commands one by one to update new password. Replace "new password" with the new password we generated.

/opt/likewise/bin/lwregshell
cd HKEY_THIS_MACHINE\services\vmdir\
set_value dcAccountPassword "new password"
quit

13. Restart the vCenter Server Appliance services by below commands.

service-control --stop --all
service-control --start --all