By far the most common Linux VDA support issue is the VDA not registering. This is often caused by misconfigured Active Directory integration or Kerberos authentication issues. In this article, I will endeavour to explain how the Linux VDA uses Active Directory and Kerberos, and what commonly goes wrong.
A nice feature of the Linux VDA is that it slots into the same StoreFront, Delivery Controller (Broker) and Active Directory infrastructure you already have in place. The familiar user SSO experience of logging on via StoreFront and getting access to apps and desktops without being re-prompted for credentials applies equally to Linux VDA.
The challenge with Linux is that it is not Windows, and making it play nicely in a Windows domain environment is inherently challenging. Whether you have selected Winbind, Quest, or Centrify as your Active Directory integration platform, or indeed gone out on a limb and tried SSSD or PowerBroker, the requirements for a clean Linux VDA setup remain the same.
There are other some other great sources of information on registration issues for the Windows VDA, including support articles CTX126992 and CTX126992.
A lot of this information is still relevant to the Linux VDA and worth a read. This article builds on these and adds the Linux flavour.
Hostname
Bad host names, that are too long or contain invalid characters, can cause either very obvious, or subtly annoying, problems in an Active Directory environment. Windows is strict with hostnames; however, many Linux distros are dangerously more liberal what they will allow. Best practise is:
- Limit hostname length to 15 characters or less to avoid NetBIOS name truncation
- Use a-z, A-Z, 0-9 and hyphen (-) characters only, especially avoid underscores, whitespace and other symbols
- Don’t start a hostname with a number and don’t end it with a hyphen.
The short-form hostname must be different from the FQDN which includes DNS domain name in dotted notation. Running the command hostname should display a short name such as myvda01, whereas running hostname -f should display its corresponding FQDN such as myvda01.workers.mycorp.net.
As the hostname and SAM Account Name of the computer in AD must match, if you ever change the hostname, you will need to rejoin the VDA the domain. Clearly it is best to get your hostname sorted out before doing anything else.
DNS
Bad DNS configuration is a common problem. Before proceeding it is good practise to ensure your VDA machine’s network settings are pointing at the correct DNS servers and your DNS servers are returning sensible results. Check that you can query the VDA’s DNS domain name. If in a multiple domain or cross forest environment, also check the other domain names. For example:
nslookup mycorp.net nslookup workers.mycorp.net nslookup central.mycorp.net
Multi-homed VDAs
For Linux VDA 1.0, support for multiple network interfaces was limited and should be avoided. The selection of network interface and IP address for communicating with the Broker was non-deterministic, which can lead to irregular registration failures.
This problem has been fixed in Linux VDA version 1.1. A configuration setting was added to allow you to specific which network interface to use for communication with the broker. For example, to make eth0 your primary network interface:
sudo ctxreg create –f -k HKLM/Software/Citrix/VirtualDesktopAgent \ -v PrimaryNetworkInterface \ -t REG_SZ \ -d eth0 sudo service ctxvda restart
If you are using multiple network interfaces (excluding loopback interfaces) on your VDAs, it is highly recommended you set this. If you leave the setting blank, the VDA will fallback to the v1.0 behaviour of unpredictable selection.
Domain Membership
Perhaps obviously, for the Linux VDA to register with the Broker, it must be joined to the domain. The join process varies between AD integration products, but essentially the end result is the same – a computer object for the VDA has been created in AD. Without being joined the domain, the VDA and Broker will not be able to establish a security context, and thus will not be able to communicate with each other.
There is nothing particularly special about a computer object in AD for Linux machines. It has a SID, a DNS name, a SAM Account Name, and the other attributes you would expect to see on a Windows computer object. If you are using Winbind, there are various commands you can run to check that the machine is joined and the computer object attributes are valid:
sudo net ads testjoin sudo net ads info sudo net ads status
Most of the other AD integration products provide similar command tools.
Cross Domain and Cross Forest
In complex environments with multiple trusted AD domains, either, the VDAs and Delivery Controllers must reside in the same domain, or reside in domains with a 2-way trust relationship. As of Linux VDA 1.1, these trust relationships may cross forest boundaries, using either forest or external trust types.
For HDX session authentication, user accounts must reside in the same domain as the VDA or in a outgoing trusted domain. Stated differently, the VDA is trusting of the users’ domain but the users’ domain does not necessarily need to trust the VDA domain.
For more information, look for the Active Directory planning guides for deploying the Linux VDA into complex AD environments.
Time Synchronization
Kerberos is very time sensitive. If clocks drift between servers by more than a few minutes, the Kerberos authentication system breaks down. It is important that all servers and domain controllers are time synchronized with each other.
A common architecture is to have all domain controllers synchronised either from a common internal time server or from the public pool time cluster, and have all domain member servers synchronized with the domain controllers.
For virtual machines, it is general best practise that time be synchronized with network time servers rather than rely purely on the hypervisor time synchronization.
For any server synchronising with public time servers, make sure you select a cluster that is geographically close. It has been observed that NTP can stop synchronizing time if network latency or jitter is too high.
System Keytab File
The VDA and Broker use Kerberos to mutually authenticate and secure communication with each other. Each side communicates using the identity of the computer account, removing the need for the admin to create service accounts in Active Directory. On Windows, the Kerberos keys used to securely identify the machine and authenticate with the domain controller are stored and managed by the LSA service. The equivalent on Linux is the system keytab file, usually called /etc/krb5.keytab.
The keytab file is typically created when the Linux machine is joined to the domain and only accessible by the root user. Note if you are using Winbind, the default configuration will not create the keytab file by default. Before joining the domain, ensure the following is set in /etc/samba/smb.conf:
kerberos method = secrets and keytab
You can verify the list of keys available, including when the keys were created, by running as root:
klist -ket
Note that since the VDA to Broker communication is bi-directional, where either side can initiate or accept connections, the VDA will use both client (UPN) and service (SPN) keys of the computer account. UPN keys (e.g. MYVDA$@WORKERS.MYCORP.NET) are used to establish a security context with the Broker, and SPN keys (e.g. host/myvda.domain.net@WORKERS.MYCORP.NET) are used to accept connection requests from the Broker.
For the VDA to decipher incoming messages, is important that the FQDN of the VDA matches the FQDN contained within the SPN exactly. The FQDN in the SPN is after the host/ prefix and before the @REALM suffix.
hostname -f
If for example, an SPN for the VDA is host/myvda.domain.net@WORKERS.MYCORP.NET, then the above command must return myvda.domain.net. If the names mismatch, VDA registration will fail. It might be necessary to rejoin the machine to the domain to correct any naming issues.
Kerberos Server Location
It is important that Kerberos on the Linux VDA has been adequately configured to locate its own domain’s domain controllers, and if working cross-domain, the domain controllers of the Broker’s domain. This is achieved by having domain name to KDC server mappings. These mappings are either statically configured in /etc/krb5.conf on each VDA, or queried via DNS SRV records.
For example, the following /etc/krb5.conf file demonstrates how a VDA, residing in the workers.mycorp.net domain, might be configured for communicating with Broker services in the foreign central.mycorp.net domain.
[libdefaults]
default_ccache_name = FILE:/tmp/krb5cc_%{uid}
default_realm = WORKERS.MYCORP.NET
dns_lookup_kdc = false
[realms]
WORKERS.MYCORP.NET = {
kdc = primarydc.workers.mycorp.net
kdc = backupdc.workers.mycorp.net
}
CENTRAL.MYCORP.NET = {
kdc = primarydc.central.mycorp.net
kdc = backupdc.central.mycorp.net
}
[domain_realm]
workers.mycorp.net = WORKERS.MYCORP.NET
.workers.mycorp.net = WORKERS.MYCORP.NET
central.mycorp.net = CENTRAL.MYCORP.NET
.central.mycorp.net = CENTRAL.MYCORP.NETThe mapping from DNS domain name to KDC servers is via two hops. The DNS domain name maps to the Kerberos realm name (under the [domain_realm] section), and the Kerberos realm name maps to the KDC servers (under the [realms] section).
Configuring this on every VDA is onerous and does not scale well. The better option is to use the DNS SRV records that were created by domain controllers. These records are identified by adding the prefix _kerberos._tcp. to the DNS domain name. For example, to verify whether the DNS SRV records for both VDA and Broker domains are available, run the following:
host -t SRV _kerberos._tcp.workers.mycorp.net host -t SRV _kerberos._tcp.central.mycorp.net
These commands should list all of the domain controllers capable of providing Kerberos ticket distribution services for the two domains. If no SRV records are returned, this might indicate DNS service records are not published for the domain, or there is some other underlying DNS problem. It might be necessary to consider using static configuration in the /etc/krb5.conf file instead.
If you wish to use the DNS SRV records, open /etc/krb5.conf and make the following change under the [libdefaults] section:
dns_lookup_kdc = true
Testing Kerberos
The easiest way to ensure your VDA is properly joined to the domain, your system keytab file is valid, and the Kerberos services are locatable and operational, is to manually authenticate the computer account. For our fictitious myvda.workers.mycorp.net VDA, we run:
sudo kinit -k MYVDA\$@WORKERS.MYCORP.NET
Since WORKERS.MYCORP.NET was configured as the default realm, we could have saved ourselves some typing by omitting the suffix:
sudo kinit -k MYVDA\$
If all goes well, the TGT ticket was freshly minted by the KDC and cached by the VDA. To view the cached ticket, run:
sudo klist
The starting time of the ticket should reflect the time kinit was run.
Ports
The default port for VDA to Broker communication is TCP port 80. As connections are established in both directions, this incoming port must be open on the firewall both on the Broker and VDA. If you are using a port other than the default, be sure to specify this port as part of the ctxsetup.sh configuration.
To verify the configured port use the ctxreg tool:
ctxreg -k HKLM/Software/Citrix/VirtualDesktopAgent \ -v ControllerRegistrarPort
This will display the port in hex. A value of 0x00000050 is 80 in decimal.
Port 1494 should also be open on the firewall for accepting HDX session traffic from the Receivers.
Test Controller Endpoint
To test whether the Broker service is reachable, listening and processing requests on its configured port, you can issue blank HTTP POST requests at the Broker’s Registrar service. A simple way is to use the curl tool:
curl -i -d "X" \ -H "Content-Type:application/soap+xml" \ -H "Expect:100-continue" \ http://mybroker.central.mycorp.net/Citrix/CdsController/IRegistrar
If the first line displayed is HTTP/1.1 100 Continue, then the Broker service responded. This will be followed by a HTTP/1.1 400 Bad Request response which can be ignored.
Another way to probe the Broker is use the new Linux XDPing tool, which I will describe later.
Rejoining the domain and stale service tickets
Under certain circumstances, when a VDA is rejoined to the domain and a fresh set of Kerberos keys are generated, the Broker fails to establish a security context with the VDA. This is often caused by the Broker using a cached out-of-date VDA service ticket based on the previous set of Kerberos keys. This won’t stop the VDA from connecting to the Broker, but the Broker will not be able to establish a secure connection back in the opposite direction to the VDA. The usual symptom, VDA registration fails.
This problem will eventually resolve itself when the VDA service ticket eventually expires and is renewed, but service tickets are usually long-lived. This could potentially be hours.
If you are experiencing registrations problem after rejoining the VDA to the domain, the solution is to clear the Broker’s ticket cache. You could simply reboot the Broker, but a less drastic measure is to run the following on the Broker from a command prompt as Administrator:
klist -li 0x3e4 purge
This will purge all service tickets in the LSA cache held by the Network Service principal under which the Citrix Broker Service runs. This will remove service tickets for other VDAs and potentially other services. However, this is harmless – these service tickets will simply be reacquired from the KDC when needed again.
LDAP Configuration
The Linux VDA will query its domain controllers to obtain information about the computer it is running on. Prior to Linux Virtual Desktop 1.1, this was done indirectly by calling specially-crafted Winbind, Quest or Centrify shell scripts. As of version 1.1, this was rationalized into a single direct LDAP query, obviating the need for specific AD integration product scripts.
For either method, this will typically just work without any Administrator intervention. However, if you are using Linux Virtual Desktop 1.1 or later, there are a few LDAP service requirements to be aware of:
- The LDAP service must be running on your domain controllers
- The LDAP service must be hosted on default port 389
- LDAP tunnelled via SSL/TLS (i.e. LDAPS) is not supported
- The LDAP service endpoints must be discoverable via DNS SRV records
Testing the last item is similar to locating Kerberos service endpoint described earlier. For example:
host -t SRV _ldap._tcp.workers.mycorp.net
If you are experiencing registration problems, look out for any LDAP errors in the VDA log files.
Known Bug
If you are using the older Linux VDA 1.0 release, be aware there is a known bug when using the OpenJDK 1.7.0.85 or later that will cause registration failures. The resolution is either upgrade the VDA to the v1.1 release or downgrade the OpenJDK back to 1.7.0.79 or earlier.
Soft Registration
If the VDA logs indicate that only “soft registration” was achieved, this likely means the Linux VDA has not been added to a machine catalog or delivery group. Soft registration means the VDA and Broker are communicating securely with each other, but are not yet setup for delivering sessions.
If your VDA is experiencing soft registration after rejoining the domain, whereas before your VDA was hard registered, either a Controller configuration change was made or the VDA has taken on a new AD identity after rejoining to the domain. You may have to re-add the new VDA computer object to the machine catalog and delivery groups.
Linux XDPing
Based on the success of the Windows XDPing tool, developed by Citrix Support, we have developed our own validation tool for Linux environments. Taking some artistic license from our colleagues in Support, we called it the Linux XDPing tool. This will run extensive tests on the system and is invaluable at detecting common issues, including many of items described in this article.
Summary
That covers the most common registration issues we know about. Anything we missed?
To read more from the Linux Virtual Desktop Team, please check out the rest of our posts here.