Customer notice: Citrix ADC Vulnerability

Citrix ADC vulnerability - what you need to know

As you may be aware, there is currently a significant Citrix ADC security vulnerability. Thomas Duryea Logicalis (TDL) strongly encourage all our Citrix customers to immediately apply the necessary mitigation steps.

TDL urge all Citrix customers to consider this an extreme threat that requires immediate action. This vulnerability poses a significant risk and is being actively exploited. TDL have already assisted several customers who have had their ADC breached and observed connections going to known malicious IPs.

What is the vulnerability?

A critical security vulnerability in the Citrix ADC (Application Delivery Controller - formerly Netscaler) which could allow an unauthenticated attacker to perform arbitrary code execution on an organisation’s network.

The vulnerability has been assigned the following CVE number:

  • CVE-2019-19781: Vulnerability in Citrix Application Delivery Controller and Citrix Gateway leading to arbitrary code execution

The vulnerability affects all supported product versions and all supported platforms:

  • Citrix ADC and Citrix Gateway version 13.0 all supported builds
  • Citrix ADC and NetScaler Gateway version 12.1 all supported builds
  • Citrix ADC and NetScaler Gateway version 12.0 all supported builds
  • Citrix ADC and NetScaler Gateway version 11.1 all supported builds
  • Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds
  • Citrix SD-WAN WANOP product versions 10.2.6 and version 11.0.3

What Citrix customers need to do

  1. TDL and Citrix strongly urge customers to immediately apply the provided mitigation described in CTX267679 .
  2. Apply permanent fixes as they are made available. 

    Permanent fixes are currently available for two versions of Citrix ADC, specifically versions 11.1 and 12.0. Schedule for fixes for other releases is available here.

     These fixes are available via download at the links provided below.

  3. TDL have observed that it is likely the ADCs that did not have the mitigations applied before 8/1/2020 are likely to have been probed for the security vulnerability and are probably compromised

    Customers should consider a clean rebuild of the ADCs (or restore from a VM backup that was taken before 8/1/2020) to sanitize the file system and bring it back to being a “known good” state.

    By compromised we mean that an attacker may have:

    1. obtained full admin access to the device
    2. modified its configuration
    3. downloaded sensitive configuration information (ns.conf file which includes sensitive information such as password hashes),
    4. downloaded certificates and private keys
    5. ran packet traces to capture user account passwords
    6. installed crypto miners and backdoors to obtain access to the internal network
  4. Customers should perform basic forensic analysis on their ADC to determine what activity has occurred.

      a. Check the log files on the appliance for malicious activity all the way back to 8/1/2020.
    1. Especially check: bash.log / sh.log / notice.log / httpaccess.log / httperror.log / ns.log files:
      • Look for shell commands that shouldn’t be there
      • Look for /vpns/cfg/smb.conf – triggers the exploit or is a probe attempt
      • Look for cron jobs that shouldn’t be there
      • Look for http POST actions that upload scripts to locations such as /vpn/../vpns/portal/newbm.pl
    2. Check for persistent outbound traffic (NetScaler > Internet) that is not expected – there might be traffic going off to known malicious IPs.
  5. Customers should modify / replace items that an attacker may have gotten access including:
    1. Change the nsroot (admin) password.
    2. LDAP Bind account - change the LDAP account to a new account with a new password. The ns.conf file contained the AD LDAP account used and a hash of the password. It might be possible for the attacker to derive the password from this hash.
    3. RADIUS Shared Key - change the RADIUS shared key. The ns.conf file contained a hash of the shared key and it might be possible for the attacker to derive the password from this hash.
    4. SSL Certs - revoke and reissue the certificates. The attacker might have downloaded the TLS certificate in PFX format and had details of the decrypt password when they downloaded the ns.conf file.
    5. Change the RPC node passwords

Need assistance?

With TDL’s expertise in Citrix we can assist customers by implementing the mitigation code, patches and performing cleanup / rebuild activities, so that your business can feel confident this critical security vulnerability is mitigated.

Please contact your TDL Account Manager for an estimate on the time and effort required to remediate this vulnerability.  If you do not know who your account manager is, then please contact your local TDL office on the phone numbers provided in the following link:

https://www.tdlogicalis.com.au/about-us/office-locations/

For further information please read the Citrix update on this matter:

Citrix Update

References