Gateway Antivirus Issue: Root Cause Analysis

Summary:
WatchGuard had an issue with Gateway Antivirus (GAV) signatures on February 7th , 2018 that affected M300, M200, and T35 appliances running firmware version 12.0 and later. An error in the signature file resulted in scan errors on the appliance. Depending on the appliance configuration, the Firebox may have prevented access to files received by email.

Root Cause
The issue was specific to PowerPC processor with 64-bit architectures (i.e., Firebox M300, M200, and T35/T35-W) running firmware version 12.0 and later with the Bitdefender Antimalware engine. A signature definition file, intended to update the Bitdefender Antimalware engine, contained two incorrectly matched code modules that resulted in faulty initialization behavior for WatchGuard’s GAV scanning capability. WatchGuard automatically tests application of each definition delivery to the various Firebox models (updates can average up to 15x per day); however this event revealed insufficient functional testing on both the vendor and WatchGuard’s testing responsibility domain.

Furthermore, the action that the Firebox takes in response to a scan error is defined by configuration. By default, HTTP proxy allows files to pass through after a scan error while the SMTP proxy “locks” the file (i.e. scrambles the binary composition of a file to prevent read-write access) from non-administrator access. Once a file is “locked” it can only be reversed by the system administrator through a dedicated file restoration utility.

To mitigate the issue, WatchGuard reverted the signatures to a previous version from earlier in February 7th , 2018. Once we tested and verified that the signature file had been fixed for all platforms, we updated the signature definition file for all platforms.

Detailed Timeline – (All times in US Pacific Time)
2:28 AM – WatchGuard receives first customer-reported incident related to GAV scan errors
3:12 AM – WatchGuard support organization begins troubleshooting operations with internal IT team
5:23 AM – WatchGuard initiates communication with vendor
8:00 AM – Rollback of definitions completed; all Fireboxes operational, however definitions levels are no longer current
9:00 AM – Verified fixed definition files
6:00 PM – Verified all Fireboxes have access to current definition levels

Corrective Measures
WatchGuard has initiated several improvements to ensure that incidents like this do not happen again:

• WatchGuard is expanding the automatic testing and monitoring service for these signature definition files to encourage greater fault tolerance in our monitoring and verification infrastructure.
• WatchGuard has identified new contact procedures, escalation paths, and service level guarantees with applicable technology partners so that we can decrease the time between initial incident reports and the time to remediation for any issues that may arise.
• WatchGuard also recommends that customers review their SMTP proxy configurations and allow files to pass through when there are scan errors. We are considering updating this default configuration in a future firmware release.

On behalf of WatchGuard, we apologize for any inconvenience this has caused our partners and customers.