Local Disruptions Due to Software Update on some Microsoft Systems: including Airports, TV Stations

Las Cruces Digest has confirmed with Tower Operations at El Paso International Airport that many flights have been “Delayed” and some cancellations have begun showing up on the board. It is recommended to check your flight status and check directly with the airlines operating the flights for details.

KOAT TV has reported the Albuerque International Sunport is also being impacted by the event. While unconfirmed, the late local news broadcast from KDBC 4 was replaced with their “Technical Difficulties” screen. There was no local broadcast between the late-running “Big Brother” and “Stephen Colbert (LIVE)”.

Crowdstrike, the company managing the software linked to the worldwide disruption made the following statement:

Statement on Falcon Content Update for Windows Hosts

CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This was not a cyberattack.
_{Source: crowdstrike.com (Published 5:49am ET, July 19, 2024)}

The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.

We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels.

Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.

Updated 1:25pm ET, July 19, 2024:

We understand the gravity of the situation and are deeply sorry for the inconvenience and disruption. We are working with all impacted customers to ensure that systems are back up and they can deliver the services their customers are counting on.

We assure our customers that CrowdStrike is operating normally and this issue does not affect our Falcon platform systems. If your systems are operating normally, there is no impact to their protection if the Falcon Sensor is installed.

Below is the latest CrowdStrike Tech Alert with more information about the issue and workaround steps organizations can take. We will continue to provide updates to our community and the industry as they become available.

Summary

CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.

Details

Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
Windows hosts which are brought online after 0527 UTC will also not be impacted
Hosts running Windows 7/2008 R2 are not impacted
This issue is not impacting Mac- or Linux-based hosts
Channel file “C-00000291*.sys” with timestamp of 0527 UTC or later is the reverted (good) version.
Channel file “C-00000291*.sys” with timestamp of 0409 UTC is the problematic version.
- Note: It is normal for multiple “C-00000291*.sys files to be present in the CrowdStrike directory – as long as one of the files in the folder has a timestamp of 0527 UTC or later, that will be the active content.

Current Action

CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
If hosts are still crashing and unable to stay online to receive the Channel File Changes, the workaround steps below can be used to address this issue.
We assure our customers that CrowdStrike is operating normally and this issue does not affect our Falcon platform systems. If your systems are operating normally, there is no impact to their protection if the Falcon Sensor is installed. Falcon Complete and Overwatch services are not disrupted by this incident.

Query to identify impacted hosts via Advanced event search

Please see this KB article: How to identify hosts possibly impacted by Windows crashes.

Workaround Steps for individual hosts:

Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
- Boot Windows into Safe Mode or the Windows Recovery Environment
  - NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.
- Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
- Locate the file matching “C-00000291*.sys”, and delete it.
- Boot the host normally.

Workaround Steps for public cloud or similar environment including virtual:

Option 1:

Detach the operating system disk volume from the impacted virtual server
Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
Attach/mount the volume to to a new virtual server
Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Detach the volume from the new virtual server
Reattach the fixed volume to the impacted virtual server

Option 2:

Roll back to a snapshot before 0409 UTC.

AWS-specific documentation:

To attach an EBS volume to an instance
Detach an Amazon EBS volume from an instance
- Note: Use a different OS version for the VirtualMachine used as the recovery VM to the Virtual Machine you are trying to recover.

Azure environments:

Please see this Microsoft article

User Access to Recovery Key in the Workspace ONE Portal

When this setting is enabled, users can retrieve the BitLocker Recovery Key from the Workspace ONE portal without the need to contact the HelpDesk for assistance. To turn on the recovery key in the Workspace ONE portal, follow the next steps. Please see this Omnissa article for more information.

Bitlocker recovery-related KBs:

Article posted by:

Levi Gwaltney

Levi Gwaltney is the Editor In Chief of Las Cruces Digest. The Editor makes editorial and posting decisions regarding content published on this site. In most cases, the Editor makes little if any contribution to content published, and takes no credit for the content of republished posts. If available, the source or author of posts is provided at the top of the article.