CrowdStrike’ ın Hatalı Güncellemesiyle Oluşan Küresel Kesinti Bilişim Sistemlerini Etkiledi 

Bugün yayınlanan Crowdstrike güncellemesi windows temelli sistemleri çökertti. Küresel BT kesintisi, havalimanları, bankalar, sağlık hizmetleri ve küresel bazı şirketlerde kaosa neden oldu.

Yanlış güncelleme Crowdstrike’ e ait olmasına rağmen sadece windows temelli bilgisayarlarda mavi ekran vererek sistemlerin açılmamasına yol açtı. 

Crowdstrike yazılımı sensör adı verilen bir sistemle, işletim sisteminden önce acıkarak, güvenlik sorunlarını topluyor.  MacOS , Linux gibi sistemlerin açılış sırası biraz daha farklıdır. Benzer bir durumu güncellemeyi geri al seçenekleri ile kolayca aşılabiliyor. 

Crowdstrike sitesindeki orijinal açıklama aşağıdadır. 

Summary
CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.

Details
Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
Windows hosts which are brought online after 0527 UTC will also not be impacted
This issue is not impacting Mac- or Linux-based hosts
Channel file “C-00000291.sys” with timestamp of 0527 UTC or later is the reverted (good) version. Channel file “C-00000291.sys” with timestamp of 0409 UTC is the problematic version.

Current Action
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.

If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:

Workaround Steps for individual hosts:
Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory

Locate the file matching “C-00000291*.sys”, and delete it.

Boot the host normally.
Note: Bitlocker-encrypted hosts may require a recovery key.

Workaround Steps for public cloud or similar environment including virtual:
Option 1:

Detach the operating system disk volume from the impacted virtual server
Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes

Attach/mount the volume to to a new virtual server

Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory

Locate the file matching “C-00000291*.sys”, and delete it.

Detach the volume from the new virtual server

Reattach the fixed volume to the impacted virtual server

Option 2:

Roll back to a snapshot before 0409 UTC.

Workaround Steps for Azure via serial
Login to Azure console –> Go to Virtual Machines –> Select the VM
Upper left on console –> Click : “Connect” –> Click –> Connect –> Click “More ways to Connect” –> Click : “Serial Console”
Step 3 : Once SAC has loaded, type in ‘cmd’ and press enter.
type in ‘cmd’ command
type in : ch -si 1
Press any key (space bar). Enter Administrator credentials
Type the following:
bcdedit /set {current} safeboot minimal
bcdedit /set {current} safeboot network
Restart VM
Optional: How to confirm the boot state? Run command:
wmic COMPUTERSYSTEM GET BootupState
For additional information please see this Microsoft article.

Latest Updates
2024-07-19 05:30 AM UTC | Tech Alert Published.
2024-07-19 06:30 AM UTC | Updated and added workaround details.
2024-07-19 08:08 AM UTC | Updated
2024-07-19 09:45 AM UTC | Updated

Türkiye’de ulusal bir bankada giriş sistemlerinin dahi etkilendiği anlatılıyor. Benzer yanlış mimarileri daha önce Twitter, Türkiye’ de yine bir banka’ da toplu kapanmaya yol açmıştı. 

Önerilen çözümler teker teker müdahale gerektiriyor.  Dolayısıyla çözüm için destek elemanları uzun mesai yapacağını tahmin ediyorum.