Are Your Data Centers and IT Closets Prepared for the Next CrowdStrike Event?

On July 19, 2024, a major IT disaster struck when a CrowdStrike update caused widespread chaos. CrowdStrike, a cybersecurity firm, inadvertently pushed a faulty “sensor configuration update” for its Falcon Sensor software. This update caused 8.5 million Windows devices to crash. The impact was severe, affecting airlines, banking systems, and healthcare networks, and the recovery process was laborious, requiring manual intervention for impacted devices.

The disaster will cost Fortune 500 companies alone more than $5 billion. To prepare for potential future scenarios of this scale, data center professionals must be vigilant and proactive.

Be Proactive and Quick to Respond with DCIM Software and Other Tools

In the wake of IT disasters like the recent CrowdStrike incident, it is beneficial for data center professionals to leverage DCIM (Data Center Infrastructure Management) software, switched intelligent rack PDUs, and KVM-over-IP switches to enhance preparedness and response capabilities. These tools provide:

• Centralized remote power control. DCIM solutions offer centralized control over intelligent power distribution units (PDUs) with remote power control, allowing administrators to power on/off individual outlets remotely. In case of emergencies, DCIM enables quick and safe shutdown or reboot of specific equipment, preventing damage and data loss.
• Remote access. To reboot and enter devices into safe mode, it is important to have embedded KVM-over-IP (e.g., Dell DRAC or HP iLO) or external KVM-over-IP switches, as well as management tools like Dell OpenManage and HP Insight Manager.
• Quick asset identification. DCIM tools provide real-time visibility into all assets across all sites. This helps in quickly identifying and addressing any issues related to specific assets, enabling faster troubleshooting and recovery during disasters.
• Streamlined change management. DCIM systems streamline change management processes by automating workflows and approval processes, ensuring changes are documented, reviewed, and implemented correctly. Before implementing changes, DCIM tools can simulate potential impacts on the data center, helping to prevent disruptions and mitigate risks.
• Thresholds and alerts. DCIM continuously monitors power and environmental conditions at a granular level. By setting thresholds for various parameters, DCIM tools can trigger alerts for potential issues, allowing for proactive measures before a disaster escalates.
• Failover reports. DCIM software lets you simulate a power failure to determine which cabinets are outside your redundancy requirements.
• Health map. Red/yellow/green color-coding helps you understand rack load levels, line currents, and environmental conditions to quickly visualize issues and aid in their correction.
• Tips to Be Prepared for the Next Data Center Disaster

With future IT disasters a matter of "when" rather than "if," data center professionals must implement strategies to ensure their readiness and resilience. Consider:

• Robust testing procedures. Implement staged rollouts where updates are first deployed to a small, controlled group of devices before a full-scale deployment. This allows for early detection of issues without affecting the entire network. Use isolated environments to test updates thoroughly to help identify potential problems without risking production systems.
• Redundancy and backups. Ensure critical systems have redundancy to take over in case of failures. Perform regular backups of critical data and systems, and ensure backups are stored in multiple locations to protect against localized failures.
• Disaster recovery plans. Develop and maintain comprehensive disaster recovery plans that outline step-by-step procedures for responding to various types of failures. Conduct regular drills to ensure that all team members are familiar with the procedures and can act quickly in an actual emergency.
• Vendor management. Regularly assess and review the reliability and performance of your software vendors. Ensure they have robust quality assurance processes and a good track record. Maintain open communication channels with vendors for timely updates and support.
• Education and training. Provide ongoing training for staff on the latest best practices in disaster recovery and incident response.
• Documentation and reporting. Keep detailed records of all incidents, including steps taken for resolution. This helps in refining future responses and improving processes. Conduct thorough post-incident reviews to identify root causes and implement changes to prevent their reoccurrence.

Bringing It All Together

Data center professionals must be proactive and prepared to handle future IT disasters. By deploying modern technologies like DCIM software to enable proactive data center management and following best practices to be prepared for the worst, you can mitigate risk.

Want to see how Sunbird’s world-leading DCIM software can help you prepare for the next data center disaster? Get your free test drive now.