From the CrowdStrike crisis to control: key strategies for IT resilience

Corporate reliance on technology frequently carries the risk of crisis when it fails, for whatever reason. The most recent example was Friday's global IT shutdown.

An IT outage like this has serious consequences. The scale of the disruption, from grounded flights to paralyzed hospital systems and stalled retail operations, highlights the vulnerability of our digital world. It has an impact on the organizations involved, particularly their credibility and reputation. It also reduces customer trust and raises questions about the company's dependability and preparedness.

For corporate leaders and IT managers in charge of managing technology risks and crisis management, it serves as an alarming reminder of what happens when businesses delegate too much control. 

As the dust settles on what has been labeled the largest IT outage in history, it is time to consider what we can all learn from it.

Prioritizing resilience over efficiency to reduce risks 

Although the outage was caused by an update and human error, rather than a cyber-attack, such vulnerabilities can still be exploited to maximize the impact of supply chain cyber-attacks.

According to 2024 Redefining Resilience: Concentrated Cyber Risk in a Global Economy Research 62% of the global external attack surface is concentrated in the products and services of just 15 companies.

Organizations such as governments, airlines, banks, and telecommunications are already counting the damage, and the situation could worsen further. 

According to CNN, costs from the global outage could top $1 billion.

Internal communication of some affected companies were disrupted as its entire network, including Outlook, Teams and SharePoint, is hosted on Microsoft 365. 

The incident serves as a wake-up call for leaders to avoid relying on a single cloud provider or becoming overly reliant on a few dominant technologies, which could lead to a single source of failure. To be resilient, businesses must adopt a diversification strategy and better manage which core competencies are outsourced. This means taking more ownership of their digital infrastructure and in-house IT skills. 

It also highlights the importance of disaster recovery plans. Companies should enhance their cybersecurity protocols and conduct consistent system audits. Additionally, implement a comprehensive crisis management strategy. This includes both technical solutions and planned communication strategies to keep all stakeholders informed.

As stated in The Guardian, most businesses run on Microsoft Windows, so corporate computing is basically a monoculture. This may be good for efficiency, standardization, training, etc, but it is also bad for resilience if anything goes wrong.

Organizations must have a reliable and backup method to manage employee, customers and partners communications in crisis and when the default method is compromised. 

For us, IT vendors, the need for resiliency is more pressing than it has ever been as we learn to navigate this digital age so we must focus on creating a more reliable, robust, and adaptable digital future based on well-trained people, effective processes, and smart technology use.

We have become utterly dependent on a complex web of technologies that few understand, created by an industry that seems indifferent to the consequences of its creations. We find ourselves in a new world, but it’s not exactly a brave one. ~ The Guardian

Ensuring reliable communication and IT infrastructure

Our team put together a few action points that can assist organizations with increasing resilience, improving security, and ensuring more reliable communication and IT infrastructure.

Implement enhanced security measures:

• Use end-to-end encryption for all communications.
• Set up multi-factor authentication (MFA) for user accounts.
• Conduct regular audits and updates to security protocols to protect against cyber threats.

Adopt decentralization and redundancy:

• Use decentralized, geographically distributed servers to avoid single points of failure.
• Ensure data redundancy to keep services running during server outages.

Ensure scalability and effective load management:

• Design systems to handle significant increases in users and message volume without sacrificing performance.
• Implement containerization and microservices to improve load distribution.

Implement real-time monitoring and incident response:

• Integrate monitoring and alerting tools to gain real-time information about system performance.
• Set up automated alerts to detect and respond to anomalies quickly.

Leverage open-source flexibility and transparency:

• Utilize open-source software for greater transparency and the ability to inspect, audit, and modify code.
• Engage in community collaboration to quickly identify and address security vulnerabilities.

Ensure interoperability with existing systems:

• Use APIs and integration tools to ensure seamless compatibility with existing IT infrastructure.
• Enhance communication capabilities without disrupting existing workflows.

Promote user training and best practices:

• Create dedicated channels for user training and security best practices.
• Regularly update users on security guidelines and promote a security-conscious culture.

Security and data privacy are built into Rocket.Chat's DNA. We believe that empowering organizations to own and control their infrastructure and critical data minimizes the risk of data breaches and unauthorized access. If you would like to learn more, talk to our experts.