CrowdStrike outage made a big AZ impact. Experts say it raises resiliency questions

Experts say a widespread technology outage that closed health care facilities, ensnarled travel and disrupted emergency services worldwide on Friday highlighted the fragility of technological systems that are increasingly globally interconnected and the need for improved user resiliency.

The issues stemmed from a software update issued by CrowdStrike , a cybersecurity firm that provides services to scores of industries. It works with tens of thousands of businesses and governments, including many Fortune 500 companies, and has become a go-to platform for large organizations seeking to avoid technological crises.

"It is one of the better, if not the best, technologies for what it does within its market space," said Kim Jones, an Arizona-based cybersecurity consultant with decades of experience in information technology and risk management.

But a flawed software deployment — not a hack or outside attack — left many people facing a "blue screen of death" on Friday morning. The issue impacted computers with some versions of the Microsoft Windows operating system, which is widely used on business and personal computers globally.

In the Valley, the outage impacted dozens of companies and government agencies . Phoenix, Mesa, Queen Creek, Goodyear and Peoria police reported issues with dispatch and communications systems. The outage caused delays and canceled flights at Phoenix Sky Harbor International Airport. It also took down Circle K point-of-service terminals, leaving convenience store clerks without a way of accepting credit and debit cards.

See what else was disrupted: Global software outage disrupts travel, emergency services in metro Phoenix

Ken Colburn , founder and president of Phoenix-based Data Doctors Computer Services, called the incident "a doomsday scenario" and "unprecedented." But he said it's also an unfortunate side effect of "the interconnectivity that we all enjoy and benefit from every single day."

"In this case, there was a single point of failure that impacted a lot of large organizations," he said.

How soon will the problem be fixed?

CrowdStrike officials deployed a fix for the flawed software on Friday morning. That doesn't solve the issue on impacted machines, but does ensure that computers that did not already download the software update won’t experience problems. Founder and CEO George Kurtz also released a statement apologizing for the disruptions .

"All of CrowdStrike understands the gravity and impact of the situation," he said. "We are working closely with impacted customers and partners to ensure that all systems are restored so you can deliver the services your customers rely on."

As of Friday afternoon, many of the affected services in the Valley were back online and returning to regular operations, though some issues still lingered. Colburn said it might take a few days for everything to entirely return to normal because the fix can't be implemented remotely on impacted computers.

"When you get a blue screen, all bets are off," he said. "The machine is completely unusable."

Some especially tech-savvy users might be able to follow instructions from CrowdStrike to get their machines up and running . But Colburn said most people with impacted machines will likely need some help from an expert, and information technology professionals have to handle the computers in person to get them back online.

"Think of the logistics of all of those computers that are on the road, or working from home or working from remote spaces," Colburn said. "That's a logistical challenge to getting this thing mitigated."

The need for more resiliency

Colburn said there wasn't much companies and government agencies could do to mitigate the impacts of the outage after computers downloaded the flawed software update.

"In this case, we would have all had to have been carrying around a second computer that was exactly duplicating the existing computer so that we could just switch to a new computer if the old one went down," he said. "That's just not practical."

But Jones said employers can make procedural changes to build resiliency and stop a similar situation from occurring in the future. Performing updates automatically and all at once might be convenient, but large organizations should consider phased software implementation and have rollback plans to minimize operational issues if something goes wrong.

"What's the resiliency plan? What's the backup plan so that organizations can recover appropriately," Jones asked. "That, I think, is the bigger concern in this environment."

Colburn said technology providers can also learn from the incident. He suspects CrowdStrike and similar cybersecurity services will do a "deep self-examination" and change their processes to ensure similar issues don't crop up again.

"The silver lining of this is maybe we as an industry become better at this type of thing," Colburn said.

Is connectivity worth the risk?

Experts say the convenience of fast technology, and interconnectivity comes with the inherent risk of widespread problems if key pieces break.

"You can't really have it both ways," Colburn said.

But Jones noted CrowdStrike has pushed "hundreds, if not thousands" of software updates without incident. He said people must compare the benefits of global connection with the small probability of major issues.

"Every day, we as individuals make risk decisions," he said. "It is the nature of individuals. It's the nature of business. It's the nature of life. What we have to do is make sure that the risks that we're taking do not exceed the value of what we're getting out of our interconnected world."

Still, the outage might offer opportunities for reflection. Colburn said he couldn't help but think about "first-world problems" as he heard some complaining about being unable to order coffee at Starbucks via an app .

"I think this is a wake-up call to everybody about how ingrained in technology we all are and how reliant we are," he said. "And, you know, maybe some of that is individual self-evaluation about reliance on technology."

This article originally appeared on Arizona Republic: CrowdStrike outage made a big AZ impact. Experts say it raises resiliency questions