Device used across U.S. caused Va.'s computer crash

By Rosalind S. Helderman and Anita Kumar

RICHMOND

The data-storage unit that failed last week, wreaking havoc in the computer networks of a number of Virginia agencies, is technology used by virtually every major company and government in the country.

The crash - still baffling to state officials - exposes the vulnerability of modern interconnected computer networks and is being closely watched by information technology professionals across the country.

"People in the industry are watching in horrified fascination as this unfolds," said Robin Harris, an Arizona technology analyst who writes a blog on computer-storage systems. "There's a lot of 'there but for the grace of God go I' kind of thinking."

It is also a political headache for Republican Gov. Bob McDonnell who promised on taking office that he would straighten out the state's troubled $2.4 billion computing contract with Northrop Grumman and is now grappling with what appears to be the worst network failure since the company took over the state's technology overhaul in 2003.

Chief Information Officer Sam Nixon said that the problem began Aug. 25 with the crash of a pair of 3-year-old memory cards, one of which was supposed to back up the other. That led to 485 of the state's 4,800 data servers being knocked offline.

"The thing that is never supposed to happen happened," he said.

Built by Massachusetts-based EMC Corp., the storage units are fundamental building blocks of the databases necessary to run complex organizations in a digital age. Industry analysts say EMC's drives are used throughout the world of finance and government and have generally been considered highly reliable. If the units were found to have flaws that caused the failure, the impact could be felt far beyond Virginia.

"This is surprising - it's a selling point for them when they talk to a major organization, that this stuff never goes down," said Bill Kreher, a senior technology analyst who monitors EMC at the investment firm Edward Jones.

Experts say that even more troubling than the failure of the initial storage drive - which was repaired by Friday - is how long it has taken state agencies, Northrop Grumman and the Virginia Information Technologies Agency to restore computer applications.

Nixon acknowledged that it took much longer than it should have for the system to be restored and said he plans to study ways to speed up the process.

In a statement this week, Northrop Grumman, which will face a state fine of more than $100,000, vowed to learn from the problems and correct them. EMC officials did not return calls for comment. State officials have said the company reported that the Virginia outage was unprecedented, an occurrence never before seen in 1 billion hours of system use.