Massive computer outage halts some Va. agencies

By Bob Lewis

RICHMOND

A failure of servers at Virginia's centralized information technology superagency has left several state agencies unable to do their work.

At least two dozen agencies were affected by the Wednesday afternoon crash at the problem-plagued Virginia Information Technologies Agency.

The state's chief information officer, Sam Nixon, said the agencies most acutely affected are the Department of Motor Vehicles and the Department of Taxation.

A failure in a primary data storage center in Chester left the DMV unable to process driver's licenses at its 74 locations statewide.

Nixon said about one-fifth of the state government's computers were affected, and it would take into today to replace the flawed components and reboot the massive storage network and the servers it supplies. Full function may not be restored until Monday.

"If we're real lucky, we will all be in here Friday morning with everything working and we'll all be giving each other high-fives, but we probably won't be that lucky," said Nixon, a former House of Delegates member whom Gov. Bob McDonnell appointed in March to take charge of VITA.

Other agencies suffering severe online outages Thursday were the Department of Social Services, which lost access to much of its benefits information, and the Department of Environmental Quality.

Other agencies at least partially dependent on the system found their computers working, but more slowly, Nixon said.

"This is a significant event, but it's not one of these situations where half of the state's computers systems are out," Nixon said.

VITA and its corporate partner, Northrop Grumman, have been criticized in reports from the General Assembly's investigative arm, the Joint Legislative Audit and Review Commission, for cost overruns, service outages, slow service and delays that have paralyzed state agencies numerous times since the agency was established in 2003.

VITA's 10-year, $2.4 billion contract with the government contracting giant Northrop Grumman is the largest in Virginia history with a single vendor.

Nixon said the failure occurred in one memory card in what is known as a "storage area network," or SAN, at VITA's suburban Richmond computing center, one of several data storage systems across Virginia.

The system was built with redundancies and backup storage. It was hailed as being able to suffer a failure to one part but continue uninterrupted service because standby parts or systems would take over. But when the memory card failed Wednesday, a fallback that attempted to shoulder the load began reporting multiple errors, Nixon said.

Failures with the type of memory card that crashed are rare, he said.

"This is supposed to be the best system you can buy, and it's never supposed to fail, but this one did," he said.

Experts who examined the system determined that no data were lost except for those being keyed into the system at the moment it failed, Nixon said.