The day The Pilot almost wasn't published

A new chapter in The Virginian-Pilot's disaster plan was written last week. A "catastrophic operational failure" of the newspaper's primary database at 2:15 p.m. on Aug. 22 widened into a scenario that the in-house disaster planners had never anticipated. Several insiders called it The Pilot's closest brush ever with the prospect of failing to publish a newspaper.

Saturday's Pilot did reach subscribers, but its diminished content and unfamiliar format created confusion for those who missed the brief front page explanation that noted, "because of technical issues, today's newspaper is incomplete."

Perhaps because the front page otherwise looked so familiar, readers were unprepared for the way inside pages departed sharply from Pilot customs. Calls clogged the newspaper's telephone lines Saturday. Readers were bewildered, alarmed or incensed.

The first call to the public editor's line came at 6:38 a.m. Walter Banks was in a fury. The newspaper "was a piece of trash today," his message declared. Another caller wondered whether some "screwball" had purchased The Pilot.

What happened to the editorial page? Why had the newspaper been reformatted without a word of warning? Where were the stock tables? How come we managed to include the classified ads when the news hole was so drastically reduced? Why was local news buried inside a section that, inexplicably, featured weather on its front page?

It all added up to a compelling snapshot of reader expectations and the distress any deviation creates. Readers, of course, were unaware that it had taken a nearly superhuman effort to produce Saturday's paper at all.

There was little to suggest that a major technological meltdown loomed when the system began sputtering Friday. Staffers pretty much shrugged and waited for the IT crew to put things right. Since glitches occasionally afflict the system, there was no immediate sense of alarm. It appeared that the electronic demons were up to their tricks again.

Like any other database, The Pilot's system is a giant electronic basket that contains all the newspaper's content - stories, photographs, page designs and advertisements. The system's "snap mirror" feature, according to Steve Dandy, newsroom technology director, copies the basket's contents to a backup system at 15-minute intervals.

In a worst-case scenario, that means only 15 minutes worth of data are lost. A problem to be sure, but hardly one that threatens publication of the newspaper. Except that the backup database also broke down that day.

Nowhere in The Pilot's thick loose-leaf notebook of disaster scenarios is there any plan for the double breakdown that hit Aug. 22. That's because, according to chief information officer Steve Peters, the chances of it happening are as remote as winning the lottery.

It took most of the night to fix the corrupted the database. The breakdown, according to Peters, was caused by a bug rather than a virus.

On the day that was soon named Black Friday, editors, designers and advertising personnel cobbled together material they could find backed up on individual computers around the building.

The normal Friday complement of 30 staffers quickly swelled to 50, all working to replace, recreate or otherwise conjure the contents of that electronic basket. They accomplished the mission with such dispatch that they missed the normal deadline by only 29 minutes.

The result, to be sure, hardly followed the blueprint for what should have been a 42- to 48-page paper.

Business news ran over 2-1/2 pages at the back of the front section. The weather landed on a section front because it required an open page where color could be accommodated. Sports news followed on six pages, and Hampton Roads news occupied two pages. The editorial pages were jettisoned in favor of obituaries on the last two pages of the section. Daily Break, the Home tab and classified ads, all printed before the crash, were also part of Saturday's package.

That so many readers complained was a perverse source of comfort for staffers who realized that readers notice when they deliver less than the very best.

Editor Denis Finley, who spent part of Saturday and Sunday answering calls from readers, is embarrassed and upset by the product that subscribers received on Saturday morning but enormously proud of the people who created it.

Black Friday is destined to become a career milestone for the staffers who faced the unthinkable: not publishing a daily newspaper. They pulled it off with a combination of imagination, creativity and abundant devotion. Improbably, and invisibly, they arm wrestled the technology gods and won.

Joyce Hoffmann, the public editor, is an associate professor in the English Department at Old Dominion University. Reach her at (757) 446-2475 or public.editor@pilotonline.com.