Wells Fargo outage has security professionals puzzled

If the problem was caused by power shutdown, why wasn't a backup activated?

Wells Fargo is still experiencing some service outages a full day after an issue at one of the bank’s facilities took much of the bank offline, including its website, mobile app, ATMs, credit and debit cards, and internal systems used by tellers.

As of Thursday night, Wells Fargo’s ATM service had been restored, bank branches were operational after being shut out of the system, and customers were once again able to make purchases with credit and debit cards. Mobile and online banking were back up, but some features, such as the ability to check credit card and mortgage balances, remained unavailable. The contact center was also restored, but Wells Fargo cautioned that customers using the phone system may have unusually long wait times.

The Wells Fargo Advisors website appears to have remained active throughout the incident, causing no disruption to Wells Fargo brokers or their clients.

“We continue to work on restoring all our services as soon as possible, and encourage customers to contact us if they have questions or concerns,” the company said in a statement Thursday.

The cause of the service outage was “a power shutdown at one of our facilities, initiated after smoke was detected following routine maintenance,” according to the statement. But information technology professionals say that explanation raises more questions than it answers. Of chief concern: If this was a power outage, why wasn’t a backup activated immediately?

Wells Fargo declined to comment beyond the official statement.

“Security engineers are looking at this cross-eyed,” said Alissa Knight, a senior analyst at Aite Group’s cybersecurity practice. “I don’t think we’re hearing everything. I don’t think we’re getting the full story.”

Part of the concern stems from conflicting stories about what happened at a Wells Fargo data farm in Shoreview, Minn. While people claiming to work at the site reported a fire to regional news outlets, the local fire department said the fire system was triggered by dust from construction. The official Wells Fargo statement simply states that there was smoke.

However, most data centers use gas systems to suppress fires rather than water sprinklers that would ruin the electronics, Ms. Knight said. If the fire system had been activated, it still doesn’t explain why the servers were powered down.

It also doesn’t explain why backups weren’t immediately turned on. The Federal Deposit Insurance Corp. recommends banks maintain a “hot failover,” or a secondary location of servers that is fully active, operational and ready to take over in the event that the primary location is taken offline.

“It’s puzzling to me why there were not backup systems or a failover site,” Ms. Knight said.

The bank’s response doesn’t sound appropriate for a power outage, she added. For security professionals, it looked more like a response to malware, a data breach or other advanced threat.

On Twitter, Wells Fargo reiterated that the system disruption was the result of “a contained issue affecting one of our facilities, and not due to any cybersecurity event.”

There’s no reason to doubt Wells Fargo’s explanation, especially considering regulations requiring financial institutions to report data breaches, Ms. Knight said. Backup systems sometimes fail, and in 2016 a fire suppression system knocked out an ING Bank data center in Romania simply because of the loud noise the system made.

(More: Crackdown showdown: Serious cybersecurity enforcement is coming in 2019, but are advisers ready?)

The bank could still be investigating the issue, but Ms. Knight said Wells Fargo hasn’t yet released enough information to debunk the speculation within the IT community.

The event should also raise serious doubts about Wells Fargo’s business continuity plan.

“There was no appropriate level of backup systems or servers in place,” Ms. Knight said. “There is clearly not regular testing going on at Wells Fargo to make sure backups were working.”

In her experience, this is unfortunately the case at many financial institutions. Companies talk a lot about cybersecurity and invest heavily in technology safeguards like firewalls and automated detection, but still ignore basic security hygiene like regular testing and holding “fire drills” to ensure protocols work.

Wells Fargo isn’t the only firm nursing bruises. BlackRock recently leaked confidential sales data online, and Summit Equities paid a fine for not restricting a former broker’s access to client data.

None of these incidents involved breaches by malicious hackers, but they all reveal weaknesses in the technology infrastructure of financial institutions.