Update: Lee Haller, who served as director of Pittsburgh’s Department of Innovation and Performance since December 2016, resigned on Aug. 3, 2018, for a job in the private sector.
The City of Pittsburgh doesn’t want to talk about its cybersecurity. Its top tech official is staying tight-lipped, saying the stakes are too high to say much publicly. “[T]here’s just too much at risk to the public sector,” said Lee Haller, director of Pittsburgh’s Department of Innovation and Performance [I&P].
But is the city just being cautious? Or are officials just avoiding a public discussion of persistent problems?
Maybe both. A cyber attack in Atlanta that’s been a living nightmare there since March has put many cities on notice. A late 2016 report on Pittsburgh’s tech systems, that has yet to be publicly reported on, illustrates many of Pittsburgh’s own vulnerabilities — and the city won’t say what’s been done to address shortcomings it still considers a threat to public safety and infrastructure.
Atlanta’s attack was near catastrophic. For more than a week, a ransomware attack knocked much of the city’s government out of the digital age. Computers were useless. As were years of files stored on their hard drives. Police dashcam footage is gone forever. Same for a decade’s worth of legal documents, rendered inaccessible after Atlanta declined to pay a Bitcoin ransom.
The response is a slow-moving crawl as software is rebuilt so a vulnerability in one program no longer puts all the others at risk.
And the cost? Likely at least $9 million, according to a recent budget document.
So prudence perhaps is wise. Or Pittsburgh could end up like Atlanta — or Baltimore or Dallas or Maricopa County, Ariz., or Allentown, Pa., or even so close to home as the Allegheny County District Attorney’s Office.
Haller explained that as threats change, so do the city’s defenses. It’s a priority. But just as he was named director in December 2016, an international consulting company’s report explained that the cybersecurity of City of Pittsburgh government operations was in pretty poor shape.
That report, commissioned for $200,000 from Deloitte Consulting, found:
- Substandard physical security for the city’s data facility
- Inadequate written plan to address “catastrophic failure”
- Limited tracking of cyber threats
- Lack of regular testing for handling a cyber disaster
- Small cybersecurity staff
- Spotty documentation of city infrastructure
A year and a half later, is it all fixed? Pittsburgh isn’t saying.
Haller’s department — with a 2018 budget of $13.6 million — has tried to fortify city networks since Atlanta. And Mayor Bill Peduto last week touted a collaboration with Carnegie Mellon University [CMU] to find system vulnerabilities. Yet officials told PublicSource that the city was wary even of a general discussion about security measures, citing the danger it could pose.
Communication Analyst Keyva Clark said in an emailed statement that “[a]s the threat landscape continues to evolve, I&P is constantly reevaluating our security efforts, including education of staff members, introduction of new monitoring tools and improvements to our systems to increase their resiliency. Because of the sensitive nature of the topic, the City does not share specifics about our cybersecurity efforts with the public.”
Clark said the city is “declining to respond” to questions about specifics, including how the city has fixed faults outlined in the Deloitte report.
The city views some problems identified in the report to be so serious that it redacted them from a copy recently provided to PublicSource. The city claimed an exemption to the state Right-to-Know Law that states disclosure could be an active threat to infrastructure or public safety.
In other words, Pittsburgh fears being Atlanta, or worse.
Howard Stern, who served as Pittsburgh’s director of technology until January 2012, described an attack on public safety systems as his “biggest nightmare.” One that created the potential for life-or-death situations, if fire crews can’t access the layout of a burning building or police in a traffic stop don’t know the driver is a dangerous suspect.
“They must protect what they have. That is crucial. Beyond crucial,” said Stern, who reviewed Deloitte’s report.
The city has employee Social Security numbers, tax information, sensitive documents from criminal investigations, not to mention documents like those related to the city’s proposal to Amazon, which officials have kept under lock and key.
It’s all there for pranksters, extortionists and criminals. Cyber experts warn that all it takes is a single misguided click by a city employee for data to be stolen, deleted or disclosed to the world.
Though Pittsburgh wouldn’t give details, Haller said Pittsburgh is “constantly evolving the tools that we’re using to try to minimize the risks to the city.”
Heed Atlanta’s pain
If any city hasn’t been mindful of the massive cost of a cyber attack, Atlanta was the wake-up call.
Atlanta City Councilman Howard Shook said more cities are probably like Atlanta than not.
Nothing seemed amiss until it seemed like everything was going wrong.
“The day before Pearl Harbor, everybody probably felt pretty good about Pearl Harbor’s defenses,” Shook said, referencing the surprise attack that thrust the United States into World War II.
At first, it seemed like a fairly routine tech hiccup. But then Shook realized his computer and the two others in his office were unusable.
“I lost, boy, 16 years’ worth of contacts and files,” Shook said, adding that the city also lost files it’s legally required to keep.
Key functions like payroll and sewage treatment kept running, keeping it from being “utterly catastrophic,” Shook said, but residents couldn’t pay their water bills and police had trouble looking up license plate information.
Citywide, he estimated the actual price of remediation at around $18 million, including upgrades that the city had already planned for, but now needs to accelerate.
David Hickton, former U.S. Attorney for the Western District of Pennsylvania and founder of the University of Pittsburgh Institute for Cyber Law, Policy and Security described the attack as a “cold splash of water for everyone.”
But in an increasingly digitized and interconnected world, it’s simply one example.
In February, Allentown, Pa., faced a malware attack that forced the city to freeze electronic bank transactions and cut police off from a state police database. Chief Information Officer Matthew Leibert said the remediation is costing about $1 million, not including ongoing expenses to build more secure systems.
The virus found an easy route into the city: It compromised an insecure laptop off city property. Then, when the laptop was brought back into city offices and joined the network, the virus spread. The city ceased online financial transactions for several weeks, and Leibert said some public safety functions (that he wouldn’t discuss publicly) still aren’t restored.
No data was stolen, but Leibert said the rebuilding process will last at least until the end of 2018.
“You can’t just restore everything from a backup and turn it back on,” Leibert said. “Because it’s just vulnerable as the day it was infected.”
Haller cited the Allentown’s attack when declining to discuss Pittsburgh’s cybersecurity efforts. Leibert discussed the attack and remediation efforts for about 45 minutes.
In March, a ransomware attack in Baltimore knocked out the city’s system to dispatch 911 calls for a full day (dispatchers were able to continue work manually). Here in Allegheny County, District Attorney Stephen Zappala’s office paid the equivalent of about $1,400 in a Bitcoin ransom to have case documents released after a 2015 ransomware attack.
According to Donald Norris, professor emeritus in public policy at the University of Maryland Baltimore County (UMBC), local governments are generally not adept at cybersecurity and should assume that threats are constant.
A survey published about a year ago by UMBC and the International City/County Management Association shows that local governments often lack staff to focus on cybersecurity, struggle to pay competitive salaries and lack training. About 400 county and municipal governments responded to the anonymous survey; Pittsburgh did not participate.
About half of the respondents weren’t keeping track of hacking attacks and only about 60 percent kept track of instances when systems were compromised; the responses also indicate local governments generally lacked formal plans to respond to attacks.
In other words, it’s fairly status quo for cities to know something bad could happen and not be ready for it.
“I would guess that our operations were probably pretty average,” Shook said of Atlanta. “Average doesn’t cut it anymore.”
So how safe is Pittsburgh?
Networks fail for any number of reasons.
A weak firewall. Overwhelmed servers. An employee who clicks a sketchy link or plugs in an infected thumb drive.
“From my perspective, it’s not a question of if it’s going to happen,” said Stern, citing attacks on major companies like Target. “It’s a question of when.”
Stern said he set up a backup system during his tenure that would go online in seconds if the city’s main data center went down from either a virtual or physical attack.
But in the event of a “catastrophic failure,” Deloitte found city staff may not actually know what to do.
“There is only a notional idea of a [Disaster Recovery] model, and no consistent [Disaster Recovery] testing occurs,” according to the report.
This text was redacted from the report, indicating that the city views it as a current vulnerability to public safety or infrastructure. PublicSource informed the city that some redacted information that explains general weaknesses would be published.
Deloitte recommends that Pittsburgh establish a clear disaster recovery plan, communicate it across city departments and test the process every year.
Deloitte also found that Pittsburgh had “limited” tracking of cyber risks. Processes to deal with threats were “not documented.”
For cybersecurity experts, protection doesn’t hinge on having an impenetrable network; it’s about knowing what to do when things go wrong. In a ransomware attack, that means isolating a virus so it won’t impact all of a city’s interconnected programs. Or if a system crashes, knowing how to quickly switch to backups.
Brandon McCrillis, an ex-U.S. government cybersecurity expert, said cybersecurity is about more than just having “a scary wall” to block threats.
“It’s [about] being effective in how quickly you can respond when someone starts scaling that wall,” said McCrillis, now CEO of Rendition InfoSec, a Georgia-based cybersecurity company.
From Stern’s perspective, the city had an adequate disaster recovery system when he left, but he said he didn’t feel “100 percent” confident about data security and that’s likely to remain an issue because of resource constraints.
“It’s not their fault,” Stern said. “It’s money, it’s expertise.”
But Deloitte also found basic problems like the lack of “well documented” security policies and lack of communication between the city’s tech experts and other departments.
As explained by Vyas Sekar, a cybersecurity expert at Carnegie Mellon University, systems are vulnerable not just to sophisticated hacking but to plain human error, like connecting infected devices or USB sticks on protected networks or even clicking on scam emails. Guarding against that requires all staff to understand vulnerabilities and to be trained to recognize common threats.
“While we did do security training, we didn’t do enough,” Leibert said of Allentown. “But how do you define what enough is?”
Deloitte also found that Pittsburgh lacked a full understanding of its systems, including older technology managed by staff outside the tech department.
In fact, much of Deloitte’s report focuses on Pittsburgh’s broader tech problems, including the city’s reliance on aging technology that is managed by departments outside of Innovation of Performance [I&P]. According to the report, Pittsburgh didn’t even have a full inventory of all those systems.
The city is moving its overall tech strategy away from some of those legacy systems, but it’s a slow-moving process, meaning aging programs are in place for now.
As systems age, Norris said, they can be more difficult to protect.
“That’s just leaving you wide open to be breached,” Norris said.
Stronger defenses
Since March, Pittsburgh has sought to discreetly learn from Atlanta’s mistakes.
Laura Meixell, an assistant director in I&P, explained right after the attack that staff put an additional focus on security, but she offered few details.
Such secrecy is to avoid attracting hackers, though the city invited a friendlier sort of scrutiny from computer science students at CMU.
“We have students try to hack into our system,” Peduto said last week. “What they find, they then report to us.”
That’s the sort of action that cyber experts applaud.
“Do this type of penetration testing almost like a fire drill to see what would happen and how you respond,” said Sekar, who wasn’t involved in the CMU effort.
McCrillis of Rendition InfoSec said his firm has a Pittsburgh-based client that they regularly try to hack to isolate vulnerabilities and to train employees how to avoid scams.
During his tenure, Stern invited similar scrutiny from CMU.
“What that showed me was that, oh my god, you can try so hard, as hard as you want,” he said. “People can still get in if they really want to.”
Hickton applauded Pittsburgh for the steps it’s taken. Commissioning a report, finding out the problems and appointing staff to work on the issue are the right steps to take, he said.
The Deloitte report recommends that Pittsburgh establishes both an I&P team led by a chief information security officer and focused on cybersecurity and a long-term cybersecurity subcommittee, so the city has a unified approach to security. The report also suggests that Pittsburgh consider seeking outside help to supplement security.
Cybersecurity requires significant and sustained investment in systems and staff training as well as buy-in from every employee with access to a city’s network, Norris said. But while Atlanta is spending millions to rehab computers and fix vulnerabilities across its network, some security measures are fairly low cost.
Clark said that Pittsburgh “takes necessary precautions” to protect systems including the use of two-factor authentication (which means logging in requires more than a single password), and a requirement for employees to regularly update passwords.
Hickton said both measures are important to security, as well the segmentation of data, so an attack on one system won’t impact other data. Clark said the city follows that philosophy as well.
Peduto said the city is also improving security of social media accounts.
Shook said Atlanta is strengthening its hardware and access requirements, so simplistic passwords — think 12345 — don’t leave the system vulnerable. And after losing more than a decade’s worth of documents, he’s going back to paper printouts to back up important files.
Atlanta’s embrace of stronger protections is costly and slow-going, and for cities with known vulnerabilities, it’s a cautionary tale.
“If nothing else, I hope we provided a public service to taxpayers elsewhere,” Shook said. “Don’t wait until you’ve been broken into to change the locks.”
Jeffrey Benzing is PublicSource’s public safety reporter. He can be reached at jeff@publicsource.org, 412-515-0062 or on Twitter @jabenzing. He can be reached securely at PGP: bit.ly/2Au8Ca9
J. Dale Shoemaker is PublicSource’s government and data reporter. You can reach him at 412-515-0060 or by email at dale@publicsource.org. You can follow him on Twitter at @JDale_Shoemaker. He can be reached securely via PGP: bit.ly/2ig07qL
This story was fact-checked by Mary Niederberger.
This story appears as a part of Open Data PGH, a joint reporting project by PublicSource and Technical.ly on open data trends in Pittsburgh.
