Update: Lee Haller, who served as director of Pittsburgh’s Department of Innovation and Performance since December 2016, resigned on Aug. 3, 2018, for a job in the private sector.
The City of Pittsburgh, despite its growing reputation as a 21st-century tech hub, doesn’t have a clear framework for making its own decisions on information technology. Individual city departments pursue IT projects in an “ad hoc” manner, in lieu of a central strategy.
The city also relies on outside vendors to build the software and supply the hardware for crucial city functions. Rather than off-the-shelf software that’s easier to maintain, 40 percent of the city’s software is custom. Ten percent is the industry average.
Underscoring these legacy problems is a concern that the city department responsible for all IT matters struggles to keep the laptops, printers and phones working for the city’s 3,400 full-time employees.
All of these assertions were made in a report the consulting firm Deloitte completed for the city in late 2016. At the time, Deloitte found that documentation for software and IT procedures was lacking or nonexistent and that the Department of Innovation & Performance [I&P] struggled to keep up with internal service complaints about malfunctioning software or hardware. Deloitte charged $200,000 for their two months of work on the report.
Now, more than a year and a half since the report was finished, I&P is beginning to make progress on fixing the problems. Under director Lee Haller, who started his job just as Deloitte was finishing its report, the department now has a strategic plan. Haller is not releasing that plan to the public, according to city spokesman Timothy McNulty — at least not until it is finalized.
The Deloitte report and interviews with current and former officials reveal a department struggling to keep the city’s IT running on a day-to-day basis while trying to modernize their operations.
“We struggle to keep up, right?” Haller said. “I mean we’re just not staffed to be able to provide kind of the reactive service to support that number of users and then all of the proactive things that are required to manage technology needs in 19 departments. So that’s been a struggle over time.”
I&P, whose budget is currently at $13.6 million, is making improvements, including upgrades to the city’s 311 information line. It has also increased its budget for staff training. When Deloitte compiled its report, the department was spending about $380 on training per employee; now, it’s investing $1,400 into each of its 70 employees, Haller said. I&P is also working to centralize how IT decisions are made.
Deloitte’s report came in three parts: two parts, totaling 262 pages, plus an illustrated “roadmap” for implementing its ideas.
Other problems still exist. Deloitte highlighted the city’s reliance on outside vendors for its essential software systems and said doing so contributed to the fragmented, siloed approach the city has taken to technology. The report levied some of the blame for the city’s shortcomings with contractor B-Three Solutions, a Plum-based IT firm that’s currently embroiled in lawsuits and controversy over its work for city police. According to the report, B-Three has built a third of the city’s software, including more than half of the software the Bureau of Police use. A former police officer has alleged in a federal lawsuit that the city paid B-Three for incomplete software projects. The city also pays B-Three $300,000 a year to maintain its software. Pittsburgh City Council in March extended B-Three’s contract for two more years at a cost of $573,000.
Both Haller and Mayor Bill Peduto said they see the Deloitte report as a roadmap, though how quickly they implement Deloitte’s ideas is tricky. Haller said Deloitte didn’t appreciate the constraints of government when it suggested that the city should have all 27 of its recommended solutions under way by June 2018. Deloitte recommended that everything should be completed by Dec. 2019. Peduto said there was a delay in implementing Deloitte’s ideas and that it could take six or seven years to get everything done. Still, the mayor praised the consultants’ work.
“We had to bring in outside assistance to be able to evaluate our entire technology portfolio,” he told PublicSource in January. “If anything, the contract that we did with Deloitte was a control-alt-delete.”
‘Ad hoc’ technology and a way to fix it
Without central decision-making on technology, directors of each city department had the authority to pursue IT projects on their own.
The result? The city entered contracts with 150 different IT vendors. I&P managed the budgeting for fewer than half of those vendors while individual departments managed the rest.
Deloitte also highlighted I&P’s lack of involvement in how other city departments budgeted for technology projects. Departments weren’t basing their annual IT budget off of anything other than what they spent the previous year, an approach that was not strategic, the report pointed out.
When the city or any entity uses so many vendors and custom-built software programs that aren’t compatible, it can stifle opportunities to merge data and find correlations, Peduto said.
He gave this example to illustrate why an “ad hoc” approach doesn’t work: One city department had a dataset that tracked property violations at various homes and businesses around the city, based on real estate parcel numbers. Another department kept a dataset to track 911 calls, based on address. The two systems couldn’t talk to each other, though.
“We [weren’t] able to look and see the number of 911 calls this house had and the number of violations because the systems weren’t able to mesh without manually going back and retyping in every single address,” he said. “We don’t want to get in that situation in the future.”
If all of the city’s data could be managed by one system, city departments could work together to solve problems holistically instead of fixing one problem at a time.
To address this concern, Haller said he helped to convene a Technology Leadership Council — one of Deloitte’s recommendations — whose members discuss technology projects that city departments need. A dozen city leaders — from I&P, the Office of Management and Budget and other departments — comprise the new council. When someone wants to purchase new software, for example, they’ll have to make a “business case” for the software to the council. The members will evaluate how much the project will cost over time, how much staff will be needed to maintain it and then vote on it.
“This is a way to try to elevate the discussion about technology projects, so that we’re helping to together prioritize what are the best projects that should move forward with limited funding,” Haller said.
Haller said I&P is also looking to hire software developers to build applications that can connect larger data systems city departments are running — to “stitch those systems together.”
Moving forward, the city will continue purchasing software from reputable vendors who update their programs as they improve over time and will work to replace old systems. The city will still have some custom needs, but Deloitte suggests that the city should reduce its reliance on custom work.
“The goal is to be able to follow the Deloitte report and then be able to have one centralized system that everything can run off of,” Peduto said earlier this month.
The Technology Leadership Council could help solve a bigger picture problem, too: a lack of accountability for technology projects.
Howard Stern, a Carlow University professor and former director of the city’s Computer Information Systems [I&P’s predecessor], bemoaned city council’s ability to ask tough questions when it came to spending money on software. Council members may be experts in their fields, he said, but they didn’t have technical backgrounds to scrutinize the projects he brought before them, Stern said. It’s important to ask questions about what platform software will run on and how compatible it is with other software already in use, he said.
“I should have been asked those hard questions,” he said. “I should have been held more accountable to making these multimillion-dollar decisions.”
Stern endorsed the idea of a council as needed oversight so long as it is transparent.
Debra Lam, the I&P director in between Stern and Haller, did not return a request to comment.
A lack of documentation and security risks
The Deloitte report also highlighted I&P’s lack of documentation on how the department operates and how systems and networks it oversees work. When a department has to manage code and software systems, documentation is vital. The consultants noted in their report that I&P’s “Infrastructure documentation is insufficient, which limits I&P’s own understanding of the hardware and network.”
The report also highlighted I&P’s lack of documentation on security policies and standards, noting that the department’s procedures to deal with security threats are not documented. It’s not clear how I&P is addressing these issues. Haller declined to discuss how the department is solving its security challenges, citing recent cyberattacks on Atlanta and Allentown, Pa.
“One thing that I would say is the threat landscape is constantly evolving and we’re constantly evolving the tools that we’re using to try to minimize the risks to the city,” he said.
Peduto said the city tasked a select group of Carnegie Mellon University students to attempt to “hack” into the city’s systems. In the past, the city has asked students to suss out vulnerabilities in code and even walk in I&P’s front door and see how hard it is to get access to important computers and documents.
Still, Stern said he’s worried by the lack of documentation. The department struggled with documentation when he was in charge, too. The Deloitte report also pointed out that nearly 40 percent of I&P’s staff, as of late 2016, had been hired in the past two years. Those two things mean that there is a lack of institutional knowledge, he said.
“You lose institutional knowledge to the city, and the management needs to really pay attention to that and make plans for it,” Stern said.
It’s often an afterthought. “Not just in the city but everywhere unfortunately, documentation is not always a priority,” Stern said.
Staff problems and structural problems
I&P’s staff could do more if it was bigger, Haller said. The staff has more work than it can accomplish, Stern said.
“I think they are so overstretched. And they’ve always been overstretched,” Stern said, adding that it was often hard to send employees away to training opportunities because there was so much to do.
A limited staff also means the I&P help desk is manned by three or four people to help other city employees fix their computers, printers and email accounts. Deloitte interviewed city employees outside of I&P for its report and highlighted some of their gripes about the help desk.
“Let’s make my computers work first and then we can talk about building a new app,” one city leader told the Deloitte auditors.
“Everything from trying to get the phone fixed to running major systems like cameras and ShotSpotter has been an issue,” said another person from Public Safety, referring to the police bureau’s use of equipment to detect gunshots.
As of late 2017, I&P had three staff members who worked full time on the help desk.
When it comes to addressing the structural problems the Deloitte report identified, Haller said having limited staff means things won’t get done as quickly.
“I certainly don’t have the staff to do everything at once,” he said. “We’ve had to kind of pick and choose and prioritize the [things] that we can actually work on that are the most critical.”
David Passmore, who worked in I&P for seven years as a web developer, said city leadership often lacked an understanding of how difficult it was to complete IT projects. One example, he said, was the open data initiative currently housed at the Western Pennsylvania Regional Data Center. Passmore, who left the city in 2014 and played a role in the project, cited one city council meeting where he said a member of Peduto’s staff testified about how easy it would be to complete the open data project.
“What always irked me was the tendency to go to press with this accomplishment before something was accomplished,” he said.
Change, however, takes time, something Haller recognizes. As he works to implement the lasting changes that will solve I&P’s structural problems, he still has to keep the city’s computers running.
“I don’t have the luxury of saying here’s something that I’m going to focus on that is going to improve things three years down the road when I have issues today.”
J. Dale Shoemaker is PublicSource’s government and data reporter. You can reach him at 412-515-0060 or by email at dale@publicsource.org. You can follow him on Twitter at @JDale_Shoemaker. He can be reached securely via PGP: bit.ly/2ig07qL
This story was fact-checked by Oliver Morrison.
This story appears as a part of Open Data PGH, a joint reporting project by PublicSource and Technical.ly on open data trends in Pittsburgh.
