Periodically, we will highlight some of the methods used in a recently released audit. Every performance audit is unique and can require creative thinking and methodologies to answer our audit objective. Some of these methods could be replicated or present valuable lessons for future projects.
Disaggregating data can improve transparency and evaluation of programs. In a recent audit of Business Oregon, the audit team divided data about business finance and forgivable loans by different programs to estimate job growth and return on investment, as well as rural investments for each program.
Business Oregon reports jobs created for all of its programs combined, in one Key Performance Measure. The audit team saw their more detailed analysis as an approach that Business Oregon’s analysts could emulate and improve on, in order to foster greater transparency and improve understanding of the investments and outcomes of individual programs.
I recently sat down with Jon Bennett, a Performance Auditor, to learn about their analysis strategy and any lessons learned.
Business Oregon’s programs contribute to job growth
Jon’s team analyzed different business finance programs at Business Oregon and calculated net job growth for participating businesses over a four year period. They found that about two-thirds of businesses had net job growth. They also found that most of the awards went to businesses that paid wages below the county average, important since Business Oregon has a mission to create living wage jobs.
They also looked at investment by geography and found that most awards go to non-rural areas. This is important because rural areas contain 40% of Oregon’s population and were struggling to get out of the recession.
While it is valuable to look at the big picture, separating data into different programs and different measures can provide greater insights into the effectiveness of each program and how each program’s investments reflect the agency’s priorities.
Lessons Learned: Time management and planning ahead
Jon had a few different lessons learned, but the big take away is one I’ve experienced before – time management. Doing data analysis always seems to take longer than you expect it to. One of the time consuming aspects Jon faced was combining data from two different data sources. He thought it would be simple because he had a unique identifier, but it turns out that some of the businesses he was looking at had multiple locations and he had to look at the data more carefully.
Next time, Jon would also like to do a better job of planning how to document his work before he does it. As auditors, we always have our work checked for accuracy, which can be challenging if there is not a clear documentation trail. That is one of the benefits of using ACL, since it automatically creates a log. But sometimes other tools can be more useful. Jon interestingly switched between Excel, ACL, and STATA to use the tool that could do the task most efficiently in the way that he knew best.