Speaking Truth to Power (Users): Why data visualization matters to researchers, auditors, and program evaluators

“I’m not a data visualization expert, but I am a data visualization enthusiast” Rebecca Brinkley joked as she started her presentation on including visual elements in reports to the State of Oregon Research Academy (SORA).  Several dozen researchers from around the state listened intently as she walked through the importance of including visual elements and gave examples from recent audit reports from the Oregon Audits Division.

Why Visuals Matter

“Our brains are visual processors not word processors” explained Rebecca Brinkley. Most people think visually rather than verbally.  Furthermore, visuals have been found to be processed at a much higher rate than text alone.  The visual elements help the reader process complex information and improve retention.

Many times visuals can help tell a story or make a connection that would be more difficult to do using text. For example, Rebecca highlighted this image of the water cycle in a recent report on water management in Oregon.

waikato

I don’t know about you, but I’d much rather see this image describing the water cycle than reading a long paragraph describing the water cycle.

We will be highlighting some more example of useful data visualization techniques in future blog posts.

Rebecca will also be presenting to the Pacific Northwest Intergovernmental Audit Forum in March.

Ian Green, CGAP and OAD Senior Auditor

Ian Green, CGAP and OAD Senior Auditor

Auditors at Work Featured Performance Audit

Talkin’ Shop With: Larry Stafford, Clark County Audit Services Manager

Join us as we talk shop with our neighbor to the North and former OAD performance auditor, Larry Stafford.

About Clark County’s Audit Services

clarkcountylogo

Per county code, internal audit is an independent appraisal activity for the review of operations within the county. The objective is to assist management in the effective discharge of their duties, and to promote efficiency and economy consistent with the public interest.

Clark County government is comprised of several separately elected officials, including the County Auditor.

The Audit Services Department conducts performance audits, internal control reviews, and provides other services to county management. Performance audits are objective and systematic reviews of program quality and the results achieved. Internal control reviews include analytical reviews, interviews, observations, and tests with the intent of evaluating the security of county assets and the accuracy/reliability of financial reports.

The Clark County Auditor

The Clark County Auditor is elected to a four-year term and our current Clark County Auditor is Greg Kimsey. In addition to being the county’s chief financial officer, the auditor also oversees several essential county services: Audits, Auto Licensing, Elections, Marriage Licensing, Financial Services, and Recording. Additionally, the Auditor works with other elected officials, various state agencies, and the state legislature regarding issues that affect these services.

ft-vancouver-clark-county-historical-musuem

Fort Vancouver (image from Clark County Historical Museum)

Working with an elected auditor is interesting because they can bring a different perspective and things to consider. We do debriefs with the auditor throughout our audits, and at the end of the Scoping, Planning, and Fieldwork phases.

Can you describe your role as Audit Services Manager?

stafford

As the Audit Services Manager for Clark County, I serve as the principle auditor and am responsible for the quality of the Audit Services team (2016 Peer review). This includes reviewing audit work and administrative responsibilities such as developing and maintaining policies and procedures, hiring staff, performing the annual quality assessment, and coordinating peer reviews.  I also serve as a liaison between County departments and external auditors, like those from the Washington State Auditor’s office.

When I left OAD, Gary Blackmer, who was the director at the time, told me that I would have to “live with my audits” at the county. He was right. At the state, we would audit an agency and leave. But at the county we see the people we audit every day. We still report to a separately elected Auditor and so we are external to those that we audit, yet we are still internal to county government. We have to build relationships while always being aware of our independence.

What got you into performance auditing?

Luck! I spent ten years in the automation/ engineering field. That was mostly robotics-related work including electrical design, project management, and programming. I also spent some time in the financial services industry.

I wasn’t happy with my work-life balance, so I went to graduate school and earned my MBA as a path to a new career. I attended a job fair with a classmate who went to speak with the Oregon Audits Division about financial auditing. At the booth they also told me about this interesting and challenging field called performance auditing. I’ve always been driven to learn new things and this sounded like just the right fit. I was fortunate to get a position with OAD and to work with some very good auditors.

Larry’s take on performance auditing

One of the most important things I learned at OAD was that performance auditing is about understanding the issues and finding solutions – not placing blame. Also, reporting when you find things are happening the right way is just as important as identifying problems. It provides objectivity to your report and delivers the assurance expected from an audit.

For example, the first audit I worked on at OAD examined if state agencies hired former employees as contractors.  We found some instances, but all of the required procurement processes were followed and the award decisions were appropriate. Saying that in an audit report certainly didn’t get headlines but it did provide assurance.

Working at OAD was really interesting.  I was involved in a lot of issues at different levels, auditing different agencies and topics.

Coordinating with state auditors

clarkcountyThe Washington State Auditor’s office is doing some work around accountability. As part of this work, they are reviewing the financial affairs of municipal governments, state and local laws, and contracts to assess risk and evaluate compliance, mostly looking at internal controls. What I help with is making sure there is no duplication of work, and help facilitate communication and work with Clark County staff. I also help provide context and clear up any confusion, similar to an internal auditor.

Providing training

Internal control failures are common issues identified in our audits. So each fall, we host a half-day seminar on fraud and internal controls for County managers. I’ve found this kind of training is helpful for managers who are often experts in their field but have not been exposed to management tools like the COSO framework or the fraud triangle.

The training doesn’t take a lot of our time, but there is a big pay off. When these fundamental issues are addressed, then our performance audits can focus on higher level issues. The trainings also help to build relationships and credibility, and we get good feedback from managers who attend. They’re very interested in this information and how they can apply it to improve their departments.

How do you choose your audit topics?

We complete a biannual risk assessment and create a schedule of proposed audits. This includes input from elected officials, employees, citizens of Clark County, and our auditors. This proposed schedule is presented to our Audit Oversight Committee, of which the Clark County Auditor is the chair. The committee recommends audit priorities before we finalize our schedule. The County Auditor can approve audit work on issues that emerge between biannual risk assessments.

What are some interesting audits you’ve worked on at Clark County?

An audit I did on economic development was probably the most interesting audit I’ve worked on. Economics has a lot of intertwined theories, so it’s hard to nail down criteria. We wanted tangible criteria because it doesn’t matter what’s happening if you can’t agree on how to measure things. In the end, we also adjusted our conclusions based on the more questionable evidence. This audit received an ALGA Knighton Award as a result of this approach and how we treated the evidence.

Audit of Clark County’s Job Creation- Fee Waiver Program 

“The Clark County Auditor’s Office evaluated an economic development program designed to spur job creation through waiving government fees. The audit found that the fee waiver program cost Clark County approximately $8 million in its first year in forgone revenue and was not cost effective. Objective in its treatment of evidence and sensitive to its policy implications, the audit makes two strong recommendations: 1) either discontinue the program, or 2) implement significant changes to improve the program. The audit won the exemplary award for its scope and methodology, significant impact and for its clear and concise report writing.”

fee-waiver-audit

We’re also completing a series of inventory audits that look at high-risk assets, like ammunition and fuel. These material management audits are more interesting than you’d think! We’ve found opportunities to improve operations while decreasing costs, risk, and liability.

Sheriff’s Office High Risk Equipment and Supplies Management Audit 

“This was an impactful audit that focused on internal controls of high-risk materials utilized by the Sheriff’s Office, taking into account public safety as well as accountability. Findings and conclusions directly tied to objectives. Recommendations when implemented will not only have a cost savings but will increase efficiency and enhance accountability. The report was well organized with effective use of photos and charts.”

What is your favorite part about your job?

When I was a kid, I would research different topics and learn all about them. In doing this work, I have the opportunity to that for a living. It’s also all across a wide spectrum of services — it’s very hard to get bored! My favorite part about this job, though, is getting to work with other auditors. They’re so intelligent and passionate about their work.

Auditors at Work Featured Performance Audit Regional Roundup: Talkin' Shop

ELGL RePost: Your department has been selected for an audit

So your department, office, division, or program has been selected for an audit. You and your coworkers might be worried that a team of humorless scolds is coming by to tell you how to do your jobs. But have no fear! As auditors, our job is to work with you to make whatever you do more efficient and effective, so that all of us can provide the best service possible to our residents. Also, we have a great sense of humor.

Sam Naik, who works for the Office of the City Auditor in Austin, Texas, eloquently summarizes for Emerging Local Government Leaders (ELGL) what we as auditors do and what agencies that are involved in an audit can expect before, during, and after an audit. It covers the standards we follow, the planning process for audits, fieldwork, and reporting, as well as follow ups after the audit.

This is a great read for anyone that fears an audit may be coming down the pike for their agency, for those interested in becoming auditors, and for those who just want to have a better understanding of what an audit looks like and what it can achieve.

Read more here.

Accountability and Media Auditors at Work Featured How To

Things to consider when choosing performance audit topics

How does a performance audit shop decide what topics, and which agencies, warrant an audit? After all, a quick scan of your morning newspaper will stir up a seemingly endless number of important public issues. We can’t get to them all- not all at the same time, anyway- so how do we pick one topic over another?

One illuminating response to that question comes from the schedule of the Auditor General of British Columbia:

Because there are more projects than we can carry out, we use a comprehensive and systematic process to select the topics that best meet our mandate and will have the greatest impact. Potential performance audit topics come from:

  • the work we’ve completed and are currently engaged in
  • discussions with stakeholders, including the public service, government and the opposition
  • information and requests from MLAs, the people of B.C. and other stakeholders
  • the work of other audit offices
  • changes to government sectors and programs

Topics

The same considerations largely apply in Oregon as well. Auditors here at the Oregon Audits Division (OAD) work on a wide range of potential topics, limited only by the range and breadth of public services state agencies deliver. In addition to the considerations outlined above, we also look at broader issues that affect multiple agencies and the general public. We consider topics where an audit may add value and clarity to the discussion, or provide guidance to an agency in need of a course correction to achieve its mission and goals. These include areas of great significance to the general population and costs to taxpayers (education, health care, environment), areas prone to risk (IT systems), and many more of the kinds of complex ‘wicked’ problems that require sustained efforts to ensure that they are properly managed.

Timing

The timing of an audit can affect the relevance and value of the audit findings and recommendations. An immediate issue that receives a lot of public attention in March may change drastically by October, so it is worth considering whether to go ahead with an audit when circumstances might render the findings obsolete by the time the report is released. For that reason, we choose topics with much care and an eye to systemic long-term benefits. The results of our audits ideally help public agencies avoid major issues to start with- not just attempt to right what has already gone wrong.

Audit Schedules

Many audit shops prepare and even publish audit schedules well in advance. Some audit topics are requested or mandated, and others are issues (or agencies) that haven’t been visited in a while and are ripe for review. That said, apart from mandated audits, an audit schedule is less a promise than it is a preview of potential coming attractions. Other, more needful, questions may arise between now and the beginning of the next audit schedule. Maintaining some amount of flexibility helps audit shops share a perspective on issues that is timely and useful.

A few audit schedule examples:

Multnomah County Auditor

Auditor General of British Columbia

Audit Scotland

 

OAD and our peers both near and far approach topic selection thoughtfully, deliberately, and with a long list of considerations in mind.

Please consult our website to delve deeper into our recent performance audits, or visit our audit snapshots page to get a taste of our recent performance audit findings.

 

Auditors at Work Featured Performance Audit

GAO WatchBlog ReBlog: Exploring alternatives to prison

While these alternatives may be less expensive than imprisonment, we found that DOJ wasn’t measuring outcomes as well as it could. For example, DOJ surveyed U.S. Attorneys in 2014 and 2015 to get information about the use of certain pretrial diversion practices—but didn’t collect data on outcomes or costs.

And it’s those outcomes—such as successful reentry into society and reduced recidivism (relapse into criminal behavior)—that could be key in reducing the federal prison population.

Read more about the GAO’s analysis of alternatives to prison here.

 

Accountability and Media Auditors at Work Featured

Talkin’ Shop with: Brian Evans, Metro Auditor

About Metro’s Audit Shop

MetroAuditor Brian Evans 2015The Metro Auditor’s Office mission is to make Metro programs more transparent, efficient and effective and to ensure that its activities are accountable to citizens. We do this by conducting independent and objective assessments and reporting our findings and recommendations.

It is our vision to be completely relevant and efficient, choosing the right areas to audit and completing audits quickly so that Metro can constantly improve its services and be accountable to the public.

The Metro Charter created the office of Metro Auditor, to be elected at large for a four-year term, beginning in 1995.

So far in 2016, reports have been released on the financial condition of Metro, a follow-up audit on travel practices and ethics, and an audit on community planning and development grants.

In addition to performance audits, the Metro auditor also publishes annual reports and peer reviews.

The Metro Accountability Hotline, administered by the Metro Auditor’s Office, gives employees and citizens an avenue to report misconduct, waste or misuse of resources in any Metro or Metropolitan Exposition Recreation Commission facility or department.

MetroAuditorLogo seems like a pretty unique government structure. Can you describe it?

It is quite unique. Metro is one of only two regional governments with elected officials in the country. It encompasses Clackamas, Multnomah, and Washington counties and is governed by a nonpartisan council, with six district councilors, each representing different parts of the region. Metro’s council president is regionally elected. The region represents 1.5 million people in 24 cities and three counties. MetroAuditorMapBoudaries

Map of the Metro area, indicated with the blue border.

When it comes to transportation planning, Metro is the federally mandated metropolitan planning organization that develops an overall transportation plan and to program federal funds through the Joint Policy Advisory Committee on Transportation (JPACT). JPACT is comprised of locally elected officials from across the region. They strategize how to use federal funds and are high-level decisions-makers.

The Metro Council also oversees the Metropolitan Exposition and Recreation Commission, or MERC. MERC Commissioners are appointed by the Metro Council based on the recommendation of their local area governments. MERC has seven members who represent the city of Portland, Metro, and one each for Clackamas, Multnomah and Washington counties. The commissioners serve four-year terms.

Services of Metro:

So, when you were a kid, what did you want to be when you grew up?

An architect.

Architecture and auditing seem very different— do you see any similarities?

Both professions focus on the details while keeping an eye on the big picture. Architects balance aesthetic considerations with the practical needs of the people who will use the building they design. Performance audits have to balance similar considerations. Effective auditing combines practical information with creativity to produce a report that the public, decision makers and managers can use to understand the need for improvements. In audit reports we choose the words, tone and data points we think will motivate change. This seem analogous with the choices an architect would make for the materials, color and details of a building.

Prior to becoming an auditor, what did you do?

In undergrad, I studied International Relations and Economics. While in school, I studied abroad in Spain, Kenya, and Tanzania. After that, I served as an AmeriCorps member working on microfinance programs at Mercy Corps. In graduate school, I pursued a Master’s degree in International Public Affairs at the University of Wisconsin-Madison. I moved back to Oregon shortly after grad school and worked in state government as the Senior Economist in the economic development department. While there, I did financial and economic analysis of industry trends and business incentives.

How did you get your start in auditing?

In my position as the Senior Economist, I found that I wanted to point out ways we could improve.  There were strategies I thought could help, and recommendations I wanted to make. Essentially, I  wanted to recommend evidence-based policy— which is similar to what auditors do. I learned about the performance auditing in graduate school when the Government Accountability Office did recruiting visits. In Oregon, I did some informational interviews with local government auditors. In 2008, I started my first auditing position with Metro as a Senior Management Auditor.MetroAuditorLandscapeCartoon

What made you stay in auditing?

When I came to Metro, I enjoyed the structured approach to auditing that the office followed. I appreciated and found helpful the office’s commitment to a high level, quality reporting based on the leadership of Jewel Lansing  and Gary Blackmer. Suzanne Flynn, the Metro Auditor I worked for, was committed to the development of staff, and this is something I strive to do as well.

As a Senior Management Auditor and then as a Principal Management Auditor with Metro, I really enjoyed the variety of topics I worked on. I also liked getting into the weeds and the structure of the workflow in performance auditing— the different phases. Programs at Metro tend to be innovative, willing to take risks to move things forward, and have long-term thinking as part of their philosophy. Prior to becoming the elected Auditor of Metro, I had the opportunity to get a good sense of the different parts that make up Metro. I conducted audits on internal services such as payroll, IT, and policy audits around solid waste planning and transportation planning.

You became the Metro Auditor fairly recently. What was the election process like? How’s it going so far? 

I decided to run at the end of 2013, and ended up running unopposed. Running unopposed takes some of the pressure off, but once you commit to running there is a method and timing involved in completing everything you need to complete—the filing deadline, creating a MetroAuditorOfficewebsite, and outreach. I took office in 2015 for a four-year term and am now about halfway though.

One of the things I am most proud of accomplishing in the past two years is putting such a strong team together. There were three vacancies when I took office, as my position needed to be filled, Mary Hull Caballero became Portland’s Auditor, and we had a retirement. I was able to rebuild the office and with my team, keep producing high quality audits.

How do you create your audit schedule

Creating the audit schedule is a process of gathering a lot of data and then analyzing it to identify the areas of greatest need.

We create a list of audit topics based on my discussions with Metro Councilors and department directors, the public, along with topics staff have identified in their audit work. As an office, we have a retreat and conduct a risk assessment, look at impact of potential audit topics, and consider timing and available resources. From there, I select the topics we will audit and publish the list on our website.

What are some current, or recently released audits that you are most excited about?

In February, we released an audit that provides a good picture of the challenge of auditing land use planning. It was also challenging to find a balance between the planning jargon and language the average reader can understand. In the audit, we found decreased geographic alignment between the grant project areas and the ones identified in the regional plan as priorities areas for development. We also found that the program did not develop clear expectations for grant monitoring. The deliverables for some projects were not fully met prior to payments being made. It was not always clear what amount of time should be spent monitoring or how much discretion there was to evaluate deliverables.

Right now, we are conducting an audit of the Zoo, looking at organizational culture. It is challenging to figure out how to audit organizational culture, and then communicate that in a public report. It is making us think about issues from a different perspective.

Follow the Metro Auditor on Twitter! @MetroAuditor 

Check out Metro’s mapping feature that uses GIS and see where the trails, neighborhoods, school districts, and more are.

MetroAuditorMapTrails

Map of the Metro area with trails indicated in green.

Auditors at Work Featured Regional Roundup: Talkin' Shop