Internal Auditor ReBlog: Let’s talk about feedback

Through frequent conversations with practitioners who are relatively new to the internal audit profession — including both people within and outside my organization — it seems there is a disconnect when it comes to feedback. Manager-level employees tell me they often provide informal feedback to the staff and senior auditors who work with them. Meanwhile, those same managers’ staff and seniors say they don’t receive enough feedback, don’t know if they are “on track,” and don’t know what they are doing well and what they can im​prove. This is where a few simple conversation areas can reap great benefits.

Laura Soileau, a director in Postlethwaite & Netterville’s Consulting Department in Baton Rouge, Louisiana, discusses the importance of ongoing communication and relationship building in the workplace when delivering – and receiving – feedback (both in formal performance evaluations and day-to-day). She provides a list of useful questions for supervisors and staff to ask each other, and to ask themselves. Maintaining healthy working relationships and keeping the lines of communication open, professional, and productive “should be a shared responsibility.” Feedback is crucial to keeping performance on track, but in this case, the ‘who and how’ is almost as important as the ‘why.’

Read more here!

Accountability and Media Featured

Methods (to our madness): Complex analysis in the public eye

The Secretary of State recently released a performance audit on the Oregon Health Authority: Oregon Health Authority Should Improve Efforts to Detect and
Prevent Improper Medicaid Payments. This audit received a lot of media exposure, in part due to an Audit Alert released in May, some months before the scheduled audit release date. Unsurprisingly, this led to more than a little pressure. How did our 4 person audit team (Ian Green, Wendy Kam, Kathy Davis, and Eli Ritchie) approach this audit, stay cool under fire and make sure their conclusions were sound? I sat down with the lead auditor, Ian Green, to find out more.

You led the OHA, Improper Medicaid Payments audit. What are your strategies when you’re faced with a complex agency and a complex topic?

When we started this audit, we knew we’d be looking at improper payments. Even that’s such a big topic, we knew we’d need to scope it down where we could. So we got as much information as we could from all levels – hundreds of interviews with officials and analysis, looking at agency documentation, research on best practices, all of that.

What methods did you use to identify improper payments?

Our primary focus was to look at process issues, but we did attempt to find some improper payments. We used audit software to analyze large data sets. We did a lot of data matching and looked for results that were outliers. For instance, we checked to see if providers were getting duplicate reimbursements. It’s a complex system, so providers and billers might make errors that should be caught before payments are set out. Another example was checking to see if there were people enrolled in the Oregon Health Plan who shouldn’t be – like if someone had moved out of state.

 What challenges did you face doing this audit, and what strategies did you use to address them?

One challenge was the sheer amount of data. We looked at over two hundred million records.  There was so much data that running tests could take a very long time. My team would run a script and leave it overnight to finish. We had to be very careful about how we set up our tests. Since we kept everything scripted out, each time we got new information, we could just update that script. That kept the testing sustainable, which is very important given all the last minute information we received.

To address the complexity of the topic, we separated our approach into three subtopics: prevention, detection, and recovery. Each person on the team focused on one area, and we’d meet to discuss weekly. That helped make sure we covered all the information while still working together closely.

Another challenge was trying to get complete data. We’d request data and be told we had it all. And then we’d find out it was incomplete. That meant we had to continue reworking our analysis constantly. Without scripting, it would have been extremely time-consuming to perform this work manually.

What was the hardest thing about completing this audit in the public eye?

It’s a very sensitive topic. We knew that we’d get a lot of scrutiny. But we did what we always do, which is to work really hard to make sure all our conclusions are accurate and well-supported, and put all our work through a thorough quality assurance process.

 Is there anything you wish non-audit folks knew about the audit process?

Generally, there’s a public perception that an audit should find everything that might be going wrong. Auditors look at a higher level to see if there are controls in place to prevent something from going wrong. If we’re concerned, we may do deeper testing to see what’s actually happening. For instance, we looked at processes to manage improper payments. Our goal wasn’t to find all of the improper payments being made. Our testing helped measure the effect of the processes that are currently in place.

Anything else?

It was a big audit. We’ve been excited to see important changes happening, even while we were still working on the audit. The Oregon Health Authority is working to address weaknesses in their processes and being more transparent. That’s a really good outcome, from our perspective.

 

Check out the audit here: http://sos.oregon.gov/audits/Documents/2017-25.pdf

Auditing and Methodology Auditors at Work Featured Performance Audit

Audit Release: DEQ Should Improve the Air Quality Permitting Process to Reduce Its Permit Backlog and Better Safeguard Oregon’s Air

Report Highlights


The Secretary of State’s Audits Division found that the Oregon Department of Environmental Quality (DEQ) should evaluate staffing and workloads among air quality permit writers and provide better guidance to both staff and businesses to help reduce the agency’s air quality permit backlog.

Background

This audit reviewed air quality permitting at the Oregon Department of Environmental Quality. Air quality permits regulate the types and amounts of air pollution businesses are allowed to emit, based on federal pollution limits set by the Clean Air Act and state limits established in state laws and DEQ rules.

Audit Purpose

The purpose of this audit was to determine how DEQ could improve its air quality permitting process to better safeguard Oregon’s air quality.

Key Findings

The Oregon Department of Environmental Quality has a significant backlog in air quality permit renewals. We found that:

  1. 43% (106 out of 246) of DEQ’s largest and most complex federal and state air quality permit renewals are overdue for renewal. Additionally, more than 40% of the most complex permits issued from 2007 to 2017 exceeded timeframes established by DEQ or the Clean Air Act, some by several years.
  2. DEQ struggles to issue timely permits and renewals due to a variety of factors, including competing priorities, vacancies, and position cuts that have created unmanageable workloads. Other factors include inconsistent support and guidance for staff; a lack of clear, accessible guidance for applicants; and increased time for the public engagement process.
  3. Untimely permits, combined with a current backlog of inspections, endanger the state’s air quality and the health of Oregonians. For example, when DEQ does not issue permit renewals on time, businesses may not provide DEQ with data showing they are complying with new or updated rules.

To reach our findings, we conducted interviews, analyzed air permit data, reviewed documents and reported practices, and researched leading practices.

Key Recommendations

Based on our review of leading practices and air quality agencies in other states, the report includes ten recommendations to the Department of Environmental Quality. Recommendations include evaluating permit writer workloads and staffing, clarifying the public engagement process, providing better guidance to permit writers and businesses, and conducting a process improvement effort.

The agency agreed with our findings and recommendations. Its response can be found at the end of the report.

Read the full report here.

Featured New Audit Release Performance Audit

Audit Release: Stronger Accountability, Oversight, and Support Would Improve Results for Academically At-Risk Students in Alternative and Online Education

Report Highlights


The Secretary of State’s Audits Division found that the Oregon Department of Education (ODE) has not focused on improving education for at-risk students in alternative and online schools and programs, though they account for nearly half the state’s high school dropouts. Sharpening Oregon’s focus would improve accountability, district oversight, and school and program performance, and would benefit at-risk students and the state’s economy.

Background

Many vulnerable students attend Oregon’s alternative schools and programs and online schools. Responsibility for improving education for those students is shared by ODE, school districts, and others.

Audit Purpose

To determine how ODE and school districts can help increase the success of academically at-risk students in alternative and online education. Online and alternative education schools and programs also serve students who are not academically at-risk. The audit did not focus on their effectiveness with these students.

Key Findings

  1. ODE has not adequately tracked and reported on the performance of alternative schools and programs. As a result, the state lacks critical information about school and program effectiveness.
  2. Enhanced state monitoring and support, and more robust district oversight could improve results for at-risk students in alternative schools and programs, and in online schools.
  3. Some states have held districts, alternative schools, and programs to high standards and provided more support to help at-risk students succeed.
  4. Other states have also increased oversight of fast-growing online schools. In contrast to these states, Oregon’s laws allow online schools to increase enrollment rapidly regardless of their performance.
  5. To reach our findings, we interviewed multiple stakeholders, reviewed documents, analyzed school performance data, researched practices in other states, visited schools, and surveyed all of Oregon’s school districts. Our office also released an audit of graduation rates recently that focuses on students in traditional high schools.

Key Recommendations

This audit includes recommendations designed to improve results for at-risk students in alternative and online schools and programs. ODE should develop a more meaningful accountability system for alternative and online education. The agency should establish and monitor standards for crucial practices, such as annual district evaluations of these schools and programs. ODE should also strengthen state attendance and funding standards for online schools.

ODE generally agreed with our recommendations. The agency’s response can be found at the end of the report.

Read full report here.

Featured New Audit Release

Audit Release: The Oregon Department of Education Should Take Further Steps to Help Districts and High Schools Increase Oregon’s Graduation Rate

Report Highlights


The Oregon Department of Education (ODE) has prioritized improving four-year graduation rates in recent years. The Secretary of State Audits Division found ODE could make further progress by helping schools and districts focus on specific groups, such as students who transfer between districts, low-income students, and middle school students. ODE can also better help districts and schools use effective improvement tools, use data to identify students in danger of not graduating, and communicate the importance of graduation to parents and the community.

Background

One in four Oregon public high school students does not graduate on time. Current steps to boost on-time graduation rates include plans to reduce chronic absenteeism, prevent students from dropping out, increase access to college-level courses in high school, and increase career-technical education.

Audit Purpose

The purpose of the audit was to determine how ODE and school districts could increase four-year graduation rates in Oregon’s public high schools.

Key Findings

Through school visits, interviews, data analysis, and document reviews, we found that:

  • Students who changed districts during high school – more than a quarter of all high school students – had graduation rates roughly 30% lower than students who did not. ODE does not analyze or report graduation performance for these students.
  • Schools with mid-range graduation rates – 67%-85% – receive limited improvement support from ODE, though most non-graduates attend these schools.
  • More than 70% of students who do not graduate on time are low-income. ODE should assess the need for services to help those students succeed.
  • The Legislature and ODE has not emphasized middle school performance or student transitions from middle school to high school, though students who struggle in middle school are already at risk of not graduating.
  • ODE does not track student grades or specific credits attained, data the agency could use to help more students graduate.
  • ODE should improve its internal communications and help districts and schools communicate the importance of graduation to parents and the community.

Recommendations

The report includes recommendations to the Oregon Department of Education on additional efforts it could take to increase on-time graduation rates. Among them: focusing on specific student groups, supporting coordination between middle schools and high schools, collecting more detailed student data, and helping districts better use improvement tools.

The Oregon Department of Education generally agreed with our findings and recommendations. The agency’s response can be found at the end of the report.

Read the full report here.

Featured New Audit Release

Audit Release: Higher Education Coordinating Commission Needs to Address Weaknesses in Procurement Practices

Report Highlights


The Secretary of State’s Audits Division found that the Higher Education Coordinating Commission state agency’s (HECC) current procurement practices are exposing the agency to legal, security, and public perception risks. Overall, HECC lacks an effective procurement system to ensure services and goods are procured in compliance with state laws.

Background

HECC is responsible for funding and coordinating public higher education in Oregon. It was established in 2011 as a volunteer commission to focus on strategic planning for public postsecondary education in the state. HECC’s structure has expanded and now consists of an established state agency with 115 budgeted full time equivalent positions in eight offices.

Purpose

The purpose of this audit was to review procurement practices at HECC and identify opportunities to improve current practices.

Key Findings

Within the context that state procurement rules are complex and intended to benefit the state as a whole, we found that:

  1. HECC leadership has not implemented a governance structure to ensure procurements are made in compliance with state laws and rules.
  2. Of the 748 HECC contracts and agreements open from November 2016 to March 2017, 65% were executed after their effective date and 53 or 7% were considered backlogged.
  3. A lack of clearly defined procurement roles and responsibilities and insufficient training has created confusion and inconsistent procurement processes and practices across the agency.
  4. HECC current practices are noncompliant with state procurement laws and rules, exposing the agency to legal, security, and public perception risks.
  5. To reach our findings we conducted interviews and reviewed agency documents, state procurement laws and rules, contract files and agreements, accounting records, and other accounting supporting documentation.

Recommendations

To establish and maintain a robust procurement process, we recommend HECC create a governance structure that clearly defines procurement roles and responsibilities and fully develops, implements, and trains staff on procurement roles, policies, processes, and procedures. We also recommend HECC management ensures consistent adherence to state procurement law and rules.

The Commission generally agrees with our findings and recommendations. Their response can be found at the end of the report.

Read the full report here.

Featured New Audit Release

TEDx ReBlog: 9 common-sense rules for getting the most out of meetings

Been in a meeting recently? Sure you have.

When it comes to making the most out of meetings (i.e., productive, clear, professional, and as brief as possible), it’s not uncommon that we drop the ball from time to time. But that doesn’t mean we can’t strive to run meetings as effectively as possible.

Ray Dalio with Bridgewater Associates has some sounds suggestions for making the most of that time you and your coworkers spend locked away in a conference room trying to change the world (or at least, make a decision on the best font for your quarterly report). Clear objectives, firm leadership, and focus top the list, but do yourself a favor and read more here.

Accountability and Media Featured