Methods (to our madness): Complex analysis in the public eye

The Secretary of State recently released a performance audit on the Oregon Health Authority: Oregon Health Authority Should Improve Efforts to Detect and
Prevent Improper Medicaid Payments. This audit received a lot of media exposure, in part due to an Audit Alert released in May, some months before the scheduled audit release date. Unsurprisingly, this led to more than a little pressure. How did our 4 person audit team (Ian Green, Wendy Kam, Kathy Davis, and Eli Ritchie) approach this audit, stay cool under fire and make sure their conclusions were sound? I sat down with the lead auditor, Ian Green, to find out more.

You led the OHA, Improper Medicaid Payments audit. What are your strategies when you’re faced with a complex agency and a complex topic?

When we started this audit, we knew we’d be looking at improper payments. Even that’s such a big topic, we knew we’d need to scope it down where we could. So we got as much information as we could from all levels – hundreds of interviews with officials and analysis, looking at agency documentation, research on best practices, all of that.

What methods did you use to identify improper payments?

Our primary focus was to look at process issues, but we did attempt to find some improper payments. We used audit software to analyze large data sets. We did a lot of data matching and looked for results that were outliers. For instance, we checked to see if providers were getting duplicate reimbursements. It’s a complex system, so providers and billers might make errors that should be caught before payments are set out. Another example was checking to see if there were people enrolled in the Oregon Health Plan who shouldn’t be – like if someone had moved out of state.

 What challenges did you face doing this audit, and what strategies did you use to address them?

One challenge was the sheer amount of data. We looked at over two hundred million records.  There was so much data that running tests could take a very long time. My team would run a script and leave it overnight to finish. We had to be very careful about how we set up our tests. Since we kept everything scripted out, each time we got new information, we could just update that script. That kept the testing sustainable, which is very important given all the last minute information we received.

To address the complexity of the topic, we separated our approach into three subtopics: prevention, detection, and recovery. Each person on the team focused on one area, and we’d meet to discuss weekly. That helped make sure we covered all the information while still working together closely.

Another challenge was trying to get complete data. We’d request data and be told we had it all. And then we’d find out it was incomplete. That meant we had to continue reworking our analysis constantly. Without scripting, it would have been extremely time-consuming to perform this work manually.

What was the hardest thing about completing this audit in the public eye?

It’s a very sensitive topic. We knew that we’d get a lot of scrutiny. But we did what we always do, which is to work really hard to make sure all our conclusions are accurate and well-supported, and put all our work through a thorough quality assurance process.

 Is there anything you wish non-audit folks knew about the audit process?

Generally, there’s a public perception that an audit should find everything that might be going wrong. Auditors look at a higher level to see if there are controls in place to prevent something from going wrong. If we’re concerned, we may do deeper testing to see what’s actually happening. For instance, we looked at processes to manage improper payments. Our goal wasn’t to find all of the improper payments being made. Our testing helped measure the effect of the processes that are currently in place.

Anything else?

It was a big audit. We’ve been excited to see important changes happening, even while we were still working on the audit. The Oregon Health Authority is working to address weaknesses in their processes and being more transparent. That’s a really good outcome, from our perspective.

 

Check out the audit here: http://sos.oregon.gov/audits/Documents/2017-25.pdf

Auditing and Methodology Auditors at Work Featured Performance Audit

Audit Release: Oregon Health Authority Should Improve Efforts to Detect and Prevent Improper Medicaid Payments

Report Highlights


Our audit found that Oregon Health Authority (OHA) recovery efforts are appropriate and reasonable, but the agency should strengthen efforts to detect and prevent improper payments in Oregon’s $9.3 billion per year Medicaid program. Prevention of improper payments is more cost-effective than attempting to recover improper payments. We also found that delays in processing eligibility for thousands of Oregon’s Medicaid recipients resulted in millions of dollars of avoidable Medicaid expenditures, a critical issue the agency failed to disclose until raised in a May 2017 Auditor Alert. Furthermore, OHA did not timely disclose relevant information, which impeded our audit work. OHA’s new management has been more proactive and transparent in addressing these issues.

Background

An improper payment is defined by the federal government as “any payment that should not have been made or was made in an incorrect amount (including overpayments and underpayments) under statutory, contractual, administrative, or other legally applicable requirements or where documentation is missing or not available.”

Purpose of Audit

The primary purpose of the audit was to determine if the Oregon Health Authority could improve processes to prevent, detect, and recover improper Medicaid payments. The secondary purpose was to follow-up on OHA’s progress to resolve issues we raised in our May 2017 Auditor Alert.

Key Findings

Within the context that Medicaid is a very complex and challenging program to administer, we found:

  1. OHA has gaps in procedures for preventing certain improper payments. Insufficient management of the agency’s processes for identifying and resolving payment and eligibility issues, prioritization of staffing resources, and efforts to address technology issues put taxpayer dollars at risk.
  2. OHA lacks well-defined, consistent, and agency-wide processes to detect certain improper payments, especially related to coordinated care. We identified approximately 31,300 questionable payments based on our review of 15 months of data. OHA needs to continue researching these claims to determine how many were improper; OHA reported that only a small percentage were improper based on preliminary research of 2,700 claims.
  3. OHA recovery efforts appear appropriate and reasonable, but may be underutilized due to OHA’s limited procedures for detecting improper payments.
  4. OHA reported completing the action plan to determine eligibility for the remaining backlog of 115,200 Medicaid recipients. Approximately 47,600 (41%) were deemed ineligible as a result, although this figure may decrease slightly through the end of November. Failure to address this issue in a timely fashion resulted in approximately $88 million in avoidable expenditures.

Recommendations

Drawing from national leading practices, our report includes eight recommendations to OHA focused on strengthening efforts to detect and prevent improper payments. Oregon Health Authority agrees with our recommendations. The agency’s response can be found at the end of the report.

Read full report here

 

Featured New Audit Release Performance Audit

Audit Release: Automated Medicaid eligibility is processed appropriately at OHA, yet manual input accuracy and eligibility override monitoring need improvement

AUDIT PURPOSE

In Oregon, over one million individuals have Medicaid coverage. Medicaid expenditures totaled $9.3 billion in fiscal year 2016, including $1.2 billion in state general funds. We conducted this audit to determine if two critical automated computer programs managed by the Oregon Health Authority accurately verify Medicaid client eligibility and accurately issue payments to healthcare providers. If these programs do not function properly, clients may inappropriately receive, or be denied, Medicaid benefits.

FINDINGS IMPACT

Manual input errors and lack of monitoring of overrides can cause inappropriate eligibility determinations and payments to providers. If agency leadership implements more effective monitoring of caseworker eligibility overrides and improves manual input accuracy, the state will better comply with eligibility requirements and increase accuracy of payments. Inaction will allow overrides and manual input errors to continue causing inappropriate payments to providers.

Read full report here.

KEY FINDINGS

  • Two critical automated computer programs appropriately determined eligibility, enrolled Medicaid clients in coordinated care organizations, and made appropriate payments to those organizations based on eligibility information received.
  • Automated computer processes appropriately validated the Social Security number and citizenship status of applicants over 99.7% of the time in our review of over 425,000 records.
  • We reviewed 30 eligibility determinations and found seven (23%) had manual input errors. While only one error resulted in a client being determined eligible when they were not, each of the errors related to application information that could have resulted in inappropriate eligibility determinations.
  • Although their volume has significantly decreased over time, overrides of eligibility are not sufficiently monitored, meaning unauthorized overrides of Medicaid eligibility could occur.
  • Our review of 72 overridden eligibility segments showed caseworkers did not take proper action to clear 25 (35%). Overridden segments are not subject to automated processes that redetermine eligibility for certain clients.
  • Our 2011 audit recommendations to OHA and DHS concerning access to the Medicaid Management Information System have not been fully implemented, increasing security risk.

RECOMMENDATIONS SUMMARY

  • OHA should continue efforts to improve caseworker manual input accuracy through additional training, and implement a review process for input where errors negatively affect eligibility determination.
  • OHA managers should monitor eligibility overrides to prevent unauthorized validation and ensure state resources are spent appropriately.
  • OHA and DHS should fully implement our 2011 audit logical access recommendations.
Featured IT Audit New Audit Release Performance Audit