Stats in Auditing: Retirement, income replacement rates, and non-linear relationships

Are you saving enough for retirement? Maybe you have given this question some serious thought. Then again, maybe not… According to research cited in a recent GAO audit report on retirement security, a large portion of workers in the United States have not done a whole lot in terms of retirement planning and many people don’t know how much they should be saving.

Data Wonk Featured Stats in Auditing

Stats in Auditing: Medicare, hospital-physician consolidation, and panel data analysis at the GAO

In an audit released last December, the Government Accountability Office (GAO) looked at whether increasing consolidation between hospitals and physicians had something to do with more evaluation and management (E/M) services being provided at hospital outpatient departments (HOPDs), rather than in the more typical setting of a physician’s office.

Why does this matter?

According to the audit, the concern is that these trends may be driving up Medicare expenditures. As the GAO notes in the report, Medicare pays providers a higher rate for E/M visits and other types of procedures if the service is provided at a hospital facility (at a HOPD) than if the same service is performed at a physician’s office. This rate premium then creates an incentive for shifting the provision of services like E/M visits from physician offices to hospitals. As more hospitals acquire physician practices or hire physicians as regular employees (also known as hospital-physician vertical consolidation or vertical integration), more hospitals are able to provide these services and be eligible for the higher rate.

The evidence

While performance audits routinely use data as evidence or background information, not many use statistical analysis in their methodology, which makes this GAO audit even more interesting. To make their analysis possible, the GAO auditors combined various sources of medical data to construct a dataset of county-level variables (for 3,121 counties) with annual observations over the 2007-2013 period.

The main variables of interest in the dataset are:

  1. The median percentage of all E/M visits in a given county that were conducted at hospital outpatient facilities.
  2. The level of hospital-physician vertical consolidation in each county, which is a measure of how much a county is served by hospitals that have consolidated with physician practices.

After grouping counties into varying levels of hospital-physician consolidation, the auditors compare the median percentage of E/M office visits conducted in hospital facilities for each of these groups.

The results for all years are in table-form in Appendix III of the report, but to make it easy on the eyes, we quickly put the data in the tables through Tableau and came up with some colorful charts (click to see the full interactive visualization):

E_M Office Visits and Vertical Consolidation

The charts show that counties with higher levels of hospital-physician consolidation tend to have higher percentages of E/M visits performed at hospital outpatient facilities. The charts also show that this positive correlation between the two variables increases over time.

But correlation does not imply causation. And it could certainly be the case that the increase in the volume of E/M visits performed at hospital outpatient facilities could be attributed to other factors, in which case the correlation would be a spurious one. This is where statistical analysis methods – such as regression analysis – come in handy because using these methods can help test whether the relationship between two variables is still there after accounting for other factors.

In this case, the GAO auditors used regressions to check whether the correlation above changes or disappears after adding other (relevant) variables into the mix, such as the level of competition between hospitals and between physicians in each county.

But what separates these regressions from the standard regression model is that they exploit the nature of the dataset used in the audit, which follows thousands of counties over several years. This type of dataset allows for the use of panel data analysis methods that have some advantages over standard regressions. In particular, it allows for the use of “fixed effects models” that can account for unobserved differences in county characteristics that remain stable over certain periods of time – such as urban/rural status, the state in which a county is located – as well as factors that vary over time but that affect all counties simultaneously, such as national boom and bust economic cycles. Note that the emphasis is on unobserved, and this is because the models automatically account for these differences without the need for the user to add data on these unobserved characteristics.

The regression results are presented in Appendix II and confirm the positive relationship between hospital-physician consolidation (‘Vertical consolidation’ in Table 4) and the percentage of E/M visits performed at hospital outpatient facilities.

Based on the results, here is what the GAO recommends: “In order to prevent the shift of services from physician offices to HOPDs from increasing costs for the Medicare program and beneficiaries, Congress should consider directing the Secretary of HHS to equalize payment rates between settings for E/M office visits–and other services that the Secretary deems appropriate–and to return the associated savings to the Medicare program.”


Luis Sandoval OAD Performance Auditor

Luis Sandoval
OAD Performance Auditor

Data Wonk Featured

Metro Auditor: Nature in Neighborhood grant program could improve performance measurement

Last week, the Metro Auditor’s Office published its latest audit report on the Nature in Neighborhoods grant program. With more than $18 million in grant awards since 2006, the program has funded habitat restoration, land acquisition, education, and trails development projects in the Metro area.

The goal of the audit was to check whether the grant program has been well-managed. This is important for Metro taxpayers because, as auditors everywhere emphasize, a well-run program improves accountability and the likelihood that program goals are met.

In their review of Nature in Neighborhoods, Metro auditors found that the program follows some of the best practices for effective management in the grant-making world.

metro audit 12.15

However, they also found that Nature in Neighborhoods could see improvements in some areas, including measuring results and grant monitoring. In particular, the program has assessed the performance of some grants, but does not have a fully developed performance measurement system that can allow managers to evaluate the project’s impact as a whole.

The auditors at Metro offer some great recommendations to improve the grant program. For example, they recommend establishing targets for each of the program’s goals and developing a system of indicators to measure grant outcomes. This may be easier said than done. However, a solid performance measurement system can be a valuable tool for Metro program managers to see where the program is meeting targets and having a positive impact. It can also help identify areas in need of improvement and where Metro’s efforts should be focused.

Read the full audit report on Metro’s website. And if you get tired of reading about audits and best practices for effective grant management (although, why should you be?), check out the cool videos about some of the projects sponsored by the Nature in Neighborhoods program (and watch one below!). The videos certainly inspired this auditor to get up and go for a walk in the forest!

Featured New Audit Release