The Evolution of New Auditor Training at OAD

How does training impact the work of new auditors?

We provide training to our newly hired auditors because government auditing is very different than business auditing. Unfortunately, no universities specialize in this field, and even a single class is rare. We try to update and improve our training each time, but what we’ve learned in the last year about our instructional design approach has given us new insights.

First, what we taught was not specifically designed to help new performance auditors conduct their day-to-day work. We explained about preparing a good workpaper, or using our software, Teammate. But there were gaps in the framework and perspective of auditing. To address those gaps, we began to develop trainings that complemented our practical nuts and bolts offerings with courses focusing on soft skills development and leadership.

Auditing has a maze-like feeling for the new hire but we sometimes forget that there are two perspectives of a maze – the bird’s eye view, and the view from inside. From above, our eye can trace the shortest route. But inside, there are no direction markers.  Success must come by assembling a clear memory of the layout of the maze through arduous trial and error.

Inside Hampton Court Maze

Inside Hampton Court Maze

Some people enjoy the puzzle of a maze and pay to wander through corn mazes. Some have no patience for mazes. The image to the right makes some folks feel like a lab rat. And not a very smart one, since I hear that lab rats are quite capable of negotiating mazes.

But even as we think about this frustration, we also need to ’embrace the maze’. Every new audit is an agency maze. We try to understand how all the program and financial pieces fit together, and build up the picture through interviews, data analysis, and observation.

On the financial side we have workpapers and team member continuity to guide us through the maze of risks each year. On the performance side, we are often entering new agency mazes and our initial, scoping phase produces a rough drawing of the maze to help us narrow down to an issue that we should audit.

Above Hampton Court

Above Hampton Court

The other maze is our audit process.  We want newly hired auditors to be able to negotiate their way through the agency maze without also getting lost in the auditing maze. The better we can prepare new auditors in our methods and procedures, how to audit, the more they can focus on what they are auditing.

Our instructional design efforts are laying out the tasks more clearly for new hires with 11 classes so far. For example, we have training on audit documentation, interviewing, elements of a finding, sufficiency of evidence, and report writing. This knowledge and skill set are spread out, presented when a new hire will encounter them, then reinforced as they do their work.

Nonetheless, we recognize that many aspects of auditing are affected by circumstances and context that can’t be set in procedure. Strategies, priorities, principles, and standards provide general guidance in these situations. Leadership is also important and we want to help our newly promoted team leads understand the new responsibilities and situations they must take on, and perform them with competence and confidence.

We are developing that team leader training in our next phase, and some intermediate auditor training, and some technical training on specific software or analytical tools.

This seems like a lot of work but we are asking our auditors to develop and offer the classes, which involves us all in the responsibility of defining expectations, developing the content, and presenting the instruction.

We want to shorten the learning curves of our auditors because that translates into highly professional work from everybody. Staff members that have had the opportunity to take part in thorough and holistic training are a key element of a learning organization.

Auditors at Work Featured Financial Audit Performance Audit

Auditors at Work: Spotlight on Sandy Hilton

Once a quarter we will be discussing the wonders of the world of auditing with (you guessed it) actual auditors! Our Summer Spotlight fell on Sandra Hilton, Performance Audit Manager with the Oregon Audits Division, and in her free time, newly adopted Cat Assistant.

What should we call you?
Call me Sandy.

How long have you been an auditor?
I’ve been an auditor for 26 years. I now work primarily on the performance side, but over the course of my career about half of that time was spent working on financial audits and hotline related work.

What led you to auditing?
I wanted to get my CPA certificate! Two years of relevant experience was required. That was the original reason I applied for the job as a staff auditor; however, once I got into it I found that I really, really liked it. I’ve been here ever since.

Many people, upon hearing the word ‘auditing,’ may assume that the job is just deadly dull. Is auditing boring?
I am never bored. Auditing is exciting! There are so many different things to look at, and it’s important work that you get to take on with a whole team. Teamwork, among other things, makes what we do interesting. Of course, after a few rounds of reading the same audit report, it does have its dull moments…

Why is your job important?
Because we can make a difference. We push things along- changing the course of a program or agency by incremental degrees and effecting some substantial long term changes. Similar to altering the trajectory of a rocket, tiny degree shifts make huge differences down the line. Audits build upon previous audit work and are often interconnected. We help the agencies and people we work with know which ‘tweaks’ can make a meaningful difference to achieving their goals.

Any personal victory stories, or tales of challenges overcome and differences made?
The TANF Audit we released a few years ago issued a number of recommendations that DHS used to  overhaul an underperforming program. In following, the most recent Governor’s Budget added $30 million to redesign and improve the program. Some of that money went to investments in support services and smoothing the transition for families off the ‘welfare cliff.’

Any auditing horror stories?
Not exactly a horror story, but it was definitely amusing. Years ago I did a site visit where I saw a safe with an ‘Open’ sign hanging on it. Apparently the safe was difficult to open, so the agency in question just left it open and put a double-sided ‘open-closed’ sign on the front. The safe was full of blank checks, and even had a signature stamp machine. Combined with the lack of segregation of duties and easy access to the safe, it was definitely a ‘one-stop’ shopping opportunity!

What do you do when you’re not auditing?
My number one hobby is gardening, and I enjoy feeding birds as well. I also adopted a cat recently- well, he adopted me. I’ve outfitted him with a break-away collar and a bell to warn the birds that a cat is around. I really enjoy being outside and seeing the new plant growth. When I’m inside I play the piano, and I am very interested in tracking genealogy.

Cats or dogs, and why?
I always thought I was a dog person. I love the affection. Cats are more independent. My husband had a cat before we were married, so I guess you could say I married the cat, too. I’ve learned to respect cats, and I respect them on their terms. I grew to understand and love our first cat. This new cat has also wiggled his way into my heart.

Any words of advice for new auditors?
That’s a tough one! But here goes… Be open to learning. Don’t be close-minded to ideas. Cultivate an attitude of willingness to hear different perspectives- it will help you realize the richness of what we do. As people we tend to focus a lot on ‘what we like.’ You may get a lot out of hearing stuff from people that you may not agree with. Be willing to learn from anybody, no matter who they are. Also, cut each other some slack. You’ll never know everything.

Final thoughts?
I’m really proud of our work here. I want to leave the world a better place, and here you have the opportunity to have a positive impact. That drives my work. When you get a whole group of smart people passionate about making a difference together, you can make change happen.

Auditors at Work Featured

Audits in the News: August 3rd

We here in the audits division are proud that the work we do makes a difference. Our work attracts the attention of the legislature, statewide news sources, and even local media outlets. Local media coverage of our audits is just another way we communicate with the people of Oregon about the work that we’re doing on their behalf to make government better. This is part of an ongoing series of posts rounding up recent instances in which the Oregon Audits Division makes a cameo in the local news.

The Oregonian/ – Ethics reforms after Kitzhaber: Which bills passed? Which bills didn’t?

Read the story here.

“More than a dozen measures dealing with ethics reforms emerged in the Legislature this year after an Oregon first: the resignation of Gov. John Kitzhaber amid a federal influence-peddling investigation. But just three bills passed, part of a measured approach championed by Gov. Kate Brown and top legislative Democrats … Senate Bill 9 orders the secretary of state to conduct an audit of state agencies’ handling of public records, and to report recommended changes.”

This audit is currently underway, with a scheduled release date of Nov. 20, 2015.

Medford Mail Tribune – Our View: Legislature left unfinished business on ethics

Read the story here.

“As lawsmakers look toward the short 2016 session, which also happens to be an election year, ethics reforms should figure prominently on their to-do list … Only three ethics bills passed … The third orders the secretary of state to conduct an audit of how state agencies handle public records and to recommend changes.

This audit is currently underway, with a scheduled release date of Nov. 20, 2015.

Portland Tribune – A victory for the Department of Administrative Services

Read the story here.
“Twelve new employees will work to fix problems identified in audits of state cyber security and IT operations, which the state has been slow to address. For example, the state has yet to fix some of the vulnerabilities auditors identified at the state data center in 2012. The data center is housed at the Department of Administrative Services, which is the central technology provider for state government and some municipal governments in Oregon.”

The audits division’s 2012 audit of the data center can be found here.
Another data center audit, conducted in 2010, can be found here.

Auditors at Work Audits in the News Featured

Auditing critical state information systems: Behind the Scenes

The Legislature just approved our request for two more IT auditors to increase our ability to examine the thousands of IT systems in the state. We now have three teams of IT auditors- a 50% increase over the previous two! We make the best of our limited resources by focusing our skilled professionals on the systems most critical to the finances and operations of state government.

We will soon start recruiting for more IT auditors so if you’re interested watch the Secretary of State website in August when applications open.

OurNeal IT Audit Manager, Neal Weatherspoon, was recently featured in the Summer 2015 newsletter of the Willamette Valley chapter of ISACA, an association of IT audit and security professionals.

Auditors at Work Featured IT Audit

A new approach in financial accountability

Portland just released its biennial financial condition report, which is a great tool that auditors in our region use. Auditors for Portland, Metro, the State of Oregon, and Multnomah County all regularly issue financial condition reports. These reports follow the same method of looking at key financial indicators over a ten year period to spot trends. These trends may show the recovery from a problem, or could be a warning that a problem is creeping up, year by year.

They use graphics in creative ways to simply tell these stories. The reports are a great way to inform the public about the sources of a government’s money, and where that money is spent.

Featured Financial Audit

GED Audit wins national award

Audit Manager Sheronne Blasi

Audit Manager Sheronne Blasi

We are very pleased to learn that our

Featured Noteworthy Performance Audit

Auditing and GIS: Visuals Help Engage Readers

In 2013 the Oregon Audits Division began expanding the use of Geographic Information System (GIS) software to improve the quality of our reports.  If you have not heard of GIS before, it is a tool used to create maps.

In our modern world, policy makers face a steady barrage of reports from multiple sources.  No one has enough time to read every page that passes by his or her desk.  The reality is that legislators and their staff skim reports.  As program evaluators, we should be aware of this fact. We must strive to capture the attention of our audience.  The use of striking visuals is a great method to do so.  A wonderful visual can engage the reader, signaling them to read the accompanying text of the report.

One important caveat: the quality of the text should match the quality of the visual.  You can have the best graphic in the world, but if the text is dense and lifeless, you will lose the reader.

Maps are a great example of a striking visual.  Everyone can easily relate to and interpret most maps.  Not only are maps easy for the public to understand, but legislators often focus on issues in the districts they serve.  By creating maps, you can effectively engage policy makers with the information they want.


Unemployment Rates, 2007-2009

Maps can tell stories

The adage goes “a picture is worth a thousand words”.  I am here to tell you that maps can also tell a persuasive story.  To the left is an example from our audit of Oregon’s Temporary Assistance for Needy Families’ program.

The program is very sensitive to economic downturns.  When millions of people lost their jobs during the great recession, they sought out public assistance in record numbers.  That should not come as a surprise since helping people during tough times is the raison d’être of these safety net programs.

By highlighting the unemployment rates, we reminded our readers of the economic conditions of the time.  This important context was essential for our readers to consider when evaluating our findings.

Working with the press

The press loves including maps and graphics with their stories. By sharing your work with reporters, you will get more attention from the press and better articles than you would otherwise. Our office recently sent the maps and other graphics we worked on to a reporter who covered our audit. The reporter loved the free materials and the audit report ended up getting a second front page article that prominently featured four of our graphics (see image below).

Remember to check in with your leadership team before working with the press. The last thing you want is that to be a surprise.

Here are some other things to consider:u.s.

(1) Deadlines matter – hand off your materials sooner rather than later.  Do your research and make sure you go directly to people who cover the topic;

(2) Give advance notice when possible. Reporters need time to brief editors, call sources, coordinate with their graphics departments, and then write and fact check their stories. Do not expect that releasing information at 4:55pm will result in an article in the morning’s paper.

(3) Make yourself available. Provide your contact number or arrange an interview. Make sure you coordinate with communication staff and leadership so that the right people are involved.

(4) Learn the lingo. When you tell a reporter that information is “on background” the reporter will not attribute the information to you. This is the origin of all the stories with lines like “Senior officials said…”. When you provide information “off the record”, the reporter can’t use the information unless they can confirm it with another source.

(5) Establish clear expectations about when the material can be published.  Providing an advanced copy of your report with an “embargo” can help ensure the information is published on the right day at the right time. Reporters respect embargos, but there are situations where other sources may leak the same material. In those circumstances, a reporter may publish their article early without your approval. This does not violate the embargo because the source of the information is someone else; and

(6) Build a working relationship with reporters between projects. It takes time to develop the trust needed for a successful relationship. Remember, it is a two-way street.

Maps can effectively highlight issues

One of the best benefits of maps is the ability to spot problems that would not be found without a geographic perspective.

Below is a classic map.  It is 1854 and London is facing a daunting epidemic of cholera.  At the time, no one knew how the disease spread.  Day by day, more Londoners succumbed to the disease. John Snow, a local physician, set out to identify the cause of the outbreak. John believed that once the cause was identified the city would be able to develop a plan of action to stop the outbreak.

John examined the relationship between the location of individuals who had died and the location of their water source.  It was clear from his analysis that the Broad Street Pump in the center of the map was the source of the outbreak. John convinced local officials to disable the pump based on this information and helped quell the epidemic.


London Cholera Epidemic, 1854

Fraud Detection

Recently GIS has found applications in fraud detection.  GIS supplements other fraud tools by looking at fraud from a spatial point of view.

Consider for example the Medicaid program.  The program has both clients and providers.  It may be suspicious if you find a provider who is serving multiple clients that live hours away. Examining the average diving distance per provider might uncover some fraud that would otherwise be missed.  This concept can extend to any program where you can analyze some geographic component.

lousianaWe can see a real world example of this type of fraud detection in the map to the left.  This analysis was conducted by the Louisiana Legislative Auditor’s office in 2013. They examined the distance clients traveled to use SNAP benefits, a.k.a. food stamps, and found some suspicious patterns.

Some stores had individuals traveling five hours across the state to purchase food. These patterns make a case that fraud is being committed. GIS not only makes your findings stronger, but it will also help law enforcement prosecute the offenders. The best-case scenario is when you provide enough evidence to law enforcement that the individuals plead out, thereby avoiding the cost of a trial.

Another example of fraud detection could include looking at ambulance billings relative to distance traveled.  If the company is billing for more time or mileage a GIS analysis could quickly detect this.

Other Audit Applications

San Diego has conducted some interesting audits of their motor pool and fuel purchases that utilized GIS. This 2011 audit looked at the number of take home vehicles from the motor pool relative to their duty location.  The audit found a number of employees had vehicles they were allowed to take home that resulted in extra maintenance and expense for the motor pool. In some cases, employees were driving over 50 miles per day in a government vehicle.  By reducing the number of take home vehicles the city was poised to save half a million dollars per year.

The audit also looked at fuel purchases.  Each time a car fills up at a retail gas station the city pays full price rather than the wholesale price for city operated facilities.  A number of individuals were filling up at retail locations even though city owned facilities were nearby.

 Exploring Demography

There is a world of free demographic information out there.  The U.S. Census Bureau as part of the decennial census and the American Community Survey collects most of the information.

Analyzing available demographics can add depth to your work.  Suppose you are tasked with assessing economic development projects.  These racial dot maps from the Cooper Center take basic census data and create engaging visuals that highlight the divided nature of many of our communities.  By overlaying economic development projects, you could test if opportunities are being enjoyed by everyone. This same principle holds for looking at educational attainment or income.  When evaluating a human services program it is very easy to pull up poverty demographics, and compare and contrast office locations relative to where clientele live.  Alternatively, it is also possible to check to see what populations are being reached and if certain areas are being underserved.


Dot maps emphasize density and grouping

Free tools

The U.S. Census Bureau is an excellent source to find data to map.  Recently, the Census has also put together an easy to use Census Data Mapper. Beyond data and interactive websites, there are a number of free analytic tools out there. A quick internet search for “open source GIS” will yield multiple products to try.  QGIS and GRASS GIS are two great options. Google also has some easy to implement interactive web maps if you feel comfortable working with basic HTML.

Pay the price

For some, the options mentioned above will meet your needs.  For others, you may have to open your checkbook to get the right tool.

The simplest to use, and one of the least expensive options, is SocialExplorer. It takes only a few seconds to create beautiful maps using the intuitive interface.  SocialExplorer even has a free version with limited functionality. One of the drawbacks is that you cannot import your own data and are limited to mostly U.S. Census surveys. It also lacks some customizable features found in other programs. If you want to start somewhere, SocialExplorer is a great place to start.

ArcGIS and Tableau are two power players in the data visualization market.  The features and analytical capabilities of their software are quite amazing.  The biggest drawback though is the cost of the licenses and the amount of staff time you need to invest to learn the software.

A more user friendly alternative is ArcGIS Online. This cloud based software is an add-in for Excel 2010 and 2013. ArcGIS Online is easier to use and much cheaper to purchase than the full desktop version, but you also lose some of the more advanced features. For example, some of the fraud detection tools such as calculating driving distance have not been implemented in the online version.

In either case, the benefits of using GIS can far outweigh the costs when you leverage the tool properly.

Good luck on your journey with GIS!


Ian Green, M.Econ, CGAP Senior Auditor with Oregon Secretary of State Audits Division

Data Wonk Featured