AI is increasingly being used to support components of the audit process, including evidence review, workpaper preparation, and first-pass analysis. This is a large efficiency increase.
However, simply doing tasks faster does not make an audit better. The Financial Reporting Council in the UK issued a formal statement to major audit firms (Deloitte, EY, KPMG, and PwC) about the incorporation of AI into the audit process without providing metrics for audit performance, oversight, and accountability. An audit firm proposing a time decrease without proof will not satisfy a regulator.
This is where human-in-the-loop AI comes in. AI should not replace audit professionals. AI can support large-volume data analysis and first-pass evaluation while the audit professional remains responsible for review and judgment. In this article, I will walk through how we go from identifying a model to designing a task to implementing human-in-the-loop AI.
What Is Human-in-the-Loop AI in an Audit Context?
Human-in-the-loop AI describes where the job involves humans maintaining some control and influence over the decisions made at particular stages within a process delegated to AI. This description fits each stage within an audit process.
The basic sequence runs like this: AI analyzes evidence or documentation → surfaces a result → auditor reviews that result → auditor validates or overrides → final conclusion is made by the engagement team.
HITL AI is useful in the context of an audit given that auditing is an exercise in forming judgments. For example, whether evidence provided is sufficient, whether a control is effective, or whether an exception represents a control deficiency; these are not necessarily matters of making a pattern match. They take professional skepticism, context, and discretion. AI can be used to make a preliminary evaluation, but auditors are still responsible for the final evaluation and conclusion.
What Can AI Do vs. What Should Auditors Do?
Here is a description of where AI can be used and where auditors must maintain responsibility:
AI Can Assist With | Auditors Should Retain |
Document and evidence review | Evaluating whether audit evidence is sufficient and appropriate |
Information extraction | Applying professional skepticism |
Control mapping to frameworks | Investigating exceptions and anomalies |
Exception identification | Validating AI-generated outputs |
First-pass control testing support | Reaching final audit conclusions |
Workpaper drafting | Approving or concluding on audit findings |
This distinction makes sense. AI is well suited to dealing with large volumes of documentation and surface issues. Professional context and judgment are required to evaluate the output and draw sufficient conclusions.
Human-in-the-Loop AI Use Cases in Auditing
Audit Evidence Review
AI reviews uploaded evidence files, screenshots, logs, exports, and policy documents. AI can assess whether each piece of evidence appears relevant to the control it supports and surface potential gaps for auditor review. The auditor then determines the sufficiency of the evidence and decides whether more documentation is required.
Control Mapping
AI extracts client policies and maps controls to the applicable frameworks and regulations (e.g., SOC 2, ISO 27001, CMMC, and HIPAA) and performs gap analysis. Auditors will review the mapping performed by AI to ensure its accuracy and completeness.
Control Testing
AI can support first-pass control testing by analyzing defined populations or samples against established testing criteria and surfacing potential exceptions and discrepancies for auditor review. Auditors then evaluate the exceptions, consider the circumstances and supporting evidence, and determine how the results should be documented.
Risk Assessment
AI can support risk assessment by identifying documentation gaps, overlapping or inconsistent controls, and areas where supporting evidence appears incomplete. Auditors can use these outputs as a starting point but remain responsible for evaluating the risks and reaching the final assessment.
ITGC Testing
In the case of IT General Controls, AI reviews access logs, change management records, and backup control documentation along with the control activities. Auditors determine whether the available evidence supports a conclusion about operating effectiveness.
Workpaper Preparation and Reporting
AI uses client documentation to generate first-pass workpapers and report sections. Auditors review draft workpapers and reports and determine if the supporting documentation is adequate.
In each case, the auditor does not review AI work as an afterthought. Review is built into the workflow.
Benefits of Human-in-the-Loop AI for Audit Teams
Well-designed human-in-the-loop AI lessens the burden of time-intensive analysis for audit teams and enables the auditor to provide review, judgment, and conclusions on the analysis.
Faster evidence review: AI can review audit evidence at a rapid pace, decreasing the amount of time auditors spend locating documents and reviewing the evidence manually.
Reduced administrative work: First-pass drafting, control mapping, and evidence sorting can be supported by AI, allowing auditors to focus on higher-value review and judgment.
More consistent analysis: When the flow of work and evaluation criteria are constant, AI is able to provide a more consistent first-pass analysis, thus reducing variability across engagements and teams when the underlying criteria and workflow are standardized.
Broader evidence coverage: AI can support analysis of larger segments or, where appropriate, entire populations within the defined audit scope and available evidence.
Faster workpaper preparation: With AI as the first draft author, auditors spend less time drafting and more time reviewing the workpapers.
More time for judgment: When AI is responsible for the repetitive analytical work, auditors are able to spend more time on the exceptions and conclusions.
How Does Human-in-the-Loop AI Differ from Fully Automated Auditing
Human-in-the-Loop AI | Fully Automated Auditing | |
Human involvement | Active review at defined checkpoints | Minimal or none |
Decision-making | Auditor validates and approves | AI performs the decision or workflow with limited human intervention |
Oversight | Built into the workflow | Passive or absent |
Risk of error | Provides opportunities to detect and correct AI errors before final decisions | Fewer opportunities for human intervention before outputs are used |
Judgment | Auditor applies professional judgment to relevant outputs | Greater reliance on predefined rules or AI-generated decisions |
Fully automated auditing, where conclusions generated by AI are acted upon without audit review, results in the audit accountability gap regulators are concerned about. HITL is not intended to slow down AI but to ensure audit results are justified.
The level of auditor oversight is proportional to the risk. Therefore, for tasks that have a low risk and high volume (such as evidence categorization and initial mapping), a lighter touch of oversight may suffice. For high-risk conclusions, such as findings, exceptions, and final reports, the level of oversight needs to be increased.
How to Implement Human-in-the-Loop AI in an Audit Workflow
Human-in-the-loop AI can bring value to your audit process without requiring a complete redesign. You should focus on where AI starts and where the review by a human auditor begins.
Identify suitable audit tasks: Start with high-volume, repetitive tasks. Think evidence organization, first-pass control testing support, or workpaper drafting. These are some tasks where auditors are likely to benefit from the use of AI.
Define where human review is required: Rather than leaving the scope of review to be determined by ad hoc decisions, build the review steps within your methodology.
Establish escalation rules: Rule-based escalation should be built into the process to avoid review backlogs for tasks with a high risk of unreliable or missing evidence or conflicting AI output.
Auditors must validate AI output: Make the auditor review a defined step in the methodology rather than an informal expectation.
Capture override scenarios: If an auditor rejects an AI output and creates their own, this must be captured for the purposes of the audit trail.
Monitor AI performance over time: Repeated AI errors, missed issues, exception patterns, confidence scores, and the usefulness of AI outputs should be monitored and documented over time. This analysis should be used to shape future audit processes using AI.
Address data governance and security: AI tools are used by audit teams with client files. Define how client data is handled, stored, accessed, and secured. Firms should understand whether inputs are retained, how they are used by the AI system, and what controls apply to client confidentiality.
What Should a Good HITL Audit System Have?
Not all AI systems consider the needs of auditors. Here are some factors worth considering during the evaluation phase.
Source-linked outputs: AI should not provide arbitrary outputs for users. All outputs should be linked to an original source, document, or evidence that supports the output.
Evidence traceability: The system should enable users to trace back the conclusion of a control through the evidence without the need for rebuilt evidence.
Human approval checkpoints: The system should require auditor approval at defined stages of the workflow rather than relying solely on discretionary review.
Ability to review or override: Auditors should be able to review AI output, document their reasoning, and record their disagreement.
Audit trails and version history: Workpapers and conclusions should be reviewed, time-stamped, and attached to a specific reviewer.
Evidence sufficiency checks: The system should support evidence sufficiency checks and notify the auditor when the evidence appears incomplete or does not fulfill the necessary control requirements.
Secure access and data controls: Client data that flows through the audit system needs to be protected by enterprise-level security. This is a deal-breaker.
How Roz Supports Human-in-the-Loop AI for Auditors
Roz is an AI-native audit fieldwork platform built for auditors and advisory firms performing control-based engagements across frameworks such as SOC 2, ISO 27001, HITRUST, HIPAA, and CMMC. Its human-in-the-loop approach supports AI-assisted analysis while keeping auditor review and professional judgment at the center of the workflow.
Here's how that works in practice:
Evidence review and sufficiency checks: Roz helps review evidence, relate documentation to controls, and surface potential gaps for auditor review.
Control extraction and mapping: Roz can extract controls from client documentation and support mapping them to relevant requirements, with auditors validating the results.
AI-powered control testing: Roz can run defined or suggested attribute checks against evidence and sample sets, returning testing results and reasoning for auditor review.
AI-assisted workpapers: Roz can export completed control activities as formatted workpapers, with supporting evidence, annotations, and audit trails.
AI-assisted search: Auditors can query engagement documentation to find relevant controls, policies, standards, or evidence without manually searching across files.
Human review throughout: Auditors remain responsible for evaluating evidence, investigating potential exceptions, exercising professional judgment, and reaching final conclusions.
Roz helps streamline repetitive fieldwork and first-pass control testing while keeping audit judgment and accountability with the engagement team.
Conclusion
Human-in-the-loop AI is a simple concept. AI can provide insights and results from quick analysis of large, complex data sets. However, AI does not replace the auditor's professional judgment or responsibility for engagement conclusions. AI can be particularly useful for supporting structured analysis related to areas such as materiality, evidence review, and control documentation, while the auditor retains responsibility for the resulting judgments.
Firms that successfully use AI as a tool rather than a replacement will be the first to realize its benefits. Accountability is a large piece of the process; thus, firms should consider building the review of the process into the workflow, articulate AI-assisted decisions, and remain responsible for all conclusions that the firm renders.
Speed matters. So does being able to stand behind the work.
Frequently Asked Questions
Why is human oversight important when using AI in audits?
AI can analyze large volumes of data quickly, but the auditor is still required to exercise professional judgment and adhere to applicable professional standards. As such, the auditor must review AI-generated output, draw conclusions, and maintain documentation, not as a requirement of an oversight framework, but because the auditor is ultimately accountable for every conclusion derived from the audit, not the technology.
What audit tasks can AI assist with?
AI can aid in document review, evidence mapping, control extraction, gap identification, preliminary control evaluations, and drafting worksheets. The judgment of the auditor is needed for determining the sufficiency of evidence, exploring exceptions, and arriving at conclusions. AI can process and organize large amounts of engagement information that can support those decisions, but responsibility for the decisions remains with the auditor.
How can audit firms implement human-in-the-loop AI?
Identify high-volume, repetitive tasks, and then set auditor checkpoints at which the auditor reviews the output of the AI. Create a plan for the next steps for "flagged" results. Document the AI-assisted decisions and auditor overrides. Select a tool that shows an output linked to its source and an audit trail, so conclusions can be traced back to the supporting evidence and documentation.
