Introduction
Buying fraud detection software is a slow decision, and it should be. The system will decide which of your customers’ claims get a second look, and it will make that decision for every claim that comes in. A wrong choice floods your investigators with noise, or misses the cases that matter, or produces decisions you cannot defend in a complaint. None of that shows up in a demonstration.
The questions below are grouped by the worry behind each one. They are written from the vendor side, which is unusual, so a note on why. The category has a problem with numbers. Detection rates and savings figures are quoted with no methodology behind them, and a buyer has no way to compare one vendor’s 90 per cent with another’s. The questions below are the ones that separate a figure with a basis from a figure without one. Any vendor worth talking to will welcome them.
First, be clear what you are buying
A detection system does not decide that a claim is fraudulent. It scores claims and applications against rules and models, and creates a referral when a score passes a threshold or a rule fires. People investigate the referral and people decide the outcome. If a vendor describes the product as catching or eliminating fraud, ask them to describe what actually happens to a claim, step by step, from scoring to decision.
Detection performance
Will this find the cases that matter?
- For any detection rate, savings figure or percentage you quote, what is the definition, the date, the line of business and the size of the portfolio it came from?
- What is the hit rate, meaning the share of referrals confirmed as fraud, and what is the false positive rate, on a comparable book?
- Which fraud types does the model perform well on, and which does it not? Organised network fraud and opportunistic exaggeration are different problems, and a vendor who claims equal strength on both should be asked to show it.
- Is the model trained on our data, on pooled data, or on neither?
A number without a definition and a scope is not a number. It is a claim.
False positives and tuning
Will this flood my investigators?
This is the question every director asks first, and rightly. The volume of referrals is set by thresholds, and thresholds are set by product, channel and claim type. Ask:
- Are claims scored in segments, so that a low-value motor claim and a high-value property claim are not judged by the same threshold?
- Who sets and changes thresholds, us or you, and how quickly?
- Can we simulate a change to the rules against historical claims before we switch it on?
- How do confirmed and dismissed outcomes feed back into the models, and how often?
- What does the referral volume look like in month one, and what should it look like after tuning?
Explainability and defensibility
Can I defend this in an audit or a complaint?
Every referral will eventually be questioned by an investigator, a customer, the Australian Financial Complaints Authority (AFCA) or a regulator. Ask:
- Does every referral carry a reason code, the evidence behind it, and the linked entities, in a form a person can read?
- Is there a complete record of actions taken on the referral, who took them and when?
- Can we show that a rule or model does not discriminate? The General Insurance Code of Practice requires insurers to review their non-genuine claims indicators at least once a year to make sure they remain relevant, appropriate and do not discriminate.¹ Ask how the vendor supports that review.
- When the model changes, is the old version and its behaviour retained, so that a decision made last year can still be explained?
Data and integration
Will this break my systems?
- What data do you need to start, at minimum, and what improves results later?
- How does the platform integrate with our claims and policy systems: two-way status updates, notes, documents and case outcomes, or a one-way feed?
- Where does our data sit, who can access it, and how is access controlled by role?
- What happens to our data and our models if we leave?
Governance and the regulators
Will this pass scrutiny?
Australian regulators have moved from principle to expectation on AI. In October 2024 ASIC reviewed 624 AI use cases across 23 licensees and warned of a governance gap, noting that nearly half of licensees had no policies covering consumer fairness or bias, and that about a third of use cases relied on third-party models.² In April 2026 APRA wrote to all regulated entities calling for a step-change, and set out expectations that boards maintain sufficient understanding and literacy to challenge AI risk, and that entities map and maintain visibility over the full AI supply chain, including third-party and fourth-party dependencies.³
For a buyer that translates into questions the vendor must be able to answer, because you will be asked them:
- What documentation exists for each model: purpose, training data, known limitations, performance by segment?
- Which components are yours, which are third-party, and which of those depend on a further supplier?
- What is your change control process, and how are we told before a model or rule set changes in production?
- What assurance can you provide, and what can our own auditors test?
The capability test
Most purchases in this category involve a test on the insurer’s own data before contract. It is the most important step, and it is often run badly. Some suggestions:
- Define what success means before the test starts: which fraud types, which segments, what hit rate and referral volume would count as a pass.
- Use a sample your own team has already worked, so confirmed outcomes exist to compare against.
- Measure by segment, not in aggregate. An impressive overall figure can hide a model that only works on one product.
- Count the false positives your investigators would have had to clear, not only the frauds found.
- Ask the vendor to explain the referrals it got wrong. The quality of that conversation tells you more than the score.
The people
Ask who you will actually deal with after the contract is signed, and how you reach them. In a category where tuning never really finishes, the answer matters more than the feature list.
What a good answer sounds like
A good answer acknowledges a trade-off, explains the mechanism, and leaves the decision with you. “Rules give you control and transparency; models find patterns rules cannot; most insurers get the best results by combining both, and here is how we would set the thresholds for your book” is a good answer. “Our AI eliminates fraud with 99 per cent accuracy” is not, and it is worth remembering that a vendor who over-claims becomes your regulatory and reputational risk the day their number is questioned.
Related Topics
Governance of AI models in insurance
XAI: Explainable AI in insurance
Threshold tuning and operational alignment
Sources & further reading
¹ General Insurance Code of Practice, October 2023 update, paragraph 195
² ASIC, Report 798: Beware the gap, governance arrangements in the face of AI innovation, and media release 24-238MR, 29 October 2024
³ APRA, Letter to Industry on Artificial Intelligence, 30 April 2026
⁴ kbs Intelligence, Detect and Investigate pages and FAQs, kbsintelligence.io
