Design payment-fraud decisions and evaluation when outcomes arrive late and the current policy determines which payments can produce labels.
Problem
Context. A hypothetical payment policy allows some attempts and declines others; allowed outcomes mature only after a declared observation window. The cohort below contains mature outcomes, pending outcomes, and declined attempts.
Task. Define labels, model and policy inputs, operating actions, and evaluation for the cohort. Explain what can be estimated now and what evidence is needed to compare a changed decision policy.
Constraints. Missing, pending, and declined outcomes are not confirmed legitimate transactions. Separate fraud risk from action costs, preserve decision-time feature availability, and account for selection and label delay.
Design payment-risk decisions for this hypothetical cohort of 1,000 attempts under policy P1. P1 allows 900 and declines 100. Of the allowed payments, 800 have completed the declared observation window; eight have confirmed fraud reports. The other 100 allowed payments remain pending.
Define training labels, operating actions and evaluation without converting missing outcomes into legitimate transactions.
Picture: Follow the Action Before Naming the Label
| Population | Count | What is observed |
|---|---|---|
| Mature allowed | 800 | Eight confirmed reports; 792 without reports in the window |
| Pending allowed | 100 | Observation window incomplete; not negative |
| Declined | 100 | No realized outcome from allowing that payment |
The confirmed-report fraction is among mature allowed payments. It is not population fraud prevalence. The 792 no-report records are negative only under the declared, noisy reporting definition. The ratio describes currently confirmed reports per attempt, not complete fraud coverage.
Keep action, maturity, outcome source and uncertainty attached to every label. Independent investigation can provide evidence about declined attempts. It cannot observe their exact losses under an action that never occurred.
One cohort, three observation states
Declined and pending payments do not supply observed negative outcomes
Derivation
Can a declined payment be labeled legitimate from an unobserved repayment?
Define the Decision Path
Start with approved rules plus a measured model baseline, such as gradient-boosted trees on point-in-time features. Use transaction amount/time, account aggregates, recent velocity and permitted device/network relationships. Every feature must have been available before its decision deadline.
A challenge is intermediate, not necessarily a declined payment. Record passed, failed or abandoned challenges and their final allow/decline actions. Challenge failure alone does not prove fraud.
Decision
transaction_id, decision_id, event_time, decision_time
policy_version, action, challenge_state, feature_version
availability_cutoff, score, calibration_population
latency, fallback_reason
Outcome
transaction_id, event_time, available_time, observation_window
maturity, report_status, label_source, uncertainty
Audit
decision_id, sampling_stratum, selection_probability
adjudication_task, evidence_refs, reviewer, available_timeChoose Actions From Consequences
Suppose a separately validated probability and the following simplified loss model apply.
| Action | Expected loss in currency units | Required evidence |
|---|---|---|
| Allow | Relevant probability of loss under allowance | |
| Decline | Cost of rejecting a legitimate payment | |
| Challenge | Friction/fee plus residual loss | Completion, abandonment, fraud and capacity estimates |
With only allow/decline, gives losses 2 versus 2.94; allow has lower modeled loss. At , losses are 10 versus 2.7; decline has lower modeled loss. The cohort’s observed report fraction is not automatically a valid for this decision.
The risk owner approves costs, constraints and capacity; ML supplies justified estimates and uncertainty. Calibration must name its target, population and observation process. PR-AUC describes ranking, not the action policy.
Change 1: The New Policy Declines More Payments
New task. P2 declines more high-risk attempts than P1.
Picture and change. Recompute action coverage and compare equally mature cohorts. Use appropriately sampled audits and validated causal designs where feasible before claiming better population detection. Preserve propensity/support assumptions; audits do not supply every missing counterfactual.
Check. Blocking nearly everyone can reduce allowed losses while destroying legitimate approvals. Track approval, challenge friction, loss, review capacity and customer impact together.
Use when. The policy changes which outcomes can become labels.
Change 2: Shared-Device Activity Rises
New task. Related accounts begin sharing more devices.
Picture and change. Verify event timing, velocity, graph construction and source integrity. Inspect legitimate shared-device cohorts, privacy constraints and segment effects before changing actions. Use bounded approved rules or investigation while testing the suspected pattern.
Check. Shared infrastructure is evidence, not proof of fraud.
Use when. Adaptive or correlated behavior changes feature meaning.
Failure: Missing Becomes Legitimate
Combining declined, pending and mature no-report transactions as negatives hides different observation processes. Semi-supervised reconstruction can learn structure; it does not identify missing allowed-payment outcomes by itself.
Transfer Check
Does high PR-AUC identify the correct blocking threshold?
Reasoned Answer
No. Use relevant calibrated evidence, consequence/friction costs, policy constraints and operating capacity. Check the resulting policy, not only its score ranking.
Audit the Observation Process
Outcome delay, reporting noise, selection and adaptive behavior require separate diagnostics. Fraud prevalence and available latency vary by product; do not substitute generic industry rates.
Sample allowed and declined traffic using recorded strata and selection probabilities. Define the adjudication target, independent evidence, reviewer uncertainty and any weighting assumptions. Active-learning cases are not automatically representative samples. An audit can estimate its defined target under justified assumptions, not every unobserved monetary loss.
Use time-based splits, decision-time feature snapshots and mature outcome windows. Prevent future reports or later graph labels from leaking into earlier features. Report precision/recall, calibration, losses and uncertainty for the population each dataset supports.
Operate a Changing Policy
Shadow scoring tests service behavior without changing actions; it does not reveal alternative payment outcomes. Use approved staged experiments only where risk, customer protection and identification assumptions permit them.
Monitor feature/score changes, maturity, action mix, segment outcomes and service deadlines. A distribution shift is a diagnostic signal, not proof that retraining solves the incident. Choose update cadence from label readiness and validated operating needs.
Name risk-policy, fraud-operations, model and service owners. Define temporary-rule expiry, incident escalation, fallback actions and model/policy rollback separately. Revalidate after changing features, calibration, thresholds or label definitions. Platt scaling or isotonic regression still requires suitable data and independent evaluation.
Explanation, retention and access requirements depend on jurisdiction, product and current rules. Feature attributions can support investigation; they are not automatically a compliant explanation.
Related: delayed and selective labels, decision thresholds and abstention, class imbalance, and calibration.