Catch the AI Lying: an audited pass/fail harness for one small free model

THE GOLD STANDARD items : 10 self-contained clinical questions file : data/gold_qa.csv ACCURACY ON THE SAME 10 ITEMS, THREE RULERS exact match : 9/10 = 90% contains gold : 8/10 = 80% judge, pass A : 7/10 = 70% judge, pass B : 6/10 = 60% IS THE JUDGE ITSELF RELIABLE? judge vs itself : agreement 60%, kappa 0.20 exact vs judge A : agreement 80%, kappa 0.4444 judge failures : pass A 2/10, pass B 4/10 API cost reported: $0 IMPORTANT PROVENANCE This run includes API errors (including rate limits). UNPARSEABLE means no valid judgment; it is not a clinical FAIL. Saved notebook output was used to reconstruct these tables. Full original answer text and raw judge reasons were not recoverable. This is not a clinical validation.

id type gold answer exact_verdict judge_a score_a judge_b score_b judge_repeated scorers_agree icd10_t2dm code_lookup E11.9 E11.9 PASS UNPARSEABLE PASS 10.0 False False icd10_ami code_lookup I21.9 I21.9 PASS PASS 10.0 UNPARSEABLE False True snomed_t2dm code_lookup 44054006 44054006 PASS PASS 10.0 PASS 10.0 True True acetaminophen_max_g numeric 4 4 PASS PASS 10.0 PASS 10.0 True True a1c_8_eag_mgdl numeric 183 183 PASS PASS 10.0 PASS 10.0 True True hypo_level1_mgdl numeric 70 FAIL FAIL 0.0 UNPARSEABLE False True hipaa_safe_harbor numeric 18 18 PASS PASS 10.0 PASS 10.0 True True warfarin_reversal_agent short_phrase 4-factor prothrombin complex concentrate Prothrombin complex concentrate PASS UNPARSEABLE UNPARSEABLE True False sjw_warfarin_yesno yes_no NO NO PASS PASS 10.0 PASS 10.0 True True atorva80_intensity one_word high high-intensity PASS PASS 10.0 UNPARSEABLE False True

exact_verdict PASS FAIL UNPARSEABLE FAIL 0 1 0 PASS 7 0 2