Jon-Paul Cacioli

A cross-domain behavioural assay of monitoring-control coupling in LLMs, grounded in the Nelson and Narens (1990) metacognitive framework. The battery comprises 524 items across six cognitive domains, each grounded in an established experimental paradigm. After every forced-choice response, dual probes adapted from Koriat and Goldsmith (1996) ask the model to KEEP or WITHDRAW its answer and to BET or decline. Applied to 20 frontier LLMs (10,480 evaluations), the battery discriminates three profiles consistent with the Nelson-Narens architecture: blanket confidence, blanket withdrawal, and selective sensitivity. Accuracy rank and metacognitive sensitivity rank are largely inverted. Scaling on metacognitive calibration is architecture-dependent. Monotonically decreasing (Qwen), monotonically increasing (GPT-5.4), or flat (Gemma). No universal scaling law.

The derivation paper for the validity screening framework. Six validity indices cross-mapped from the PAI and MMPI-3 (L, K, F, Fp, RBS, TRIN), calibrated against the 20-model MMB derivation sample with family-level comparison groups. A tiered classification system, validated against eight synthetic response policies, identifies four models as construct-level invalid. Valid-profile models produce item-sensitive confidence (mean r = .18). Invalid-profile models do not (mean r = -.20, d = 2.17). Chain-of-thought training produces two opposite response distortions. Two latent dimensions, corresponding to the under-reporting and over-reporting blocs of clinical assessment, account for 94.6% of index variance.

The protocol paper. Extracts from the derivation study a portable, benchmark-agnostic screening procedure: three core indices (L, Fp, RBS) computed from a single correctness-by-confidence contingency table, plus a structural indicator (TRIN) and an item-sensitivity statistic. Three-tier classification (Invalid, Indeterminate, Valid) with subsampling analysis showing stable classification at 100-150 items. Cross-benchmark validation on external data (Yang et al., 2024) and an MMLU/verbalized-confidence variant confirms the protocol transfers. Validity is a property of the model-probe-task interaction, not an intrinsic property of the model.

The criterion-validation paper. Tests whether the validity tiers predict deployment-relevant selective prediction behaviour. Valid models achieve mean Type-2 AUROC = .624 [.604, .647]. Invalid models mean AUROC = .357 [.031, .522]. Tiers order monotonically (d = 2.81, p = .002). Split-half cross-validation yields median d = 1.77 across 1,000 splits with P(d > 0) = 1.0. The three-tier classification accounts for 47.0% of variance in AUROC. DeepSeek-R1, classified Invalid by massive inversion, drops from 85.3% accuracy at full coverage to 11.3% at 10% coverage. Exactly the failure pattern selective prediction is supposed to avoid.

The first applied use of the validity screening protocol on a published benchmark. Seven instruction-tuned open-weight models (3-9B parameters, four families) tested on 524 TriviaQA items under numeric (0-100) and categorical (10-class) confidence elicitation. All seven instruct models classified Invalid on numeric confidence, with mean ceiling rate 91.7%. Two models produced zero low-confidence responses across 500+ trials. Categorical elicitation did not rescue validity. It disrupted task performance, with most models dropping from 59-76% accuracy under numeric to 0.2-4.2% under categorical. Token-level logprobability did not usefully predict verbalised confidence (mean cross-validated R² < 0.01). Within the reasoning-distilled model, reasoning-trace length showed a strong negative partial correlation with confidence (ρ = -.36), consistent with the Reasoning Contamination Effect. Internal item-level information exists (item difficulty correlates with mean confidence at ρ = .50) but does not survive the verbalised readout under minimal elicitation.

Tests whether confidence-conditioned supervised fine-tuning (CSFT) with self-consistency-derived targets can close the gap between internal information and verbal readout in small instruct models. The pre-registered Phase 0 protocol on Gemma 3 4B-it, with a modal filter restricting training to items with correct modal answers, produced a negative result: AUROC2 dropped from 0.554 to 0.509 due to label-entropy collapse in the training targets. An exploratory post-hoc rescue removed the filter, training on all 2,000 calibration items. This produced a binary verbal correctness discriminator with AUROC2 = 0.774 on held-out TriviaQA, compressing a 10-sample self-consistency signal (AUROC2 = 0.999) into a single-pass readout exceeding logit entropy (0.701). The shuffled-target control showed no improvement (0.501). On MMLU, accuracy improved from 54.2% to 77.4% with the shuffled model at baseline (56.1%), supporting a target-dependent interpretation. Two design lessons: confidence training requires label entropy, and correct targets regularise output format. The result is exploratory, binary rather than continuously calibrated, and observed at a single scale.

Scales the Phase 0 CSFT protocol to eight models (7B-72B, four families) across TriviaQA, GSM8K, and ARC-Challenge. PT-CSFT uses a linear probe on the model's own hidden states to generate continuous confidence targets for LoRA fine-tuning. At 7-32B, PT-CSFT recovers 91-115% of probe discrimination in verbal confidence. At 70B, a two-stage curriculum closes 66% of the gap (AUROC2 = 0.797), producing the first VRS-Valid confidence signal at 70B. Controlled activation patching at the confidence-token position supports the interpretation that verbal confidence failure is a routing problem: the intervention is position-specific, bidirectional, selective (83% of answers unchanged), and follows a near-monotonic layer-depth gradient (Spearman rho = 0.976). The logit readout universally rescues where text confidence fails. Multi-seed replication across three families confirms stability.

Predictive coding networks (PCNs) are an architecturally attractive substrate for structural metacognitive probing. An approximate decomposition shows that under standard discriminative PC, the K-way energy margin reduces to a monotone function of the log-softmax margin plus a residual that is not trained to correlate with correctness. The decomposition predicts the probe should track softmax from below. Confirmed empirically across six conditions (deterministic PC, BP+decoder control, matched-budget PC vs BP, Langevin temperature sweep, trajectory-integrated MCPC). In every condition the structural probe sat below softmax. MCPC versus final-state training differed by 6x10⁻⁴ AUROC2 at deterministic evaluation.

A pre-registered follow-up that isolates which assumptions of the K-way decomposition are empirically load-bearing. Three conditions across 10 seeds on CIFAR-10: standard discriminative PC with cross-entropy (baseline), standard PC with MSE replacing CE, and bidirectional PC (Oliviers et al., 2025). The original negative result replicates on CE-trained PC (Δ = -0.082, p < 10⁻⁶). Removing CE alone halves the probe-softmax gap (Δ = -0.037). Bidirectional PC flips the gap positive across all 10 seeds (Δ = +0.008), though a pre-registered manipulation check shows bPC does not produce materially greater latent movement at this scale. A post-hoc temperature scaling ablation decomposes the gap: approximately 66% is attributable to CE-induced logit-scale inflation, 34% to a scale-invariant ranking advantage of proper scoring rules. Confirms that the earlier negative result is correctly scoped to CE-trained discriminative PC rather than a general limitation of structural probing.

A pre-registered intervention study that set out to improve domain-specific metacognition through targeted SFT, and instead discovered that M-ratio profiles are format-dependent. Quantisation restructures domain-level metacognitive geometry completely (Spearman ρ = 0.00 across Q5_K_M and f16), while Type-2 AUROC rankings are perfectly stable (ρ = 1.00). The dissociation localises to the M-ratio normalisation: quantisation shifts d′ non-proportionally across domains, which propagates through division into an apparent profile restructuring absent in the raw discrimination signal. Diagnostic systems relying on M-ratio profiles have an unexamined dependency on inference format. AUROC2-based diagnostics do not.

Standard calibration metrics (ECE, Brier score, AUROC) conflate how much a model knows (Type-1 sensitivity) with how well it knows what it knows (Type-2 metacognitive sensitivity). We introduce meta-d′ and M-ratio as an evaluation framework across four LLMs and 224,000 factual QA trials. Mistral achieves the highest Type-1 sensitivity but the lowest metacognitive efficiency. Gemma-2 shows near-optimal efficiency despite lower sensitivity. AUROC2 and M-ratio produce fully inverted model rankings. Temperature manipulation dissociates confidence policy from metacognitive capacity in instruction-tuned models, suggesting RLHF shifts criterion without improving metacognitive signal quality.

Standard LLM calibration metrics such as ECE conflate sensitivity (discriminating correct from incorrect) and bias (the tendency toward confident or cautious responding). This pre-registered study treats three LLMs as psychophysical observers performing factual discrimination across 168,000 trials, applying the full parametric SDT framework (unequal-variance model fitting, criterion estimation, z-ROC analysis) for the first time to language models. Temperature does not function as a clean criterion shift. It simultaneously increases sensitivity and shifts criterion. Models occupying distinct positions in sensitivity-bias space were indistinguishable by calibration metrics alone.

The largest standardised profile dataset for LLM metacognitive monitoring to date: 33 frontier LLMs from eight families, 1,500 MMLU items across six cognitive domains, 47,151 total observations. Every model with above-chance aggregate monitoring showed non-trivial domain-level variation. Applied/Professional was reliably easiest to monitor (mean AUROC = .742, top-2 in 21/33 models); Formal Reasoning and Natural Science were reliably hardest (.658 and .652). Three models classified Invalid under binary KEEP/WITHDRAW probes produced normal profiles under verbalised confidence, confirming probe-format specificity. The Anthropic family showed the highest and narrowest metacognitive quality range (.708–.806 across eight models). Gemma 4 31B showed a +.202 AUROC leap over Gemma 3 27B — the largest single-generation improvement in the dataset.

Pre-registered study (Stage 1 of 2) testing whether thinking mode improves the discriminative power of token-probability-derived confidence signals. Three 7-8B reasoning models answered 2,000 TriviaQA items in thinking and non-thinking mode. Thinking mode improved accuracy but significantly reduced NLP-based AUROC2 in all three models (Qwen3-8B: delta = -0.231, 95% CI [-0.262, -0.198]). The mechanism is NLP compression: thinking-mode NLP variance dropped to 16% of non-thinking levels.

Tests whether symptom validity testing logic from clinical malingering detection can identify sandbagging through below-chance performance on forced-choice items. Pre-registered pilot at the 7-9B instruction-tuned scale: 3 models, 4 MMLU-Pro domains, 4 conditions, 500 items per cell, 24,000 total trials. The plausibility gate failed. Zero of 12 model-domain cells showed significant below-chance performance under sandbagging instruction. Exploratory analyses revealed three qualitatively distinct failure modes. Qwen-2.5-7B and Phi-3.5-mini largely ignored the sandbagging instruction (62-88% response identity with honest baseline). Llama-3-8B complied substantially but implemented underperformance as a positional heuristic, collapsing its response distribution onto middle-alphabet options (E at 31.8%, F at 26.1%) regardless of where the correct answer fell. An explicit anti-task instruction ("pick the least likely answer") drove two of three models below chance, with accuracy as low as 0.024. The capability for answer-aware avoidance exists but is not activated by "deliberately underperform." Positional-distribution shift may be a more effective behavioural signature than below-chance accuracy for detecting prompted underperformance at this scale.

Pre-registered follow-up (3 models, 2,000 MMLU-Pro items, 4 conditions, 24,000 primary trials) adding cyclic option-order randomisation as the critical control. The pre-registered item-level same-letter diagnostic was ambiguous (37.3%, below the 50% threshold), but pre-specified supporting analyses revealed a soft distributional attractor: the response-position distribution under sandbagging was highly stable under complete content rotation (Pearson r = 0.9994; Jensen-Shannon divergence = 0.027, compared to 0.386 between honest and sandbagging conditions). Accuracy spiked to 72.1% when the correct answer coincidentally occupied the preferred position E, and fell to 4.3% at position A. Response-position entropy is a promising black-box behavioural signature of this sandbagging mode at the 7-9B parameter scale.

Pre-registered study mapping the boundary between content-aware and content-blind adversarial compliance in two 8B instruction-tuned LLMs. A six-condition specificity gradient (29,600 trials) reveals three regimes rather than a monotonic transition: content-engaged compliance, shortcut with residual engagement, and extreme positional collapse. The only multi-step instruction (answer-aware avoidance) produced near-total concentration on a single response position (99.9% and 87.4%), matching the model's content-absent null-prompt default. Distributional screening and difficulty-accuracy correlation showed 50% concordance, indicating they capture partially independent dimensions of response validity. A two-stage validity screening architecture, adapted from clinical psychometric practice, provides richer evaluation information than either criterion alone.

Adapts the Reliable Change Index (Jacobson and Truax, 1991) from clinical psychology to item-level LLM version comparison on 2,000 MMLU-Pro items (K = 10 samples at T = 0.7, 80,000 trials). Two within-family pairs tested: Llama 3 to 3.1 (+1.6 pp) and Qwen 2.5 to 3 (+2.8 pp). Among analysable items, change was bidirectional with large effect sizes: 34% improved and 28% deteriorated for Llama; 47% improved and 39% deteriorated for Qwen (median |Δp| = 0.50 and 0.90). Churn was asymmetric by difficulty: low-accuracy items improved, high-accuracy items deteriorated. Domain-level decomposition revealed family-specific reversals: Llama lost physics while Qwen lost law. Greedy single-shot evaluation missed 42% of reliably changed items and falsely flagged 25% of unchanged items. K = 3 samples suffice for reliability above .90.

Applies formal tools of psychophysics to resolve conflicting accounts of how transformers represent magnitude. Across four converging paradigms (RSA, behavioural discrimination, precision gradients, causal intervention) in three 7-9B instruction-tuned models, representational geometry is consistently log-compressive, consistent with Weber's Law, but dissociated from behaviour. Early layers are functionally implicated in magnitude processing while later layers, where geometry is strongest, are not causally engaged.

Scalar variability, noise proportional to magnitude producing a constant coefficient of variation, is a hallmark of biological magnitude systems. This pre-registered companion to the Weber study tests whether transformers share this property. They show the opposite. Representational variability decreases with magnitude (α ≈ -0.19; 0/48 model-layer cells with α > 0). Corpus frequency strongly predicts per-magnitude variability (ρ = .84), consistent with a distributional account. Distributional learning produces the geometry of biological magnitude representation but not the noise.

Categorical perception, enhanced discriminability at category boundaries, is among the most studied phenomena in perceptual psychology. Using RSA across six models from five architecture families, a CP-additive model fits the representational geometry better than a purely continuous model at 100% of primary layers in every model tested. The effect is specific to structurally defined boundaries (digit-count transitions at 10 and 100), absent at non-boundary controls, and absent in the temperature domain. Two distinct signatures emerge. Classic CP (Gemma, Qwen), where models both categorise explicitly and show geometric warping. Structural CP (Llama, Mistral, Phi), where geometry warps but models cannot report the distinction.

Pre-registered evaluation of a neurosymbolic architecture combining BabyBERTa with a differentiable PCFG auxiliary loss derived from Minimalist Grammar. Structural priors produce phenomenon-specific effects. Filler-gap dependencies improve by 9-13 pp (d = 2.41-2.82) while locally-cued phenomena are damaged regardless of whether the grammar is real or random.

Pre-registered test of whether autoregressive transformers trained on synthetic child-scale corpora induce overhypotheses (second-order generalisations about category structure). Models achieve perfect first-order exemplar retrieval but remain at chance on second-order generalisation across 120 runs and three model sizes. Feature-swap diagnostics reveal template matching rather than structured abstraction as the operative mechanism.

First systematic evaluation of mutual exclusivity in text-only language models trained on child-directed speech. Across 45 GPT-2-architecture models spanning a 12x parameter range, all models show robust repetition priming rather than ME. Priming attenuates with better language modelling but never crosses zero. A context-dependence diagnostic rules out embedding artefacts as an alternative account.

A research programme formalising the analogy between binocular rivalry and LLM ambiguity resolution. The visual system resolves competing representations through mutual inhibition, adaptation, and goal-conditioned top-down modulation. The same computational problem faced by language models during ambiguous generation. Paper 1 validates the GC-LCA model against rivalry data. Paper 2 tests whether LLM internals exhibit metastable dynamics consistent with rivalry. Paper 3 applies the framework as an ambiguity detection mechanism.