Jon-Paul Cacioli

A cross-domain behavioural assay of monitoring-control coupling in LLMs, grounded in the Nelson and Narens (1990) metacognitive framework. The battery comprises 524 items across six cognitive domains, each grounded in an established experimental paradigm. After every forced-choice response, dual probes adapted from Koriat and Goldsmith (1996) ask the model to KEEP or WITHDRAW its answer and to BET or decline. Applied to 20 frontier LLMs (10,480 evaluations), the battery discriminates three profiles consistent with the Nelson-Narens architecture: blanket confidence, blanket withdrawal, and selective sensitivity. Accuracy rank and metacognitive sensitivity rank are largely inverted. Scaling on metacognitive calibration is architecture-dependent. Monotonically decreasing (Qwen), monotonically increasing (GPT-5.4), or flat (Gemma). No universal scaling law.

The derivation paper for the validity screening framework. Six validity indices cross-mapped from the PAI and MMPI-3 (L, K, F, Fp, RBS, TRIN), calibrated against the 20-model MMB derivation sample with family-level comparison groups. A tiered classification system, validated against eight synthetic response policies, identifies four models as construct-level invalid. Valid-profile models produce item-sensitive confidence (mean r = .18). Invalid-profile models do not (mean r = -.20, d = 2.17). Chain-of-thought training produces two opposite response distortions. Two latent dimensions, corresponding to the under-reporting and over-reporting blocs of clinical assessment, account for 94.6% of index variance.

The protocol paper. Extracts from the derivation study a portable, benchmark-agnostic screening procedure: three core indices (L, Fp, RBS) computed from a single correctness-by-confidence contingency table, plus a structural indicator (TRIN) and an item-sensitivity statistic. Three-tier classification (Invalid, Indeterminate, Valid) with subsampling analysis showing stable classification at 100-150 items. Cross-benchmark validation on external data (Yang et al., 2024) and an MMLU/verbalized-confidence variant confirms the protocol transfers. Validity is a property of the model-probe-task interaction, not an intrinsic property of the model.

The criterion-validation paper. Tests whether the validity tiers predict deployment-relevant selective prediction behaviour. Valid models achieve mean Type-2 AUROC = .624 [.604, .647]. Invalid models mean AUROC = .357 [.031, .522]. Tiers order monotonically (d = 2.81, p = .002). Split-half cross-validation yields median d = 1.77 across 1,000 splits with P(d > 0) = 1.0. The three-tier classification accounts for 47.0% of variance in AUROC. DeepSeek-R1, classified Invalid by massive inversion, drops from 85.3% accuracy at full coverage to 11.3% at 10% coverage. Exactly the failure pattern selective prediction is supposed to avoid.

The first applied use of the validity screening protocol on a published benchmark. Seven instruction-tuned open-weight models (3-9B parameters, four families) tested on 524 TriviaQA items under numeric (0-100) and categorical (10-class) confidence elicitation. All seven instruct models classified Invalid on numeric confidence, with mean ceiling rate 91.7%. Two models produced zero low-confidence responses across 500+ trials. Categorical elicitation did not rescue validity. It disrupted task performance, with most models dropping from 59-76% accuracy under numeric to 0.2-4.2% under categorical. Token-level logprobability did not usefully predict verbalised confidence (mean cross-validated R² < 0.01). Within the reasoning-distilled model, reasoning-trace length showed a strong negative partial correlation with confidence (ρ = -.36), consistent with the Reasoning Contamination Effect. Internal item-level information exists (item difficulty correlates with mean confidence at ρ = .50) but does not survive the verbalised readout under minimal elicitation.

Predictive coding networks (PCNs) are an architecturally attractive substrate for structural metacognitive probing. An approximate decomposition shows that under standard discriminative PC, the K-way energy margin reduces to a monotone function of the log-softmax margin plus a residual that is not trained to correlate with correctness. The decomposition predicts the probe should track softmax from below. Confirmed empirically across six conditions (deterministic PC, BP+decoder control, matched-budget PC vs BP, Langevin temperature sweep, trajectory-integrated MCPC). In every condition the structural probe sat below softmax. MCPC versus final-state training differed by 6x10⁻⁴ AUROC2 at deterministic evaluation.

A pre-registered follow-up that isolates which assumptions of the K-way decomposition are empirically load-bearing. Three conditions across 10 seeds on CIFAR-10: standard discriminative PC with cross-entropy (baseline), standard PC with MSE replacing CE, and bidirectional PC (Oliviers et al., 2025). The original negative result replicates on CE-trained PC (Δ = -0.082, p < 10⁻⁶). Removing CE alone halves the probe-softmax gap (Δ = -0.037). Bidirectional PC flips the gap positive across all 10 seeds (Δ = +0.008), though a pre-registered manipulation check shows bPC does not produce materially greater latent movement at this scale. A post-hoc temperature scaling ablation decomposes the gap: approximately 66% is attributable to CE-induced logit-scale inflation, 34% to a scale-invariant ranking advantage of proper scoring rules. Confirms that the earlier negative result is correctly scoped to CE-trained discriminative PC rather than a general limitation of structural probing.

A pre-registered intervention study that set out to improve domain-specific metacognition through targeted SFT, and instead discovered that M-ratio profiles are format-dependent. Quantisation restructures domain-level metacognitive geometry completely (Spearman ρ = 0.00 across Q5_K_M and f16), while Type-2 AUROC rankings are perfectly stable (ρ = 1.00). The dissociation localises to the M-ratio normalisation: quantisation shifts d′ non-proportionally across domains, which propagates through division into an apparent profile restructuring absent in the raw discrimination signal. Diagnostic systems relying on M-ratio profiles have an unexamined dependency on inference format. AUROC2-based diagnostics do not.

Standard calibration metrics (ECE, Brier score, AUROC) conflate how much a model knows (Type-1 sensitivity) with how well it knows what it knows (Type-2 metacognitive sensitivity). We introduce meta-d′ and M-ratio as an evaluation framework across four LLMs and 224,000 factual QA trials. Mistral achieves the highest Type-1 sensitivity but the lowest metacognitive efficiency. Gemma-2 shows near-optimal efficiency despite lower sensitivity. AUROC2 and M-ratio produce fully inverted model rankings. Temperature manipulation dissociates confidence policy from metacognitive capacity in instruction-tuned models, suggesting RLHF shifts criterion without improving metacognitive signal quality.

Standard LLM calibration metrics such as ECE conflate sensitivity (discriminating correct from incorrect) and bias (the tendency toward confident or cautious responding). This pre-registered study treats three LLMs as psychophysical observers performing factual discrimination across 168,000 trials, applying the full parametric SDT framework (unequal-variance model fitting, criterion estimation, z-ROC analysis) for the first time to language models. Temperature does not function as a clean criterion shift. It simultaneously increases sensitivity and shifts criterion. Models occupying distinct positions in sensitivity-bias space were indistinguishable by calibration metrics alone.

Applies formal tools of psychophysics to resolve conflicting accounts of how transformers represent magnitude. Across four converging paradigms (RSA, behavioural discrimination, precision gradients, causal intervention) in three 7-9B instruction-tuned models, representational geometry is consistently log-compressive, consistent with Weber's Law, but dissociated from behaviour. Early layers are functionally implicated in magnitude processing while later layers, where geometry is strongest, are not causally engaged.

Scalar variability, noise proportional to magnitude producing a constant coefficient of variation, is a hallmark of biological magnitude systems. This pre-registered companion to the Weber study tests whether transformers share this property. They show the opposite. Representational variability decreases with magnitude (α ≈ -0.19; 0/48 model-layer cells with α > 0). Corpus frequency strongly predicts per-magnitude variability (ρ = .84), consistent with a distributional account. Distributional learning produces the geometry of biological magnitude representation but not the noise.

Categorical perception, enhanced discriminability at category boundaries, is among the most studied phenomena in perceptual psychology. Using RSA across six models from five architecture families, a CP-additive model fits the representational geometry better than a purely continuous model at 100% of primary layers in every model tested. The effect is specific to structurally defined boundaries (digit-count transitions at 10 and 100), absent at non-boundary controls, and absent in the temperature domain. Two distinct signatures emerge. Classic CP (Gemma, Qwen), where models both categorise explicitly and show geometric warping. Structural CP (Llama, Mistral, Phi), where geometry warps but models cannot report the distinction.

Pre-registered evaluation of a neurosymbolic architecture combining BabyBERTa with a differentiable PCFG auxiliary loss derived from Minimalist Grammar. Structural priors produce phenomenon-specific effects. Filler-gap dependencies improve by 9-13 pp (d = 2.41-2.82) while locally-cued phenomena are damaged regardless of whether the grammar is real or random.

Pre-registered test of whether autoregressive transformers trained on synthetic child-scale corpora induce overhypotheses (second-order generalisations about category structure). Models achieve perfect first-order exemplar retrieval but remain at chance on second-order generalisation across 120 runs and three model sizes. Feature-swap diagnostics reveal template matching rather than structured abstraction as the operative mechanism.

First systematic evaluation of mutual exclusivity in text-only language models trained on child-directed speech. Across 45 GPT-2-architecture models spanning a 12x parameter range, all models show robust repetition priming rather than ME. Priming attenuates with better language modelling but never crosses zero. A context-dependence diagnostic rules out embedding artefacts as an alternative account.

A research programme formalising the analogy between binocular rivalry and LLM ambiguity resolution. The visual system resolves competing representations through mutual inhibition, adaptation, and goal-conditioned top-down modulation. The same computational problem faced by language models during ambiguous generation. Paper 1 validates the GC-LCA model against rivalry data. Paper 2 tests whether LLM internals exhibit metastable dynamics consistent with rivalry. Paper 3 applies the framework as an ambiguity detection mechanism.