External Corpus Loss/Complexity Tradeoff

Generated by scripts/run_external_corpus_tradeoff.py.

This is a bounded first pass over comp-tc-mark and CATENA. Rows and features may be sampled; the selected panels are written to outputs/tables/external_corpus_tradeoff/.

Row cap: 60
Feature cap per panel: 30
Random panels per dataset: 0
Target branch path depth cap: none
p-adic subset sizes: 1
p-adic random subsets per size: 6

target_absent_folds is important for sampled full-depth tree targets: it counts held-out rows whose exact parent branch is not represented in the training sample. In those folds, models can still be close p-adically but cannot predict the exact unseen branch label.

Negative panel-level log-log slopes: 2/4
Exact p-adic loss between dummy and best high-capacity model: 0/4
Exact p-adic parameters between dummy and largest high-capacity model: 4/4

Diagnostics

dataset	panel_id	family_points	slope_log_loss_vs_log_params	r_squared	linear_p_value	spearman_rho	padic_loss	padic_parameters	best_high_capacity_loss
catena_ephesians	panel_coverage	5	0.0697	0.0221	0.8114	0.6000	0.0000	12.0000	0.0001
catena_galatians	panel_coverage	5	-0.0002	0.0015	0.9509	0.6669	0.0000	12.0000	0.0000
catena_romans	panel_coverage	5	0.0415	0.0076	0.8892	0.6669	0.0007	12.0000	0.0013
comp_tc_mark_stemma_backbone	panel_coverage	5	-0.0442	0.1375	0.5389	-0.3000	0.3500	12.0000	0.2333

Family-best rows

dataset	panel_id	model_group	model	mean_padic_loss	adjusted_parameters_used	target_absent_folds
catena_ephesians	panel_coverage	Dummy	dummy_training_padic_medoid	0.0000	1.0000	44
catena_ephesians	panel_coverage	Exact p-adic	padic_exact_s1_k6	0.0000	12.0000	44
catena_ephesians	panel_coverage	Decision tree	opaque_tree_depth5_featuresnone_leaf1	0.0014	28.5778	44
catena_ephesians	panel_coverage	Random forest	opaque_forest_trees20_depth5_featuressqrt_leaf1	0.0001	1468.8387	44
catena_ephesians	panel_coverage	Logistic	logistic_l2_c1	0.0001	3742.4667	44
catena_galatians	panel_coverage	Dummy	dummy_training_padic_medoid	0.0000	1.0000	46
catena_galatians	panel_coverage	Exact p-adic	padic_exact_s1_k6	0.0000	12.0000	46
catena_galatians	panel_coverage	Decision tree	opaque_tree_depth5_featuresnone_leaf1	0.0000	30.9944	46
catena_galatians	panel_coverage	Random forest	opaque_forest_trees20_depth5_featuressqrt_leaf1	0.0000	1536.7676	46
catena_galatians	panel_coverage	Logistic	logistic_l2_c1	0.0000	4335.3667	46
catena_romans	panel_coverage	Dummy	dummy_training_padic_medoid	0.0007	1.0000	52
catena_romans	panel_coverage	Exact p-adic	padic_exact_s1_k6	0.0007	12.0000	52
catena_romans	panel_coverage	Decision tree	opaque_tree_depth5_featuresnone_leaf1	0.0353	21.5422	52
catena_romans	panel_coverage	Random forest	opaque_forest_trees20_depth5_featuressqrt_leaf1	0.0013	1522.9248	52
catena_romans	panel_coverage	Logistic	logistic_l2_c1	0.0020	3693.9333	52
comp_tc_mark_stemma_backbone	panel_coverage	Dummy	dummy_training_padic_medoid	0.3167	1.0000	35
comp_tc_mark_stemma_backbone	panel_coverage	Exact p-adic	padic_exact_s1_k6	0.3500	12.0000	35
comp_tc_mark_stemma_backbone	panel_coverage	Decision tree	opaque_tree_depth5_featuresnone_leaf1	0.7171	25.8868	35
comp_tc_mark_stemma_backbone	panel_coverage	Random forest	opaque_forest_trees20_depth5_featuressqrt_leaf1	0.3338	1303.7918	35
comp_tc_mark_stemma_backbone	panel_coverage	Logistic	logistic_l2_c1	0.2333	4689.0000	35

Figure

outputs/figures/external_corpus_tradeoff_facets.png