Add LoRA fine-tuning to the MLX backend

[ovuruska]

Stacked on #49 (MLX backend) — only the feat: add LoRA fine-tuning to the MLX backend commit is new here. Replaces #51 (closed by a history rewrite that removed CLA-blocking co-author trailers).

Summary

Parameter-efficient LoRA fine-tuning for the MLX backend. The pre-trained base weights are frozen and never updated — training optimizes only low-rank adapter matrices.

• lora.apply_lora(model, rank, alpha, target_keys, scope) — freezes the model first, then wraps target attention projections (default q_proj/v_proj in the ICL predictor) with LoRALinear; trainable_parameters() is exactly the adapter set.
• lora.train_lora(...) — in-context episode training: rows split into context (labels visible) and targets, loss on targets only, target labels masked with the -100 sentinel (leakage structurally impossible).
• lora.fit_lora(estimator, X, y, ...) — runs the sklearn wrapper's ordinary fit() (encoders + ensemble prep; never touches weights), then trains adapters on the same preprocessed matrix the wrapper feeds the model at predict time.
• lora.merge_lora(model) — folds adapters into the base for zero-overhead inference; save_adapters/load_adapters round-trip only adapter tensors.
• Adapters in float32 over the bf16 base; B starts at zero → adapted model is bit-equivalent to base until trained.

Tests (full suite: 82 passed)

Base-frozen invariant (bit-identical after training) · loss descent · merge parity (<1e-4) · adapter save/load round-trip · wrapper end-to-end (fit_lora → predict) · regression path smoke.

Validation on real weights (Apple M1 Pro, 6.5GB checkpoint)

• Adapter inference overhead: ~zero (unmerged within noise of zero-shot; merged identical by construction).
• Accuracy on tasks where zero-shot is at ceiling (0.94–1.0): neutral with 50 steps @ lr 1e-4; overfits with 200 steps @ 3e-4 — as expected, no headroom.
• See the validation results below.

Validation results (real weights, Apple M1 Pro) — honest assessment

The mechanism is verified; an out-of-the-box accuracy gain is not. Across every regime tested, LoRA with this recipe (rank 8, ICL scope, episode training) tracked zero-shot within noise:

Setting	Zero-shot	LoRA	XGBoost ref
Synthetic easy, 200 rows	0.960–1.000	−0.02 (overfit @200 steps) / neutral (@50 steps)	—
Synthetic hard, 20k rows, 512-row context	0.992	0.988	0.994
Adult census income (ACI), 20k rows	acc 0.854 / AUC 0.913	acc 0.856 / AUC 0.909	acc 0.876 / AUC 0.927
Kaggle Titanic, 712 rows	acc 0.799 / AUC 0.852	acc 0.804 / AUC 0.849	acc 0.793 / AUC 0.807

Observations:

• Training loss always decreases healthily (e.g. 0.63→0.25 on ACI) — optimization works; it just does not transfer to test metrics because zero-shot ICL already extracts what the context offers.
• Context scaling beats gradient adaptation on these tasks: growing the context 200→512 rows lifted a hard synthetic from 0.77 to 0.99 zero-shot.
• TabFM zero-shot itself is strong: it beats XGBoost on small data (Titanic AUC 0.852 vs 0.807) with zero training.
• Adapter inference overhead is ~zero (and exactly zero after merge_lora).

Where gains may still exist (untested): distribution shift / domain adaptation (train adapters on one distribution, deploy on drifted data), datasets far outside the pre-training prior, larger adaptation budgets (higher rank, scope="all", more epochs), and regression tasks. Keeping this PR as draft until a gain regime is demonstrated is a reasonable call.