Explainability · ENG work package 5

Where the model looks, and how we checked.

An explanation is a claim about the model, so we tested it like one. Four methods produced maps of where the detector looks; five tests asked whether those maps are faithful, stable, on the peak, free of artefacts, and tied to what the model learned. The analyzer shows the methods that held up.

  1. 01
    The maps reflect learning. The models score 0.98–0.99; a model retrained on shuffled labels gets maps unrelated to the real one, for all four methods.
  2. 02
    PFI is the most reliable. It sits on the peak for 4 of 4 analytes, is the most stable across models (0.89) and shows no edge bias.
  3. 03
    The other three depend on the fill. Occlusion, SHAP and SAGE are faithful when tested with their own fill, but the map changes when the fill changes.
  4. 04
    "On the peak" is the weak spot. No method sits firmly on the peak window (best 0.50); on P4VP no peak window can even be defined.
  5. 05
    The edges carry real information. 9–20% of the model's split gain sits on coefficients that only see the ends of the scan.
Two questions

Is the explanation true to the model, or true to chemistry?

They sound alike and are often mixed up. A map can match chemistry and still misdescribe the model, or describe the model faithfully and contradict chemistry.

Question A · asked here

Why did the model decide this?

Faithfulness and stability: does the map reflect what the model actually uses, and does it stay the same when the model is retrained? This is what work package 5 measures.

Question B · read alongside

Does the evidence agree with what we know?

Plausibility: does the map point at the analyte's known peak? We have that outside reference for PVDF — single-analyte peak windows — so the E3 test reads the maps against it. A match is a hint, not a proof of chemistry.

Four methods

Four ways to ask where it looked.

All four treat a 20 or 40 mV window of the curve as one player. Two explain a single curve, two explain the model as a whole. Three of them must replace a window with something — the fill — and that choice turns out to matter.

Occlusion

this curve

Replace one window, re-score the curve, record how much the probability drops.

Cover part of a photo and see whether the answer changes.

Fillzero · training mean · blank · straight line

KernelSHAP

this curve

Share the step from a reference curve to this curve fairly among the windows, over many combinations of windows.

Split a team's score fairly among its players.

Fillthe same four, as the reference for absent windows

Grouped PFI

whole model

Shuffle one window across all held-out curves and measure how much worse the predictions get.

Swap one ingredient between dishes and see which dishes suffer.

Fillnone — real values from other curves

SAGE

whole model

Share the model's accuracy fairly among the windows, averaged over curves.

Ask which players win matches, not which scored in one game.

Fillthe same four fills

Five tests

Each map had to pass the same exams.

Five seeds × six models per seed. The table reads one cell at a time: does this method pass this test, and by how much.

Held = passed; Conditional = passes under some settings; Weak = mostly fails. Numbers from the ENG XAI run summaries (5 seeds × 6 models), default setting 20 mV, R1 fill, log-loss; maps and E3 on PVDF.

Where it looks

The most reliable map, for this exact model.

Grouped PFI of the model running on this site (seed 0), from the XAI run. Pick a coating and an analyte; bars under the axis are the E3 peak windows.

    Permutation importance per window
    The edges

    Why some maps light up the ends.

    A six-level wavelet transform on 200 points reaches the ends of the scan. Some of its coefficients see almost nothing but the first or last ten points — and the model does use them. So part of the weight that maps put on the edges is real.

    The other part comes from the fill. Bridging a masked window with a straight line (R3) is harmless in the middle of a peak but, at the edge, it has only one neighbour and flattens the start of the scan; in the tests that fill pushed the edge share to 35% and was flagged.

    Coefficients that only see the ends—
    Computed in your browser from the wavelet filters and this model's split gain. A coefficient is edge-only when more than 90% of its weight comes from the first or last 10 points; grey cells sit exactly on 90%, so the count is a range — the XAI run's own computation counted 30.
    In the analyzer

    What you see there, and why.

    See it on a curve