Instead of Opening the Black Box, I Asked the Model to Explain Itself
A summer internship project. Most explainability methods try to reverse-engineer a model after the fact. I tried training one to just tell you why.
A sentiment model reads “My internship was incredibly amazing!” and says positive. The question I’ve spent this summer on, as an intern at Novetta, is simple to ask and annoying to answer: why? Which words, what reasoning? That’s the text explainability problem, and it matters for two boring-but-real reasons — debugging models that are confidently wrong, and giving an analyst a reason to trust, or distrust, a prediction instead of staring at a bare label.
I started where everyone starts, by surveying the open-source methods. Attention-based approaches are fast but easy to fool with adversarial inputs. LIME is model-agnostic and popular, but slow and inconsistent run to run. Anchors has a nice interface and is painfully slow on anything paragraph-length. They mostly work by perturbing the input many times and watching the output move, which is expensive — and running the same explanation twice can hand you two different answers. I wrote them all up in a guide and came away unconvinced I’d recommend any of them.
So I tried a different idea. Instead of reverse-engineering the model after it makes a decision, what if the model just hands you the explanation as part of its output?
The whole trick is in how you format the training data. I rewrote each labeled example so the explanation was part of the target, roughly like this:
(BEGIN TEXT) The selection on the menu was great and so were the prices. (END TEXT) (BEGIN EXP) It is positive because it contains: [‘great’]. (END EXP)
Train a generative model — GPT-2 — on a whole dataset formatted that way, and at inference you feed it new text and it generates the label and the words it’s keying on, in one shot. No perturbations, no second explainer model, no running it ten times and averaging. The explanation falls out of the same forward pass as the answer.
On a mix of movie, Amazon, and Yelp reviews it gets sentiment right about 94% of the time, and — the part I actually care about — 80% of the time the words it cites are genuinely present in the source text. The explanations are usually real, not invented.
“Usually” is doing some work in that sentence, and the failures are the fun part. Sometimes it calls a review negative “because it contains ‘worst’” when the word “worst” appears nowhere in the text. It has learned the shape of an explanation and will confabulate a plausible-sounding word to fill the slot — “No buyers remorse on this one!” comes back positive “because of ‘good’,” a word that isn’t there but fits the vibe. The model has, in a small way, learned to rationalize: produce a confident-looking reason that isn’t quite the real one.
I also ran an informal survey of the other interns — gave them the text, asked which single word best carried the sentiment, without showing them the model’s pick. They agreed with the model about 56% of the time. For a task where humans don’t fully agree with each other, that feels like a real signal rather than noise.
There’s plenty left to try. I only used GPT-2 medium; GPT-2 large, or GPT-3, which was just announced this summer, might sharpen both the accuracy and the explanations. But the part I find most interesting isn’t the numbers — it’s the bet underneath the method. Most explainability work tries to pry a model open from the outside, after the fact. This is the opposite: train the model to tell you why in the same breath as the answer. I think that’s the more promising direction, and I want to keep pulling on it.
