Retrace

What it does

You give Retrace a photograph. It gives you back one of three answers, a picture of where it thinks the image was changed, and a plain-English reason for every measurement it took.

The three answers are clear, flag, and send this to a person. Three rather than two, because a system that must choose between “fine” and “fraud” on every image will be wrong loudly, and the useful product for a claims team is a queue sorted by how much a human needs to look.

The setting is insurance. Someone photographs hail damage to a roof, or a dented bumper, and the photo is the evidence the payout rests on. Adding damage that was not there is cheap. Checking a few thousand of those a day by eye is not.

The cover image on this page is real output. Two figures were pasted into a courtyard photograph, and the highlighted region is what the system found.

How it works

Most tools for this train a classifier on a pile of real and fake images. That works until someone uses an editor the training set never saw, which in practice is immediately.

Retrace does not look at the picture, and it never sees an original. It is given one photograph and asks whether that photograph is internally consistent with having come from a single sensor in a single exposure. Editing breaks that consistency in ways that are properties of the hardware and the file format rather than of any particular editing tool, which is why the same checks apply to an editor that did not exist when the system was written.

The two that carry the measured result both work on the image alone:

Sensor structure. A camera sensor measures only one colour at each pixel and interpolates the other two from its neighbours, which leaves a fine, regular structure across every genuine photograph. Pixels that were painted, rendered, or pasted from a rescaled source were never interpolated, so they do not carry it. Retrace measures that structure in roughly 870 blocks and reports the blocks that disagree with the other blocks of the same image. On a hand-made forgery that reads as something like 59 of 872 blocks carry demosaicing structure inconsistent with the rest of the image, at 7.9 sigma.

Duplicated regions. Most “clone out the scratch” edits copy nearby pixels over the thing being hidden, so the giveaway is a region that appears twice in one frame. The check finds groups of keypoints that all shift by the same distance, which produces a claim someone can verify by hand: 212 keypoint pairs share a single displacement of (400, -10) pixels. On the example above, that is a decorative window clone-stamped along a wall, in a file with no original of any kind attached to it.

Two more tiers sit around those:

The file, before the image. When was it taken, what software wrote it, and is the file actually the format its name claims? A damage.jpg whose bytes are really a PNG is a finding on its own, so containers are identified by their leading bytes and never by the extension. This tier also catches the thing no pixel analysis can: a perfectly authentic photograph taken three weeks before the policy started.

Compression history. JPEG compresses in 8x8 blocks locked to the corner of the image. Paste in a region from a different photo and it arrives carrying its own grid, and only one alignment in 64 happens to land back in step with the frame around it.

And one lucky shortcut, worth describing precisely because it is the thing people assume the whole system is. Most image files carry a second, smaller copy of themselves, an EXIF thumbnail or the embedded JPEG inside a RAW file, and editors routinely rewrite the main image while forgetting to regenerate it. When that happens the preview is a photograph of the pre-edit original, so the system recovers it and differences the two. Those are real earlier pixels out of the same file, never a reconstruction, and the type system refuses to let a synthesised guess present itself as recovered evidence. It is also luck rather than method: on the 224 benchmark photographs that path is applicable zero times, so none of the accuracy below comes from it.

Each detector returns three separate things, and keeping them apart is the design decision I would most want to defend: can I speak to this image at all, what did I find, and how much do I trust myself here. A detector that cannot run is dropped from the vote entirely rather than folded in as a neutral 0.5, because “I have nothing to say about this” and “I looked and found nothing” are different claims, and averaging them together is the fastest way to make a multi-detector system produce confident nonsense.

The votes are then pooled in log-odds, weighted by confidence, with findings of absence deliberately discounted. Finding a trace is informative. Failing to find one mostly is not. Without that asymmetry, a handful of detectors correctly reporting nothing in their own domains would average away a near-certain positive from the one detector that actually saw something.

The question I was not asking

Everything above asks whether a real photograph was altered after it was taken. That question quietly assumes a photograph was taken at all.

Ask ChatGPT to edit an image and it does not patch your original, it regenerates the whole frame. Every pixel is synthetic, evenly, so nothing is locally inconsistent with anything else and the strongest detector abstains rather than firing. I ran a real ChatGPT export through it: cleared at 0.288, which is lower than a genuine untouched photograph scored by the same pipeline. Not a near miss. A clean pass.

My first fix was to report the abstention itself, since an image carrying no sensor fingerprint was never produced by a camera. I built the measurement before building the detector, and the measurement killed it: an honest photo re-saved at JPEG quality 75 already fails that test 93% of the time, so it cannot tell generated from forwarded through WhatsApp. It does not ship.

I had been looking in the pixels. The answer was in the bytes.

That ChatGPT file carries a 29 KB signed C2PA manifest, issued by OpenAI, asserting digitalSourceType: trainedAlgorithmicMedia with softwareAgent: gpt-image. The image declares under cryptographic signature that a model made it, and my pipeline was reading straight past it to hunt for demosaicing artefacts that were never going to be there.

So Retrace now has a provenance tier that reads and verifies that manifest against its certificate chain. Not trusting a metadata string, checking a signature. Same file:

before   AUTO CLEAR   0.288
after    FLAG         0.876
         "this image says it was generated by an AI model by gpt-image,
          and the claim is cryptographically signed, signed by OpenAI OpCo, LLC"

It is also the only tier that can produce positive evidence of authenticity. A manifest asserting camera capture scores below 0.5. Everything else in the system can only fail to find a problem.

The limit is severe, and I measured it rather than caveating it. Content credentials do not survive contact with anything:

what happened to the file	credentials survive
as supplied	100%
re-saved as PNG	0%
re-encoded to JPEG, q95 or q75	0%
resized by a messaging app	0%
screenshotted	0%

One save strips it. That is not a flaw in C2PA, it is what metadata is: the bytes are not the pixels, and nothing binds them together once the chunk is dropped. So the detector is deliberately one-sided. A valid manifest is close to conclusive; its absence means nothing at all, and it abstains rather than reporting innocence, because a stripped generated image and an honest photo that went through WhatsApp look identical on this signal.

Which leaves the case I genuinely cannot do: a generated image whose credentials have been stripped. Nothing in the project catches that today, and I would rather say so than imply otherwise.

Where it stops working

The demo shows edits it catches, which tells you nothing on its own. So I went looking for the boundary by trying to defeat it: stage a clone into a real photograph, then vary how big it is, how hard it is to see, and whether it was transformed on the way in.

Visibility turned out not to be the axis. The check measures duplication, not how obvious something looks, so degrading the image afterwards does almost nothing:

applied after the clone	result
saved at JPEG q75 / q50 / q30	caught, 62 / 53 / 30 matching pairs
blurred, noise added	caught
rotated 3 degrees, rescaled 1.08x	caught

Three things do defeat it:

evasion	result
region smaller than about 1% of the frame	missed below 120px
heavy feathering, a 60px blend on a 250px patch	missed, zero matches
mirroring the copy	missed, zero matches

Two of those three are ordinary editing habits rather than attacks, which is the part worth knowing. A healing brush feathers aggressively by default, and retouchers mirror clones precisely so the result does not look repetitive. Neither requires knowing this system exists.

So the claim I can actually support is narrow, and narrower than the demo suggests: a hard-edged or lightly-blended, un-mirrored copy covering more than roughly 1% of the frame, after which it survives essentially any degradation. Mirroring is the fixable one, by matching against flipped descriptors at about double the cost, and it is unbuilt.

What it scores

Headline numbers come from the Korus Realistic Tampering Dataset, which is 112 forgeries made by hand in GIMP and Affinity Photo plus their 112 matched pristine originals. Real cameras, someone else’s data, pixel-exact ground truth.

ROC AUC                 0.792   95% CI [0.731, 0.847]
threshold 0.75          14.3% of forgeries caught, 0.0% false positives
clean photographs auto-cleared                74.1%
localisation on flagged images       11.2x better than the mask area alone

The 0.75 row is the one that matters. A high-confidence tier that catches one forgery in seven without a single false accusation across 112 clean photographs is worth more to a claims team than a higher hit rate that occasionally calls an honest customer a fraud.

The part I would want to be asked about

An earlier version of this system was chance performance, and the repository says so. Synthetic fixtures scored 0.99 localisation. On real photographs it scored 0.010, with a false-positive rate above its true-positive rate and score distributions identical to three decimal places.

The lesson turned out to be about the fixture, not the detector. My noise_splice() generator built a scene with uniform noise and swapped in a patch at a different level, which is precisely the assumption the noise detector made. The test could not have failed, and the detector could not have failed it. Real photon shot noise scales with the square root of signal, so bright regions of an untouched photo are genuinely noisier than dark ones, and hundreds of blocks legitimately deviate in both versions of the same scene.

Every synthetic number in the repository is now labelled an implementation check, meaning it only asks whether a detector fires in the right place given its own assumptions. Only the real photographs are allowed to be a performance claim.

Four detectors that did not survive measurement

They are still in the tree, fully documented, and excluded from the pipeline:

• Compression grid required two windows to agree on a foreign JPEG phase. Across N confident windows and 64 possible phases, chance supplies about N/64, roughly ten on a full-frame photo. The threshold sat below the noise floor, so it fired on three of four pristine images. Raised to beat chance, it finds nothing.
• Focus consistency fires at 7.2 sigma on a photo with a hand-replaced licence plate, and localises to the crumpled bumper and the tree line. Zero percent of flagged pixels inside the actual edit.
• Rendered content goes straight from flagging every clean photograph to detecting nothing. There is no operating point in between.
• Sensor noise survived one repair. I replaced the global-noise assumption with a fitted noise level function, a genuinely better model of a photograph, and it still scored AUC 0.494, below chance, across all 224 images. A better model of the wrong quantity does not become the right quantity.

The pattern is the point. Physically-motivated cues keep looking convincing on one image and failing to be selective across several. Every detector that ships earns its place by how much removing it costs the pipeline, and the full set is the best-scoring subset.

Calibration, and why the score still is not a probability

The system ordered images well, but its output was not a probability, so the threshold was a rank cut-off wearing a probability’s clothing. I fitted the missing map and validated it two ways, because the two protocols answer different questions:

protocol	question	result
fit on one camera, test on the other	does it transfer to an unseen sensor?	no, nothing significant, every interval spans zero
within one camera, out-of-fold	is it learnable at all?	yes, every metric improved, 20/20 shuffles

The fitted slopes explain the split: 2.05 on the D7000 against 1.02 on the D90, a difference of +1.03 [+0.04, +2.08], p = 0.038. The D7000’s fusion is genuinely under-confident. The D90’s is about right and merely off-centre.

So the miscalibration is a property of the sensor, not of the fusion, and the shipped system still defaults to no calibration. Applying one globally would export one camera’s correction to another and make the number look more trustworthy while making it less true. Running only the cross-camera test would have confused does not transfer with does not exist. Running only the within-camera one would have claimed a calibration deployment cannot use.

Two things I took from it that generalise. Calibration cannot change AUC, because a monotonic map preserves every pairwise ordering, and the measured change was bit-identical zero. Anyone reporting a calibration that improved their AUC has a bug. And binning is not cosmetic: equal-width bins put 119 of 224 images into a single bin, and switching to equal-mass bins changed the headline error and flipped the apparent sign of one result.

Using it

There is a local review tool, a FastAPI service with a drop zone, that shows the verdict beside every measurement that produced it, including the ones that did not apply. Hiding the abstentions would make the tool look more decisive and be worth less, because knowing that four of seven checks could not speak to an image is most of what tells a reviewer how much to trust the eighth.

It is styled with mk-ui, my own element collection, which is the point of having built one. The three verdicts map onto its semantic colours, so changing the accent in the collection changes what a cleared claim looks like here.

126 tests, continuous integration across every install shape, and the camera fingerprint detectors abstain cleanly rather than crashing when PyTorch is absent.

What is still open

The score needs labelled forgeries from more than two camera models before anyone can say whether the calibration slope tracks something predictable, like sensor generation or read-noise level, in which case a calibrator could be chosen from the file’s own metadata rather than fitted per camera. Splicing between cameras that share a noise level function remains undetected. Removal by filling in a region needs a “too smooth” cue that is not built. And every Korus image is a TIFF, so the compression-tier detectors have never met data that could falsify them.