Your Golden Tests Are Passing for the Wrong Reason

The last golden suite we inherited had 214 images in it. All green, for about eight months, across a few hundred pull requests. Then someone opened one of the PNGs to see what it was protecting.

Every glyph was a black rectangle.

The suite had been written by copying the first tutorial that came up, which is not a criticism — the tutorial works, in the sense that flutter test --update-goldens produces files and the next run compares against them. It just compares the wrong thing: a layout made of filled boxes, with its shadows flattened into hard blocks, on a machine nobody had pinned. Swap a word for another of the same length, change the font family, and the pixels do not move. The suite was not measuring the UI. It was measuring that nothing had moved.

That is the whole problem with golden tests in one example: they are simultaneously the cheapest regression net in Flutter and the most convincing fake. A green checkmark looks identical in both cases.

Key takeaways

• A golden is a snapshot of the render tree at one logical size, one theme, one locale, one text scale — and nothing else. Not a screenshot of your app on a phone.
• The test environment ships no app fonts. Unspecified text renders as solid boxes, deliberately, so layout stays deterministic. Without loading fonts, you are golden-testing geometry and calling it a visual test.
• Shadows are flattened into hard-edged blocks (debugDisableShadows): the framework drops the blur. Elevation is still in the picture, but blur radius, spread and gradient are not — so shadow regressions are invisible.
• Pixels are tied to the engine, the host OS and the Flutter version. A suite generated on macOS fails on Linux CI. Pin the toolchain and generate on the platform CI runs on.
• tester.view is the surface, not the device. The default test surface is 800×600 logical pixels at a pixel ratio of 3; without setting a size, every phone screen is rendered at a resolution no user has.
• pumpAndSettle hangs on a spinner — an indeterminate progress indicator schedules frames forever. Loading states need pump(duration).
• The matrix multiplies. 12 screens × 2 themes × 3 widths × 2 text scales × 2 locales is 288 images, all regenerated on every Flutter bump. Pick the axes you will actually review.
• A golden suite's value is the probability someone looks at the diff. Everything else is plumbing. An unreviewed --update-goldens commit converts a test into a lockfile.

What a golden test actually does

One test, one image, one comparison:

testWidgets('cart is empty', (tester) async {
  await pumpScreen(tester, const CartScreen(items: []));

  await expectLater(
    find.byType(CartScreen),
    matchesGoldenFile('goldens/cart_empty.light.390.ts1.0.en.png'),
  );
});

flutter test --update-goldens writes that PNG and passes. Every later run decodes it and compares it pixel by pixel against what the engine just rendered. Not a tolerance, not a perceptual hash — exact equality, until you install a custom goldenFileComparator. When it fails, the default comparator drops four artifacts into the failures/ directory beside the test file: the reference, the new render, an isolated diff and a masked diff. (A size mismatch writes only the first two.) Those are your diff review, and they belong in .gitignore.

Two details in that snippet matter more than they look.

find.byType(CartScreen) does not capture CartScreen. The matcher walks up from the found element to the nearest RepaintBoundary ancestor and captures that layer. Usually that ancestor is the whole app, which is why goldens so often come out as a full-window image with a debug banner in the corner. Wrap the subject in a RepaintBoundary and you capture exactly what you meant.

The rendered image is not your app. It is flutter_tester on your host machine: no platform fonts, no system UI, no keyboard, a mocked network that only returns 400s, shadows reduced to hard-edged blocks, and whatever surface size the test happened to have. A golden that passes tells you this tree, at this size, with these inputs, has not changed. That is genuinely valuable — and much narrower than "the UI works".

The five reasons your first golden lies to you

1. There are no fonts, so every glyph is a box

flutter test runs without your pubspec.yaml fonts loaded. Text with no available family falls back to a test font whose every glyph is a filled rectangle — a design that makes layout deterministic and font files unnecessary, and that means the PNG in your repository contains no readable text at all.

The fix is test/flutter_test_config.dart, which Flutter runs before the suites under that directory. Anything declared there — font loading, a custom comparator, clock setup — applies to every test below it:

// test/flutter_test_config.dart
import 'dart:async';
import 'package:flutter/services.dart';

Future<void> testExecutable(FutureOr<void> Function() testMain) async {
  await _loadFonts();
  await testMain();
}

Future<void> _loadFonts() async {
  // The family name must match what the theme asks for. If `fontFamily` is
  // null, register the same faces as 'Roboto' — that is what Material falls
  // back to, and a mismatch loads nothing and fails silently.
  // Both faces are declared under `assets:` in pubspec.yaml, and `flutter test`
  // can load them from there — a pure `dart test` cannot.
  final text = FontLoader('Roboto')
    ..addFont(rootBundle.load('assets/fonts/Roboto-Regular.ttf'))
    ..addFont(rootBundle.load('assets/fonts/Roboto-Medium.ttf'));

  // Icons are fonts too. Skip this and every `Icon` in every golden is a box.
  final icons = FontLoader('MaterialIcons')
    ..addFont(rootBundle.load('assets/fonts/MaterialIcons-Regular.otf'));

  await Future.wait([text.load(), icons.load()]);
}

Two things make this fail quietly rather than loudly. A wrong family name loads a font that no widget asks for, so nothing changes. And google_fonts downloads its faces at runtime, which the test environment does not do — vendor the .ttf files you actually ship, or your goldens are permanently pending a network call that never happens.

There is a subtler consequence, and it is the one that bites teams who skip this section: with box glyphs you cannot see what the text says, and every property the test font normalizes disappears. Family, weight, kerning, glyph shape and the real font's ascent and descent are replaced by identical squares, so switching from Roboto to Inter renders the same boxes, and a copy change of equal character count moves nothing at all. Character count and font size still move the boxes — a longer Russian string produces a wider bar — so overflow and size changes are not invisible. Only the things that make type legible in production are.

2. Shadows are flattened, so blur does not exist

debugDisableShadows does not remove shadows; it replaces them with solid blocks. BoxShadow.toPaint() drops its mask filter, and a Material surface paints its elevation as a hard outline (strokeWidth: elevation * 2), so what lands in the golden is an unblurred shape in the shadow color. The test binding turns it on because shadow rendering is not guaranteed to be pixel-identical from version to version, or even from run to run.

The consequence is narrower than "no shadows" and more misleading: the card is still there, but its blur radius, spread and gradient are not, so a shadow regression — a blur that doubled, a spread that broke the layout — renders identically in the golden. The Flutter framework's own tests set debugDisableShadows = false when they are specifically testing shadows, and restore it at the end of the test; the binding asserts the flag is back to its default (true) afterward. That is the right shape of decision: turn real shadows back on where the shadow is the thing under test, accept the flakiness there, and leave the flag alone everywhere else.

3. Every network image is a placeholder

Image.network in a test hits a mocked HttpClient that returns an empty 400 response. What happens next depends on the widget: with no errorBuilder, the Image widget re-throws the load error in debug and the test fails with a NetworkImageLoadException; with one, the golden quietly records your placeholder instead. cached_network_image and friends need their own fake. Either wrap calls in HttpOverrides (older recipes do), or do the thing that survives a refactor: take an ImageProvider in the widget and hand it a MemoryImage of a 1×1 PNG in the test.

Avatars and product shots are most of the visual weight of a real screen, so a suite where all of them are placeholders is testing the layout around the pictures and nothing about the pictures.

4. The clock, the locale and the randomness are not yours

DateTime.now() in a widget, DateFormat without a fixed date, an unseeded Random, a uuid generated during build — each one makes a golden that passes today and fails on the second Tuesday of some month, or fails only in CI where the machine's locale differs.

The fix is not in the test, it is in the widget: take a now function or a Clock, take a Locale, take a seed. That is worth doing anyway — a widget that reads the wall clock directly is a widget you cannot test at any layer, golden or otherwise. Set Intl.defaultLocale explicitly in flutter_test_config.dart and you remove a whole class of "works on my machine, wrong language in CI" failures; for non-English dates, also call initializeDateFormatting(locale), because DateFormat throws for a locale it has no data for.

5. pumpAndSettle never returns on a loading state

pumpAndSettle pumps until no frame is scheduled. An indeterminate CircularProgressIndicator schedules frames for as long as it spins, so the call times out with the famously unhelpful "pumpAndSettle timed out", and the loading-state golden — the one state QA never looks at because you cannot get the app to stay in it — is the one you cannot write.

// Not this: it waits for an animation that never ends.
// await tester.pumpAndSettle();

await pumpScreen(tester, const CartScreen.loading());
await tester.pump(const Duration(milliseconds: 100)); // one fixed phase of the spinner
await expectLater(find.byType(CartScreen), matchesGoldenFile('goldens/cart_loading.light.390.ts1.0.en.png'));

The golden now captures the spinner at 100ms, consistently, forever. That is fine: you are testing that the surrounding layout holds while something indeterminate spins, not the spinner itself.

The setup that makes goldens reproducible

Four decisions, all boring, all of which you make once:

Pin the toolchain. A .fvmrc in the repo, and CI reading the same file. A Flutter upgrade changes text rendering, theme defaults, CardTheme-to-CardThemeData renames and shadow painting, so every golden in the repository changes at once. That is not a reason to avoid upgrades; it is a reason to make the upgrade a deliberate commit that contains nothing but regenerated goldens, marked as such in the message.

Generate on the platform CI runs on. Text anti-aliasing and hinting differ between macOS and Linux — the same widget at the same size produces different pixels. A suite generated on a developer's Mac will fail on a Linux runner, and the usual "fix" is --update-goldens somewhere it should not be. Run goldens on Linux locally too, in the same container as CI if the team is mixed.

Never let CI regenerate. The guard is one line:

- uses: subosito/flutter-action@v2
  with:
    flutter-version-file: .fvmrc   # one version, repo and CI
    cache: true
- run: flutter test
# A "helpful" --update-goldens in CI or a stale golden written by a bad merge
# would make the suite green while comparing nothing. Fail instead. `git diff`
# alone misses untracked files, which is exactly what a regenerated golden is.
- run: test -z "$(git status --porcelain -- '*.png')"

That last line is not paranoia. The most common way a golden suite stops testing anything is a CI step, added during a red build at 2am, that updates goldens instead of failing.

Set the surface explicitly. The default test surface is 800×600 logical pixels at a pixel ratio of 3. Test a phone screen on it and you get the layout of a small tablet, with the app's own MediaQuery insisting otherwise. One helper solves it for every test, and if you cannot read the golden's name and know the five axes it was rendered at, the file name is wrong:

// test/helpers/golden.dart
Future<void> pumpScreen(
  WidgetTester tester,
  Widget child, {
  Size size = const Size(390, 844), // iPhone 14 — pick your own baselines
  double textScale = 1,
  Locale locale = const Locale('en'),
  Brightness brightness = Brightness.light,
}) async {
  tester.view.physicalSize = size * 3;   // logical size × device pixel ratio
  tester.view.devicePixelRatio = 3;
  addTearDown(tester.view.reset);        // never leak a view into the next test

  await tester.pumpWidget(
    MaterialApp(
      debugShowCheckedModeBanner: false,
      theme: brightness == Brightness.dark ? AppTheme.dark : AppTheme.light,
      locale: locale,
      // Without the app's own delegates a locale it does not support resolves
      // back to English.
      localizationsDelegates: AppLocalizations.delegates,
      supportedLocales: AppLocalizations.supportedLocales,
      home: RepaintBoundary(
        child: Builder(
          builder: (context) => MediaQuery(
            // Merge, don't replace: a fresh MediaQueryData has size zero,
            // devicePixelRatio 1 and no padding.
            data: MediaQuery.of(context).copyWith(
              textScaler: TextScaler.linear(textScale),
            ),
            child: child,
          ),
        ),
      ),
    ),
  );
  await tester.pump();
}

Two details here are easy to get wrong. MediaQueryData(...) instead of copyWith builds a fresh data object with size: Size.zero and devicePixelRatio: 1, so MediaQuery.sizeOf lies to every widget below it. And a MaterialApp rebuilt inside the helper resolves any locale it does not support back to English unless you hand it the app's own delegates (which is what AppLocalizations stands in for here); otherwise the locale axis silently tests English.

addTearDown(tester.view.reset) is not optional. tester.view is global for the test file, so one test that sets a 320pt width leaks into every test after it, and the failure shows up attached to an innocent test later in the same file.

What to golden, and what the matrix costs

The trap is that the matrix looks free. It is not: every image is a file to regenerate, a file to review, and a file that adds a line to the diff of your next unrelated pull request.

That is a suite of roughly 60–90 images: about twenty seconds of CI and a pull request whose visual diff a person can actually look at. The version that multiplies everything by every axis — 12 screens × 2 themes × 3 widths × 2 text scales × 2 locales — is 288 images, and nobody reviews 288 images.

Three things are not worth goldening at all:

• Animations and transitions. A golden captures one frame at one fixed time. It tells you the frame at 100ms looks the same, not that the animation is smooth, and the marginal value over the initial and final states is near zero.
• Anything with real content in it. Long product names, prices, dates, generated IDs — every one is a reason for the golden to change for a non-visual reason. Golden the shell with fixed fixture data, and test content rendering with a widget test that asserts on text, not pixels.
• Whole flows. Three screens in one image is a third as reviewable as one screen per image.

And the sharpest thing to golden, in our experience, is not a screen at all: it is the design system at 320pt in dark mode with text at 200%. That combination is where real apps break, it is nearly impossible to check by hand across every component, and it is exactly what a golden is good at.

What goldens do not test

A golden is a picture. It knows nothing about meaning, and a few common assumptions are worth correcting explicitly.

Behavior. A button wired to the wrong callback renders identically. So does a form that no longer validates, a list that lost its scroll controller, a dialog that fails to dismiss. Those are widget tests — tap, pump, assert on what happened — and no amount of pixels substitutes for them.

Accessibility. The semantics tree is invisible to a screenshot: missing labels, wrong roles, a tap target reduced to 24pt, contrast under 4.5:1 for body text (3:1 for large text). Flutter ships the assertions, and they are cheap:

testWidgets('cart meets a11y guidelines', (tester) async {
  final semantics = tester.ensureSemantics();
  addTearDown(semantics.dispose);

  await pumpScreen(tester, const CartScreen(items: []));

  // Tap targets (48dp Android / 44pt iOS) and text contrast.
  await expectLater(tester, meetsGuideline(androidTapTargetGuideline));
  await expectLater(tester, meetsGuideline(textContrastGuideline));

  // The label and role the golden cannot see. `isSemantics` checks only the
  // values you pass; `matchesSemantics` would also require every other flag
  // to be false, and a real Material button sets `hasEnabledState`.
  // (`containsSemantics` is the same matcher on Flutter before 3.40.)
  expect(
    tester.getSemantics(find.byType(CheckoutButton)),
    isSemantics(label: 'Checkout', isButton: true, hasTapAction: true),
  );
});

Real-device rendering. Goldens run on the host engine, so they do not tell you what Impeller on iOS does with the same tree, or how the platform's own text shaping differs. If a difference matters, that is integration_test on a device, with a screenshot captured there — slower, flakier, and the only thing that answers the question.

Jank. Frame budget is measured on a profile build on a real device, not inferred from a PNG.

The rules that decide whether it was worth it

The suite's value is proportional to the probability that a human being looks at the diff. That is the entire return on the investment, and it is a cultural property, not a technical one.

So, concretely:

1. --update-goldens only in a commit whose message names the visual change. "Update goldens" as a commit message means the diff was not reviewed. "Update goldens: checkout button is 48pt and the error state uses the new palette" means it was.
2. A golden diff in a screen you did not touch is a finding, not noise. That is the suite doing its job. Batching it into "chore: update goldens" is the suite being switched off.
3. If the reviewer cannot see the images, the tooling failed. Upload the failures/ artifacts from CI, or attach the diff to the pull request. A review that consists of "trust me, the pixels changed" is not a review.
4. Delete the goldens you would rubber-stamp. A suite of forty images that are all looked at beats four hundred that are all approved in one click. Coverage is not the metric; reviewed coverage is.
5. When goldens change on unrelated pull requests, stop and find the cause — an unpinned Flutter version, a font that is not loaded, a DateTime.now() that reached a widget. Flakiness in goldens is almost never the mechanism; it is a real source of non-determinism you now know about.

What it costs, six months in

Setting the suite up is a day, maybe two, most of it fonts and CI plumbing. Writing each golden is minutes. The recurring cost is the one nobody estimates: every Flutter upgrade regenerates every image. With the matrix above that is a commit of 60–90 PNGs, two to four times a year, each one needing its own justification. If you cannot afford that review, you cannot afford the suite — because an unreviewed regeneration is indistinguishable from no test at all, except that it takes longer.

In exchange you get the regressions that no other test type catches: the empty state that lost its padding in dark mode, the Russian translation that overflowed a 320pt button, the design system drifting because someone changed one TextStyle in the theme. And the case that made us care in the first place — an agent or a teammate changing a shared component and being genuinely unable to see the four screens it broke. A golden is the closest thing Flutter has to handing a code change a pair of eyes, which is exactly the argument we made in why AI agents struggle with Flutter.

The short version

A golden test is a pixel comparison against a PNG, and everything that makes it useful is about the conditions of that comparison. Load your fonts, load your icon font, set the surface, pin the toolchain, regenerate on the platform CI uses, and never let a machine approve the diff.

Then keep the suite small enough that a person still looks at it. The failure mode of golden testing is not a flaky test — it is 214 images that all pass because they compare boxes to boxes, and a green checkmark that has been telling the same comfortable story for eight months.

Sources

• matchesGoldenFile — what the matcher captures, and the goldenFileComparator hook: api.flutter.dev
• FontLoader — registering a font face by family name inside a test: api.flutter.dev
• debugDisableShadows — declared in painting/debug.dart, set by the test binding, and flipped back on by Flutter's own shadow tests: github.com/flutter/flutter
• TestFlutterView — physicalSize, devicePixelRatio and reset(): api.flutter.dev
• WidgetTester.pumpAndSettle — "repeatedly calls pump with the given duration until there are no longer any frames scheduled": api.flutter.dev
• AccessibilityGuideline — androidTapTargetGuideline, iOSTapTargetGuideline, textContrastGuideline: api.flutter.dev
• Framework goldens triaged on Skia Gold, and the flutter/goldens repo that holds the ones that cannot use it: skia.org, github.com/flutter/goldens
• alchemist — per-platform golden folders, grouped multi-widget goldens, CI-aware configuration: pub.dev
• golden_screenshot — screenshot tests with device previews and platform-specific goldens: pub.dev
• golden_toolkit — the package most golden-test tutorials still recommend; discontinued, last released February 2023: pub.dev
• spot — golden-style widget tests that compare a serialized widget tree instead of pixels, when pixel instability is the bigger problem: pub.dev
• integration_test — screenshots taken on a real device (call convertFlutterSurfaceToImage() first on Android), the only answer to "what does it look like on a phone": integration_test.dart

Frequently asked questions

If you are setting up the testing layer from scratch, the surrounding decisions are in our write-up of the Flutter app development process, and the state-model problem that makes half of a UI untestable is in five states that cannot exist. The theme in all three is the same: a check that passes is not the same thing as a check that looks.

Maxim is a Senior Software Engineer at Nerdy Production, a Flutter-first agency that builds and maintains apps across fintech, healthcare, and retail. We also do Flutter app development for teams who would rather find out what their test suite is comparing before the next release.