The Test With No Known Answer

A test usually starts with a known answer. Give a function an input, then compare its output with the expected result. That plan becomes hard when the output is a paragraph, a summary, or a judgment about meaning. A human may be able to assess each answer, but writing an exact expected response for every new input takes time. The research on metamorphic testing for language models starts with this problem: useful test inputs are easy to find, while reliable labels are scarce.

Check a relationship between answers

Metamorphic testing asks what should happen when an input changes in a controlled way. Run the original input and a related input through the same system. Then check a relationship between their outputs. The relationship is the test rule. The paper calls it a metamorphic relation: a condition on the inputs implies a condition on the outputs. It can expose a failure without requiring the exact right answer to either individual input. The paper defines this method formally.

Consider a simple square function. If the input changes from a number to its negative, the result should stay the same. A test can compare the two results even if it never stores the expected square. This example appears in the paper. Its value lies in the precision of the transformation and comparison. The test knows exactly what changed and exactly what equality means. The authors use it to explain numerical and language tests.

Try it on a language task

Imagine a system that classifies short support requests into categories. Start with “My invoice shows two charges for the same order.” Change it to “I was charged twice for one order.” Keep the classification instruction and all other settings fixed. If both sentences express the same request, their categories should match. A mismatch deserves inspection. This is an illustrative test design, based on the paper’s rule that a change preserving meaning should leave a suitable output unchanged. Its examples include paraphrases and synonymous wording.

The input condition matters. A rewrite can quietly change who did what, whether an event happened, or how strong a statement is. “The payment failed” and “The payment might fail” should not be treated as interchangeable. A test that assumes they are equivalent will flag a correct difference as a failure. Check the transformed input before blaming the system.

The output condition also needs care. Comparing category names is relatively clear if the system must return one label from a fixed set. Comparing free text is harder. Two answers can use different words and convey the same meaning. The paper found that free form question answering and relation extraction made output comparison especially difficult, while a task with a small set of labels allowed a simpler comparison. Those differences appear in its manual review.

Read a failed relation carefully

A failed relation points to an inconsistency. It does not identify which answer is wrong. It can also mean the input change failed to preserve the intended property, or that the output comparison was too strict. In the paper’s manual review, roughly 62% of sampled violations were judged true faults. The most common source of false alarms was a flawed input transformation. The authors report these findings and their review categories.

A passed relation has a limit too. A system may give the same wrong answer to both inputs. In the study, some incorrect original answers were accompanied by outputs that still satisfied the relation. The researchers therefore treat these tests as a complement to checks against known answers. Their comparison separates these cases.

This distinction changes how a team should use the signal. Save the original input, the transformed input, both outputs, the expected relation, and the reason the transformation was allowed. Review a failure against that record. If the relation itself was unsound, repair the test. If the behavior was faulty, turn the case into a regression test with a human checked expectation. This follows the paper’s proposal to use relation failures to select cases for more costly labeling. The authors describe that workflow.

Keep the experiment within its scope

The study gathered 191 relations from earlier work on natural language tasks and implemented 36 of them. It ran the tests on three language models and four tasks. Its average relation violation rate was 18%, with large differences across relations. That figure describes the tested combinations. It is not a general failure rate for language models. The experiment and results are reported here.

The authors also repeated the initially failing cases. Some failures changed across runs, so a single mismatch can overstate a stable defect. Repeating a case helps show whether it recurs, but repetition cannot fix a bad transformation. The paper found that input errors could produce persistent false alarms. Its repeated run analysis makes both points.

What to do

1. Pick one task with outputs you can compare clearly, such as a fixed set of classification labels. Keep a small set of ordinary inputs.
2. Write a transformation rule in plain language. State what it is allowed to change and what it must preserve. Make each transformed input visible for review.
3. Write the expected output relation before running the system. Hold the task instruction and other settings steady, as in the paper’s setup.
4. Run each input pair and record both responses. Treat a violation as a case to inspect. Repeat unstable cases with the same saved inputs.
5. Review the transformation and the comparison first. Then check the system’s behavior. Add confirmed faults to a labeled regression set, and keep those checks alongside the relation tests.

The useful question is small: when this input changes in a way we understand, what must happen to the answer? A precise relationship gives that question a testable form, even when the exact answer is still unknown.