Testing nonexistence

Testing that something is not there can be unreliable

Today I am writing a test that something does not happen, like this:

Given that the A setting is "off"

When I start the app

Then the warning about A being on IS NOT displayed

It's quite hard to check that something is not displayed in a robust way. Initially it's fine:

assert!(page.does_not_contain("Setting A should normally be turned off!"));

But later if we change the wording of the warning, then this test becomes useless: we could introduce a bug that makes the warning appear, and this test would still pass, because it's looking for the wrong text.

Pair it with testing that something is there

The best way I have found to guard against this is to make sure every "not there" test is paired with an "is there" test as a companion:

Given that the A setting is "on"

When I start the app

Then the warning about A being on IS displayed

Now when we change the wording of the warning, this positive test fails, and we can update both tests to use the new wording.

So it's all fine?

But, of course, some poor programmer who was working on changing "normally" to "usually" across 800 source code files will probably not be thinking much about this pair of tests, and will be perturbing the code until all 10,800 tests pass, and could easily miss the pairing.

They might fix the wording in the "is there" test, and miss it in the "not there" test, making it useless again.

How to make this better

It might be nice to establish a naming convention to show that tests are paired, to help avoid this problem. Suggestions?

Even better, can we make some kind of "existence determined by X" test that merges this pair of tests into a single one? Something like this:

Given that the A setting can be either "on" or "off"

When I start the app

Then the warning is controlled by A:
    "on"  = displayed
    "off" = not displayed

I'm not aware of this pattern being used anywhere. Anyone? What would the code in the test look like?