Running a subset of reftests
One aspect of running reftests that periodically annoyed me was that I would often want to run a subset of all the reftests within a particular directory. Sometimes it would be a single test that I wanted to run, other times a particular set of tests. With the mochitest harness, it’s easy to run an individual test:
$ TEST_PATH=path/to/the/test.html make -C $(OBJDIR) mochitest-plain
or just:
$ ./mach mochitest path/to/the/test.html
if you’re using mach. This wasn’t possible with the reftest harness, since you need to give a manifest file that has information about what the document that provides the reference image is, whether it’s a test for equality or inequality, and so on.
Usually I would deal with this by either writing up a new manifest file, copying the relevant entries from the real reftest.list manifest, or by temporarily commenting out all the entries but the ones I was interested in. This was a bit of pain, so now with bug 809312 landed, you can pass a --filter command line option to runreftest.py to select which tests to run.
--filter takes a regular expression against which to test the first (test) URL of each test entry in the manifest file you are using. It uses JS regular expression syntax. So if you are still using the make targets to run reftests, you could do this:
$ EXTRA_TEST_ARGS=--filter=mask TEST_PATH=layout/reftests/svg/ \
make -C $(OBJDIR) reftest
and it will run only the reftests whose test URL matches /mask/. And with mach it’s simpler:
$ ./mach reftest layout/reftest/svg/ --filter=mask
If you just want to run a single reftest, you can write the whole filename there:
$ ./mach reftest layout/reftest/svg/ --filter=mask-basic-01.svg
It’s still a regular expression, so the dot will match any character, but it’s unlikely to matter given the simple filenames we use.
Note that the regular expression is matched against the absolute URL of the test document, which is going to be something like file:/path/to/your/repo/layout/reftests/.... So if you specify --filter=file don’t be surprised if all tests run!
Specifying default preferences in a reftest manifest
Recent features I have been working on, such as @supports, live behind a pref until we decide that their relevant specification is stable enough. We default the pref to true for Firefox Nightly and Aurora builds, and false for Beta and Release. Initially, I had a number of reftests for @supports that were skipped when the feature was not enabled, but it was pointed out that it would be better to force the pref on while running the tests so that we didn’t lose test coverage on the Beta and Release branches. My reftest manifest then looked like this:
pref(layout.css.supports-rule.enabled,true) == css-supports-001.xht support/pass.xht
pref(layout.css.supports-rule.enabled,true) == css-supports-002.xht support/pass.xht
pref(layout.css.supports-rule.enabled,true) == css-supports-003.xht support/pass.xht
pref(layout.css.supports-rule.enabled,true) == css-supports-004.xht support/pass.xht
pref(layout.css.supports-rule.enabled,true) == css-supports-005.xht support/pass.xht
...
It seemed a bit silly to have to repeat this pref() setting for each test, so with bug 788967 landed, you can now use a default-preferences statement in a reftest manifest to specify preferences to use for the remaining tests in the manifest:
default-preferences pref(layout.css.supports-rule.enabled,true)
== css-supports-001.xht support/pass.xht
== css-supports-002.xht support/pass.xht
== css-supports-003.xht support/pass.xht
== css-supports-004.xht support/pass.xht
== css-supports-005.xht support/pass.xht
...
pref(), test-pref() and ref-pref() can all be used in a default-preferences statement.
See the reftest documentation for very slightly more details.