Wesnoth add-on tests and sanity checking
Maintaining Wesnoth add-ons of the size of Invasion from the Unknown and After the Storm isn’t a small task by any means. Over the years, I have had to rely on user feedback to detect critical problems in a release, because testing becomes cumbersome and tedious as the scenario count increases.
My usual release procedure simply involves—at least since I acquired the habit of testing before releasing—running the game, starting each episode of the campaign with the medium difficulty level and making sure the WML preprocessor and parser don’t throw any warnings or errors. Before Wesnoth 1.9.x, the preprocessor didn’t abort when encountering a missing macro or file during a brace substitution, so I had to pay close attention to stderr to ensure nothing is wrong.
The WML preprocessor in Wesnoth 1.10 became more strict, aborting on the aforementioned situations. It was also exposed for command-line usage (for testing or debugging) through the -p or --preprocess switch, also explained in detail under PreprocessorRef in the wiki.
At first I thought that wasn’t very useful beyond diagnosing complicated preprocessor issues, but today I realized I can also do this:
This can be easily accomplished with a simple shell script here embedded in the AtS Makefile. The only major shortcoming is that it doesn’t cover every possible problem because it’s merely running the WML preprocessor, which doesn’t consume and produce WML — all it sees is plain text mixed with some preprocessor directives. The task of reading actual WML (which is potentially found in the preprocessor’s output) is left to the WML parser proper, which creates internal objects in memory corresponding to the internal representation of WML handled by Wesnoth (config class objects).
UPDATE: After investigating the issue further with timotei (who exposed this functionality through --preprocess in the first place), it turns out the preprocessor output with --preprocess is indeed parsed — the real problem is that the preprocessor and parser use different logging facilities, and the former doesn’t even throw errors directly, so a preprocessor-only failure will make the game exit successfully (exit status of zero), while a parser error (potentially induced by a previous preprocessor error) causes a more appropriate non-zero exit status. This and other jarring inconsistencies make add-on test automation rather difficult, to say the least, so things have been simplified in the Makefile as a result.
It would be nice to be able to run the parser unit on the For now, it seems I’ll have to stick to my primitive and inelegant manual method before making new AtS releases, plus the unbelievably clumsy --preprocess output to detect syntax issues like missing end tags or unterminated string literals in the future, as part of a fully automated test suite.wmllint.