Is it over already?

Finally, it's January! The New Year celebrations are mostly over and fading away, and people all around the world are going back to regular business and everything should be back to normal in a few days.

I used to be fond of the Christmas and New Year celebrations as a child as I could spend time with my family and eat delicious food. That is not the case anymore, because, even if I still live with my parents, there's no longer a sense of family here and we only want to throw sharp stuff at each other. There's not much enthusiasm by the end of the year anymore, and phrases such as “Merry Christmas” and “Happy New Year” (in Spanish, though) are truly unheard of in this house. Recent disagreements amongst us indicate that this is not going to be a good year for anyone. To add insult to injury, one of our cats died in a rather tragic and violent fashion on December 22th — it's a tradition here that one or more pets must always die in December. While we have many of them, the first ones to die are those whom we are most attached to.

To mark the actual start of 2010 (as far as the Gregorian calendar is involved, of course), there was a black-out on the area about 6 minutes 7 seconds past midnight, which left us with no Internet or tap water until around 1:50 AM. What a great way to start the first day of the year.

But there's still some hope at the moment. Some days before Xmas, my creativity returned from its long, chaotic journey and my Wesnoth add-on, After the Storm (sequel to Invasion from the Unknown has seen steady progress and two new releases were published in less than two weeks. Keep in mind that this add-on had not seen any public releases for almost a year.

After the last released version of AtS (0.2.1) including 5 of 12 planned scenarios in Episode I, there has been more progress in the Wesnoth-UMC-Dev repository. Just yesterday, I finished the two-part cutscene that is the sixth scenario of Episode I, one of the most important points of the plot's development, in which two forest elves finally make contact with the desert/Quenoth elves.

I won't be able to release AtS 0.2.2 or 0.3.0 until scenario 7 and the next cutscene (appropriately named “Resolutions”) are finished, since I'd be teasing the players otherwise. However, those who are really interested on it can always check AtS out from the repository's trunk into their <wesnoth preferences dir>/data/add-ons dir and play using the latest development version of Wesnoth:

svn co https://wesnoth-umc-dev.svn.sourceforge.net/svnroot/wesnoth-umc-dev/trunk/After_the_Storm

It's really exciting to work with several plot elements from quartex's Under the Burning Suns in new, creative manners — kind of like Fanfic production taken to a new level using the power of the GNU General Public License (version 2 or later!). Nevertheless, I am fairly sure he deliberately left much details unresolved in the original campaign, and that he'd fry us (Espreon, AI0867 and me) alive if he found out what we are doing with his story.

One week before Xmas, the Wesnoth.org forums saw another upgrade on which Turuk and I worked hard and quickly to improve forum usability by not only upgrading the codebase to phpBB 3.0.6, but also tweaking the templates, adding modifications and a couple of new forum styles to take advantage of the new features implemented by the phpBB devs in this iteration of their software. The main points were highlighted in this forum post (originally a Global Announcement).

This year should also bring us a new stable series (1.8) of the Battle for Wesnoth game itself. There are currently some problems delaying the first Release Candidate and getting us flooded with generic beta releases, but the developers in charge of them will (hopefully!) find a solution so we can get 1.8 released and trunk “thawed” soon, to work on new features and allow new code contributors to join the project. As for me, I can't wait for the new stable series — development series players seem to be scarce and the new versions of IftU and AtS are receiving little feedback on the forums because of this! I suppose Multiplayer content authors are similarly eager to get more fresh meat to play their add-ons.

I also recently talked about how I can finally suspend my laptop to RAM using Linux, and run some basic OpenGL-based software without crashing or destroying anything. That's something I didn't expect to be able to do in the near future, so the Mesa, libdrm and X.org radeon driver developers have my thanks for improving the Linux experience of those unfortunate enough to own onboard ATI graphics controllers!

In summary, as usual, a new year brings good and bad news. I guess it's up to us to take what's good and fix what's wrong. So, anyway (although I guess it's pretty much unnecessary at this point): happy New Year and have fun!

Annoying spambot behavior

While checking the Wesnoth.org forums ensuring that everything is working properly, I occasionally find spambots who have managed to make it past our hacky anti-bot registration measures, probably with help from humans. Most of these spambots go into a hit-and-run attack leaving one spammy post in some random place of the forums — sometimes in existing threads, sometimes starting new threads themselves.

Two nights ago (November 30th, 4:45 AM), some guy came into our Off-topic section asking this:

Post subject: Linux

How long does it take to become a Linux Administrator? Also is 1 year of linux study enough to get a job as System linux administrator? What exactly do I need to learn to achieve that goal?

It was his first post, and there were no signature or links whatsoever. He joined the forums two days before that (November 28th, 8:08 AM). You may note that the message above is not in proper English, but there are many underage or foreign users in the board. Nonetheless, some regulars and I replied to this guy's message in good faith.

Spam screenshot

On the next day, December 1st 8:36 AM, “he” edits his message adding spammy links after a fake signature separator similar to the phpBB 3 subsilver2 template's, except a bit longer. The post spent approximately 14 hours as search-engine bait because no other Wesnoth developers noticed the cheat (or read the Off-Topic forum, I guess) and I was away for the whole day until I went in and found this surprise after a third look at the thread.

Fascinating. 😐

In one occasion I had one of these drones join the forums and make their first spam post after 6 weeks (and shortly get kicked out of the house). I guess it must have been collecting posts and feeding them to a learning engine in order to enhance future spam posts by fitting them into the board's context for the time being. Not that its own attempt at posting succeeded.

The problem is that they sometimes do manage to produce reasonable posts — as long as you use a very flexible definition of “reasonable”, which is more or less required around the Wesnoth.org forums these days especially since there are real people who join the board and never post anything constructive or interesting and roam just the Off-Topic forum asking weird, unimportant questions and answering other weird, unimportant questions with bogus, misleading or uninformed answers or opinions (no, not pointing at anyone in particular here). On the other hand, it's still possible that those weird guys or gals will eventually drop their masks and reveal their true identities as spambots. Or maybe someone else will remove it for them.

libpng's complexity

libpng is the reference implementation of the PNG file format, which has become quite popular during the last decade thanks to the benefits of lossless compression coupled with multiple storage formats — including indexed pixmaps, RGB pixmaps with optional alpha channel, or 256 grayscale colors, adding Adam7 interlacing as an option to allow displaying incomplete drafts of images before reading all the pixmap data in the stream. It also allows for some metadata such as text comments and internal timestamps, or even unknown chunks of (meta)data, allowing it to become a container format of sorts, most infamously abused by Mozilla Firefox to implement animated PNGs through a slightly modified version of the library — because lossless RGB pixmaps with alpha transparency are so cool compared to indexed pixmaps with binary opacity...

As cool as this sounds, libpng by itself is a trap for the unexperienced and/or impatient programmer, of which I fall into the latter category.

The API allows for many different applications, resulting in a convoluted interface and some poor or missing documentation for it. Reading the manual and examples included with it does not help a lot; when skimming through the header one can find some undocumented interfaces. It's a hell of flexibility that makes me feel lost in middle of a big forest without a map.

I foolishly wrote Wesnoth-TC without knowing any of this beforehand, and ended up creating a monster known as version 1.0, which is a mess of quickly crafted ugly code that breaks under big-endian platforms and leaves huge memory leaks when running on anything. I could have used a different library but I didn't have more documentation than libpng's at the time. Now that I'm preparing to release Wesnoth-TC 1.5.0 and have learned to keep away from this library, I'm turning my attention to wrapper libraries that should be hopefully much easier to work with if I just want to read and write ARGB streams from/to PNG files.

But Wesnoth-TC 1.5.0 will still use libpng, although I'll rewrite the code that uses it. Why? Because this time, it's personal.

The smallest bugs are the nastiest

Preparing the release of a fixed demo revision for my Wesnoth add-on After the Storm, which is the immediate sequel to Invasion from the Unknown, I ran into a rather nasty bug in one of the scripts I made for easy creation of .tar.bz2 packages for the Wesnoth-UMC-Dev Project, build-external-archive.sh.

The script in question is one of my earliest experiments with bash scripting, and as such suffers from the quirks of a novice programmer trying to get work done fast. It's some sort of crapload of hacks that just ‘‘works’’, somehow. I have an odd habit of tempting fate and using my earliest programming experiments in production environments — and this includes Shikadibot and Soradoc, which is the PHP-based layout code generator used both here and at the Wesnoth-UMC-Dev website.

The only reasons for not killing it are the following features offered to the selected few who know how to use it:

  • It reliably produces a .tar.bz2 archive for add-on distributions on regular websites — including SF.net — provided a few simple arguments.
  • It roughly* conforms to the Wesnoth .ign file syntax, although it doesn't provide the same defaults as the game engine.
  • It knows how to cleanly handle SVN check-outs.
  • There's built-in, automatic support for XDelta patch generation, provided the xdelta tool is installed and in PATH.
  • Provides MD5 and SHA1 digests of produced packages and writes them to .MD5 and .SHA1 files for distribution.
  • No suitable replacement has been provided yet.

(* Espreon pointed out on the #wesnoth-umc-dev channel at chat.freenode.net, that `build-external-archive.sh` may match .ign patterns only in the add-on's root directory, and not in its subdirectories. This, of course, a bug.)

build-external-archive.sh has got a few inoffensive bugs in the past, perhaps the greatest of which was a set of dependencies upon Linux distribution-specific conventions (why, hello there, openSUSE!). However, yesterday I hit the nastiest, yet unseen bug.

Since I usually use actual SVN tag check-outs for creating the packages, or get someone else to do it for me, I had not checked the interactions of this script with an input directory that is actually a symbolic link pointing to a directory containing files or paths matched by a _server.ign file. This tool needs to copy the input directory to a separate temporary directory in order to physically remove the files (actually copies) that match the aforementioned .ign patterns, and then proceed to run tar to wrap the modified temp directory into a nifty bzip2 tarball.

And how was it copying the input directory to /tmp when the input is not a SVN check-out? Without going into much detail, it is similar to this:

cp -a ${input_directory} /tmp/${output_file.fakeroot}.dir/${addon_name}

The -a switch to GNU coreutils' cp is equivalent to -dpR, which roughly means ‘‘be recursive, preserve timestamps, access masks and never dereference symlinks and preserve them!’’ So you can start to imagine why this would become a huge problem when the $input_directory is a symlink.

Such a small overlooked mistake ended up destroying the git-svn tree contained within my add-on's directory, which was indeed specified as a symlink in ~/.wesnoth-1.7/data/add-ons/After_the_Storm pointing to ~/src/wesnoth-umc-dev/git/trunk/After_the_Storm. The symlink itself was copied to the temporary directory and the removal of unwanted files took place on the original location instead of a temporary tree. This gave me that disturbing feeling of having a bucket with cold water dropped on my head whenever I accidentally remove files I wasn't supposed to remove. Thankfully, I had committed all my changes to the upstream SVN repository and only had to regenerate a git-svn tree to continue working after making the release using a fallback SVN check-out.

The correct pseudo-code, forcing symlinks to be dereferenced and timestamps, file modes and ownerships to be preserved, is actually:

cp -LRp ${input_directory} /tmp/${output_file.fakeroot}.dir/${addon_name}

As you can see, it's perfectly easy to screw up your own files with a technically small mistake in a command line. However, in this case some idiot (me!) didn't read the manual correctly and didn't notice that the -a switch had additional effects, and that's a big logical mistake. Fortunately, only Espreon, AI0867 and I rely on this script, as far as I know.

This tool is going to be eventually replaced by umcdist (codenamed Blackwater), which is written in manageable Perl, provides the same functionality plus archive signing with OpenPGP, and would be done in a few days if I actually concentrated on it — don't take me wrong, it exists and there's code written for it, but it still needs to be completed. Whereas build-external-archive.sh is a deliberately undocumented IftU-specific tool that was later inherited by the wesnoth-umc-dev project, umcdist is targeted to the project as a whole, it will have proper online and built-in documentation and it'll be readable by more than one person to allow easier bug-fixing and maintenance.

Half-assed commits

During my work on the Coordinated Wesnoth User-Made Content Development Project (which we dub "wesnoth-umc-dev" for short), I came up with an interesting concept related to Subversion's standard workflow. Half-assed commits are revision commits to the Subversion repository that are not completed due to the subversion client (or server!) process dying unexpectedly, usually due to anything but a SIGTERM.

The obvious symptom of a half-assed commit in your local file system is a bunch of 'L' flags in the svn st command output. These can be removed with svn cleanup. So, most half-assed commits are harmless to you. However, according to the (holy) Subversion Book, it may leave garbage, half-assed transactions in the repository. These are not viewable to anyone but the repository admin of course, and should not harm anyone provided the filesystem on which it resides does not run out of space.

😐 Last afternoon I ran into a more harmful and painful sort of half-assed commit. I renamed some files in my working copy, invoked svn ci, and my crappy Wireless LAN connection burped just when it was about to update the working copy with the changes introduced to the repository:

Transmitting file data ...svn: Commit failed (details follow):
svn: MERGE request failed on '/svnroot/wesnoth-umc-dev/trunk/Invasion_from_the_Unknown'
svn: MERGE of '/svnroot/wesnoth-umc-dev/trunk/Invasion_from_the_Unknown': Could not read status line: Connection reset by peer
(https://wesnoth-umc-dev.svn.sourceforge.net)
svn: Your commit message was left in a temporary file:
svn: '/home/shadowm/src/wesnoth-umc-dev/trunk/Invasion_from_the_Unknown/svn-commit.2.tmp'

Unsurprisingly, I was left with my files in an awful state that caused local conflicts with the repository. That is, next svn update failed because the commit above was successful for the server, leaving the renamed files in the repository. SVN just didn't like that at my end, because I had those renamed files already in the working copy as result of the svn move result I just (half-ass) commited.

Thanks for nothing SVN! Seriously, the protocol should have the server request for a final confirmation from the client to check-in the transaction after its changes have been merged in the client's working copy. Or the inverse: have the client react in a smarter fashion to these situations that people like me often run into.