After being in Development Hell for...I forgot how many months, Wesnoth-TC, the advanced Team-Colorizer version 1.5.0 is finished and released.
This version introduces a few changes in the output files' naming scheme, and adds support for palette switches a la ~PAL() image function from Wesnoth's engine (which I implemented, by the way). The old Makefile is gone and there's a normal autotools build system in place, which seems to work well on Linux systems, as well as Mac OS X and even Mingw32-based cross compiling environments!
Thanks to this improvement and a small help from loonycyborg, Wesnoth-TC also has a experimental “official” Win32 build this time, which I have successfully tested on Windows XP SP3, and Debian Squeeze using WINE. It was cross-compiled using the Mingw32 environment packages from Debian. There are no important functionality differences between the Win32 build and normal Unix-based builds. It remains a command-line application as it's supposed to be.
Windows (Win32) binary:wesnoth-tc-1.5.0-win32.zip (373.3 KiB)
(SHA1: 311adf05b3855bf74eb782fe020992e61aac80eb)
In an ideal world, the availability of a CLI Win32 binary would raise interest on this tool and someone would volunteer for working with me on a GUI front-end. That's very unlikely to happen in the real world, so I'm researching Qt4 and messing around with some ideas for a cross-platform front-end based on penguin's Mac OS X applet.
Instructions
Both distributions come with a README file, and the source code distro includes a INSTALL file with detailed instructions on configuring, compiling and installing wesnoth-tc. The Win32 binary distribution doesn't require any installation besides unpacking it into an appropriate directory — which you may optionally add to PATH.
EDIT: some people may have problems compiling under certain platforms or configurations because the configure script in this version enables strict mode by default, treating compiler warnings as errors. To solve this, pass the --disable-strict option to configure. Git master already solves this problem, plus a particular warning reported on Ubuntu 9.10.
Behind the scenes
As I mentioned before, I tried using gd for PNG manipulation but ultimately discovered that its limitations weren't nice for wesnoth-tc's particular case. I rewrote the PNG reading and writing code again afterwards, using the SDL_image library as a model instead.
The Win32 build doesn't have support for libpng exception handling (via longjmp/setjmp) because the compiler complains about it for some reason:
warning: variable ‘height’ might be clobbered by ‘longjmp’ or ‘vfork’
This shouldn't be a big deal since libpng won't complain unless someone feeds a corrupted or invalid PNG file to it. (EDIT: figured out the cause with some help, but haven't fixed it in Git master yet.) The other functionality change is the lack of support for checking file access permissions considering effective user and group ids, etc.
Supposedly, this release should work properly on PowerPC and other big-endian platforms. However, I don't have such a machine and therefore can't test.
My own software projects tend to be very much like pets to me. I take care of them, carry them with me anywhere if it's possible, I feel horribly sad when something bad happens to them and, even when I go mad at them for something, in a few hours we are together like a neat happy family again.
Invasion from the Unknown has been theWesnoth add-on project of mine since around September 10th 2007 and it has evolved throughout time and endured 3 mainline development cycles introducing drastic game engine changes, receiving little automated help from the likes of wmllint. Instead, it has been kept on shape by me and a few people who have helped with the huge maintenance burden that this epic-length campaign is.
With an initial goal of 30 playable scenarios, 29 of them were made at first and I later shrank the campaign to approx. 26 stages per suggestions from various people including ESR, and split it into two halves of roughly 13 scenarios each. IftU outlasted two other projects of mine which finally rotted among my hard disk backups for good — one after 5 years of development and little progress, and the other after three months of development and decent progress. It also directly or indirectly created or inspired the following Wesnoth-related projects:
For a few years since ESR established the new mainline conventions in Wesnoth for tagging and adding new contributed music tracks, I've been in charge of helping with the process of importing tracks since ESR doesn't seem to be available for this anymore, and our “Lords” of Music never get familiarized with SVN to do it themselves. Normally I'm asked to do this whenever I'm available, but for some reason, another developer did it this time on March 13th — that is, two days ago — and I didn't find it out until today.
This minor thing reminded me of a substantial problem with campaign music in Wesnoth in regards to originality, which I've been perceiving for a few months already.
Since I'm fed up with libpng's complexity (which borders on obscurity), I asked for recommendations on libraries for reading, editing and writing PNG files in #defocus, freenode's social channel. Someone mentioned gd, which I mostly knew for being a dependency of PHP-based software that deals with graphics, so I decided to give it a try.
It turns out to be incredibly easy to learn and use — I quickly started the gd_writer branch for the Wesnoth Team Colorizer tool and nuked hundreds of convoluted lines of libpng calls to replace them with short, concise and clean code.
However, I then noticed something in the (thorough and easy to read) documentation that spelled trouble for my usage:
gd retains only 8 bits of resolution for each of the red, green and blue channels, and only 7 bits of resolution for the alpha channel. The former restriction affects only a handful of very rare 48-bit color and 16-bit grayscale PNG images. The second restriction affects all semitransparent PNG images, but the difference is essentially invisible to the eye. 7 bits of alpha channel resolution is, in practice, quite a lot.
No justification is provided in the documentation as far as I could find. However, the main header file (gd.h, which is perfectly readable unlike libpng's png.h) has this precious missing bit explaining the reasoning here:
If 'truecolor' [in a gdImage struct] is set true, the image is truecolor; pixels are represented by integers, which must be 32 bits wide or more.
True colors are repsented (sic) as follows:
ARGB
Where 'A' (alpha channel) occupies only the LOWER 7 BITS of the MSB. This very small loss of alpha channel resolution allows gd 2.x to keep backwards compatibility by allowing signed integers to be used to represent colors, and negative numbers to represent special cases, just as in gd 1.x.
Ah, good old backwards compatibility, always biting the programmer in the ass after a while — Windows 9x being a major commercial example of backwards compatibility gone mad.
While I agree that the difference might not be noticeable for the human eye, this still means that wesnoth-tc/gd produces output that differs from the original images in more than just team colors. It'd be often silly to use team-colored semi-transparent in Wesnoth unit sprites, but team color and palette switches can, thanks to the flexibility of the image path functions mechanism, be applied on virtually any kind of image that SDL_image can read into a surface pixmap, including things such as transparent haloes for visual effects.
Experimenting with wesnoth-tc/gd, a linear sequence of pixels with these alpha channel values:
255, 254, 253, 252, 251, 250, 249, 258 [...]
...turns into this in the application's output:
255, 255, 253, 253, 251, 251, 249, 249 [...]
What happens here is that values below 0x7F — which is 127 in base 10 and 1111111 in base 2, and as you can see, the maximum positive integer that can be represented with 7 bits — can only be even, and values above are odd. There's no 0x7F.
I could just ignore this issue and merge the gd_writer branch, but no, I think I'll just try yet another library. It's a pity because this experiment had created the possibility of making a web interface for recoloring/team-coloring Wesnoth artwork, but I believe this kind of loss of information isn't any good for our purposes, even if it should be virtually unnoticeable at the standard 72x72 sprite scale. At least unit shadows shouldn't be affected by this limitation since their alpha value is supposed to be 153, which would remain unchanged under this scheme.
And so, Wesnoth-TC 1.5 stays in Development Hell for now, and the search for a simple enough libpng wrapper continues.
I've been working on cleaning up the code for Wesnoth-TC's PNG reader/writer functionality and so far I've only partially succeeded. I cleaned the code up, sure, but I also messed up somewhere and mixed up something breaking Wesnoth-TC.
The new_writer branch is already in the Git repository but users will definitely want to use master instead. I haven't yet decided if new_writer will ever be merged back into master or I'll just throw it away.
But at this point I'm tempted to give up and try a different library for reading and writing PNG files — if I keep trying to understand how to use libpng correctly (if that's even possible), Wesnoth-TC 1.5 will never be finished.
More than one year ago I commented on the consequences of interrupted commit transactions with the Subversion version control system. Back then, SVN was the only VCS I was familiarized with, but nowadays I also have a basic grasp of Git for local and remote repository management.
The thing is, SVN is pretty simple and easy to learn for novice users — which is one of the reasons I haven't decided yet, as the founder admin of the Wesnoth-UMC-Dev Project, to switch to Git. A distributed version control system such as Git or Mercurial are not “better” than SVN, just like Linux cannot be “better” than Windows — they are completely different models for both users and site admins, and switching your version control system isn't as easy as switching from KDE to GNOME as your desktop environment or buying a new printer, especially when you have lots of users and the model conversion isn't easily reversible.
But let's not forget that there's more to SVN, or any other revision tracking system than just the philosophy and the model behind. There is an official client which ships in major Linux distributions such as Debian GNU/Linux, Ubuntu and openSUSE, and which also has shared library code used by third-party GUI front-ends such as kdesvn, or other SVN clients such as the git-svn infrastructure.
I have not seen the code, and I believe I do not want to see it with my eyes, but SVN's network code seems to be crap.
The issue mentioned in my second blog post remains the same after several versions of the vanilla SVN client. Then there are other issues that have been here for a long time, and an issue I only discovered some days ago:
It is possible in middle of a networked transaction (commit, update) for the Subversion client process to get stuck if a network error occurs. Subversion normally traps SIGTERMs (and apparently SIGHUPs and SIGINTs too) to perform cleanup routines after such interruptions. However, when it gets stuck, the SIGTERM handler becomes useless and the client ignores the signal forever. This means that SVN can get stuck and sit idle on the terminal (most likely waiting for data from the remote host) until it a SIGKILL is sent to force the destruction of its process. Since SIGHUPs are also trapped, killing the terminal leaves a hidden, waiting SVN process. In other words, you can get a dead SVN client running for months if your power source is stable enough. Wonderful.
Clients that terminate abnormally (SIGKILL and such) may leave random crap hidden in your SVN checkout's control directories that are normally cleaned up by the SIGTERM handler. While this is often inoffensive and svn cleanup or svn cleanup .. can handle it all, there are times when this is not enough and SVN gets confused for missing/extra files or directory metadata and refuses to update or cleanup a path. In such cases, removing the path and its contents and re-checking it out with svn update (or, if it was the whole working dir, svn checkout) is necessary.
There seems to be a lot of overhead in the SVN subprotocol on any transport class, be it http, https, svn or svn+ssh. Commits containing simple changes to file/dir object properties can take as long as a regular commit diff when they should probably contain less data (if they contain as much data, then...?!). This is very noticeable on low-bandwidth connections for me. In comparison, a SVN commit of about 20 property changes can take longer than a Git push through SSH of about 10 large commits introducing whole new files.
Then there are some odd things with the SVN client library (libsvn), specifically the Perl bindings, namely the issues I mentioned at the start of this month, that leave me very disappointed at this version control system, rumored to be better than CVS (which I haven't ever used...imagine!). The Debian version in Squeeze, and possibly Lenny or upstream too, has a really nasty bug which caused a massive memory leak with Wesnoth-UMC-Dev's umcpropfix tool when I ran it to set properties on a version of Extended Era for Wesnoth, manipulating over 1300 files on multiple dirs. The parent Perl interpreter process allocated over 2 GB of overall virtual memory, making Linux page most memory out to swap, thus hurting performance.
The cause? Pool management. libsvn's Perl bindings are supposed to do automatic memory management unless the client wants to do their own pool management with the library's facilities, but that somehow causes the aforementioned leak instead. The solution turns out to be doing custom pool management by allocating a new pool for every libsvn call, and forgetting the old one. r6567 in the Wesnoth-UMC-Dev repository applies this workaround for our SVN property setting tool.
Honestly, I'm tired of SVN. I use git-svn wherever I can but this isn't a magic solution for the crappy design of SVN's innards. git-svn can skip upstream commits if the connection interrupts during a fetch operation, and it forgets about local commits during pushes (git svn dcommit) after it has sent the first commit and fetched missing ones, which can cause loss of commits and history if the connection to the remote host breaks at that point or git-svn exits in any other fashion.
git-svn doesn't replace SVN's network code either (it uses it instead), so it's still subject to the perceived overhead, but at least it doesn't get stuck forever ignoring SIGTERMs.
What Wesnoth-UMC-Dev needs at the moment is a distributed (yes) version control system that's as more user-friendly as than SVN and has a nice, well documented Windows front-end that's easy to setup, learn and understand. If we can't find that, I'll continue complaining about SVN at any time and on every place where I see fit, here or in IRC.
Long ago, I wrote a little Bash script — set-properties — for the Wesnoth-UMC-Dev project, to ensure the correctness of SVN properties such as svn:keywords and svn:executable on files. It was pretty simple:
#! /bin/sh
# Set properties on PNG files
for f in $(find -iname *.png); do
svn propdel svn:executable $f
svn propset svn:mime-type image/png $f
done
# Set properties on Ogg files
for f in $(find -iname *.ogg); do
svn propdel svn:executable $f
svn propset svn:mime-type audio/x-vorbis
done
# Set properties on PCM files
for f in $(find -iname *.wav); do
svn propdel svn:executable $f
svn propset svn:mime-type audio/x-wav
done
# Set properties on JPEG files
for f in $(find -iname *.jpg); do
svn propdel svn:executable $f
svn propset svn:mime-type image/jpeg $f
done
for f in $(find -iname *.jpe); do
svn propdel svn:executable $f
svn propset svn:mime-type image/jpeg $f
done
for f in $(find -iname *.jpeg); do
svn propdel svn:executable $f
svn propset svn:mime-type image/jpeg $f
done
# Set properties on CFG file
for f in $(find -iname *.cfg); do
svn propdel svn:executable $f
done
# Set properties on scripts
for f in $(find -iname *.sh); do
svn propset svn:executable '*' $f
done
for f in $(find -iname *.cmd); do
svn propset svn:executable '*' $f
done
for f in $(find -iname *.bat); do
svn propset svn:executable '*' $f
done
for f in $(find -iname *.py); do
svn propset svn:executable '*' $f
done
Some time after I learned Perl with my work on Shikadibot 0314, I rewrote that script in Perl to arrange the “ideal” property values in a neat table (hash), check current properties instead of blindly overwriting them in the working copy, and cover plenty of other file types. It also gained a blinking progress bar to display the search progress for some reason.
To have an idea of how known file types are defined in the source, let's take a look at these bits:
# properties set to the empty string '' (except svn:executable) are left unchanged;
#
my %proptab = (
cfg => [FALSE, '', 'native', ''],
ign => [FALSE, '', 'native', ''],
"map" => [FALSE, '', 'native', ''],
# [...]
pl => [TRUE, '', 'native', 'Author Date Id Revision'],
# [...]
gif => [FALSE, 'image/gif', '', ''],
png => [FALSE, 'image/png', '', ''],
# [...]
xcf => [FALSE, '', '', ''],
# [...]
);
The table we have used since then (around Sept. 2008) has always contained more than 10 extensions with their minimum required property sets. As of this writing, it covers approximately 57 file types. Keep this on mind.
It would be overkill to fork-exec find processes to discover paths that could require SVN property changes, right? So, instead, I used find2perl to generate File::Find client code to embed it into set-properties. So far, so good. But how about running that code (scalar keys %proptab) times (e.g. number-of-extensions-times) anyway? Overkill?
No! It's plain stupid. But definitely less stupid than what you are about to read.
I apparently decided, for some reason, that any matching paths in each cycle should be added to a plain scalar (a text string to be exact) separating individual paths with newlines, of all things. Then, another cycle is performed at the end, (scalar keys %proptab) times again, to put each array of paths matching a certain extension into another hash, then iterating over the newly inserted hash element (array of paths) checking and fixing SVN properties in the same cycle.
Very roughly summarized as the following pseudocode:
FOREACH extension FROM file_extensions
FIND IN ./ AS file
; all while displaying a cute blinking bar!
IF file MATCHES extension
APPEND file TO file_list_string
END IF
END FIND
END FOREACH
FOREACH extension FROM file_extensions
; split 'file_list_string' every newline
FOREACH path FROM (SPLIT /\n/ file_list_string)
INSERT path IN file_index[extension]
END FOREACH
FOREACH path FROM file_index[extension]
fix_properties( extension, path )
END FOREACH
END FOREACH
Careful readers will quickly realize that something is horribly wrong with this pseudocode. I wish I was making it up. This algorithm has actually been in use by the Wesnoth-UMC-Dev set-properties script for one year and 4 months! I must have been on something when I wrote this shit. I've honestly never seen any program so awful as this in my life. Not even build-external-archive.sh (a.k.a. “Scrappy”) can compete with this abomination.
So, yesterday, I took a look at set-properties after noticing how much CPU time it ate working with very scarcely populated directories of the project — and I was blaming the usage of backticks (svn propset foo bar baz and such) as a possible cause of overhead. Then I slowly realized what I had written. There isn't any emoticon here for the expression in my face at that moment.
Thus, set-properties got rewritten with a much cleaner and simpler algorithm:
FOREACH file FROM (FIND IN ./)
; no more useless cute blinking bar!
FOREACH extension FROM file_extensions
IF file MATCHES extension
fix_properties( extension, path )
END IF
END FOREACH
END FOREACH
So yeah. 🙁
But wait, there's more! While optimizing replacing the script I also replaced svn foobar backtick code with invocations of libsvn, via the SVN::Client module. This worked very well at the end, but I discovered a few things in the process:
Those SVN::Client methods I used choke on non-absolute path specifications for some reason, causing an assertion failure in libsvn's C back-end and terminating the execution of Perl and the script with a SIGABRT.
Despite the documentation's claims for SVN::Client::url_from_path() returning undef if the specified path is not under version control, it actually causes the module to invoke die and get the client script terminated. Which means that I cannot even check if a file is versioned or not safely (e.g. without resorting to .svn/text-base/<FileName>.svn-base existence checks). What the hell?
Turns out the solution is to wrap SVN::Client method calls in eval blocks and handle whatever crap SVN comes up with. Oh, and make sure all paths are absolute using Cwd::realpath so that libsvn doesn't hit an assertion failure killing us, eval or no eval. How nice.
# No point in working with unversioned files.
my $svn_ret = undef;
eval { $svn_ret = $svn->url_from_path($path) };
if($@) {
# fucking libsvn dies if $path isn't under version control;
# the documentation says it should return undef above instead!
return;
}
With this, I have absolutely lost my faith in Subversion's excellence as a version control system not only as a normal user, but also as a programmer integrating it into my own client applications. And I know I lost my faith as a user after seeing svn sit for two days doing fucking nothing because the damned connection died after 1 minute of running time — and then svn spent the next days ignoring SIGTERMs all the time, no less. It's been like this for so many versions that I'm almost convinced it's intentional.
(Ah, that was relaxing. I should do this more often.)
Wesnoth-UMC-Dev has to continue using SVN because we have several add-on authors using Windows and the few alternatives for using git (I love git, I don't think I could try anything else at the moment) on Windows seem to be rather awkward to install and/or use for the average Non-Computer Person. That's a pity. It's really a pity.
Finally, set-properties got renamed to umcpropfix for a change, to mark its rebirth after I solved the algorithmic mess above last night. It doesn't have a nifty codename like the rest, though, but the new umc-prefixed name is still something to be celebrated now that we are going to have umcdist, umcstat, umcreg and umcbotd. Yes, I know I'm crazy, thank you.
It has become increasingly common for me to come up with a program for an amazing task one day, to rewrite it the next day.
umcdist, the Wesnoth-UMC-Dev Distribution Tool, has been in development hell for a year mostly for this reason; the other reason is that it seems like it will perform worse than build-external-archive.sh a.k.a. “Scrappy” due to an excessive usage of Perl's system function. I cannot make up my mind and choose between performance and maintainability; Espreon knows that build-external-archive.sh is broken, but I can't be bothered to try to understand that unholy abomination again to fix it.
Meanwhile, umcstat (Wesnoth-UMC-Dev Statistics Service) is still a work in progress, but with more emphasis in progress; there's actual code written already, and I've been using freenode's Eir bot framework to test it.<
While Eir could possibly be a nice way to get rid of umcreg's Net::IRC dependency and code, it's actually a C++ program that can be compiled only with GCC 4.4 at minimum, due to at least one C++0x feature used throughout the code: auto. The target machine runs Debian lenny, unlike my laptop (Squeeze), and therefore doesn't have GCC 4.4!
Instead of sticking with Eir, I'm refactoring umcreg's IRC code into a custom Perl-only framework, umcbotd, and making creative use of eval writing an abomination code-named “Naia”, which I'm rewriting again because the first version I wrote, which worked, was very badly designed and ugly. I know it's a problem when I have three classes or modules all depending upon each other's internals.
The goal of umcbotd/Naia is producing a Net::IRC-based abstraction layer for our bots that treats them as “services” with multiple “modules” (not in the Perl sense, though) that can be easily inserted and removed from the system by adding/removing their files. umcreg already runs with a prototype implementation of this mechanism, but it needs to be generalized further before it can be usable with Naia.
Writing our bots could be this simple, if Naia gets completed:
#!/bin/true
CO_SERVICE_COMPONENT('umcreg');
sub ctcp_version_reply () { "Wesnoth-UMC-Dev Registry Service, using " . Naia::version_string() }
my ($parent, $nick, $hostmask, $svaname) = (shift, shift, shift, shift);
if(!@_) {
$parent->notice($nick, "Not enough arguments for \002RAW\002.");
return;
}
$parent->sl(join(' ', @_));
broadcast_to_log("RAW [$hostmask]");
}
... And of course it would still involve lots of ugly stuff under the hood (eval magic), but if done right I shouldn't have to touch it whenever I wanted to add or remove a feature from any of our two services.
At last, umcreg, Wesnoth-UMC-Dev's Registry Service, is finished, deployed and announced in the forums, thus completing the first part of introducing the Registry system to the project.
There were some changes from its original incarnation, but everything turned out pretty well for a bot written from scratch in approximately 6 days by a Perl fanatic with no knowledge of object-oriented Perl — I actually learned some object-oriented Perl while at this and I feel like I can do anything with it now. 😄
I was in a hurry to get this done right before freenode deployed ircd-seven today (yays!), for no particular reason; this resulted in a security feature not working with hyperion-ircd until I introduced a quickie hack that I'll be retiring later today.
I have already registered our current members using some basic, known information about them — even including their join date. Old projects' registration will be a little slow as I need to research the current structure of the repository, of which I lost track a year ago, and retrieve original timestamps. The registry's web interface, provided by the umcreg::Web and Thoria::Web packages, can be found here.
umcreg is already working at the project's admin channel on freenode too. It only obeys the project staff's orders, though, so there's no point in trying to send messages to it.
The next step in implementing the Registry model is writing the Statistics service, codenamed “Listra”, and most likely going to be named umcstat. That will definitively take much longer than umcreg's development. Meanwhile, Espreon is trying to convince me to take care of umcdist (codename “Blackmore”) first.
After some weeks of inactivity, I have finally completed the first arc (not the first episode, though) of After the Storm in the Wesnoth-UMC-Dev SVN repository, comprising 7 scenarios out of planned 11; this means that a 0.3.0 release is coming soon. It was about time!
With this new arc completed, I have introduced new background story elements that could be considered controversial if any mainline purist is actually paying attention to the campaign — that's okay, I never intended IftU or any sequels to be mainlined. I still struggle to keep everything as fuzzy as possible to give a certain degree of flexibility to any content authors who decide to take what is said in IftU and AtS as “canon”. It's harder than it sounds, particularly because it must still be clear enough to allow the plot to progress; so I cannot just throw a bunch of nonsense into the campaign and say “hey, look at this, this is our vague excuse for this pathetic plotline!”.
Now that the characters have a decidedly vague excuse for the plot of the next arc, I face a problem that I knew I'd have to handle sooner or later: artwork.
I am not a good pixel artist, but I don't have any loyal slav- pixel artist that could help me either. And even if I could get one, I'm not completely sure I could describe the concepts I have in mind in plain words to tell them what unit sprites I require. There's also the spoileriffic factor; there's a reason that I removed the original (clumsy) storyboard from the SVN repository and departed from the original plans, most notably by removing a main character and introducing two new sidekicks instead. So, it's up to me to create any pixel art needed to make the campaign work; baseframes are enough for this purpose, although I still wish IftU and AtS' original units had animations.
At least I don't need to write a game engine from scratch too, thanks to Wesnoth's scripting flexibility.