Since I'm fed up with libpng's complexity (which borders on obscurity), I asked for recommendations on libraries for reading, editing and writing PNG files in #defocus, freenode's social channel. Someone mentioned gd, which I mostly knew for being a dependency of PHP-based software that deals with graphics, so I decided to give it a try.
It turns out to be incredibly easy to learn and use — I quickly started the gd_writer branch for the Wesnoth Team Colorizer tool and nuked hundreds of convoluted lines of libpng calls to replace them with short, concise and clean code.
However, I then noticed something in the (thorough and easy to read) documentation that spelled trouble for my usage:
gd retains only 8 bits of resolution for each of the red, green and blue channels, and only 7 bits of resolution for the alpha channel. The former restriction affects only a handful of very rare 48-bit color and 16-bit grayscale PNG images. The second restriction affects all semitransparent PNG images, but the difference is essentially invisible to the eye. 7 bits of alpha channel resolution is, in practice, quite a lot.
No justification is provided in the documentation as far as I could find. However, the main header file (gd.h, which is perfectly readable unlike libpng's png.h) has this precious missing bit explaining the reasoning here:
If 'truecolor' [in a gdImage struct] is set true, the image is truecolor; pixels are represented by integers, which must be 32 bits wide or more.
True colors are repsented (sic) as follows:
ARGB
Where 'A' (alpha channel) occupies only the LOWER 7 BITS of the MSB. This very small loss of alpha channel resolution allows gd 2.x to keep backwards compatibility by allowing signed integers to be used to represent colors, and negative numbers to represent special cases, just as in gd 1.x.
Ah, good old backwards compatibility, always biting the programmer in the ass after a while — Windows 9x being a major commercial example of backwards compatibility gone mad.
While I agree that the difference might not be noticeable for the human eye, this still means that wesnoth-tc/gd produces output that differs from the original images in more than just team colors. It'd be often silly to use team-colored semi-transparent in Wesnoth unit sprites, but team color and palette switches can, thanks to the flexibility of the image path functions mechanism, be applied on virtually any kind of image that SDL_image can read into a surface pixmap, including things such as transparent haloes for visual effects.
Experimenting with wesnoth-tc/gd, a linear sequence of pixels with these alpha channel values:
255, 254, 253, 252, 251, 250, 249, 258 [...]
...turns into this in the application's output:
255, 255, 253, 253, 251, 251, 249, 249 [...]
What happens here is that values below 0x7F — which is 127 in base 10 and 1111111 in base 2, and as you can see, the maximum positive integer that can be represented with 7 bits — can only be even, and values above are odd. There's no 0x7F.
I could just ignore this issue and merge the gd_writer branch, but no, I think I'll just try yet another library. It's a pity because this experiment had created the possibility of making a web interface for recoloring/team-coloring Wesnoth artwork, but I believe this kind of loss of information isn't any good for our purposes, even if it should be virtually unnoticeable at the standard 72x72 sprite scale. At least unit shadows shouldn't be affected by this limitation since their alpha value is supposed to be 153, which would remain unchanged under this scheme.
And so, Wesnoth-TC 1.5 stays in Development Hell for now, and the search for a simple enough libpng wrapper continues.
I've been working on cleaning up the code for Wesnoth-TC's PNG reader/writer functionality and so far I've only partially succeeded. I cleaned the code up, sure, but I also messed up somewhere and mixed up something breaking Wesnoth-TC.
The new_writer branch is already in the Git repository but users will definitely want to use master instead. I haven't yet decided if new_writer will ever be merged back into master or I'll just throw it away.
But at this point I'm tempted to give up and try a different library for reading and writing PNG files — if I keep trying to understand how to use libpng correctly (if that's even possible), Wesnoth-TC 1.5 will never be finished.
It's been almost 3 days since the earthquake in Chile which affected many regions of the central-south areas, including the region of Santiago. While things are mostly normal here — besides the many closed stores, the many damaged buildings, unusable highway infrastructure in some places, some fallen buildings, zones without electric power, unstable mobile phone networks and a bit more than 30 dead in the region, the disaster here pales in comparison to what can be seen following the coastal line to the south near the region of Maule and Bío-Bío. There's no way I could properly summarize what happened there, so you'd better use Google if you are curious. Many locations look like ruined, flooded, destroyed battlefields.
And yet, opportunists have pillaged around 16 supermarkets in our region so far; some succeeded, but most failed. There was not enough damage on this region to justify such vandalism. Naturally, Concepción and many other locations in the south are also being constantly vandalized, enough to justify passing the control of two regions over to the army to try to keep the order. They haven't completely succeeded, and the vandals have started fires, given false alarms of tsunamis and destroyed or damaged private and public property.
The aftershocks have been scarce today and they've mostly faded out from our point of view in the Santiago region — that is to say, they haven't stopped in Maule and Bío-Bío and it doesn't let those people rest or sleep for much time. A few stronger aftershocks have reached us as well. Fortunately, they are not anything like the original earthquake.
However, they seem not to stop for me. I've been feeling dizzy since the moment I got out of the house during the earthquake and, no matter if I'm on my bed or sitting on a chair or on the toilet, everything feels the same as in the aftershocks, like a big boat on the sea that never stays still. It's very annoying but I still work on stuff and chat on IRC despite of this.
As sad as this is, and as grim as everything may seem, Santiago is mostly alright and we must move on and continue our lives and help however it is possible. But an earthquake with epicenter near our city isn't a terribly far-fetched possibility since we live next to several (relatively inactive) volcanoes, so let's not relax too much either.
More than one year ago I commented on the consequences of interrupted commit transactions with the Subversion version control system. Back then, SVN was the only VCS I was familiarized with, but nowadays I also have a basic grasp of Git for local and remote repository management.
The thing is, SVN is pretty simple and easy to learn for novice users — which is one of the reasons I haven't decided yet, as the founder admin of the Wesnoth-UMC-Dev Project, to switch to Git. A distributed version control system such as Git or Mercurial are not “better” than SVN, just like Linux cannot be “better” than Windows — they are completely different models for both users and site admins, and switching your version control system isn't as easy as switching from KDE to GNOME as your desktop environment or buying a new printer, especially when you have lots of users and the model conversion isn't easily reversible.
But let's not forget that there's more to SVN, or any other revision tracking system than just the philosophy and the model behind. There is an official client which ships in major Linux distributions such as Debian GNU/Linux, Ubuntu and openSUSE, and which also has shared library code used by third-party GUI front-ends such as kdesvn, or other SVN clients such as the git-svn infrastructure.
I have not seen the code, and I believe I do not want to see it with my eyes, but SVN's network code seems to be crap.
The issue mentioned in my second blog post remains the same after several versions of the vanilla SVN client. Then there are other issues that have been here for a long time, and an issue I only discovered some days ago:
It is possible in middle of a networked transaction (commit, update) for the Subversion client process to get stuck if a network error occurs. Subversion normally traps SIGTERMs (and apparently SIGHUPs and SIGINTs too) to perform cleanup routines after such interruptions. However, when it gets stuck, the SIGTERM handler becomes useless and the client ignores the signal forever. This means that SVN can get stuck and sit idle on the terminal (most likely waiting for data from the remote host) until it a SIGKILL is sent to force the destruction of its process. Since SIGHUPs are also trapped, killing the terminal leaves a hidden, waiting SVN process. In other words, you can get a dead SVN client running for months if your power source is stable enough. Wonderful.
Clients that terminate abnormally (SIGKILL and such) may leave random crap hidden in your SVN checkout's control directories that are normally cleaned up by the SIGTERM handler. While this is often inoffensive and svn cleanup or svn cleanup .. can handle it all, there are times when this is not enough and SVN gets confused for missing/extra files or directory metadata and refuses to update or cleanup a path. In such cases, removing the path and its contents and re-checking it out with svn update (or, if it was the whole working dir, svn checkout) is necessary.
There seems to be a lot of overhead in the SVN subprotocol on any transport class, be it http, https, svn or svn+ssh. Commits containing simple changes to file/dir object properties can take as long as a regular commit diff when they should probably contain less data (if they contain as much data, then...?!). This is very noticeable on low-bandwidth connections for me. In comparison, a SVN commit of about 20 property changes can take longer than a Git push through SSH of about 10 large commits introducing whole new files.
Then there are some odd things with the SVN client library (libsvn), specifically the Perl bindings, namely the issues I mentioned at the start of this month, that leave me very disappointed at this version control system, rumored to be better than CVS (which I haven't ever used...imagine!). The Debian version in Squeeze, and possibly Lenny or upstream too, has a really nasty bug which caused a massive memory leak with Wesnoth-UMC-Dev's umcpropfix tool when I ran it to set properties on a version of Extended Era for Wesnoth, manipulating over 1300 files on multiple dirs. The parent Perl interpreter process allocated over 2 GB of overall virtual memory, making Linux page most memory out to swap, thus hurting performance.
The cause? Pool management. libsvn's Perl bindings are supposed to do automatic memory management unless the client wants to do their own pool management with the library's facilities, but that somehow causes the aforementioned leak instead. The solution turns out to be doing custom pool management by allocating a new pool for every libsvn call, and forgetting the old one. r6567 in the Wesnoth-UMC-Dev repository applies this workaround for our SVN property setting tool.
Honestly, I'm tired of SVN. I use git-svn wherever I can but this isn't a magic solution for the crappy design of SVN's innards. git-svn can skip upstream commits if the connection interrupts during a fetch operation, and it forgets about local commits during pushes (git svn dcommit) after it has sent the first commit and fetched missing ones, which can cause loss of commits and history if the connection to the remote host breaks at that point or git-svn exits in any other fashion.
git-svn doesn't replace SVN's network code either (it uses it instead), so it's still subject to the perceived overhead, but at least it doesn't get stuck forever ignoring SIGTERMs.
What Wesnoth-UMC-Dev needs at the moment is a distributed (yes) version control system that's as more user-friendly as than SVN and has a nice, well documented Windows front-end that's easy to setup, learn and understand. If we can't find that, I'll continue complaining about SVN at any time and on every place where I see fit, here or in IRC.
The Wi-Fi router I use for connecting to the Internet stopped working last night (that dreaded “No route to host”). It's back now, but I'd have liked to post an update yesterday. Anyway, no news is good news, no? Er, well, assuming that the communication systems work, that is...which sadly seems not to be the case for the location of the epicenter, Cobquecura.
It's been more than 30 hours after the earthquake that caught me in the bathroom some minutes after 3:30 am yesterday. The aftershocks, while generally small, continue every 5-90 minutes, and at this point it just feels like one long ride on a bus through a badly deteriorated road — and it's getting fucking annoying.
I'd said that we were okay before. First I was scared when all this happened, and then I was sad after the news. Next, I was worried about the aftershocks. Now I'm just annoyed. Really annoyed.
Nonetheless, I'm not letting an earthquake and a never-ending sequence of aftershocks stop me. There's some groundbreaking work going on here — which you can't see, but I can, and I must say it looks really nice.
Now, could we just stop this madness and allow the rescue groups do something in the most affected locations near the epicenter without having the ruins of the buildings shaken every few minutes by a whimsical force of nature?
As you probably know (if not, Google “earthquake in Chile”), Chile's been struck by an earthquake approx. 8.8 in the Richter scale in the region near Concepción. I live in Santiago, and we have also been affected by this unfortunate event.
(Following comes from this Wesnoth.org forum post, now missing due to the former autopruning mechanism.)
I am OK right now, but I got trapped in the house during the earthquake (I was in the bathroom and some really heavy objects were in the way to the kitchen, which is the closest way to exit from my bedroom) and thought I wouldn't be able to get out in time. I did (about right at the end of the earthquake, not knowing at the moment if its intensity would continue increasing...), but we are not sure whether the house deteriorated further or not and do not really believe it'd resist a real earthquake with epicenter near Santiago.
There's a good distance between Concepción and Santiago, so it was about 7 in the Richter scale in Santiago according to the authorities last time I checked — about 10 am via FM radio on the car, we didn't have electricity, tap water or Internet at all until around 2 pm and I fell asleep around 1:16 pm after being unable to sleep the whole night with the strong and continuous tremors that followed. I originally posted this around 6:40 pm.
While everything's fine for us here right now, sadly, other areas of this same region didn't have this luck. Including areas where some of our family lives.
Naturally, everything to the south is chaos according to the news and there are still isolated people in coastal areas closer to the epicenter.
***
There are still tremors as of this writing. There was a small one which cut the power lines for 1 second some minutes ago, followed by a stronger one with lots of underground noise. The movement pattern continues being the same as the original earthquake. Internet is flaky.
If anyone's really interested in speaking to me, I've temporarily opened ##shadowm on irc.freenode.net.
Long ago, I wrote a little Bash script — set-properties — for the Wesnoth-UMC-Dev project, to ensure the correctness of SVN properties such as svn:keywords and svn:executable on files. It was pretty simple:
#! /bin/sh
# Set properties on PNG files
for f in $(find -iname *.png); do
svn propdel svn:executable $f
svn propset svn:mime-type image/png $f
done
# Set properties on Ogg files
for f in $(find -iname *.ogg); do
svn propdel svn:executable $f
svn propset svn:mime-type audio/x-vorbis
done
# Set properties on PCM files
for f in $(find -iname *.wav); do
svn propdel svn:executable $f
svn propset svn:mime-type audio/x-wav
done
# Set properties on JPEG files
for f in $(find -iname *.jpg); do
svn propdel svn:executable $f
svn propset svn:mime-type image/jpeg $f
done
for f in $(find -iname *.jpe); do
svn propdel svn:executable $f
svn propset svn:mime-type image/jpeg $f
done
for f in $(find -iname *.jpeg); do
svn propdel svn:executable $f
svn propset svn:mime-type image/jpeg $f
done
# Set properties on CFG file
for f in $(find -iname *.cfg); do
svn propdel svn:executable $f
done
# Set properties on scripts
for f in $(find -iname *.sh); do
svn propset svn:executable '*' $f
done
for f in $(find -iname *.cmd); do
svn propset svn:executable '*' $f
done
for f in $(find -iname *.bat); do
svn propset svn:executable '*' $f
done
for f in $(find -iname *.py); do
svn propset svn:executable '*' $f
done
Some time after I learned Perl with my work on Shikadibot 0314, I rewrote that script in Perl to arrange the “ideal” property values in a neat table (hash), check current properties instead of blindly overwriting them in the working copy, and cover plenty of other file types. It also gained a blinking progress bar to display the search progress for some reason.
To have an idea of how known file types are defined in the source, let's take a look at these bits:
# properties set to the empty string '' (except svn:executable) are left unchanged;
#
my %proptab = (
cfg => [FALSE, '', 'native', ''],
ign => [FALSE, '', 'native', ''],
"map" => [FALSE, '', 'native', ''],
# [...]
pl => [TRUE, '', 'native', 'Author Date Id Revision'],
# [...]
gif => [FALSE, 'image/gif', '', ''],
png => [FALSE, 'image/png', '', ''],
# [...]
xcf => [FALSE, '', '', ''],
# [...]
);
The table we have used since then (around Sept. 2008) has always contained more than 10 extensions with their minimum required property sets. As of this writing, it covers approximately 57 file types. Keep this on mind.
It would be overkill to fork-exec find processes to discover paths that could require SVN property changes, right? So, instead, I used find2perl to generate File::Find client code to embed it into set-properties. So far, so good. But how about running that code (scalar keys %proptab) times (e.g. number-of-extensions-times) anyway? Overkill?
No! It's plain stupid. But definitely less stupid than what you are about to read.
I apparently decided, for some reason, that any matching paths in each cycle should be added to a plain scalar (a text string to be exact) separating individual paths with newlines, of all things. Then, another cycle is performed at the end, (scalar keys %proptab) times again, to put each array of paths matching a certain extension into another hash, then iterating over the newly inserted hash element (array of paths) checking and fixing SVN properties in the same cycle.
Very roughly summarized as the following pseudocode:
FOREACH extension FROM file_extensions
FIND IN ./ AS file
; all while displaying a cute blinking bar!
IF file MATCHES extension
APPEND file TO file_list_string
END IF
END FIND
END FOREACH
FOREACH extension FROM file_extensions
; split 'file_list_string' every newline
FOREACH path FROM (SPLIT /\n/ file_list_string)
INSERT path IN file_index[extension]
END FOREACH
FOREACH path FROM file_index[extension]
fix_properties( extension, path )
END FOREACH
END FOREACH
Careful readers will quickly realize that something is horribly wrong with this pseudocode. I wish I was making it up. This algorithm has actually been in use by the Wesnoth-UMC-Dev set-properties script for one year and 4 months! I must have been on something when I wrote this shit. I've honestly never seen any program so awful as this in my life. Not even build-external-archive.sh (a.k.a. “Scrappy”) can compete with this abomination.
So, yesterday, I took a look at set-properties after noticing how much CPU time it ate working with very scarcely populated directories of the project — and I was blaming the usage of backticks (svn propset foo bar baz and such) as a possible cause of overhead. Then I slowly realized what I had written. There isn't any emoticon here for the expression in my face at that moment.
Thus, set-properties got rewritten with a much cleaner and simpler algorithm:
FOREACH file FROM (FIND IN ./)
; no more useless cute blinking bar!
FOREACH extension FROM file_extensions
IF file MATCHES extension
fix_properties( extension, path )
END IF
END FOREACH
END FOREACH
So yeah. 🙁
But wait, there's more! While optimizing replacing the script I also replaced svn foobar backtick code with invocations of libsvn, via the SVN::Client module. This worked very well at the end, but I discovered a few things in the process:
Those SVN::Client methods I used choke on non-absolute path specifications for some reason, causing an assertion failure in libsvn's C back-end and terminating the execution of Perl and the script with a SIGABRT.
Despite the documentation's claims for SVN::Client::url_from_path() returning undef if the specified path is not under version control, it actually causes the module to invoke die and get the client script terminated. Which means that I cannot even check if a file is versioned or not safely (e.g. without resorting to .svn/text-base/<FileName>.svn-base existence checks). What the hell?
Turns out the solution is to wrap SVN::Client method calls in eval blocks and handle whatever crap SVN comes up with. Oh, and make sure all paths are absolute using Cwd::realpath so that libsvn doesn't hit an assertion failure killing us, eval or no eval. How nice.
# No point in working with unversioned files.
my $svn_ret = undef;
eval { $svn_ret = $svn->url_from_path($path) };
if($@) {
# fucking libsvn dies if $path isn't under version control;
# the documentation says it should return undef above instead!
return;
}
With this, I have absolutely lost my faith in Subversion's excellence as a version control system not only as a normal user, but also as a programmer integrating it into my own client applications. And I know I lost my faith as a user after seeing svn sit for two days doing fucking nothing because the damned connection died after 1 minute of running time — and then svn spent the next days ignoring SIGTERMs all the time, no less. It's been like this for so many versions that I'm almost convinced it's intentional.
(Ah, that was relaxing. I should do this more often.)
Wesnoth-UMC-Dev has to continue using SVN because we have several add-on authors using Windows and the few alternatives for using git (I love git, I don't think I could try anything else at the moment) on Windows seem to be rather awkward to install and/or use for the average Non-Computer Person. That's a pity. It's really a pity.
Finally, set-properties got renamed to umcpropfix for a change, to mark its rebirth after I solved the algorithmic mess above last night. It doesn't have a nifty codename like the rest, though, but the new umc-prefixed name is still something to be celebrated now that we are going to have umcdist, umcstat, umcreg and umcbotd. Yes, I know I'm crazy, thank you.
It has become increasingly common for me to come up with a program for an amazing task one day, to rewrite it the next day.
umcdist, the Wesnoth-UMC-Dev Distribution Tool, has been in development hell for a year mostly for this reason; the other reason is that it seems like it will perform worse than build-external-archive.sh a.k.a. “Scrappy” due to an excessive usage of Perl's system function. I cannot make up my mind and choose between performance and maintainability; Espreon knows that build-external-archive.sh is broken, but I can't be bothered to try to understand that unholy abomination again to fix it.
Meanwhile, umcstat (Wesnoth-UMC-Dev Statistics Service) is still a work in progress, but with more emphasis in progress; there's actual code written already, and I've been using freenode's Eir bot framework to test it.<
While Eir could possibly be a nice way to get rid of umcreg's Net::IRC dependency and code, it's actually a C++ program that can be compiled only with GCC 4.4 at minimum, due to at least one C++0x feature used throughout the code: auto. The target machine runs Debian lenny, unlike my laptop (Squeeze), and therefore doesn't have GCC 4.4!
Instead of sticking with Eir, I'm refactoring umcreg's IRC code into a custom Perl-only framework, umcbotd, and making creative use of eval writing an abomination code-named “Naia”, which I'm rewriting again because the first version I wrote, which worked, was very badly designed and ugly. I know it's a problem when I have three classes or modules all depending upon each other's internals.
The goal of umcbotd/Naia is producing a Net::IRC-based abstraction layer for our bots that treats them as “services” with multiple “modules” (not in the Perl sense, though) that can be easily inserted and removed from the system by adding/removing their files. umcreg already runs with a prototype implementation of this mechanism, but it needs to be generalized further before it can be usable with Naia.
Writing our bots could be this simple, if Naia gets completed:
#!/bin/true
CO_SERVICE_COMPONENT('umcreg');
sub ctcp_version_reply () { "Wesnoth-UMC-Dev Registry Service, using " . Naia::version_string() }
my ($parent, $nick, $hostmask, $svaname) = (shift, shift, shift, shift);
if(!@_) {
$parent->notice($nick, "Not enough arguments for \002RAW\002.");
return;
}
$parent->sl(join(' ', @_));
broadcast_to_log("RAW [$hostmask]");
}
... And of course it would still involve lots of ugly stuff under the hood (eval magic), but if done right I shouldn't have to touch it whenever I wanted to add or remove a feature from any of our two services.
At last, umcreg, Wesnoth-UMC-Dev's Registry Service, is finished, deployed and announced in the forums, thus completing the first part of introducing the Registry system to the project.
There were some changes from its original incarnation, but everything turned out pretty well for a bot written from scratch in approximately 6 days by a Perl fanatic with no knowledge of object-oriented Perl — I actually learned some object-oriented Perl while at this and I feel like I can do anything with it now. 😄
I was in a hurry to get this done right before freenode deployed ircd-seven today (yays!), for no particular reason; this resulted in a security feature not working with hyperion-ircd until I introduced a quickie hack that I'll be retiring later today.
I have already registered our current members using some basic, known information about them — even including their join date. Old projects' registration will be a little slow as I need to research the current structure of the repository, of which I lost track a year ago, and retrieve original timestamps. The registry's web interface, provided by the umcreg::Web and Thoria::Web packages, can be found here.
umcreg is already working at the project's admin channel on freenode too. It only obeys the project staff's orders, though, so there's no point in trying to send messages to it.
The next step in implementing the Registry model is writing the Statistics service, codenamed “Listra”, and most likely going to be named umcstat. That will definitively take much longer than umcreg's development. Meanwhile, Espreon is trying to convince me to take care of umcdist (codename “Blackmore”) first.
After some weeks of inactivity, I have finally completed the first arc (not the first episode, though) of After the Storm in the Wesnoth-UMC-Dev SVN repository, comprising 7 scenarios out of planned 11; this means that a 0.3.0 release is coming soon. It was about time!
With this new arc completed, I have introduced new background story elements that could be considered controversial if any mainline purist is actually paying attention to the campaign — that's okay, I never intended IftU or any sequels to be mainlined. I still struggle to keep everything as fuzzy as possible to give a certain degree of flexibility to any content authors who decide to take what is said in IftU and AtS as “canon”. It's harder than it sounds, particularly because it must still be clear enough to allow the plot to progress; so I cannot just throw a bunch of nonsense into the campaign and say “hey, look at this, this is our vague excuse for this pathetic plotline!”.
Now that the characters have a decidedly vague excuse for the plot of the next arc, I face a problem that I knew I'd have to handle sooner or later: artwork.
I am not a good pixel artist, but I don't have any loyal slav- pixel artist that could help me either. And even if I could get one, I'm not completely sure I could describe the concepts I have in mind in plain words to tell them what unit sprites I require. There's also the spoileriffic factor; there's a reason that I removed the original (clumsy) storyboard from the SVN repository and departed from the original plans, most notably by removing a main character and introducing two new sidekicks instead. So, it's up to me to create any pixel art needed to make the campaign work; baseframes are enough for this purpose, although I still wish IftU and AtS' original units had animations.
At least I don't need to write a game engine from scratch too, thanks to Wesnoth's scripting flexibility.