After the Storm: Big Merge done, 0.8.90 on the horizon

The first After the Storm commit to Wesnoth-UMC-Dev’s trunk happened on May 17th, year 2008.

Today, February 19th 2013, after years and years of development, with various real life and non-real life issues getting in the way and dooming the campaign to Development Hell until version 0.4.0 with a completed Episode I finally happened on October 16th 2011, I can say for certain that...

After the Storm is finally complete.

With the Big Merge done and all the Episode III post-Divergence content (sans the Epilogue scenario) finally landed in Wesnoth-UMC-Dev trunk, the next step is releasing AtS version 0.8.90 to the public.

r17177 | AtS: bump version from 0.8.5+svn to 0.8.90-svn in the changelog after the Big Merge
r17176 | AtS: [Big Merge] workaround issues with the test suite and macros from data/core/units.cfg
r17175 | AtS E3: [Big Merge] land post-Divergence maps and scenarios
r17174 | AtS E3: [Big Merge] land post-Divergence ancillary macros, story text, character macros, and death handlers
r17173 | AtS: [Big Merge] land post-Divergence units WML, baseframes, halos, and animations
r17172 | AtS: [Big Merge] merge macros used for post-Convergence content
r17171 | AtS: [Big Merge] remove conditional loading of finale-stage scenarios and units
r17170 | AtS: bump version from 0.8.5+svn to 0.8.90-svn before the Big Merge
r17169 | AtS E3S6: enable scenario in regular gameplay

There will be a delay between this and the actual public 0.8.90 release while my primary playtester (vultraz) does the playtesting thing with the scenarios. Some balancing changes before 0.8.90 may also be necessary.

In the meantime, people can check out After the Storm from Wesnoth-UMC-Dev trunk using the Subversion client of their choice, and provide me with feedback via forum PM (in particular, about any possible bugs or balance issues that might plague some specific scenarios), or private messaging on IRC — I am on irc.freenode.net, channel ##shadowm most of the time, but I would rather avoid people dropping spoilers in the presence of my aforementioned playtester, who also hangs around there.

UPDATE: Since I haven’t gotten around to publish version 0.2.0 of the AtS Music add-on, you will also need to obtain the latest version from SVN separately, since it introduces a few music tracks used in the new AtS Episode III scenarios. Of course, this is only necessary if you want/need to have in-game background music.

UPDATE 2: AtS Music version 0.2.0 is now in the 1.10 and 1.11.x add-ons servers. The previous version was 12.3 MiB in size, whereas the new version is 22.7 MiB. I actually had to do some re-encoding to bring it down from 33.1 MiB, but there shouldn’t be any noticeable compression artifact build-up — or at least, I cannot perceive any with my headphones on.

svn co https://wesnoth-umc-dev.svn.sourceforge.net/svnroot/wesnoth-umc-dev/trunk/After_the_Storm
svn co https://wesnoth-umc-dev.svn.sourceforge.net/svnroot/wesnoth-umc-dev/trunk/AtS_Music

You can also grab tarballs of the latest trunk snapshots through SourceForge.net’s SVN web interface:

This last alternative is probably not the best, though, since you will not be able to track future updates. Subversion makes it far easier to update every time a changeset is committed, without having to download the whole thing every time.

I am not going to provide any further instructions for installing using either of these methods, so I’m leaving this to people who actually know their way around Subversion tools or manually installing add-on content. I am not going to post any spoilers either; in particular, I am not going to reveal the Epilogue sequence until version 0.9.0.

Special thanks go to Espreon, Gambit, and vultraz for making all of this possible in their own ways. I will probably explain the deal with AtS’ troubled development in a new post in the near future (probably after 0.8.90 is properly published).

Thanks to all those who waited this long for this to happen. For those who are eager to playtest this and don’t know their way around the aforementioned things, I can only promise that the wait for 0.8.90 will be much shorter. A couple of weeks, tops.

Finally, the usual disclaimer applies: this campaign is not for everyone.

Late After the Storm development roadmap

It is February 16th already, and it has been a long while since my last update on AtS’ development on this blog, unless we count the January 1st teaser.

Well, here is another teaser in the form of a more concrete roadmap for AtS 0.9.0’s development:

Development diverged around the release of version 0.8.3 resulting in only bug-fixes and minor features landing in Wesnoth-UMC.-Dev trunk, which is where AtS’ releases come from. During all this time, I have been (mostly) silently working on the final set of scenarios (E3S7A onwards) in a separate ‘branch’ of sorts, with occasional code and other assets landing in trunk to ease the upcoming Big Merge work that will take place “soon”.

And by “soon”, this time I really mean it.

This development model may have given some people the impression that I stopped working on AtS altogether, but this was never the case. IftU and AtS have historically been solo projects, and I have dedicated a lot of time to programming, creating original artwork, and writing prose for both campaigns. AtS has been a more demanding energy sink because of the different standards compared to IftU, and Episode III has been even more demanding than Episode II was in every possible sense.

However, it is finally starting to pay off as I’m about to complete the last scenario of the sequence referred to in the chart above as Finale C, which is pretty much the end of the After the Storm proper. Once that is done, I will go back to some of the previous unreleased scenarios to deal with various rough edges in programming, balancing, and prose, and then perform the Big Merge on trunk, leading to the 0.8.90 release (also known as 0.9.0 Release Candidate 1). Whether this release will be public or just limited to whoever expresses interest in playtesting it is something I have yet to decide.

Since at this point all the artwork for Finale C is done and most of the programming components in place, I expect the Big Merge to take place around the end of the month. No promises on when 0.8.90 will be announced, though, nor to whom it will be announced.

After that I’m facing the challenge of writing the combined epilogue for both campaigns, which is a whole different can of worms because of some architectural restrictions in Wesnoth’s design that I need to overcome, somehow. While I am already know how the epilogue flows in advance, making it not look like shit could well be an unprecedented feat.

Once the epilogue is done, After the Storm 0.9.0 will be released. Theoretically, this should not take too long, but we shall see.

To recap, I am literally one scenario away from the epilogue, and this campaign is definitely not abandoned.

Also, if there are any campaign authors out there making assumptions about the AtS E3 plot progression, you should probably be prepared to either revise them or stick to your own alternate-universe continuity theories or whatever floats your boat.

Wesbreak, IRC-break

Just wanted to note that I am pausing my online Wesnoth and IRC related activities for an indeterminate amount of time while I tend to some matters of greater priority. In case you are wondering, yes, everything is fine, but I do need a break.

You can still message me through Twitter, email (preferably!), or private message through the Wesnoth forums (which result in email notifications) if something absolutely important crops up in the meantime other than the forums being down. I have provided the other Wesnoth.org administrators with instructions in case I’m needed for something specific, and for other issues you can always PM the Forum Moderators or Administrators groups or email the forums support address, which is displayed whenever things go completely wrong.

And of course, I may continue to post about non-Wesnoth things here as I see fit.

On Wesnoth’s Game Development forum

From a recent topic in Wesnoth.org’s Game Development forum:

[...] this is also quite close to advertisement, but you don't have the traits of a spammer.

This is one of my major qualms about the current organization of the Wesnoth.org forums. The non-indicative Game Development section was indeed created for that specific purpose of serving as a venue for advertising other games and potentially recruiting contributors. The definition of spam for this section is sketchy at best, but common sense suggests that the following forms of promotion would be off-limits:

  • Advertising things that are not games
  • Blatant advertising by posting links to different topics
  • Misleading information/scam

If advertising in GD in general was considered spam, then the forum would be nearly or entirely empty. The question is, why do we provide this marketing channel in the first place? It seems to me like the primary goal is promoting open-source game development, but the forum description implies that this is not a strict requirement for posting there. I am not terribly comfortable with the idea that it should be our mission to do this seeing as how there are larger and better organized communities for this kind of thing, but I can see how some people might prefer to have means to preach to the world at large without having to maintain a blog or register accounts on other boards.

Quite lazy, if you ask me.

But all this is fine by me as long as I’m not forced to read every single topic that crops up in there.

irker-svnpoller: Subversion poller and mail filter application for irker

Just as irker’s adoption rate is increasing, I have just completed work on a very simple application for Subversion repositories — two applications, in fact.

irker-svnpoller is a very simple script that polls a single commit log (not data) from a Subversion repository and delivers notifications to any number of channels using an irkerd running on the same host. It mimics the CIA bots’ formatting, much like nenolod’s irker CIA proxy, from which I borrowed a small amount of code.

irker-svnpoller → irkerd

But exactly how is this supposed to be useful to anyone, you may be wondering right now? Well, irker-svnpoller is not really intended to be used stand-alone. A timed poller script that tracks the last notified revision could come in handy, but people could get impatient waiting for their commits to appear in their IRC channels minutes later. I am well familiarized with the defects, quirks, and virtues of my primary audience—the Battle for Wesnoth and Wesnoth-UMC-Dev projects—and this approach would simply not scale well over time.

Enter the first companion script, svnmail-filter. It reads email message headers from stdin to determine a commit’s revision number and the pertinent repository to probe using irker-svnpoller. Configuration is mostly done through a ruleset file using the JSON format.

Of course, svnmail-filter is not that useful on its own either. The idea is that procmail or some other MDA should pipe incoming email headers through svnmail-filter — and preferably, only those from legitimate sources, such as subscribed commit mailing lists. This is actually simpler than it sounds, and it is more or less inspired by CIA.vc’s perpetually broken mail-based SVN poller.

MDA → svnmail-filter → irker-svnpoller → irkerd

Since no service in the pipeline other than irkerd runs persistently in the background, this should be significantly more fault-resilient than CIA.vc’s approach, which apparently required a poller service to listen and act upon local requests. The downside is that the host running irker-svnpoller may need to create many short-lived SVN repository connections for individual commits in a chain. In Wesnoth’s case, SVN commit chains are rare enough, but their size often goes around a dozen individual commits or so. Regardless, this shouldn’t be terribly concerning for a production server with a decent low-latency uplink, and the overhead on the repository provider should be rather small compared to pushing massive commit diffs across the network.

Right now, the Wesnoth and Wesnoth-UMC-Dev projects are using this service as a stopgap measure until their respective providers—Gna.org and SourceForge.net—allow installing a hook that can either talk directly to an irkerd server, or to an instance of the aforementioned CIA proxy using the CIA XML-RPC method.

I am not all that keen on other people using a piece of software I developed and tested within less than three days without any prior experience working with Python. There are also various problems inherent to any application depending upon Subversion and its incompetent network transport layer.

Nonetheless, I published a Git repository on GitHub including a small amount of documentation to get started:

I am open to possible improvements coming from people intending to use this on production servers. In particular, if someone out there works with a commit mailing list where revision numbers can’t be found in mail subjects it would be necessary to adapt svnmail-filter a little to handle that case. Perhaps it might even be possible to skip the irker-svnpoller step for mailing lists where the notification message structure is constant and well documented.

Exit CIA.vc, enter irker

Following CIA.vc’s untimely demise, ESR and a small ad hoc group of coders and testers including nenolod (from Atheme) and our very own AI0867 (who has led Wesnoth-UMC-Dev in my absence) finally completed the work required to get irker 1.0 out. irker itself has been a work in progress for a while since the last CIA outage in August.

It’s advertised as a CIA.vc replacement, but in reality it is something far less ambitious in scope: a write-only IRC bot that serves its own message bus. From its own README:

irkerd is a specialized IRC client that runs as a daemon, allowing other programs to ship IRC notifications by sending JSON objects to a listening socket.

It is meant to be used by hook scripts in version-control repositories, allowing them to send commit notifications to project IRC channels. A hook script, irkerhook.py, supporting git and Subversion is included in the dustribution (sic); see the install.txt file for installation instructions.

The author’s intention is for existing code forges to adopt this service, and perhaps optionally run it on their own facilities alongside their VCSes, allowing repository admins to opt in for using hooks that deliver notifications to those internal irker instances. irker’s pipeline is extremely flexible and can be summed up as follows:

Repository hook → irker instance → IRC server

CIA.vc’s pipeline is not entirely clear to me and I did not have the opportunity to inspect it from inside, unlike ESR. However, there’s enough evidence suggesting that it was more or less like the following:

Repository hook → CIA.vc XML-RPC or mail provider → CIA.vc database manager → IRC front-end → IRC server

Note that there was also a web front-end, which was integral to CIA’s mission as it was the only way to define projects and bots. A commit notification occurred for a given project; say, Wesnoth-UMC-Dev. The IRC portion of the pipeline made sure that all relevant bots (each one associated to a single channel from the model standpoint) would report the same commit. A less relevant Web front-end in the pipeline took care of adding the commit to the project page (including statistics and an XML feed).

The IRC portion was flexible enough to accommodate the simplest use case (notifying a single project’s commits in a single channel), and more elaborate yet still reasonable use cases (notifying commits from multiple projects in a single channel) without much hassle, all done by tweaking the bots’ configuration in the web-based configuration front-end. Even more advanced use cases were possible by choosing the Advanced Filtering option in the same front-end. This allowed me to have a bot in ##shadowm on freenode report commits as follows:

  • Commits from wesnoth-umc-dev (already reported in #wesnoth-umc-dev) with paths matching */After_the_Storm/* and */Invasion_from_the_Unknown/*, regardless of author
  • Commits from my own CIA-registered projects (morningstar, weldyn, dorset, etc.) regardless of author
  • Commits from any other CIA-registered project (such as Wesnoth or Frogatto) with an author matching my real name or any of my preferred screen names (fun fact: never got any false positives since I set it up a couple of years ago)

I should emphasize that this required no changes to hooks in each repository. Hooks delivered just a minimal set of information, including the commit hash or number, the commit message, affected path, affected branch (when applicable), affected module (when applicable), the author name, and the project name. Everything else was done on CIA’s side, including deciding which channels should get notified of individual commits.

irker does not do this.

irker’s perceived elegance stems from its very basic and versatile design. Essentially, it serves as a mechanism for a repository hook to interact with IRC without having to establish a short-lived connection to a server for every individual commit or commit batch — an approach that GitHub currently allows via a separate, seldom used IRC service hook. irker is not novel in design by any means, and the hype around it is only justified by the fact that nobody bothered to create and advertise any other service that could properly replace CIA.vc before and be inherently extensible maintainable over time.<

irker’s extensibility and maintainability stems from the fact that a good portion of the work is done by the repository hooks, and irker is near completely stateless — the obvious opposite of CIA.vc’s architecture.

Unfortunately, this renders advanced use cases such as the above ##shadowm CIA ruleset completely incompatible with the irker pipeline.

From ESR’s post on CIA.vc’s design and its shortcomings:

[...] the original designer fell in love with the idea of data-mining and filtering the notification stream. It is quite visible on the CIA site how much of the code is concerned with automatically massaging the commit stream into pretty reports. I’m told there is a complicated and clever feature involving XML rewrite rules that allows one to filter commit reports from any number of projects by the file subtrees they touch, then aggregate the result into a synthetic notification channel distinct from any of the ones those projects declared themselves.

(He somehow got this part slightly wrong. Incidentally, it was me who brought it up in #cia around August 25th in the first place. The projects’ own notification channels were as synthetic as any others from CIA.vc’s point of view. That is to say, not at all. Additionally, they weren’t XML rewrite rules, but rather commit matching rules.)

His opinion is, naturally...

Bletch! Bloat, feature creep, and overkill!

Yes, I admit that it is overkill, but it was a nice thing from our point of view as users of the system. There’s a line between using a service, and administrating it.

On the plus side, seeing as how irker aims to become an actual standard for IRC feeds of any sort (not just for VCSes), it is good that it only implements the most basic functionality by itself. This should later allow us to come up with ingenious applications such as nenolod’s CIA proxy for irker (delivers CIA.vc XML-RPC requests in a format suitable for irker). Some people have even proposing building new services using irker’s protocol, adding an authentication layer on top and integrating it to IRC networks as a hosted service!

But replicating the end-user functionality a few people like me enjoyed will invariably take some additional effort. ESR suggests:

Filtering? Aggregation? As previously noted, they don’t need to be in the transmission path. One or more IRC bots could be watching #commits, generating reports visible on the web, and aggregating synthetic feeds. The only agreement needed to make this happen is minimal regularity in the commit message formats that the hooks ship to IRC, which is really no more onerous than the current requirement to gin up an XML-RPC blob in a documented format.

Of course, if the #commits channel on freenode ever regains its former glory, this would require a bot to listen to and filter possibly thousands of messages per minute, all coming from multiple clients. I don’t think I am fit to become the pioneer who’ll conquer this new land.

Furthermore, since the task of formatting messages for individual commits is exclusive to individual hooks, we may end up with a highly fragmented and inconsistent ecosystem. For example, as things stand right now, no-one is required to include #commits on freenode as a destination for commit notifications, and I imagine very few people will bother to do so in the future.

All in all, it was our own incompetence that allowed CIA.vc to die prematurely despite the multiple calls for replacing it, and the obviously deplorable service conditions. We can’t really complain now.

CIA.vc is dead

I normally don’t comment or report on other sites’ statuses in here since this is my personal blog, but this situation actually impacts Wesnoth, Wesnoth-UMC-Dev, and me directly; especially me, considering I went to ridiculous lengths the other day to solve a related issue on GitHub.

The point is literally the title of this post: CIA.vc is dead.

You know, CIA.vc; that amazing service which provided real-time VCS commit notifications on various IRC networks and that everyone took for granted. This is by no means the first time it bites the dust, but in this opportunity it’s suspected that nobody really bothered to make backups.

nenolod (who was merely hosting the instance running CIA.vc) explained the situation in freenode’s #cia channel about an hour ago.

Assuming the other people who had admin access don’t have their own recent backups, CIA.vc’s future looks particularly bleak right now. Here’s hoping that a dedicated team of competent coders with access to a suitable server for hosting will quickly build a better replacement within the next few days. (Ha, ha, ha. Right.)