Controlling a differential drive robot in Gazebo from ROS

Gazebo is a simulation suite targeted at robotics projects, used by DARPA to implement the virtual part of its Robotics Challenge. It's a great companion to the Robot Operating System, supporting several ways to interface with it.

One recurrent question regarding Gazebo is how to control robots described in Simulation Description Format (SDF) models from ROS. The more involved case of porting a URDF model to Gazebo and manipulating it through ROS Control interfaces is well documented, but what about a robot that already has a full SDF model available? Isn't it possible to just paste it into a Gazebo workspace and get some ROS topics to manipulate it?

The tutorial below covers this simpler case. It is based on the SDF model for the Pioneer 2DX differential drive robot, which is available from Gazebo's default library. It also assumes ROS and Gazebo are already installed, as well as the necessary integration packages and plugins.

First, save the SDF world model below to your local filesystem as pioneer2dx_ros.world:

<?xml version="1.0" ?>
<sdf version='1.4'>
  <world name='default'>
    <light name='sun' type='directional'>
      <cast_shadows>1</cast_shadows>
      <pose>0 0 10 0 -0 0</pose>
      <diffuse>0.8 0.8 0.8 1</diffuse>
      <specular>0.2 0.2 0.2 1</specular>
      <attenuation>
        <range>1000</range>
        <constant>0.9</constant>
        <linear>0.01</linear>
        <quadratic>0.001</quadratic>
      </attenuation>
      <direction>-0.5 0.1 -0.9</direction>
    </light>
    <model name='ground_plane'>
      <static>1</static>
      <link name='link'>
        <collision name='collision'>
          <geometry>
            <plane>
              <normal>0 0 1</normal>
              <size>100 100</size>
            </plane>
          </geometry>
          <surface>
            <friction>
              <ode>
                <mu>100</mu>
                <mu2>50</mu2>
              </ode>
            </friction>
            <bounce/>
            <contact>
              <ode/>
            </contact>
          </surface>
          <max_contacts>10</max_contacts>
        </collision>
        <visual name='visual'>
          <cast_shadows>0</cast_shadows>
          <geometry>
            <plane>
              <normal>0 0 1</normal>
              <size>100 100</size>
            </plane>
          </geometry>
          <material>
            <script>
              <uri>file://media/materials/scripts/gazebo.material</uri>
              <name>Gazebo/Grey</name>
            </script>
          </material>
        </visual>
        <velocity_decay>
          <linear>0</linear>
          <angular>0</angular>
        </velocity_decay>
        <self_collide>0</self_collide>
        <kinematic>0</kinematic>
        <gravity>1</gravity>
      </link>
    </model>
    <physics type='ode'>
      <max_step_size>0.001</max_step_size>
      <real_time_factor>1</real_time_factor>
      <real_time_update_rate>1000</real_time_update_rate>
      <gravity>0 0 -9.8</gravity>
    </physics>
    <scene>
      <ambient>0.4 0.4 0.4 1</ambient>
      <background>0.7 0.7 0.7 1</background>
      <shadows>1</shadows>
    </scene>
    <spherical_coordinates>
      <surface_model>EARTH_WGS84</surface_model>
      <latitude_deg>0</latitude_deg>
      <longitude_deg>0</longitude_deg>
      <elevation>0</elevation>
      <heading_deg>0</heading_deg>
    </spherical_coordinates>
    <model name='pioneer2dx'>
      <link name='chassis'>
        <pose>0 0 0.16 0 -0 0</pose>
        <inertial>
          <mass>5.67</mass>
          <inertia>
            <ixx>0.07</ixx>
            <iyy>0.08</iyy>
            <izz>0.1</izz>
            <ixy>0</ixy>
            <ixz>0</ixz>
            <iyz>0</iyz>
          </inertia>
        </inertial>
        <collision name='collision'>
          <geometry>
            <box>
              <size>0.445 0.277 0.17</size>
            </box>
          </geometry>
          <max_contacts>10</max_contacts>
          <surface>
            <bounce/>
            <friction>
              <ode/>
            </friction>
            <contact>
              <ode/>
            </contact>
          </surface>
        </collision>
        <collision name='castor_collision'>
          <pose>-0.2 0 -0.12 0 -0 0</pose>
          <geometry>
            <sphere>
              <radius>0.04</radius>
            </sphere>
          </geometry>
          <surface>
            <friction>
              <ode>
                <mu>0</mu>
                <mu2>0</mu2>
                <slip1>1</slip1>
                <slip2>1</slip2>
              </ode>
            </friction>
            <bounce/>
            <contact>
              <ode/>
            </contact>
          </surface>
          <max_contacts>10</max_contacts>
        </collision>
        <visual name='visual'>
          <pose>0 0 0.04 0 -0 0</pose>
          <geometry>
            <mesh>
              <uri>model://pioneer2dx/meshes/chassis.dae</uri>
            </mesh>
          </geometry>
        </visual>
        <visual name='castor_visual'>
          <pose>-0.2 0 -0.12 0 -0 0</pose>
          <geometry>
            <sphere>
              <radius>0.04</radius>
            </sphere>
          </geometry>
          <material>
            <script>
              <uri>file://media/materials/scripts/gazebo.material</uri>
              <name>Gazebo/FlatBlack</name>
            </script>
          </material>
        </visual>
        <velocity_decay>
          <linear>0</linear>
          <angular>0</angular>
        </velocity_decay>
        <self_collide>0</self_collide>
        <kinematic>0</kinematic>
        <gravity>1</gravity>
      </link>
      <link name='right_wheel'>
        <pose>0.1 -0.17 0.11 0 1.5707 1.5707</pose>
        <inertial>
          <mass>1.5</mass>
          <inertia>
            <ixx>0.0051</ixx>
            <iyy>0.0051</iyy>
            <izz>0.009</izz>
            <ixy>0</ixy>
            <ixz>0</ixz>
            <iyz>0</iyz>
          </inertia>
        </inertial>
        <collision name='collision'>
          <geometry>
            <cylinder>
              <radius>0.11</radius>
              <length>0.05</length>
            </cylinder>
          </geometry>
          <surface>
            <friction>
              <ode>
                <mu>100000</mu>
                <mu2>100000</mu2>
                <slip1>0</slip1>
                <slip2>0</slip2>
              </ode>
            </friction>
            <bounce/>
            <contact>
              <ode/>
            </contact>
          </surface>
          <max_contacts>10</max_contacts>
        </collision>
        <visual name='visual'>
          <geometry>
            <cylinder>
              <radius>0.11</radius>
              <length>0.05</length>
            </cylinder>
          </geometry>
          <material>
            <script>
              <uri>file://media/materials/scripts/gazebo.material</uri>
              <name>Gazebo/FlatBlack</name>
            </script>
          </material>
        </visual>
        <velocity_decay>
          <linear>0</linear>
          <angular>0</angular>
        </velocity_decay>
        <self_collide>0</self_collide>
        <kinematic>0</kinematic>
        <gravity>1</gravity>
      </link>
      <link name='left_wheel'>
        <pose>0.1 0.17 0.11 0 1.5707 1.5707</pose>
        <inertial>
          <mass>1.5</mass>
          <inertia>
            <ixx>0.0051</ixx>
            <iyy>0.0051</iyy>
            <izz>0.009</izz>
            <ixy>0</ixy>
            <ixz>0</ixz>
            <iyz>0</iyz>
          </inertia>
        </inertial>
        <collision name='collision'>
          <geometry>
            <cylinder>
              <radius>0.11</radius>
              <length>0.05</length>
            </cylinder>
          </geometry>
          <surface>
            <friction>
              <ode>
                <mu>100000</mu>
                <mu2>100000</mu2>
                <slip1>0</slip1>
                <slip2>0</slip2>
              </ode>
            </friction>
            <bounce/>
            <contact>
              <ode/>
            </contact>
          </surface>
          <max_contacts>10</max_contacts>
        </collision>
        <visual name='visual'>
          <geometry>
            <cylinder>
              <radius>0.11</radius>
              <length>0.05</length>
            </cylinder>
          </geometry>
          <material>
            <script>
              <uri>file://media/materials/scripts/gazebo.material</uri>
              <name>Gazebo/FlatBlack</name>
            </script>
          </material>
        </visual>
        <velocity_decay>
          <linear>0</linear>
          <angular>0</angular>
        </velocity_decay>
        <self_collide>0</self_collide>
        <kinematic>0</kinematic>
        <gravity>1</gravity>
      </link>
      <joint name='left_wheel_hinge' type='revolute'>
        <pose>0 0 -0.03 0 -0 0</pose>
        <child>left_wheel</child>
        <parent>chassis</parent>
        <axis>
          <xyz>0 1 0</xyz>
          <limit>
            <lower>-1e+16</lower>
            <upper>1e+16</upper>
          </limit>
        </axis>
      </joint>
      <joint name='right_wheel_hinge' type='revolute'>
        <pose>0 0 0.03 0 -0 0</pose>
        <child>right_wheel</child>
        <parent>chassis</parent>
        <axis>
          <xyz>0 1 0</xyz>
          <limit>
            <lower>-1e+16</lower>
            <upper>1e+16</upper>
          </limit>
        </axis>
      </joint>
      <!-- Replaced Gazebo's differential drive plugin with the
           ROS-friendly variant -->
      <!--
      <plugin filename="libDiffDrivePlugin.so" name="diff_drive">
        <left_joint>left_wheel_hinge</left_joint>
        <right_joint>right_wheel_hinge</right_joint>
        <torque>5</torque>
      </plugin>
      -->
      <plugin name='differential_drive_controller'
              filename='libgazebo_ros_diff_drive.so'>
        <alwaysOn>true</alwaysOn>
        <updateRate>100</updateRate>
        <leftJoint>left_wheel_hinge</leftJoint>
        <rightJoint>right_wheel_hinge</rightJoint>
        <wheelSeparation>0.39</wheelSeparation>
        <wheelDiameter>0.15</wheelDiameter>
        <torque>5</torque>
        <commandTopic>cmd_vel</commandTopic>
        <odometryTopic>odom</odometryTopic>
        <odometryFrame>odom</odometryFrame>
        <robotBaseFrame>chassis</robotBaseFrame>
      </plugin>
      <pose>0 0 0 0 -0 0</pose>
      <static>0</static>
    </model>
    <state world_name='default'>
      <sim_time>85 304000000</sim_time>
      <real_time>85 849190220</real_time>
      <wall_time>1432260579 736436496</wall_time>
      <model name='ground_plane'>
        <pose>0 0 0 0 -0 0</pose>
        <link name='link'>
          <pose>0 0 0 0 -0 0</pose>
          <velocity>0 0 0 0 -0 0</velocity>
          <acceleration>0 0 0 0 -0 0</acceleration>
          <wrench>0 0 0 0 -0 0</wrench>
        </link>
      </model>
      <model name='pioneer2dx'>
        <pose>-0.103826 0.027961 2e-06 -2e-06 -8e-06 -0.094162</pose>
        <link name='chassis'>
          <pose>-0.103827 0.027961 0.160002 -2e-06 -8e-06 -0.094162</pose>
          <velocity>
            -0.000797 0.002229 0.003363 -0.024657 -0.007301 0.0023
          </velocity>
          <acceleration>0 0 0 0 -0 0</acceleration>
          <wrench>0 0 0 0 -0 0</wrench>
        </link>
        <link name='left_wheel'>
          <pose>0.011714 0.187806 0.110002 1.38258 1.57033 2.85912</pose>
          <velocity>
            -0.000816 0.001565 0.000275 -0.014307 -0.006987 0.002012
          </velocity>
          <acceleration>0 0 0 0 -0 0</acceleration>
          <wrench>0 0 0 0 -0 0</wrench>
        </link>
        <link name='right_wheel'>
          <pose>
            -0.020254 -0.150688 0.110002 -1.56886 1.51434 -0.092322
          </pose>
          <velocity>
            -0.000347 0.001559 0.008433 -0.014387 -0.003243 0.000126
          </velocity>
          <acceleration>0 0 0 0 -0 0</acceleration>
          <wrench>0 0 0 0 -0 0</wrench>
        </link>
      </model>
    </state>
    <gui fullscreen='0'>
      <camera name='user_camera'>
        <pose>5 -5 2 0 0.275643 2.35619</pose>
        <view_controller>orbit</view_controller>
      </camera>
    </gui>
  </world>
</sdf>

Open a terminal window and start the ROS middleware by entering:

roscore

Open another terminal window, cd to the folder containing pioneer2dx_ros.world and enter:

rosrun gazebo_ros gazebo -file pioneer2dx_ros.world

You should now have a Gazebo window opened with a Pioneer 2DX placed in the middle of an empty world.

Finally, open a third terminal window (it's the last one, I promise) and check if the differential drive topics have been published:

rostopic list

The following topics should be visible, among others:

/pioneer2dx/cmd_vel
/pioneer2dx/odom

You should now be able to get the robot moving by publishing messages to the /pioneer2dx/cmd_vel topic, e.g.

rostopic pub -1 /pioneer2dx/cmd_vel geometry_msgs/Twist \
'{linear: {x: 1.0, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 1.0}}'

Should get the robot running in a loop.

How to download all Gazebo models

Gazebo is a wonderful simulation tool widely used in robotics projects. The project includes a library of 3D models that are automatically downloaded on demand. However this makes the application pause each time a model is used for the first time, which can take a considerable time depending on connection speed and interfere with the flow of work.

In order to avoid this issue I wrote the below script to do a bulk download of all models in the Gazebo library, then copy them to the local model folder. Simply copy the lines below to a script file on a convenient local folder and run it from a command prompt.

#!/bin/sh

# Download all model archive files
wget -l 2 -nc -r "http://models.gazebosim.org/" --accept gz

# This is the folder into which wget downloads the model archives
cd "models.gazebosim.org"

# Extract all model archives
for i in *
do
  tar -zvxf "$i/model.tar.gz"
done

# Copy extracted files to the local model folder
cp -vfR * "$HOME/.gazebo/models/"

Building OpenCV's HTML documentation

OpenCV has become the de facto standard vision processing library, but the project remains surprisingly obscure in several areas, particularly in relation to building and installation tasks. For example, it took me a couple hours to discover how to build the project's HTML documentation (though to be fair, I was looking into it concurrently with other work).

For those also struggling with this, see this question in the OpenCV Q&A site. In short, it gives the following instructions:

1. Install Sphinx;
2. Run cmake on your OpenCV source base as explained in the installation instructions, perhaps adding the -D BUILD_DOCS=ON option for good measure;
3. Run make html_docs to build the documentation in HTML format.

If everything goes well the HTML documentation will be built to the subfolder doc/_html/ inside the build output folder. If not, check if the output of the cmake call reported items Build Documentation and Sphinx both set to YES. You may also check the cmake script cmake/OpenCVDetectPython.cmake inside your OpenCV source base for potential problems.

ROS and my first pull request

I had been putting it off for a long time, but a couple weeks ago I finally decided to bite the bullet and port my research code to ROS. After a bit of a rough start reinstalling my OS from scratch – I had recently upgraded my notebook to Kubuntu 14.10, but the latest ROS distribution (Indigo) wouldn't install on anything past 14.04 – and coming to terms with the catkin build system, I quickly fell in love with its modular approach to software development.

One of the many little conveniences ROS offers are parameters, which among other things greatly simplify management of program arguments. The ROS C++ client library provides several ways to retrieve parameter values, and optionally provide defaults for when they are not defined. One of those ways is the ros::param::param template function, which can be used to write initialization code like this:

std::string path;
std::string format;
double fps;

param("~path", path, "video.mpg");
param("~format", format, "MPEG");
param("~fps", fps, 30.0);

VideoRecorder recorder(path, format, fps);

However I disliked having to declare variables to get to the parameter values, just to pass them straight to a constructor or function call and then disregard them. So I wrote an alternative version of the template function that retrieved the value as a return instead of an output argument:

template<class T> T param(const std::string& name, const T& fallback)
{
  T value;
  param(name, value, fallback);
  return value;
}

So I could rewrite the code above as:

VideoRecorder recorder(param("~path", "video.mpg"),
                       param("~format", "MPEG"),
                       param("~fps", 30.0));

At first I had separate copies of this function on each of my ROS projects, but soon I realized it would be great if I could contribute it to the official roscpp code base. After a bit of stumbling about looking for the right place to propose it, I ended up submitting the change myself as a pull request. Overall the process went very smoothly, though there are some tips I wish I had been given before I started:

• You'll want to compile and run tests on the code you intend to submit (believe me, even a change as small as the one above was not without a couple coding errors), so remember to create a catkin workspace into which to clone the forked repository;
• You may think that your change is trivial and will be done in a single commit, but it's likely your submission will go through several revisions before it's accepted. To keep things organized, create a new branch before starting your work;
• Beware of line-ending whitespace – there are a few lurking around in source files, and if your editor automatically removes them you'll end with an unnecessarily large (and likely unacceptable) diff;
• That said if you do mess up, don't bother trying to amend the initial commit – simply submit additional commits with the required changes or reversals (this is where having a separate branch may come in handy).

Open communities are not always noted for being welcoming to beginners, even less so in technical circles. But I have nothing but praise for the ros_comm community and the fairly friendly and constructive exchange we had. Keep up the good work people!

batcher - simple shell wrappers for Python

One of my favorite Linux features is how repetitive tasks can be automated via shell scripts. Unfortunately, the Bourne shell language and its variations (e.g. Bash) are bogged down by a number of idiosyncrasies – such as the nearly constant need to enclose variable references in quotes, the awkwardness of conditional constructs, the difficulties involved in performing simple arithmetics and so on – which needlessly complicate development. On the other hand, the ability to directly invoke user shell commands, scripts and command-line applications is a convenience seldom matched by alternatives, which explains why we mostly put up with it.

A while ago I wanted to write a relatively complex automation script, and not looking forward to doing the required text processing in Bash, decided instead to try my luck with Python, using Popen objects to invoke the needed command-line applications and collect their outputs. In order to make this task simpler, I came up with the function below:

def batch(*args, **options):
    options.setdefault('stdout', PIPE)
    options.setdefault('stderr', STDOUT)
    process = Popen(args, **options)
    return process.stdout

This allowed me to interface to shell tools in a straightforward way, for example:

for line in batch('git', 'status', '-s', cwd=path):
    # Do some line-oriented processing with the command's output

While it was simple and worked well enough, this interface surely had room for improvement. It bothered me that I had no access to the Popen object driving the shell process, but only to its stdout output object; I also felt it was wasteful that the process would run to completion even if control broke off the loop before it finished. Finally, on an aesthetic note, I thought it would be nice if there was some "automagic" way to turn command names into callables, so I could e.g. do git('status', '-s') instead of batch('git', 'status', '-s').

After some tinkering with Python's reflection API, I came up with the code below:

#! /usr/bin/env python
#coding=utf-8

# batcher.py
# Dynamic interfaces to shell commands

from subprocess import Popen, PIPE, STDOUT
from sys import modules

class batch(Popen):
    r'''A running shell command or script.
    '''
    def __init__(self, command, args, options):
        args = (command,) + args
        options.setdefault('stdout', PIPE)
        options.setdefault('stderr', STDOUT)
        Popen.__init__(self, args, **options)
        self.daemon = options.get('daemon', True)

    def __del__(self):
        if self.daemon:
            self.close()
        else:
            Popen.__del__(self)

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        self.close()

    def __iter__(self):
        return self

    def close(self):
        if self.returncode != None:
            return

        try:
            self.kill()
        except:
            pass

    def next(self):
        return self.stdout.next().rstrip()

class batcher_module(object):
    r'''An extension to the default module object, which creates new
        command-calling functions as they are imported.
    '''
    def __init__(self, module):
        self.module = module

    def __getattr__(self, name):
        try:
            return getattr(self.module, name)
        except AttributeError:
            pass

        def batcher(*args, **options):
            return batch(name, args, options)

        setattr(self.module, name, batcher)
        return batcher

modules[__name__] = batcher_module(modules[__name__])

When a symbol is imported from the batcher module, the batcher_module object first checks whether it already exists. If it doesn't (which is most likely), the symbol is created and bound to a function that will run the correspondingly-named shell command when called. That function works by instantiating an object of the batch class, which inherits from Popen and adds the following customizations and extensions:

• By default both standard and error outputs are collected in the stdout attribute. This can be changed at instantiation time using the named arguments stdout and stderr;
• Whereas Popen objects try to remain alive for as long as the underlying process is active, batch objects work the other way around by killing the process (if not yet finished) when they're selected for garbage collection (the original behavior can be restored by passing the daemon named argument with a value of False);
• batch objects are iterable, returning the next (right-trimmed) line of output at each iteration;
• batch objects are also their own context managers, killing the underlying process (if not yet finished) when the context is exited;
• Finally, batch objects implement the close() method, which kills the underlying process if it hasn't yet finished, and does nothing otherwise.

Using the batcher module makes it possible to write Python scripts that seamlessly interface with shell applications, for example:

#! /usr/bin/env python
#coding=utf-8

# repo_dirty.py - scans a repo base for git projects containing non-commited changes

from re import search
from batcher import git, repo

def gitdirty(path):
    for line in git('status', cwd=path):
        if search(r'(On branch master)|(nothing to commit)', line) != None:
            print 'Project "%s" is dirty' % path
            return True

    return False

def repodirty():
    projects = 0
    dirty = 0
    for path in repo('forall', '-c', 'pwd'):
        projects += 1
        dirty += 1 if gitdirty(path) else 0

    print 'Finished checking dirty projects'
    print 'Total projects checked: %d' % projects
    print 'Dirty projects: %d' % dirty

def main():
    repodirty()

if __name__ == '__main__':
    main()

For the past few days I have been using this interface to rewrite some of my own shell scripts in Python, and I am startled with the results. So far it worked without a hiccup; being able to use Python on shell automation duties brought great improvements to performance and reliability, as well as my own personal satisfaction with the resulting code. Between Python's power and this newfound seamless integration to the shell environment, I now see seldom reason to ever bother writing a shell script again.

A Python Implementation of The Log-Polar Transform, Redux

One of my first posts to this blog presented a Python implementation of the log-polar transform. Back then I was just beginning to learn NumPy, and didn't know many of its more useful tricks; in particular, I had no domain of fancy indexing, and did not know that it could be used to avoid the need for time-consuming Python loops. Besides that, I later realized that the algorithm I chose to implement, taken from an article I had read at the time, was unnecessarily complicated.

A couple of months ago I was considering the use of the log-polar transform for another project, and thought I should go back and rewrite my early implementation. I wanted to make use of the NumPy expertise I've earned since then, but also clean up my early blunder and present a simpler algorithm, that could be easily ported to other languages. The results are available below, and also for download in a nice package, with test code and thoroughly commented sources.

The first listing presents a "naive" implementation, which relies on Python loops:

def logpolar_naive(image, i_0, j_0, p_n=None, t_n=None):
    (i_n, j_n) = image.shape[:2]
    
    i_c = max(i_0, i_n - i_0)
    j_c = max(j_0, j_n - j_0)
    d_c = (i_c ** 2 + j_c ** 2) ** 0.5

    if p_n == None:
        p_n = int(ceil(d_c))
    
    if t_n == None:
        t_n = j_n
    
    p_s = log(d_c) / p_n
    t_s = 2.0 * pi / t_n
    
    transformed = zeros((p_n, t_n) + image.shape[2:], dtype=image.dtype)

    for p in range(0, p_n):
        p_exp = exp(p * p_s)
        for t in range(0, t_n):
            t_rad = t * t_s

            i = int(i_0 + p_exp * sin(t_rad))
            j = int(j_0 + p_exp * cos(t_rad))

            if 0 <= i < i_n and 0 <= j < j_n:
                transformed[p, t] = image[i, j]

    return transformed

The second version constructs a pair of NumPy fancy indices first – cached for later reference – and then uses them to perform the transform:

_transforms = {}

def _get_transform(i_0, j_0, i_n, j_n, p_n, t_n, p_s, t_s):
    transform = _transforms.get((i_0, j_0, i_n, j_n, p_n, t_n))

    if transform == None:
        i_k = []
        j_k = []
        p_k = []
        t_k = []

        for p in range(0, p_n):
            p_exp = exp(p * p_s)
            for t in range(0, t_n):
                t_rad = t * t_s

                i = int(i_0 + p_exp * sin(t_rad))
                j = int(j_0 + p_exp * cos(t_rad))

                if 0 <= i < i_n and 0 <= j < j_n:
                    i_k.append(i)
                    j_k.append(j)
                    p_k.append(p)
                    t_k.append(t)

        transform = ((array(p_k), array(t_k)), (array(i_k), array(j_k)))
        _transforms[i_0, j_0, i_n, j_n, p_n, t_n] = transform

    return transform

def logpolar_fancy(image, i_0, j_0, p_n=None, t_n=None):
    (i_n, j_n) = image.shape[:2]
    
    i_c = max(i_0, i_n - i_0)
    j_c = max(j_0, j_n - j_0)
    d_c = (i_c ** 2 + j_c ** 2) ** 0.5
    
    if p_n == None:
        p_n = int(ceil(d_c))
    
    if t_n == None:
        t_n = j_n
    
    p_s = log(d_c) / p_n
    t_s = 2.0 * pi / t_n
    
    (pt, ij) = _get_transform(i_0, j_0, i_n, j_n, p_n, t_n, p_s, t_s)

    transformed = zeros((p_n, t_n) + image.shape[2:], dtype=image.dtype)

    transformed[pt] = image[ij]
    return transformed

Both implementations are based on a "reverse transform" algorithm: rather than iterate over the Cartesian image calculating where each pixel fits in the transform (which resulted in all sorts of complications), iterate over the transform's cells and calculate which, if any, pixel from the Cartesian image should be put there.

According to my tests, the "fancy" version using a cached pair of indices runs about 10 times faster than the "naive" version – but even when it has to construct the indices before using them, it's still slightly faster.

In Defense of Java Over C++

A while ago Tim Robertson elaborated on his opinion that Java compares rather unfavorably to C++, to the point it's not worth learning. His arguments are well known to the seasoned Java professional: limited features, poorer performance, and a sense of missing out on new developments and becoming more of a legacy platform (the infamous "new COBOL" argument).

I beg to disagree. I have worked - and by "work" I mean professional, 8-to-5 programming work - on both C++ and Java projects for years now. They're both very good languages, and when used together can realize great things (hello Android). That said, I do find quite a few advantages to Java over C++:

• Less cruft. This has become a little less true since 1.5 with "generics", enums and the like, but still Java remains much more concise a language than C++. Arguing that C++ can do everything Java does and then some completely misses this point - often Java is a better tool than C++ precisely because it does less, while also requiring less;
• A cleaner OO implementation. This is a continuation of my previous point but really deserves its own mention. I learned OO in C++ before getting to know Java, and I was relieved to see how much more straightforward Java's model was;
• A unified memory model. In C++ you have a dozen ways to manage memory - local variables, malloc / free, the myriad variants of new / delete, etc - while in Java you have a couple primitive numeric types and then everything is dynamically allocated with new and garbage collected;
• No invalid pointers. In Java an object variable is either null or pointing to a valid object. Whereas in C++ - well, do you feel lucky, punk?
• No need for header files. It may be just laziness, but I'm really bothered by having to define every functional symbol twice, decide whether to #include dependencies on the header or source file, and so on;
• Better namespace management. Java's package / import mechanism is a no-brainer, while C++'s namespace feature is more complicated, verbose, and incomplete - besides using symbols you still have to #include headers (did I mention headers suck?);
• A standard, cross-platform mechanism for dynamic linking built right into the language. Whereas in C++ building a dynamic library is a project in itself (and woe if the thing fails to load), in Java you get it for free, as far as manual labor is concerned;
• A more complete standard library. Sure, C++ is kind of catching up now - only that "now" actually means "a good ten years later".

What all this adds up to, is that even as C++ is in many respects a more powerful language than Java, programming in Java is a lot easier and quicker; things more often turn out right the first time around; and baffling, mysterious build and/or runtime issues are less frequent. Even when things do go haywire, the tight memory model and pervasive exception mechanism guarantee that you'll at least have a good idea as to where the problem manifested - compare this to C++ error handling, which often leaves programmers staring at a "segmentation fault" error message as though we were supposed to know from that what went wrong.

Because of that, Java is more productive for daily programming, which makes it the preferred tool whenever it is up to the task. Perhaps it's not so much so if you're in academia or NASA, but for contractors implementing custom systems, such ease of development is a pretty big deal: most contracts have far stricter schedule than performance constraints. Getting something without glaring bugs and with acceptable performance now is much more valuable than getting it perfect in two months time.

As for the "new COBOL" argument: first off, COBOL was only ever good to write business rules and the odd input form. It survives today at the expense of financial institutions who've grown so fat and lazy (or ignorant of their own processes) that rather than modernize their operations, they prefer to pay big bucks for a COBOL programmer and tools to keep everything as it ever was. Java on the other hand is all around, from home appliances and handsets to big computer clusters. I myself have used it for pretty much everything, from web service servers to serial port controllers and the odd AI gimmick. So please don't hold your breath for Java's collapse (or better yet, do hold your breath, by all means!) - it's not going to tail off any time soon, and surely no sooner than C++.

Haters gonna hate, but professionals still have to weight on the right tool for the job. Whether it's Java, C++ or something else (I personally I'm very fond of Python), it should be decided on the languages' relative strengths and the task at hand, and not on nerd bigotry.

How a Bash Script Can Know Where It Is

A colleague of mine wrote a shell script that is called from various other scripts, from different locations. He wanted his script to be able to discover the directory at which it is stored, but didn't know how; of course, calling pwd inside the script would yield the current working directory (by default, the directory from which the original script was called), not the script's directory.

After a little googling, I found this solution by netizen apokalyptik:

#!/bin/bash
LSOF=$(lsof -p $$ | grep -E "/"$(basename $0)"$")
MY_PATH=$(echo $LSOF | sed -r s/'^([^\/]+)\/'/'\/'/1 2>/dev/null)

MY_PID=$$
MY_ROOT=$(dirname $MY_PATH)
MY_NAME=$(basename $0)

echo -e "PATH\t$MY_PATH"
echo -e "FILE\t$MY_NAME"
echo -e "CWD \t$MY_ROOT"
echo -e "PID \t$MY_PID"

The trick lies in the use of the lsof command, which returns a list of currently open files. From there, the script uses grep to locate the entry corresponding to the currently running script, sed to separate the full path from the entry, then basename & dirname to split the base directory and file name.

This solution could be further enhanced by adding the current user to the grep regular expression: as it is, it could return the path to a same-named file opened by another user – but unless you work on a multi-user environment, it shouldn't make much difference.

To C Something in Less Words...

Thanassis Tsiodras makes a good case on C++ for C programmers. His main point – which I wholeheartedly share – is that C++ allows algorithms to be generalized in ways not easily available in C (if at all), whithout the performance hit found in most other alternatives.

Futhermore, both C programs and programmers can benefit from a more smooth transition, since the languages are mostly compatible at source-level. While it is my experience that, for new projects, one should not try to mingle C and C++ – it's better to go full C++ from the start, lests one risks creating an ugly mess of conflicting memory management paradigms – for ongoing projects hindered by the limitations of the C language, moderate doses of C++ may well be the right medicine.

Hacking configurable indent width into BeautifulSoup

Many a time I have been saved by the HTML parsing prowess of BeautifulSoup. When it comes to beautifying HTML dumps for easier analysis it is king; however I have always been bothered by its unorthodox choice of indentation width (just one space per level). So the other day I took the time to hack configurable indent width into the code for version 3.2.0.

The complete modified file is here; so long you're using version 3.2.0, you can just paste it over the original file at your installation. For details on the changes see the diff below:

***************
*** 544 ****
--- 545,547 ----
+         # Adjustable indentation patch
+         self.indentWidth = parser.indentWidth
+ 
***************
*** 752 ****
!             space = (' ' * (indentTag-1))
--- 755 ----
!             space = (self.indentWidth * (indentTag-1))
***************
*** 813 ****
!                     s.append(" " * (indentLevel-1))
--- 816 ----
!                     s.append(self.indentWidth * (indentLevel-1))
***************
*** 1080,1082 ****
!     def __init__(self, markup="", parseOnlyThese=None, fromEncoding=None,
!                  markupMassage=True, smartQuotesTo=XML_ENTITIES,
!                  convertEntities=None, selfClosingTags=None, isHTML=False):
--- 1083,1088 ----
!     def __init__(
!         self, markup="", parseOnlyThese=None, fromEncoding=None,
!         markupMassage=True, smartQuotesTo=XML_ENTITIES,
!         convertEntities=None, selfClosingTags=None, isHTML=False,
!         indentWidth=' '
!     ):
***************
*** 1111 ****
--- 1118,1121 ----
+ 
+         # Adjustable indentation patch
+         self.indentWidth = indentWidth
+ 

Now setting a custom indent width is as simple as doing:

soup = BeautifulSoup(html_string.encode('utf-8'), indentWidth='    ')
beautiful_html_string = soup.prettify()

Making numpy ndarray's Hashable

I am doing some research work using numpy and scipy, and I have been amazed by how fast they run; the other day, though, I stumbled on what seemed a roadblock. I wanted to use these quite large (length 1024) arrays as keys for storing data in Python dictionaries, however numpy's ndarray objects are not hashable; converting them to tuples would take forever, besides eating up an unbelievably large amount of memory. Searching the web provided no satisfactory answers.

After a while, though, I realized that it would be simple to implement a hashable wrapper for my array objects. So here you have it:

from hashlib import sha1

from numpy import all, array, uint8


class hashable(object):
    r'''Hashable wrapper for ndarray objects.

        Instances of ndarray are not hashable, meaning they cannot be added to
        sets, nor used as keys in dictionaries. This is by design - ndarray
        objects are mutable, and therefore cannot reliably implement the
        __hash__() method.

        The hashable class allows a way around this limitation. It implements
        the required methods for hashable objects in terms of an encapsulated
        ndarray object. This can be either a copied instance (which is safer)
        or the original object (which requires the user to be careful enough
        not to modify it).
    '''
    def __init__(self, wrapped, tight=False):
        r'''Creates a new hashable object encapsulating an ndarray.

            wrapped
                The wrapped ndarray.

            tight
                Optional. If True, a copy of the input ndaray is created.
                Defaults to False.
        '''
        self.__tight = tight
        self.__wrapped = array(wrapped) if tight else wrapped
        self.__hash = int(sha1(wrapped.view(uint8)).hexdigest(), 16)

    def __eq__(self, other):
        return all(self.__wrapped == other.__wrapped)

    def __hash__(self):
        return self.__hash

    def unwrap(self):
        r'''Returns the encapsulated ndarray.

            If the wrapper is "tight", a copy of the encapsulated ndarray is
            returned. Otherwise, the encapsulated ndarray itself is returned.
        '''
        if self.__tight:
            return array(self.__wrapped)

        return self.__wrapped

Using the wrapper class is simple enough:

>>> from numpy import arange
>>> a = arange(0, 1024)
>>> d = {}
>>> d[a] = 'foo'
TypeError: unhashable type: 'numpy.ndarray'
>>> b = hashable(a)
>>> d[b] = 'bar'
>>> d[b]
'bar'

In my profiling sessions, adding the wrapped-up 1024-long arrays as keys to a dictionary amounted to no more overhead than adding the naked arrays to a list.

A Python API for Lego MINDSTORMS NXT Alpha Rex Design

Update: a properly-commented version of the code below made it to the nxt-python repository. Yay me! :P

Last week a friend of mine asked me to take his Lego MINDSTORMS NXT kit, build the Alpha Rex design, and then demonstrate it to his father, who was a few days from returning to his home in Japan. Needless to say, having a chance to once again play with Lego – and in particular with the mind-blowingly nerdy, robotics-enabled NXT – was hardly unpleasant. In fact, while I was building the iconic bipedal robot – carefully and patiently following the instructions, taking the time to visualize how the pieces would fit together before actually connecting them, building the major parts before assembling them into the final model – I recognized just how similar to playing with Lego programming is, and how, during all those years of playtime, my mind was shaped for my current career in computing.

So it's no surprise that when it came to putting Alpha Rex to work, I snubbed NXT's Windows-based GUI tools, and looked instead for some way to program it from my Linux netbook, preferably in Python. Sure enough, after a little Googling, I stumbled at nxt-python, "a Python driver/interface for the Lego Mindstorms NXT robot". Though at first I was a little discouraged by the motor interface (I was expecting something more in line with NXT-G's, where you can send commands to up to all three motors simultaneously), eventually I was able to find my way around the API, and even come to admire its simplicity and straightforwardness.

The best part about nxt-python is Python itself. In just a couple hours I was able to write a rather sophisticated, object-oriented API to drive Alpha Rex and sample its various sensors – all in under 100 lines of code:

# alpharex.py

from time import sleep

from nxt.brick import Brick
from nxt.locator import find_one_brick
from nxt.motor import Motor, PORT_A, PORT_B, PORT_C
from nxt.sensor import Light, Sound, Touch, Ultrasonic
from nxt.sensor import PORT_1, PORT_2, PORT_3, PORT_4


FORTH = 100
BACK = -100


class AlphaRex(object):
    def __init__(self, brick='NXT'):
        if isinstance(brick, basestring):
            brick = find_one_brick(name=brick)
    
        self.brick = brick
        self.arms = Motor(brick, PORT_A)
        self.legs = [Motor(brick, PORT_B), Motor(brick, PORT_C)]

        self.touch = Touch(brick, PORT_1)
        self.sound = Sound(brick, PORT_2)
        self.light = Light(brick, PORT_3)
        self.ultrasonic = Ultrasonic(brick, PORT_4)

    def echolocate(self):
        return self.ultrasonic.get_sample()
    
    def feel(self):
        return self.touch.get_sample()

    def hear(self):
        return self.sound.get_sample()

    def say(self, line, times=1):
        for i in range(0, times):
            self.brick.play_sound_file(False, line + '.rso')
            sleep(1)

    def see(self):
        return self.light.get_sample()

    def walk(self, secs, power=FORTH):
        for motor in self.legs:
            motor.run(power=power)
        
        sleep(secs)

        for motor in self.legs:
            motor.idle()

    def wave(self, secs, power=100):
        self.arms.run(power=power)
        sleep(secs)
        self.arms.idle()


def wave_and_talk():
    robot = AlphaRex()
    robot.wave(1)
    robot.say('Object')


def walk_forth_and_back():
    robot = AlphaRex()
    robot.walk(10, FORTH)
    robot.walk(10, BACK)


def walk_to_object():
    robot = AlphaRex()
    while robot.echolocate() > 10:
        robot.walk(1, FORTH)

    robot.say('Object.rso', 3)


if __name__ == '__main__':
    wave_and_talk()

Playing with Alpha Rex was real fun, but now I'm afraid my next move (since my friend allowed me to keep the kit a while longer) will be to dismantle it. Unable to turn around or move its ultrasound sensors, it's rather limited in its navigation capabilities – and I'm dying to check out whether I can build a SLAM-capable robot out of the basic kit. At any rate, it'll certainly be pleasant to try.

Finding a Laser Dot in An Image with OpenCV

Earlier today a friend of mine, who is working on an OpenCV project, asked me for a good way to locate a laser pointer in an image of a blank wall. I showed him two options:

1. The function cvMatchTemplate() generates a likelyhood map for the position of a template within a larger image. See this tutorial for a sample program;
2. Alternatively, since the searched object (the laser dot) distinguishes itself from the background by a primary color's (Red) concentration, it may be simpler to just isolate the image's Red channel with cvSetImageCOI(), then use cvMinMaxLoc() to return the position of the biggest-valued pixel:

/*
Returns the most likely position of a laser dot within the input image. This is
defined as the position of the pixel with the biggest value in the Red
channel.

Arguments:

image
  Pointer to the input image, assumed 32-bit RGB.

dot
  Pointer to the CvPoint instance to receive the output location.
*/
void cvSearchLaserDot(IplImage* image, CvPoint* dot) {
  double minVal, maxVal;

  // Select's the image's first channel. Assuming this is an RGB image, the
  // Red channel will be selected.
  cvSetImageCOI(image, 1);

  // Looks for the biggest-valued pixel on the above selected channel.
  cvMinMaxLoc(image, &minVal, &maxVal, NULL, dot, NULL);

  // Resets the channel selection performed in the previous call to
  // cvSetImageROI().
  cvSetImageCOI(image, 0);
}

I did think of a third option – fitting a 2D gaussian function to the Red channel, then taking the gaussian's center as the location of the dot – but that would be a bit of an overkill, wouldn't it?

Never Found A Programming Language I Couldn't Love

Notice: this was originally posted to the CodeProject's Lounge in 2010-10-25, and featured in the Daily News the next day. Yay me! :P

Some say that "when the lady can't dance, she complains the band can't play". Generally that's my feeling towards complaints that such-and-such language is "bad", or that it produces "bad" code.

I do agree that some languages lend themselves more than others to certain good (or bad) practices, and that some languages are better/worse suited than others for some problems. However, in my experience the most crippling problems are the result of poor programming style: the respective programmer community either doesn't know, or seems to forget how to write clear code once they join in.

As way of example, this is my list of good practices for several widely-belittled languages. It's mostly about how you can use each language's specific features to uphold the principles of structured / OO programming:

Visual Basic

• Always use Option Explicit for enforcing variable declaration;
• Shun Variant variables – always use definite types;
• Horribly misnamed as they are, embrace "Classes" as the way to go for behavior encapsulation;
• Use the Object type sparingly, but learn to recognize where it can be leveraged for generalizing algorithms;
• Well employed, the On Error machinery can make do as an effective Exception system;
• Prefer Collections over Arrays, and learn to explore their associative features.

C

• Just because the language doesn't enforce a type-oriented form of programming doesn't mean you shouldn't. Always think about problems in terms of data types and accompanying operation (function) sets;
• Structure your code as a collection of header / source pairs, where the header defines (ideally) one data type and its API, and the source contains the implementations;
• Create more complex types by composing simpler ones, and likewise implement their API's in terms of the lower types' interfaces;
• When designing a type's API, remember to provide functions for dynamic instantiation, initialization of stack-defined instances, and deletion. Consider providing macros for access to struct fields, instead of conceding to direct dereference;
• Preserve the global namespace by defining symbols as static whenever they don't have to be seen outside their compilation units;
• Use function pointers for generalizing complex algorithms;
• Though dreadful when used carelessly, macros have great potential for simplifying programming interfaces – know when and how to use them.

C++

• Forget struct's, always use classes;
• Forget memory buffers and pointer arithmetic, use the STL collections instead;
• Prefer inline functions to macros;
• Avoid using pointers whenever possible. Use references for function arguments, inline definitions for class members, STL collections for dynamic instantiation, and locally-defined variables for automatic memory management;
• Leverage variable scope control for automating common operations, as logging function calls;
• C++ exception support is poor, but it's better than nothing – learn to take advantage of it;
• In general terms, do not stand with one foot on C and the other on C++ – jump in with both feet.

Scripting Languages (Perl, Javascript, PHP, etc)

• Just because you're writing to a script language doesn't mean you should forget how to program – structure your code around data types and/or classes, concise functions and specialist modules, just as you'd do on a "proper" systems programming language;
• Do not mingle your code to HTML. Write everything on separate modules; where inline code can't be avoided, provide as concise an interface as possible, ideally a single function call.

What about you? Do you think there are inherently "bad" languages, or is it essentially a matter of poor practice?

WAC On Track to Deliver

The Wholesale Applications Community (WAC) is the heir to BONDI and the Joint Innovation Lab (JIL), which in turn were mobile application platforms based on extended versions of W3C's Web Widgets standard. The idea is enabling the use of web technologies (HTML, CSS and Javascript) to write full-blown mobile apps, as well as small homescreen-hanging "widgets". This is achieved by providing a web client runtime environment with an extended Javascript API, allowing access to underlying handset functions such as soft keys, camera, messaging system etc – though in my experience you can go a long way without ever crossing the limits of standard web programming.

Ever since its inception early this year there has been quite reasonable doubts on WAC's ability to deliver. Following Android's avalanching success and a surge of me-too mobile platforms – Samsung's BADA, HP's plans for webOS, Microsoft's Windows Phone 7 and the unnamed (and apparently stillborn) DOCOMO-led platform were all announced around the same time – the whole enterprise seemed somewhat reactive. Add the mobile network operators' (WAC's principal sponsors) historical inability to agree on anything and poor innovation record, and it looked highly unlikely that WAC-enabled handsets would ever see the light of day, let alone take on already established smartphone platforms.

It seems, however, that the rumors of WAC's unavoidable doom were highly exaggerated. Not only has the organization found consistent support all across the mobile industry, it has recently launched version 1.0 of its runtime specification (though I feel this was rather rushed, as just two weeks ago it was still requesting comments for an alleged "first draft"). Since both BONDI and JIL managed to get shipped into actual feature phones before their somewhat forceful merger, I'd say there's now a definite chance we'll see WAC-enabled phones in the wild sooner or later.

And I say the sooner the better. I have recently spent a couple weeks working on a proof-of-concept mobile widget for the BONDI platform, and I loved it. As it's mostly web programming with a couple extra Javascript objects, there is virtually no learning curve; it's also very portable, as whatever extensions you use can be abstracted into a Javascript file and swapped out when you want to run it on a desktop browser (which I actually did, as a shortcut for testing during development). Deployment was quite troublesome, as I had to upload the packaged application to the handset manufacturer's web portal, sign it, and then download it back; but overall the experience was very positive.

Despite all the hype around smartphones, feature phones still dominate the market. Currently JME and Flash Light rule its app development roost; however, with the right promotion the WAC could very well topple them in no time, appealing as it would to the current generation of web-savvy developers with a platform they not only already know how to program, but for which they could effortlessly port already existing applications. If only the WAC associates can take it to the next step and provide enabled devices and developer support in a timely fashion, this might just be the beginning of the open, cross-vendor, cross-platform mobile application ecosystem we've been waiting for ever since the first days of WAP.

Real-world Functional Programming

There is a recurrent discussion in programming circles about the state-of-practice of functional programming languages, and the (presumably undeserved) little attention they get. It's curious to me that such arguments are usually centered on so-called "pure functional" languages such as Lisp, Scheme or Haskell (which isn't that "pure", but anyway), while totally ignoring how functional programming features have been consistently pouring into the mainstream programming landscape for more than a decade, through comparatively conservative efforts such as Javascript and Python.

As a matter of fact, I do functional programming in Python a lot; only not right away. Often, when working on a problem, I'll start with a simple imperative solution based on top-level functions. Then, as the domain creeps ever larger and passing around values becomes awkward, I factor in a basic object model with only essential behavior moved to methods, while the bulk of processing remains on the functions.

Finally, as recurring algorithms emerge, I start to refactor them into general implementations, customizable through object or function arguments. Eventually, most of my code is collected into highly general "portfolio" functions, while the "real" functions are mostly constructed by mixing and matching those. As a side-effect, closures and lambda functions abound.

This approach doesn't have much of the obsessive recursivity found in classical functional programming (although I've at times marveled at the mind-blowing length of the function invocation chains it produces), but it does rely on the first-citizenship status Python grants to functions – in fact it wouldn't work otherwise. It isn't "pure" functional, as it isn't "pure" OO either; but it enables widespread code reuse, naturally leans toward cohesion & loose coupling, and is highly productive.

Purity is a fool's errand. Just as it's not practical to have a "pure" object-oriented language (even Java has allowed for something akin to top-level function with static methods and imports), successful functional languages will have to compromise in order to accommodate its design features and the pragmatic needs of daily coding.

Object Models of Microsoft Office

Despite its hideous looks and varied shortcomings, the Component Object Model (COM) technology from Microsoft does make interfacing to Windows applications a lot easier. Notably, programmatic manipulation of Microsoft Office files (Word documents and Excel spreadsheets, mostly) is almost always implemented by way of automation – i.e. driving the corresponding Office application through its COM API to perform the desired actions.

COM is language-agnostic and widely supported, making it a readily-available option in most programming environments. Accordingly, the web is littered with tutorials on the basics of setting up a COM session and performing simple MS Office automation tasks; almost anything available on Windows is covered, from C++ through C# all the way into Python. However, a more in-depth API reference is often missing; small articles, forum posts and code snippets can be found for any one particular topic, but learning even moderately complex tasks can quickly turn into a scavenging hunt for bits and pieces of information scattered across a dozen websites.

Microsoft Office Development with Visual Studio is MSDN's misleadingly-titled article on Office automation. It was last updated in 2001, so it even lacks a .NET section; despite that, it remains an invaluable resource, as it contains one of the most complete Object Models of the Office suite still available – and those seldom changed (if at all) since then. It's still not the definitive reference one would hope for – I had trouble finding information on how to open an Excel file, and what I did eventually find was barely enough – but it's certainly a very valuable addition to the information pool.

DeWBXML: Simple WBXML Decoder

At work I have a recurrent need to decode WAP Binary XML (WBXML) files, to check whether problems with network provisioning, WAP page rendering etc. are the handset's or the content provider's fault. Doing it by hand is tedious and error-prone, and I couldn't get the WBXML Library to work under Windows; so after a while I decided to write my own WBXML decoder.

DeWBXML is a Python application for decoding WBXML files. It was written with extensibility in mind: applications (e.g. WAP provisioning, WML, Push) are specified in Python using a style reminiscent of JSON, and plugged into a generic WBXML parser. It doesn't give much attention to XML namespaces or DTD's, so it's relatively forgiving on well-formedness.

DeWBXML was created out of personal need for a simple, convenient WBXML decoder. As such, development follows a rather inconstant pace; new features are added only as their necessity arises. Contributions and bug reports are welcome, but no guarantees are made on responses, timely or otherwise.

Reverse-engineering the XMF Audio Format

Some time ago I was investigating a bug in a mobile media player, that crashed when it attempted to play certain (not all) XMF audio files. In order to determine why some files crashed the player while others didn't, I decided to inspect their contents – only to discover no matter how hard I tried, I couldn't find any information about the XMF format on the web.

Eventually I was able to wrangle enough information from the player's source code to build a simple XMF inspector, which allowed me to quickly determine the differences between playable and crash-inducing files. After that, I thought that, while XMF is a pretty obscure format these days, that information might be useful for others – it would certainly have made things easier for me, had it been readily available back then. So I wrote an article containing the results of my research, and published it to the MultimediaWiki.

You can read the article here. As I said, I don't expect it to be a hit, but I hope others will find it useful at some point.

Clay Programming Language

Recently we've seen an upsurge of interest in new programming languages. The latest addition to the pack is Clay, which intends to marry C's tight memory model and small runtime overhead to generic programming semantics and a concise syntax. It's a tall order, but it's long past due someone took it.

There is no particular reason why modern features such as dynamic array instantiation, managed strings and a module system (to name a few deeply-felt C absences) couldn't be efficiently integrated into a native programming language. For a while now I've felt the need for an "updated C" that didn't represent a radical departure from the original, but brought it up to speed with the state-of-practice; Clay may or may not turn out to be that language, but in that respect I find quite encouraging its dismissing (up to this point) of classic object-orientation in favour of a simpler ADT-based model.

Details on Clay are sparse at this time (the project page provides no documentation whatsoever, only a terse feature list and some example programs), but a fully functioning compiler is already available for download. It will be interesting to see how far it goes, and whether other languages will follow on its theme of native-compiled languages with high-level abstraction features.