/home/jeff/blog

Archive for the ‘Development’ Category

I have been recently been trying to redesign FreeMED (my opensource GPL’d EMR/PM system) in order to work in a “clustered” environment, so that I could support scenarios where multiple application servers were load-balanced to handle larger quantities of traffic. The latest piece of this has been to move filesystem-based storage into the database layer so that I don’t have to mess around with clustered file storage and replication. Some of the highlights of this have been:

• Database-based session handling – At least when dealing with stateless applications like FreeMED, session information has a bad habit of being confined to the local filesystem. We ended up using PEAR’s HTTP_Session2 to support database-backed sessions. I can’t help but think this would be so much easier using JAAS …
• API abstraction – Don’t write this one off too quickly. A good API layer means that whenever you have to make these sweeping changes, you’re only playing with a few files of source code, as opposed to completely rewriting a good chunk of your codebase. Without the few layers of API which I had been using, I would not have been able to implement the database backed information store which I just completed.
• Database backed file store – This was the part I had just finished up. Storing files locally and relying on filesystem-based replication from some of the new clustering filesystems may *seem* like a good idea, but it’s not. Database gives you a single authoritative source, and unless we’re looking at multi-gigabyte files, there should not be an appreciable toll. Besides, you indexed your tables properly, right … ?
• Design for concurrency – That record locking may seem like a pain, but two people working on the same object can be catastrophic. Take the time to add a locking field if you have to.
• Singletons – Take “singleton” processes into account, so you’re only running a single iteration of something which can only be run once.

Those are the major bullet points, although I’m sure that upon review of this, I’ll think of some other design paradigms I had to play with to get this to function properly. Off to enjoy the weekend …

In 2007, I had posted a patch for the ADS DVD Xpress DX2 device to work on Linux, but it had been based on an antiquated kernel version, etc.

Since then, someone was nice enough to post an updated version of the driver, but without DVD Xpress DX2 support. I put together a patch which ensures that the drivers now compile and use the new I2C and V4L2 APIs. I can’t guarantee that it works, only that it compiles the driver properly now. Theoretically it should work, but I can’t find my DVD Xpress DX2 to try out the hardware properly.

Original driver : http://go7007.imploder.org/wp-content/uploads/2009/11/wis-go7007-linux-098-5b.tgz
Patch : http://jeff.ourexchange.net/wp-content/uploads/2010/03/wis-go7007-linux-098-5-20100325.diff

I have just wrapped up (no pun intended) work on an initial version of a CLI JasperReports wrapper, based heavily off the work of jasperCall. It’s also quite similar to the work being done on RunJasperReports, although it was specifically designed to be integrated into FreeMED’s reporting engine, as it is put together as a fatjar. It currently supports PDF, XML, XLS and HTML output, and should theoretically support parameter passing, though I haven’t tried it out yet. It has a newer version of MySQL Connector/J baked in as well, so it shouldn’t require any external libraries outside a standard JRE install to function properly.

The code can be check out of anonymous Subversion from https://svn.master.freemedsoftware.org/freemed-utilities/org.freemedsoftware.util.JasperWrapper/ if anyone is interested in building it. (It obviously should require Java 6 JDK to compile properly.) Hopefully I’ll get around to tagging and releasing versions of it for download shortly, if anyone shows interest in it.

The simple recipe is to add this to root’s cron:


* * * * * /usr/bin/gmetric -c /etc/gmond.conf -n pf_states -v $(/usr/local/sbin/pftop -b | grep pfTop | cut -d/ -f2 | cut -d, -f1) -t int32 -d 120 2>&1 | logger -t pf_states


and install the pftop package along with a gmetric binary and a working /etc/gmond.conf configuration file. It might be advantageous to check for the maximum number of states as well.

In addition, you might want to know which pf rules are passing how much traffic. A nice easy way of doing this is to create this file as ./pfstates (make it executable, of course):

#!/usr/bin/perl
# pfstates
# jeff@ourexchange.net
my $limit = shift  || 0;

my $seg = 0;
my @s = [];

while (chomp( my $line = <>)) {
        $s[$seg] = $line;
        if ($seg == 2) {
                $seg = 0;
                if ($s[1] !~ /States\: 0/) {
                        my $states = 0;
                        if ($s[1] =~ m/States\: (\d+)/) {
                                $states = $1;
                        }
                        if ($states >= $limit) {
                                print "[$states] $s[0]\n";
                        }
                }
        } else {
                $seg++;
        }
}

…. then you would pipe pfctl’s state output to it:

pfctl -v -s rules | ./pfstates

Optionally you could add a “minimum level” of connections you want to see:

pfctl -v -s rules | ./pfstates 100

for example to see only rules passing 100 or more active connections.

I have a hate/hate relationship with Exchange Server. I hate it, and I’m pretty sure it hates me.

Why someone would design a system to expose every bit of data for a system through a nice standard protocol like IMAP, then only allow certain things to be viewed through a piece of crap proprietary protocol like MAPI just boggles the mind. I’m sure it’s part of their “vendor lock-in” thing, but it just pisses me off.

Anyway, I found out that to completely backup an Exchange account (as I have been doing over the last week or so for different accounts), you also have to backup the non-mail portions. I ended up using a deprecated library called Jakarta Slide for WebDAV client support, which helpfully came with a SearchMethod call which was capable of running the specialized XML query required to backup the data.

The XML I ended up using was:

<?xml version="1.0"?>
<a:searchrequest xmlns:a="DAV:">
<a:sql>
SELECT * FROM "URL"
</a:sql>
</a:searchrequest>

in case anyone is interested. Again, I have a fatjar of this utility, but I have to check with work to make sure they’re okay with me releasing it before I can post it anywhere.

I hate it. IMAP Synchronization, that is.

In an effort to migrate users from one *shudder* Exchange provider to another (after getting shot down for proposing first Zimbra, then standard mail server stuff, then Openchange), I have been going through all of the available IMAP sync software that I could find.

mbsync (http://isync.sourceforge.net/mbsync.html) – We use this for IMAP backup, so I figured it would be a good idea to try it for syncing between two IMAP servers. *Crash*. *Segfault*. This is one of the reasons I’m not a big fan of really low-level apps written in C.

imapsync (http://www.linux-france.org/prj/imapsync/) – Written using a Perl library, this offered the promise of fast syncing, and appeared to work in test runs. That is, until it encountered folders with spaces in them, and threw up all over the terminal. Really, it wasn’t a use-case for *anyone* to have spaces in folder names? I could have put some effort into fixing the library and/or script, but I figured that if something that basic wasn’t working for a fairly mature library, this wasn’t going to be pretty.

The Imap Migration Tool (http://migrationtool.sourceforge.net/) – Written in PHP, with a two or three step process. Was meant for moving stuff, not keeping them in sync, so running multiple times would cause additional copies of messages to be created. Also, it seems to have been written in some prehistoric form of PHP, which required hacking to get it even marginally functional. Even then, it spat tons of warnings and nasties, which made me a little nervous to use it on any important mailboxes.

mailsync (http://mailsync.sourceforge.net/) – I don’t even remember why this one didn’t work, only that I wasted a fair amount of time playing with it to get it to try to migrate using the scheme we were working with to no avail. Scratch another one.

Solution … I had to roll my own. This is *stupid* to the highest degree, but I rolled something using javamail into a self-contained “fat jar” using fatjar. It works sickeningly well, and actually respects existing messages by message id, so that if for any reason I have to break the sync process, it’ll skip past anything it has already done. I made it non-destructive because it seemed like seriously bad juju to have a sync tool purge out email…. If I get permission from work, I’ll post the jar for anyone with Java 1.6 to use.

As an aside, it took a little bit of hackery to get fatjar running outside eclipse. The relevant section of build.xml was:

  <path id="fatJarPath" location="lib/fatjar.jar"/>
  <property name="ant.reuse.loader" value="true"/>

  <taskdef name="fatjar.build" classname="net.sf.fjep.anttask.FJBuildTask" classpathref="fatJarPath"/>
  <typedef name="fatjar.manifest" classname="net.sf.fjep.anttask.FJManifestType" classpathref="fatJarPath"/>
  <typedef name="fatjar.exclude" classname="net.sf.fjep.anttask.FJExcludeType" classpathref="fatJarPath"/>
  <typedef name="fatjar.jarsource" classname="net.sf.fjep.anttask.FJJarSourceType" classpathref="fatJarPath"/>
  <typedef name="fatjar.filesource" classname="net.sf.fjep.anttask.FJFileSourceType" classpathref="fatJarPath"/>

  <target name="main" depends="build">
    <fatjar.build output="ImapSynchronizer.jar">
      <fatjar.manifest mainclass="...myclasspath...ImapSynchronizer"/>
      <fatjar.filesource path="bin" relpath=""/>
      <fatjar.jarsource file="lib/mail-1.4.3.jar" relpath=""/>
    </fatjar.build>
  </target>

I have recently had cause to proxy a J2EE CXF service through an apache 2.2 instance, and thought it would be nice to share my findings. (This was all done on a Debian system.)

First of all, the mod_proxy pieces have to be enabled using a2enmod proxy.

A fragment has to be added with the proxying bits and some limitation:

<Proxy proxy:http://SERVER:PORT/URL>
     Order allow,deny
     Allow from all
</Proxy>

ProxyPass /EXPOSEDURL http://SERVER:PORT/URL
ProxyPassReverse /EXPOSEDURL http://SERVER:PORT/URL

Reloading apache configuration should enable the proxy properly. The only other possible issue is that in addition to the WSDL URL, you’re going to have to specify a “proxy” URL, which is just the service URL without ?wsdl at the end of it. A fragment of PHP’s SoapClient would look like this:

$url = "http://server.domain:8180/services/MySoapService?wsdl";
$sc = new SoapClient( file_get_contents($url), array(
          'login' => $my_username // optional for basic auth
        , 'password' => $my_password // optional for basic auth
        , 'compression' => SOAP_COMPRESSION_ACCEPT | SOAP_COMPRESSION_GZIP
        , 'location' => str_replace('?wsdl', '', $url) // force this to work through proxies
));

The magic in that last segment was the “location” parameter, as it specifies the proxy. Good luck!

I really like Ubuntu — I use it for all of my workstations/laptops, and have for the last several years. I do not, however, like it when people tell me that something that’s clearly and demonstrably a bug in a system simply isn’t a bug. Fie on your crappy PPA build system, I’m just not going to build more Ubuntu packages for things. I’ll just go back to only packaging Debian server packages.

Way to build community, Canonical.

I quote:

The buildds[sic] have no internet access, so you can’t use installer packages (packages which need to download the undistributable archive from the official location) on buildds (e.g. as build dependencies).

I swear, it’s a feature, not a bug. </sarcasm>

To sum up before I even begin: they *suck*.

One of Java’s exceptionally weak points is serial and parallel interfaces, since it’s not within the normal bundled class library specification. In developing for Linux systems, I found that there are two basic routes you can take, being Java Communications API and RXTX, with the latter being a reimplementation/extension of the former. To say that these are painful to work with doesn’t adequately state the gravity of the situation.

For example, the javax.comm.properties file needed by both libraries has to exist in the classpath or buried deep in the JRE. This is an offshoot of the issue that they require JNI libraries to function, which debian handles nicely in the form of a librxtx-java package which does most of the heavy lifting with the exception of that nasty properties file. In dealing with a J2EE app, this becomes a pain if you want to stick with the “single file” WAR deployment. Eventually I baked an additional instruction into my build.xml file to relocate the properties file on execution.

The documentation for rxtx, at least in terms of practical examples, is horrid. The wiki server isn’t up and running (at least when I went to look) and the only evidence that anyone had ever tried to use this library for the purposes I was looking to use it for were a few old javadocs and a handful of mailing list entries.

This really should *not* be this painful. Virtually every system targeted by the JRE has I/O devices which should be covered by the scope of the basic class libraries. I should not have to practically reinvent the wheel simply to push data out to a peripheral.

By the look of the documentation, this wouldn’t have been any easier doing it with Windows, and I believe I managed to kill its ability to be cross-platform in any meaningful sense when I decided to use RXTX. This is just as bad as those asshats who make completely crossplatform applications then tie them intrinsically to Windows in some completely brain-dead way for no apparent reason.

Hey Sun, when you made this specification, could you have possibly thought to include support for devices which have been around since 1969 …

I’ve been in the process of teaching myself the Google Android platform over the last few weeks, as my work has had a contest devoted to creating a mobile phone app, with a prize attached. I’m the only entrant who doesn’t work in the Engineering department, so this is a bit of a professional stretch for me as far as my job is concerned.

I have found quite a few caveats in the development process which are worth noting, as I have spent a fair amount of time researching them to figure them out. (I should also note at this point that this is in no way downplaying the important role Android plays in the future of mobile phone platforms, and is definitely *not* advocating either WinMo, which is a horrible abortion of a platform, or iPhone, which is not only locked to a single vendor, but *requires* purchasing a Crapitosh for development as well as a minimum 99$ development license to get going …)

1) Camera Access – It took some digging to find that you have to allow permission android.permission.CAMERA, or else it’ll throw out of memory errors and crash the app. I think it’s a running bug that in Android 1.6 it isn’t throwing a proper “PermissionDeniedException” or whatever else they feel like calling it. Otherwise the Camera API is pretty powerful.

2) Getting your own phone number – This is very interesting, as Android just moved to using something called “TelephonyManager” to deal with all of this. Unfortunately documentation hasn’t quite caught up to it. An example:

TelephonyManager mTelephonyMgr = (TelephonyManager)context.getSystemService(Context.TELEPHONY_SERVICE);
String imei = mTelephonyMgr.getDeviceId();
String phoneNumber=mTelephonyMgr.getLine1Number();
String country = mTelephonyMgr.getNetworkCountryISO(); 

3) Saving state – If you want to save state for your app, it makes no sense to go any further than SharedPreferences. It keeps you from having to create a content provider to deal with keeping that information stored somewhere else …

4) AppWidgets are awesome — The idea of a “home screen” applet is really easy to implement simply by extending AppWidgetProvider and add a <receiver> XML element to the AndroidManifest.xml file corresponding with the class.

5) Opening an intent from a menu item – The documentation says to use *a* Context, but doesn’t mention that you should be passing the Activity subclass that you’re coding to use as the Context.