/home/jeff/blog

Archive for the ‘Development’ Category

I have just wrapped up (no pun intended) work on an initial version of a CLI JasperReports wrapper, based heavily off the work of jasperCall. It’s also quite similar to the work being done on RunJasperReports, although it was specifically designed to be integrated into FreeMED’s reporting engine, as it is put together as a fatjar. It currently supports PDF, XML, XLS and HTML output, and should theoretically support parameter passing, though I haven’t tried it out yet. It has a newer version of MySQL Connector/J baked in as well, so it shouldn’t require any external libraries outside a standard JRE install to function properly.

The code can be check out of anonymous Subversion from https://svn.master.freemedsoftware.org/freemed-utilities/org.freemedsoftware.util.JasperWrapper/ if anyone is interested in building it. (It obviously should require Java 6 JDK to compile properly.) Hopefully I’ll get around to tagging and releasing versions of it for download shortly, if anyone shows interest in it.

The simple recipe is to add this to root’s cron:


* * * * * /usr/bin/gmetric -c /etc/gmond.conf -n pf_states -v $(/usr/local/sbin/pftop -b | grep pfTop | cut -d/ -f2 | cut -d, -f1) -t int32 -d 120 2>&1 | logger -t pf_states


and install the pftop package along with a gmetric binary and a working /etc/gmond.conf configuration file. It might be advantageous to check for the maximum number of states as well.

In addition, you might want to know which pf rules are passing how much traffic. A nice easy way of doing this is to create this file as ./pfstates (make it executable, of course):

#!/usr/bin/perl
# pfstates
# jeff@ourexchange.net
my $limit = shift  || 0;

my $seg = 0;
my @s = [];

while (chomp( my $line = <>)) {
        $s[$seg] = $line;
        if ($seg == 2) {
                $seg = 0;
                if ($s[1] !~ /States\: 0/) {
                        my $states = 0;
                        if ($s[1] =~ m/States\: (\d+)/) {
                                $states = $1;
                        }
                        if ($states >= $limit) {
                                print "[$states] $s[0]\n";
                        }
                }
        } else {
                $seg++;
        }
}

…. then you would pipe pfctl’s state output to it:

pfctl -v -s rules | ./pfstates

Optionally you could add a “minimum level” of connections you want to see:

pfctl -v -s rules | ./pfstates 100

for example to see only rules passing 100 or more active connections.

I have a hate/hate relationship with Exchange Server. I hate it, and I’m pretty sure it hates me.

Why someone would design a system to expose every bit of data for a system through a nice standard protocol like IMAP, then only allow certain things to be viewed through a piece of crap proprietary protocol like MAPI just boggles the mind. I’m sure it’s part of their “vendor lock-in” thing, but it just pisses me off.

Anyway, I found out that to completely backup an Exchange account (as I have been doing over the last week or so for different accounts), you also have to backup the non-mail portions. I ended up using a deprecated library called Jakarta Slide for WebDAV client support, which helpfully came with a SearchMethod call which was capable of running the specialized XML query required to backup the data.

The XML I ended up using was:

<?xml version="1.0"?>
<a:searchrequest xmlns:a="DAV:">
<a:sql>
SELECT * FROM "URL"
</a:sql>
</a:searchrequest>

in case anyone is interested. Again, I have a fatjar of this utility, but I have to check with work to make sure they’re okay with me releasing it before I can post it anywhere.

I hate it. IMAP Synchronization, that is.

In an effort to migrate users from one *shudder* Exchange provider to another (after getting shot down for proposing first Zimbra, then standard mail server stuff, then Openchange), I have been going through all of the available IMAP sync software that I could find.

mbsync (http://isync.sourceforge.net/mbsync.html) – We use this for IMAP backup, so I figured it would be a good idea to try it for syncing between two IMAP servers. *Crash*. *Segfault*. This is one of the reasons I’m not a big fan of really low-level apps written in C.

imapsync (http://www.linux-france.org/prj/imapsync/) – Written using a Perl library, this offered the promise of fast syncing, and appeared to work in test runs. That is, until it encountered folders with spaces in them, and threw up all over the terminal. Really, it wasn’t a use-case for *anyone* to have spaces in folder names? I could have put some effort into fixing the library and/or script, but I figured that if something that basic wasn’t working for a fairly mature library, this wasn’t going to be pretty.

The Imap Migration Tool (http://migrationtool.sourceforge.net/) – Written in PHP, with a two or three step process. Was meant for moving stuff, not keeping them in sync, so running multiple times would cause additional copies of messages to be created. Also, it seems to have been written in some prehistoric form of PHP, which required hacking to get it even marginally functional. Even then, it spat tons of warnings and nasties, which made me a little nervous to use it on any important mailboxes.

mailsync (http://mailsync.sourceforge.net/) – I don’t even remember why this one didn’t work, only that I wasted a fair amount of time playing with it to get it to try to migrate using the scheme we were working with to no avail. Scratch another one.

Solution … I had to roll my own. This is *stupid* to the highest degree, but I rolled something using javamail into a self-contained “fat jar” using fatjar. It works sickeningly well, and actually respects existing messages by message id, so that if for any reason I have to break the sync process, it’ll skip past anything it has already done. I made it non-destructive because it seemed like seriously bad juju to have a sync tool purge out email…. If I get permission from work, I’ll post the jar for anyone with Java 1.6 to use.

As an aside, it took a little bit of hackery to get fatjar running outside eclipse. The relevant section of build.xml was:

  <path id="fatJarPath" location="lib/fatjar.jar"/>
  <property name="ant.reuse.loader" value="true"/>

  <taskdef name="fatjar.build" classname="net.sf.fjep.anttask.FJBuildTask" classpathref="fatJarPath"/>
  <typedef name="fatjar.manifest" classname="net.sf.fjep.anttask.FJManifestType" classpathref="fatJarPath"/>
  <typedef name="fatjar.exclude" classname="net.sf.fjep.anttask.FJExcludeType" classpathref="fatJarPath"/>
  <typedef name="fatjar.jarsource" classname="net.sf.fjep.anttask.FJJarSourceType" classpathref="fatJarPath"/>
  <typedef name="fatjar.filesource" classname="net.sf.fjep.anttask.FJFileSourceType" classpathref="fatJarPath"/>

  <target name="main" depends="build">
    <fatjar.build output="ImapSynchronizer.jar">
      <fatjar.manifest mainclass="...myclasspath...ImapSynchronizer"/>
      <fatjar.filesource path="bin" relpath=""/>
      <fatjar.jarsource file="lib/mail-1.4.3.jar" relpath=""/>
    </fatjar.build>
  </target>

I have recently had cause to proxy a J2EE CXF service through an apache 2.2 instance, and thought it would be nice to share my findings. (This was all done on a Debian system.)

First of all, the mod_proxy pieces have to be enabled using a2enmod proxy.

A fragment has to be added with the proxying bits and some limitation:

<Proxy proxy:http://SERVER:PORT/URL>
     Order allow,deny
     Allow from all
</Proxy>

ProxyPass /EXPOSEDURL http://SERVER:PORT/URL
ProxyPassReverse /EXPOSEDURL http://SERVER:PORT/URL

Reloading apache configuration should enable the proxy properly. The only other possible issue is that in addition to the WSDL URL, you’re going to have to specify a “proxy” URL, which is just the service URL without ?wsdl at the end of it. A fragment of PHP’s SoapClient would look like this:

$url = "http://server.domain:8180/services/MySoapService?wsdl";
$sc = new SoapClient( file_get_contents($url), array(
          'login' => $my_username // optional for basic auth
        , 'password' => $my_password // optional for basic auth
        , 'compression' => SOAP_COMPRESSION_ACCEPT | SOAP_COMPRESSION_GZIP
        , 'location' => str_replace('?wsdl', '', $url) // force this to work through proxies
));

The magic in that last segment was the “location” parameter, as it specifies the proxy. Good luck!

I really like Ubuntu — I use it for all of my workstations/laptops, and have for the last several years. I do not, however, like it when people tell me that something that’s clearly and demonstrably a bug in a system simply isn’t a bug. Fie on your crappy PPA build system, I’m just not going to build more Ubuntu packages for things. I’ll just go back to only packaging Debian server packages.

Way to build community, Canonical.

I quote:

The buildds[sic] have no internet access, so you can’t use installer packages (packages which need to download the undistributable archive from the official location) on buildds (e.g. as build dependencies).

I swear, it’s a feature, not a bug. </sarcasm>

To sum up before I even begin: they *suck*.

One of Java’s exceptionally weak points is serial and parallel interfaces, since it’s not within the normal bundled class library specification. In developing for Linux systems, I found that there are two basic routes you can take, being Java Communications API and RXTX, with the latter being a reimplementation/extension of the former. To say that these are painful to work with doesn’t adequately state the gravity of the situation.

For example, the javax.comm.properties file needed by both libraries has to exist in the classpath or buried deep in the JRE. This is an offshoot of the issue that they require JNI libraries to function, which debian handles nicely in the form of a librxtx-java package which does most of the heavy lifting with the exception of that nasty properties file. In dealing with a J2EE app, this becomes a pain if you want to stick with the “single file” WAR deployment. Eventually I baked an additional instruction into my build.xml file to relocate the properties file on execution.

The documentation for rxtx, at least in terms of practical examples, is horrid. The wiki server isn’t up and running (at least when I went to look) and the only evidence that anyone had ever tried to use this library for the purposes I was looking to use it for were a few old javadocs and a handful of mailing list entries.

This really should *not* be this painful. Virtually every system targeted by the JRE has I/O devices which should be covered by the scope of the basic class libraries. I should not have to practically reinvent the wheel simply to push data out to a peripheral.

By the look of the documentation, this wouldn’t have been any easier doing it with Windows, and I believe I managed to kill its ability to be cross-platform in any meaningful sense when I decided to use RXTX. This is just as bad as those asshats who make completely crossplatform applications then tie them intrinsically to Windows in some completely brain-dead way for no apparent reason.

Hey Sun, when you made this specification, could you have possibly thought to include support for devices which have been around since 1969 …

I’ve been in the process of teaching myself the Google Android platform over the last few weeks, as my work has had a contest devoted to creating a mobile phone app, with a prize attached. I’m the only entrant who doesn’t work in the Engineering department, so this is a bit of a professional stretch for me as far as my job is concerned.

I have found quite a few caveats in the development process which are worth noting, as I have spent a fair amount of time researching them to figure them out. (I should also note at this point that this is in no way downplaying the important role Android plays in the future of mobile phone platforms, and is definitely *not* advocating either WinMo, which is a horrible abortion of a platform, or iPhone, which is not only locked to a single vendor, but *requires* purchasing a Crapitosh for development as well as a minimum 99$ development license to get going …)

1) Camera Access – It took some digging to find that you have to allow permission android.permission.CAMERA, or else it’ll throw out of memory errors and crash the app. I think it’s a running bug that in Android 1.6 it isn’t throwing a proper “PermissionDeniedException” or whatever else they feel like calling it. Otherwise the Camera API is pretty powerful.

2) Getting your own phone number – This is very interesting, as Android just moved to using something called “TelephonyManager” to deal with all of this. Unfortunately documentation hasn’t quite caught up to it. An example:

TelephonyManager mTelephonyMgr = (TelephonyManager)context.getSystemService(Context.TELEPHONY_SERVICE);
String imei = mTelephonyMgr.getDeviceId();
String phoneNumber=mTelephonyMgr.getLine1Number();
String country = mTelephonyMgr.getNetworkCountryISO(); 

3) Saving state – If you want to save state for your app, it makes no sense to go any further than SharedPreferences. It keeps you from having to create a content provider to deal with keeping that information stored somewhere else …

4) AppWidgets are awesome — The idea of a “home screen” applet is really easy to implement simply by extending AppWidgetProvider and add a <receiver> XML element to the AndroidManifest.xml file corresponding with the class.

5) Opening an intent from a menu item – The documentation says to use *a* Context, but doesn’t mention that you should be passing the Activity subclass that you’re coding to use as the Context.

As I have been delving deeper into the J2EE Servlet specification during the rewrite of REMITT, I have been learning some very interesting, and sometimes very painful, lessons about trying to package something without requiring complicated installations.

To get authentication working in Tomcat, the normal way is to define a Context and Realm in TOMCAT_HOME/conf/server.xml. A relatively undocumented file called META-INF/context.xml allows fragments to be distributed with war files. This would be a great solution if it didn’t mean that authentication was now deeply dependent on editing a file in the web archive, which kind of takes the advantage completely out of having one. D’oh!

Next up was the idea of using JAAS, which is a way of setting all of the security information in a single configuration file, and using a custom security module. Which would work very well, if I wanted to pre-extract the war file and pass an archaic system property file through JAVA_OPTS. This seems, again, too difficult to make any sort of sense.

I eventually found a solution by creating my own Filter and LoginModule, then setting the system property for the JAAS configuration manually in the init() portion of the Filter. A little filter and realm information in WEB-INF/web.xml, and everything seems to work somehow. I understand that the Servlet guys wanted to make sure that the J2EE server context and app context were at two different levels (hence the reason a DataSource in META-INF/context.xml or tomcat’s conf/server.xml isn’t writeable at all), but it makes it a mess for clean deployment.

As it stands now, I can run the server with an override file like like:
JAVA_OPTS="-Dproperties=/my/remitt.properties" ./bin/catalina.sh start

And I think that’s worth the hassle to set up, since it makes deployment a breeze.

For those people who are unfamiliar with my programming ventures, I had written a three stage extensible billing system a few years back called REMITT (which stands for REMITT Electronic Medical Information Translation and Transmission). I had created the majority of the code in about a week and the XSL transforms required for it to function in another two weeks or so.

Since then, the project has been somewhat languishing, due to it being written in Perl, which is a minor pain to get running for most people on a good day, compounded with it using XML-RPC with sessions as a remote procedural call dialect.

I decided to embark on a complete rewrite of the core of REMITT to make it more portable, more extensible, better designed and far more stable than it had been in past. I have gotten about 3/4 of the way through rewriting the core of REMITT as a J2EE application using MySQL as a database backend. There are a number of improvements I have been working on, including:

1. True crossplatform nature. – Being J2EE, you can even run it on *shudder* Windows if you felt the need to. I wouldn’t recommend it, but you could do it.
2. Better threading model. – Instead of a single executor thread pool, each stage of transformation gets its own pool, maintained and reaped by a single control thread. As Java has a very nice threading model, this was much easier than I had anticipated.
3. Transport scripting. – Instead of requiring heavy-duty programming expertise to write transport plugins, I moved to using Rhino do do embedded Javascript scripting for them, as well as allowing SFTP transport.
4. Namespaced configuration. – Each user can have their own configuration *per* plugin, which is a lot easier than before. It also has the ability for each plugin to present its configuration options to a frontend, so there’s no more worrying about configuration variable names.
5. SOAP. – I’m using CXF for web services, and it took me all of a second to generate a proxy class using wsdl2php which worked perfectly the first time around.
6. Web Interface. – Through the magic of JSP, the REMITT server has its own set of administration pages and monitoring functions, so it’s not quite as much of a black box as before.
7. IDE Support. – Eclipse is a beautiful IDE, and with Javadoc completion support, makes anything I could have used with Perl look like crap. ‘Nuff said.
8. OO Model. – Perl OO is to Java OO as a Pinto is to a Tesla Roadster. This thing actually has some architecture behind it. It’s about time.

I’ll be posting the code into REMITT’s SVN trunk shortly, as soon as I’m positive that it does *something* in case anyone else feels the need to work on it. In the meantime, I’ll continue to play with it until I get something useful out of it.