Showing posts with label integration. Show all posts
Showing posts with label integration. Show all posts

Tuesday, March 8, 2011

How to use Java to call an IBM i program

This post is a late update to a post I wrote a couple of years ago about exposing RPGLE programs so that we could call them from our webapps. That worked fine, but I didn't particularly like the workflow we used to produce the wrapper, so I tried to get rid of some muda.
The worst one was, in my opinion, the need to use two different IDEs, thus necessarily involving the collaboration of two persons (not every developer has IBM tools) that creates a bottleneck.
After some researches I found some official documentation (the problem with IBM is not scarcity but abundance), dug for our jt400.jar and started to experiment a little.

A small word of caution: the article is written in an introductory style, showing the TDD process that lead to the implementation of the functionality I needed. If you're only interested in the results you can easily skip to the end of the article (I will not know it, so I won't get offended).

The guinea pig for my experiment is a simple program that given a table name returns a serial number which is unique for that table. As I had have to use this program in an application that already uses a serial number generator, first I need an interface, so I use the NetBeans Extract Interface refactoring on my existing service:
public interface SerialsService {
long getSerialNumber(String tableName);
}
That done I create an implementing class that gets the serial from the iSeries:
public class MyIseriesSerialsService implements SerialsService {
public long getSerialNumber(String tableName){
throw new UnsupportedOperationException();
}
}
OK now we can start the real stuff. First thing first, we write a couple of simple tests in a new test class:
@Test(expected = "IllegalArgumentException.class"
public void testWithWrongTableNameShouldThrowException() {
MyIseriesSerialsService instance = new MyIseriesSerialsService();
instance.getSerialNumber("WrongName");
}
CTRL + F6 to start the test and NetBeans gives me a red bar. I change the type of exception thrown, CTRL + F6 again and here's a nice green bar. That was easy, wasn't it? now we have a service that behaves correctly when you don't invoke it correctly. This is good, but if your application is like mine I suppose you expect your service to behave correctly even when it is correctly invoked (yes, sometimes it happens).

To call an IBM i program you first have to connect to an iSeries, and for this you use the AS400 class, which IBM documentation actually calls AS400 IBM® Toolbox per Java™: I'll stick to that, so that is what I mean when I simply write AS400 - the same holds true for all other registered trademarks. If this is enough to keep lawyers at bay, as I hope, this ends the disclaimer.

The AS400 class, amongst other things, manages socket connections on behalf of a user, so at least we have to tell our service what is the iSeries we want to connect to, and which user we want to impersonate. This requires to change the constructor of the service (obviously it is not the only way, but I'd rather use an immutable object). We could pass in the name of the server (or the ip address if you don't want to rely on a DNS service), a username and a password, or we could create an AS400 object and pass it to the constructor of the service. This option separates responsibilities better (and can be tested more easily), so I choose it and write a simple test to check the connection to our system:
@Test
public void testConnection() throws Exception {
AS400 system = new AS400(host, username, password);
assertNotNull(system.getRelease());
System.out.println(system.getRelease());
}
I am just experimenting, so as always I defer all the exception dealings. Green bar. I can now add a field to the constructor of the service...
public class MyIseriesSerialsService implements SerialsService {

private AS400 system;

public MyIseriesSerialsService(AS400 system) {
this.system = system;
}

public long getSerialNumber(String tableName) {
throw new IllegalArgumentException();
}
}
...and modify the tests accordingly, also removing a small duplication:
@Before
public void setUp() {
instance = new MyIseriesSerialsService(createValidSystem());
}

private AS400 createValidSystem() {
return new AS400(host, username, password);
}

@Test(expected = IllegalArgumentException.class)
public void testWithWrongTableNameShouldThrowException() {
instance.getSerialNumber("WrongName");
}
Skipping (for the sake of the article) all tests on invalid systems and other exceptions we can now start writing a more interesting test:
@Test
public void testGetSerialNumberReturnsPositiveNumber() throws Exception {
long result = instance.getSerialNumber(tableName);
assertTrue(result > 0);
}
CTRL + F6, red bar (as expected). This is where the rubber hits the road.

The MYPGM program is contained in the MYLIB library in the QSYS, and it also uses the MYOTHERLIB library (by the way, watch for the maximum length of the names of the libraries, which is 10). The wrapper class for a program call (the Command pattern in action) is ProgramCall (fantasy does not abound in IBM, but that's good as I easily found what I was looking for). It needs to know the system we're operating on, the path to the program and the parameter list. We have the system, so we need the path to the program, for which we use the QSYSObjectPathName class, and the parameter list, for which we use the ProgramParameter class.

The QSYSObjectPathName constructor takes as parameters the name of the library, the name of the program and it extension. Once we have a QSYSObjectPathName object we can ask it the path to the program, which is what precisely what we need:
QSYSObjectPathName pgmName = new QSYSObjectPathName(myLib, myPgm, pgmExtension);
The parameter list is actually an array of ProgramParameter objects, which cannot be null. Each parameter wraps an array of bytes, so we have to use the proper converter; luckily there are some converters ready for us.

Our program has an input/output parameter (the name of the table) and an output parameter (the serial number for the given table), so we have the following:
ProgramParameter[] paramList = new ProgramParameter[2];
The first parameter is the name of the table, which is a String. To deal with it we can use the AS400Text converter:
AS400Text textConverter = new AS400Text(10, system);
byte[] key = textConverter.toBytes(tableName);
paramList[0] = new ProgramParameter(key);
The first parameter for the constructor is the length of the IBM i text, the second one is the system we're operating on.

The output parameter is a long, so we'll have to use a different converter. Up to now let's just initialize it:
paramList[1] = new ProgramParameter(32);
Our command is now ready to be born:
ProgramCall pgm = new ProgramCall(system, pgmName.getPath(), paramList);
To execute it we simply call the run method, which returns a boolean. If the result is true we can extract data from the output parameter and convert it:
byte[] data = paramList[1].getOutputData();
AS400PackedDecimal pdconverter = new AS400PackedDecimal(12, 0);
long result = ((BigDecimal) pdconverter.toObject(data)).longValue();
And this is it. Let's see:
Testcase: testGetSerialNumberReturnsPositiveNumber(testserialias.MyIseriesSerialsServiceTest):
Caused an ERROR
6
Wow that's illuminating... Luckily we aldready know what's going on: if you check above you'll notice I wrote that the program needs another library, which is not loaded when we first connect to the system. To add the library we need to issue a command, thus we use the CommandCall class:
CommandCall cc = new CommandCall(system);
cc.setCommand("ADDLIBLE " + myotherLib);
cc.run();
Now we're going somewhere... As we trust but also want to control we add another simple test to check that the service returns bigger numbers on consecutive calls:
@Test
public void testGetSerialNumberReturnsBiggerNumbersOnFurtherCalls() {
firstResult = instance.getSerialNumber(tableName);
long secondResult = instance.getSerialNumber(tableName);
assertTrue(firstResult < secondResult);
}
If we want to have more informations we can ask the program wrapper for an array of AS400Message objects:
private String buildMessage(final ProgramCall pgm) {
AS400Message[] messageList = pgm.getMessageList();
String message = "";
for (int i = 0; i < messageList.length; i++) {
AS400Message aS400Message = messageList[i];
message += aS400Message.getText();
message += "\r\n";
}
return message;
}
The RPGLE program already ensures that no serial can be duplicated, but I thought that adding a little bit of safety wouldn't be too much damage, so the main call is synchronized. The final draft:
import com.ibm.as400.access.AS400;
import com.ibm.as400.access.AS400Message;
import com.ibm.as400.access.AS400PackedDecimal;
import com.ibm.as400.access.AS400Text;
import com.ibm.as400.access.CommandCall;
import com.ibm.as400.access.ProgramCall;
import com.ibm.as400.access.ProgramParameter;
import com.ibm.as400.access.QSYSObjectPathName;
import java.math.BigDecimal;
import java.util.logging.Level;
import java.util.logging.Logger;

public class MyIseriesSerialsService implements SerialsService {

private final String myLib = "MYLIB";
private final String myOtherLib = "MYOTHERLIB";
private final String myPgm = "MYPGM";
private final String pgmExtension = "PGM";
private final AS400 system;
private QSYSObjectPathName pgmName;

public MyIseriesSerialsService(final AS400 system) {
this.system = system;
addLibraries();
createPathName();
}

public long getSerialNumber(final String tableName) {
synchronized (this) {
ProgramParameter[] paramList = createInputParameters(tableName);
ProgramCall pgm = createProgramCall(paramList);

try {
pgm.run();
} catch (Exception ex) {
Logger.getLogger(MyIseriesSerialsService.class.getName()).
log(Level.SEVERE, ex.getMessage(), ex);
String errorMessage = buildMessage(pgm);
throw new RuntimeException(errorMessage, ex);
}

long result = extractResult(paramList);

return result;
}
}

private void addLibraries() {
try {
CommandCall cc = new CommandCall(system);
cc.setCommand("ADDLIBLE " + myOtherLib);
cc.run();
} catch (Exception ex) {
Logger.getLogger(MyIseriesSerialsService.class.getName()).
log(Level.SEVERE, ex.getMessage, ex);
throw new RuntimeException("Unable to inizialize service", ex);
}
}

private void createPathName() {
pgmName = new QSYSObjectPathName(myLib, myPgm, pgmExtension);
}

private ProgramParameter[] createInputParameters(final String tableName) {
ProgramParameter[] paramList = new ProgramParameter[2];
paramList[0] = createInputParameter(tableName);
paramList[1] = new ProgramParameter(32);
return paramList;
}

private ProgramParameter createInputParameter(final String tableName) {
AS400Text textConverter = new AS400Text(10, system);
byte[] key = textConverter.toBytes(tableName);
return new ProgramParameter(key);
}

private ProgramCall createProgramCall(final ProgramParameter[] paramList) {
return new ProgramCall(system, pgmName.getPath(), paramList);
}

private long extractResult(final ProgramParameter[] paramList) {
byte[] data = paramList[1].getOutputData();
AS400PackedDecimal pdconverter = new AS400PackedDecimal(12, 0);
return ((BigDecimal) pdconverter.toObject(data)).longValue();
}

private String buildMessage(final ProgramCall pgm) {
AS400Message[] messageList = pgm.getMessageList();
String message = "";
for (int i = 0; i < messageList.length; i++) {
AS400Message aS400Message = messageList[i];
message += aS400Message.getText();
message += "\r\n";
}
return message;
}
}
And the test class which assisted me in all the small refactorings:
import com.ibm.as400.access.AS400;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import static org.junit.Assert.*;

public class MyIseriesSerialsServiceTest {

private final String host = "myVeryExpensiveHost";
private final String username = "myUsername";
private final String password = "myPassword";
private final String tableName = "myTable";
//
private MyIseriesSerialsService instance;

@Before
public void setUp() {
instance = new MyIseriesSerialsService(createValidSystem());
}
private AS400 createValidSystem() {
return new AS400(host, username, password);
}

@Test
public void testConnection() throws Exception {
AS400 system = new AS400(host, username, password);
assertNotNull(system.getRelease());
}

@Test(expected = IllegalArgumentException.class)
public void testWithWrongTableNameShouldThrowException() {
instance.getSerialNumber("WrongName");
}

@Test
public void testGetSerialNumberReturnsPositiveNumber() throws Exception {
long result = instance.getSerialNumber(tableName);
assertTrue(result > 0);
}

@Test
public void testGetSerialNumberReturnsBiggerNumbersOnFurtherCalls() {
long firstResult = instance.getSerialNumber(tableName);
long secondResult = instance.getSerialNumber(tableName);
assertTrue(firstResult < secondResult);
}
}
As I said, this is but a draft, and could be improved in many ways, e.g. the tests are quite coarse and don't consider all the small things that could go wrong, some of which I discovered with a quick debugging while I was setting up the tests. Calling a system.disconnectAllServices() when you finish would not be bad either. Yet, it is easily readable and hopefully understandable, so I hope this will help everyone to get rid of the extra stack required by the application server when it is not needed (does that ring a bell?).

Monday, December 20, 2010

Meet Flatworm

I'm OK with flat files, but why should you use a fixed-length record when every row is formed by 1320 chars, most of which are blank? Isn't it an enormous waste of resources?

Anyway, should you need to deal with flat files (it seems like no programmer can keep them at bay... does this ring a bell?) after some investigation I stumbled upon Flatworm, an interesting library that lets you read from a flat file, be it fixed-length or separated by a separator character, and instantiate the appropriate beans. It also supports repeating segments or multi-line records. Nicely enough, it also work the other way around, which it what I was primarily interested in.

All you have to do is provide a descriptor in XML format, sit back and relax. Let's see how it works.

Let's suppose we need to produce a fixed-length file with the following format:
XXvalueOne  valueTwo  
i.e. a fixed record identifier and two fields of 10 chars each.

First of all you have to provide the descriptor:
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE file-format SYSTEM "http://www.blackbear.com/dtds/flatworm-data-description_1_0.dtd">
<file-format>
<converter name="char" class="com.blackbear.flatworm.converters.CoreConverters" method="convertChar" return-type="java.lang.String"/>
<record name="whatever-record">
<record-ident>
<field-ident field-start="0" field-length="2">
<match-string>XX</match-string>
</field-ident>
</record-ident>
<record-definition>
<bean name="whatever" class="my.package.Whatever"/>
<line>
<record-element length="2"/>
<record-element length="10" beanref="whatever.propOne" type="char">
<conversion-option name="justify" value="left"/>
</record-element>
<record-element length="10" beanref="whatever.propTwo" type="char">
<conversion-option name="justify" value="left"/>
</record-element>
</line>
</record-definition>
</record>
</file-format>
Then you can write a simple class that, given an iterator of whatever you have to export, creates a file and populates it:
public class SimpleExporter {

FileCreator fileCreator;
Iterator<Whatever> iterator;

public SimpleExporter(
Iterator<Whatever> iterator,
final String configFile,
final String outputFile)
throws FlatwormCreatorException {
this.iterator = iterator;
InputStream config = Thread.currentThread().
getContextClassLoader().
getResourceAsStream(configFile);
fileCreator = new FileCreator(config, outputFile);
}

public void execute() {
try {
fileCreator.setRecordSeperator("\r\n");
fileCreator.open();
while (iterator.hasNext()) {
Whatever whatever = iterator.next();
fileCreator.setBean("whatever", whatever);
fileCreator.write("whatever-record");
}
fileCreator.close();
} catch (IOException ex) {
Logger.getLogger(SimpleExporter.class.getName()).log(Level.SEVERE, null, ex);
} catch (FlatwormCreatorException ex) {
Logger.getLogger(SimpleExporter.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
Exceptions should be managed row by row, but just stay with me for the example, OK?

For the sake of this example the Whatever class is just a POJO, so it's not worth reporting it here. So what have I done? I just created a FileCreator object passing it an InputStream to the descriptor and the path for the output file. That was not hard, was it?

If your bean has inner properties you can simply use a dot notation:
<record-element length="10" beanref="whatever.outerProperty.innerProperty" type="char">
Playing around I had some little tricks to learn: maybe there's a better way, but they work :-) for example I had to write several fields which are not present in my beans. For this I simply added a "filler" property of type String and used it in all such cases, adding a comment in the desctriptor to specify what I was substituting.

Another problem emerged when the properties in my bean were null; to fix this once and for all I simply extended the CoreConverters class adding null-safe operations:
@Override
public String convertChar(String str, Map<String, ConversionOption> options) {
return super.convertChar(str == null ? "" : str, options);
}
That said, I found the library really useful as it saved me a lot of time.

Friday, October 15, 2010

Meet Dozer

Integration between systems is one of the most typical reasons of failure in complex projects. Yet, we find ourselves doing it day after day, so after some strategic mapping you have to get your hands dirty and map your wonderful objects to that incredible garbage other vendors still insist to call code. Or was it the other way round?

Let's assume a simple interaction via web services exposed on, say, an i-Series platform. I don't know whether all RPGLE programmers work in the same way, but almost all services of that kind I've seen so far have a single port with a thousand parameters for a hundred of different purposes and it is up to you to try and understand which of them must be used in which occasion. If that was not enough (and for me it is) it seems there is a contest for the use of the most cryptic parameter names, for which vowels seem to be banished and six characters is considered more than ehough (when not a waste of space).

One of the many things you learn with DDD is that code should be as close as possibile to your mental representation of the domain: well, LDLSDC does not make me think of a list of documents, but maybe it's my fault.

That said, before I use those objects in my code I'll have to convert them. The typical mapper has a signature similar to the following one:
public MyObject convert(YourObject yours, MyObject mine)
or
public MyObject convert(YourObject yours, MyObject.class)
The implementation is normally something like this:
myObject.setPropertyOne(yourObject.getPropertyOne);
myObject.setPropertyTwo(yourObject.getPropertyTwoButHasDifferentName)
(repeat ad lib for each property to be mapped)

This is very frustrating and most of all error prone. This is where several libraries step forward, most of them based on reflection, each of them having pluses and minuses. So far, the most interesting one I've found is Dozer, a bean to bean mapper that recursively copies data from one object to another (of a different type, of course! where would the fun be otherwise?). By the way, integrating different systems is just one of many reasons you have for mapping between different objects.

Using Dozer is just as easy as downloading the jar file, adding it to the classpath (don't forget the dependencies!) and coding right away:
Mapper mapper = new DozerBeanMapper();
Destination destination = mapper.map(source, Destination.class);
(almost utterly copied from the official tutorial). That's it. All fields with the same name are automagically mapped, with Dozer taking care of the necessary conversions (at least the more common ones). Should you need to customize mappings you may add some xml files that define how Dozer should behave:
<?xml version="1.0" encoding="UTF-8"?>
<mappings xmlns="http://dozer.sourceforge.net"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://dozer.sourceforge.net
http://dozer.sourceforge.net/schema/beanmapping.xsd">

<configuration>
<stop-on-errors>true</stop-on-errors>
<date-format>MM/dd/yyyy HH:mm</date-format>
<wildcard>true</wildcard>
</configuration>

<mapping>
<class-a>yourpackage.yourSourceClassName</class-a>
<class-b>yourpackage.yourDestinationClassName</class-b>
<field>
<A>yourSourceFieldName</A>
<B>yourDestinationFieldName</B>
</field>
</mapping>

other custom class mappings would go here.......

</mappings>
And this is copied from the official manual.

You can also introduce a custom converter that you can reuse through your application:
<converter type="org.dozer.converters.TestCustomConverter" >
<class-a>org.dozer.vo.CustomDoubleObject</class-a>
<class-b>java.lang.Double</class-b>
</converter>
and its corresponding class:
public class TestCustomConverter implements CustomConverter {

public Object convert(Object destination, Object source, Class destClass, Class sourceClass) {
if (source == null) {
return null;
}
CustomDoubleObject dest = null;
if (source instanceof Double) {
// check to see if the object already exists
if (destination == null) {
dest = new CustomDoubleObject();
} else {
dest = (CustomDoubleObject) destination;
}
dest.setTheDouble(((Double) source).doubleValue());
return dest;
} else if (source instanceof CustomDoubleObject) {
double sourceObj = ((CustomDoubleObject) source).getTheDouble();
return new Double(sourceObj);
} else {
throw new MappingException("Converter TestCustomConverter used incorrectly. Arguments passed in were:" + destination + " and " + source);
}
}
}
And now a personal reminder: if you have to map an array of objects, but you know the types that you will find in specified positions AND want to avoid class cast exceptions (and I guess you do), you can use this syntax:
<field>
<a>myObjects[0]</a>
<b>other.propOne</b>
<a-hint>java.lang.Long</a-hint>
</field>
<field>
<a>myObjects[1]</a>
<b>other.propTwo</b>
<a-hint>java.lang.Short</a-hint>
</field>
Next thing I'm gonna try is the mapping for POJOs and JAXB objects...