Monthly Archives: November 2019

Repeated code

Introduction

It is usually not good to have copy/paste code in our Java application but sometimes it is unavoidable. For example the project License3j provides a method isXXX in the Feature class for each XXX type it supports. In that case, we can do no better than write

    public boolean isBinary() {
        return type == Type.BINARY;
    }

    public boolean isString() {
        return type == Type.STRING;
    }

    public boolean isByte() {
        return type == Type.BYTE;
    }

    public boolean isShort() {
        return type == Type.SHORT;
    }

and so on

for each and every feature type the application supports. And there are some types there: Binary, String, Byte, Short, Int, Long, Float, Double, BigInteger, BigDecimal, Date, UUID. It is not only a boring task to type all the very similar methods, but it is also error-prone. A very few humans are good at doing such a repetitive task. To avoid that we can use the Java::Geci framework and as the simplest solution we can use the generator Iterate.

POM dependency

To use the generator we have to add the dependency

<dependency>
    <groupId>com.javax0.geci</groupId>
    <artifactId>javageci-core</artifactId>
    <scope>test</scope>
    <version>1.4.0</version>
</dependency>

The library is executed only during when the tests run, therefore the use of it does not imply any extra dependency. Whoever wants to use the library License3j does not need to use Java::Geci. This is only a development tool used in test scope.

Unit Test running it

The dependency will not run by itself. After all the dependency is not a program. It is a bunch of class files packaged into a JAR to be available on the classpath. We have to execute the generator and it has to be done through the framework creating a unit test:

    @Test
    @DisplayName("run Iterate on the sources")
    void runIterate() throws IOException {
        Geci geci = new Geci();
        Assertions.assertFalse(
            geci.register(Iterate.builder()
                              .define(ctx -> ctx.segment().param("TYPE", ctx.segment().getParam("Type").orElse("").toUpperCase()))
                              .build())
                .generate()
            , geci.failed()
        );
    }

It creates a Geci object, instantiates the generator using a builder and then invokes generate() on the configured framework Geci object. The define() call seems a bit cryptic as for now. We will shed light on that later.

Source Code Preparation

The final step before executing the build is to define a template and the values to insert into the template. Instead of writing all the methods all we have to do is to write a template and an editor fold segment:

    /* TEMPLATE
    LOOP Type=Binary|String|Byte|Short|Int|Long|Float|Double|BigInteger|BigDecimal|Date|UUID
    public boolean is{{Type}}() {
        return type == Type.{{TYPE}};
    }

     */
    //<editor-fold id="iterate">
    //</editor-fold>

When we execute the generator through the framework it will evaluate the template for each value of the placeholder Type and it will replace each {{Type}} with the actual value. The resulting code will be inserted into the editor-fold segment with the id “iterate”.

Looking at the template you can see that there is a placeholder {{TYPE}}, which is not defined in the list. This is where the unite test define() comes into the picture. It defines a consumer that consumes a context and using that context it reads the actual value of Type, creates the uppercased version of the value and assigns it to the segment parameter named TYPE.

Generally, that is it. There are other functionalities using the generator, like defining multiple values per iteration assigned to different placeholders, escaping or skipping lines and so on. About those here is an excerpt from the documentation that you can read up-to-date and full az https://github.com/verhas/javageci/blob/master/ITERATE.adoc

Documentation Excerpt

In the Java source files where you want to use the generator you have to annotate the class with the annotation @Geci("iterate").
You can also use the @Iterate annotation instead, which is defined in the
javageci-core-annotations module.
This will instruct the Geci framework that you want to use the iterate generator in the given class.

TEMPLATE

A template starts after a line that is /\*TEMPLATE or TEMPLATE.
There can be spaces before and after and between the /* and the word
TEMPLATE but there should not be anything else on the line.
When the generator sees such a line it starts to collect the following lines as the content of the template.

The end of the template is signaled by a line that has */ on it and nothing else (except spaces).

The content of the template can contain parameters between {{ and }}
characters similarly as it is used by the mustache template program.
(The generator is not using mustache, template handling is simpler.)

LOOP

While collecting the lines of the template some of the lines are recognized as parameter definitions for the template.
These lines do not get into the trunk of the template.
(Command names on these lines are always capital.)

As you could see in the introduction the line

    LOOP type=int|long|short

is not part of the template text.
It instructs the generator to iterate through the types and set the parameter {{type}} in the text to int
first, long the second and short the last.
That way you can iterate over multiple values of a single parameter.

A more complex template may need more than one parameter.
In that case, you can list them in the LOOP line as

    LOOP type,var=int,aInt|long,aLong|short,aShort

This will tell the generator to set the parameter {{type}} the same way as before for the three iterations but the same time also set the parameter {{var}} to aInt in the first loop, to aLong in the second loop and aShort in the last loop.

If the list of the values is too long it is possible split the list into multiple LOOP lines.
In this case, however, the variables have to be repeated in the second, third and so on LOOP lines.
Their order may be different, but if there is a variable undefined in some of the LOOP
lines then the placeholder referring to it will be be resolved and remains in the {{placeholder}} form.

The above example can also be written

    LOOP type,var=int,aInt
    LOOP var,type=aLong,long
    LOOP type,var=short,aShort

and it will result into the same values as the above LOOP repeated here:

    LOOP type,var=int,aInt|long,aLong|short,aShort

Default editor-fold

The templates are processed from the start of the file towards the end and the code generated is also prepared in this order.
The content of the generated code will be inserted into the editor-fold segment that follows the template directly.
Although this way the id of the
editor-fold segment is not really interesting you must specify a unique id for each segment.
This is a restriction of the the
Java::Geci framework.

Advanced Use

EDITOR-FOLD-ID

It may happen that you have multiple templates looping over different values and you want the result to go into the same editor-fold
segment.
It is possible using the EDITOR_FOLD_ID.

In the following example

package javax0.geci.iterate.sutclasses;

public class IterateOverMultipleValues {
    /* TEMPLATE
    {{type}} get_{{type}}Value(){
      {{type}} {{variable}} = 0;
      return {{variable}};
    }

    LOOP type,variable=int,i|long,l|short,s
    EDITOR-FOLD-ID getters
     */
    //
            // nothing gets here
    //

    //
    int get_intValue(){
      int i = 0;
      return i;
    }

    long get_longValue(){
      long l = 0;
      return l;
    }

    short get_shortValue(){
      short s = 0;
      return s;
    }

    //
}

the generated code gets into the editor-fold that has the id name
getters even though this is not the one that follows the template definition.

Use this feature to send the generated code into a single segment from multiple iterating templates.
Usually, it is a good practice to keep the template and the segment together.

ESCAPE and SKIP

The end of the template is signaled by a line that is */.
This is essentially the end of a comment.
What happens if you want to include a comment, like a JavaDoc into the template.
You can write the */
characters at the end of the comment lines that still have some characters in it.
This solution is not elegant and it essentially is a workaround.

To have a line that is exactly a comment closing or just any line that would be interpreted by the template processing, like a LOOP line you should have a line containing nothing else but an ESCAPE on the previous line.
This will tell the template processing to include the next line into the template text and continue the normal processing on the line after.

Similarly, you can have a line SKIP to ignore the following line altogether.
Using these two commands you can include anything into a template.

An example shows how you can include a JavaDoc comment into the template:

package javax0.geci.iterate.sutclasses;

public class SkippedLines {
    /* TEMPLATE
    /**
     * A simple zero getter serving as a test example
     * @return zero in the type {{type}}
    ESCAPE
     */
    // SKIP
    /*
    {{type}} get_{{type}}Value(){
      {{type}} {{variable}} = 0;
      return {{variable}};
    }
    LOOP type,variable=int,i|long,l|short,s
    EDITOR-FOLD-ID getters
     */
    //
    /**
     * A simple zero getter serving as a test example
     * @return zero in the type int
     */
    int get_intValue(){
      int i = 0;
      return i;
    }
    /**
     * A simple zero getter serving as a test example
     * @return zero in the type long
     */
    long get_longValue(){
      long l = 0;
      return l;
    }
    /**
     * A simple zero getter serving as a test example
     * @return zero in the type short
     */
    short get_shortValue(){
      short s = 0;
      return s;
    }
    //
}

The template starts with the comment and a comment can actually contain any other comment starting.
Java comments are not nested.
The end of the template is, however the line that contains the */ string.
We want this line to be part of the template thus we precede it with the line
ESCAPE so it will not be interpreted as the end of the template.
On the other hand, for Java, this ends the comment.
To continue the template we have to get “back” into comment mode since we do not want the Java compiler to process the template as code.
(Last but not least because the template using placeholders is probably not a syntactically correct Java code fragment.) We need a new /* line, which we do not want to get into the template.
This line is, therefore, preceded with a line containing // SKIP.
(Skip lines can have optional // before the command.)

The result you can see in the generated code.
All methods have the proper JavaDoc documentation.

SEP1 and SEP2

Looping over the values you have to separate the names of the placeholders with , and | the list of the values.
For example, the sample above contains

    LOOP type,variable=int,i|long,l|short,s

two placeholder names type and variable and three values for each.
Placeholders do not need to contain special characters and it is the best if they are standard identifiers.
The values, however, may contain a comma or a vertical bar.
In that case, you can redefine the string (not only a single character) that the template LOOP command can use instead of the single character strings , and |.

For example the line

    SEP1 /

says that the names and the values should be separated by / instead of only one and

    SEP2 &

the list of the values should be separated by one character &amp;
string.
The SEP1 and SEP2 will have effect only if they precede the
LOOP command and they are effective only for the template they are used in.
Following the above commands, the LOOP example would look like

    LOOP type/variable=int/i&long/l&short/s

That way there is nothing to prevent us to add another value list

    LOOP type/variable=int/i&long/l&short/s&byte,int/z

which eventually will result in a syntax error with the example template, but demonstrates the point redefining the name and the value list separators.

Configuration

The generator is implemented the configuration tools supported by the Geci framework and all the parameters are configurable. You can redefine the regular expressions that match the template start, end, skip and so on lines in the unit test where the generator object is created, in the annotation of the class or in the editor-fold parameters.

Takeaway

The iterate generator is an extremely easy to use generator to create code that is repetitive. This is also the major danger: you should be strong enough to find a better solution and use it only when it is the best solution.

Supporting Java 8

Although Java has version 13 released as for now, there are a lot of production installations running with Java 8. As a professional, I develop Java 8 code many times even these days and I have to be happy that this is not Java 6. On the other hand as an open-source developer, I have my liberty to develop my Java code using Java 11, 12 or even 13 if that pleases me. And it does.

On the other hand, though, I want my code to be used. Developing a tool like License3j or Java::Geci, which are kind of libraries releasing Java 11 compatible byte code cuts off all the Java 8 based applications that may use these libraries.

I want the libraries to be available from Java 8.

One solution is to keep two branches parallel in the Git repo and have a Java 11+ and a Java 8 version of the code. This is what I have done for Java::Geci 1.2.0 release. It is cumbersome, error-prone and it is a lot of work. I had this code only because my son, who is also a Java developer starting his career volunteered.

(No, I did not pressure him. He speaks and writes better English than I do, and he regularly reviews these articles fixing my broken languages. If he has different opinion about the pressure, he is free to insert here any note till the closing parenthesis, I will not delete or modify that. NOTE: )

Anything above between the NOTE: and ) is his opinion.

The other possibility is to use Jabel.

In this article, I will write about how I used Jabel in the project Java::Geci. The documentation of Jabel is short but still complete and it really works like that for simpler projects. For example I really only had to add a few lines to the pom.xml in case of the Licenese3j project. For more complex projects that were developed over a year without thinking about any compromise for Java 8 compatibility, it is a bit more complex.

About Jabel

Jabel is an open-source project available from https://github.com/bsideup/jabel. If you have a Java 9+ project source you can configure Jabel to be part of the compilation process. It is an annotation processor that hooks into the compilation process and kind of tricks the compiler to accept the Java 9+ features as they were available for Java 8. The compiler will work and generate Java 8, Jabel does not interfere with the byte code generation, so this is as genuine as it can be out of the Java compiler fresh and warm. It only instructs the compiler not to freak out on Java 9+ features when compiling the code.

The way it works and why it can work is well written on the project’s GitHub page. What I wrote above may not even be precise.

Backport issues

When creating Java code using Java 9+ features targeting a Java 8 JVM it is not only the byte code version that we should care about. The code executed using the Java 8 JVM will use the Java 8 version of the JDK and in case we happen to use some classes or methods that are not available there then the code will not run. Therefore we have two tasks:

  • Configure the build to use Jabel to produce Java 8 byte-code
  • eliminate the JDK calls that are not available in Java 8.

Configure Build

I will not describe here how to configure Jabel to be part of the build using Maven. It is documented on the site and is straightforward.

In the case of Java::Geci I wanted something different. I wanted a Maven project that can be used to create Java 8 as well as Java 11 targets. I wanted this because I wanted Java::Geci to support JPMS just as before and also to create state-of-the-art byte code (class nesting instead of bridge methods for example) for those projects that run on Java 11 or later.

As the first step, I created a profile named JVM8. Jabel is only configured to run only when this profile is active.

This profile also sets the release as

<release>8</release>

so the very first time the compiler was freaking out when it saw the module-info.java files. Fortunately, I can exclude files in the POM file in the JVM8 profile. I also excluded javax0/geci/log/LoggerJDK9.java and I will talk about that later.

I also tried to use Maven to automatically configure the version number to have the -JVM8 postfix if it runs with the JVM8 profile but it was not possible. Maven is a versatile tool and can do many things and in case of a simpler project doing that should be the approach. In the case of Java::Geci I could not do that because Java:Geci is a multi-module project.

Multi-module projects refer to each other. At least the child module reference the parent module. The version of the child module may be different from the version of the parent module. It is kind of logical since their evolution and development are not necessarily tied together. However, usually, it is. In projects, like Java::Geci that has seven child modules and each child module has the very same version number as the parent the child modules can inherit all the parameters, dependencies, compiler options and so on, from the parent but the version. It cannot inherit the version because it does not know which parent version to inherit it from. It is a catch 22.

The Java::Geci development goes around this problem using the Jamal preprocessor maintaining the eight pom.xml files. Whenever there is a change in the build configuration it has to be edited in one of the pom.xml.jam files or in one of the included *.jim files and then the command line mvn -f genpom.xml clean will regenerate all the new pom.xml files. This also saves some repetitive code as the preprocessed Jamal files are not so verbose as the corresponding XML files. The price for this is that the macros used have to be maintained.

Java::Geci has a version.jim file that contains the version of the project as a macro. When targeting a Java 8 release then the version in this file has to be changed to x.y.z-JVM8 and the command mvn -f genpom.xml clean has to be executed. Unfortunately, this is a manual step that I may forget. I may also forget to remove the -JVM8 postfix after the Java 8 target was created.

To mitigate the risk of this human error I developed a unit test that checks the version number is coherent with the compilation profile. It identified the compilation profile reading the /javax0/geci/compilation.properties file. This is a resource file in the project filtered by Maven and contains

projectVersion=${project.version}
profile=${profile}

When the test runs the properties are replaced by the actual values as defined in the project. project.version is the project version. The property profile is defined in the two profiles (default and JVM8) to be the name of the profile.

If the version and the profile do not match the test fails. Following the philosophy of Java::Geci, the test does not just order the programmer to fix the “bug” when the test itself can also fix the bug. It modifies the version.jim file so that it contains the correct version. It does not, however, run the pom file generating Jamal macros.

As a result of this I will get release files with version x.y.z and also x.y.z-JVM8 after the second build with some manual editing work.

Eliminate Java 8+ JDK calls

Simple calls

This is a simple task at first sight. You must not use methods that are not in Java 8 JDK. We could do anything using Java 8 so it is a task that certainly possible.

For example every

" ".repeat(tab)

has to be eliminated. To do that I created a class JVM8Tools that contain static methods. For example:

public static String space(int n){
    final StringBuilder sb = new StringBuilder(/*20 spaces*/"                    ");
    while( sb.length() < n){
        sb.append(sb);
    }
    return sb.substring(0,n).toString();
}

is defined there and using this method I can write

space(tab)

instead of the invocation of String::repeat method. This part was easy.

Mimicking getNestHost

What was a bit more difficult is to implement the getNestHost() method. There is no such thing in Java 8, but the selector expressions included in the Tools module of Java::Geci lets you use expressions, like

Selector.compile("nestHost -> (!null & simpleName ~ /^Map/)").match(Map.Entry.class)

to check that the class Entry is declared inside Map, which it trivially is. It makes sense to use this expression even in Java 8 environment someone chooses to do so and I did not want to perform amputation dropping this feature from Java::Geci. It had to be implemented.

The implementation checks the actual run-time and in case the method is there in the JDK then it calls that via reflection. In other cases, it mimics the functionality using the name of the class and trying to find the $ character that separates the inner and the enclosing class name. This may lead to false results in the extremely rare case when there are multiple instances of the same class structures loaded using different class loaders. I think that a tool, like Java::Geci can live with it, it barely happens while executing unit tests.

There is also a speed drawback calling the method Class#getNestHost reflectively. I decide to fix it if there will be real demand.

Logging support

The last issue was logging. Java 9 introduced a logging facade that is highly recommended to be used by the libraries. Logging is a long-standing problem in the Java environment. The problem is not that there is not any. Quite the opposite. There are too many. There is Apache Commons Logging, Log4j, Logback, the JDK built-in java util logging. A standalone application can select the logging framework it uses, but in case a library uses a different one then it is difficult if not impossible to funnel the different log messages into the same stream.

Java 9 thus introduced a new facade that a library can use to send out its logs and the applications can channel the output through the facade to whatever logging framework they want. Java::Geci uses this facade and provides logging API for the generators through it. In case the JVM8 environment this is not possible. In that case Java::Geci channels the log messages into the standard Java logger. To do that there is a new interface LoggerJDK implemented by two classes LoggerJVM8 and LoggerJDK9. The source code for the latter is excluded from the compilation in case the target is Java 8.

The actual logger tries to get the javax0.geci.log.LoggerJDK9#factory via reflection. If it is there, then it is possible to use the Java 9 logging. If it is not there then the logger falls back to with the factory to javax0.geci.log.LoggerJVM8#factory. That way only the logger factory is called via reflection, which happens only once for every logger. Logging itself is streamlined and uses the target logging without any reflection, thus without speed impediment.

Takeaway

It is possible to support Java 8 in most of the library project without unacceptable compromise. We can create two different binaries from the same source that support the two different versions in a way that the version supporting Java 9 and later does not “suffer” from the old byte code. There are certain compromises. You must avoid calling Java 9+ API and in case there is an absolute need, you have top provide a fall-back and you can provide a reflection-based run-time detection solution.