Monthly Archives: July 2019

Annotation Handling and JPMS

TLDR; Instead of annotation.getClass().getMethod("value") call
annotation.annotationType().getMethod("value").

(If you do not know what JPMS is, then you are not a Java developer. Stop reading, go ahead buy and read the book of the genius Nicolai Parlog https://www.manning.com/books/the-java-module-system about JPMS and become a Java developer again!)

All Java developers have heard about annotations. Annotations are with us since Java 1.5 (or only 1.6 if you insist). Based on my experience interviewing candidates I feel that most Java developers know how to use annotations. I mean, most developers know that it looks like @Test, or @Override and that they come with Java or with some library and they have to be written in front of a class, method, or variable.

A few developers know that you can also define an annotation in your code using @interface and that your code can do some metaprogramming using the annotation. Even fewer know that annotations can be processed by annotation processors and some of them can be processed during run time.

I could continue but long story short is that annotations are a mystery for most Java developers. If you think I am wrong stating how clueless related to the annotations most of the Java developers are, then consider that the number of programmers, (or coders, generally) was growing exponentially during the last 30 years and Java developers, especially, was doing so during the last 20 years and it is still growing exponentially. The exponential function has this feature: If the number of whatnots is growing exponentially then most of the whatnots are young.
That is the reason why most Java developers are not familiar with annotations.

To be honest, annotation handling is not something simple. It deserves its own article, especially when we want to handle annotations while using module systems.

During the final touches of the release 1.2.0 of the Java::Geci code generation framework I ran into a problem that was caused by my wrong use of annotations and reflection. Then I realized that probably most of the developers who handle annotations using reflection are doing so the same wrong way. There was hardly any clue on the net to help me understand the problem. All I found was a GitHub ticket and based on the information there I had to figure out what is really happening.

So let’s refresh a bit what annotations are and after that let’s have a look at what we may be doing wrong that was okay so far but may cause trouble when JPMS comes into the picture.

What is an annotation?

Annotations are interfaces that are declared using the interface keyword preceded with the @ character. This makes the annotation usable in the code the way we got used to. Using the name of the annotation interface with the @ in front of it (e.g.: @Example). The most frequently used such annotation is @Override that the Java compiler is using during compile time.

Many frameworks use annotations during run-time, others hook into the compilation phase implementing an annotation processor. I wrote about annotation processors and how to create one. This time we focus on the simpler way: handling annotations during run-time. We do not even implement the annotation interface, which is a rarely used possibility but is complex and hard to do as the article describes.

To use an annotation during run-time the annotation has to be available during run-time. By default, the annotations are available only during compile-time and do not get into the generated byte-code. It is a common mistake to forget (I always do) to put the @Retention(RetentionPolicy.RUNTIME) annotation on the annotation interface and then starting to debug why I cannot see my annotation when I access it using reflection.

A simple run-time annotation looks like the following:

@Retention(RetentionPolicy.RUNTIME)
@Repeatable(Demos.class)
public @interface Demo {
    String value() default "";
}

The annotations have parameters when used on classes, on methods, or on other annotated elements. These parameters are methods in the interface. In the example, there is only one method declared in the interface. It is called value(). This is a special one. This is a kind of default method. If there are no other parameters of an annotation interface, or even if there are but we do not want to use the other parameters and they all have default values then we can write

@Demo("This is the value")

instead of

@Demo(value="This is the value")

If there are other parameters that we need to use then we do not have this shortcut.

As you can see annotations were introduced on top of some existing structure. Interfaces and classes are used to represent annotations and it was not something totally new introduced into Java.

Starting with Java 1.8 there can be multiple annotations of the same type on an annotated element. You could have that feature even before Java 1.8. You could define another annotation, for example

@Retention(RetentionPolicy.RUNTIME)
public @interface Demos {
    Demo[] value();
}

and then use this wrapper annotation on the annotated element, like

@Demos(value = {
    @Demo("This is a demo class"),
    @Demo("This is the second annotation")})
public class DemoClassNonAbbreviated {
}

To ease the tendinitis, caused by excessive typing, Java 1.8 introduced the annotation Repeatable (as you can see on the annotation interface Demo) and that way the above code can be written simply as

@Demo("This is a demo class")
@Demo("This is the second annotation")
public class DemoClassAbbreviated {
}

How to read the annotation using reflection

Now that we know that the annotation is just an interface the next question is how can we get information about them. The methods that deliver the information about the annotations are in the reflection part of the JDK. If we have an element that can have an annotation (e.g. a Class, Method or Field object) then we can call getDeclaredAnnotations() on that element to get all the annotations that the element has or getDeclaredAnnotation() in case we know what annotation we need.

The return value is an annotation object (or an annotation array in the first case). Obviously, it is an object because everything is an object in Java (or a primitive, but annotations are anything but primitive). This object is the instance of a class that implements the annotation interface. If we want to know what string the programmer wrote between the parenthesis we should write something like

final var klass = DemoClass.class;
final var annotation = klass.getDeclaredAnnotation(Demo.class);
final var valueMethod = annotation.getClass().getMethod("value");
final var value = valueMethod.invoke(annotation);
Assertions.assertEquals("This is a demo class", value);

Because value is a method in the interface, certainly implemented by the class that we have access to through one of its instances we can call it reflectively and get back the result, which is "This is a demo class" in this case.

What is the problem with this approach

Generally nothing as long as we are not in the realm of JPMS. We get access to the method of the class and invoke it. We could get access to the method of the interface and invoke it on the object but in practice, it is the same. (Or not in case of JPMS.)

I was using this approach in Java::Geci. The framework uses the @Geci annotation to identify which class needs generated code inserted into. It has a fairly complex algorithm to find the annotations because it accepts any annotation that has the name Geci no matter which package it is in and it also accepts any @interface that is annotated with a Geci annotation (it is named Geci or the annotation has an annotation that is Geci recursively).

This complex annotation handling has its reason. The framework is complex so the use can be simple. You can either say:

@Geci("fluent definedBy='javax0.geci.buildfluent.TestBuildFluentForSourceBuilder::sourceBuilderGrammar'")

or you can have your own annotations and then say

@Fluent(definedBy="javax0.geci.buildfluent.TestBuildFluentForSourceBuilder::sourceBuilderGrammar")

The code was working fine up until Java 11. When the code was executed using Java 11 I got the following error from one of the tests

java.lang.reflect.InaccessibleObjectException: 
Unable to make public final java.lang.String com.sun.proxy.jdk.proxy1.$Proxy12.value() 
accessible: module jdk.proxy1 does not 
"exports com.sun.proxy.jdk.proxy1" to module geci.tools

(Some line breaks were inserted for readability.)

The protection of JPMS kicks in and it does not allow us to access something in the JDK we are not supposed to. The question is what do we really do and why do we do it?

When doing tests in JPMS we have to add a lot of --add-opens command-line argument to the tests because the test framework wants to access the part of the code using reflection that is not accessible for the library user. But this error code is not about a module that is defined inside Java::Geci.

JPMS protects the libraries from bad use. You can specify which packages contain the classes that are usable from the outside. Other packages even if they contain public interfaces and classes are only available inside the module. This helps module development. Users cannot use the internal classes so you are free to redesign them so long as long the API remains. The file module-info.java declares these packages as

module javax0.jpms.annotation.demo.use {
    exports javax0.demo.jpms.annotation;
}

When a package is exported the classes and interfaces in the package can be accessed directly or via reflection. There is another way to give access to classes and interfaces in a package. This is opening the package. The keyword for this is opens. If the module-info.java only opens the package then this is accessible only via reflection.

The above error message says that the module jdk.proxy1 does not include in its module-info.java a line that exports com.sun.proxy.jdk.proxy1. You can try and add an add-exports jdk.proxy1/com.sun.proxy.jdk.proxy1=ALL_UNNAMED but it does not work. I do not know why it does not work, but it does not. And as a matter of fact, it is good that it is not working because the package com.sun.proxy.jdk.proxy1 is an internal part of the JDK, like unsafe was, that caused so much headache to Java in the past.

Instead of trying to illegally open the treasure box let’s focus on why we wanted to open it in the first place and if we really need to access to that?

What we want to do is get access to the method of the class and invoke it. We can not do that because the JPMS forbids it. Why? Because the Annotation objects class is not Demo.class (which is obvious since it is just an interface). Instead it’s a proxy class that implements the Demo interface. That proxy class is internal to the JDK and so we can not call annotation.getClass().
But why would we access the class of the proxy object, when we want to call the method of our annotation?

Long story short (I mean a few hours of debugging, investigating and understanding instead of mindless stackoverflow copy/paste that nobody does): we must not touch the value() method of the class that implements the annotation interface. We have to use the following code:

final var klass = DemoClass.class;
final var annotation = klass.getDeclaredAnnotation(Demo.class);
final var valueMethod = annotation.annotationType().getMethod("value");
final var value = valueMethod.invoke(annotation);
Assertions.assertEquals("This is a demo class", value);

or alternatively

final var klass = DemoClass.class;
final var annotation = klass.getDeclaredAnnotation(Demo.class);
final var valueMethod = Demo.class.getMethod("value");
final var value = valueMethod.invoke(annotation);
Assertions.assertEquals("This is a demo class", value);

(This is already fixed in Java::Geci 1.2.0) We have the annotation object but instead of asking for the class of it we have to get access to the annotationType(), which is the interface itself that we coded. That is something the module exports and thus we can invoke it.

Mihály Verhás, my son, who is also a Java developer at EPAM usually reviews my articles. In this case, the “review” was extended and he wrote a non-negligible part of the article.

Dr Heinz M. Kabutz posted recently an article that is about a related matter, worth reading: https://www.javaspecialists.eu/archive/Issue273.html

Java hexadecimal floating point literal

How I met hexadecimal floating point numbers

I was developing a new functionality into Java::Geci to make it less prone to code reformatting. The current release of the code will overwrite an otherwise identical code if it was reformatted. It is annoying since it is fairly easy to press the reformatting key shortcut and many projects even require that developers set their editor to automatically format the code upon save. In those cases Java::Geci cannot be used because as soon as the code is reformatted the generator thinks that the code it generates is not the same as the one already in the source file, updates it and signals the change of the code failing the unit tests.

The solution I was crafting compares the Java source files first converting them to a list of lexical elements. That way you can even reformat the code inserting new-lines, spaces, etc. so long as long the code remains the same. To do that I needed a simplified lexical analyzer for Java. Writing a lexical analyzer is not a big deal, I created several for different reasons since I first read the Dragon Book in 1987. The only thing I really needed is the precise definition of what are the string, character, number literals, the keywords and so on. In short: what is the definition of the Java language on the lexical level and how is it processed. Fortunately, there is a precise definition for that, the Java Language Specification, which is not only precise but also readable and has examples. So I started to read the corresponding chapters.

To my bewilderment, I could see there that there is a possibility in the Java language to express a floating point in hexadecimal. Strange, is it? Since I have not ever seen it, first I thought that this was something new introduced in Java 12 but my investigation showed that it probably was introduced in Java 1.5 That was the very first Java version I really liked but not because of hexadecimal floating points. So this was how I met this beast in the standard face to face. I started to wonder if this beast can be found at all in the wild or is it only something that can be seen captive in the confinements of the text of the JLS. So…

I put up a vote on Twitter


As you can see nine decent humans answered the question, mostly saying that they have had no idea about this feature.

Probably hexadecimal floating points are the least known and used feature of the Java language right after lambdas and streams (just kidding… hexadecimal floating points are important, no?)

Even though I did some scientific study in the past, I cannot see any use of hexadecimal floating point literals.


What is a floating point number?

We will get to hexadecimal floating point numbers, but to understand that we have to know first what a floating point number, generally is.

Floating point numbers have a mantissa and exponent. The mantissa has an integer and a fractional part, like iii.ffff. The exponent is an integer number. For example, 31.415926E-1 is a floating point number and an approximation for the ratio of the diameter and the circumference of a circle.

Java internally stores the float numbers on 32 bit and double number on 64 bit. The actual bits are used according to the IEEE 754 standard.

That way the bits store a sign on a single bit, then the exponent on 8 or 11 bits and finally the mantissa on 23 or 52 bits for 32- or 64-bit float/double respectively. The mantissa is a fractional number with a value between 1 and 2. This could be represented with a bit stream, where the first bit means 1, the second means 1/2 and so on. However, because the number is always stored normalized and therefore the number is always between [1 and 2) the first bit is always 1. There is no need to store it. Tus the mantissa is stored so that the most significant bit means 1/2, the next 1/22 and so on but when we need the value we add to it 1.

The mantissa is unsigned (hence we have a separate signum bit). The exponent is also unsigned, but the actual number of bitshifts are calculated subtracting 127 or 1023 from the value to get a signed number. It specifies how many bits the mantissa should virtually be shifted to the left or right. Thus when we write 31.415926E-1f then the exponent will NOT be -1. That is the decimal format of the number.

The actual value is 01000000010010010000111111011010. Breaking it down:

  • 0 sign, the number is positive. So far so good.
  • 10000000 128, which means we have to shift the mantissa one bit left (multiply the value by two)
  • 10010010000111111011010 is 4788186/2^23+1 \approx 1.570796251296997. The hex representation of this bit stream is 0x490FDA

And here comes the

Hexadecimal floating point literal

We can write the same number in Java as 0x0.C90FDAP2f. This is the hexadecimal floating point representation of the same number.

The mantissa 0xC9aFDA should be familiar to the hexadecimal representation of the number above 0x490FDA. The difference is that the first character is C instead of 4. That is the extra one bit, which is always 1 and is not stored in the binary representation. C is 1100 while the original 4 is 0100. The exponent is the signed decimal representation of the actual bitshifts needed to push the number to the proper position.

The format of the literal is not trivial. First of all, you HAVE TO use the exponent part and the character for the exponent is p or P. This is a major difference from the decimal representation. (UPDATE: If the exponent was optional you could not tell if, for example, 0.55 is a decimal floating point or a hexadecimal floating point. A hexadecimal number can, by accident, contain only decimal characters and still be hexadecimal.)

After a little bit of thinking it becomes obvious that the exponent cannot be denoted using the conventional e or E since that character is a legitimate hexadecimal digit and it would be ambiguous in case of numbers like 0x2e3. Would this be a hexadecimal integer or 2\times 2^3. It is an integer because we use p and not e.

The reason why the exponent part is mandatory I can only guess. Because developers got used to decimal floating point numbers with e or E as exponent it would be very easy to misread 0xC90F.0e+3 as a single floating point number, even though in case of hexadecimal floating point p is required instead of e. If the exponent were not mandatory this example would be a legit sum of a floating point number and an integer. The same time it looks like a single number, and that would not be good.

The other interesting thing is that the exponent is decimal. This is also because some hexadecimal digits were already in use for other purposes. The float and double suffix. In case you want to denote that a literal is a float, you can append the f or F to the end. If you want to denote that this literal is double then you can append d or D to the end. This is the default, so appending D is optional. If the exponent were hexadecimal we would not know if 0x32.1P1f is a float literal or a double and having a lot of magnitudes different value. This way, that that exponent is decimal it is a float number (32+1/2)\times 2^1.

Java and IEEE 754

Java implemented the IEEE 754 standard strictly until Java 1.2 This standard defines not only the format of the numbers when stored in memory but also defines rules how calculations should be executed. After the Java release 1.2 (including 1.2) the standard was released to make the implementations more liberal allowing to use more bits to store intermediate results. This was and it still is available on the Intel CPU platforms and it is used heavily in numeric calculations in other languages like FORTRAN. This was a logical step to allow the implementations to use of this higher precision.

The same time to preserve backward compatibility the strictfp modifier was added to the language. When this modifier is used on a class, interface or method the floating point calculations in those codes will strictly follow the IEEE 754 standard.

Takeaway

  • There are hexadecimal floating point literals in Java. Remember it and also what strictfp is because somebody may ask you about it on a Java interview. No practical use in enterprise programming.
  • Do not use them unless it makes the code more readable. I can barely imagine any situation where this would be the case. So, simply put: do not use them just because you can.
  • Follow me on Twitter @verhas to get notification about new articles.

I think that is it, nothing more. By the time this article is published, I will probably be swimming across the lake of Zürich along with ten thousand people. This is a big event here.

Oh… and yes: if you have ever used hexadecimal floating point literals in Java to make it more readable, please share the knowledge in the comments. I dare say in the name of the readers: we are interested.

UPDATE: Joseph Darcy, (Engineer, OpenJDK developer at Oracle, marathoner, fast walker, occasional photographer, lots of other things.) provided feedback on Twitter. I copied his reply to here as it is absolutely valuable and adds value to this article for the benefit of the reader:

The mapping between decimal strings and particular settings of binary floating-point values is often non-obvious. Hexadecimal floating-point literals provide a straightforward text to binary fp mapping when needed, such as in tests. See https://blogs.oracle.com/darcy/hexadecimal-floating-point-literals