Inject-able only in test?

This article is about some thoughts of test design and testability. Some questions that we discussed with my son, who is a junior Java developer and currently is employed and studies at EPAM Hungary (the same company but a different subsidiary where I work). All the things in this article are good old knowledge, but still, you may find something interesting in it. If you are a junior then because of that. If you are a senior then you can get some ideas on how to explain these things. If neither: sorry.

Introduction to the problem

The task they had was some roulette program or some other game simulation code, they had to write. The output of the code was the amount of simulated money lost or won. The simulation used a random number generator. This generator caused a headache when it came to testing. (Yes, you are right: the very basis of the problem was lack of TDD.) The code behaved randomly. Sometimes the simulated player was winning the game, other times it was losing.

Make it testable: inject mock

How to make this code testable?

The answer should be fairly obvious: mock the random number generator. Make the use of the source of randomness injected and inject a different non-random source during tests. Randomness is not important during testing and there is no need to test the randomness. We have to believe that the random number generator is good (it is not, it is never good, perhaps good enough, but that is a totally different story) and was tested by its own developers.

Learning #1: Do not test the functionality of your dependency.

We can have a field of type Supplier initialized to something like () -> rnd() lambda and in case of test it is overwritten using a setter.

Is testable good?

Now we changed the structure of the class. We opened a new entry to inject a random number generator. Is this okay?

There is no general yes or no answer to that. It depends on the requirements. Programmers like to make their code configurable and more general than they are absolutely needed by the current requirements. The reason that… well… I guess, it is because many times in the past programmers experienced that requirements have changed (no kidding!) and in case the code was prepared for the change then the coding work was easier. This is fair enough reasoning but there are essential flaws in it. The programmers do not know what kind of future requirements may come. Usually, nobody really knows, and everybody has some idea about it.

Programmers usually have the least knowledge. How would they know the future? Business analysts know a bit better, and at the end of the chain, the users and customers know it the best. However, even they do not know the business environment out of their control that may require new features of the program.

Another flaw is that developing of a future requirement now has extra costs that the developers a lot of times do not comprehend.

Practice shows that the result of such ‘ahead of time’ thinking is usually complex code and flexibility that’s hardly ever needed. There is even an acronym for that: YAGNI, “You Aren’t Gonna Need It”.

So, is implementing that injectability feature a YAGNI? Not at all.

First of all: a code has many different uses. Executing it is only one. An equally important one is the maintenance of the code. If the code cannot be tested, it cannot be reliably used. If the code cannot be tested, it cannot be reliably refactored, extended: maintained.

A functionality that is only needed for testing is like a roof bridge on a house. You do not use it yourself while you live in the house, but without them, it would be hard and expensive to check the chimneys. Nobody questions the need for those roof bridges. They are needed, they are ugly and still, they are there. Without them, the house is not testable.

Learning #2: Testable code usually has better structure.

But that is not the only reason. Generally, when you create a code testable the final structure will usually be more useable as well. That is, probably, because testing is mimicking the use of the code and designing the code testable will drive your thinking towards the usability to be on the first place and implementation to be only on the second place. And, to be honest: nobody really cares about implementation. Usability is the goal, implementation is only the tool to get there.

Responsibility

Okay, we got that far: testability is good. But then there is a question about responsibility.

The source of randomness should be hard-wired into the code. The code and the developer of the code are responsible for the randomness. Not because this developer implemented it, but this developer selected the random number generator library. Selecting the underlying libraries is an important task and it has to be done responsibly. If we open a door to alter this selection of implementation for randomness then we lose control over something that is our responsibility. Or don’t we?

Yes and no. If you open the API and provide a possibility to inject a dependency then you are not inherently responsible for the functioning of the injected functionality. Still, the users (your customers) will come to you asking for help and support.

“There is a bug!” they complain. Is it because of your code or something in the special injected implementation the user selected?

You essentially have three choices:

1. You may examine the bugs in each of those cases and tell them when the error is not your bug and help them select a better (or just the default) implementation of the function. It will cost you precious time either paid or unpaid.
2. The same time you can also exclude the issue and say: you will not even examine any bug that cannot be reproduced using the standard, default implementation.
3. You technically prevent the use of the feature that is there only for the testability.

The first approach needs good sales support or else you will end up spending your personal time fixing customers problem instead of spending your paid customer time. Not professional.

The second approach is professional, but customers do not like it.

The third is a technical solution to drive users from #1 to #2.

Learning #3: Think ahead about users’ expectations.

Whichever solution you choose the important thing is to do it consciously and not just by accident. Know what your users/customer may come up with and be prepared.

Prevent production injecting

When you open the possibility to inject the randomness generator into the code how do you close that door for the production environment if you really must?

The first solution, which I prefer, is not to open it wide in the first place. Use it via the initialized field holding the lambda expression (or some other way) that makes it injectable, but do not implement injection support. Let the field be private (but not final, because that may cause other problems in this situation) and apply a bit of reflection in the test to alter the content of the private field.

Another solution is to provide a package private setter, or even better an extra constructor to alter/initialize the value of the field and throw an exception if it is used in the production environment. You can check that many different ways:

• Invoke `Class.forName()` for a test class that is not on the classpath in the production environment.
• Use `StackWalker` and check that the caller is test code.

Why do I prefer the first solution?

Learning #4: Do not use a fancy technical solution just because you can. Boring is usually better.

First of all, because this is the simplest and puts all testing code into the test. The setter or the special constructor in the application code is essentially testing code and the byte codes for them are there in the production code. Test code should be in test classes, production code should be in production classes.

The second reason is that designing functionality that is deliberately different in the production and in the test environment is just against the basic principles of testing. Testing should mimic the production environment as much as economically feasible. How would you know that the code will work properly in the production environment when the test environment is different? You hope. There are many environmental factors already that may alter the behavior in the production environment and let bug manifest there only and silently remaining dormant in the test environment. We do not need extra such things to make our testing even riskier.

Summary

There are many more aspects of programming and testing. This article was addressing only a small and specific segment that came up in a discussion. The key learnings also listed in the article:

• Test the system under test (SUT) and not the dependencies. Be careful, you may think you are testing the SUT when actually you are testing the functionality of some dependencies. Use stupid and simple mocks.
• Follow TDD. Write the test before and mixed with the functionality development. If you don’t because just you don’t, then at least think about the tests before and while you write the code. Testable code is usually better (not just for the test).
• Think about how fellow programmers will use your code. Imagine how a mediocre programmer will use your API and produce the interfaces of your code not only for the geniuses like you, who understand your intentions even better than you.
• Do not go for a fancy solution when you are a junior just because you can. Use a boring and simple solution. You will know when you are a senior: when you no longer want to use the fancy solution over the boring one.

Advertisement

Converting objects to Map and back

In large enterprise applications sometimes we need to convert data objects to and from Map. Usually, it is an intermediate step to a special serialization. If it is possible to use something standard then it is better to use that, but many times the architecture envisioned by some lead architect, the rigid environment or some similar reason does not make it possible to use JOOQ, Hibernate, Jackson, JAX or something like that. In such a situation, as it happened to me a few years ago, we have to convert the objects to some proprietary format being string or binary and the first step towards that direction is to convert the object to a Map.

Eventually, the conversion is more complex than just

Map myMap =  (Map)myObject;

because these objects are almost never are maps by their own. What we really need in the conversion is to have a Map where each entry corresponds to a field in the “MyObject” class. The key in the entry is the name of the field, and the value is the actual value of the field possibly converted to a Map itself.

One solution is to use reflection and reflectively read the fields of the object and create the map from it. The other approach is to create a toMap() method in the class that needs to be converted to a Map that simply adds each field to the returned map using the name of the field. This is somewhat faster than the reflection-based solution and the code is much simpler.

When I was facing this problem in a real application a few years ago I was so frustrated writing the primitive but numerous toMap() methods for each data object that I created a simple reflection-based tool that to do it just for any class we wanted. Did it solve the problem? No.

This was a professional environment where not only the functionality matters but also the quality of the code and the quality of my code, judged by my fellow programmers, was not matching. They argued that the reflection-based solution is complex and in case it becomes part of the code base then the later joining average developers will not be able to maintain it. Well, I had to admit that they were correct. In a different situation, I would have said that the developer has to learn reflection and programming in Java on a level that is needed by the code. In this case, however, we were not speaking about a specific person, but rather somebody who comes and joins the team in the future, possibly sometime when we have already left the project. This person was assumed to be an average developer, which seemed to be reasonable as we did not know anything about this person. In that sense, the quality of the code was not good, because it was too complex. The quorum of the developer team decided that maintaining the numerous manually crafted toMap() method was going to be cheaper than finding senior and experienced developers in the future.

To be honest, I was a bit reluctant to accept their decision but I accepted it even though I had the possibility to overrule it based simply on my position in the team. I tend to accept the decisions of the team even if I do not agree with that, but only if I can live with those decisions. If a decision is dangerous, terrible and threatens the future of the project then we have to keep discussing the details until we get to an agreement.

Years later I started to create Java::Geci as a side project that you can download from http://github.com/verhas/javageci

Java::Geci is a code generation tool that runs during the test phase of the Java development life cycle. Code generation in Java::Geci is a “test”. It runs the code generation and in case all the generated code stays put then the test was successful. In case anything in the code base changed in a way that causes the code generator to generate different code than before and thus the source code changes then the test fails. When a test fails you have to fix the bug and run build, including tests again. In this case, the test generates the new, by now fixed code, therefore all you have to do is only to run the build again.

When developing the framework I created some simple generators to generate equals() and hashCode(), setters and getters, a delegator generator and finally I could not resist but I created a general purpose toMap() generator. This generator generates code that converts the object to Map just as we discussed before and also the fromMap() that I did not mention before, but fairly obviously also needed.

Java::Geci generators are classes that implement the Generator interface. The Mapper generator does that extending the abstract class AbstractJavaGenerator. This lets the generator to throw any exception easing the life of the generator developer, and also it already looks up the Java class, which was generated from the currently processed source. The generator has access to the actual Class object via the parameter klass and the same time to the source code via the parameter source, which represents the source code and provides methods to create Java code to be inserted into it.

The third parameter global is something like a map holding the configuration parameters that the source code annotation @Geci defines.

package javax0.geci.mapper;

import ...

public class Mapper extends AbstractJavaGenerator {

...

    @Override
    public void process(Source source, Class<?> klass, CompoundParams global)
                                                             throws Exception {
        final var gid = global.get("id");
        var segment = source.open(gid);
        generateToMap(source, klass, global);
        generateFromMap(source, klass, global);

        final var factory = global.get("factory", "new {{class}}()");
        final var placeHolders = Map.of(
                "mnemonic", mnemonic(),
                "generatedBy", generatedAnnotation.getCanonicalName(),
                "class", klass.getSimpleName(),
                "factory", factory,
                "Map", "java.util.Map",
                "HashMap", "java.util.HashMap"
        );
        final var rawContent = segment.getContent();
        try {
            segment.setContent(Format.format(rawContent, placeHolders));
        } catch (BadSyntax badSyntax) {
            throw new IOException(badSyntax);
        }
    }

The generator itself only calls the two methods generateToMap() and generateFromMap(), which generate, as the names imply the toMap() and fromMap() methods into the class.

Both methods use the source generating support provided by the Segment class and they also use the templating provided by Jamal. It is also to note that the fields are collected calling the reflection tools method getAllFieldsSorted() which returns all the field the class has in a definitive order, that does not depend on the actual JVM vendor or version.

    private void generateToMap(Source source, Class<?> klass, CompoundParams global) throws Exception {
        final var fields = GeciReflectionTools.getAllFieldsSorted(klass);
        final var gid = global.get("id");
        var segment = source.open(gid);
        segment.write_r(getResourceString("tomap.jam"));
        for (final var field : fields) {
            final var local = GeciReflectionTools.getParameters(field, mnemonic());
            final var params = new CompoundParams(local, global);
            final var filter = params.get("filter", DEFAULTS);
            if (Selector.compile(filter).match(field)) {
                final var name = field.getName();
                if (hasToMap(field.getType())) {
                    segment.write("map.put(\"%s\", %s == null ? null : %s.toMap0(cache));", field2MapKey(name), name, name);
                } else {
                    segment.write("map.put(\"%s\",%s);", field2MapKey(name), name);
                }
            }
        }
        segment.write("return map;")
                ._l("}\n\n");
    }

The code selects only the fields that are denoted by the filter expression.

Extending abstract classes with abstract classes in Java

The example issue

When I was creating the Java::Geci abstract class AbstractFieldsGenerator and AbstractFilteredFieldsGenerator I faced a not too complex design issue. I would like to emphasize that this issue and the design may seem obvious for some of you, but during my recent conversation with a junior developer (my son, Mihály specifically, who also reviews my articles because his English is way better than mine) I realized that this topic may still be of value.

Anyway. I had these two classes, fields and filtered fields generator. The second class extends the first one

abstract class AbstractFilteredFieldsGenerator
                  extends AbstractFieldsGenerator {...

adding extra functionality and the same time it should provide the same signature for concrete implementation. What does it mean?

These generators help to generate code for a specific class using reflection. Therefore the input information they work on is a Class object. The fields generator class has an abstract method process(), which is invoked for every field. It is invoked from an implemented method that loops over the fields and does the invocation separately for each. When a concrete class extends AbstractFieldsGenerator and thus implements this abstract method then it will be called. When the same concrete class is changed so that it extends AbstractFilteredFieldsGenerator then the concrete method will be invoked only for the filtered method. I wanted a design so that the ONLY change that was needed in the concrete class is to change the name.

Abstract class problem definition

The same problem described in a more abstract way: There are two abstract classes A and F so that F extends A and F provides some extra functionality. Both declare the abstract method m() that a concrete class should implement. When the concrete class C declaration is changed from C extends A to C extends F then the invocation of the method m() should change, but there should be no other change in the class C. The method m() is invoked from method p() defined in class A. How to design F?

What is the problem with this?

Extending A can be done in two significantly different ways:

• F overrides m() making it concrete implementing the extra functionality in m() and calls a new abstract method, say mx()
• F overrides the method p() with a version that provides the extra functionality (filtering in the example above) and calls the still abstract method m()

The first approach does not fulfill the requirement that the signature to be implemented by the concrete class C should remain the same. The second approach throws the already implemented functionality of A to the garbage and reimplements it a bit different way. In practice this is possible, but it definitely is going to be some copy/paste programming. This is problematic, let me not explain why.

The root of the problem

In engineering when we face a problem like that, it usually means that the problem or the structure is not well described and the solution is somewhere in a totally different area. In other words, there are some assumptions driving our way of thinking that are false. In this case, the problem is that we assume that the abstract classes provide ONE extension “API” to extend them. Note that the API is not only something that you can invoke. In the case of an abstract class, the API is what you implement when you extend the abstract class. Just as libraries may provide different APIs for different ways to be used (Java 9 HTTP client can send() and also sendAsync()) abstract (and for the matter of fact also non-abstract) classes can also provide different ways to be extended for different purposes.

There is no way to code F reaching our design goal without modifying A. We need a version of A that provides different API to create a concrete implementation and another, not necessarily disjunct/orthogonal one to create a still abstract extension.

The difference between the APIs in this case is that the concrete implementation aims to be at the end of a call-chain while the abstract extension wants to hook on the last but one element of the chain. The implementation of A has to provide API to be hooked on the last but one element of the call-chain. This is already the solution.

Solution

We implement the method ma() in the class F and we want p() to call our ma() instead of directly calling m(). Modifying A we can do that. We define ma() in A and we call ma() from p(). The version of ma() implemented in A should call m() without further ado to provide the original “API” for concrete implementations of A. The implementation of ma() in F contains the extra functionality (filtering in the example) and then it calls m(). That way any concrete class can extend either A or F and can implement m() with exactly the same signature. We also avoided copy/paste coding with the exception that calling m() is a code that is the same in the two versions of ma().

If we want the class F extendable with more abstract classes then the F::ma implementation should not directly call m() but rather a new mf() that calls m(). That way a new abstract class can override mf() giving again new functionality and invoke the abstract m().

Takeaway

1. Programming abstract classes is complex and sometimes it is difficult to have a clear overview of who is calling who and which implementation. You can overcome this challenge if you realize that it may be a complex matter. Document, visualize, discuss whatever way may help you.
2. When you cannot solve a problem (in the example, how to code F) you should challenge the environment (the class A we implicitly assumed to be unchangeable by the wording of the question: “How to implement F?”).
3. Avoid copy/paste programming. (Pasta contains a lot of CH and makes your code fat, the arteries get clogged and finally, the heart of your application will stop beating.)
4. Although not detailed in this article, be aware that the deeper the hierarchy of abstraction is the more difficult it is to have a clear overview of who calls whom (see also point number 1).

• Find a sample demo application at https://github.com/verhas/abstractchain
• Find the original, a tad more complex application that has this pattern at https://github.com/verhas/javageci

Reflection selector expression

Java::Geci is a code generator that runs during unit test time. If the generated code fits the actual version of the source code then the test does not fail. If there is a need for any modification then the tests modify the source code and fail. For example, there is a new field that needs a setter and getter then the accessor generator will generate the new setter and getter and then it fails. If there is no new field then the generated code is just the one that is already there, no reason to touch the source code: the test that started the generator finishes successfully.

Because Java::Geci generators run as tests, which is run-time and because they need access to the Java code structures for which they generate code Java reflection is key for these generators.

To help the code generators to perform their tasks there are a lot of support methods in the javageci-tools module.

com.javax0.geci
javageci-tools
1.1.1

In this article, I will write one class in this module: Selector that can help you select a field, method or class based on a logical expression.

Introduction

The class javax0.geci.tools.reflection.Selector is a bit like the regular expression class Pattern. You can create an instance invoking the static method compile(String expression). On the instance, you can invoke match(Object x) where the x object can be either a Field a Method or a Class or something that can be cast of any of those (Let’s call these CFoMs). The method match() will return true if x fits the expression that was compiled.

Selector expression

The expression is a Java String. It can be as simple as true that will match any CFoM. Similarly false will not match anything. So far trivial. There are other conditions that the expression can contain. public, private volatile and so on can be used to match a CFoM that has any of those modifiers. If you use something like volatile on a CFoM that cannot be volatile (class or method) then you will get IllegalArgumentException.

For classes you can have the following conditions:

• interface when the class is interface
• primitive when it is a primitive type
• annotation when it is an annotation
• anonymous
• array
• enum
• member
• local

Perhaps you may look up what a member class is and what a local class is. It is never too late to learn a bit of Java. I did not know it was possible to query that a class is a local class in reflection until I developed this tool.

These conditions are simple words. You can also use pattern matching. If you write extends ~ /regex/ it will match only classes that extend a class that has a name matching the regular expression regex. You can also match the name, simpleName and canonicalName against a regular expression. In case our CFoM x is a method or field then the return type is checked, except in case of name because they also have a name.

Conditions

There are many conditions that can be used, here I list only a subset. The detailed documentation that contains all the words is at https://github.com/verhas/javageci/blob/master/FILTER_EXPRESSIONS.md

Here is an appetizer though:

protected, package, static, public, final, synthetic,
synchronized, native, strict, default, vararg, implements,
overrides, void, transient, volatile, abstract

Expression Structure

Checking one single thing would not be too helpful. And also calling the argument of the method compile() to be an “expression” suggests that there is more.

You can combine the conditions to full logical expression. You can create a selector Selector.compile("final | volatile") to match all fields that are kind of thread safe being either final or volatile or both (which is not possible in Java, but the selector expression would not mind). You can also say Selector.compile("public & final & static") to match only those fields that are public, final and static. Or you can Selector.compile("!public & final & static") to match the final and static fields that are private, protected or package private, also as “not public”. You can also apply parenthesis and with those, you can build up fairly complex logical expressions.

Use

The usage can be any application that heavily relies on reflection. In Java::Geci the expression can be used in the filter parameter of any generator that generates some code for the methods or for the fields of a class. In that case, the filter can select which fields or methods need code generation. For example, the default value for the filter in case of the accessor generator is true: generate setters and getter for all the fields. If you need only setters and getters for the private fields you can specify filter="private". If you want to exclude also final fields you can write `filter=”!final & private”. In that case, you will not get a getter for the final fields. (Setters are not generated for final fields by default and at all. The generator is clever.)

Using streams it is extremely easy to write expressions, like

Arrays.stream(TestSelector.class.getDeclaredFields())
.filter(Selector.compile("private & primitive")::match)
.collect(Collectors.toSet());

that will return the set of the fields that are private and primitive. Be aware that in that case, you have some selector compilation overhead (only once for the stream, though) and in some cases, the performance may not be acceptable.

Experiment and see if it suits your needs.

I just forgot to add: You can also extend the selector during run-time calling the selector(String,Function) and/or selectorRe(String,Function) methods.