# HTTP/2 Server Push

The new version of the HTTP protocol, HTTP/2 lets the server to push content to the client before the client requests the particular content. There are many other modifications in the protocol if we compare the previous version 1.1 with the new version 2, but in this article, I will focus on the push functionality. I will discuss briefly how it can be used in a servlet, and I will also discuss a bit about how to test and see that it really works at all or not. Before writing this article my original intention was to create a demonstration of HTTP/2 showing how faster the sample page load is with the push than it is without. It is going to be one chapter in my video tutorial that is published by PACKT. During the development of the sample application I faced several problems, I have read some tutorials and debugged the sample code a bit gathering experience. In this article, I share this experience with you. That way this article is a bit more than just a simple introductory tutorial. Nevertheless, it is also a bit longer, so TLDR; if you are impatient.

# HTTP versions

HTTP/2 is a new version of the HTTP protocol. The protocol had three versions prior to 2. They were 0.9, 1.0 and 1.1. The first one was only an experiment starting in 1991. The first real version was 1.0 released in 1996. This was the version that you probably met if you were using the internet that time and you still remember the Mosaic browser. This version was soon followed by the version 1.1 next year, 1997. The major difference between 1.0 and 1.1 was the Host header field that made it possible to operate several websites on one machine, one server, one IP address, and one port.

## HTTP 1.1

Both versions 1.0 and 1.1 are extremely simple. The client opens a TCP channel to the server and writes the request into it as a plain text. The request starts with a request line, it is followed by header lines, an empty line and the body of the request. The body may be binary. The response has the same structure except the first line is not a request specific line, but rather a response obviously.

Imho, the simple approach was extremely important these days because it was inevitable to aid the spread and use of the protocol. You can find this type of pattern in the industry in different areas. This is how the world is developing in an agile way. First, you develop something simple that works and then you go on making it more and more complex as the environment requires it to be more powerful longing for more complexity. Making a long-shot and aiming for the complex and perfect solution the first time usually does not work. If you remember the film series Star Wars you know that death stars are never finished and they are razed at the end. But it does not mean that the simple version that we start with has no problems.

## HTTP/1.1 problems

HTTP/1.1 had a lot of problems. A real Englishman could say it was far from optimal using the network. It was wasting bandwidth. Early times when HTTP was invented a web page was text. Today it is text, CSS, JavaScript, images, sounds, videos and digitized smell, taste, and touch samples not to mention the direct neural stimulation command files. These last content types following videos in this taxative list are still rare but you should be prepared and that is what HTTP/2 is aiming. Preparing for the future. With HTTP/1.1 the browser downloads these resources one after the other in separate TCP channels. The number of the TCP channels the server or for that matter the client can handle is finite, therefore the browsers limit themselves not to open more than four TCP channels. It means that while four content elements for the page is downloading the other elements wait in a queue to be downloaded. If we have some very slow downloading content it may choke the download of the whole web page. You can run experiments with that writing some simple servlet that sends a complex HTML page to the browser and then you let some of the content elements download slow and others fast. Doing it in debug mode using Chrome you will get a nice Gantt chart of the download timings.

The network is also wasted creating a new TCP channel for each new HTTP connection. Creating a TCP channel require an SYN package traveling from the client to the server, then an SYN-ACK from the server to the client and then an ACK from the client to the server again before we can start sending data. In addition to these, the TCP protocol limits itself not to flood a lossy channel with a lot of data that is going to be lost anyway, so it starts slow, sending only a few data packages at the start and increases the speed only when it sees that packages are arriving in good shape and content. Let’s just think about that three-way handshake that starts the TCP channel. The network between the client and the server has a certain lag and has a certain bandwidth. The lag is the time needed for a package to travel from the client to the server or the other way around. The bandwidth is the number of bits that can be pushed through the network between the client and the server during a given time (like one second). Imagine a hose that you use to fill buckets. If it is long it may take two or three seconds until the water starts to pour after you opened the tap. If it is narrow filling a bucket can take a minute. In that case, 2-3 seconds at the start is not a big deal, you have to wait a minute anyway. On the other hand, if the hose is wide the water may run in amount filling the bucket in ten seconds. Three seconds delay is 30%. It is significant. Similarly the TCP channel slow-start lag is significant in fast networks, where the bandwidth is abundant. Fortunately, we go in this direction. We get fibers in our homes replacing the 19.2kbps dial-up modems. However, at the same time, HTTP/1.1 lag is an increasing problem.

There is a header field Keep-alive that can tell the server not to close the TCP channel, but rather reuse it for the next HTTP request and this patch on the scar of HTTP/1.1 helps a lot, but not enough. The blocking slow resource problem is still there. There are other problems with HTT/1.1 that HTTP/2 addresses but this, using many TCP channels to download several content pieces to the client from the same server is the main issue related to server push, which is indeed the topic of this article.

# HTTP/2

HTTP/2 uses a single TCP channel between the server and the client. When the content is downloaded from different servers the client will eventually open separate TCP channels to each server, but for the content pieces that come from the same server, HTTP/2 uses only one TCP channel. Using multiple TCP channels between a certain client and a single server does not speed up the communication, it was only a workaround in HTTP/1.1 to partially mitigate the choking effect of slow resources.

A request and a corresponding response do not use exclusively the TCP channel in HTTP/2. There are frames and the request and the response travels in these frames. If a content piece is created slowly and does not use the channel for some time then other resources can get frames and can travel in the same TCP channel. This is a significant boost of download speed in many cases. The change is also transparent for the browser programs being either JavaScript or WebAssembly. In the case of WebAssembly, the change is extremely simple: WebAssembly does not directly handle XMLHttpRequest, it uses JavaScript implemented and imported functions. In the case of JavaScript, the browser hides the transport complexity. JavaScript network API is just the same as it was before. You request a resource, you get one and you do not need to care if that was traveling in its exclusive TCP channel, mixed into HTTP/2 frames or was sent to you by pigeon post. On the server side, the approach is the same. In case of Java, a servlet application gets a request and creates a response. It is up to the container, the web server and to the client browser how they make it travel through the net.

The only difference, where server application may change is server push. This is a new feature and the new API gives an extra possibility for a server application to initiate a content push to the client.

# When to Push, What to Push?

The server push typically can be used in a situation when the application prepares some content slowly and the application knows that this is going to be slow and it also knows that there are other resources, typically many small icon images that can be downloaded fast and will be needed by the client. Thus the conditions for the server push are (PRECONDITIONS)

• the content requested is slow,
• the application knows it is going to be slow,
• there are other resources that will be needed by the client after this resource was downloaded,
• the application knows what these resources are,
• these resources can be downloaded fast.

In this case, the servlet can initiate one or more push that may deliver the content to the browser while the main content is prepared. When the main content is ready, and the client browser realizes that it needs the extra resources they are already there. If we do not push the resources they will be downloaded only after the original resource was processed when the browser realizes that the extra resources are needed. In a simple example, when the browser sees all the img tags and knows that it needs the icon images to render the page. This is what the above animation tries to show in a simple way.

# How to push

To initiate a push it is more than simple. The servlet standard 4.0 extends the HttpServletRequest to create a new PushBuilder whenever the servlet calls req.newPushBuilder() on the request object req. The push builder can be used to create a push and then invoking the method push() on it will initiate the sending towards the client. It is as simple as that. The only parameter that you have to set is the path of the resource to be pushed.

var builder = req.newPushBuilder();
...
builder.path(s).push();


# Sample application

The sample application to test server push is a servlet that responds with an HTML page that contains one hundred image references in a 10×10 table. Essentially these are small icons from the http://www.flaticon.com website

The first thing the servlet does is to initiate the icon downloads via server push. To do that it creates one push builder and this single object is used to initiate 100 pushes. After that, the servlet goes to sleep. This sleep simulates the slow inner working of the servlet response. A real application in this time would gather the information needed to send the response from the different database, from other services and so on. During this time the server and the client have enough time to download the PNG files. When the response arrives the files are there and the images are displayed instantaneously. At least that is what we expect.

The servlet has a parameter, named push that can be 1 or 0. If this parameter is 1 the servlet pushes the PNG files to the client, it is 0 then it is not. This way we can easily compare the speed of the two different behaviors.

    public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
final String titleText;
var builder = req.newPushBuilder();
var etag = now();
if (builder == null) {
titleText = "Good old HTTP/1.1 download";
} else {
var pushRequested = parse(req).get("push", 1) == 1;
if (pushRequested) {
titleText = "HTTP/2 Push";
sendPush(req, builder, etag);
} else {
titleText = "HTTP/2 without push";
}
}
var lag = parse(req).get("lag", 5000);
var delay = parse(req).get("delay", 0);
var sleep = new ThrottleTool.Sleeper();
sleep.till(lag);
resp.setContentType("text/html");
sendHtml(req, resp, titleText, etag, delay, sleep);
}


The servlet can also be parametrized with query parameters lag and delay.

lag is the number of milliseconds the servlet sleeps counting from the start of the servlet before it starts to send the HTML page to the client. The default value is 5000, which means that the HTML sending will start 5seconds after the servlet started.

delay is the number of milliseconds the servlet sleeps between each image tags. The default value is zero, that means the servlet sends the HTML as fast as it can send.

For the push, it is interesting how we get the builder object. The line

var builder = req.newPushBuilder();


returns a new builder object that can also be null. It is null if the environment does not allow push. The simplest case is when the servlet is queried through normal HTTP and not HTTPS. HTTP/2 works only over TLS secure channel and that way if we open the servlet via HTTP it will not be able to push anything.

After this, the method sendPush() sends the push contents as the name implies. Here is the method:

    private void sendPush(HttpServletRequest req, PushBuilder builder, long etag) {
for (var i = 0; i < 10; i++) {
for (var j = 0; j < 10; j++) {
var s = imagePath(i, j, req, etag);
builder.path(s);
builder.push();
}
}
}


The method imagePath() calculates the relative URL for the png based on the indices and this path is specified through the push builder. The builder is finally asked to push the content. This call to push() initiates the push on the server.

The builder is used literally one hundred times in this example. We do not need new builder objects for each push. We can safely reuse the same objects. The only requirement is that we set all the parameters before the next push we need. This is usually only the path.

# Image Server

The icon images are not directly served by the server in the demo application. The URL for the icon resources is mapped to a servlet that reads the icon PNG from the disk and writes it to the response. The reason for it is twofold. The chronologically first reason is that during the application debugging I needed information when and how the resource is collected and in case the resource is just a plain file that the server directly handles I do not have many possibilities to debug or log anything. The second reason is that the demo needs network throttling. Just as in case of the main HTML resource that waits 5 seconds before it vomits out the HTML text images also need to slow down to show a good demonstration effect. There is a throttling functionality in the debug mode of the browsers. However, it seems that this throttling in Chrome is architecturally between the cache where the pushed resources are temporarily stored and the DOM displaying engine. Implementing throttling on the demo server in our code is certainly at a good place from the experiment point of view and it can not happen that the already pushed and downloaded resource is loading throttled into the screen.

For this reason, the image servlet has two query parameters. imglag is the number of milliseconds the servlet waits after a hit arrives and before it starts to do anything. The default value in case the parameter is missing is 300 milliseconds. The other parameter is imgdelay that specifies the number of milliseconds the server consumes sending the bytes of the image to the client. This is implemented in a way that the server sends each byte individually one after the other and in case the current time is proportionally too soon to serve the next byte then the server sleeps a bit. The default value is 1000, which means that each icon is delivered from the server in approximately 1.3sec. The code itself is not too educational from the server push point of view. You can see it in the GitHub repository https://github.com/verhas/pushbuilder/blob/master/src/main/java/javax0/pushbuilder/demo/ImageServer.java

# PushBuilder methods

The PushBuilder class has many other methods in case you need to build a request that has some special characteristics. When you initiate the push of a resource you actually define an HTTP request, for which the resource would be the answer. The actual resource is provided by the server. If it is a normal static resource, like an image then the server will pick it up from the file system. If it is some dynamically calculated resource, then the server will start the servlet that is targeted by the request. What the push builder really builds up is a request that stays and is used on the server and the client will actually never issue this request after the resource was already pushed. The setter methods of the class set request attribute, like parameters, headers etc.

## method()

You can set the method of the request that the resource is a response to. By default, this is GET and this should be OK most of the time. If the resource is a response to a
POST, PUT, DELETE, CONNECT, OPTIONS or TRACErequest then we can not push the resource anyway because the HTTP/2 standard does not allow resources of those kinds to be pushed. What remains is the HEAD method that does not seem to be meaningful. I believe that the method method() is in the interface

• for compatibility reason if ever there will be some new method,
• for the sake of some application that uses some proprietary non-standard method
• or as a joke (a method called method).

# sessionId()

This method can be used to set the session id that is usually carried in the JSESSIONID cookie or in a request parameter.

# set, add and remove + Header()

You can modify the header of the request using these methods. When you invoke setHeader() the previous value is supposed to be replaced. The addHeader() adds a new header. When you reuse the same object adding the same header again and again may result the same header in the request multiple times. Finally the removeHeader() removes a header.

Note that some tutorials also add the header Content-type and the value in some of the tutorials is image/png. This is erroneous even though it does not do any harm. The headers we set on the push builder are used to request the resource. In our example, the images are not directly accessed by Tomcat, but rather are delivered through a servlet. This servlet reads the content of the PNG file from the disk and sends it to the writer that it acquires from the response object. This servlet will see the headers set in the push builder. If we set the content type to be image/png, it may think that we do are a special kind of stupid sending an HTTP GET request body that is PNG format. Usually, it does not matter, servlets and servers tend to ignore the content type for GET request especially when the content has zero bytes. What format does a non-existent picture have?

# get methods

What you can set on the push builder object you can also get out. You can use these getHeader(), getMethod(), getHeaderNames(), getPath() and so on methods to see what the current value of the values is in the push builder.

# Experiments

During the experiments, I could not see any difference between HTTP/2 with and without the push. The download using HTTP/2 was inevitably faster than HTTP/1.1, but there was no difference between the push and the non-push version. I used Chrome Version 67.0.3396.99 (Official Build) (64-bit). The browser supports the push functionality but the developer tools do not support the debugging of the push. Pushed contents are displayed as normally downloaded contents. There is some secret internal URL (well, not really a secret since it is published on the net), that will show the actual frames, but it is not too handy. What I could see there finally, that the push itself worked. The images were pushed to the browser while the main HTML page serving servlet was having its 5 seconds sleep, but after that, the browser was just downloading the images again, even if I switched on browser caching.

Since HTTP/2 was started by SPDY protocol and it was pushed by Google (pun indented (again)) I strongly believed that it is my code that does something wrong. Finally, I gave up and fired up a Firefox and here you go, this is what you can see on the screen captures:

# Summary Takeaway

Server push is an interesting topic and a powerful technology. There are a lot of preconditions to use it (see listed above, search for “PRECONDITIONS”). If these are all met then you can think about implementing it, but you should do experiments with the different network setups, delays and with the different client implementations. If the client implementation is less than optimal then you may end up with a slower download with a push than without one. There is a lot of room for development in the clients. Some of these developments will just mature the way the browsers handle server pushed resource, but I am almost sure that soon we will also have JavaScript API that may register call-back functions to be triggered when a push starts so that it will not only be the browser autonomously who can refuse a push stream but also the client-side JavaScript. Keep the eyes and your mind open for the development of HTTP/2.

Advertisements

# Upgrade to Java non-LTS version

There was a bug discovered in Java 9

https://bugs.openjdk.java.net/browse/JDK-8204322

and Richard Warburton tweeted about it

If you look at the twitter thread you can see that following this tweet I had some friendly fencing-bout with Nicolai Parlog about upgrading or not upgrading to non-LTS Java versions for commercial production applications (what a Shakespearian dilemma!). Part of the bouts was about misunderstandings as usually when using a limited communication channel. (Just a side note: I know Nicolai irl, he publishes a lot, a nice guy, great Java expert and in case you follow me on the blog, twitter or just in the street you definitely should also follow him, .. better not in the street, that is creepy.)

Since Twitter is a bit short of characters I decided to collect here my thoughts.

# What is the question at all?

For a lot of people, who use a computer and support themselves, install new software the question is obvious: use the latest version! Why would anyone want to use an outdated version of a software? I do not say that their answer is wrong for the situation they are in, but it is not a rationally calculated decision. You can decide based on rations or based on feelings or even random. Your decision may be sufficiently good, though science says that decisions based on well-established reasoning are usually more reliable and more often sufficiently good than other decisions. That is the reason why there are so many divorces.

The well-established reasonings about the upgrades and when to upgrade consider the cost and the benefit of the upgrade. What costs do we have?

# Costs and benefits

The first and most obvious cost item of the software upgrade is the installation cost: $C_i$.

It may be small but it can also be significant when the organization has several servers. The installation has to be tested. The new version of the software may not be compatible with the previous version from which we upgrade and it has to be tested. Proof of the pudding is eating. The testing should be done on the actual application that we use. For example, we may not care about the bug that falsely calculates an x += value expression if our code never uses the += operator. The testing cost is $C_t$.

The benefit $B$, the reason we do the upgrade is that the software runs better and thus it produces more income $\Delta I_u$ for the company and/or needs less maintenance cost $C_m$.

$B = \int_0^T \Delta I_u - C_m dt$

The summing up should be considered from the time of the installation until the time while the software is used.

# Assumptions, simplifications

In case of a Java version upgrade, it is extremely difficult to know the $\Delta I_u$ and the best estimate is to model it as zero. That essentially leads us to cost reduction. Estimating the cost reduction we usually ignore the short-term effects and consider only the long-term ones. Using Java 9 instead of Java 8 in a short time period rather increases than decreases the support costs. The support people have to learn the small intricacies of the new version and they are likely to make more mistakes shortly after the upgrade. This is essentially a temporary increase that we could consider to be part of the upgrade cost. In real life, if ever these calculations are performed these costs are not identified and do not get an estimated value. Rather they are ignored lessening the detected cost of an upgrade. This is understandable if you consider that these cost estimations are made by IT people and we love the newest versions and we want to upgrade. (As a demonstration read the Twitter thread.)

In the long run, when we consider the $T$ in the range of 5 to 10 years the maintenance cost is drastically increasing. A few years ago an old friend of mine was offered a year-long contract to maintain RPG application (not the role-playing game, google it, it eas before COBOL) for 400EUR per hour and he refused saying that the little time he has left he wants to spend with the family. (Old I meant literally in his case.) You cannot find a developer who would joyfully jump on the possibility to maintain a Java 6 application and if they accept they will not be shy to ask the price. You will see the same moue on their face when you mention that time age-old Java 8 or Java 9 version.

That means that the $\int_0^T C_m dt$ part is estimated as infinite for large $T$ values and zero for small ones. In layman terms, it implies that sooner or later we have to upgrade. Upgrade necesse est. Vivere non est necesse.

What we have left is the cost of the upgrade $C_i$.

# Upgrade cost

When we consider Java long term and non-long term support versions we have to compare the small steps upgrade process and the three stairs jump climbing from one LTS to the next LTS version.
Every upgrade cost has a constant part $C_{ic}$. This is the cost of downloading the media, installing on the server, running the tests, managing the meetings, communicating the out-of-service times and so on.

There is another part of the cost that depends on the previous version from which we upgrade, a variable part of the cost $C_{iv}$. When you upgrade from Java 8 to Java 9 you have to fight each and every compatibility issue that affects your application. Say this is $C_{iv}(J_8,J_9)$. Similarly, you will have $C_{iv}(J_9,J_{10})$ and $C_{iv}(J_{10},J_{11})$.

The big question is

$C_{iv}(J_8,J_9) + C_{iv}(J_9,J_{10}) + C_{iv}(J_{10},J_{11}) + 3 C_{ic} < C_{iv}(J_8,J_{11}) + C_{ic}$

?

It is certain that the variable part of the cost of the upgrade from Java 8 to Java 11 is larger than the same adding up from Java 8 to Java 9 and then from Java 9 to Java 10 and from Java 10 to Java 11. Using the notation for the difference

$\Delta C_{iv} =C_{iv}(J_8,J_{11}) - \big(C_{iv}(J_8,J_9) + C_{iv}(J_9,J_{10}) + C_{iv}(J_{10},J_{11})\big)$

the above question becomes simpler and easier to interpret:

$2 C_{ic} < \Delta C_{iv}$

# Interpretation and conclusion

The calculation mainly is playing around with latex and mathematical expressions, and they result in the obvious: if the constant cost of the upgrade that happens three times in one case versus one time in the other case is bigger than the extra cost of the big leap upgrading from Java 8 to Java 11 then you should not upgrade to the intermediate versions. If you work in a big organization with a lot of meetings, lot of administration, which is needed to keep the organization alive, then you do not upgrade for each an every version. If you are a small company, doing things fast and flexible then you will upgrade.

# Post Scriptum

Creating the formulas and expressing the calculation in a mathematicized way helps engineers to avoid some mistakes like ignoring some cost that should not be.

Do not think that being small, flexible and fast is always better than big, slow. Both have advantages and disadvantages and both have the place to live in the economy. A small company will never build a railway, a telecom network, or an ocean cruiser. The dinosaurs are extinct but there are whales as well as viruses. You may certainly enjoy more to work for a virus than for a whale.

# Comparing Golang and Understanding Value Types Vilnius DevDays Video

I have edited the video I recorded during the conference where I presented Comparing Golang and Understanding Value Types. The conference was DevDays Vilnius, the talk was May 24, 2018. The video shows the slides and, just for the sake of completeness and to increase the enjoyment factor, my slender myself presenting in PIP.

I delivered the talk also April, the same year in Mainz at W-JAX and it seems that this talk is doomed. In Mainz, another talk finished sooner and the presenter’s wireless mic was on the same frequency as mine. They forgot to switch it off and coming closer to our room the noise of the coffee break mixed with the signal of my mic.

Here in Vilnius, there was a presentation in the room next to my presentation where apparently there was no air conditioning and all doors and mobile walls were kept open all times. The presentation delivered by Sam Bellen was titled “Channel Your Inner Rockstar with the WebAudio API”. Don’t laugh! It was not funny!

# Comparing Golang and understanding Java Value Types

I start to talk about

Comparing Golang and understanding Java Value Types

The talk compares the memory model of the Go programming language to the memory model of Java. This comparison will help Java developers understand the planned Java 10 feature: Value Types. The talk will describe how these are implemented in Go, and why they so much needed for the Java language. At the end of the presentation, the audience will also understand why Value Types cannot be extended, immutable and passed always by value.
This talk is very lucrative for the audience because it is about Go language as well as a future of the upcoming Java version that is not available yet. Both are a gem for the audience.

right now.

You can look at the slides at https://github.com/verhas/compare-go-java-devdays2018-peter-verhas

# Prevent Hacking with Modules in Java 9

I start to talk about

Prevent Hacking with Modules in Java 9

Before Java 9 there was a lot of room to do tricky things mainly using reflection. Some of these possibilities were even considered as security holes. With the advent of Java 9, the module systems close these secret doors in Java runtime library and also allows library developers to do the same for their libraries.
The presentation will demonstrate some shocking and funny examples what you could do using Java 8 and then tries to do the same obviously failing using Java 9.

right now.

You can look at the slides at https://verhas.github.io/preventHack-J9-devdays2018-peter-verhas/#/

# Source Code Generation is not Good

The most important statement in this topic before we would even start to discuss anything else is that source code generation is a suboptimal solution. It may be needed and it may be a viable solution, but whenever source code is generated it could have been done some way better. It is just that the environment, the available tools, developers are not fit for the purpose. Let me give some examples.

When you program Java you use Eclipse, IntelliJ or NetBeans. Each of these IDEs is capable of generating hashCode(). What is wrong with it? The language could provide a declarative description of how to compute the function. The hash code depends on the hash code of the fields and the calculation is fairly standard. Why can’t we just define which fields should be taken into account and the language would implicitly provide us with the method? In this case, the language is insufficient for the purpose. I do not say that Java should provide such a feature. Maybe it should, maybe it should not.

In case of setters and getters the case is more prominent. Java needs them and we have to generate them whenever there is a need. Other languages, like C#, Swift or even Groovy support the feature on the language level.

Another example from my practice when I needed several business object classes converted to Map<String,String> with a special format. I created some utility classes that listed the fields using reflection and performed the conversion. This solution, however, was rejected during code review. The code was too complex and later teams who will be responsible for the maintenance may not be able to cope with the code. I could have said that they should hire cleverer people, but that costs more money and they wanted code that is cheap to maintain. The solution was to write extremely similar code for each and every business objects class. It could have been generated if there was any tool that could do that and, which could have been part of the build process, which again increases maintenance cost. In this case, the human environment was insufficient.

Please do not start flame war on this part of the article. This example is partially made up for NDA reasons, and after all it is not the major topic of the article.

# Navigare Necesse Est

The above examples clearly depict that source code generation is a must. We may not like it though, but it is a must. The next question is when to generate code, which phase of the development process?

It is fairly obvious that source code can only be generated before the compilation phase. You can generate source code after the compilation phase, but that is like calling a doctor after the patient is dead: no use. We can generate code during the build process, just before the compilation phase or as part of the editing process. Both have advantages and disadvantages.

## Editing Phase Source Code Generation

When you generate code while you edit the code the code generation does not need to be part of the build process. This means that the rebuild of the code is simpler, there are fewer
potential deviations from the standard build process and thus you are more likely to be able to do it when you work in a restricted enterprise environment. An example is when you use your IntelliJ to generate hashCode(). The generated method is available immediately in the editing environment, and functions like auto-complete will take the generated code into account.

The disadvantage is that the process is triggered manually. The more manual the process is the more room there is for human errors. You create a new field and you forget to update the hashCode() in the class. The generated code also gets into the source code repository that may not be optimal. Source code repository is for the source code and generated source code is not really source-code, is it?

## Build Process Source Code Generation

When you generate the source code during the build process the code generation tool will certainly rely on the last version of the source code. In our example there will not be any field left out from the hashCode() method.

The disadvantage is that the build process is more complex. Your favorite code generation tool may not be available or allowed in the environment you work in. The tools that can be hooked into the build process usually generate whole files. It is not likely that you will generate a hashCode() method into the middle of a class using a tool that runs on the build server in batch mode. Also, you will not have the generated code in your IDE and you may lose some of the code editing support.

Build time source code generation tools are usually also environment specific. You may have a tool that works for Java but does not work for Rust or Python projects.

There is no clear “one is better than the other” decision. Sometimes build time source code generation is better, other purposes are fit better with edit time source code generation. I created tools like Fluflu mentioned in my article “Named parameters in Java“, or Scriapt Java annotation processing tool described in the article “Don’t write boilerplate, use scriapt“. These tools are Java specific and build time executable. They are annotation processors, that hook into the Java compilation process and thus interestingly the IDEs continuous builds also handle them.

# Source Code Generation In-line

This time I want to write about a Python written tool Pyama that can be used to generate code not only for Java but also for Go, Rust, Markdown or just anything else. It is an editing phase tool and it was designed with editing in mind. The major idea was to automate the part of the editing process that can be automated.

## My Demanding Need

The demanding need was my editing the new edition of my book Java 9 Programming by Example published by Packt. The first edition of the book was edited in MS Word and I had to copy paste the source code samples from the IDE. However, book and code development is not a linear work. Sometimes the code was edited and modified after it was copied. It was a huge work to revisit each code sample in the book to see if the latest version is included in the document. I wanted something else, something more automatic. Luckily the second edition that will address Java 11 is edited with a different format that I can convert from Markdown. I edit the text in Markdown and I needed a tool that copies the code samples into the text.

The first idea was to create a tool that converts a .md.pre file that contains markdown and special directives controlling the source code inclusion into .md containing the code snippets. Such a solution, however, would not allow me to see the full rendered document in a Markdown WYIWYG editor. IntelliJ lets me render the markdown document text on the left side of the screen and see the result on the right side, which is a great help when I forget closing a backtick. Thus I decided to create a tool that can copy the snippets into my edited text file. It is also very handy that IntelliJ keeps the file almost all the time saved and reloads it when it is modified on the disk. Therefore I can edit the file in the editor and I can safely edit the file with any external tool. To develop this tool was also a nice Python learning project.

I also wanted to create something that was more general than just fetching snippets from code files and insert them into markdown documents. The outcome was a framework that, by now, has several extensions. One is handling snippets and markdown, others generate Java code (setters, getters, equals, hashCode, constructors, builder methods), handle text macros, execute Python scripts in any code files and so on. These extensions are samples and you can create other extensions with a few lines of Python code. As far as the book writing and Markdown Pyama proved to be an extremely valuable tool.

## Pyama Architecture

When generating code into already existing source files, it is evident that the unit of editing should be something more granular than a file. We should not overwrite a whole file with something new. The tool has to distinguish between the lines that need to be altered, or rather that are allowed to be altered and those that must not be touched. Pyama introduces the notion of a segment when processing files. The tool splits up the source files it works with into segments. Segments contain lines of the text files. Thus a pyama project works with files, each file contains segments and each segment contains lines. The segments of a file make up the whole file. In other words, there are no lines outside of segments. Pyama reads the contents of the files into the memory and then it invokes configured handlers (Python objects) to do whatever they should with the individual segments. When invoked, a handler works with a single segment. It can collect information from it, it can build up data structures to use later and it can read and modify the lines that are in the segment. This way the code of a handler is extremely simple, because it does nothing else but processes a list of strings and it does not need to care for anything else.

To decide where a segment starts an ends pyama asks the handler objects for regular expressions to identify lines that start and end segments. Different handlers may work with different segments and they may have different start and end patterns.

The segments in all files are processed a few times invoking the handlers in several passes. For example, the snippet reader may collect the code snippets from the configured source files into a snippet store where each snippet is identified with a name. In the next pass, the snippet writer handler looks at segments that start with a line referencing a named snippet and it replaces the lines of the segment with the current version of the collected snippet.

The snippet reader says that each line that contains START SNIPPET starts a new segment and such a segment lasts till a line containing END SNIPPET or till the end of the file. Then the code

// START SNIPPET main_java
System.out.println("Hello, world!");
// END SNIPPET


will collect a snippet that contains the code sample. The snippet writer manages segments that start with a line that contains USE SNIPPET and the name of the snippet and end with a line containing END SNIPPET. If there is a line in a file that the snippet writer processes that reads

USE SNIPPET main_java
System.out.println("Hello, outdated string world!");
END SNIPPET


it will replace it with

USE SNIPPET main_java
System.out.println("Hello, world!");
END SNIPPET


The lines with the USE SNIPPET and END SNIPPET remain in the code, but in most formats, it is possible to hide them into some comment field that the output (HTML renderer, or Java compiler) will ignore.

This is only the tip of the iceberg of this code generation, text processing tool. There are handlers that can number the snippet lines, trim the code, skip certain lines that may not be interesting for the printout, apply regular expression search and replace, or even execute small Python scripts that can create the segment text.

For example the following code

/* PYTHON SNIPPET xxx
fields = ["String name", "String office", "BigDecimal salary"]
print("    public void setParameters(",end="")
print(", ".join(fields), end="")
print("){")
for field in fields:
field_name = field.split(" ")[1]
print("        this." + field_name + " = " + field_name + ";")
print("        }")

print("""
public Map getMap(){
Map retval = new HashMap();\
""")
for field in fields:
field_name = field.split(" ")[1]
print("        retval.put(\""+field_name+"\", this."+field_name+");")
print("        return retval;\n        }")

END SNIPPET*/

public class SimpleBusinessObject {
//USE SNIPPET ./xxx
public void setParameters(String name, String office, BigDecimal salary){
this.name = name;
this.office = office;
this.salary = salary;
}

public Map getMap(){
Map retval = new HashMap();
retval.put("name", this.name);
retval.put("office", this.office);
retval.put("salary", this.salary);
return retval;
}
//END SNIPPET
}


can easily be changed to contain another field, just adding to the type and the name of the field to the array named fields. In real life examples the source printing code would be in some external file and imported, and probably the generated code would also be more complex than this sample. This code, however, enlightens that with minimal Python knowledge such manual tasks can be automated.

Please feel free to try and use pyama available from GitHub.

# DevDays Vilnius 2018

Starting today I participate DevDays in Vilnius for the whole week.

Come and see me on May 24, 2018, in Hall 2 at 14:00 and also at 15:55.

I will talk about

14:00 Prevent Hacking with Modules in Java 9 and
15:55 Comparing Golang and Understanding Java Value Types

https://devdays.lt/peter-verhas/

https://devdays.lt/