In Java there is your standard Iterator that has been around “forever” (since Java 1.2) and has hasNext() and next() methods, to check if the iterator has more elements and to get the next one, respectively. This is often sufficient, and can work well (though imperatively). Java 5 added a new Iterable interface along with the “enhanced for” loop, from which you can get an Iterator and a couple other things (primarily a forEach() method). ...
As a preview in JDK 23 and about to be final in the upcoming JDK 24, Java has added a capability to its Stream API to create new collectors using a new interface called Gatherer. Java has had a Collector API for a while now, but the Gatherer interface addresses some limitations, including the issue that there was no way to tell a Collector that processing was complete. If there were still some data the Collector was processing, this data was lost. You will see an example of this later. ...
How would you define “regression testing”? According to Wikipedia, Regression testing (rarely non-regression testing[1]) is re-running functional and non-functional tests to ensure that previously developed and tested software still performs after a change. In the modern age of test automation and continuous integration, tests are run (or should be) automatically by the build system. Whether testing functional or non-functional features, there really are only three categories of tests to run: Unit tests Integration tests System tests Let’s discuss each in turn. ...
Recently I was trying to do essentially a “map-reduce” using the Java Stream API … counting the number of occurrences of words in some input. This wasn’t for some huge “big data” input set. Using Java Stream API was sufficient. But the Stream API doesn’t have a groupBy() operation. While it does have map() and reduce() I couldn’t add a groupBy() … at least not directly. Since it was not obvious I thought I’d write a quick post on how to do it. ...
Since Java 7, the JDK includes a set of classes implementing a fork-join pattern. This is an approach decomposing work into multiple tasks that can be executed in parallel. Java provides ForkJoinPool and ForkJoinTask as the two primary classes implementing the approach. This post will cover an example of using these, by converting a Mergesort implementation from a recursive implementation to one using fork-join. Mergesort is a classic divide-and-conquer approach to sorting. The data to be sorted are split into two halves, and those halves each are split into two until the data can no longer be split. Then the resulting arrays are merged and sorted. ...
In my previous post about Using off-heap memory in Java programs I showed how to set up a memory-mapped file. Now that we have a memory-mapped file, let’s write something to the file. There are different approaches to how to serialize Java objects obviously including the built-in Java serialization. But in this post I’m going to use Protocol Buffers (AKA protobuf) for serialization, because why not? It will also give an opportunity to show how to automate the use of protobuf in your automated build using Gradle. ...
One of the nice things about modern programming languages is Garbage Collection. As a developer you don’t have to worry much about allocating and freeing memory for your objects. With Java you just ’new’ your class and voila a new instance of the class. And when the instance is no longer referenced, Java will take care of freeing the memory. When you create objects this way, the JVM allocates memory from ‘heap’ memory — memory it manages for you. ...
In this post I will explore the ksqlDb API for creating streams. This is the second post about using ksqlDb with Spring Boot. Read the first post here. A stream in ksqlDb is analogous to a stream in Kafka Streams. One difference is how you create the stream. As described in the first post, a stream in Kafka Streams is created programmatically with an API. StreamsBuilder streamsBuilder = new StreamsBuilder(); KStream<Integer, Order> ordersStream = streamsBuilder.stream("orders"); ordersStream .filter((key, value) -> value.orderType == Electronics) .groupByKey() .aggregate(() -> 0.0f, (key, value, sum) -> sum += value.orderAmount) .toStream() .to("output"); Topology topology = streamsBuilder.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, new Properties()); kafkaStreams.start(); ksqlDb uplevels this to a more familiar SQL syntax. Here’s an example from the ksqlDb quickstart. ...
In this post I will show how to use the ksqlDb Java REST client to interact with the ksqlDb server using a Spring Boot application as a foundation. The Boot app will also use Spring Webflux to help expand the use of reactive programming. I will also show the use of the Testcontainers framework for integrating deployment of Docker containers as part of integration testing, in this case with JUnit 5. What this will not cover is details of Apache Kafka or Spring Boot, or even ksqlDb and streaming systems, except to provide background to understand the code. There are plenty of other examples and posts that cover those details. ...