Spring Boot supports the use of YAML files to externalize configuration. YAML is a superset of JSON, human-readable and very convenient for specifying hierarchical configuration data. This is an example of yaml
You may also want to use Spring Cloud ZooKeeper Config and having your configuration distributed along your cluster. To get it you have to create a tree structure of nodes in ZooKeeper that is equivalent to the content of your yaml configuration files. It can be done through creation scripts for zkCli but is easier to use zookeeperdump. Zookeeperdump is a tool intended to help loading configurations for Spring Cloud ZooKeeper Config.
It can be downloaded with this maven command.
Zookeeperdump is a groovy script which requires that groovy binaries have been installed previously in your system. I recommend the use of gvm for managing groovy tools and versions.
To view all the tool options execute zookeeperdump.groovy without parameters:
For example to import a yaml file into ZooKeeper is only needed this command.
Apache ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. It is used as a cluster coordination service for demanding systems such as Hadoop, Kafka, Storm, Akka, Neo4j, Talend ESB and so on.
ZooKeeper is a very versatile tool and a perfect partner for a microservice architecture. These are its most common uses:
Spring Cloud ZooKeeper Config has a specific Spring PropertySource that loads transparently the configuration properties from ZooKeeper. It enables me to switch the source of my properties without affecting my application.
As a software engineer I usually find myself deciding whether to use existing tools or develop them on my own and to what extent. I have made my own checklist in order to help me to take those decisions.
Best option: use an existing tool. These are some of the benefits versus developing your own tool
Re-writing code that exists is wasted effort and makes the project takes longer than is necessary.
Your code is more likely to have more bugs/unforeseen issues than general widely used tools because those tools have been tested and patched multiple times. It's hard and time-consuming encountering the corner-cases by yourself.
Your code is more likely to miss features.
Your code has probably worse documentation.
Your code probably will not have community support (e.g. on stackoverflow, github, linkedin, ...).
Your code has to be maintained even when no new features are added (e.g. upgrading and adapting to a new version of a library).
If you use a well known tool, you may hire developers that had already used it.
Studying and truly understanding good existing solutions is a great way to learn from others and become a better developer.
Second best option: extend an existing tool
You could choose a widely used tool and extend it for adapting to your requirements.
You may share your extension (e.g. creating a pull request) and take advantage of the community (e.g. through bug reporting and bug fixing ).
You could be benefiting from the eventual improvements of the tool.
Third option: rethink your requirements
Perhaps you have the same issue than others but you intend to solve it in a weird way. Try to rethink your requirements in a more standard manner.
Software world is subject to trends. If you follow them you will have the software community with you: "Never be wrong alone, if you are to be wrong, better be wrong with most everyone else"
Nevertheless create your own software if ...
It's a part of the core business: "It's ok to reinvent the wheel when you're a wheel manufacturer".
Your requirements are specific and inflexible, hence you need to have a complete control with your own solution.
There is no tool that fits the requirements or that can be extended (with a proportional cost)
in order to satisfy them.
Existing tools are black boxes, are bad documented, expensive or with a poor community support.
therore-concurrent provides self-tuning thread-pools helping you to make the most of your system.
Recently, I have been working in the optimization of an OLTP system. The software has a SEDA architecture (Staged Event Driven Architecture) with lots of threads doing little works. I had to fight with the hard task of adjusting a hundred of parameters. Each of those parameters affected some others and so on.
For example if the number of concurrent database connections is set too low, it would cause a contention in getting connections. On the contrary, if that number is set too high, it could cause a lock-contention in the database when the threads want to access to some shared resources (index, row, block, etc.)
Even more, not always the processing of an event requires the same type of resources. A sudden change in the type of events that are being treated, can turn an optimal configuration into a suboptimal.
One of the most significant parameters is the number of threads assigned for each component. It is difficult to choose a good value if you don’t know how much the threads use each type of resource and how much are they coupled between each other.
Usually certain tasks have a higher priority and should be processed as soon as possible. This further complicates the choice of the configuration. Enforcing priorities and maximizing throughputs are opposite goals therefore it is necessary to define the scope of both.
In my experience a huge configurability can work against you. In a medium/big SOA system with a lot of service communications and complex workload profiles that even change over time, is almost impossible to get the optimal value for each of those parameters. Because of that I found interesting to develop a library that might be able to adapt quickly at runtime in order to make the most of the system.
Nowadays creating threads manually is not very common. Instead of that, thread-pools are frequently used. A thread-pool manages the creation and allocation of threads. JDK comes with some interesting and useful classes for managing threads. I list two of the most important:
ThreadPoolExecutor is a very flexible and configurable thread-pool that supports customization of queue size, minimum and maximum pool size, keep-alive time, etc.
Executors is a convenient class that creates thread-pools for the most usual cases.
I have developed the library therore-concurrent that takes advantage of those classes and extends some functionalities. The library contains analogous to the above classes.
SelfTuningExecutorService is a thread-pool that implements a mechanism for searching a good value for the pool size. The algorithm tries to maximize the throughput respecting the thread-pool priorities.
SelfTuningExecutors acts as the factory of SelfTuningExecutorService. It is recommended to use it as a singleton.
The following charts show how quickly SelfTuningExecutorService finds the optimal value.
Using SelfTuningExecutors directly
Add the dependency to the pom
The following snippet shows how can it be used.
The only new parameters are initPoolSize and priority.
initPoolSize is the initial amount of threads assigned to the pool.
priority is a positive number that works for SelfTuningExecutorService to limit the number of threads of this pool regarding others.
Integrating SelfTuningExecutors with Quartz Scheduler
Quartz-Scheduler has his own thread-pool interface and its name is "ThreadPool" (not surprise). The class SelfTuningThreadPool that is in the artifact therore-concurrent-quartz implements such interface. Integrating it is very easy, follow these steps:
Add the dependency to the pom
Change the configuration properties of quartz
Integrating SelfTuningExecutors with Apache Camel
I love Apache Camel. It offers a lot of components supporting integration with different technologies. But if none of them actually help you yet, it's pretty easy to make your own component.
Camel's team has thought very well the threading model. They use the concept (and interface) of ThreadPoolProfile which is a kind of thread-pool-template that you can use to instantiate several pools with the same configuration. If that is not enough, you can program your own implementation of ExecutorServiceManager, the Camel's thread-pool provider. Simplifying, think about it like the Executors class of the JDK.
I've just done that, SelfTunigExecutorServiceManager is the name of my own implementation of ExecutorServiceManager. It is located in other maven module therore-concurrent-camel. I'll explain how to use it.
Add the dependency to the pom
The following snippet contains two connected routes with SEDA component and SelfTunigExecutorServiceManager
I have figured out that there are many elements that might turn into selftuning ones. I chose ThreadPool because from my point of view is one of the most important, used and easy to test element.
Moreover, most of the modern libraries and frameworks feature different ways to extend their factories, providers and templates. All of that aims to develop general purpose classes and integrate them with lots of frameworks.
Many libraries use system properties for settings. For example, you usually have to set a property to enable debug traces or to activate certain behaviors. However, sometimes system properties that are used are not well documented or you are not sure that they are having some effect. In such cases it would be useful to have a way to know which system properties are queried.
There are several manners to do this but I have found a simple way that is not so intrusive. By creating a custom Security Manager you can define a method that checks property accesses. This method is able to send to the logger or to the standard output what property has been queried.
This class should be compiled and available in the classpath.
Now we create in the current directory a file named “java.policy” with the following content:
We just have everything we need to trace calls to System.getProperty. Next we add these parameters to the java call: