Friday, March 12, 2010

The OSGi Bundle Buddy

The Problem

As probably every developer, I want to keep my compile-test-deploy cycle as short and fast as possible. It shouldn't take longer than a few seconds to test a new feature or a bug fix. As long as I have unit tests I can directly run the tests in my IDE or use a tool like sbt (when writing Scala code) to have continuous test execution. However, often I want to evaluate the final product as well, and not only an isolated piece covered by a unit test. Do all my bundles work together? How does the UI look like? How responsive is the AJAX search box?

In order to do this, I need to combine my build artifacts and my target runtime. Since I am a full time OSGi developer, 99% of my build artifacts are bundles. The target runtime varies and depends on the project and task. Sometimes I deploy on Eclipse Equinox, sometimes (and more often recently) I deploy on Apache Felix Karaf and every now and then I embed a OSGi runtime within a WAR and deploy everything in Apache Tomcat. Obviously, executing this cycle takes much longer than running a few unit tests:

  1. Compile the source code
  2. Package all bundles
  3. Assemble/collect all bundles
  4. Copy bundles to target runtime
  5. Make sure the target runtime applies the changes
Especially 3 and 4 are most annoying. What if I only updated one bundle? I can easily do a "mvn package" in the corresponding module directory but I am still forced to execute the top level assembly process. This will collect all bundles, regardless of which I changed, and put them in a common location. From there, I have to manually copy the bundles to the target runtime. Given the presence of a tool like "mvn jetty:run", which automatically watches my class files and restart the web application on changes, I always thought that this process is way to complex. One of OSGi strengths is the dynamic, fine-grained update mechanism so with a bit of plumbing we should be able to improve our build process with it. Tools like fileinstall only help with step 5, since they rely on someone who collects all bundles and copies them to the watched directory.

The Solution

What I wanted is a tool that takes a running target runtime and my build environment and synchronizes them whenever the build changes. This tool should be independent from a specific target runtime (Equinox, Felix, ...) and independent from the build tool (Maven, Ant, Bnd in Eclipse, ...) and it only took approx. 150 lines of code (probably my best LOC/usefulness ratio so far) to implement this: The Bundle Buddy (TBB)

TBB is one simple bundle that you add to your target runtime during the development. Before you start the target runtime, you have set the environment variable "tbb" to the top-level directory of the project you are working on, e.g.

export tbb=/home/roman/workspace/myfancyproject

When you now start the runtime, TBB will recursively collect all jar files in that directory and align the result with the bundles already installed. Whenever a jar file in the build directory gets changed, TBB will update the corresponding bundle in the running OSGi framework automatically. It doesn't matter that the bundle location is different from the build directory path. TBB will synchronize by using the Bundle-SymbolicName and Bundle-Version.


You can get the TBB bundle file at github:


Currently I have these additional features in mind. Feel free to contact me if you have a fancy idea!
  • Specify more than one directory
  • Watch the class files as well and re-create the bundle jar file on the fly (only works if the manifest does not need to change)

Thursday, February 11, 2010

Why I hate everything you love about Java

A couple of days ago, Nick Kallen, the author of, wrote an interesting blog entry titled "Why I love everything you hate about Java" (link). He starts with addressing the "hipster programmers who loves Clojure, Ruby, Scala, Erlang, or whatever" and why he thinks that their sympathy for convention-over-configuration is a fallacy. He continues to discuss why his favorite design pattern (dependency injection, factories and decorators) are key features for modular applications.

The blog entry is a very good read and every Java/OO programmer should be familiar with these concepts. However, I disagree with most of his statements about convention-over-configuration and functional programming in particular. Overall, the blog reads a bit like "Why OO design pattern are better than functional programming and why those hippie trends like convention-over-configuration are overrated". It's always hard to cover everything in detail in a blog entry and often you simply have to set a focus (that's why it's called a blog and not an academic paper, right?) and here the focus clearly was set on object-orientation and design pattern. Within the OSGi enterprise expert group we had a short off-topic discussion about this blog entry and someone pointed out that it would be nice to see some counter examples.

Challenge accepted! I will use Clojure to convert the examples to a more functional style.


In the first part of the blog entry Nick presented an API usage where he always parametrizes the use of a ThreadPoolExecutor instead of using a more simplistic approach driven by convention-over-configuration. The used configuration "depends entirely on the nature of the problem you're solving and how callers of this code behave". Nick states that this boilerplate code "is really important when you work at massive scale" and that it provides "a way to configure the behavior of the system". He further describes that this style of API usage leads to modular software.

I think that this has nothing to do with the language or even the paradigm you choose since it's simply a matter of good API design. Provide meaningful defaults but do not tie the user to unchangeable behavior. However, I think that we can already improve his code and that his requirements even show some weaknesses of object-oriented programming! First, lets define the requirements:

1. Create a parametrized ThreadPoolExecutor
2. Execute a future task
3. The configuration depends on the problem and the caller

To keep the code simple, I will limit the configuration to PoolSize (just to demonstrate it, I won't change the size) and ThreadFactory. I will use the ThreadFactory to assign different names to the created threads. This provides us with an easy way to validate our configuration during task execution.

First, we create variables to define the default values for the ThreadPoolExecutor and ThreadFactory. We will later override the values when we want to parametrize:

(def *pool-size* 10)
(def *thread-name* "default name")

We need a ThreadFactory that uses the *thread-name* variable every time it creates new threads:

(def thread-factory (proxy [ThreadFactory] []
(newThread [runnable] (Thread. runnable *thread-name*))))

For testing purposes we will need several ThreadPoolExecutors so lets create a factory function that creates and returns a new executor. The created executor will use the *pool-size* and the thread-factory which in turn uses the *thread-name*:

(defn create-executor
(ThreadPoolExecutor. *pool-size*
(long 500)

We also need a function that will get called by the executor. We create another factory function that returns FutureTasks. Since every Clojure function implements the java.lang.Runnable interface we can easily pass a Clojure function to the FutureTask constructor. The factory function takes a task-id as parameter. We will use this ID to identify the created task later. When the created tasks get called by the executor, they simply print their ID and the name of the current thread:

(defn create-demo-task
(FutureTask. (fn []
(println "Task ID:" task-id
"Thread Name:" (.getName (Thread/currentThread))))))

Lets try it (note: The print to stdout will happen in a different thread so you might not see the output in your normal console):

(.execute (create-executor) (create-demo-task 1))

=> Task ID: 1 Thread Name: default name

Nothing special so far. The task gets executed and prints its ID and the name of the thread. So far we followed the convention-over-configuration way since we simply used our default value of "default name" for the thread. Our executor usage is very simple and we only implemented requirement 1 and 2. Nick argued that this simplicity is only an illusion and that you therefore always have to write the boilerplate code (for requirement 3). However, Clojure has a simply yet powerful mechanism to get a simple API and a flexible configuration at the same time: Bindings! Lets assume our problem demands a different thread name and that therefore the caller wants to influence the configuration:

(binding [*thread-name* "my new name"]
(.execute (create-executor) (create-demo-task 2)))

=> Task ID: 2 Thread Name: my new name

Bindings can be nested and each binding has its own scope. In the next example, the inner binding overrides the name with "my second name":

(binding [*thread-name* "my first name"]
(.execute (create-executor) (create-demo-task 3))
(.execute (create-executor) (create-demo-task 4))

(binding [*thread-name* "my second name"]
(.execute (create-executor) (create-demo-task 5)))

(.execute (create-executor) (create-demo-task 6)))

=> Task ID: 3 Thread Name: my first name
=> Task ID: 4 Thread Name: my first name
=> Task ID: 5 Thread Name: my second name
=> Task ID: 6 Thread Name: my first name

As you can see, the use of binding is completely optional and we only need to use it when we want to provide a different configuration. Requirement 3 is a very important one: "The configuration depends on the problem and the caller" and this is where object-oriented programming shows some weaknesses:
  • The caller (who knows what configuration makes sense) is often not the one who creates the object (where we need to apply the configuration)! Sometimes the objects get created later in the call chain, sometimes the objects were already created long before the caller knew something about it.
  • Once our tangled web of objects is instantiated, it's quite hard to change it. We either have a global dependency injection configuration or we pass factories around.
  • Whenever we create a new object (to apply a new configuration) we in fact create a new tree of objects since the object creation might imply the creation of child objects. This means that we often have a redundant set of trees with only slight differences. These trees often need to share some other objects so we again pass factories and references around etc. The beloved OO-design pattern are often just workarounds for problems introduced by object-oriented programming!
So far my examples mixed the concepts of executor creation, defining new thread names and running tasks. To add the concept of layers, we now assume that we have an application-wide configuration that creates the executor. For whatever app-specific reason, it's common in our application that new threads should get a random name in some contexts. We only need to capture this behavior once in a new function:

;; Create application executor
(def app-executor (create-executor))

;; Capture "random name" behavior
(def with-random-name #(binding [*thread-name* (str (rand))]

The application code now only needs to use the with-random-name function and the *thread-name* variable is now an implementation detail of our application-wide configuration:

#(.execute app-executor (create-demo-task 7)))

=> Task ID: 7 Thread Name: 0.5414362847600062

Easy, isn't it? Obviously, you can create as many functions, compose them, pass them to other functions etc. This is a very powerful programming concepts and I fail to see how "function composition is a degenerate case of the Decorator pattern", as Nick called it.

This style of programming is called context-oriented programming and I hope that we will see more use of it in the future.

The design patterns of modularity

The blog continues to describe the use of dependency injection, decorators and factories and shows why those pattern are important for modular application. Again, I argue that those pattern can be easily implemented in functional languages and that the result is often more flexible and probably easier to use. The examples heavily depend on OO constructs (e.g. class hierarchies) and therefore do not translate 1:1 to a functional style. However, this shouldn't be a big problem since the purpose is to demonstrate the design pattern, not the API of a query framework. We assume that we have a query function and simulate a long running operation with Thread/sleep:

(defn query [q]
(Thread/sleep (rand 1000))
(println "Run query:" q))

In practice, the function would contact the database, return the query data structure etc. The first task is to add a TimingOutQuery that cancels the query after a certain amount of time. To keep our Clojure code simple we instead create a timing decorator that prints how long the operation took (a timing function is already implemented in the Clojure core library):

(defn timing-query [q]
(time (query q)))

This function simply delegates to the original query function. With this design we would need to create a wrapper function for each database function (query/select/cancel) which is clearly not an option. Fortunately it's quite easy to generalize this abstraction:

(defn timing-fn [f & args]
(time (apply f args)))

The function timing-fn takes a function and an arbitrary number of arguments as parameters, calls the function with the arguments and wraps everything in Clojure's time function. We simply created a generic timing decorator that we can use to wrap any function:

(timing-fn query "query string")

Nick stated that "function composition is a degenerate case of the Decorator pattern" but it's the object-oriented implementation which is limited because e.g. only Query objects can be decorated. The TimingOutQuery and Query are even tied together via class inheritance, one of the most strict coupling you can use! Making the OO version generic would involve another abstraction, maybe new interfaces, etc.

One of the advantages of decorators is that you can test the code in isolation, no matter if you use the OO or the functional way. As described above, the timing-fn is ever easier to test since it does not depend on a specific type that it will wrap. For example, we could easily wrap the println function:

(timing-fn println "Hello World!")

After we decomposed everything into small functions, we need factories to put everything together again. The idea is that user code should never know what type of query (normal query, timing query, ...) it is using. We therefore pass factories to the user code. These factories return the actual query function that the user code should use. First we need to create 2 factory function, one for normal queries and one for timing queries:

(defn query-factory []
(fn [q] (query q)))

(defn timing-fn-factory [factory]
(fn [q] (timing-fn (factory) q)))

It is important to note that while the query-factory has no parameters, the timing-fn-factory takes another factory as parameter. Our timing factory knows nothing about the query function and is completely generic. Next we need a user code function that takes a factory as parameter and invokes it to get and use the actual query function:

(defn load-users [factory]
((factory) "load users"))

Lets use it with the normal query function:

(load-users query-factory)

Using the timing-fn-factory is a bit more tricky, since a) we want that the user code calls the factory and b) the user code does not know that the factory takes another factory as parameter. Hence we need something like "here is your factory and I already fill in the parameters for you" feature. This pattern is called partial application in functional languages and is probably one of the most basic operation:

(load-users (partial timing-fn-factory query-factory))

We simply call partial with the function and parameters.

As I wrote in the beginning, I think that every Java/OO programmer should be familiar with the concepts Nick presented and they indeed are very important. However, I hope that I was able to show that the functional programming world is powerful as well and that there are more reasons than just convention-over-configuration why "hipster programmers who loves Clojure, Ruby, Scala, Erlang, or whatever" are adopting functional programming languages.

Tuesday, November 17, 2009

OSGi Enterprise Management Suite

For almost 10 years, ProSyst is actively involved in the OSGi standardization process and focused on offering products and services around the OSGi technology. Our flag ship products are mBedded Server, and highly optimized OSGi framework implementation for embedded systems, and mPower Remote Manager (mPRM), a remote management system for OSGi-based system. The mPRM is used in a variety of deployments and is probably managing thousands and thousands of OSGi container by now.

Around 3 years ago, Java enterprise vendors got interested in OSGi. As a result of this, the OSGi enterprise expert group was created (EEG) and for the last 2 years I've had the pleasure to participate in this group. Furthermore, I got the chance to join and lead ProSyst's enterprise team to adopt the successful mPRM system to OSGi enterprise requirements. Java developers and architects quickly adopted the OSGi technology (server as well as applications) and started to profit from OSGi. While this improves the application's runtime and design, OSGi also brings enhanced remote server and application management features. Multi version support, dynamic updates and fine-grained management are well-known strengths of OSGi.

Our enterprise team worked hard in the last 2 years and I am proud to say that we finally finished the development and are about to release the "OSGi Enterprise Management Suite" (EMS). EMS is based on mPRM and focuses on enterprise OSGi requirements and use cases. Unlike our application stack for embedded environments, EMS focuses on managing Apache Felix and Eclipse Equinox based runtimes. Nevertheless, EMS is framework agnostic and will work with any OSGi 4.1 compliant implementation.

The key use case for EMS is remote server and application management. Let's say you finished your OSGi-based web application that you now want to bring into production. Even though the initial deployment might be easy, controlling the running application and using OSGi's fine-grained management abilities requires a sophisticated managed system. Another scenario is where you have dozens of deployments at e.g. customer side and want to update certain modules for important security bugfixes. Doing this by hand with in a remote shell is clearly not an option. EMS fulfills these needs by providing a central managing system. Some of my favorite features are:
  • Bundle management
    Typical bundle install/update/remove operations.

  • Multi-server management
    Install bundles on hundreds of servers in one operation. Useful for e.g. SOA and grid deployments. EMS will align the operation with each server individually.

  • Software repository
    EMS has its own software repository used to store bundle artifacts. EMS will use the repository and it's metadata for proper dependency management. Maven user can directly import their Maven artifacts (specified in a POM, including transitive dependencies, support for Maven repositories) into the repository.

  • Snapshots
    Whenever the server state changes (e.g. bundles), EMS will store a copy of the new state. This allows "roll backs" of e.g. faulty deployments even weeks after the deployment happend.
  • Logging
    EMS will collect all log messages on the server and transfer to them to the central EMS system. Support for e.g. OSGi LogService, log4j, commons logging, Java Logging
  • JMX representation
    EMS will register JMX MBeans in the managed server to present the OSGi state for JMX clients (e.g. Hyperic)
  • Configuration
    Front-end for Configuration Admin
  • Web UI frontend

We will officially announce EMS in the next couple of weeks. However, I just wanted to share this information with you already today! Please feel free to contact me (r.roelofsen at if you want to get more information! We also have a online evaluation version running if you want to give it a try!

Tuesday, June 9, 2009

Zen of OSGi

Even though my favorite languages are Scala and Java (yes, Scala comes first ;-) ), I used to use the Python programming language quite a lot. Python has some very well thought out design elements and programming patterns. The Python community actively communicates these patterns and continues to promote their use. As a result of this, the Zen of Python ( was created to capture them in a jokey, but still useful way. There is even an easter egg in the Python interpreter (hint: "import this").

While beeing a platform technology in its core, the OSGi technology seems to use and promote certain patterns as well. The Whiteboard and Extender pattern are well-known examples for this. So why shouldn't we put these patterns in a more jokey and useful way?

Zen of OSGi
  1. Graphs are better than trees.
  2. Prefer services over objects.
  3. Import is better than required.
  4. Although required is better than classpath.
  5. Only export stuff you want to provide support for.
  6. Once you support something, support it forever.
  7. Import your exports.
  8. Minimise your dependencies.
  9. But do not omit imports you really use.
  10. Do not rely on start ordering.
  11. Listen to your customer, do not make them listen to you.
  12. Communicate your version policy.
  13. Be as permissive as possible but no more.
  14. Use the obvious OSGi way.
  15. Although that way may not be obvious at first unless you're Dutch.
  16. Expect the unexpected (especially when you are dealing with services).
You can even discover some interesting similarities between these two Zens.

Any important rules missing?

Thursday, July 3, 2008

OSGi management tool mToolkit free edition released

ProSyst Software ( announced today that the mToolkit Basic Edition is now available free of charge.

mToolkit extends the Eclipse IDE with the necessary tools for easier adoption of the OSGi technology. It provides the developers with remote management capabilities of OSGi containers (currently Equinox and ProSyst's mBedded Server are supported). From within the Eclipse IDE the developer can execute bundle operations like installing bundles on a remote OSGi container. Additionally, mToolkit enables access to the OSGi service registry in order to validate the bundle's service dependencies and registrations.

Screenshot of the Eclipse plugin in action:

For the integration of ProSyst’s mBedded Server Professional and Equinox Edition, mToolkit supports the creation of framework specific launch configurations to e.g. define security parameters and lazy initialization properties.

Please take a look at the following screencasts for feature demonstration and usage instructions.

mToolkit OSGi Remote Management:

mToolkit mBS Framework Launcher:

The Eclipse plugin and the necessary bundles for the target OSGi container can be downloaded at:

Sunday, March 2, 2008

Creating a Domain Specific Language for OSGi

Probably the most interesting language currently is Scala, "a general purpose programming language designed to express common programming patterns in a concise, elegant, and type-safe way". Scala combines the object-oriented and functional programming paradigms. Besides several very interesting programming constructs, Scala provides a 100% interoperability with Java. Scala code can instantiate Java classes, call methods, etc. Since Java code can also access classes written in Scala, application written in Scala can easily use framework-style libraries where the framework needs to use callbacks provided by the user. In fact, the Scala compiler even generates Java byte code. So at runtime there is no distinction between code written in Java and code written in Scala.

Whereas Scala is probably the most interesting language, the OSGi Technology provides a good candidate for the best framework within the Java system ;-) So what would be more obvious than looking at a combination of them? Of course, since Scala generates Java byte code and the OSGi framework is dedicated to Java this is already possible. Neil Bartlett wrote a nice introduction on how to build an OSGi bundle in Scala.

However, we can do even better! The Scala language syntax provides several mechanisms to create a domain specific language. Lets start with a simple bundle activator written in Scala:

class MyBundleActivator extends BundleActivator {
def start(context: BundleContext) {}
def stop(context: BundleContext) {}

Nothing special here. Now lets assume we have the following definitions of a service interface and implementation:

trait SuperService {
def msg: String

class SuperServiceImpl extends SuperService {
def msg = "My super message!"

To add an instance of the service implementation to the service registry we normally would write:

context.registerService(classOf[SuperService].getName, new SuperServiceImpl,

But with some work and leveraging the syntactic elements of Scala, we can write:

context service classOf[SuperService] add new SuperServiceImpl

This doesn't save a lot of typing but is already much more readable. A nice side effect is, that the compiler ensures that the implementation is a subclass of the interface. So this statement would not compile:

context service classOf[SuperService] add new String("A fake super service")

Using this kind of syntax really pays off if we want to access a service:

context service classOf[SuperService] get {s =>

Again, the compiler ensures that everything is correctly typed. Thanks to the type inference mechanism, we do not need to declare that the type of variable s is SuperService. So we can directly call s.msg and pass it to the println methods. If we would have tried to call a nonexistent method, e.g. msg2, the compiler would have thrown an error. The Java equivalent to the above code is:

ServiceReference ref = context.getServiceReference(SuperService.class);
if (ref != null) {
SuperService s = (SuperService) context.getService(ref);

While the DSL version is much more readable and less error-prone, it also has the advantage of following the RAII design pattern. We do not need to do the null-check and the DSL library ensures to "unget" the service.

So how does this work? First, lets remove some of Scala's syntactic sugar: In Scala, a method that only has one parameter can also be used as an infix operator. So


can be used as

obj method parameter

If we reverse this rule to the DSL version for using a service we get

context.service(classOf[SuperService]).get({s =>
The get method is a higher-order function that passes the reference to the service to a closure. You might find this in future versions of Java (BGGA, CICE, FCM, C3S). More interesting however, is the context.service(...) part of the statement. context is an instance of the BundleContext class that clearly does not have a service(...) method. The trick here is to use an implicit conversion from BundleContext to RichBundleContext. The RichBundleContext has the method service(...) which in turn returns an instance of the class ServiceInformation. This class provides the add and get methods:

class RichBundleContext(context: BundleContext) {
def service[T](s: Class[T]) = new ServiceInformation[T](context, s)

class ServiceInformation[T](...) {
def add(...) = ...
def get(...) = ...

To convert the BundleContext instance to an RichBundleContext we could either use the constructor directly or we could use a factory method:

// constructor
new RichBundleContext(context).service(classOf[SuperService]).get({s =>

// factory method
implicit def bc2rbc(bc: BundleContext): RichBundleContext = {
new RichBundleContext(bc)
bc2rbc(context).service(classOf[SuperService]).get({s => println(s.msg)})

You may have noticed that the bc2rbc method is marked "implicit". So as long as this method is in the local scope (e.g. imported) and "implicit", Scala will automatically apply this method. So context.service(...) automatically becomes bc2rbc(context).service(...). Several conditions must match to make this happen. Please take a look at the Scala documentation for more informations.

There are several use-cases for a custom DSL in OSGi:

// sending events
context event "APP/TOPIC" send new MyEvent(...) // Version 1
context event "APP/TOPIC" ! new MyEvent(...) // Version 2

// receiving events
// Version 1
context event "APP/TOPIC" receive match {
case MyEvent(e) => ...
case _ => ...
// Version 2
context event "APP/TOPIC" ? match {
case MyEvent(e) => ...
case _ => ...

// Wire admin
context wire classOf[MyType] connect new MyReciever

What do you think? Which cases would you like to see covered by a DSL?

Saturday, February 16, 2008

Fun with OSGi

OSGi clearly has its root in the embedded world. You can use it on your mobile phone, network router, home gateways, etc. Recently, the OSGi technology received a lot of attention in the enterprise world.

However, maybe someone should tell EA about OSGi since the author of this blog discover an interesting new use case ;-)