GGBLOG – Just In Time MapReduce with OSGi

A couple of years ago I was thrown into a team lead role where I was responsible for distributing workload across a number of developers. During the first few months I found that as the workload increased so did my issues whilst trying to increase the teams scale. It took me a few more months to figure out that my team leading technique needed to be turned upside down.

Instead of pushing the work out, I found it was much better for me to set up a work queues and have the developers pick up tasks from the queue. With this very minor change in technique I was able to double the team’s size without taking stress leave.

I’ve taken this concept and attempted to implement it on OSGi whilst borrowing heavily from the world famous MapReduce research made available by the ingeniously generous folk over at Google.

If you’d like to get my demo running or want to know what the most popular starting letter is on your favorite web pages, you can always follow the steps below. The steps should take around 5 – 10 minutes to complete.

Pre Requisites

1) Install eclipse 3.4 (http://www.eclipse.org/)

2) Install the Rich Client Platform (http://www.eclipse.org/rap/gettingstarted.php)

Running the Demo

Download the mapreduce.zip example source code from:

https://sourceforge.net/project/showfiles.php?group_id=228168&package_id=303652

Open 2 instances of eclipse and create 2 new workspaces, eg. Node1 and Node2.

In both instances configure RAP as the target platform.

Click window -> preferences -> Plug in development -> target platform -> Select the RAP target platform location which is located at [ECLIPSE_HOME]/ configuration/org.eclipse.rap.target-1.1.1/eclipse

Import the demo code into both instances of eclipse.

Click file -> Import -> Existing Projects into workspace -> Select archive file -> click browse and select the mapreduce.zip file which you downloaded earlier.

In both instances of eclipse, Expand galang.research.rap.hello -> double click plugin.xml then click launch RAP application.

In any eclipse instance click Add URL content to memory.

To add a few more pages, Paste the following URL’s and click Add URL content to memory after each one.

http://cnn.com

http://slashdot.com

http://engadget.com

http://smh.com.au

Once youve added the above urls, click run map reduce. If you see both instances of eclipse showing the map reduce output, You’ve setup the demo as expected.

What’s going on under the covers?

A wise/lazy man once said a picture says a thousand words so below is a sequence diagram of what is going on under the hood. You can also step through the code if diagrams aren’t your thing.

Can you use this algorithm for anything else?

I think there are more uses for this algorithm this than just counting the starting characters of words on web pages. You could use this to run a lot of sql queries in parallel then reduce the output. If you want to do more with this then I’d recommend playing around with MyMapper.java and MyReducer.java.

I’d love to throw this code on a large number of nodes to see how quick I can get it run with gigabytes of data. If you’re a big iron or grid/cloud vendor with a few hundred nodes to spare you are more than welcome to drop me a line :) (glenn.galang@gmail.com).

GGBLOG – Dynamic Distributed Data Clustering on OSGi

If you’ve followed some of my past posts you’ll notice I try to keep my examples small and as simple as they need to be. If I were to describe my programming style its very much back to basics and I’m a huge proponent of simplicity.

I’ve whipped up a small example of how you can distribute data dynamically on OSGi using Hazelcast 1.4. I’ve really only scratched the surface of what you can do with this but thought I’d write up my first experiences with this light weight simple yet powerful framework.

Running the Example.

Download the cluster.zip example source code from:

https://sourceforge.net/project/showfiles.php?group_id=228168&package_id=302726&release_id=647048

Unzip the cluster.zip and start up 2 instances of eclipse 3.4. Open up the node1 and node2 directories in different instances of eclipse.

Click run light weight cluster in both instances of eclipse and Jetty should start up on port 9091 and 9092.

Open up two web browsers and in the first browser paste the following URL which will set cVal in a distributed Map on the first node.

http://localhost:9091/sample?cVal=Glenn

In the second browser paste the following:

http://localhost:9092/sample?check=true

If you see the following you have successfully setup a dynamically distributed data cluster.

Where’s the magic?

If you have a look at SampleServlet.java the magic is on line 27 and 32 where we get the map and set a value. Hazelcast makes it look like we are dealing with an ordinary map but the reality is that it possesses extraordinary distributed capability.

If you have any questions on the demo feel free to drop me a line (glenn.galang@gmail.com) or head on over to hazelcast.com.

GGBLOG – Single Source OSGi Enterprise Architecture Pattern

The following article documents an architectural solution to a problem which I’ve encountered in many enterprise computing environments.

Problem:

There are a wide variety of deployment artifacts which cannot be reused between different deployment environments. Copies of the artifacts are taken and modifications are made to enable compatibility for each different execution environments. Component dependencies on different deployment environments cause the code base to diverge over time and become specific to a particular execution environment. This causes the enterprise code base to grow exponentially for only a linear increase in functionality.

Solution and Consequence:

Utilise OSGi as the execution back plane for service, web and desktop applications. This solution delivers a unified approach to the creation, deployment and management of enterprise bundles. The consequence is that a linear relationship is formed between adding enterprise functionality and the size of the code base that enables it.

Structure:

The diagram below shows the architectural structure of this pattern.

Example Strategy and Code:

Example code is available for download at:

https://sourceforge.net/project/showfiles.php?group_id=228168&package_id=301845

Download OSGiEAPattern.zip

The example strategy and code makes use of:

1) Equinox as the OSGi execution environment,

2) Rich Ajax Platform as the rich web application

3) Rich Client Platform as the rich desktop application,

4) Apache CXF as the Web Service consumer

5) JAX-WS as the service layer

Installing the example code:

Pre requisites:

1) Install eclipse 3.4 (http://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/ganymede/SR1/eclipse-java-ganymede-SR1-win32.zip)

2) Install the Rich Client Platform (http://www.eclipse.org/rap/gettingstarted.php)

Running the Rich Desktop Application

Start eclipse and new workspace.

Click window -> prefererences -> Plug in development -> target platform -> tick group plug-ins by location, Select or ensure that the target platform location is the standard directory and not pointing to the rich ajax platform plugins.

Click File -> import -> Existing projects into workspace:

Click next -> select archive file -> browse and select the OSGiEAPattern.zip which you can download from https://sourceforge.net/project/showfiles.php?group_id=228168&package_id=301845

Deselect galang.research.rap.hello and click finish

Select Run configurations -> select OSGi -> select new –> deselect workspace and target platform -> tick galang.research.esb -> add required bundles, you should now have 13 bundles selected

Click Run then double click plugin.xml within galang.research.rcp.hello -> click overview

click Launch eclipse application and type in your name and click “Call web service”

Running the Rich Web Application

Ensure that you have installed the Rich Ajax Platform.

http://www.eclipse.org/rap/gettingstarted.php

Start eclipse and a new workspace.

Click window -> prefererences -> Plug in development -> target platform -> tick group plug-ins by location, Select or ensure that the target platform location is the pointing to the rich ajax platform plugins which is located at [ECLIPSE_HOME]/ configuration/org.eclipse.rap.target-1.1.1/eclipse

Click File -> import -> Existing projects into workspace:

Click next -> select archive file -> browse and select the workspaces.zip which you can downloaded from

https://sourceforge.net/project/showfiles.php?group_id=228168&package_id=301845

Download OSGiEAPattern.zip

Deselect galang.research.rcp.hello and click finish

Right click galang.research.rap.hello.launch

type in your name click Call web service

If you have any questions or want to work with me to implement a strategy for this pattern based on your java technology stack feel free to drop me a line (glenn.galang@gmail.com).

Related Articles:

OSGi

http://ggalangblog.blogspot.com/2008/11/ggblog-light-weight-reporting-server-on.html

RAP

http://ggalangblog.blogspot.com/2008/11/ggblog-chat-on-rap.html

RCP

http://ggalangblog.blogspot.com/2008/11/ggblog-dynamic-jface-xml-tableviewer.html

GGBLOG – Light weight reporting server on Equinox OSGi

I’ve been building software for quite a few years now and as software programming techniques have advanced, so has the number of layers.

Every once in a while a technology comes along which changes the way we think. Of late I’ve been hearing a lot about how cloud computing will allow us to scale our applications out. I think OSGi can give us similar results at a micro level by removing the traditional boundaries and layers on which our java applications are currently built.

OSGi heavily encourages the breaking up of our applications into bundles or modular components which can be readily plugged into mobile, desktop and server architectures.

I’m forever trying to update my 90’s programming style, so in attempt to get up to speed with OSGi I’ve put together a light weight reporting server based on Equinox OSGi and Jasper Reports 3.1.2.

If you’d like to have a look at code and get the demo running feel free to follow the steps below.

Running the Demo

Download the the LWJSOSGi.zip file from:

https://sourceforge.net/project/showfiles.php?group_id=228168&package_id=300872

Once downloaded, unzip it onto your local pc and open Eclipse 3.4 and select the directory which you unzipped to.

Click the down arrow next to the run button and select Run Light Weight Jasper Server

If you see the following the server is now running.

Copy test.jasper, test.jrxml and test.xml from LWJOSGi.zip file to /temp/ (c:\temp on windows).

Open up a web browser and paste the following url.

http://localhost/reportInput?INPUT_XML=/temp/test.xml&JASPER_FILE=/temp/test.jasper&PARENT_XPATH=/RowCollection/Row&EXPORT_TYPE=pdf

You should get a prompt like the following.

Open with adobe reader and if you see the following pdf, you have the demo successfully running.

Optional Step: if you want to change the report download iReport 3.1.2 and open up /temp/test.jrxml and compile the report back to /temp/test.jasper, no need to restart the server after you recompile the jrxml.

Performance

I ran a quick burn test and right out of the box with no tweaking or tuning, the server managed to generate 56 pdfs/second or 3,358 pdfs/minute on a Core 2 Duo 2.4 ghz with 2 gig ram (Win XP).

If I ever need upgrade/dowgrade the jasper distribution I simply create a new jasper reports bundle and configure the runtime dependencies via Equinox. If this were a traditional webapp I’d have to worry about breaking other modules that live inside WEB-INF/lib or within the shared app server lib directory.

If you have any troubles/questions with the demo or want to know a bit more detail on any specific area feel free to drop me a line (glenn.galang@gmail.com).