Monday, December 22, 2008

GGBLOG – Just In Time MapReduce with OSGi

A couple of years ago I was thrown into a team lead role where I was responsible for distributing workload across a number of developers. During the first few months I found that as the workload increased so did my issues whilst trying to increase the teams scale. It took me a few more months to figure out that my team leading technique needed to be turned upside down.

Instead of pushing the work out, I found it was much better for me to set up a work queues and have the developers pick up tasks from the queue. With this very minor change in technique I was able to double the team’s size without taking stress leave.

I’ve taken this concept and attempted to implement it on OSGi whilst borrowing heavily from the world famous MapReduce research made available by the ingeniously generous folk over at Google.

If you’d like to get my demo running or want to know what the most popular starting letter is on your favorite web pages, you can always follow the steps below. The steps should take around 5 – 10 minutes to complete.

Pre Requisites

1) Install eclipse 3.4 (http://www.eclipse.org/)

2) Install the Rich Client Platform (http://www.eclipse.org/rap/gettingstarted.php)

Running the Demo

Download the mapreduce.zip example source code from:

https://sourceforge.net/project/showfiles.php?group_id=228168&package_id=303652

Open 2 instances of eclipse and create 2 new workspaces, eg. Node1 and Node2.

In both instances configure RAP as the target platform.

Click window -> preferences -> Plug in development -> target platform -> Select the RAP target platform location which is located at [ECLIPSE_HOME]/ configuration/org.eclipse.rap.target-1.1.1/eclipse

Import the demo code into both instances of eclipse.

Click file -> Import -> Existing Projects into workspace -> Select archive file -> click browse and select the mapreduce.zip file which you downloaded earlier.

In both instances of eclipse, Expand galang.research.rap.hello -> double click plugin.xml then click launch RAP application.

In any eclipse instance click Add URL content to memory.

To add a few more pages, Paste the following URL’s and click Add URL content to memory after each one.

http://cnn.com

http://slashdot.com

http://engadget.com

http://smh.com.au


Once youve added the above urls, click run map reduce. If you see both instances of eclipse showing the map reduce output, You’ve setup the demo as expected.

What’s going on under the covers?

A wise/lazy man once said a picture says a thousand words so below is a sequence diagram of what is going on under the hood. You can also step through the code if diagrams aren’t your thing.

Can you use this algorithm for anything else?

I think there are more uses for this algorithm this than just counting the starting characters of words on web pages. You could use this to run a lot of sql queries in parallel then reduce the output. If you want to do more with this then I’d recommend playing around with MyMapper.java and MyReducer.java.

I’d love to throw this code on a large number of nodes to see how quick I can get it run with gigabytes of data. If you’re a big iron or grid/cloud vendor with a few hundred nodes to spare you are more than welcome to drop me a line :) (glenn.galang@gmail.com).

42 comments:

Steve Hanov said...

Hi, I'm just googling for everything to do with sequence diagrams and adding comments to promote my tool www.websequencediagrams.com.

I think it could have let you draw your diagram a lot more quickly.

lluis said...

The diagram shown here is created with Enterprise Architect, a very good UML tool. The language to express UML is already invented, it's called XMI and it's a standard. Happy new year!

3d電影 said...

睇完之後覺得有d頓悟..感謝分享..........................

無尾熊可愛 said...

上床圖片av圖片免費色情圖台灣性網陰蒂淫妹貼圖色情激突成人論壇成人色情網激情成人聊天室情色影音鹹濕女生自慰影片淫慾自慰色情a片成人色情圖片網情人視訊網成人話題成人色情台灣成人淫蕩a圖片情色留言板女生奶頭情色成人全裸美女圖片情色寫真裸體照大奶子台灣色情成人網站一夜正妹美女裸體寫真g罩杯美女sex520情趣娃娃視訊打槍視訊辣妹uthome淫亂走光照av情色網女生高潮成人自拍火辣眉眉太太陰毛美女視訊聊天全裸入鏡火辣av三點全露

志義 said...

thank for share, it is very important . ̄︿ ̄..................................................

季玉 said...

dvd 日本色情片新做愛色情片做愛色情影片免費色情論壇色情網小說尋夢中部聊天尋夢中部聊天室尋夢元聊天是尋夢園大學生尋夢園聊天館尋夢園遊戲尋夢聯盟廁所盜攝免費看援交中壢援交平台援高雄晴色小說無碼日本女優甜心寶貝貼影片區情色大全情色a影片情色人成情色小文學情色小說上看情色小說成人學情色小說圖貼情色小舖情色文張情色文章sex情色文學頁gogogirl視訊美女 網路小說 情色論壇聊天室ut

明秋明秋 said...

Birthdays are good for you. The more you have, the longer you live.............................................

AmiraE_Tao0614 said...

It's great!!..........................................

JesseniaT_Orndorff1021 said...

分享笑話三則~
唐僧徵婚
原來還是個母鷹
俺的愛人不是豬

香君 said...

變天了~~注意身體,別感冒囉! .........................................

CasandraC文行 said...

生活盡可低,志氣當高潔........................................

IrmaOrbison said...

I do like ur article~!!!..................................................

冠慧 said...

很用心的blog~很喜歡~願您一切順心 ....................................................

Avri221lE_Pryce1 said...

三分之一的人生,可以決定另外三分之二的人生!!共勉哦!.............................................

韋于倫成 said...

當一個人內心能容納兩樣相互衝突的東西,這個人便開始變得有價值了。.........................

MargaritoJ淑珍_Arent0 said...

you two make a lovely couple............................................................

josema said...

感覺很好的blog,祝你開心喔........................................

均琇均琇 said...

好的開始並不代表會成功,壞的開始並不代表是失敗......................................................................

明文吳 said...

看看blog放鬆一下,工作累死了....<.................................................................

貢慧 said...

一棵樹除非在春天開了花,否則難望在秋天結果。....................................................................

恩如 said...

喜歡你的部落格,留言請您繼續加油.................................................................

皇銘 said...

成熟,就是有能力適應生活中的模糊。.................................................................

啟佐啟佐 said...

如果成為一支火柴,也要點亮一個短暫的宇宙;如果是一隻烏鴉,也要叫疼閉塞的耳膜。.................................................................

丞虹 said...

友誼能增進快樂,減少痛苦............................................................

PearleY建佑 said...

人生匆匆-把握當下,支持鼓勵~事事如意~..................................................................

宜欣宜欣 said...

憤怒,是片刻的瘋狂。..................................................

幸雨幸雨 said...

花花公子的責任就是要很平均的愛他認識的每一個女孩子~~我們的責任則在於公平的回應每一篇blog............................................................

郁如郁如 said...

憤怒,是片刻的瘋狂。..................................................

明儒 said...

從來愛都不知它的深度,非得等到別離的時候.................................................................

潘凱花潘凱花 said...

人應該做自己認為對的事,而不是一味跟著群眾的建議走。..................................................

曾法幸 said...

Fancy may kill or cure...................................................

RicoLisi0802志竹 said...

很用心的部落格 祝你人氣百分百 期待您的新文章...........................................................

黃子軒 said...

It is never too late to learn.......................................................................

凱v胡倫 said...

文章不求沽名釣譽,率性就是真的..................................................................

家唐銘 said...

It is easier to get than to keep it.......................................................................

偉曹琬 said...

Judge not of men and things at first sight................................................

麗王王珠 said...

獨居時,要反省自己的過錯;在社會大眾之間,則要忘卻別人的過失。..................................................

慧佳蓉佳蓉桓 said...

來看看你囉~blog很棒! ............................................................

士文婷文婷松 said...

來看看你逛逛blog囉,加油!..................................................................

孫邦柔 said...

在莫非定律中有項笨蛋定律:「一個組織中的笨蛋,恆大於等於三分之二。」..................................................

孫邦柔 said...

朋友是一面鏡子......................................................................

孫邦柔 said...

一個人的價值,應該看他貢獻了什麼,而不是他取得了什麼............................................................