Jong Choi CGL Blog

Too frequent update of Symantec Endpoint Protection in Mac

2011-01-16T22:27:00.003-05:00

I have installed Symantec Endpoint Protection (SEP) for Mac in an unmanaged mode. The problem is its LiveUpdate runs too frequently and I can't disable it or modify. Although it provides Symantec Scheduler UI, it doesn't look like working at the beginning.

It turns out SEP installs a default schedule owned by a root user, which a normal user can't see in his/her Symantec Scheduler UI. A solution is as follows:

1. Open a terminal

2. Type sudo symsched -d all

3. Setup a new schedule by using Symantec Scheduler UI or a command like,

symsched LiveUpdate "Update All Daily" 1 1 -daily 13:00 "All Products" -quiet

Reference:

http://www.symantec.com/connect/forums/sep-mac-live-update-bouncing-dock

http://www.symantec.com/business/support/index?page=content&id=TECH134203&locale=en_US

http://www.symantec.com/business/support/index?page=content&id=TECH105502

Time measurement methods

2011-01-09T10:09:00.004-05:00

I've just recently tried to decide what is the best way to measure times. The followings are a short summary I got after browsing a few sources in the Internet (See references).

1. gettimeofday

-. Resolution : 1 microsecond

-. Have a negative time issue due to clock reset or NTP

2. clock_gettime(CLOCK_MONOTONIC)

-. Resolution : 1 nanosecond

-. Use HPET, an external chipset on board.

-. An optimal choice, considering both reliability and portability

3. RDTSC

-. Resolution : 1 nanosecond

-. Use RDTSC, a cpu instruction

-. Very light-weight but lots of issues on reliability

-. For a short run and if it is clear on the reliability issues (see Ref 2.), it's good to use.

References:
1. http://aufather.wordpress.com/2010/09/08/high-performance-time-measuremen-in-linux/
2. http://juliusdavies.ca/posix_clocks/clock_realtime_linux_faq.html

3. http://www.kerrywong.com/2009/05/28/timing-methods-in-c-under-linux/

HDF5 MPIIO's Hyperslab_by_chunk.c error in Mac

2010-07-07T11:44:00.004-04:00

In Mac, I got a runtime error after compiling hyperslab's chunk example, Hyperslab_by_chunk.c

As mentioned in here, the error is Mac-specific but the solution what I found is a little different.

Use

hdf5_cv_mpi_complex_derived_datatype_works='no',

instead of

hdf5_mpi_complex_derived_datatype_works='no'

I have tested with HDF5 version 1.8.5.

A few useful Windows HPC commands

2009-11-02T12:46:00.007-05:00

0. Set Default scheduler

set CCP_SCHEDULER=headnode_name

1. clusrun

One can run the following command from his desktop to setup compute node environments

clusrun /user:domain\username dir

clusrun /user:domain\username xcopy \\headnode_name\dir .

clusrun /user:domain\username setx PATH "somepath"

2. job

job list /scheduler:headnode_name

3. node

node list /scheduler:headnode_name

5. cluscfg

cluscfg setcreds

More details are available at http://technet.microsoft.com/en-us/library/cc972841(WS.10).aspx

Imporve serialize() in R for windows

2009-11-01T16:02:00.003-05:00

serialize() function of R is very slow in Windows. My observation is that calling realloc is the bottle neck. The workaround can be replacing realloc with Rm_realloc in $R_HOME/src/main/serialize.c. More specifically, I have updated as follows:

1. In function, resize_buffer(...), replace

mb->buf = realloc(mb->buf, newsize);

with

mb->buf = Rm_realloc(mb->buf, newsize);

2. In function, free_mem_buffer(...), replace

free(buf);

with

Rm_free(buf);

Directions for compiling R in Windows are in http://cran.r-project.org/bin/windows/base/rw-FAQ.html#How-can-I-compile-R-from-source_003f

The following is a short result running on my Windows 7 desktop:

The original:

> system.time(serialize(matrix(0, 1000, 1000), NULL))

user system elapsed

5.74 4.39 10.15

> system.time(serialize(matrix(0, 2000, 2000), NULL))

user system elapsed

85.40 74.80 161.62

After updating:

> system.time(serialize(matrix(0, 1000, 1000), NULL))

user system elapsed

0.78 0.30 1.10

> system.time(serialize(matrix(0, 2000, 2000), NULL))

user system elapsed

6.21 4.13 10.54

Compile Rmpi with Windows MPI (HPC pack)

2009-10-31T21:47:00.007-04:00

Compile Rmpi (http://www.stats.uwo.ca/faculty/yu/Rmpi/) in Windows

1. Install HPC Pack 2008 SDK with SP1 and modify line 314 in mpi.h (installed_dir\Include) as follows:

//typedef __int64 MPI_Offset;

typedef long long MPI_Offset;

2. Download Rmpi from http://www.stats.uwo.ca/faculty/yu/Rmpi/download/linux/Rmpi_0.5-7.tar.gz

3. Untar the source and modify src/Makevars.win by changing directory and library option (-lmsmpi) as follows:

PKG_CFLAGS = -I"C:\Program Files\Microsoft HPC Pack 2008 SDK\Include" -DMPI2 -DWin32

PKG_LIBS = -L"C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\i386" -lmsmpi

4. Compile and install

R CMD INSTALL Rmpi

5. Build for re-distribution

R CMD build --binary Rmpi

Setting up a non-admin SVN repository shared with multiple users

2009-10-28T14:36:00.004-04:00

A trick is using SSH with public key as described in

http://svn.collab.net/repos/svn/trunk/notes/ssh-tricks

As a quick summary, I have set up a SVN repository as follows:

1. create a svn root directory, which will share with others, and create a repository:

$ mkdir /path/svnshare

$ svnadmin create /path/svnshare/project

2. Get public keys of user to share svn repository and insert to ~/.ssh/authorized_keys the following line:

command="/path/to/svnserve -t -r /path/svnshare --tunnel-user=[USERID]",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty [KEY_TYPE] [PUB_KEY] [COMMENT]

[.] should be replaced with users' information

3. Now users can access the shared svn repository remotely as follows:

svn co svn+ssh://[MYID]@[SERVER IP or DNS]/project

Note that use relative path names of repository after server ip or dns

More tips for using R with GotoBLAS

2009-06-29T01:56:00.001-04:00

After building R with GotoBLAS (See http://jychoi-report-cgl.blogspot.com/2009/04/compile-r-with-gotoblas.html), a few things we can do for verification.

1. Download a R benchmark script (http://r.research.att.com/benchmarks/R-benchmark-25.R) and run it to check if every step works ok. (You can compare the performance with ones from normal R build too.)

2. If R is hang in calling eigen() function, try to rebuild GotoBLAS and R by using the same fortran compiler.

3. If nothing works, one can use ATLAS instead of GotoBLAS

Compile R with GotoBLAS

2009-04-24T08:31:00.001-04:00

I want to share my experience to use GotoBLAS as an external multithread library of R. I didn’t make through performance test with other libraries (such as ATLAS) but I did got lot of performance gains with GotoBLAS in using R.

1. GotoBLAS from http://www.tacc.utexas.edu/resources/software
Follow instructions in 02QuickInstall.txt.

2. CBLAS from http://www.netlib.org/blas/blast-forum/cblas.tgz
This is not required for using R but you may need this for using GSL(GNU Scientific Library)
a. If the architecture is Linux, type
$ ln –s Make.LINUX Make.in
b. In Make.in, Modify BLLIB, CBDIR and –fPIC –lpthread to LOADER option.
c. Type make all for building libraries and testing
d. After completing, go to lib/LINUX and type
$ ld -melf_x86_64 -shared -soname libgotocblas.so -o libgotocblas.so cblas_LINUX.a

Note. try to use –m64 or –m32 if you are working with powerpc

3. R from http://cran.r-project.org/
Run configure as follows:
$ export GOTOBLAS_LIB=PATH/TO/GOTOBLAS_LIB
$ mkdir build_goto; cd build_goto;
$ ../configure --prefix=$HOME/usr/R/ --with-blas="-L$GOTOBLAS_LIB -lgotoblas -lpthread" --enable-R-shlib --enable-R-static-lib --enable-BLAS-shlib

Create AVI from R plot

2009-04-07T11:09:00.001-04:00

1. In R, save plots as postscript (or png) files by adding sequence number. For example,

postscript(sprintf("%04d.eps", i))

2. By using ImageMagick’s convert command line tool, convert images from eps to png (You may skip if you have png files)

for f in *.eps; do convert -rotate 90 $f png32:$f.png; done

You may want to add the following options:

-resize 1280x720 : change image size
-bordercolor white -border 0x0 : add white background

3. Create AVI by using ffmpeg

ffmpeg -r 15 -sameq -i %04d.eps.png out.avi

You can control frame rate (-r rate) and quality (-sameq or –b bitrate)

Note:

If you need speed-up in conversion, you may try to use a bash script parallel.sh in http://pebblesinthesand.wordpress.com/2008/05/22/a-srcipt-for-running-processes-in-parallel-in-bash/

Summary of the recommendation system survey paper

2009-02-01T21:52:00.001-05:00

I’ve found a good survey paper about recommendation systems as follows:

Gediminas Adomavicius, Alexander Tuzhilin, "Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions," IEEE Transactions on Knowledge and Data Engineering, vol. 17, no. 6, pp. 734-749, June, 2005.

A short summary:

-. A short definition of recommendation system: a problem of extrapolation for predicting unknown values

-. 3 approaches: i) Content-based, ii) Collaborative, and iii) Hybrid

i) Content-based RS(Recommendation System): a user’s feature is computed solely based on the user’s activity history. Can have the following limitations:
a. Feature extraction can be hard in some domain, such as multimedia or image
b. Over specialization: Diversity is required. Randomness, genetic algorithms, or some adjustment (remove too similar, or too different outputs)
c. New user problem: No information to consider

Known algorithms: (Naive) Bayesian classifier, Rocchio, winnow, ANN, …

ii) Collaborative RS: a user’s feature is computed by a group of like-mined people or peers. Limitations:
a. New user problem [83][89] : The same with content-based RS
b. New item problem
c. Sparsity: a few workaround ideas -- use of demographic information, dimension reduction, …

Known algorithms: clustering, Bayesian network, SVD, maximum entropy, …

iii) Hybrid RS: utilize both content-based and collaborative system.

Flash 3D Engine: Sandy3D Vs. Papervision3D

2008-12-21T17:26:00.001-05:00

Inspired by an article compared performance between Away3D vs. Papervision3D, I’ve just wanted to compare simple performance Sandy 3D (3.1 AS3) vs. Papervision 3D (2.0. Revision 849. Code name: Greate White) in rendering 1,000 objects.

As a result, the papervision3D is faster than Sandy3D by roughly 2~3 times or even more. My benchmark implementations of both Sandy and Papervision3D and source codes are available.

Optimization advice of Sandy can be found here.

How to Parallelize

2008-12-21T16:46:00.001-05:00

I’ve found a very good introduction about how to parallelize applications from http://www.cs.princeton.edu/courses/archive/spr08/cos598A/parallelization_course.pdf

Especially, a few tools are very helpful to analyze the code at the beginning stage:

1. gprof – profiler. Help to decide which part should look at. Consider to use a pthread wrapper available at http://sam.zoy.org/writings/programming/gprof.html

2. helgrind – To detect race conditions

GCC 4.3.2 Compilation and OpenMP

2008-12-09T01:35:00.001-05:00

In order to try OpenMP3.0, I had been trying to install gcc-4.3.2 on an x86_64 Linux box without any luck. I got the following error in building gcc-4.3.2

/usr/bin/ld: crti.o: No such file: No such file or directory

It turns out that I need 32bit libc for cross-compilation which I can’t install since I’m not a superuser. The workaround is to disable this feature by using “--disable-multilib” option as follow:

./configure –-disable-multilib
make

After installation, if you meet the following error in compiling OpenMP file:

gcc: libgomp.spec: No such file or directory,

find libgomp.spec under /path/to/gcc/lib64 and make a symbolic link under /path/to/gcc/lib/gcc/x86_64-unknown-linux-gnu/4.3.2 (See [2])

Reference:
[1] gcc-help mailing list
[2] OpenMP in 30 Minutes

Collective Collaborative Tagging System (Abstract)

2008-11-04T11:04:00.001-05:00

Currently in the Internet many collaborative tagging sites exist, but there is the need for a service to integrate the data from the multiple sites to form a large and unified set of collaborative data from which users can have more accurate and richer information than from a single site. In our paper, we have proposed a collective collaborative tagging (CCT) service architecture in which both service providers and individual users can merge folksonomy data (in the form of keyword tags) stored in different sources to build a larger, unified repository. We have also examined a range of algorithms that can be applied to different problems in folksonomy analysis and information discovery. These algorithms address several common problems for online systems: searching, getting recommendations, finding communities of similar users, and finding interesting new information by trends. Our contributions are to a) systematically examine the available public algorithms’ application to tag-based folksonomies, and b) to propose a service architecture that can provide these algorithms as online capabilities.

cross-domain problem in making ajax call by using XMLHttpRequest

2008-10-27T15:05:00.001-04:00

I’ve just faced with a cross-domain problem in making a mash-up site by using ajax call. In particular, I’m developing an igoogle gadget which should use cross-domain calls to retrieve data from a server outside of google domain.

I’ve found a good article to workaround this problem: http://developer.yahoo.com/javascript/howto-proxy.html

Among the solutions suggested in the article, I’m very pleased with JSONP approaches. Intuition of JSONP approach is that cross-domain problems will not occur for <script> tag. By exploiting this, call a cross-domain function, which designed to return json data enclosed in a function call, by using "src” attribute of <script> tag. Details can be found at http://bob.pythonmac.org/archives/2005/12/05/remote-json-jsonp/

Summary of “Evaluation Collaborative Filtering Recommender Systems”

2008-07-30T13:36:00.001-04:00

The following is a short summary of:
J. Herlocker, J. Konstan, L. Terveen, and J. Riedl. Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems (TOIS), 22(1):5--53, 2004.

Commonly observed user tasks from recommender systems are as follow:
-. Annotation in context : Uses a recommender in an existing context. E.g., Overlay prediction information on top of existing links
-. Find good items : Video recommender, Item recommender, etc
-. Find All Good Items : purpose is to lower false negative rate
-. Recommend sequence : can be interesting
-. Find credible recommender : making the system more credible

Ratings can be explicit (users rate directly) or implicit (inferred from users behavior or preferences)

Commonly used evaluation metrics are:
1. Predictive accuracy
-. MAE (Mean Absolute Error) = (\sum (p_i – r_i))/N
-. MSE (Mean Square Error) = (\sum (p_i – r_i)^2)/N
-. RMSE (Root Mean Square Error) = \sqrt (MSE)
2. Classification accuracy
-. Precision : the ratio of relevant items selected to number of items selected
-. Recall : the ratio of relevant items selected to total number of relevant items available (Note: Precision and recall are inversely related.)
-. MAP(Mean Average Precision, aka F1) = 2PR/(P+R)

Netflix Prize for the best collaborative filtering algorithm

2008-07-16T01:52:00.001-04:00

Collaborative filtering, also known as social tagging, is much more popular these days in the Internet but it’s not so clear yet how much and what kind of information we can extract from the system. Regarding this problem, I think Netflix prize would be a great challenge to make those questions clear out.

@. What to predict?

Simple. From the training set, provided by Netflix, which contains over 100M movie 1-to-5 scale ratings, we need to predict unknown movie ratings for the given qualifying set. More specifically, each data in the training set is quadruple of <user, movie, date of grade, grade> and the qualifying set is given <user, movie, date of grade, unknown grade>. We need to fill out the unknown grades by prediction and submit them for evaluation.

Besides two data sets, the training and qualifying set, Netflix provides the probe set which is a problem set with answers. With this, we can roughly estimate the accuracy without consulting with the scoring oracle.

@. How to predict?

Hard. However, we can learn from the front-runners, posted at the Leaderboard. The first annual progress winner is BellKor and they wrote about their algorithm.

@. Number-wise story

#. of data in the training set = 100,480,507
#. of users in the training set = 480,189
#. of movies in the training set = 17,770
#. of data to predict in the qualifying set = 2,817,131
#. of data in the probe set = 1,408,395

@. Research problems

As a student, trying to apply machine learning algorithms to various applications, it is interesting to study:

What kind of machine learning algorithms can be applied to analyze the Netflix collaborative filtering data? Furthermore, Can we find more general algorithms can be used for the data in the net?
How such computations can be expedited by using parallel, multi-core platform? Interestingly, is it possible to use other computing powers, such as cloud computing?
Can we improve accuracy by adding other information easily accessible from the Internet? What kind of infrastructure of the Internet can help this? Web2.0 or else?

@. Reading list

J. Herlocker, J. Konstan, L. Terveen, and J. Riedl. Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems (TOIS), 22(1):5--53, 2004.

R. Bell, Y. Koren, and C. Volinsky. Modeling relationships at multiple scales to improve accuracy of large recommender systems. Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 95--104, 2007

R. Bell and Y. Koren. Improved Neighborhood-based Collaborative Filtering. KDD-Cup and Workshop.

R. M. Bell and Y. Koren, Scalable Collaborative Filtering with Jointly Derived Neighborhood Interpolation Weights, Proc. IEEE International Conference on Data Mining (ICDM'07), 2007

Possible useful papers can be found from the Internet. I will keep this list up-to-date, as I read through.

PMPP workshop at NCSA

2008-07-15T23:46:00.001-04:00

I attended PMPP (Programming Massively Parallel Processors Agenda) workshop at NCSA last week (July 10th, 2008). It was great opportunity to see what we can do with parallel processors and modern powerful GPUs. Recently Multi-core or many-core has been drawing great attentions in various research communities and people are still struggling to find a way to maximize its capabilities in the coming 80-core era. In other side, Peoples have been also trying to utilize GPUs as a parallel computing unit. Modern GPUs are equipped with tens of, or hundreds of cores, which can be used for computing intensive jobs. The workshop was about all of these: Multicores and GPUs.

Here are some highlights:

Parallel GPU: Mostly two venders, nVidia and ATI, are actively developing this market by continually supplying powerful hardware and SDKs. nVidia provides CUDA and ATI does Stream SDK for programming kits.
nVidia's parallel GPU with CUDA: Lightweight threads running on nVidia GPU (8 to 240 cores). Designed with parallel execution in mind, while general CPU is not. GPU can be considered as massively parallel manycore machine.
Autotuning for multicore : Multicore tuning parameter spaces are so huge to investigate one by one. We can overcome this by systematic approaches. Details can be seen from the papers by Samuel Williams and Kaushik Datta. In their paper, many multicore tuning techniques are introduced.
Cell broadband engine (Cell BE): Interesting demonstration was shown by Hema Reddy from IBM. Look at this article and clip. I saw the more interesting clip in the workshop but I can't find the exact one in the Internet.
Multicore/GPU researchers : look at Pedro Trancoso, John Stone, and

MDS graph of Connotea

2008-04-23T00:36:00.001-04:00

MDS graph with Connotea data by using GGobi.

Status report for 02/07/2008 to 02/13/2008

2008-02-13T17:02:00.001-05:00

I have been doing tag analysis to find underlying relationships between tags, users, and resources in folksonomies. I think there are possibly two different approaches: i) frequency analysis based on resource vectors spanning over the term space and ii) graph-based analysis based on tag graphs.

In the frequency analysis, we can use Principal Component Analysis(PCA) or Independent Component Analysis(ICA). Similar approaches have been done in the field of IR. An example can be found in here, where PCA and ICA were used for tag-advertising matching.

With PCA and ICA, I produced the following figures:

(a) PCA with Tag-graph

(b) ICA with Tag-graph (number of component = 3)

It is very interesting that we can see some tag relationships (or tag clusters) in both two pictures but reasonable interpretation is not so easy to get. I think it's a good starting point to cluster terms for extract meaningful information. I will keep working on those graph.

As for the next step, I'm going to do analysis with graph-based methods.

Status report for 12/13/2007 to 12/19/2007

2007-12-19T14:56:00.001-05:00

1. Listing up papers for Oral Qual.

I've been making a reading list for oral qualify exam. The focus of my oral exam will be (1) Folksonomies and Web 2.0 architecture, (2) Machine Learning, and (3) Multicore and parallel programming of machine learning algorithms.

2. Paper reading - HMM

I've been reading a paper about Hidden Markov Models:

L. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition," Proc IEEE, vol. 77, pp. 257-286, 1989.

A short summary is as follow:

-. HMM: A Markov chain with hidden states. HMM can be used to guess a transitions between hidden states based on the observation

-. The basic 3 problems in HMM: (a) Given a HMM model, compute probability of an observation sequence, (b) Given the observation sequence, guess a probable state sequence, and (c) Adjust HMM model parameters to maximize P(Observations|HMM Model)

-. Viterbi Algorithm can be used to solve the problem of type (b)

-. Expecation-Maximization algorithm can be applied to (c)

Status report for 11/08/2007 to 11/14/2007

2007-11-14T17:18:00.001-05:00

I've been reading a paper about ranking in folksonomy, called FolkRank, proposed in Trend detection in folksonomies, Information Retrieval in Folksonomies: Search and Ranking.

FolkRank is based on well-known PageRank algorithm used in Google (The Anatomy of a Large-Scale Hypertextual Web Search Engine and The PageRank Citation Ranking: Bringing Order to the Web).

In a nut shell, the basic idea of PageRank is that a page is important if there are many pages linking to it. This can be formulated by the following matrix equation:

R(u) = c \sum R(v)/N or R(u) = c \sum R(v)/N + cE(u)

where

u is a web page and v is a page referring u,

R(u) and R(v) are PageRank of u and v repectively,

N is a number of all links in page u, and

E(u) is called a source of rank.

Folksonomy can also be represented as a graph by using the similar concept of PageRank but a few exceptions:

Folksonomy graph is undirected, while web graph in PageRank is directed.
Nodes are heterogeneous and the graph is triadic, while in PageRank the nodes are homogeneous.

I think it's a good start to begin with applying PageRank algorithm to folksonomy recommendation system. For this purpose, I'm planing to dig into more details on the PageRank algorithm and find some sample code for better understand.

Status report for 11/01/2007 to 11/07/2007

2007-11-08T09:42:00.001-05:00

1. I made a reading list about folksonomy at my CGL blog or del.icio.us. I will update the list frequently by adding related articles.

2. Characteristics of Folksonomy Network

The folksonomy networks show the following characteristics:

a. Small-world Network

Almost similar with a random network but having much larger clustering co-efficient factor. I.e., Small shortest path but a larger clustering coefficient.

b. Scale-free Network

A scale-free network is a network in which any two nodes can be connected no matter what the system size is. This is because there is a node called "hub"which is a highly connected node than any others. In folksonomies, popular keywords can act like this hub and thus the network shows scale-free network properties.

A scale-free network shows the following rules:

(1) Power law : Degree distribution follows the Yule-Simon distribution, which is called a power law: P(k) ~ k^(-r), where k is a degree of connectivity of a node and P(k) is it's probability

(2) Preferential Attachment : A way to build a scale-free network. The idea is to make a connection with a more connected node.

3. Next step

a. Build a simple graph by using CITEAM tag data

b. Find some algorithms about recommendation

Reading list about folksonomy analysis

2007-11-04T12:31:00.001-05:00

(Also available at http://del.icio.us/yyalli/folksonomy)

Formal Model, Graph

The Complex Dynamics of Collaborative Tagging, HT 07
Network Properties of Folksonomies, WWW 07
Mining Association Rules in Folksonomies, Data Science and Classification
Ontologies are us: A unified model of social networks and semantics, Web Semant. 5, 1 (Mar. 2007)
Folksonomy as a Complex Network
Trend detection in folksonomies, LNCS, 2006
A Social Networking Model of a Web Community,

Semantic Web

Bridging the Gap Between Folksonomies and the Semantic Web: An Experience Report, ESWC 07

Pattern, Structure

Recommendation, Ranking, Trend

Tag Generation

P-TAG: large scale automatic generation of personalized annotation tags for the web, WWW 07

PageRank

The Anatomy of a Large-Scale Hypertextual Web Search Engine, Sergey Brin and Lawrence Page, 1998
The PageRank Citation Ranking: Bringing Order to the Web, 1998

Clustering (referred from Filippo Menczer's class material)

Web Page Recommender System based on Folksonomy Mining for ITNG ’06 Submissions, ITNG '06, 2006
Finding related pages in the World Wide Web, Computer Networks, 1999
Self-Organization and Identification of Web Communities, IEEE Computer, 2002
Finding replicated Web collections, ACM SIGMOD '00, 2000
Community structure in social and biological networks, PNAS, 2002
Defining and identifying communities in networks, PNAS, 2004

Learning and Classification (referred from Filippo Menczer's class material)

Text Categorization with Support Vector Machines: Learning with Many Relevant Features, 1997
Support Vector Machines, 1998
A Tutorial on Support Vector Machines for Pattern Recognition, 1998
A re-examination of text categorization methods, ACM SIGIR, 1999

Status report for 10/04/2007 to 10/10/2007

2007-10-10T11:38:00.001-04:00

I have been searching articles about folksonomies in the Internet. Among them the followings are useful to understand:

Folksonomies - Cooperative Classification and Communication Through Shared Metadata : Pros and cons about folksonomies. Future research areas.
Ontology of Folksonomy : Roles of ontologies and folksonomies.

Some researches about folksonomy applications by using machine learning algorithms.

Visualizing Folksonomies using Machine Learning Algorithms
Learning from a Visual Folksonomy: Automatically Annotating Images from Flickr
Optimizing web search using social annotations

I have also started reading an introductory machine learning book, "Introduction to machine learning" by Ethem Alpaydin. During the next week, I will search and read more papers about folksonomies and machine learning.

Status Report for 06/07/07 to 06/13/07

2007-06-14T22:31:00.000-04:00

Issues in using NWS
1. Hostnames
NWS’s memory service, which provides memory service to store NWS sensors’ data, saves data based on NWS sensors’ hostnames. (A hostname is a system name returned by gethostname or ‘hostname -–fqdn’) This can be problem when a master and sensors installed in different clusters and a sensor’s hostname is not a public domain name, i.e., internal hostname. For example, BigRed's hostname is not for publicly accessible from the Internent, rather hostnames are for only internal uses. In this case, NWS’s information extractor, nws_extract , can not show correct data to a user. If the memory service is in the same cluster with sensors, it will be ok but no other sensors outside clusters will be allowed.

2. Manager nodes and computing nodes
NWS’s bandwidth measurement may be incorrect depending on cluster’s topology. For instance, NCSA’s TeraGrid cluster consists of four manager nodes, which have all public hostnames and thus can be accessible from the Internet, and hundreds of computing nodes, whose hostname is not known unless submitting a job. To measure bandwidth before knowing which node will run a job can be difficult. To overcome this possible problem, knowledge on a cluster’s topology or managing policy will be required, if possible, by contacting a help desk.

Status Report for 05/31/07 to 06/06/07

2007-06-07T02:34:00.000-04:00

NWS installation in TeraGrid

I've installed a NWS client in a few TeraGrid clusters and in my local machine to measure network bandwidth information between them. To sum up, we need to run the followings for a master node and a client node

a. Mater node
nws_nameserver
nws_memory
nws_sensor

b. Client node
nws_sensor

To start measuring, we need to submit the following command:

start_activity -f file_name name_server

To extract measured values, type the following:

nws_extract -N name_server -f time,measurement band from_node to_node

Here is some outputs for network bandwidth:

## Units: Megabits/second
## E.g.,
## 1181197165 (<= This is time) 6.148710000(<= This is bandwidth, Megabits/second)
## Network bandwidth between a.ufo.edu and b.ufo.edu
$ nws_extract -N nameserver.ufo.edu -f time,measurement,source,destination band a.ufo.edu b.ufo.edu
Time Measure Source Destination
1181197165 6.148710000 a.ufo.edu b.ufo.edu
1181197285 6.142370000
1181197405 6.135040000

Status Report for 05/24/07 to 05/30/07

2007-05-31T16:54:00.000-04:00

1. STFILTER and Analyze_Tseri service
I’ve finished updating STFILTER and Analyze_Tseri service to process multiple GPS stations at the same time. Now multiple stations data is shown in one graph. The overhead, I guess, may be linearly increased as the number of stations is increased. However, for time being, it looks fine a few stations. More details will be measured later. In addition, there are a few minor bugs, which will be fixed soon.

Status Report for 05/17/07 to 05/23/07

2007-05-23T16:32:00.000-04:00

1. STFILTER portlet and Analyze_Tseri service
I’ve been updating STFILTER and Analyze_Tseri web service to make the system deal with multiple sites (Previously only single site can be supported). The implementation is almost done except displaying multiple graphs by using BFO library. Since the output from analyze_tseri application is a union of multiple outputs of single execution with a single site, the graph should be multiplied or multiple graphs should be displayed in one graph by merging them.

2. Network Bandwidth Information
Yili preferred to install NWS server/client in the nodes of TeraGrid. So, I’m making a list of nodes to install NWS client and server. There might be restrictions using a special port or network traffics. I will keep posted when I have problem.

Status Report for 05/10/07 to 05/16/07

2007-05-17T13:56:00.000-04:00

1. STFILTER portlet and Analyze_Tseri service
During the last week, I’ve been updating STFILTER portlet and Analyze_Tseri to make the system to deal with multiple sites. To make sure correct operation, I started with making a test case to run standalone tseri_analysis program. Now, I’m updating STFILTER portlet and Analyze_Tseri webservice and will complete in the early of next week.

2. Network Bandwidth Information
I’ve been looking at again NWS (Network Weather Service) to find out how to get network bandwidth information in Teragrid from NWS but it seems like there is no publicly available method to share. Although we can install NWS server/clients for each node for testing purpose, it will suffer from low scalability.

Status Report for 05/03/07 to 05/09/07

2007-05-09T16:34:00.000-04:00

STFILTER portlet and Analyze_Tseri service :
During the last week, I’ve been updating STFILTER portlet by using Db4o to support project management feature. Now STFILTER portlet can save query parameters as a project and reload later. I’ve also updated Analyze_Tseri service by changing interface parameters to receive GRWS URL as an input. Thus, STFILTER portlet doesn’t need to send data as String type. Instead, only GRWS URL is enough. Analyze_Tseri will download by connecting to GRWS service. In the future, Analyze_Tseri will be updated to query GRWS directly with given parameters. The next thing to do is to provide support for processing data from multiple stations.

Status Report for 04/26/07 to 05/02/07

2007-05-02T16:17:00.000-04:00

I’m updating STFILTER to work with the object database, Db4o and almost ready to release. I’m planning to demonstrate STFILTER portlet to Marlon in a few days. A few minor issues needs to be considered are as follows (including previous ones):

1. Problems in implementing custom serializer/de-serializer of analyze_tseri service. Making webservice exchange user-defined complex type has some errors.
2. Lock/Un-lock features. I have lock related errors in using Db4o. Currently, I disabled lock features to prevent such errors. But, I don’t know which is better way for STFILTER portlet.
3. Deleting temporary files. Analyze_tseri web service is now saving user’s input and output under a directory in a way a user can download files directly through http protocol. However, at some point, such files should be deleted to save disk spaces. I didn’t decide yet when and which is the best way to do that.

Status Report for 04/19/07 to 04/25/07

2007-04-25T21:04:00.000-04:00

I have updated STFILTER by adding project management function and input and output file downloading feature. For this purpose, new STFILTERProjectBean class has been added like RDAHMM and an object database, Db4o, is used to save and restore STFILTERProjectBean object. I also cleared out most of dependencies on old context manager class. A few minor issues I'm facing are:

Lock related errors in using Db4o. Currently, I disabled lock features to prevent such errors. But, I don’t know which is better way for STFILTER portlet.
Analyze_tseri web service is now saving user’s input and output under a directory in a way a user can download files directly through http protocol. However, at some point, such files should be deleted to save disk spaces. I didn’t decided yet when and which is the best way to do that.

During the next week, I will do more test to find out any bug and improve some cosmetic things.

Status Report for 04/012/07 to 04/18/07

2007-04-18T22:32:00.000-04:00

1. STFILTER portlet
I’ve updated STFILTER portlet by adding more time series parameter filters. In addition, time series analysis service (analyzeTseri) and STFILTER portlet has be updated with easy installation procedure so that most user dependent system settings can be done in installation step As for the next step, I will add project management features in STFILTER portlet.
2. Network Weather Service
I’m still searching for how to get list of hosts served as Network Weather Service hosts. There is no open information to list up such hosts. I think we need to ask either TeraGrid help desk or Network Weather Service development team

Status Report for 04/05/07 to 04/11/07

2007-04-11T21:24:00.000-04:00

1. STFILTER portlet
I’m still updating both analyze-tseri-exec webservice and STFILTER portlet to work with addition parameters for time series analysis. Although no technical difficulties yet, it takes some time to make them more general as they are sharing same code in many cases. I will finish this update in a few days. After updating, I will modify the project management part of STFILTER portlet.

2. TeraGrid 07 final paper submission
I’ve been updating TeraGrid 07 paper for camera ready, co-worked with Mehmet and Marlon. The final due is April 12th.

3. Network Weather Service
I sent an email to NWS’s developer mailing list (nws-dev@cs.ucsb.edu) to query about any possibility to get network bandwidth service but haven’t get any reply yet. To prepare worst case, I think I need to find other way to get such information.

Status Report for 03/29/07 tto 04/04/07

2007-04-04T22:53:00.000-04:00

1. STFILTER portlet
I'm updating analyze_tseri web service to use a bean as an input parameter. Accordingly STFILTER portlet will be updated soon.
As for the next step, I'm planning to upgrade STFILTER portlet as follow:
a. Adding more options on filter parameter.
b. Using property file, instead of web.xml or portlet.xml
c. Modifying project management
d. Fixing some bugs when clicking "edit" mode button.

2. Network Weather Service
After installing nws package from NWS website, I found a NWS command for extracting inter-machine bandwidth and latency measurements as follow:

$ html-hosts -N nws.cs.ucsb.edu sperm.cs.ucsb.edu beluga.cs.ucsb.edu

Bandwidth values are measured in Megabits/second; latency in milliseconds
Source machine		Destination machine
Source machine		sperm.cs.ucsb.edu	beluga.cs.ucsb.edu
sperm.cs.ucsb.edu	bw forecast error lt forecast error	- - - - - -	86.10412000 86.10059000 1.759016353 0.429000000 0.370000000 0.091088105
beluga.cs.ucsb.edu	bw forecast error lt forecast error	85.42371000 85.52822000 4.072307805 1.648000000 1.702000000 0.320489947	- - - - - -

However, I haven't found yet if I can use this command more generally, for example, in TeraGrid clusters. It seems like if I know a name server of a cluster, it will be possible to get network bandwidth information. But, I don't know how I can get list of such name server. I will dig into that more.

Moab dashboard or memejob portlet configuration

2007-03-29T11:36:00.000-04:00

1. memejob-portlet
a. Copy memejob-portlet.war to $CATALINA_HOME/webapps
b. Modify ./WEB-INF/portlet.xml as follows:
Basically, by using memejob-portlet, memejob can be send to multiple sites as shown in "hostnames" parameters. If you don't need multiple site, only one host name in "hostnames" parameter will be enough. The other values should be set with similar way.
-. hostnames : list of hosts to submit meme job. Comma separated and no space between them.
-. batchhostnames : list of hosts names for batch job submission
-. commandset: list of meme execution file path. The order must be the same with list of hostnames
-. pathset : list of path env. The order must be the same with list of hostnames.
-. gridftpservers : list of ftp servers for each host shown in hostnames.

2. moab-dashboard
a. Copy moab-dashboard.war to $CATALINA_HOME/webapps
b. Modify ./WEB-INF/portlet.xml as follows:
moab-dashboard portlet can support executing moab command in multiple sites. You can specify either list of sites in "hostnames" or one site if you want only support only one site.
-. hostnames: list of host names
-. commandset: list of path of moab command. The order should be same with hostnames.
-. checkjob: list of path of 'checkjob' command.

3. Displaying in OGCE portal

When the deployed portlet is shown as a non-portlet web application in “Adminitration” tab of OGCE, we can change it as a portlet application as follow:

a. In “Administration” tab, type “moab-dashboard”(or memejob-portlet) in “Deploy new portlet webapp” and click “Deploy” button. If successful, you can see “moab-dashboard” in “Portlet web applications” section. You may need to refresh the screen by visiting other tab or re-logging.

b. Go to “Welcome” tab, check “moab-dashboard”(or memejob-portlet) in “Configure group membership” and click “Save” button. You can see a new tab named “MOAB Dashbarod”(or “Meme job”)

Status Report for 03/22/07 to 03/28/07

2007-03-28T21:54:00.000-04:00

1. STFILTER and AnalyzeTseriService
I have made AnalyzeTseriService more self-contained that the service can get environmental parameters as well as array of time series data. Thus, STFILTER application can query more dynamically from the AnalyzeTseriService service. However, the better way is to make the service receive a bean as an input parameter so that a client only needs to send only a bean.

During the last week, I have been busy in doing some class project. I'm planning to upgrade STFILTER portlet and investigate on Network Weather Service.

Status Report for 03/15/07 to 03/21/07

2007-03-21T14:16:00.000-04:00

1. STFILTER portlet
I have developed STFILTER portlet by converting stand alone STFILTER application which is using both JSF and JSTL tag. In the conversion process, I faced a problem for some reason that JSTL tag didn’t work with portlet environment. Especially, JSTL’s EL expression didn’t parse correctly, such as ${bean.param} or ${0}, to evaluate bean property or expressions. To overcome the problem, I put the following code inside JSP scriplet in .jsp file:

FacesContext context = FacesContext.getCurrentInstance();
Application app = context.getApplication();
ValueBinding vbName = app.createValueBinding("#{stfilterBean.graphName}");
String[] strName = (String[]) vbName.getValue(context);

Then, use JSTL tag as follows:

2. Network Weather Service
I'm still looking around to get network bandwidth information from Network Weather Service.

STFILTER portlet configuration

2007-03-21T12:43:00.000-04:00

To make STFILTER portlet work, the following values should be set properly.

Replace Google key according to the site url in the following files: allstations.jsp, map.jsp, and NewProject.jsp which are in the installed directory. For example, replace _new_key_ in
http://maps.google.com/maps?file=api&v=2&key=_new_key_
You can have new Google key from http://www.google.com/apis/maps/signup.html.
In ./WEB-INF/faces-config.xml, values should be set as follow:
-. contextUrl : context manager web service url
-. contextBasePath : context manager base path
work.dir value in ./WEB-INF/web.xml should be replaced. Note that this directory should be accessible by using httpd
Specify SOPAC GPS parameters in ./WEB-INF/classes/sensorgrid.properties.

Status Report for 03/08/07 to 03/14/07

2007-03-12T21:09:00.001-04:00

1. User Interface using JavaScript
I have been trying to improve user interface of st_filter, mostly focusing on using JavaScript and I found that a few useful sites:

Yahoo UI: A few ready-to use components such as calendar and dialog box. A few basic tools for developing UI are also available such as event handling and DOM.
JavaScript Image Cropper UI: If one need to handle images, this tool might be helpful. A very handy script to make an image cropper.
script.aculo.us : You can make lots of effects on objects in HTML.
Prototype: A framework. I was surprised how JavaScript is complicated and can be improved by applying OOP and functional programming language concept. More details on JavaScript as a programming language can be found in here . Especially video clips by Douglas Crockford are my recommendation.

2. st_filter portlet
I’ve started to convert standalone st_filter web application into a portlet. I’ve encountered a few problems but I think it can be solved soon. I will put some summary in the next post.

3. Workflow mapping in Grids
I discussed with Yili about gathering information from Network Weather Service and GPIR to schedule workflow. Although simple information can be obtainable, more complicated and detail information is hard to get. I will look over again beyond documentation and look at the source code, if possible.

Status Report for 03/01/07 to 03/07/07

2007-03-07T14:29:00.000-05:00

1. Time series analysis graph
I have developed drawing graphs by using BHO's graph library for the last week. To make the graph interactively, we need to combine javascript with the BHO's graphic lib.

The code will draw a line graph with the vector data obtained by stfilterBean.getFilteredList(). To patch data iteratively, JSTL's c:foreach statement is used. To use JSTL, we make two jar files, jstl.jar and standard.jar, available in the library path. For better UI interface, the next step will be using Yahoo's UI library (http://developer.yahoo.com/yui/)

2. Workflow mapping in Grids
a. GPIR client (Done last week)
b. Network Weather Service (Done) : I had trouble in generating a client from WSDL of Network Weather Service by running WSDL2JAVA. I figured it out by installing a newest version of axis library. Now it's working correctly. I will discuss with Yili for more details
c. Running time prediction (Not yet) : I will discuss with Yili.

Status Report for 02/22/07 to 02/28/07

2007-02-28T15:24:00.000-05:00

1. Time series analysis graph
To improve interface of the time series analysis portlet, I have been trying to use BHO’s graphic library in a way a user can click a graph to change inputs. However, due to some deficiency in BHO’s graphic library, I need to modify some behavior in the portlet and this requires some changes in data structures in the existing portlet. I haven’t finished yet but will finish soon.

2. Workflow mapping in Grids
As a part of implementation of workflow mapping developed by Yili, I have been making test programs as follows:
a. GPIR client (Completed)
GPIR is a web service as defined in http://gridport.net/services/gpir/webservices.html. It supports two operations: GPIR Query to patch information and GPIR Ingester for updating. By using a query, we can get server information about load, jobs, downtime, queues, queue, and so on. A part of codes as follows:

call.setTargetObjectURI(WS_NAME); // Set a web service API
call.setMethodName(WS_METHOD); // Set method name
Vector params = new Vector();
params.addElement(new Parameter("strQuery", String.class, strQuery, null));
params.addElement(new Parameter("strVO", String.class, strVO, null));
call.setParams(params);
Response resp = call.invoke(url, ""); // Web service invocation

Whole sample code can be found in the same URL above.
b. Network Weather Service (Not yet. A few bugs need to be cleared out)
This is also a web service which provides forecasting service to predict on performance considering network conditions and resources. A sample code can be found in http://nws.cs.ucsb.edu/ewiki/nws.php?id=BQP+Web+Service

c. Running time prediction (Not yet)

After finishing those, I will discuss Yili to define interfaces to give those data.

Status Report for 02/14/07 to 02/21/07

2007-02-21T14:29:00.000-05:00

1. Time series analysis graph
I have been working with drawing time series analysis graph by using BFO’s graphic library (http://big.faceless.org/). By using this java library, we can draw graphs with ease. Since the library can be used as a tag library, we can use this with JSP or JSF. Especially, with one of JSTL tag library, , BFO’s graph library can make a graph iteratively without explicit operation. The variable used in this case can be an array or list in the bean class. Another advantage of using BFO’s graphic library, we can make the graph interactive so that a user can click data inside the graph and verify its contents. For this purpose, we need to use both javascript and png graph format. As for the next step, I will put more interactive features to improve user interface in time series analysis portlet of QuakeSim2 project.

2. Workflow mapping in Grids
I’ve started to help Yili to implement workflow mapping scheme. My work is to supply information such as GPIR, Network Weather Service, running time prediction to a match maker which will construct an efficient DAGMan workflow based on the inputs. Currently I’m looking at what kind of interfaces will be used between the match maker and data suppliers

Status Report for 02/08/07 to 02/14/07

2007-02-15T12:14:00.000-05:00

1. Analyze-tseri web service
I have been working with building a web service to execute Time Series Analysis (analyze_tseri). A few bugs have been removed during the last week and now it works very well so far. Using the new web service, a user needs to send mainly parameters, in contrast to the previous case a user needs to generate a drive file to execute Time Series Analysis application (analyze_tseri_linux). A test client has been implemented and is running at http://gf1.ucs.indiana.edu:8888/STFILTER/EstimatedParams.jsf. A button “Run ST_FILTER_WS” will execute the analyze_tseri web service, while another button “Run ST_FILTER” still invokes the old method.
To refer to the definition of the service, a user can see the definition (WSDL) from here (http://gf1.ucs.indiana.edu:8888/analyze-tseri-exec/services/AnalyzeTseriExec)

As an output, the Time Series Analysis web service generates 3 picture files and returns the urls of the files so that a user can download the output from the server. A sample output is as follows:

http://gf1.ucs.indiana.edu:8888//analyze-tseri-exec/public_project-TEST/public_project-TEST.input.xyz.X.png http://gf1.ucs.indiana.edu:8888//analyze-tseri-exec/public_project-TEST/public_project-TEST.input.xyz.Y.png http://gf1.ucs.indiana.edu:8888//analyze-tseri-exec/public_project-TEST/public_project-TEST.input.xyz.Z.png

2. Paper submission to TeraGrid 07
Working with Mehment Nacar and Marlon, a paper for TeraGrid07 conference has been submitted. The paper mainly focuses on how OGCE and portlets was used to build Big Red portal to support customized application execution such as MEME command and showq command and how Grid Tags Libraries And Beans (GTLAB), developed by Mehmet Nacar, can be used for the best portlet production. A file is available to access at here (http://www.cs.indiana.edu/~jychoi/files/Teragrid_Big_Red_Portal-final.pdf)

Status Report for 02/01/07 to 02/07/07

2007-02-07T22:01:00.000-05:00

I’ve just finished implementing a client (STFILTER) which will invoke time series analysis web service (AnalyzeTseriService). I added a web service version for time series analysis, called execATS. WSDL of execATS can be accessed at http://gf1.ucs.indiana.edu:8888/analyze-tseri-exec/services/AnalyzeTseriExec.

execATS needs the following 4 arguments as input:

String siteCode – Site code such as dhlg
String dataURL – Data file URL from which the server will download data file
double[][] globalParam – Parameters for global station. Array of (double Parameter type, double param_1, … , double param_n)
double[][] siteParam – Parameters for the specific station . Array of (double Parameter type, double param_1, … , double param_n)

A very simple example to create a client to invoke web service can be found at http://ws.apache.org/axis/java/user-guide.html#BasicsGettingStarted. In a nut shell, execATS web service can be executed from a client by using the following code:

String siteCode = "dhlg";
String dataUrl = "http://server:port/datafile";
double[][] globalParam = {
{ 7, 0.0, 50.0, 2005.0, 2008.0 },
{ 9, 0.0, 50.0, 2005.0, 2008.0 } };
double[][] siteParam = {
{ 8, 0.0, 50.0, 2005.0, 2008.0 } };

Service service = new Service();
Call call = (Call) service.createCall();

call.setTargetEndpointAddress(new java.net.URL(endpoint));
call.setOperationName(new QName("http://soapinterop.org/",
"execATS"));

String[] ret = (String[]) call.invoke(new Object[] {siteCode, dataUrl, globalParam, siteParam});

Status Report for 01/18/07 to 01/24/07

2007-01-24T12:02:00.000-05:00

During the last week, I mainly focused on the following tasks:
1. Implementation of Time Series Analysis as a web service -- Most implementation is working now. I’m making the implementation to work as a web service.

To execute analyze_tseri_linux, we need to create a few input files. Among them, a parameter file needs more attention on creation. No blank line is allowed for separating sites. In the previous implementation, new line was added automatically, which caused error.
To server as a web service, type of inputs should be carefully considered. If a complex data type is used such as object, the data type should be serializable or mashalable.

2. Preparing to submit a paper to TeraGrid 07 about Big Red portal.

The paper will contain the experience of Big Red portal and portlet implementation for SC 06.
Grid Tag Library and Grid Beans made by Mehmet Nacar will be discussed too.

Status Report for 01/11/07 to 01/17/07

2007-01-17T12:17:00.001-05:00

I’m working on QuakeSim2 project by implementing a web service for Time Series Analysis. Currently deployed Time Series Analysis service needs to be improved for providing better user interfaces. For this purpose some functionality in the old implementation should be moved into a new service implementation. The following works should be done:

Implementing a stand-alone Time Series Analysis application. The new application will generate drive files for given user input and execute analyze_tseri_linux utility. (will be finished at the end of this week)
Converting the application into a web service application.

After finishing this implementation, I will help out to develop another module in QuakeSim2 project and start with Polar Grid project.

Status Report for 12/21/06 to 12/27/06

2006-12-27T11:44:00.000-05:00

I've installed CEMon Client APIs in my local machine. The following is my summary of CEMon Client API installation.
1. Download the most recent files from http://grid.pd.infn.it/cemon/ClientRPMs
2. Install RPMs as follow:
rpm -ivh c-ares-1.3.0-1.slc3.i686.rpm
rpm -ivh glite-security-gsoap-plugin-1.4.0-0_U200612071426.i386.rpm
rpm -ivh gpt-VDT1.2.2rh9-1.i386.rpm
rpm -ivh glite-security-gsoap-plugin-1.4.0-0_U200612071426.i386.rpm
rpm -ivh glite-ce-monitor-client-api-c-1.7.5-0_U200612071517.i386.rpm
3. Set up library path:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/glite/lib:/opt/globus/lib:/opt/c-ares/lib
4. Testing
cd /opt/glite/bin
CEMonitorGetInfo

Although I haven't completed verifying that CEMon Client APIs are working in my environments, I'm looking at how CEMon is working in client side.

Status Report for 12/13/06 to 12/20/06

2006-12-20T17:02:00.000-05:00

Basically CEMon is used to provide/get information for helping resource selection. The service consists of two parts: a CEMon server and a client. While the CEMon server produces information and publishes to its subscribers, the CEMon client consumes the events. CEMon is based on a web service providing WSDL and SOAP message is needed for subscription.

To implement CEMon as a web client using OGCE, I need to do the followings:

Install CEMon command line tools and try to use them to know how CEMon service is working
Analyze CEMon’s WSDL to find commands between CEMon server and clients
Implement CEMon web client by using Java

Now I’m ready to submit a job to OSG since I got a valid certificate and registered a member of OSG VO. For the next week, I will complete step 1.

Status Report for 12/06/06 to 12/12/06

2006-12-13T11:48:00.001-05:00

Currently I’m playing around OSG and reading some documents about CEMon. Especially, I found that the following website is useful: http://vdt.cs.wisc.edu/components/cemon.html. After understanding CEMon, I’m planning to build a dashboard portlet which can work with CEMon.

Status Report for 11/22/06 to 11/28/06

2006-11-29T08:44:00.000-05:00

1. OGCE portlets
I’m still updating source codes of MEME portlets and Jobqueue portlets by adding comments and removing debugging messages. The source codes and war files will be exported to the SourceForge repository. The following portlets will be added:
a. memejob-portlet : A portlet for MEME job execution
b. moab-dashboard : A portlet for job queue status query
c. serialjob-portlet : A general portlet to execute a series of jobs

2. Open Science Grid
Meanwhile waiting for a user certificate from OSG CA, I’m working on setting up OSG client in my local machine. I think my installation was successful but $VDT_LOCATION/post-install/README reports a few warnings I should fix. I will clear up those as soon as possible and start to look at OGCE in OSG and CEMon.

Status Report for 11/15/06 to 11/21/06

2006-11-22T09:39:00.000-05:00

1. Open Science Grid
After coming back Supercomputing '06 at Tampa, I'm looking around Open Science Grid (OSG) to verify whether OGCE portal can also work with OSG environments. For this test, I’ve installed OSG client by following the client installation instructions and been trying to setup correct configuration which includes certificates for GSI settings.

2. OGCE portlets
I’m preparing to export source codes of MEME and job queue portlets to CVS repository by adding comments into the codes and removing verbose debugging messages.

Status Report for 11/1/06 to 11/7/06

2006-11-08T10:49:00.000-05:00

During the last week, I was working on Big Red portal by customizing for Supercomputing 06 and updating key portlets such as MEME job submission portlet and MOAB job queue portlets. Major works on portlets are as follows

1. MEME job submission portlet
a. File downloading: Basically, in portlets, file downloading feature is not working well since html header cannot be set by portlets but, instead, portlet container usually control html header such as "Content type" or "Content-Disposition". Thus, using servlet than portlet is more ideal in this situation. Good news here is that servlet and portlet can share data by using session. Thus, I created a servlet for file downloading which can get proxy information for authorization by using shared session.
b. Output retrieval: Outputs executed by a job managers can be retrieved later by using job handler which is an URL string starting with "http" or "https". Under the hood of this function, GASS server is in charge to save data and hand it over to the client upon request.

2. MOAB job queue viewer

a. Job query belongs to the user: Since each machine in Teragrid has different user map file on user certificate, username(id) can be different system by system even in using the same certificate. Simply, we can submit a GRAM job before starting query to get back a correct username by executing “whoami” command. Later, I’m planning to maintain information service to map usernames on different machines file by using RDF and Jena from HP labs.

Status Report for 10/25/06 to 10/31/06

2006-10-31T23:35:00.000-05:00

1. Introduction - A Big Red portal development project
The mission is to create portlets for Big Red portal which will be open to the public for those who want to use newly installed supercomputer in IU, Big Red. Most of portlets come directly from OGCE Grid portlets and a few portlets should be created to deal with Big Red's special needs. Currently the following portles are under development:
a. MEME job submission portlet : a portlet to execute a remote MEME command
b. Job status dashboard : a portlet to query job queues managed by MOAB job queue manager
c. IFrame for IU's Knowledge Base : a iframe portlet displaying IU's Knowlege Base webpages.

2. Week activity (10/25 ~ 10/31)
a. Datafile uploading in MEME portlet
-. Datafile uploading function is done.
-. Output downloading will be implemented soon

b. Job queue forecast image portlet

-. A fixed image can be refreshed continually in a specific interval (60 seconds or so).

c. MOAB dashboard

-. A radio button is added to specify a query condition. It can show all jobs or only user's jobs.

3. Next activity
a. Downloading functionality will be implemented in MEME job submission portlet. Then, users can download output directly from the same portlet
b. Job submission portlet enhancement : Adding a "Serial Job" and a "Parallel Job" portlet. A portlet will rather be more general in a way the portlet can be included in OGCE.