Status Report for 05/17/07 to 05/23/07

1. STFILTER portlet and Analyze_Tseri service
I’ve been updating STFILTER and Analyze_Tseri web service to make the system deal with multiple sites (Previously only single site can be supported). The implementation is almost done except displaying multiple graphs by using BFO library. Since the output from analyze_tseri application is a union of multiple outputs of single execution with a single site, the graph should be multiplied or multiple graphs should be displayed in one graph by merging them.

2. Network Bandwidth Information
Yili preferred to install NWS server/client in the nodes of TeraGrid. So, I’m making a list of nodes to install NWS client and server. There might be restrictions using a special port or network traffics. I will keep posted when I have problem.

Status Report for 05/10/07 to 05/16/07

1. STFILTER portlet and Analyze_Tseri service
During the last week, I’ve been updating STFILTER portlet and Analyze_Tseri to make the system to deal with multiple sites. To make sure correct operation, I started with making a test case to run standalone tseri_analysis program. Now, I’m updating STFILTER portlet and Analyze_Tseri webservice and will complete in the early of next week.

2. Network Bandwidth Information
I’ve been looking at again NWS (Network Weather Service) to find out how to get network bandwidth information in Teragrid from NWS but it seems like there is no publicly available method to share. Although we can install NWS server/clients for each node for testing purpose, it will suffer from low scalability.

Status Report for 05/03/07 to 05/09/07

STFILTER portlet and Analyze_Tseri service :
During the last week, I’ve been updating STFILTER portlet by using Db4o to support project management feature. Now STFILTER portlet can save query parameters as a project and reload later. I’ve also updated Analyze_Tseri service by changing interface parameters to receive GRWS URL as an input. Thus, STFILTER portlet doesn’t need to send data as String type. Instead, only GRWS URL is enough. Analyze_Tseri will download by connecting to GRWS service. In the future, Analyze_Tseri will be updated to query GRWS directly with given parameters. The next thing to do is to provide support for processing data from multiple stations.

Status Report for 04/26/07 to 05/02/07

I’m updating STFILTER to work with the object database, Db4o and almost ready to release. I’m planning to demonstrate STFILTER portlet to Marlon in a few days. A few minor issues needs to be considered are as follows (including previous ones):

1. Problems in implementing custom serializer/de-serializer of analyze_tseri service. Making webservice exchange user-defined complex type has some errors.
2. Lock/Un-lock features. I have lock related errors in using Db4o. Currently, I disabled lock features to prevent such errors. But, I don’t know which is better way for STFILTER portlet.
3. Deleting temporary files. Analyze_tseri web service is now saving user’s input and output under a directory in a way a user can download files directly through http protocol. However, at some point, such files should be deleted to save disk spaces. I didn’t decide yet when and which is the best way to do that.

Status Report for 04/19/07 to 04/25/07

I have updated STFILTER by adding project management function and input and output file downloading feature. For this purpose, new STFILTERProjectBean class has been added like RDAHMM and an object database, Db4o, is used to save and restore STFILTERProjectBean object. I also cleared out most of dependencies on old context manager class. A few minor issues I'm facing are:

1. Lock related errors in using Db4o. Currently, I disabled lock features to prevent such errors. But, I don’t know which is better way for STFILTER portlet.
   
2. Analyze_tseri web service is now saving user’s input and output under a directory in a way a user can download files directly through http protocol. However, at some point, such files should be deleted to save disk spaces. I didn’t decided yet when and which is the best way to do that.

During the next week, I will do more test to find out any bug and improve some cosmetic things.

Status Report for 04/012/07 to 04/18/07

1. STFILTER portlet
I’ve updated STFILTER portlet by adding more time series parameter filters. In addition, time series analysis service (analyzeTseri) and STFILTER portlet has be updated with easy installation procedure so that most user dependent system settings can be done in installation step As for the next step, I will add project management features in STFILTER portlet.
2. Network Weather Service
I’m still searching for how to get list of hosts served as Network Weather Service hosts. There is no open information to list up such hosts. I think we need to ask either TeraGrid help desk or Network Weather Service development team

Status Report for 04/05/07 to 04/11/07

1. STFILTER portlet
I’m still updating both analyze-tseri-exec webservice and STFILTER portlet to work with addition parameters for time series analysis. Although no technical difficulties yet, it takes some time to make them more general as they are sharing same code in many cases. I will finish this update in a few days. After updating, I will modify the project management part of STFILTER portlet.

2. TeraGrid 07 final paper submission
I’ve been updating TeraGrid 07 paper for camera ready, co-worked with Mehmet and Marlon. The final due is April 12th.

3. Network Weather Service
I sent an email to NWS’s developer mailing list (nws-dev@cs.ucsb.edu) to query about any possibility to get network bandwidth service but haven’t get any reply yet. To prepare worst case, I think I need to find other way to get such information.

Status Report for 03/29/07 tto 04/04/07

1. STFILTER portlet
I'm updating analyze_tseri web service to use a bean as an input parameter. Accordingly STFILTER portlet will be updated soon.
As for the next step, I'm planning to upgrade STFILTER portlet as follow:
a. Adding more options on filter parameter.
b. Using property file, instead of web.xml or portlet.xml
c. Modifying project management
d. Fixing some bugs when clicking "edit" mode button.

2. Network Weather Service
After installing nws package from NWS website, I found a NWS command for extracting inter-machine bandwidth and latency measurements as follow:

$ html-hosts -N nws.cs.ucsb.edu sperm.cs.ucsb.edu beluga.cs.ucsb.edu

Bandwidth values are measured in Megabits/second; latency in milliseconds

Source machine			Destination machine
			sperm.cs.ucsb.edu	beluga.cs.ucsb.edu
sperm.cs.ucsb.edu		bw forecast error lt forecast error	- - - - - -	86.10412000 86.10059000 1.759016353 0.429000000 0.370000000 0.091088105
beluga.cs.ucsb.edu		bw forecast error lt forecast error	85.42371000 85.52822000 4.072307805 1.648000000 1.702000000 0.320489947	- - - - - -

However, I haven't found yet if I can use this command more generally, for example, in TeraGrid clusters. It seems like if I know a name server of a cluster, it will be possible to get network bandwidth information. But, I don't know how I can get list of such name server. I will dig into that more.

Moab dashboard or memejob portlet configuration

1. memejob-portlet
a. Copy memejob-portlet.war to $CATALINA_HOME/webapps
b. Modify ./WEB-INF/portlet.xml as follows:
Basically, by using memejob-portlet, memejob can be send to multiple sites as shown in "hostnames" parameters. If you don't need multiple site, only one host name in "hostnames" parameter will be enough. The other values should be set with similar way.
-. hostnames : list of hosts to submit meme job. Comma separated and no space between them.
-. batchhostnames : list of hosts names for batch job submission
-. commandset: list of meme execution file path. The order must be the same with list of hostnames
-. pathset : list of path env. The order must be the same with list of hostnames.
-. gridftpservers : list of ftp servers for each host shown in hostnames.

2. moab-dashboard
a. Copy moab-dashboard.war to $CATALINA_HOME/webapps
b. Modify ./WEB-INF/portlet.xml as follows:
moab-dashboard portlet can support executing moab command in multiple sites. You can specify either list of sites in "hostnames" or one site if you want only support only one site.
-. hostnames: list of host names
-. commandset: list of path of moab command. The order should be same with hostnames.
-. checkjob: list of path of 'checkjob' command.

3. Displaying in OGCE portal

When the deployed portlet is shown as a non-portlet web application in “Adminitration” tab of OGCE, we can change it as a portlet application as follow:

a. In “Administration” tab, type “moab-dashboard”(or memejob-portlet) in “Deploy new portlet webapp” and click “Deploy” button. If successful, you can see “moab-dashboard” in “Portlet web applications” section. You may need to refresh the screen by visiting other tab or re-logging.

b. Go to “Welcome” tab, check “moab-dashboard”(or memejob-portlet) in “Configure group membership” and click “Save” button. You can see a new tab named “MOAB Dashbarod”(or “Meme job”)

Status Report for 03/22/07 to 03/28/07

1. STFILTER and AnalyzeTseriService
I have made AnalyzeTseriService more self-contained that the service can get environmental parameters as well as array of time series data. Thus, STFILTER application can query more dynamically from the AnalyzeTseriService service. However, the better way is to make the service receive a bean as an input parameter so that a client only needs to send only a bean.

During the last week, I have been busy in doing some class project. I'm planning to upgrade STFILTER portlet and investigate on Network Weather Service.

Status Report for 03/15/07 to 03/21/07

1. STFILTER portlet
I have developed STFILTER portlet by converting stand alone STFILTER application which is using both JSF and JSTL tag. In the conversion process, I faced a problem for some reason that JSTL tag didn’t work with portlet environment. Especially, JSTL’s EL expression didn’t parse correctly, such as ${bean.param} or ${0}, to evaluate bean property or expressions. To overcome the problem, I put the following code inside JSP scriplet in .jsp file:

FacesContext context = FacesContext.getCurrentInstance();
Application app = context.getApplication();
ValueBinding vbName = app.createValueBinding("#{stfilterBean.graphName}");
String[] strName = (String[]) vbName.getValue(context);

Then, use JSTL tag as follows:

2. Network Weather Service
I'm still looking around to get network bandwidth information from Network Weather Service.

STFILTER portlet configuration

To make STFILTER portlet work, the following values should be set properly.

1. Replace Google key according to the site url in the following files: allstations.jsp, map.jsp, and NewProject.jsp which are in the installed directory. For example, replace _new_key_ in

   http://maps.google.com/maps?file=api&v=2&key=_new_key_

   You can have new Google key from http://www.google.com/apis/maps/signup.html.
   
2. In ./WEB-INF/faces-config.xml, values should be set as follow:
   -. contextUrl : context manager web service url
   -. contextBasePath : context manager base path
   
3. work.dir value in ./WEB-INF/web.xml should be replaced. Note that this directory should be accessible by using httpd
   
4. Specify SOPAC GPS parameters in ./WEB-INF/classes/sensorgrid.properties.

Status Report for 03/08/07 to 03/14/07

1. User Interface using JavaScript
I have been trying to improve user interface of st_filter, mostly focusing on using JavaScript and I found that a few useful sites:

• Yahoo UI: A few ready-to use components such as calendar and dialog box. A few basic tools for developing UI are also available such as event handling and DOM.
• JavaScript Image Cropper UI: If one need to handle images, this tool might be helpful. A very handy script to make an image cropper.
• script.aculo.us : You can make lots of effects on objects in HTML.
• Prototype: A framework. I was surprised how JavaScript is complicated and can be improved by applying OOP and functional programming language concept. More details on JavaScript as a programming language can be found in here . Especially video clips by Douglas Crockford are my recommendation.

2. st_filter portlet
I’ve started to convert standalone st_filter web application into a portlet. I’ve encountered a few problems but I think it can be solved soon. I will put some summary in the next post.

3. Workflow mapping in Grids
I discussed with Yili about gathering information from Network Weather Service and GPIR to schedule workflow. Although simple information can be obtainable, more complicated and detail information is hard to get. I will look over again beyond documentation and look at the source code, if possible.

Status Report for 03/01/07 to 03/07/07

1. Time series analysis graph
I have developed drawing graphs by using BHO's graph library for the last week. To make the graph interactively, we need to combine javascript with the BHO's graphic lib.



The code will draw a line graph with the vector data obtained by stfilterBean.getFilteredList(). To patch data iteratively, JSTL's c:foreach statement is used. To use JSTL, we make two jar files, jstl.jar and standard.jar, available in the library path. For better UI interface, the next step will be using Yahoo's UI library (http://developer.yahoo.com/yui/)

2. Workflow mapping in Grids
a. GPIR client (Done last week)
b. Network Weather Service (Done) : I had trouble in generating a client from WSDL of Network Weather Service by running WSDL2JAVA. I figured it out by installing a newest version of axis library. Now it's working correctly. I will discuss with Yili for more details
c. Running time prediction (Not yet) : I will discuss with Yili.

Status Report for 02/22/07 to 02/28/07

1. Time series analysis graph
To improve interface of the time series analysis portlet, I have been trying to use BHO’s graphic library in a way a user can click a graph to change inputs. However, due to some deficiency in BHO’s graphic library, I need to modify some behavior in the portlet and this requires some changes in data structures in the existing portlet. I haven’t finished yet but will finish soon.

2. Workflow mapping in Grids
As a part of implementation of workflow mapping developed by Yili, I have been making test programs as follows:
a. GPIR client (Completed)
GPIR is a web service as defined in http://gridport.net/services/gpir/webservices.html. It supports two operations: GPIR Query to patch information and GPIR Ingester for updating. By using a query, we can get server information about load, jobs, downtime, queues, queue, and so on. A part of codes as follows:

call.setTargetObjectURI(WS_NAME); // Set a web service API
call.setMethodName(WS_METHOD); // Set method name
Vector params = new Vector();
params.addElement(new Parameter("strQuery", String.class, strQuery, null));
params.addElement(new Parameter("strVO", String.class, strVO, null));
call.setParams(params);
Response resp = call.invoke(url, ""); // Web service invocation

Whole sample code can be found in the same URL above.
b. Network Weather Service (Not yet. A few bugs need to be cleared out)
This is also a web service which provides forecasting service to predict on performance considering network conditions and resources. A sample code can be found in http://nws.cs.ucsb.edu/ewiki/nws.php?id=BQP+Web+Service

c. Running time prediction (Not yet)

After finishing those, I will discuss Yili to define interfaces to give those data.

Status Report for 02/14/07 to 02/21/07

1. Time series analysis graph
I have been working with drawing time series analysis graph by using BFO’s graphic library (http://big.faceless.org/). By using this java library, we can draw graphs with ease. Since the library can be used as a tag library, we can use this with JSP or JSF. Especially, with one of JSTL tag library, , BFO’s graph library can make a graph iteratively without explicit operation. The variable used in this case can be an array or list in the bean class. Another advantage of using BFO’s graphic library, we can make the graph interactive so that a user can click data inside the graph and verify its contents. For this purpose, we need to use both javascript and png graph format. As for the next step, I will put more interactive features to improve user interface in time series analysis portlet of QuakeSim2 project.

2. Workflow mapping in Grids
I’ve started to help Yili to implement workflow mapping scheme. My work is to supply information such as GPIR, Network Weather Service, running time prediction to a match maker which will construct an efficient DAGMan workflow based on the inputs. Currently I’m looking at what kind of interfaces will be used between the match maker and data suppliers

Status Report for 02/08/07 to 02/14/07

1. Analyze-tseri web service
I have been working with building a web service to execute Time Series Analysis (analyze_tseri). A few bugs have been removed during the last week and now it works very well so far. Using the new web service, a user needs to send mainly parameters, in contrast to the previous case a user needs to generate a drive file to execute Time Series Analysis application (analyze_tseri_linux). A test client has been implemented and is running at http://gf1.ucs.indiana.edu:8888/STFILTER/EstimatedParams.jsf. A button “Run ST_FILTER_WS” will execute the analyze_tseri web service, while another button “Run ST_FILTER” still invokes the old method.
To refer to the definition of the service, a user can see the definition (WSDL) from here (http://gf1.ucs.indiana.edu:8888/analyze-tseri-exec/services/AnalyzeTseriExec)

As an output, the Time Series Analysis web service generates 3 picture files and returns the urls of the files so that a user can download the output from the server. A sample output is as follows:

http://gf1.ucs.indiana.edu:8888//analyze-tseri-exec/public_project-TEST/public_project-TEST.input.xyz.X.png http://gf1.ucs.indiana.edu:8888//analyze-tseri-exec/public_project-TEST/public_project-TEST.input.xyz.Y.png http://gf1.ucs.indiana.edu:8888//analyze-tseri-exec/public_project-TEST/public_project-TEST.input.xyz.Z.png

2. Paper submission to TeraGrid 07
Working with Mehment Nacar and Marlon, a paper for TeraGrid07 conference has been submitted. The paper mainly focuses on how OGCE and portlets was used to build Big Red portal to support customized application execution such as MEME command and showq command and how Grid Tags Libraries And Beans (GTLAB), developed by Mehmet Nacar, can be used for the best portlet production. A file is available to access at here (http://www.cs.indiana.edu/~jychoi/files/Teragrid_Big_Red_Portal-final.pdf)

Status Report for 02/01/07 to 02/07/07

I’ve just finished implementing a client (STFILTER) which will invoke time series analysis web service (AnalyzeTseriService). I added a web service version for time series analysis, called execATS. WSDL of execATS can be accessed at http://gf1.ucs.indiana.edu:8888/analyze-tseri-exec/services/AnalyzeTseriExec.

execATS needs the following 4 arguments as input:

1. String siteCode – Site code such as dhlg
2. String dataURL – Data file URL from which the server will download data file
3. double[][] globalParam – Parameters for global station. Array of (double Parameter type, double param_1, … , double param_n)
4. double[][] siteParam – Parameters for the specific station . Array of (double Parameter type, double param_1, … , double param_n)
   

A very simple example to create a client to invoke web service can be found at http://ws.apache.org/axis/java/user-guide.html#BasicsGettingStarted. In a nut shell, execATS web service can be executed from a client by using the following code:

String siteCode = "dhlg";
String dataUrl = "http://server:port/datafile";
double[][] globalParam = {
{ 7, 0.0, 50.0, 2005.0, 2008.0 },
{ 9, 0.0, 50.0, 2005.0, 2008.0 } };
double[][] siteParam = {
{ 8, 0.0, 50.0, 2005.0, 2008.0 } };

Service service = new Service();
Call call = (Call) service.createCall();

call.setTargetEndpointAddress(new java.net.URL(endpoint));
call.setOperationName(new QName("http://soapinterop.org/",
"execATS"));

String[] ret = (String[]) call.invoke(new Object[] {siteCode, dataUrl, globalParam, siteParam});

Status Report for 01/18/07 to 01/24/07

During the last week, I mainly focused on the following tasks:
1. Implementation of Time Series Analysis as a web service -- Most implementation is working now. I’m making the implementation to work as a web service.

• To execute analyze_tseri_linux, we need to create a few input files. Among them, a parameter file needs more attention on creation. No blank line is allowed for separating sites. In the previous implementation, new line was added automatically, which caused error.
• To server as a web service, type of inputs should be carefully considered. If a complex data type is used such as object, the data type should be serializable or mashalable.

2. Preparing to submit a paper to TeraGrid 07 about Big Red portal.

• The paper will contain the experience of Big Red portal and portlet implementation for SC 06.
• Grid Tag Library and Grid Beans made by Mehmet Nacar will be discussed too.

Status Report for 01/11/07 to 01/17/07

I’m working on QuakeSim2 project by implementing a web service for Time Series Analysis. Currently deployed Time Series Analysis service needs to be improved for providing better user interfaces. For this purpose some functionality in the old implementation should be moved into a new service implementation. The following works should be done:

1. Implementing a stand-alone Time Series Analysis application. The new application will generate drive files for given user input and execute analyze_tseri_linux utility. (will be finished at the end of this week)
2. Converting the application into a web service application.

After finishing this implementation, I will help out to develop another module in QuakeSim2 project and start with Polar Grid project.