TheGridBLOG

Grids are on the rise - w00t!

The Grid Looking Glass tracks the number of "grid computing" google queries.

Summary of Some Grid Services Workflow Papers


Grid Workflow
Hugh P. Bivens
Sandia National Laboratories

This paper proposes an XML meta-langauge with custom tags and his own DTDs.
example
< !DOCTYPE WorkFlow SYSTEM "gale.dtd">
< WorkFlow id="Hawaii">
...
< DataTransfer id="stage_camera_file">
< Argument name="source" value="/home/hpbiven/hawaii/camera.param"/>
< Argument name="destination"
value="@target1.hostname@:@target1.scratchdir@/@grid.user@.camera.param"/>
</ DataTransfer>
...
This work appears to be pre-web services and does not provide for functionality such as concurrent job submission, conditional statements, and the OGSA framework.


Workflow Management Coalition
http://www.aiim.org/wfmc
They provide a nice definition of workflow, and propose their own standards.

The workflow management coalition defines workflow to be the following:
The automation of a business process, in whole or part, during which documents, information or tasks
are passed from one participant to another for action, according to a set of procedural rules.
Workflow Management Coalition Terminology & Glossary (WFMC-TC-1011, v3.0),
http://www.aiim.org/wfmc/standards/docs/glossy3.pdf


GSFL: A Workflow Framework for Grid Services
Sriram Krishnan, Patrick Wagstrom, Gregor von Laszewski

Mathematics and Computer Science Division, Argonne National Laboratory
Indiana University
Illinois Institute of Technology

Defines yet another XML meta language for grid services workflow. Pays homage to all the previous XML-meta languages for WSFL, XLANG, WSCL, WSCI, etc

Elements of the language provide separate models for control flow and data flow, allowing for peer-to-peer communication of data; very useful for data-intensive scientific applications.

They propose an engine to interpret the GSFL documents and dynamically generate WSDLs. So services can be bound to consumers at run-time.

This has possibly been abandoned as it is complex and no quality implementation will exist in the foreseeable future. This workflow also leaves out conditional statements and loops.



Abstracting the Grid
G. von Laszewski 2003

This paper lays out the plans for a software toolkit, Grid SDK. This toolkit provides a layer of abstraction on top of a specific grid software platform such as Globus 2, Globus 3, Unicore, and Legion.

The paper says:
... we propose a set of initial data structures, interfaces, and the patterns
of their interaction. We call such a standard client development environment Open Grid Computing Environment (OGCE).

By building the front-end of grid service consumer application on top of OGCE, the authors hope to encourage the development of graphical end-user tools.

There are tasks, that corespond to grid services, and supertasks, which corespond to a collection of talks. Tasks have 'connects' adapter objects specific to the underlying software. For example to submit a job for a service on a Globus grid you would use the Gram connects.

The paper promises an "extensible “lingua franca” for Grid application development." The question is will this set of interfaces and design patterns be adopted, and will it hold up "when the rubber meets the road?"


myGrid: Personalised Bioinformatics on the Information Grid
Robert Stevens, Alan Robinson, Carole Goble
University of Manchester, European Bioinformatics Institute

Claim to have prototypes of BioServices are available from the myGrid web site for NCBI & WU BLAST and the complete EMBOSS application suite, MEDLINE & SRS.
Claims that these services are OGSA-complient and have published WSDLs via UDDI.
Claims that workflow is possible through a Web Service Flow Language engine.
Claim each user keeps personalized data (workflows and experiments) in the Grid Information Repository.

The one million dollar question is, can someone else invoke these services?

All About BPEL

OASIS working group:
http://www.oasis-open.org/committees/join.php?wg_abbrev=wsbpel

IBM Spec:
http://www-106.ibm.com/developerworks/webservices/library/ws-bpel/

Microsoft Spec:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbiz2k2/html/bpel1-1.asp

Pattern Based Analysis of BPEL4WS
Stockholm University/The Royal Institute of Technology, Sweden
http://gunther.smeal.psu.edu/papers/E-Commerce/435/http:zSzzSztmitwww.tm.tue.nlzSzresearchzSzpatternszSzqut_bpel_rep.pdf/pattern-based-analysis-of.pdf

Some BPEL examples:
http://www.erc.msstate.edu/~ammari/

Email to Vouk, Singh, Zhengang, and Brent

I'm been in contact with the principal investigator of the Globus CoG project, and he's invited me to join the team of people working on the revision of the grid services flow language.

Since the Globus project proposed the grid services flow language several years ago, much has changed, and work is now underway to stabilize and standardize workflow and ease-of-use wrappers for grid services. Specifically, they are working on updating the specification and Java engine to take into account the new Business Process Execution Language (BPEL) and a not-yet-public version of the Globus CoG kit.

This is too large of a project for a single person to do in 8 months, but we can make progress in defining the problems, re-shaping the spec, and participate in the early stages of creating the software. The work that is most needed to be done to advance the release of better grid services tools, is to isolate the differences between the Grid Service Workflow and BPEL to express how the differences will be reflected in the new software release.

An extensive literature review and code integration project will be getting underway in the next month.

I believe that research on Grid Service Composition should be undertaken at NC State. I would like to invite all of you participate.

I would also like to request a meeting with Dr. Vouk so we could add Gregor von Laszewski as an advisor in an official capacity. This is necessary so that he can share the not-yet-public documents and source code from the Globus Project and Argonne National Laboratory.

Regards,
Daniel

spoke to Gregor today.

He's doing bioinformatics stuff too.

GSFL hasn't been looked at in a year and needs to be updated to take into account the new WS specs and BPEL.

There is another effort GridANT that we may leverage. There is a new CoG kit that is really good for actual use.

One person can't hope to straighten all this stuff out, but I can assist Gregor and his team with a problem definition and liturature review, so that other people can.

I'm going to start collecting all the stuff I can find in regrads to grid services, BPEL, and workflow. And there is a ton of it.

Description of the Independent Study

The goal of this research project is to make grid services easier to use for both the programmers and application users. Grid services have the potential to make future applications more powerful by providing invisible access to remote processing power and storage.

The most prominent effort to make grid services easier to use is the Globus Commodity Grid Kits. These kits, such as the Java CoG, package together a number of useful tools under a common framework. The Java CoG toolkit has a number of active projects underway that, once completed, will be integrated into the toolkit.

The specific CoG toolkit that I’m working with is the Grid Services Flow Language project. The GSFL project was begun last year when a team drafted a specification for the GSFL. GSFL is an XML Meta Language that allows for the aggregation and orchestration of grid services.

Besides the GSFL, there are several other prominent workflow technologies. There is Business Process Execution Language for Web Services (BPEL4WS) which has superseded several previous specification for Web Services Flow Language (WSFL) and the WSCI and XLANG specifications. There are a large number of XML meta-languages which aim to define abstract workflows and processes, including process-based approaches, notification-based approaches, and agent-based approaches.

Current research in workflow for grid services is centered on integrating these various technologies.

Specifically, my short term goal is to work with the GSFL team to leverage the BPEL4WS technology for the revision of the GSFL specification and the implementation of a GSFL Java engine.

My final deliverable is a prototype application that executes the workflow defined ina workflow meta-document. The XML document adopted by the Globus Project CoG toolkit will most likely be some hybrid of BPEL and GSFL.

Another deliverable is some documentation that would help other students pick up this technology faster.

The problem is that there are many underlying specifications such as WS-Coordination, WS-Transactions, XSD, and GWSDL/WSDL that are evolving. We are tying to hit a moving target.

Here’s another working title
Exploratory Research in XML Based Meta Languages for Service Workflow Construction

How’s that for a thesis title?

The tasks I’m working on now are:

(1) Investigate and create a list of methods for dynamic swapping of grid services endpoint references. This is a major shortcoming of BPEL and many other workflow languages that needs to be address when I have grid service factories that can dynamically create new services, and hence new endpoints.

(2) Answer: What components of GSFL support stateful services? What components support service data?

I will also solicit guidance from the CoG team, Dr. Singh, and Dr. Vouk.

w00t! -- I'm joining the Globus Project

I exchanged emails with Gregor von Laszewski, a Scientists at Argonne National Laboratory who is heading up the Grid Services Flow Language GSFL effort for the Globus project and the CoG toolkit. Someone on the team just quit, so I'm joining up.

This means that instead of coming up with my own little workflow architecture, I'm going to contribute to one of the most prominent workflow projects and the open grid services community can benefit from my work.

What he says is that they're revising the current proposals for grid services workflow to take into account the new developments in workflow from business process execution language for web services BPEL4WS.

I think the goal is to integrate the GSFL with the BPEL, or at least try to figure out how to leverage both technologies, so they do not compete. So I want to help shape the spec and contribute code. My thesis project may be to build a prototype to illustrate the emerging workflow language.

Met with Dr. Vouk today. We discussed the purpose of The Grid, OGSA, and why it's not being used yet. Using grid services is too complex, too much "hassle".

I want to make grid services easy to use for the service consumer. The service consumer can be a programmer or an application user. Dr. Vouk says he wants to make grid services easy enough so non-computer scientists (i.e. application users) can use them.
Grid services should hide their enormous complexity from non-programmers, and be more of a black box. One day we'll have applications that relay on grid services for improved processing power, storage, fault tolerance, and security. All the user will see is better, more powerful, applications.

It's oh-so easy to slip into the tar pit of grid service construction. We agree that it would be a huge and unrewarding project to build a production grid hosting real services.

There is also the interesting topic of service discovery, I may have made some progress advocating WSDL thought WSIL as a better approach than UDDI.

I'll narrow it down further to only looking at application that aggregate multiple grid services.

Workflow Construction with Stateful Services and Service Data

How's that for a thesis title? In the next 2 weeks, I'm going to write a statement of the problems in this area and a proposal for how to go about solving one of them. I'm also going to do some more research to convince Dr. Vouk (and myself) that there are new software engineering methods being created when you work with a service-oriented model.

Service Data -- w00t!!

I created and deployed a new grid service which retrieves service metadata; it tells you the name of the last operation called and the total number of operations available.

The Grid Service Handle is
http://context1.vnet.ncsu.edu:8080/ogsa/services/tutorial/core/servicedata/MathFactoryService

I can't call this service yet, because I'm having trouble generating stubs.

The client needs to have access to an XSD schema representing the service data, and generate and compile stubs to use. I have doubts about the usefulness of remotely accessing such data.

The claim is that a Service Data Set can contain zero or more Service Data Elements which correspond to different 'Types' of instances of the same service. The same Factory can create different Types of service Instances.

Does Service Data have a place in this project? Can a Factory Service have service data in the same way the instances can? There is currently no way for service data to maintain transactional integrity ( I think). The a bunch of service data interface methods have to be implemented and you have to write the wsdl/xsd. It just looks like a more standardized way of accessing reflection-type instance data.

So how useful is servicedata?

That, remain to be seen.

prepared some Powerpoint Slides on OGSA and what part I'm focusing on.

get them here

I have some examples grid services deployed from The Globus Toolkit 3 Programmer's

Tutorial
http://www.casa-sotomayor.net/gt3-tutorial/

I have a 'Math' Service the Grid Service Handle is
http://context1.vnet.ncsu.edu:8080/ogsa/services/tutorial/core/first/MathService

There's a 'Math Factory' service at
http://context1.vnet.ncsu.edu:8080/ogsa/services/tutorial/core/factory/MathFactoryService

And there's an instance of the service that the factory creates at
http://context1.vnet.ncsu.edu:8080/ogsa/services/tutorial/core/factory/MathFactoryService/math

I've also worked with Zhengang and wrote some grid service actors for Ptolemy, and
integrated these actors with the CVS project.

The big thing that these grid services add is the ability to dynamically create and
deploy new web services that maintain state.

The only problem is when you have a new grid service you generate stubs to use it, and
recompile. Zhengang, Brent, and I are now working on methods of run-time invocation, but
this is a very tough problem. See:
http://www1.bcs.org.uk/DocsRepository/03700/3795/mukhi.htm

I'm also reading about software engineering practices and how they are
modified when you have a grid services framework. With traditional web
services you usually start with a JAVA interface and some functionality
that you would like, and build the web service from the "bottom-up."
The new OGSA framework, adds some new dimensions to the process of
creating services because you should start with the WSDL since the
Grid-WSDL allows for inheritance. This will be supported in WSDL 1.2 in
a few months. So, this added functionality allows for a more "top-down"
approach to web service creation because you can now leverage existing
service functionality, and have true meta-services that control your
other services.

I'd eventually like to write a paper about
(1) methods for stub-less grid service invocation
(2) software engineering and design with grid services

Also, I'm in the process of submitting a paper on "High-Throughput Web
Services" to the International Conference on Information Technology:
Coding and Computing http://www.itcc.info