OGSA Teleconference - December 15, 2003 ====================================== * Participants Andrew Grimshaw (UoV) Dave Snelling (Fujitsu) Frank Sievenlist (ANL) Chris Smith (Platform) Bill Horn (IBM) Hiro Kishimoto (Fujitsu) Fred Maciel (Hitachi) Ravi Subramaniam (Intel) Jem Treadwell (HP) Ali Anjomshoaa (EPCC) Minutes: Hiro Kishimoto 2) Program Execution Discussion (100min) 2-1) Define "job" and "job service" Andrew: primary topic what exactly is the "job service" two things discussed so far (A) instances created when job submitted, even before resources are allocated. This is not the job itself, but an overseer (B) a service much more tightly bound to application to control the job, including application interfaces. Chris (A) is job proxy than job manager like that one. If we go (A) how to deal with (B)? (B) is necessary especially for legacy one. (Andrew) Ravi: we should call (B) as "job" (B) encapsulates application as manageable entity Ali: I agree with Chris, I explain job services at GGF9 "job service" manages job job I/O kind of thing is application services not job service Ravi: we will separate "job" and "job service" "job service" is not executable. Andreas send out slide in the F2F minutes. Andrew: Do we agree "job service" definition is (A)? Ravi: we should separate job and job service Chris: everything is "service" in OGSA world. Ravi: Job is a document "job manager" manages job (as document) and "job service" in case of "job cancel" the manager acts Is there hierarchy issue? No real discussion on this question. Ali: "job service" is document, "job manager" interact "job service" 1:n mapping. "job service" shows the state of the job. we also need "application service" job might depend on (multiple) "application services" i.e. workflow. Ravi: want to separate into three parts - job (as document) job is an entity there is factory job document comes first - job service - job manager Andrew: we need some use cases. like platform has several legacy jobs like BLASTs Chris: job manager is a client in this case Ravi: job manager has multiple roles. there is JSDL document Ali: JSDL document is a part of "job document" Andrew: someone constructs "job document" Ravi: "job service" should have a handle and can be resolved to "job manager" Ali: job document is a representation of "job service" and can be read from all related services around. Andrew: "job service" have several status but does not mean has handle. Ravi: "job service" has multiple handles. single handle is very inefficient Dave: it is implementation issue Ravi: handle may not have service Dave: Yes, handle should have service but may not have physical entity there is service if you have handle. it could be slow but it is just an implementation issue. Ravi: concept here is that handle has document at the end. Andrew: Steve says service is super light weight and you can have zillions of services. Ali: what is exactly job document, memory? disk? Chris: this is a concept. Ali: we don't need additional concept, just say we can get this "document" from "job service." Andrew will make a decision. (A) contains the "job document" it begins to exist when create entity one to one mapping to "job" "job" may be workflow (multiple job step) (B) contains the actual job it's created when the job starts. Ravi: job has multiple personality. Shall we call "job service" as "task"? Call (A) as "job" Call (B) as "task" and is the actual job consumes cpu, resources. Andrew: this is identical picture what I draw Ravi: I just rename things. Chris: after initiating job, i still need to interact the job. Andrew: we only have/expose one grid handle and use this handle through all stages but implementation is free to deal with maybe multiple handles Ravi: if we manage this entity thought? Ali: I understand Chris: it is container Ravi: you have job and container which have jobs inside. Ali: is container (application service) have just one job or may have multiple jobs? Container may have multiple jobs. (A) job manager: has GSH and "job document" created when client want to run job (B) job wrapper: correspond to application (like BLAST) created when job started. may bind to specific container job wrapper may or may not expose through job manager if job wrapper fails, job manager may take care and recreate new job wrapper for recovery. Ali: I agree Andrew and like this picture. Andrew: Name? Dave: (A) should be "job" Ravi: (A) may have higher level i/f also (like restart) if we mix single step job and workflow... Ravi: we have two level structure Chris: we should have one more bubble for workflow or the job manager orchestrates workflow Hiro: job manager should be job factory not workflow engine workflow engine is different service(manager) Ravi: I agree with Hiro on this aspect. Jem: what is the difference between job wrapper and container? Andrew: Container is hosting/execution environment corresponding to workstation or cluster job wrapper is a interface to running job and application specific Ravi: container have some aspect of job job and job wrapper is one to one. job wrapper and container may be multiple to one. Ali: can you contact job through container? Ravi: you can contact through job wrapper not container. Container does not have wide variety of interface Andrew: container should have another functionality like deployment... but this discussion is not for today. Ravi: container is a factory of "job" or "job wrapper"? Andrew: neither Andrew: take 5min break and create word document to explain 5 bubbles on whiteboard - job wrapper: A job wrapper is a grid service to an application. A job wrapper will have a standard set of portTypes. Two different examples may highlights this. First, consider a legacy code like blast. the wrapper may allow TTAY redirection, reading and writing of local files, etc. A second example may be a wrapper for a conferencing application - it may have a portType to allow creating a new white board. Note thought that application specific interfaces, e.g., create new white board,... - container: - job document: a job document describes the set of the job e.g. the agreements that have been acquired, the JSDL, how many times it has been started. By state we do not mean for example the internal memory of a blast job. the job document is encapsulated by a job and exposed as service date of the job. - job: A job is a grid service (named by a distinct GSH) and is created at the instant requested ... even through no resources have been committed, the job does not necessarily "run" on/in the container where the actual "job" is run, whether the "job" is a legacy or not. A job encapsulates a job document. A job is not a workflow. A job grid service may be instantiated by a job factory. A single physical job service may handle many distinct job documents i.e. there may be many jobs, each with a distinct GSH, multiplexed to a single physical service. - job manager: the manages jobs. for example it may be a workflow manager, it may be a portal that interacts with users, may deal with failures and restarts, it may schedule them to resources, it may collected agreements and reservations. It is responsible or making the job - or set of jobs do the voodoo that they do so well. It is very likely to be a subtype of the WSDM collection, which is a collection of manageable entities. A WSDM collection can expose as its methods some of the methods exposed by the members of its collection. There may be a stop, kill, nuke, nuked till they glow and let god sort them out. - flow manager: Ravi: still uncomfortable with the name "Wrapper." How about "application proxy"? Ravi: I think inward focus, not outward focus. Dave: do you want to use "task"? Ravi: let's go back to "Job Wrapper" Ravi: Who will handle checkpoiting? Job wrapper or container. About February F3F meeting. Andrew: San Diego Super Computer Center confirm our F2F meeting room. And also hotels are ready to reservation. Will send out hotel information to the list.