OGSA F2F minutes ================ Date: July 29-31, 2003 Place: Argonne National Lab. Building 221 room C101 Meeting materials: https://forge.gridforum.org/docman2/ViewCategory.php?group_id=42&category_id=379 * Participants - Dave Berry, UK National Science Centre, Edinburgh - Ravi Subramaniam, Intel (29&30) - Shel Finkelstein, SUN (30&31) - Jeffrin J. Von Reich, HP - Hiro Kishimoto, Fujitsu - Ian Foster, ANL - Frank Siebenlist, ANL - Jay Unger, IBM - Takuya Mori, NEC - Andreas Savva, Fujitsu - Fred Maciel, Hitachi - Ming Xu, Platform Minutes: Fred(29) & Andreas(30,31 & merge) Note: Actions marked with '#' Note: Not in chronological order. * Early discussion ** Last Teleconf's minutes - By approving the minutes are we approving the conclusions? - We are mainly approving that they are an accurate record of what happened. - If there are any decisions we are also approving them. - (But GGF policy seems to be that final decisions are only possible on the mailing list.) - (Andreas) The only decision in these minutes was for Jeffrin to become the editor of the next version of the Use Case document. The rest is discussion. - (People want to raise issues with some of the content; the agenda already includes time for such discussion.) # Minutes approved. # (Ravi) Make it easy to see what has been decided. E.g., Arrange # for the last 10-15 minutes on the call for a review of actions, # decisions. ** OGSA-WG co-chairs - (Fred) Has Hiro officially become co-chair of the WG? - (Ian) Yes; the GFSG has reached consensus on the issue, will announce officially later. # (Ian) Hiro can go ahead and edit the ogsa-wg gridforge page. ** GridForge usage: - (Jay) Can we hang off discussions from a document, rather than just have them separately on the mailing list? - (Lotus Notes can do it.) - It wouldn't be nice if you had to be online to access threads - People used to receiving (and dealing with) lots of email like to download & process things offline. - Benefits: - Easier to catchup to discussions. Don't have to wade through the mailing list. - Might be able to reduce email volume (or at least size of emails) - (Hiro) Some of this is already possible/happening. Can upload files to ogsa-wg gridforge site and create trackers for them. # Hiro to check(change settings so) that group members can # upload documents without requiring approval. Get ogsi-wg # settings from Dave Snelling. (> done) - (Fred, Andreas) There was some talk about being able to integrate the mailing list with the tracker in the next (current?) version of gridforge. # Look into integration of tracker and mailing list. >Hiro? - Ultimately this stuff, and other instructions, should become "best practices" on how to use GridForge # Fred volunteers to help with this. * Agenda bashing & general discussion ** Taxonomy - Why does Jay's taxonomy have just Core, Data & Program execution? - (Jay) All(90%?) grids are either computation, data, or both. Hence the "inverted T" figure. - Taxonomy as "inverted T" - So as not to imply that everything is through the top level (program/data) layers only. - Figures - (Ravi) A layer figure is not enough. - (Jay) Tried a number of different figures. Using scattered and related boxes to explain to people what the OGSA Platform is doesn't work, so a figure with the concept of layers is needed. It may not be completely right, but it is easy for people to relate to. - Taxonomy & hierarchy - (Shel) There was also a (>1?) service taxonomy presented at WSDM (HP, IBM) in WSDM. We might be able to use them. # Shel sent references to ogsa-wg list already. # Agreed: # - Taxonomy & hierarchy are not exclusive. # - We need more than one picture. ** OGSA general *** What is OGSA? (Note: A very long discussion that amounted to clarification, with consensus.) - (Jay) OGSA is the interfaces and dependencies between them. Need to take care with optional features, so that people will not implement features internally (duplication, etc), which will reduce usefulness of the framework. For interoperability to be possible we need portTypes, bindings, service data schema, semantics on portTypes. - (Ravi) Describing boundaries is restrictive. Some company may want to bundle many functionalities in a single component. - (Jay) Try to implement(define) the minimum (discrete units), and services say which compositions are possible. # Add text on this subject to the platform document. >who? *** OGSA-compliance - (Ravi) What does "OGSA compliant" mean? - (Jay) For now, "OGSI compliant" is the only well defined term. Perhaps nobody will implement the whole of OGSA, so compliance by "parts" is needed (i.e., product says which parts of the spec it is compliant with). - (Jeffrin) Add a section that explains OGSA compliance. - (Fred) "compliance" is too strong. It implies, for instance, testing by a third party. - (Jeffrin) We should say what we mean, e.g., that OGSA compliance for now does not mean such testing. *** Resources (Note: Long discussion on this subject.) - (Ravi) OGSA has to clarify how to deal with resources (that's the main difference between Web Services and Grids). CMM is part of it, but not everything. ** Infrastructure (or the "blob" between ogsi & services) - (See also 'refactoring' email from Dave, Hiro, Andreas.) - (Ravi) OGSI and CMM are more than infrastructure. - (Ravi) Infrastructure can be different things to different people. OGSA itself can be thought of as infrastructure. The term is misleading/overloaded. - (OGSI already uses it; why not use it by extension....) - (Jay) OGSI has value beyond grid; ideally should disappear into WS space. OGSA is a different story. # Decide on a better name later (plumbing, core plumbing, # infrastructure,...). - Maybe grouping/orchestration is low level enough to be at ogsi level. - (Andreas) ServiceDomain and orchestration was put at this level in the 'refactoring' email. ServiceDomain is more like a pattern. (Part of) orchestration was put here intuitively. Other part was placed higher up. - Some discussion on the difference between orchestration/composition/workflow. Jay offered a definition & explained difference. # Put definition into our glossary. >Glossary-owner. - Also we should address - how services build on top of other services - environment settings - context (shared between services), e.g., VO context (See also *** Factoring ("Expected Output" slide 7-11)) ** Discussion focused on Hiro's "Expected Output" presentation *** Autonomic category ("Expected Output" slide 5) - (Jay) Autonomic does not work as a box, it is a philosophy that's present at all levels. - "Autonomic" category contains "fault handling" and "problem determination". These look more appropriately placed into "Services Management." # Agreed to move these to "Services Management" and delete # autonomic. *** Resource Management category ("Expected Output" slide 5) - Standardize provisioning? - (Ming) Yes, there is demand. - (Jay) Provisioning is ultimately service provisioning (factory), hierarchical (cascading to services to which it depends), which ultimately provides networking, storage, etc (as services). - Discussion on whether the lowest level (bootstrapping) has to be defined or not. - (Jay) Agree that a basic set of factories is needed; suggests for other people to write a paper on bootstraping OGSI/OGSA. Also, for platform as a whole: will have to find the dependence graph, and get to "well known" (or implementation-dependent) portTypes. - (Jay) What is resource management beyond provisioning? - (Hiro) Scheduling and resource brokering. - Discussion on what a scheduler and a broker are. - (Jay) workload manager [broker for Hiro] virtualizes resources (hardware, licenses, etc.), which are reserved by the scheduler. - (Ravi, Ming) Process can be even more detailed (3 or more steps, say temporal mapping then reservation). *** VO ("Expected Output" slide 5) (Note: Long discussion on meaning of VO.) - (Frank) Where is Virtual Organization(VO) control in this picture? - Discussion on whether VO is a service or scope (for policy, users, services, etc). # Conclusion (?): need VO management as a service, to pull # information (policies, etc.), to provide VO context in requests, # etc. # Conclusion: meaning of VO is not clear enough. # (Jay) suggested Frank (and others) to investigate relationship # between VOs, relationship providers, etc., to serve as basis for # this discussion. # Jay later did a draft for section 5 of the platform that # seemed to capture the basic meaning of VO. # - "VO is context to associate resources & members and requests..." # - First part looks like function explanation. *** Factoring ("Expected Output" slide 7-11) - Maybe we should talk about DAIS-WG and not OGSA-DAIS. OGSA-DAIS refers to one implementation of the work done in DAIS-WG. - uses/extends relationship - (Andreas) The comment on "extends/uses" in the presentation is meant to be explanation to the last teleconference. The refactoring email uses "extends" when it should (sometimes) say "uses." Don't read it as a general statement. - (Ming et al) Maybe we should focus on "uses" initially. - "extends" may be more meaningful for implementations(?). - (Ravi, Andreas) "extends" here mainly refers to port type extension. - functionality extension may be a part of it. - (Ming, Andreas) In any case it is too much to put both use/extends concepts in one figure. # Agreed that multiple figures to separate concepts (e.g., one # for uses, one for extends, more...) are a good idea. *** TOC ("Expected Output" slide 12) - Focus mainly on sections 3 & 4 - Discussion on what we want to show and expected audience (In jest: "if you are reading it you are the audience") - We are expressing the same contract but provider(implementor) and consumer will still have different views - OGSA user is domain services, or end user. - Difference between service and specification of (model). - All stuff that extend web services framework (ogsi+?; don't say "infrastructure") - infrastructure services vs infrastructure model/specification interfaces, schemas - Sec.3 - layer of functions Include stuff like: - taxonomy; different view for user's/provider's(implementator's) - high level view - Sec.4 - what should the name of this be? hierarchy is not right? - WS extensions - Don't talk about services - The gory details - Glossary # We should also do a glossary so as to gather common # terminology. First candidates: interface, service, brokering, # discovery. > Glossary-owner? Brokering in particular raised an extended terminological discussion on 2 of the 3 days. E.g., - What is it (definition) - Is it a function or a service (or both) - (Andreas) Use the GGF scheduling dictionary definition as starting point rather than work from zero. - This helped a bit, but people did not seem particularly happy. ** Security - (Frank) Do not think of it as an add-on; bake it in. It is essential. - There is nothing specifically "Grid" about security. It should end up just being plumbing. We use what exists. - Make it "on" by default. - Make sure that there is a choice to bypass/turn it off, if a particular environment does not need it (conscious choice; implies that you know what you are doing). - logging vs secure logging. ** Agreements [On a tangent: "Everything addressable is a service" - (Jay) This is the Frey doctrine. The motivation is to avoid re-inventing the same stuff over and over again (e.g., stuff already in ogsi). Also the assumption is that services can be lightweight and that there is no scalability problem with having lots of them.] - (Andreas) So are policies or agreements addressable? (Are they services?) - (Jay) Yes & no. Depends on what you are doing. e.g., are we talking about a document or not. - Are there any base terms required by all agreements? ... - Define schemas: WS-security stuff descibes some terms for negotiation - Terms of negotiation: probably as appendices to the GRAAP document. - Negotiation is in band while enforcement/monitoring of agreed terms is out of band. - WS-Policy: (Shel) IP status is still unclear # Agreed to try putting policy and agreements in one section in the # Platform. ** Side discussion on data services - Replication can be thought of as a mechanism to fulfill a certain performance agreement. - Relationship of data services to data movement (replication) - Also added some Data functionality in "Revised Function List". * Functions ** Functions list Worked from the function list appearing in the Platform(& Matrix). Re-organized the list(Andreas) and added functionality (e.g, Security(Frank), Data(DaveB)) that people expect, but wasn't explicitly referred to by use cases; maybe because it wasn't in the platform in the first place. - See "Revised Function List" section further down. Note that it is still only partly worked through. - We should try to avoid using the same terms in functions and mechanisms/services; it leads to confusion. # General agreement that "Use Cases" don't drive the specification # but are used to validate it. - Functions (requirements) vs mechanisms - Example problem function --- Discovery: - What do we want to discover, and how (mechanisms) - Can define single interface, but maybe have different functionality/semantics. e.g., define class of registries, and what you do with them. - ogsi definition (service group) or service collection - registry as shorthand for mechanism to do discovery - registry as mechanism to advertise - discovery indicates a wider functionality, while registry talks about a certain specific way to do it. ** How to go from functions (requirements) to services - What detail do we expect in descriptions - factor or identify what uses what - high level requirements - don't step on anyone's toes by doing architecture in detail. - Service Template - rename to function description? - a function may map to multiple services - talk about what are the services later * Platform document ("The Next Generation") ** Purpose - (Jay) Ultimately the platform document is like a TOC; it is the entry point leading to more detailed documents. - (Jeffrin) The document should include its purpose. - Platform specifies "what" not "how" (realization) - data (service data) - ?protocols? - ... - port types -> service description template - Platform audience - (Jay) Two use classes: spec (for implementers) and guidelines. - Focus on functionality; we have certain services that we agree on. We should go on and assign people to do them. # The assignments list is under "Platform Assignments" further down. ** Service template & Sec.5 reorganization # Agreed that the level of detail suggested in Hiro's template seems # right. # Agreed to use the template to refactor section 5. - Don't delete anything at this stage. Just move things around to expose gaps. - Assign owners. # Level of detail (in general) - Requirements level only - Don't go into architecture of the "function" - "function" architecture, if we do it, is suggestion. - Be prescriptive of function (boundaries, dependencies) but don't go into port types - Be careful of relation with existing groups. # (Hiro, Fred, Andreas) did an initial draft (Wed night). # Based on this draft Jay did another revision (Thur morning). ** Review by group of resulting document - Did not delete anything yet; everything that was there is still there, but place might have moved. - Identified overlap of "administrative" section and "service mgmt" - probably delete "administrative" if it is a complete overlap - Or move "Service Mgmt" a level up; as separate category? - "Policy" includes "agreement." - "Agreement" includes reservation. - Move "Transactions" under "Service interactions" - Move "Provisioning" down to be near "Installation & Deployment" - Probably combine "Provisioning" with "Installation & Deployment." Provisioning is more general. - Too many core services? - We'll prioritize services, so it is not a big issue; not everything has to be included in all solutions. In other words not everything is fundamental. - In this sense, "Core" is not the right word. Suggestions for an alternative term to "core" - Foundation - Nothing. Promote everything instead. (DaveB) - How to show that we need some of the things from each (taxonomy) box some of the time? - What you need depends on purpose: what you want to build. - Need to define what is a set of services; eg. if you do this kind of "thing" you have to have "X, Y, Z" services. - Solution dependency graph - Also think of performance and other requirements that have to be fulfilled. - (Back to profiles.) ** Prioritization of services *** Service granularity Services (or what passed for them at the moment) are very granular. When we decompose them, we also should prioritize the sub-services. E.g., security, contains authorization. (And authorization may still be too granular.) *** Priority Suggestion to use the following tags: required, recommended, optional, deferred. - Should we do dependency first? *** Profiles - (Ming) What do we mean by profile? - types/class of grid: storage, computation, or - operational aspect: e.g. management grid, user grid - encompasses different types/classes of grid # Agreed that by profile we mean the first variety (type/class). - How to do Profiles? - Look at 2-3 use cases and pick out the main items. *** UK eScience categorization Input from DaveB; see presentation for more details. # DaveB to send pointer to this presentation. (>uploaded to gridforge) - User needs for OGSA grid - Absolutely minimal: simple registry (e.g., serviceGroups), Data Access/ movement (RFT), job submission, access control (assumes a basic level of AA) - Key services (application writer's viewpoint): authorization, notification, workflow, dependency management (deployment), (UDDI) registry (or equivalent) - Types: info/portals/compute grids - Info: DQP, Cache/Replication - Compute: scheduling, dynamic deployment - Portals: session mgmt, state mgmt - Later: mining, reservation, etc - Infrastructure: deployment (versioning, etc), registry (service factory, data set), user mgmt, monitoring *** Production grid systems - (Jeffrin) Best examples of fully realized grid? - (Ian) Fully realized in what sense? - Some examples (missed some of them): UK CMS, NEES grid, etc. - NEES grid uses GT3 (Ian showed us a live view). * CMM - Explanation of what is CMM (Fred) - Some discussion on lifecycle - CMM allows you to define your own lifecycle model to capture specific requirements; e.g., a lifecycle for job submission and management. - WSDM relation: - Need cooperation to avoid overlapping work in different standards bodies. - Comments that an alliance between the GGF and OASIS (like the one between GGF and DMTF) might be tricky due to differences in confidentiality and membership of both bodies. * Related activities in other bodies (WSMF, WSDM) - WSDM is related to CMM - Shel, Ellen attended - Shel gave us a quick overview. - CMM is lower level than WSDM - We have to decide how we are related and how to collaborate. - WSDM management specification stack (shel) - similarities with ogsa - there was also talk of ogsi as lowe(st/r) layer - WSMF HP presentations (components) - what is needed to manage services - identified some items (7,8) that are needed for mgmt - build a stack : - how to use/integrate legacy tools; - compliant with other mgmt standards - lots of other people are doing work in this area; need to pay special attention. # Shel sent to the list some material from WSMF, WSDM. # Fred also sent to the list a pointer to the WSMF specs. * Use cases (Jeffrin) ** Document direction - Avoid UC proliferation. Make sure we have a good set that covers a wide area, with little overlap - UC based on existing stuff or having clear customers are ok. - "Imaginary" ones, without a clear customer have a higher burden of proof. We don't reject them unconditionally but they have to prove that they are not contrived. - We won't separate UC vertically. But we do want to have people working in other domains be able to relate to some of these. - Show dependencies between UC where possible. - The UC document is not an educational/marketing document. - The idea of doing a separate marketing (market awareness) document was raised. # Hiro will provide Jeffrin an up to date UC list. (Note: Everything # submitted is on the list.) ** New UC Some new use cases have been submitted recently: GRAAP, RealityGrid, AuthZ, .... We did not go into details of these UC. Instead Jeffrin went through some of the HP proposed UC. *** Intergrid UC - Data centers have a mix of grid and non-grid resources - Customers won't be named but they are real - Motivator: Sharing resources with suppliers/collaborators - Collaboration cycles - Want to be able to specify that certain jobs will not share machines with other specific groups/companies. - Service grid functionality: on-demand, etc; provide a service # Get list of applications that are already grid enabled from the # manufacturing industry. (>Ming) *** Financial data center UC - Migration from non-grid to grid environment while providing continuous service. - Need to define terminology - Manual process (deploy 7 days, etc) - Requirement that automating process must not take longer. *** Maintenance UC - (Jeffrin) This UC could be included in existing "Commercial Grid Center" UC. - (Hiro) Prefer to keep separate. # Decided to keep it separate and make the Grid Center one refer # to it. *** Interactive grid UC - (Hiro) Not clear on how it is different from existing Online Media UC. (Online Media also includes interactive component, e.g., game playing.) - (Jay) Game playing requires fairly loose consistency. Other collaboration scenarios may require stricter consistency. Maybe this UC could be used to expose different consistency requirements? # Rewrite to expose differences relative on Online Media UC and # review again. >Jeffrin *** Grid lite UC - Using small devices (pdas, phones, layer 2 devices etc) - It is about virtualization, not sharing cpu cycles. - (Shel) Includes a number of different things - small device - mobile - occasionally connected - offline & online & sync *** Global grid UC - GESA relation, etc ** Other UC ideas - Scheduling(parallel) Model: Partitioning a program to ensure good performance needs information on what resources the program will receive for execution. It implies a certain division of scheduling functionality, that may be visible in the Platform document, under Program Execution (Jay, Ming). - (Andreas) Might make a good use case; perhaps also have some explanatory text in the platform document to motivate the program execution discussion. - Portals Use Case (DaveB) - online credential repository - state/session mgmt * F2F wrap up # Next F2F will be held Sep 3-4 (Wed, Thu) # **Morning to Evening** (rather than noon to noon). # Decided by majority voting. - For next F2F - Platform Document - Work on Sec.5 assignments - Check for holes in first teleconf after this F2F - (Fred, Andreas) Where to put CMM, and other (don't say "infrastructure") stuff is still not addressed. - Review first draft 2 weeks after this F2F on teleconf - Produce reasonably stable draft of Sec.5 by the teleconf before the next F2F. - Use cases - For GGF9 - Plan to have solid draft of Platform to show people. - Identify 1-2 people from potentially related wg and arrange a discussion with them. - Plan to talk to something like 10 groups, so 3 sessions may have to be dedicated to this. - If no WG exists we have to decide next step, e.g., BoF? - Aim: Come to an agreement on their relation with OGSA and 'their' portion of the Platform document. # Agreed to chop up each OGSA session in 2,3 segments (e.g., 30 # min each) and have a chat with specific groups during that # time. Groups to be decided later. - Initially focus on relations with groups in GGF. - We also have to address other groups, e.g., OASIS, DMTF. * Platform Assignments ====================== Overall editing - Hiro Services -------- Handle resolution - Frank Policy & Agreements - Jay VO - Jay Security - Frank Discovery - Ian Data - Ian Program Execution - Ming ServiceDomain, -\ Service composition,--\ - Shel orchestration, --/ workflow -/ note: Jay et al to forward info & offer assistance. Metering & Accounting - Andreas Installation & deployment - (Jeffrin) Fault detection/management - Jeffrin Problem determination - (Jeffrin) Logging - Hiro? >Andreas messaging & queueing - Shel two aspects: as service and as binding topic based notification ?routing: WS-routing + security implications (routing through intermediaries) > +Frank? events - (Jay) Note: "(PERSON)" means that the person assigned has taken on the job of finding someone to do it. * Revised Function List ======================= (Note that this was based on a section of the OGSA Platform. It is still incomplete.) ** Core *** ??? - Discovery - Brokering - Virtual organizations - Notification/Messaging - Monitoring - Metering - Accounting - Logging - relation with secure logging - Instantiate new services (includes provisioning) - Grouping / Aggregation of Services --- based on policy and functional requirements *** Policy *** Data - management - sharing - access - movement - replication - consistency - provenance - indexing - discovery - federation *** Security - Multiple security infrastructures & mechanisms - Discover & negotiate security policy - Trust relationships - Perimeter security solutions - Application and Network-level Firewalls - Authentication - Authorization - Identification - Integrity - Signatures - Confidentiality - Delegation - Data Privacy - Encryption - communication payload - data in filesystems/storage - Auditing - Secure logging - Certification - trust - correct semantic behaviour ** Resource management - Provisioning - Resource virtualization - Optimization of resource usage while meeting cost targets - CPU scavenging - Transport management - Usage models (batch or interactive) - Agreement-based interaction (e.g., negotiate Service level agreements) - Extended Service Level Agreements (incorporate cost) - Advanced Reservation - Management and monitoring - Service Level Management - Scheduling - Workload Management - Load balancing - Workflow management ** System properties - Fault tolerance - Disaster Recovery - Self-healing capabilities - Strong monitoring - Legacy application management - Administration