Fig 2 - uploaded by Diego Scardaci
Content may be subject to copyright.
Source publication
In this chapter, we describe a successful methodology to support e-Science applications on e-Infrastructures put in practice
in the EELA-2 project co-funded by the European Commission and involving European and Latin American countries. The heterogeneous
requirements of the e-Science applications, coming from several scientific fields, makes diffi...
Similar publications
In a distributed Grid environment with ambitious service demands the job submission and management interfaces provide functionality of major importance. Emerging e-Science and Grid infrastructures such as EGEE and DEISA rely on highly available services that are capable of managing scientific jobs. It is the adoption of emerging open standard inter...
Citations
... Before launching the experiment, the platform has to take decisions on file replication, finding a trade-off between distribution and availability. This use-case applies to various science gateways such as the Virtual Imaging Platform [2] and other similar science gateways345 deployed on production grids. ...
In spite of numerous works studying file replication strategies on distributed systems, data management policies remain mostly handled by manual operators or very basic algorithms on production grids. Among other causes, this situation is due to the lack of models taking job reliability into account. In this paper, we study file replication using new metrics to evaluate the reliability of distributed storage configurations. A stateful storage availability model is introduced to cope with the inability of the stateless model to account for the commonsense intuition that limiting the number of storage hosts involved in the execution of an application improves reliability. We describe the job success probability and the brittleness entropy, a metric describing the uncertainty of the job failure rate associated to a storage configuration. Results, obtained on synthetic data and on traces extracted from the European Grid Infrastructure, show that the stateful model is more accurate than the stateless on real data, and that it can describe the consequences of limiting the number of storage hosts on application reliability. These findings open the door to the design of new file replication strategies taking storage availability into account. Copyright © 2014 John Wiley & Sons, Ltd.
... Software-as-a-service (SaaS) platforms deployed on production grids, for instance the Virtual Imaging Platform (VIP [1]) and other science gateways [2,3,4] , usually have no a-priori model of the execution time of their applications because (i) task costs depend on input data with no explicit model, and (ii) characteristics of the available resources, in particular network and RAM, depend on background load. Modeling application execution time in these conditions requires cumbersome experiments which cannot be conducted for every new application in the platform. ...
Controlling the granularity of workflow activities executed on widely distributed computing platforms such as grids is required to reduce the impact of task queuing and data transfer time. Most existing granularity control approaches assume extensive knowledge about the applications and resources (e.g. task duration on each resource), and that both the workload and available resources do not change over time. We propose a granularity control algorithm for platforms where such clairvoyant and offline conditions are not realistic. Our method groups tasks when the fineness degree of the application, which takes into account the ratio of shared data and the queuing/round-trip time ratio, becomes higher than a threshold determined from execution traces. The algorithm also de-groups task groups when new resources arrive. The application's behavior is constantly monitored so that the characteristics useful for the optimization are progressively discovered. Experimental results, obtained with 3 workflow activities deployed on the European Grid Infrastructure, show that (i) the grouping process yields speed-ups of about 2.5 when the amount of available resources is constant and that (ii) the use of de-grouping yields speed-ups of 2 when resources progressively appear.
... Freesurfer and the Functional MRI of the Brain Software Library (FSL 3 ). Some scientific gateways such as EUMEDGrid 4 , DECIDE 5 , and GISELA [32] are built from a set of reusable portlets providing basic functions such as authentication and data transfer. Compared with the reviewed alternatives, VIP is distinguished by (i) an interface dedicated to medical image simulation , including simulators of 4 image modalities and a repository to store physical and biological models, (ii) a workflow-based methodology that allows fast porting of new simulators with minor adaptations, (iii) a web interface that totally relieves users from resource management decisions. ...
This paper presents the Virtual Imaging Platform (VIP), a platform accessible at http://vip.creatis.insa-lyon.fr to facilitate the sharing of object models and medical image simulators, and to provide access to distributed computing and storage resources. A complete overview is presented, describing the ontologies designed to share models in a common repository, the workflow template used to integrate simulators, and the tools and strategies used to exploit computing and storage resources. Simulation results obtained in 4 image modalities and with different models show that VIP is versatile and robust enough to support large simulations. The platform currently has 200 registered users who consumed 33 years of CPU time in 2011.
Fairly allocating distributed computing resources among workflow executions is critical to multi-user platforms. However, this problem remains mostly studied in clairvoyant and offline conditions, where task durations on resources are known, or the workload and available resources do not vary along time. We consider a non-clairvoyant, online fairness problem where the platform workload, task costs and resource characteristics are unknown and not stationary. We propose a fairness control loop which assigns task priorities based on the fraction of pending work in the workflows. Workflow characteristics and performance on the target resources are estimated progressively, as information becomes available during the execution. Our method is implemented and evaluated on 4 different applications executed in production conditions on the European Grid Infrastructure. Results show that our technique reduces slowdown variability by 3 to 7 compared to first-come-first-served.
