An Evaluation of the Cost and Performance of Scientific Workflows on Amazon EC2

Gideon Juve; Ewa Deelman; G. Bruce Berriman; Benjamin P. Berman; Philip Maechling

doi:https://doi.org/10.1007/s10723-012-9207-6

An Evaluation of the Cost and Performance of Scientific Workflows on Amazon EC2

Gideon Juve, Ewa Deelman, G. Bruce Berriman, Benjamin P. Berman, Philip Maechling

Research output: Contribution to journal › Article › peer-review

Abstract

Workflows are used to orchestrate data-intensive applications in many different scientific domains. Workflow applications typically communicate data between processing steps using intermediate files. When tasks are distributed, these files are either transferred from one computational node to another, or accessed through a shared storage system. As a result, the efficient management of data is a key factor in achieving good performance for workflow applications in distributed environments. In this paper we investigate some of the ways in which data can be managed for workflows in the cloud. We ran experiments using three typical workflow applications on Amazon's EC2 cloud computing platform. We discuss the various storage and file systems we used, describe the issues and problems we encountered deploying them on EC2, and analyze the resulting performance and cost of the workflows.

Original language	American English
Pages (from-to)	5-21
Number of pages	17
Journal	Journal of Grid Computing
Volume	10
Issue number	1
DOIs	https://doi.org/10.1007/s10723-012-9207-6
State	Published - Mar 2012
Externally published	Yes

Keywords

Cloud computing
Scientific workflows

All Science Journal Classification (ASJC) codes

Software
Information Systems
Hardware and Architecture
Computer Networks and Communications

Access to Document

https://doi.org/10.1007/s10723-012-9207-6

Cite this

@article{003f9bcdcc534cff8475f1e7a8b5c19e,

title = "An Evaluation of the Cost and Performance of Scientific Workflows on Amazon EC2",

abstract = "Workflows are used to orchestrate data-intensive applications in many different scientific domains. Workflow applications typically communicate data between processing steps using intermediate files. When tasks are distributed, these files are either transferred from one computational node to another, or accessed through a shared storage system. As a result, the efficient management of data is a key factor in achieving good performance for workflow applications in distributed environments. In this paper we investigate some of the ways in which data can be managed for workflows in the cloud. We ran experiments using three typical workflow applications on Amazon's EC2 cloud computing platform. We discuss the various storage and file systems we used, describe the issues and problems we encountered deploying them on EC2, and analyze the resulting performance and cost of the workflows.",

keywords = "Cloud computing, Scientific workflows",

author = "Gideon Juve and Ewa Deelman and Berriman, {G. Bruce} and Berman, {Benjamin P.} and Philip Maechling",

note = "Funding Information: Acknowledgements This work was supported by the National Science Foundation under the IntelData (IIS-0905032) and Pegasus (OCI-0722019) grants. This research made use of Montage, funded by the National Aeronautics and Space Administration{\textquoteright}s Earth Science Technology Office, Computation Technologies Project, under Cooperative Agreement Number NCC5-626 between NASA and the California Institute of Technology.",

year = "2012",

month = mar,

doi = "https://doi.org/10.1007/s10723-012-9207-6",

language = "American English",

volume = "10",

pages = "5--21",

journal = "Journal of Grid Computing",

issn = "1570-7873",

publisher = "Springer Netherlands",

number = "1",

}

TY - JOUR

T1 - An Evaluation of the Cost and Performance of Scientific Workflows on Amazon EC2

AU - Juve, Gideon

AU - Deelman, Ewa

AU - Berriman, G. Bruce

AU - Berman, Benjamin P.

AU - Maechling, Philip

N1 - Funding Information: Acknowledgements This work was supported by the National Science Foundation under the IntelData (IIS-0905032) and Pegasus (OCI-0722019) grants. This research made use of Montage, funded by the National Aeronautics and Space Administration’s Earth Science Technology Office, Computation Technologies Project, under Cooperative Agreement Number NCC5-626 between NASA and the California Institute of Technology.

PY - 2012/3

Y1 - 2012/3

N2 - Workflows are used to orchestrate data-intensive applications in many different scientific domains. Workflow applications typically communicate data between processing steps using intermediate files. When tasks are distributed, these files are either transferred from one computational node to another, or accessed through a shared storage system. As a result, the efficient management of data is a key factor in achieving good performance for workflow applications in distributed environments. In this paper we investigate some of the ways in which data can be managed for workflows in the cloud. We ran experiments using three typical workflow applications on Amazon's EC2 cloud computing platform. We discuss the various storage and file systems we used, describe the issues and problems we encountered deploying them on EC2, and analyze the resulting performance and cost of the workflows.

AB - Workflows are used to orchestrate data-intensive applications in many different scientific domains. Workflow applications typically communicate data between processing steps using intermediate files. When tasks are distributed, these files are either transferred from one computational node to another, or accessed through a shared storage system. As a result, the efficient management of data is a key factor in achieving good performance for workflow applications in distributed environments. In this paper we investigate some of the ways in which data can be managed for workflows in the cloud. We ran experiments using three typical workflow applications on Amazon's EC2 cloud computing platform. We discuss the various storage and file systems we used, describe the issues and problems we encountered deploying them on EC2, and analyze the resulting performance and cost of the workflows.

KW - Cloud computing

KW - Scientific workflows

UR - http://www.scopus.com/inward/record.url?scp=84859741475&partnerID=8YFLogxK

U2 - https://doi.org/10.1007/s10723-012-9207-6

DO - https://doi.org/10.1007/s10723-012-9207-6

M3 - Article

SN - 1570-7873

VL - 10

SP - 5

EP - 21

JO - Journal of Grid Computing

JF - Journal of Grid Computing

IS - 1

ER -

An Evaluation of the Cost and Performance of Scientific Workflows on Amazon EC2

Abstract

Keywords

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this