PUMA publications for /group/simtech/ReproducibilitySat Nov 21 14:36:13 CET 2020Science62421422--1425Promoting an open research culture3482015reproducibility Wed Nov 27 15:03:04 CET 20192017 IEEE 13th International Conference on e-Science (e-Science)10488-489Artifact Evaluation: Is It a Real Incentive?2017forschungsdaten software reproducibility diss berlin It is well accepted that we learn hard lessons when implementing and re-evaluating systems, yet it is also acknowledged that science faces a crisis in reproducibility. Experimental computer science is far from immune, although it should be easier for CS than other sciences, given the emphasis on experimental artifacts, such as source code, data sets, workflows, parameters, etc. The data management community pioneered methods at ACM SIGMOD 2007 and 2008 to encourage and incentivize authors to improve their software development and experimental practices. Now, after 10 years, the broader CS community has started to adopt Artifact Evaluation (AE) to review artifacts along with papers. In this paper, we examine how AE has incentivized authors, and whether the process is having a measurable impact. Our answer can help guide CS, and more broadly, other computationally-oriented sciences, in encouraging peer-review of software artifacts and developing additional community practices for incentives.Wed Nov 27 14:57:56 CET 2019Communications of the {ACM}09105--5Incentivizing reproducibility592016forschungsdaten software reproducibility diss berlin Wed Oct 23 10:57:34 CEST 2019International Journal of Digital Curation061263--291Committing to Data Quality Review92014forschungsdaten software reproducibility diss Wed Apr 10 16:09:18 CEST 2019Papers From the 12th Advanced Summer School of Service-Oriented Computing (SummerSoC 2018)Oktober31--42Towards Deployable Research Object Archives Based on TOSCAKonferenz-Beitrag2018projekt TOSCA myown Reusability publist Reproducibility lebenslauf Deployment roar Research In science, reproducibility means that a scientific experiment can be repeated by another scientist with the same result. This is of particular importance to verify the results as well as to show the usefulness and reusability for further research. However, the exclusive publication of the research results in a scientific journal is usually not sufficient. In addition to research results, also research data as well as research software need to be published and made public available in order to enable researcher to gain new insights and thus advance research. However, the reproducibility and reusability of research data and research software typically is hindered by several barriers. Therefore, this work intends to first provide an overview of the current situation and issues in this particular topic and furthermore sketch our vision of standards-based Research Object Archives containing scientific publications, software, data, metadata and licenses in order to tackle the existing problems.Thu May 24 10:44:08 CEST 2018Future Generation Computer Systems354 - 367Reproducibility of execution environments in computational science using Semantics and Clouds672017workflow reproducibility metadaten In the past decades, one of the most common forms of addressing reproducibility in scientific workflow-based computational science has consisted of tracking the provenance of the produced and published results. Such provenance allows inspecting intermediate and final results, improves understanding, and permits replaying a workflow execution. Nevertheless, this approach does not provide any means for capturing and sharing the very valuable knowledge about the experimental equipment of a computational experiment, i.e., the execution environment in which the experiments are conducted. In this work, we propose a novel approach based on semantic vocabularies that describes the execution environment of scientific workflows, so as to conserve it. We define a process for documenting the workflow application and its related management system, as well as their dependencies. Then we apply this approach over three different real workflow applications running in three distinct scenarios, using public, private, and local Cloud platforms. In particular, we study one astronomy workflow and two life science workflows for genomic information analysis. Experimental results show that our approach can reproduce an equivalent execution environment of a predefined virtual machine image on all evaluated computing platforms.Mon Mar 19 14:36:40 CET 2018Collaborative Librarianship2Reproducibility Librarianship 92017forschungsdaten bibliothek reproducibility Fri Feb 23 20:45:57 CET 2018IEEE Data Eng. Bull.454-59A Computational Reproducibility Benchmark.362013reproducibility Fri Feb 23 20:44:32 CET 2018Scientific Programming243180:1-243180:11Towards Reproducibility in Scientific Workflows: An Infrastructure-Based Approach.20152015reproducibility Fri Feb 23 20:43:17 CET 2018Computing in Science and Engineering164-72The Limits of Reproducibility in Numerical Simulation.142012reproducibility Wed Jun 21 17:24:22 CEST 2017ScienceDez63171240–1241Enhancing reproducibility for computational methods3542016forschungsdaten software reproducibility Wed Jun 21 06:11:26 CEST 2017studieRepeatability and Benefaction in Computer Systems Research2015software reproducibility zitieren e describe a study into the extent to which Computer Systems researchers share their code and data and the extent to which such code builds. Starting with 601 papers from ACM conferences and journals, we examine 402 papers whose results were backed by code. For 32.3% of these papers we were able to obtain the code and build it within 30 minutes; for 48.3% of the papers we managed to build the code, but it may have required extra effort; for 54.0% of the papers either we managed to build the code or the authors stated the code would build with reasonable effort. We also propose a novel sharing specification scheme that requires researchers to specify the level of sharing that reviewers and readers can assume from a paperWed May 10 12:01:30 CEST 2017Eos, Transactions American Geophysical Union16163--163The R software environment in reproducible geoscientific research932012re-use forschungsdaten software reproducibility Wed May 10 11:54:59 CEST 2017D-Lib Magazine011/2Opening the Publication Process with Executable Research Compendia232017oa forschungsdaten software reproducibility