PUMA publications for /group/researchcode/bigdataMon Aug 21 13:26:42 CEST 201722nd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing317-321Device-Driven Metadata Management Solutions for Scientific Big Data Use Cases2014bigdata forschungsdaten metadata Big Data applications in science are producing huge amounts of data, which require advanced processing, handling, and analysis capabilities. For the organization of large scale data sets it is essential to annotate these with metadata, index them, and make them easily findable. In this paper we investigate two scientific use cases from biology and photon science, which entail complex situations in regard to data volume, data rates and analysis requirements. The LSDMA project provides an ideal context for this research, combining both innovative R&D on the processing, handling, and analysis level and a wide range of research communities in need of scalable solutions. To facilitate the advancement of data life cycles we present preferred metadata management strategies. In biology the Open Microscopy Environment (OME) and in photon science NeXus/ICAT are presented. We show that these are well suited for the respective data life cycles. To facilitate searching across communities we discuss solutions involving the Open Archive Initiative - Protocol for Metadata Harvesting (OAI-PMH) and Apache Lucene/Solr.Mon Aug 21 13:03:30 CEST 2017Journal of Physics: Conference Series3032047Optimization of data life cycles5132014bigdata forschungsdaten Data play a central role in most fields of science. In recent years, the amount of data from experiment, observation, and simulation has increased rapidly and data complexity has grown. Also, communities and shared storage have become geographically more distributed. Therefore, methods and techniques applied to scientific data need to be revised and partially be replaced, while keeping the community-specific needs in focus. The German Helmholtz Association project "Large Scale Data Management and Analysis" (LSDMA) aims to maximize the efficiency of data life cycles in different research areas, ranging from high energy physics to systems biology. In its five Data Life Cycle Labs (DLCLs), data experts closely collaborate with the communities in joint research and development to optimize the respective data life cycle. In addition, the Data Services Integration Team (DSIT) provides data analysis tools and services which are common to several DLCLs. This paper describes the various activities within LSDMA and focuses on the work performed in the DLCLs.Tue Jul 18 09:41:03 CEST 2017SwitzerlandWhither turbulence and big data in the 21st century?27497-507Turbulence in the Era of Big Data: Recent Experiences with Sharing Large Datasets2017bigdata forschungsdaten obib engineering iag In the context of the contemporary push for “big data” in many fields, we review recent experiences building large databases for turbulence research. We consider data from direct numerical simulations (DNS) of various canonical flows and from experimental studies and related numerical simulations of wall-bounded turbulence, where the data storage needs are particularly challenging due to the very large range of length and time scales that exists in these flows at high Reynolds numbers. The focus is on a move from the traditional approach of data-handling and analysis where datasets are moved to individual computers, to one where much of the analysis is moved to the hosting system that stores these data. In this context we give a summary of a unique open numerical laboratory that archives over 200 Terabytes of DNS data, including full spatio-temporal flow fields of various canonical flows. Particular attention is given to the unique access requirements for large datasets to become open to the research community and the success the system has had in democratizing access to large datasets.Tue Jul 18 09:33:00 CEST 2017SwitzerlandWhither turbulence and big data in the 21st century?28509-515Public Dissemination of Raw Turbulence Data2017bigdata forschungsdaten reuse dissemination simulation obib engineering iag It is argued that there is a certain urgency to the discussion of whether raw data should be made publicly available within the turbulence community, and about the best ways, technology and rules for possible dissemination. Besides expressing the personal opinion that such sharing would be advantageous for the field, the urgency mostly arises from the danger that funding agencies or other institutions would otherwise set standards without proper community input. This paper is in part a plea for community action in that direction. As an example, the experience of the Madrid school of Aeronautics with the dissemination of numerical simulation results is briefly reviewed, including the present technological solutions and usage statistics.