High Throughput Environmental Assessment Pipeline

HiTEA (The High Throughput Environmental Assessment Pipeline) is a HPC (High Performance Computing) focused software project that combines emerging digital container technologies, in memory column storage (Apache Arrow) and interactive (Jupyter) notebooks to build the standard tool for quantitative sustainability research for conducting high throughput environmental assessments. With HiTEA we aim to enable researchers to routinely conduct sophisticated (HPC-dependent) solution-space estimates rather than single?point estimates of results and policy options in Sustainable Development Goals (SDG) relevant analysis. In its core, HiTEA consists of a data pipeline which accepts data from various quantitative sustainability accounting frameworks. Several already existing parsers will be bundled in HiTEA to provide a generic entry point into the HiTEA work flow. After passing the data through the distributed/cloud-based calculation engine the result data will be gathered in a designated module which will allow to export the results in various formats. HiTEA will run on various e-infrastructure systems ranging from local multi-core servers, to HPC infrastructure at NTNU (OpenStack), Sigma2 (Nird Toolbox) and to commercial Kubernetes based cloud services like Amazon AWS and MS Azure. After reaching a mature software product, we aim to include HiTEA into the service catalogue of the EOSCHub. The focus of the first year was to investigate the IT stack which will be used in the project and develop a first prototype of the pipeline. As part of this efforts we developed and published an Open Source software module for Environmentally Extended Multi-Regional Input Output Analysis, Pymrio. The software is available on Github and the article was published in the Journal of Open Research Software. In the next year, this software will be extended to run on the various cloud- and HPC infrastructure as outlined above. Further development of this software will be driven by case studies, providing the feedback needed to ensure software which thoroughly meets scientific user requirements. To do so, the project adopts industry standard Agile Software Development practices for the use in scientific software development. These case studies will be conducted by the PhD student financed through the HiTEA project. To select the student we posted job announcements in February 2021 on jobbnorge, various social networks (Twitter, LinkedIn) and on specific job-boards related to the PhD topic (forum of the International Society for Industrial Ecology, is4ie.org, and on Woman in HPC organization, womaninhpc.org). In August 2021 we hired Mohamed Badr (https://www.ntnu.edu/employees/mhelsaye), an highly qualified researcher combining knowledge on globalization issues with programming experience. In the next months our efforts will focus on running the first uncertainty analysis for the case studies with the developed prototype. Further information about the project, including links to the code repository and publications, can be found at https://hitea.iedl.no/ .

HiTEA combines emerging digital container technologies, in memory column storage and interactive notebooks to build THE standard tool for quantitative sustainability research for conducting high throughput environmental assessments. This directly addresses the current lack of a readily available tool for researchers to explore solution spaces of environmental life-cycle assessments. HiTEA will make it simple, fast and cost-effective for researchers to perform thousands of model runs when they would previously only undertake one. As such, HiTEA will operationalize sophisticated sensitivity and uncertainty analysis as a standard implementation for sustainability research, pushing the field well beyond its current state. Case studies directly connected to SDGs will be conducted during the project to proof the usability of the HiTEA. These run in parallel to the software development, providing the feedback needed to ensure software which thoroughly meets scientific user requirements. To do so, the project adopts industry standard Agile Software Development practices for the use in Scientific Software Development. HiTEA embraces Open Source development. Besides maximizing outreach and exploitation, this will provide feedback on the usability and required capabilities already during the development phase. In addition, it actively encourages code contributions and usage from outside the project team to deliver a tool fully owned by the sustainability science community. The abstraction layers of the software architecture allow HiTEA to run on various e-infrastructure systems. During the project, HiTEA will be tested on local multi-core servers, HPC infrastructure at NTNU (OpenStack) and Sigma2 (Nird Toolbox) and commercial Kubernetes based cloud services like Amazon AWS and MS Azure. Reaching a mature software product and using the modern Infrastructure as a Service (IaaS) paradigm will finally lead to the inclusion of HiTEA in the service catalogue of the EOSCHub.