Application preparation for H2020 project proposal DataCloud for the call id ID: ICT-40-2020

Cloud computing has been a major disruptive technology in the last decade providing resources-as-a-service for diverse Internet applications, poised to grow by around 27.5% and expected to reach $1,250 billion by 2025 . While Cloud offers elastic capacity and customisable connectivity over a large-scale network, the resilience, sustainability and human-centric collaborative requirements of Big Data pipeline applications demand an interoperable end-to-end ecosystem design that pushes the infrastructure services, traditionally bounded within data centres, towards remote nodes closer to the data sources. These concerns are partially addressed by the computing continuum federating Cloud services with emerging Edge and Fog computing paradigms with reduced overheads of transferring distributed data into remote data centres. Unfortunately, eminent challenges in supporting Big Data pipeline applications still remain, including effective Big Data pipeline discovery, domain-specific languages and simulation, deployed over trustworthy services over heterogeneous resources from different providers DataCloud enables Big Data pipeline discovery through the utilization of AI-based process mining algorithms capable of learning the structure of data pipelines across multiple data silos. Furthermore, Data Cloud will provide a high-level tool for workflows definition and a framework for efficient simulation of complex Big Data pipelines for improved resources utilization and performance increment. These complex tools and frameworks will be supported by a decentralized pool of resource, managed through a blockchain technology, with a capability for resource provisioning in interoperable Fog/Cloud/Edge infrastructure, and flexible and automated deployment of Big Data pipelines.