Efficient Execution of Large Workloads on Elastic Heterogeneous Resources

In the current and future industry and society, there will be an increasing number of systems storing and processing of large amounts of data. This is the next frontier for innovation, competition and productivity where there are currently large initiatives both in the EU and the US. Example data sets stem from areas like medicine, meteorology, physics, biology, environmental research, Internet search, finance and governmental informatics. Here, there are massive computational demands, but also huge dem ands for I/O and communication and for timeliness and deadlines. Additionally, there are often dependencies between processing steps. In a large computing cluster like a grid or a cloud, an important challenge is thus to execute the many concurrent computations in an efficient and dynamic manner where available resources, processing, communication, dependencies and timeliness must be taken into account when mapping tasks to processing cores. As such, the aim of the EONS research project is to perform research in the area of development of distributed large-scale heterogeneous systems, in particular medical systems where we aim at automatic disease detection. We aim at a complete, high accuracy end-to-end disease-detection system receiving and analysing video and giving feedback to doctors in real-time. The processing is parallelised over multiple heterogeneous processing cores to reach the required speed, scale and resource efficiency.

In the current and future industry and society, there will be an increasing number of systems storing and processing of large amounts of data. This is the next frontier for innovation, com- petition and productivity where there are currently large initiat ives both in the EU and the US. Example data sets stem from areas like medicine, meteorology, genomics, connectomics, physics, biology, environmental research, Internet search, finance and governmental informatics. Here, there are massive computational de mands, but also huge demands for I/O and communication and for timeliness and deadlines. Additionally, there are often dependencies between processing steps. In a large computing cluster like a grid or a cloud, an important challenge is thus to execute th e many concurrent computations in an efficient and dynamic manner where available resources, processing, communication, dependencies and timeliness must be taken into account when mapping tasks to processing cores. As such, the aim of the EONS research p roject is to perform basic research in the area of development of parallel programming and parallel processing in the context of future distributed large-scale heterogeneous systems. We aim to research and develop concepts and mechanisms that will enable the development of software for these next-generation big data applications. This is achieved by solving fundamental challenges for the dispatching, division, scheduling and identification of tasks that can run correctly in parallel in a shared distribute d system of heterogeneous computing resources in complex topologies.