D2.5 Conceptual model and Reference architecture - II

BigDataStack aims to deliver a complete stack including an infrastructure management solution that drives decisions according to live and historical data, thus being fully scalable, runtime adaptable and highly performant. The overall objective is for BigDataStack to address the emerging needs of big data operations and data-intensive applications. The solution will base all infrastructure management decisions on data aspects, monitoring data from deployments and logic derived from data operations that govern and affect storage, compute and network resources. On top of the infrastructure management solution, the “Data as a Service” will be offered to data providers, decision-makers, private and public organisations. Approaches for data quality assessment, data skipping and efficient storage, combined with seamless data analytics will be realised holistically across multiple data stores and locations.

Enhanced infrastructure management with BigDataStack services

To provide the required information towards enhanced infrastructure management BigDataStack will provide a range of services:

• Application dimensioning workbench: facilitates data-focused application analysis and dimensioning in terms of predicting the required data services, their interdependencies with the application micro-services and the necessary underlying resources. This will allow the identification of the applications data-related properties and their data needs, thereby enabling BigDataStack to provision deployment with specific performance and quality guarantees. 
• Data toolkit: will enable data scientists to ingest their data analytics functions and to specify their preferences and constraints, which will be exploited by the infrastructure management system for resources and data management. 
• Process modelling framework: will be delivered, to enable functionality-based modelling of processes, which will be mapped in an automated way to concrete technical-level data analytics tasks. 

The aforementioned key outcomes of BigDataStack are reflected in a set of main building blocks in the corresponding overall architecture of the stack. This deliverable describes the key functionalities of the overall architecture, the interactions between the main building blocks and their components, while also providing a first version of the internals of these components regarding research approaches to be realised during the course of the project.  

What’s new? 

This deliverable is a refinement of the key functionalities of the overall architecture, the interactions between the main building blocks and their components, as they were described in the previous version of the architecture (Deliverable D2.4 - Conceptual model and Reference architecture). 

Key changes refer to:

• interplay between the application and data dimensioning and the components that manage the deployment lifecycle (i.e. deployment patterns generation and ranking and deployment management);
• dynamic orchestrator and the overall quality and performance assessment during runtime;
• changes in the specifications of several components, reflecting their latest implementation status.  

Download the full report