ELEGANT
THE PROJECT
Motivation
Nowadays, the main driving force of extreme analytics is the business decisions that derive from processing vast amounts of data streams obtained from IoT devices. Big Data infrastructures cannot cope efficiently with the amount of raw data that IoT devices can generate, due to scalability or operational cost challenges. Therefore, on-edge computations, by means of lightweight computations such as aggregations, filtering or even lightweight AI models, are performed in order to enable a first order of data manipulation and filtering before they reach the backend of the Big Data stacks. Ideally, developers should regard an IoT/Big Data deployment as a single system that can be uniformly programmed and takes automatically intelligent decisions of where to execute code based on the PESRD optimization space.
Currently, no Big Data/IoT deployment provides a unified programming environment that allows:
-
Automated deployment of existing code in the form of edge-operators onto IoT devices,
-
Bi-directional communication channels between IoT and Big Data to control data streams (velocity, volume, variety) and code motion,
-
Dynamic self-adaptation of both the correct amount and type/placement of resources for code execution according to user defined SLAs or PESRD constraints, and
-
Intelligent application of security mechanisms throughout the different layers of the unified ecosystems.
Vision
ELEGANT’s vision is summarized by the following four points:
-
Unification of programming environments: Through virtualized execution, both Big Data and IoT devices will offer the same application programming interface (API) to programmers. This will allow us to define operators, code them once, and deploy them both at the data analytics (Big Data) and the IoT sides.
-
Dynamic Code Motion: By unifying the programming environments between Big Data and IoT we will enable code reuse between the different platforms. This in turn, will enable the dynamic code motion across the different layers of the deployments. This will be achieved by enabling the lightweight IoT runtimes to dynamically accept and apply incoming operators without restarting their execution.
-
Intelligent resource selection and allocation: By enabling the dynamic code motion between Big Data and IoT we essentially unify the two systems, thereby allowing intelligent resource selection and allocation. By establishing a global monitoring system, we will enable decisions such as where to execute code, how much and which data we can process in-situ or stream to the data analytics side, IoT energy savings, etc.
-
Secure, Reliable, and Dependable code deployment: The unification of operators that can execute on both IoT and Big Data will enable a single code validation and security point for all running code. In addition, the same runtime verification and security techniques will be applied on both sides. Finally, through dynamic code motion we will be able to transition between in-situ on-edge processing and data streaming completely dynamically based on the data sensitivity, network stability, and software and hardware reliability.
Objectives
01
Dynamic code motion between cloud data analytics and IoT.
-
Enable code sharing between Big Data and IoT
-
Enhanced Security and Verification of IoT deployed code
02
Big Data/IoT interoperability and orchestration for improved performance, energy efficiency, dependability, and reliability.
-
Automated and easy management of large scale IoT deployments
-
Elasticity in data generation and computation rate
A new software to overcome IoT and big data fragmentation concerns
Our Technologies
ELEGANT’s unified operators’ API
This component is responsible for creating a unified API for data operators that can be executed on both the IoT devices and the data analytics side. The operators will be initially designed to fulfil the requirements of the use cases. In addition, a generalized framework regarding the extraction, packaging and deployment of operators that will aid the development of third-party operators will be designed. Finally, these operators will form the basic blocks of the code motion between IoT and Big Data.
ELEGANT IoT Application Virtualization
By integrating low-power application virtualization on IoT devices (in the form of embedded Java Virtual Machines) we will enable them to execute the unified data operators of the unified opeartos' API. In addition, the virtualization layer will be designed with dynamic code motion at its core, allowing the dynamic application of incoming data operators without restarting the application. This will allow the whole ELEGANT architecture to work as a single unit that enables dynamic data management based on the PESRD requirements. The aforementioned capabilities will lead to elastic Big Data/IoT deployments where decisions can be made at real-time, while achieving the desired pareto optimal optimization point.
ELEGANT-enabled Data Analytics Platforms and Gateways
Byintegratinglow-power application virtualization on IoT devices (in the form of embedded Java Virtual Machines) we will enable them to execute the unified data operators of Component 1. In addition, the virtualization layer will be designed with dynamic code motion at its core, allowing the dynamic application of incoming data operators without restarting the application. This will allow the whole ELEGANT architecture to work as a single unit that enables dynamic data management based on the PESRD requirements. The aforementioned capabilities will lead to elastic Big Data/IoT deployments where decisions can be made at real-time, while achieving the desired pareto optimal optimization point.
ELEGANT Intelligent Orchestrator
This component runs horizontally to the whole ELEGANT architecture and it entails all the envisioned software artefacts that will enable the dynamic code motion and intelligent resource scheduling decisions. All hardware platforms (IoT, Gateways, Cloud analytics) will be augmented with subsystems of the Intelligent Orchestrator to:
-
profile the whole system,
-
make scheduling decisions regarding where (edge or not, the amount and type of resources) code should be executed,
-
instruct code motion and take actions based on the scheduling decisions, and
-
enable or disable the security features of the ELEGANT Security Layer (Component
-
All the above will be based on an AI-assisted layer integrated into the orchestrator that will be trained based on the PESRD values and SLAs of the executed applications.
Depending on the execution scenario, different optimization points will be used to guide the scheduling decisions.
ELEGANT Security Layer
Each layer of the ELEGANT architecture will have built-in security features that can be dynamically enabled or disabled based on the security requirements of the executed applications. Techniques in the form of code verification both at the edge and at the cloud analytics side will be implemented and invoked on demand by reusing the code motion techniques of the previous components. In addition, state-of-the-art cybersecurity techniques at the IoT and networking side will be developed.
