##########################################
HLO Data Aggregation and Alert System
##########################################

.. contents::
  :local:
  :depth: 1

Component of the HLO responsible for interacting with aerOS domain data fabric to retrieve all data needed for defining the most suitable IE accros the continuum for each workload to be  executed. 

Triggered by HLO frontend component, exposing and handling HLO Endpoint's, through message broker in protobuf format, including newly arrived service request id.
Retrieves all service runtime and computing resources requirments and  queries data fabric for all IEs which capable to support requirements.
Merges service requirements data and candidate IEs data in a protobuf message and forwards message to HLO Allocator by pushing the message to domain broker. 
The same process can also be initiated internally from self-orchestration component whcih might request service migration.

Data Aggregator interacts with HLO frontend (blue component) and allocator (in green).
Also events for service migration initiated by self-orchestration are present.

.. image:: aggregator.png
   :alt: Data-aggregator HLO interactions

Prerequisities
==============

This component requires make, helm and docker for building and installation.  

Installation
============

Note that this component is part of HLO common deployment and used in aerOS domain installation. You can install as part of this standard installation.

For development installation :

   * 
      .. code-block:: bash 
      
        git clone https://gitlab.aeros-project.eu/wp3/t3.3/hlo-data-aggregator.git
   * 
      .. code-block:: bash 
      
        cd hlo-data-aggregator
   * 
      .. code-block:: bash 
      
        python3 -m venv venv
   * 
     .. code-block:: bash 
      
        pip install -r requirments
   * 
      .. code-block:: bash 
      
        source venv/bin/activate
   * 
      .. code-block:: bash 
      
        kubectl port-forward -n redpanda svc/redpanda 9093:9093
   * 
     .. code-block:: bash 
      
        python3 main.py

For isolated component installation in aerOS domain, you can use the Makefile in the repository :

Edit variables, if needed, and :


* Build and tag docker image    

.. code-block:: bash

  make build

* Push to aerOS repository

.. code-block:: bash

  make push

* Package and upload helm chart

.. code-block:: bash

 make helm-upload

* Deploy against cluster chosen as current k8s context

.. code-block:: bash

 make deploy

* All the above

.. code-block:: bash

 make all

* Removes docker image

.. code-block:: bash

 make clean-build

* Removes the helm deployment from the cluster

.. code-block:: bash

 make clean-deploy 

* Removes all the above

.. code-block:: bash

 make clean-all 


Configuration options
=====================
The component can be configured from docker container environment variables :

- KRAKEND : KrakenD Present or not.
- API_URL : Context Broker URL.
- API_PORT : Context Broker Port.
- BOOTSTRAP_SERVERS : Redpanda bootstrap servers.
- GROUP_ID : Redpanda group id.
- AUTO_OFFSET_RESET : Auto Offset Reset option for Redpanda.
- CONSUMER_TOPIC : The Redpanda topic used to read from the HLO Frontend.
- PRODUCER_TOPIC : The Redpanda topic used to publish to the HLO Allocation Engine.

Developer guide
===============
As introduced before, The HLO Data Aggregation and Alert system interacts with HLO frontend by receiving the allocated service id. It transfers then the service component requirements to the HLO Allocation Engine. In what follows, the concerned data definitions :  

.. list-table:: Messages exchanged and Primitives definition
   :header-rows: 1

   * - Messages exchanged
     - Primitives definition

   * - 
       
       .. code-block:: protobuf
    
           syntax = "proto3";

           import  "hlo.proto";

            message HLOFEInput {
              Service service = 1;
            }
 
            message ServiceComponentRequirement {
              ServiceComponent service_component_definition = 2;
              repeated InfrastructureElement infrastructure_element_candidates = 1;
            }

            message HLOAllocatorOutput {
               repeated ServiceComponentRequirement service_component_requirements  = 1;
            }

     - 
        
        .. code-block:: protobuf

           syntax = "proto3";

           message Domain {
              string id = 1;
           }

           message Service {
              string id = 1;
           }

           message LowLevelOrchestrator {
              string id = 1;
              string orchestration_type = 3;
              Domain domain = 2;
           }

           message InfrastructureElement {
              string id = 1;
              float total_ram = 2;
              float cpu_cores = 3;
              float avg_power_consumption = 4;
              bool real_time_capable = 5;
              float current_ram = 6;
              float current_cpu_usage = 7;
              float current_power_consumption = 8;
              Domain domain = 9;
              LowLevelOrchestrator low_level_orchestrator = 10;
           }

           message Port {
              int32 number = 1;
           }

           message ServiceComponent {
              string id = 1;
              string image = 2;
              repeated Port ports = 3;
              Service service = 4;
              ServiceComponentConstraints service_component_constraints = 5;
              optional InfrastructureElement infrastructure_element = 6;
           }

           message ServiceComponentConstraints {
              map<string, float> constraints = 1;
           }


Authors
=======
The authors are:

- Pitsilis, Vasilis  (vpitsilis@dat.demokritos.gr).
- Benmerar, Tarik Zakaria (tarik.benmerar@ictficial.com).