This blog post describes the high level architecture for proposed self-organising self-managing heterogeneous cloud.
The life-cycle of a proposed cloud service is initiated by an Enterprise Application Developer (EAD) by submitting a Blueprint to a Gateway Service. A Blueprint is a graph representing a workflow of Services collectively composed to automate a particular task or business process.
The Gateway Service is a front-facing component of the CloudLightning system, abstracting the inner workings of the backend components and providing a unified access interface to the CloudLightning system. The Gateway Service provides a graphical user interface for EADs. As well as providing a workflow and an enabling process for Blueprint development, it also sends requests for resource options capable of running the Blueprint, receives resource deployment option(s), presents options to the EAD and ultimately enables the EAD to select and deploy the Blueprint to the chosen resources automatically.
The Gateway Service communicates to a self- contained self-organising self-managing system, which we define as a Cell. We envisage a Cell being typically associated with one geographical region and the CloudLightning system as a whole being composed of multiple Cells.
Each Cell is composed of resource fabric comprising heterogeneous compute, storage and network resources organised in to groups, each of which is called a vRack. A vRack Manager is used to maintain information about groups of servers with the same type of physical resources and manages a logical group of these resources which is called a Coalition. Coalitions can be formed either statically or dynamically.
Each vRack is self-managed. Local decisions are made by the vRack Manager, to decide on how coalitions are optimally formed (or identified) to deliver the service – reflecting self-optimisation at the vRack Manager Level. Self-healing and Self-protection may be addressed by feedback loops and fault tolerance, respectively. A vRack Manager (or vRack Manager Group) can deliver one type of the heterogeneous options for implementing a service.
vRack Managers cooperate to form a vRack Manager Group. vRacks Managers in the same Group are capable of self-organising to meet the objective of efficient power consumption based on their local knowledge of underlying resource utilisations. Each vRack Manager (or vRack Manager Group) autonomously monitors itself and the environment. Similarly, vRack Managers are aware of changes in the environment including new and disappearing resources and adapts on a negotiated basis with other vRack Managers within the same vRack Manager Group to meet system objectives.
Back at the Cell-level, having received a request from the Gateway Service for resource options capable of running the Blueprint, a resource discovery service propagates this request to appropriate vRack Manager Groups. The Cell Manager has the knowledge of the resource capacity in each vRack Manager Group and therefore can immediately communicate back to the Gateway Service informing it of the various delivery options available. These options can be agreed upon, using a Resource Selection Service, or selected automatically without the need to reserve the underlying resources. Having been agreed, the resources necessary for the service implementation can be commissioned by the appropriate vRack Manager (or vRack Manager Groups). This commissioning process involves either the identification of pre-defined, statically created, coalitions or by dynamically creating new coalitions in consultation with the Coalition Formation Services.