Multi-Tenancy on VMware Cloud Foundation with vRealize Automation and Cloud Director

Multi-Tenancy on VMware Cloud Foundation with vRealize Automation and Cloud Director

Nov 30, 2020 | AWS, Azure, Cloud Director, Google Cloud, NSX, Tanzu, VCPP, VMC, VMware Cloud, VMware Cloud Foundation, vSAN, vSphere

In my article VMware Cloud Foundation And The Cloud Management Platform Simply Explained I wrote about why customers need a VMware Cloud Foundation technology stack and what a VMware cloud management platform is.

One of the reasons and one of the essential characteristics of a cloud computing model I mentioned is resource pooling.

By the National Institute of Standards and Technology (NIST) resource pooling is defined with the following words:

The provider’s computing resources are pooled to serve multiple
consumers using a multi-tenant model, with different physical and virtual
resources dynamically assigned and reassigned according to consumer demand.
There is a sense of location independence in that the customer generally has no
control or knowledge over the exact location of the provided resources but may be
able to specify location at a higher level of abstraction (e.g., country, state, or
data center).

This time I would like to focus on multi-tenancy and how you can achieve that on top of VMware Cloud Foundation (VCF) with Cloud Director (formerly known as vCloud Director) and vRealize Automation, which both could be part of a VMware cloud management platform (CMP).

Multi-Tenancy

There are many understandings around about multi-tenancy and different people have different definitions for it.

If we start from the top of an IT infrastructure, we will have application or software multi-tenancy with a single instance of an application serving multiple tenants. And in the past even running on the same virtual or physical server. In this case the multi-tenancy feature is built into the software, which is commonly accessed by a group of users with specific permissions. Each tenant gets a dedicated or isolated share of this application instance.

Coming from the bottom of the data center, multi-tenancy describes the isolation of resources (compute, storage) and networks to deliver applications. The best example here are (cloud) services providers.

Their goal is to create and provide virtual data centers (VDC) or a virtual private cloud (VPC) on top of the same physical data center infrastructure – for different tenants aka customers. Normally, the right VMware solution for this requirement and service providers would be Cloud Director, but this is maybe not completely true anymore with the release of vRealize Automation 8.x. 

To make it easier for all of us, I’ll call Cloud Director and vCloud Director “vCD” from now on.

VMware Cloud Director (formerly vCloud Director)

Cloud Director is a product exclusively for cloud service providers via the VMware Cloud Provider Program (VCPP). Originally released in 2010, it enables service providers (SPs) to provision SDDC (Software-Defined Data Center) services as complete virtual data centers. vCD also keeps resources from different tenants isolated from each other.

Within vCD a unit of tenancy is called Organization VDC (OrgVDC). It is defined as a set of dedicated compute (CPU, RAM), storage and network resources. A tenant can be bound to a single OrgVDC or can be composed of multiple Organization VDCs. This is typically known as Infrastructure as a Service (IaaS).

A provider virtual data center (PVDC) is a grouping of compute, storage, and network resources from a single vCenter Server instance. Multiple organizations/tenants can share provider virtual data center resources.

Cloud Director Resource Abstraction

A lot of customers and VCPP partners have now started to offer their cloud services (IaaS, PaaS, SaaS etc.) based on VMware Cloud Foundation. For private and hybrid cloud scenarios, but also in the public cloud as a managed cloud service (VMware Cloud on AWS, Azure VMware Solution, Google Cloud VMware Engine, Alibaba Cloud VMware Solution and more).

Important: I assume that you are familiar with VCF, its core components (ESXi, vSAN, NSX, SDDC Manager) and architecture models (standard as the preferred).

Cloud Director components are currently not part of the VCF lifecycle automation, but it is a roadmap item!

Cloud Director Resource Hosting Models

vCD offers multiple hosting models:

  • In the shared hosting model, multiple tenant workloads run all together on the same
    resource groups without any performance assurance
  • In the reserved hosting model, performance of workloads is assured by resource
    reservation.
  • In the physical hosting model, hardware is dedicated to a single tenant and performance
    is assured by the allocated hardware

Tenant Using Shared Hosting on VCF Workload Domain

In this use case a tenant is using shared hosting backed by a VMware Cloud Foundation workload domain. A workload domain, which is mapped to a provider VDC.

vCD VCF Shared

Tenant Using Shared Hosting and Reserved Hosting on Multiple VCF Workload Domains

This use case describes the example of customer using shared and reserved hosting backed by multiple VCD workload domains. Here each cluster has a single resource pool mapped to a single PVDC.

vCD VCF Shared Reserved

Tenant Using Physical Hosting and Central Point of Management (CPOM)

The last example shows a single customer using physical hosting. You will notice that there is also a vSphere with
Kubernetes workload domain. VMware Cloud Foundation automates the installation of vSphere with Kubernetes (Tanzu) which makes it incredibly easy to deploy and manage.

You can see that there is an “SDDC” box on top of the Kubernetes Cluster vCenter, which is attached to
the “SDDC Proxy” entity. vCD can act as an HTTP/S proxy server between tenants and the
underlying vSphere environment in VMware Cloud Foundation. An SDDC proxy is an
access point to a component from an SDDC, for example, a vCenter Server instance, an ESXi host, or
an NSX Manager instance.

The vCD becomes the central point of management (CPOM) in this case and the customer gets a complete dedicated SDDC with vCenter access.

vCD VCF Physical CPOM

Note: Since vCD 9.7 it is possible to present for example a vCenter Server instance securely to a tenant’s organization using the Cloud Director user interface. This is how you could build your own VMC-on-AWS-like cloud offering!

Cloud Director CPOM

All 3 Tenants Together

Finally, we put it all together. In the first use case we can see that different customers are sharing resources from a
single PVDC. We can also see that resources from a single vCenter can be split across different provider virtual datacenters and that we can mix and match multi-tenants workload domains and workload domains offering dedicated private cloud all together.

vCD VCF All Together

Cloud Director Service and VMware Cloud on AWS

If you don’t want to extend or operate your own data center or cloud infrastructure anymore and provide a managed service to multiple customer, there are still options for you available backed by VMware Cloud Foundation as well.

Since October 2020 you have Cloud Director Service globally available, which delivers multi-tenancy to VMware Cloud on AWS for managed service providers (MSP).

VMware sees not only new, but also existing VCPP partners moving towards a mixed-asset portfolio, where their cloud management platform consists of a VCPP and MSP (VMware SaaS offerings) contract. This allows them for example to run vCD on-premises for their current customers and the onboarding of new tenants would happen in the public cloud with CDS and VMC on AWS.

vCD CDS Mixed Mode

Enterprise Multi-Tenancy with vRealize Automation

With the release of vRealize Automation 8.1 (vRA) VMware offered support for dedicated infrastructure multi-tenancy, created and managed through vRealize Suite Lifecycle Manager. This means vRealize Automation enables customers or IT providers to set up multiple tenants or organizations within each deployment.

Providers can set up multiple tenant organizations and allocate infrastructure. Each tenant manages its own projects (team structures), resources and deployments.

Enabling tenancy creates a new Provider (default) organization. The Provider Admin will create new tenants, add tenant admins, setup directory synchronization, and add users. Tenant admins can also control directory synchronization for their tenant and will grant users access to services within their tenant. Additionally, tenant admins will configure Policies, Governance, Cloud Zones, Profiles, access to content and provisioned resources; within their tenant. A single shared SDDC or separate SDDCs can be used among tenants depending on available resources.

vRealize Automation 8.1 Multi-Tenancy

With vRealize Automation 8.2, provider administrators got the ability to share infrastructure by creating and assigning Virtual Private Zones (VPZ) to tenant organizations.

Think of VPZs as a kind of container of infrastructure capacity and services which can be defined and allocated to a Tenant. You can add unique or shared cloud accounts, with associated compute, flavors, images, storage, networking, and tags to each VPZ. Each component offers the same configuration options you would see for a standalone configuration.

vRealize Automation 8.2 Multi-Tenancy

vRealize Automation and VMware Cloud Foundation

With the pretty new multi-tenancy and VPZ capability a new consumption model on top of VCF can be built. You (provider) would map the Cloud Zones (compute resources on vSphere (or AWS for example)) to a VCF workload domain.

The provider sets these cloud zones up for their customers and provides dedicated or shared infrastructure backed by Cloud Foundation workload domains.

This combination would allow you to build an enterprise VPC construct (like AWS for example), a logically isolated section of your provider cloud.

vRealize Automation and VMware Cloud Foundation

SDDC Manager Integration and VMware Cloud Foundation (VCF) Cloud Account

Since the vRA 8.2 release customers are also able to configure a SDDC Manager integration and on-board workload domains as VMware Cloud Foundation cloud accounts into the VMware Cloud Assembly service.

VMware Cloud Director or vRealize Automation?

You wonder if vRealize Automation could replace existing vCD installations? Or if both cloud management platforms can do the same?

I can assure you, that you can provide a self-service provisioning experience with both solutions and that you can provide any technology or cloud service “as a service”. Both have in common to be backed by Cloud Foundation, have some form of integration (vRA) and can be built by a VMware Validated Design (VVD).

vCD is known to be a service provider solution, where vRA is more common in enterprise environments. VMware has VCPP partners, that use Cloud Director for their external customers and vRealize Automation for their internal IT and customers.

If you are looking for a “cloud broker” and Infrastructure as Code (IaC), because you also want to provision workloads on AWS, Azure or GCP as well, then vRealize Automation is the better solution since vCD doesn’t offer this deep integration and these deployment options yet.

Depending on your multi-tenant needs and if you for example only have chosen vCD in the past, because of the OrgVDC and resource pooling feature, vRealize Automation would be enough and could replace vCD in this case.

It is also very important to understand how your current customer onboarding process and operational model look like:

  • How do you want to create a new tenant? 
  • How do you want to onboard/migrate existing customer workloads to your provider infrastructure?
  • Do you need versioning of deployments or templates?
  • Do customers require access to the virtual infrastructure (e.g. vCenter or OrgVDC) or do you just provide SaaS or PaaS?
  • Do customers need a VPN or hybrid cloud extension into your provider cloud?
  • How would you onboard non-vSphere customers (Hyper-V, KVM) to your vSphere-based cloud?
  • Does your customer rely on other clouds like AWS or Azure?
  • How do you do billing for your vSphere-based cloud or multi-cloud environment?
  • What is your Kubernetes/container strategy?
  • And 100 other things 😉

There are so many factors and criteria to talk about, which would influence such a decision. There is no right or wrong answer to the question, if it should be VMware Cloud Director or vRealize Automation. Use what makes sense.

Which could also be a combination of both.

VMware Carbon Black Cloud Workload – Agentless Protection for vSphere Workloads

VMware Carbon Black Cloud Workload – Agentless Protection for vSphere Workloads

Nov 26, 2020 | Carbon Black, ESXi, Horizon, Tanzu, VMware Cloud Foundation, vSphere

At VMworld 2020 VMware announced Carbon Black Cloud Workload (CBC Workload) as part of their intrinsic security approach.

For me, this was the biggest and most important announcement from this year’s VMworld. It is a new offering, which is relevant for every vSphere customer out there – even the small and medium enterprises, which maybe still just rely on ESXi and vCenter only for their environment.

CBC Workload introduces protection for workloads in private and public clouds. For vSphere, there is no additional agent installation needed, because the Carbon Black sensor (agent) is built into vSphere. That’s why you may hear that this solution is “agentless”.

Carbon Black Cloud Workload Bundles

This cloud-native (SaaS) solution provides foundational workload hardening and vulnerability management combined with prevention, detection and response capabilities to protect workloads running in virtualized private cloud and hybrid cloud environments.

Carbon Black Cloud Workload Protection Bundles

Note: Customers, that are using vSphere and VMware Horizon, should take a look at Workspace Security VDI, which has also been announced at VMworld 2020. A single-vendor solution with the combination of VMware Horizon and Carbon Black.

If you would like to know more about the interoperability of Carbon Black and Horizon, have a look at KB79180.

Carbon Black Cloud Workload Overview

Customers and partners have now the possibility to provide a workload security solution for Windows and Linux virtual machines. The complete system requirements can be found here.

“You can enable Carbon Black in your data center with an easy one-click deployment. To minimize your deployment efforts, a lightweight Carbon Black launcher is made available with VMware Tools. Carbon Black launcher must be available on the Windows and Linux VMs.”

Carbon Black enable via vCenter

Carbon Black Cloud Workload consists of a few key components that interact with each other:

CBC Workload Components

You must first deploy an on-premises OVF/OVA template for the Carbon Black Cloud Workload appliance (4 vCPU, 4GB RAM, 41GB storage) that connects the Carbon Black Cloud to the vCenter Server through a registration process. After the registration is complete, the Carbon Black Cloud Workload appliance deploys the Carbon Black Cloud Workload plug-in and collects the inventory from the vCenter Server.

The plug-in provides visibility into processes and network connections running on a virtual machine.

As a vCenter Server administrator, you want to have visibility of known vulnerabilities in your environment to understand your security posture and schedule maintenance windows for patching and remediation. With the help of vulnerability assessment, you can proactively minimize the risk in your environment. You can now monitor known vulnerabilities from the Carbon Black Cloud Workload plug-in:

vSphere Client Carbon Black

The infosec guys in your company would do the vulnerability assessment from the CBC console:

CBC Vulnerabilities

Carbon Black Cloud Workload protection provides vSphere administrators a full inventory, appliance health and vulnerability reporting from one console, the already well-known vSphere Client.

Carbon Black vSphere Client Summary

Cybersecurity Requirements

According to the NIST Cybersecurity Framework the security lifecycle is made of five functions:

  1. Identify – Cloud & Service Context, Dynamic Asset Visibility, Compliance & Standards, Cloud Risk Management
  2. Protect – Services / API Defined, Cloud Access Control, Network Integrity, Data Security, Change Control & Guardrails
  3. Detect – Cloud-Speed, Inter-connected Services, Events & Anomalies, Continuous Monitoring
  4. Respond – DevOps Collaboration, Real-time Notifications, Automated Actions, Response as Code
  5. Recover – Templates / Code Review, Shift Left / Pipeline, Exceptions and Verification

Workload Security Lifecycle

CBC Workload focuses on identifying the risks with workload visibility and vulnerability management, which are part of the “Workload Essentials” edition.

If you would like to prevent malicious activities to protect your workloads and replace your existing legacy anti-virus (AV) solution, then “Workload Advanced” would be the right edition for you as it includes Next-Gen AV (NGAV).

Behavioral EDR (Endpoint Detection & Response), also part of the “Advanced” bundle, belongs to “detect & respond” of the security lifecycle.

Workload Security for Kubernetes

Carbon Black Guardrails and Runtime Security

You just learned that Carbon Black Cloud gives workload protection for virtualized Windows or Linux virtual machines running on vSphere. What about container security for Kubernetes?

In May 2020 VMware officially closed its acquisition of Octarine, a SaaS security platform for protecting containers and Kubernetes. VMware bought Octarine to enable Carbon Black to secure applications running in Kubernetes.

Traditional security is no longer relevant for the security of Kubernetes, because Kubernetes is so powerful and hence risky, networking is very complex and a total different game, because static IPs and ports are no longer relevant. And you need a new security approach which is compatible with IT’s organizational shift from traditional to a DevSecOps approach.

VMware’s solution covers the whole lifecycle of the application from building the container to the app running in production. It is a two-part solution with the first one being “Guardrails“. It is able to scan container images for vulnerabilities and Kubernetes manifests for any misconfigurations.

Carbon Black Cloud Guardrails Module

The second part is runtime protection. When the workloads are deployed in production, the Carbon Black security agent is able to detect malicious activities.

Carbon Black Cloud Runtime Module 

Let’s have a look at the different features the Kubernetes “Guardrails” provide for each phase of the application:

  • Build: Image vulnerability scanning, Kubernetes configuration hardening
  • Deploy: Policy governance, compliance reporting, visibility and hardening
  • Operate: Threat detection and response, anomaly detection and least privilege runtime, event monitoring

And these were the key capabilities and benefits, which have been mentioned at VMworld 2020 for “Guardrails”:

Carbon Black Kubernetes Guardrails Features

For “runtime” security the following key capabilities and benefits were mentioned:

  • Visibility of network traffic
  • Coverage of workloads and hosts activity
  • Network policy management
  • Threat detection
  • Anomaly detection
  • Egress security
  • SIEM integration

Customers will be able to have visibility of all the workloads running in the local or cloud-native production clusters and how they interact with each other. They will also see which services are exposed to ingress traffic, which services are exiting the cluster and where this egress traffic is going to. It is also going to be visible which communication is encrypted and what type of encryption is used.

Note: The Carbon Black Cloud module for hardening and securing Kubernetes workloads is expected to be generally available until the end of 2020.

The launch of Carbon Black Workload was the first important step to let the intrinsic security vision become more a reality (after VMware acquired Carbon Black). Moving on with Kubernetes and bringing new container security capabilities is going to be the next big move forward, that VMware can become a major security provider. 

Stay tuned for more security announcements!

Additional Resources

If you would like to know more about Carbon Black Cloud Workload and security for Kubernetes, have a look at:

Introduction to Alibaba Cloud VMware Solution (ACVS)

Introduction to Alibaba Cloud VMware Solution (ACVS)

Nov 11, 2020 | Alibaba Cloud, HCX, Horizon, NSX, VCPP, VMware Cloud, VMware Cloud Foundation, vSAN, vSphere

VMware’s hybrid and multi-cloud strategy is to run their Cloud Foundation technology stack with vSphere, vSAN and NSX in any private or public cloud including edge locations. I already introduced VMC on AWS, Azure VMware Solution (AVS), Google Cloud VMware Engine (GCVE) and now I would like to briefly summarize Alibaba Cloud VMware Solution (ACVS).

VMware Multi-Cloud Offerings

A lot of European companies, this includes one of my large Swiss enterprise account, defined Alibaba Cloud as strategic for their multi-cloud vision, because they do business in China. The Ali Cloud is the largest cloud computing provider in China and is known for their cloud security, reliable and trusted offerings and their hybrid cloud capabilities.

In September 2018, Alibaba Cloud (also known as Aliyun), a Chinese cloud computing company that belongs to the Alibaba Group, has announced a partnership with VMware to deliver hybrid cloud solutions to help organizations with their digital transformation.

Alibaba Cloud was the first VMware Cloud Verified Partner in China and brings a lot of capabilities and services to a large number of customers in China and Asia. Their current global infrastructure operates worldwide in 22 regions and 67 availability zones with more regions to follow. Outside Main China you find Alibaba Cloud data centers in Sydney, Singapore, US, Frankfurt and London.

As this is a first-party offering from Alibaba Cloud, this service is owned and delivered by them (not VMware). Alibaba is responsible for the updates, patches, billing and first-level support.

Alibaba Cloud is among the world’s top 3 IaaS providers according to Gartner and is China’s largest provider of public cloud services. Alibaba Cloud provides industry-leading flexible, cost-effective, and secure solutions. Services are available on a pay-as-you-go basis and include data storage, relational databases, big-data processing, and content delivery networks.

Currently,  Alibaba Cloud has been declared as a Niche player according to the actual Gartner Magic Quadrant for Cloud Infrastructure and Platform Services (CIPS) with Oracle, IBM and Tencent Cloud.

Alibaba Gartner CIPS MQ

Note: If you would like to know more about running the VMware Cloud Foundation stack on top of the Oracle Cloud as well, I can recommend Simon Long’s article, who just started to write about Oracle Cloud VMware Solution (OCVS).

This partnership with VMware and Alibaba Cloud has the same goals like other VMware hybrid cloud solutions like VMC on AWS, OCVS or GCVE – to provide enterprises the possibility to meet their cloud computing needs and the flexibility to move existing workloads easily from on-premises to the public cloud and have highspeed access to the public cloud provider’s native services.

ACVS vSphere Architecture

In April 2020, Alibaba Cloud and VMware finally announced the general availability of Alibaba Cloud VMware Solution for the Main China and Hongkong region (initially). This enables customers to seamlessly move existing vSphere-based workloads to the Alibaba Cloud, where VMware Cloud Foundation is running on top of Aliyun’s infrastructure.

As already common with such VMware-based hybrid cloud offerings, this let’s you move from a Capex to a Opex-based cost model based on subscription licensing.

Joint Development

X-Dragon – Shenlong in Chinese – is a proprietary bare metal server architecture developed by Alibaba Cloud for their cloud computing requirements. It offers direct access to CPU and RAM resources without virtualization overheads that bare metal servers offer (built around a custom X-Dragon MOC card). The virtualization technology, X-Dragon, behind Alibaba Cloud Elastic Compute Service (ECS) is now in its third generation. The first two generations were called Xen and KVM.

X-Dragon NIC

VMware works closely together with the Alibaba Cloud engineers to develop a VMware SDDC (software-defined data center based on vSphere and NSX) which runs on this X-Dragon bare metal architecture.

The core of the MOC NIC is the X-Dragon chip. The X-Dragon software system runs on the X-Dragon chip to provide virtual private cloud (VPC) and EBS disk capabilities. It offers these capabilities to ECS instances and ECS bare metal instances through VirtIO-net and VirtIO-blk standard interfaces.

Note: The support for vSAN is still roadmap and comes later in the future (no date committed yet). Because the X-Dragon architecture is a proprietary architecture, running vSAN over it requires official certification. 

Project Monterey

Have you seen VMware’s announcement at VMworld 2020 about Project Monterey which allows you to run VMware Cloud Foundation on a SmartNIC? For me, this looks similar to the X-Dragon architecture 😉

Project Monterey VMware Cloud Foundation Use Cases

Data Center extension or retirement. You can scale the data center capacity in the cloud on-demand, if you for example don’t want to invest in your on-premises environment anymore. In case you just refreshed your current hardware, another use case would be the extension of your on-premises vSphere cloud to Alibaba Cloud.ACVS Disaster Recovery

Disaster Recovery and data protection. Here we’ll find different scenarios like recovery (replication) or backup/archive (data protection) use cases. You can use your ACVS private clouds as a disaster recovery (DR) site for your on-premises workloads. This DR solution would be based on VMware Site Recovery Manager (SRM) which can be also used together with HCX. At the moment Alibaba Cloud offers 9 regions for DR sites.

Cloud migrations or consolidation. If you want to start with a lift & shift approach to migrate specific applications to the cloud, then ACVS is the right choice for you. Maybe you want to refresh your current infrastructure and need to relocate or migrate your workloads in an easy and secure way? Another perfect scenario would be the consolidation of different vSphere-based clouds.

ACVS Migration to Alibaba Cloud

Multicast Support with NSX-T

Like with Microsoft Azure and Google Cloud, an Alibaba Cloud ECS instance or VPC in general doesn’t support multicast and broadcast. That is one specific reason why customers need to run NSX-T on top of their public cloud provder’s global cloud infrastructure.

Connectivity Options

For (multi-)national companies Alibaba Cloud has different enterprise-class networking offerings to connect different sites or regions in a secure and reliable way.

Cloud Enterprise Network (CEN) is a highly-available network built on the high-performance and low-latency global private network provided by Alibaba Cloud. By using CEN, you can establish private network connections between Virtual Private Cloud (VPC) networks in different regions, or between VPC networks and on-premises data centers.  The CEN is also available in Europe in Germany (Frankfurt) and UK (London).

Alibaba Cloud Cloud Enterprise Network

Alibaba Cloud Express Connect helps you build internal network communication channels that feature enhanced cross-network communication speed, quality, and security. If your on-premises data center needs to communicate with an Alibaba Cloud VPC through a private network, you can apply for a dedicated physical connection interface from Alibaba Cloud to establish a physical connection between the on-premises data center and the VPC. Through physical connections, you can implement high-quality, highly reliable, and highly secure internal communication between your on-premises data center and the VPC. 

Alibaba Cloud Express Connect

ACVS Architecture and Supported VMware Cloud Services

Let’s have a look at the ACVS architecture below. On the left side you see the Alibaba Cloud with the VMware SDDC stack loaded onto the Alibaba bare metal servers with NSX-T connected to the Alibaba VPC network.

This VPC network allows customers to connect their on-premises network and to have direct acccess to Alibaba Cloud’s native services.

Customers have the advantage to use vSphere 7 with Tanzu Kubernetes Grid and could leverage their existing tool set from the VMware Cloud Management Platform like vRealize Automation (native integration of vRA with Alibaba Cloud is still a roadmap item) and vRealize Operations.

Alibaba Cloud VMware Solution Architecture

The right side of the architecture shows the customer data centers, which run as a vSphere-based cloud on-premises managed by the customer themselves or as a managed service offering from any service provider. In between, with the red lines, the different connectivity options like Alibaba Direct Connect, SD-WAN or VPN connections are mentioned with different technologies like NSX-T layer 3 VPN, HCX and Site Recovery Manager (SRM).

To load balance the different application services across the different vSphere-based or native clouds, you can use NSX Advanced Load Balancer (aka Avi) to configure GSLB (Global Server Load Balancing) for high availability reasons.

Because the entire stack on top of Alibaba Cloud’s infrastructure is based on VMware Cloud Foundation, you can expect to run everything in VMware’s product portfolio like Horizon, Carbon Black, Workspace ONE etc. as well.

You can also deploy AliCloud Virtual Edges with VMware SD-WAN by VeloCloud.

Node Specifications

The Alibaba Cloud VMware Solution offering is a little bit special and I hope that I was able to translate the Chinese presentations correctly.

First, you have to choose the amount of hosts which gives you specific options.

1 Host (for testing purposes): vSphere Enterprise Plus, NSX Data Center Advanced, vCenter

2+ Hosts (basic type): vSphere Enterprise Plus, NSX Data Center Advanced, vCenter

3+ Hosts (flexibility and elasticity): vSphere Enterprise Plus, NSX Data Center Advanced, vCenter, (vSAN Enterprise)

Site Recovery Manager, vRealize Log Insight and vRealize Operations need to be licensed separately as they are not included in the ACVS bundle.

The current ACVS offering has the following node options and specifications (maximum 32 hosts per VPC):

ACVS Node Specifications

All sixth-generation ECS instance come equipped with Intel® Xeon® Platinum 8269CY processors. These processors were customized based on the Cascade Lake microarchitecture, which is designed for the second-generation Intel® Xeon® Scalable processors. These processors have a turbo boost with an increased burst frequency of 3.2 GHz, and can provide up to a 30% increase in floating performance over the fifth generation ECS instances.

Component Version License
vCenter 7.0 vCenter Standard
ESXi 7.0 Enterprise Plus
vSAN (support coming later) n/a Enterprise
NSX Data Center (NSX-T) 3.0 Advanced
HCX n/a Enterprise

Note: Customers have the possibility to install any VIBs by themselves with full console access. This allows the customer to assess the risk and performance impacts by themselves and install any needed 3rd party software (e.g. Veeam, Zerto etc.).

If you want to more about how to accelerate your multi-cloud digital transformation initiatives in Asia, you can watch the VMworld presentation from this year. I couldn’t find any other presentation (except the exact same recording on YouTube) and believe that this article is the first publicy available summary about Alibaba Cloud VMware Solution. 🙂

VMware Cloud Foundation And The Cloud Management Platform Simply Explained

VMware Cloud Foundation And The Cloud Management Platform Simply Explained

Nov 3, 2020 | AWS, Azure, Google Cloud, HCX, NSX, VMware Cloud, VMware Cloud Foundation, vSAN, vSphere

I think that it is pretty clear what VMware Cloud Foundation (VCF) is and what it does. And it is also clear to a lot of people how of where you could use VCF. But very few organizations and customers know why they should or could use Cloud Foundation and what its purpose is. This article will give you a better understanding about the “hidden” value that VMware Cloud Foundation has to offer.

My last contributions focused on VMware’s multi-cloud strategy and how they provide consistency in any layer of their vision:

VMware Strategy

The VMware messaging is clear. By deploying consistent infrastructure across clouds, customers gain consistent operations and intrinsic security in hybrid or multi-cloud operating models. The net result is, that the intricacies of infrastructure fade, allowing IT to focus more on deploying applications and providing secure access to those applications and data from any device.

The question is now, what are the building blocks and how can you fulfill this strategy? And why is VMware Cloud Foundation really so important?

Cloud Computing

To answer these questions we have to start with the basics and look at the NIST definition of cloud computing first:

Cloud computing is a model for enabling convenient, on-demand network access to a shared
pool of configurable computing resources (e.g., networks, servers, storage, applications, and
services) that can be rapidly provisioned and released with minimal management effort or
service provider interaction. This cloud model promotes availability and is composed of five
essential characteristics, three service models, and four deployment models.

Data Center Cloud Computing

Let’s start with the three service models and the capabilities each is aiming to provide:

  • Software as a Service (SaaS). Centrally hosted software, which is licensed on a subscription basis. They are also known as web-based or hosted software. The consumer of this service does not manage or control the underlying cloud infrastructure (servers, network, storage, operating system)
  • Platform as a Service (PaaS). This application platform allows the consumer to build, run and manage applications without the complex building of the application infrastructure to launch the applications. Like with SaaS, the consumer doesn’t manage or control the underlying cloud infrastructure, but has the control over the deployed applications.
  • Infrastructure as a Service (IaaS). IaaS provides the customer fundamental resources like compute, storage and network where they are able to deploy and run software in virtual machines or containers. The consumer doesn’t manage the underlying infrastructure, but manages the virtual machines including the operating systems and applications.

Deployment Models

There are four cloud computing deployment models defined today and mostly we talk only about three (I excluded the community cloud) of them. Let’s consult the VMware glossary for each definition.

  • Private Cloud. Private cloud is an on-demand cloud deployment model where cloud computing services and infrastructure are hosted privately, often within a company’s own data center using proprietary resources and are not shared with other organizations. The company usually oversees the management, maintenance, and operation of the private cloud. A private cloud offers an enterprise more control and better security than a public cloud, but managing it requires a higher level of IT expertise.
  • Public Cloud. Public cloud is an IT model where on-demand computing services and infrastructure are managed by a third-party provider and shared with multiple organizations using the public Internet. Public cloud service providers may offer cloud-based services such as infrastructure as a service, platform as a service, or software as a service to users for either a monthly or pay-per-use fee, eliminating the need for users to host these services on site in their own data center.
  • Hybrid Cloud. Hybrid cloud describes the use of both private cloud and public cloud platforms, which can work together on-premises and off-site to provide a flexible mix of cloud computing services. Integrating both platforms can be challenging, but ideally, an effective hybrid cloud extends consistent infrastructure and consistent operations to utilize a single operating model that can manage multiple application types deployed in multiple environments.

Hybrid Cloud Model

Multi-Cloud is a term for the use of more than one public cloud service provider for virtual data storage or computing power resources, with or without any existing private cloud and on-premises infrastructure. A multi-cloud strategy not only provides more flexibility for which cloud services an enterprise chooses to use, it also reduces dependence on just one cloud vendor. Multi-Cloud service providers may host three main types of services IaaS, PaaS and SaaS.

With IaaS, the cloud provider hosts servers, storage and networking hardware with accompanying services, including backup, security and load balancing. PaaS adds operating systems and middleware to their IaaS offering, and SaaS includes applications so that nothing is hosted on a customer’s site. Cloud providers may also offer these services independently.

Note: It is very important to understand which cloud computing deployment is the right one for your organization and which services your IT needs to offer to your internal or external customers.

Essential Characteristics

If you look at the five essential cloud computing characteristics from the NIST (National Institute of Standards and Technology), you’ll find attributes which you would also consider as natural requirements for any public cloud (e.g. Azure, Google Cloud Platform, Amazon Web Services):

  • On-demand self-service. A consumer can unilaterally provision computing capabilities,
    such as server time and network storage, as needed automatically without
    requiring human interaction with each service’s provider.
  • Broad Network Access. Capabilities are available over the network and accessed through
    standard mechanisms that promote use by heterogeneous thin or thick client
    platforms (e.g. PCs, laptops, smartphones, tablets).
  • Resource Pooling. The provider’s computing resources are pooled to serve multiple
    consumers using a multi-tenant model, with different physical and virtual
    resources dynamically assigned and reassigned according to consumer demand.
    There is a sense of location independence in that the customer generally has no
    control or knowledge over the exact location of the provided resources but may be
    able to specify location at a higher level of abstraction (e.g., country, state, or
    data center).
  • Scalability and Elasticity. Capabilities can be rapidly and elastically provisioned, in some cases
    automatically, to quickly scale out and rapidly released to quickly scale in. To the
    consumer, the capabilities available for provisioning often appear to be unlimited
    and can be purchased in any quantity at any time.
  • Measure Service. Cloud systems automatically control and optimize resource use by
    leveraging a metering capability at some level of abstraction appropriate to the
    type of service (e.g., storage, processing, bandwidth, and active user accounts).
    Resource usage can be monitored, controlled, and reported providing
    transparency for both the provider and consumer of the utilized service.

And besides the five essentials, you look for security, flexibility and reliability. With all these properties in mind, you would follow the same approach today, if you build a new data center or have to modernize your current cloud infrastructure. A digital foundation, or a platform, which can adopt to any changes and serve as expected.

5 Characteristics of Cloud Computing

This is why VMware has built VMware Cloud Foundation! This is why we need VCF, which is the core of VMware’s multi-cloud strategy.

To be able to meet the above characteristics/criteria, you need a set of software-defined components for compute, storage, networking, security and cloud management in private and public environments – also called the software-defined data center (SDDC). VCF makes operating the data center fundamentally simpler by bringing the ease and automation of the public cloud in-house by deploying a standardized and validated architecture with built in lifecycle management and automation capabilities for the entire cloud stack.

As automation is already integrated and part from the beginning, and not something you would integrate later, you are going to be able to adopt to changes and have already one of the elements in place to achieve the needed security requirements. Automation is key to provide security through the whole stack.

In short, Cloud Foundation gives you the possibility and the right tools to build your private cloud based on public cloud characteristics and also an easy path towards a hybrid cloud architecture. Consider VCF as VMware’s cloud operating system, which enables a hybrid cloud based on a common and compatible platform that stretches from on-premises to any public cloud. Or from public cloud to another public cloud.

Note: VMware Cloud Foundation can also be consumed as a service (aka SDDC as a service) through their partners like Google, Amazon Web Services, Microsoft and many more.

Why Hybrid or Multi-Cloud?

A hybrid cloud with a consistent infrastructure approach enables organizations to use the same tools, policies and teams to manage the cloud infrastructure, which hosts the virtual machines and containers.

Companies want to have the flexibility to deploy and manage new and old applications in the right cloud. They are looking for an architecture, which allows them to migrate on-premises workloads to the public cloud and modernize these applications (partially or completely) with the cloud provider’s native services.

Customers have changed their perception from cloud-first to a cloud-appropriate strategy where they choose the right cloud for each specific application. And to avoid a vendor lock-in, you suddenly see two or three additional public clouds joining the cloud architecture, which by definition now is a multi-cloud environment.

Now you have a mix of a VMware-based cloud with AWS, Azure and GCP for example. It is possible to build new applications in one of the VMware “SDDC as a service” (e.g. VMware Cloud on AWS, Azure VMware Solution, Google Cloud VMware Engine) offerings, but customers also want deploy and use cloud-native service offerings.

Multi-Cloud Reality

How you deal with this challenge with the different architectures, operational inconsistencies, varying skill sets or your people, different management and security controls and incompatible technology formats?

Well, the first answer could be, that your IT needs to be able to treat all clouds and applications consistently and run the VCF stack ideally in any (private or public) cloud.

But this is not where I want to head to. There is something else, which we need to transform in this multi-cloud environment.

We only have consistent infrastructure with consistent operations, because of VMware Cloud Foundation, so far.

  • How does your deployment and automation model for your virtual machines and containers look like now?
  • How would you automate the provisioning these workloads and needed application components?

With your current tool set you have to talk four “languages” via the graphical management console or API (application programming interface).

In an international organization, where people come from different countries and talk different languages, we usually agree to English as corporate language. VMware is following the same approach in this case and puts an abstraction layer above the clouds and expose the APIs.

VMware Cloud-Agnostic CMP

This helps to manage the different objects and workloads you have deployed in any cloud. You don’t have to use your cloud accounts anymore and can define a consistent and centralized team and permission structure as well.

On top of this cloud-agnostic API you can provide all means for a self-service catalog, use programmable provisioning and provide the operations (e.g. cost or log management) and visibility (powered by artificial intelligence where needed) tool set (e.g. application and networks) to build, run, manage, connect and protect your applications.

Your applications, which are part of the different main services (IaaS, PaaS, SaaS) and most probably many other services (like DaaS, DBaaS, FaaS, DRaaS, CaaS, Backup as a Services, MongoDB as Service etc.) you are going to offer to your internal consumers or customers, are deployed via this cloud abstraction layer.

VMware CMP and Services

This abstraction layer forms the VMware cloud management platform (CMP), which consists of the vRealize Suite and VMware Cloud Services. This CMP also provides you with the necessary interfaces and integration options to other existing backend services or tools like a ticketing system, change management database (CMDB), IP address management (IPAM) and so on.

In short this means, that the VMware cloud operation model treats each private or public cloud as a landing zone.

VMware Cloud Foundation Is More About Business Value

Yes, Cloud Foundation is a very technical topic and most people see it only like that. But the hidden and real value are the ones nobody sees or talk about. The business values and the fact, that you can operate your private cloud with the ease like a public cloud provider and that you can follow the same principles for any cloud delivery model.

On-Demand self-service is offered through the lifecycle management capabilities VCF has included in combination with the cloud-agnostic API from VMware’s cloud management platform.

Broad network access starts with VMware’s digital workspace offerings and ends in the data center, at the edge or any cloud with their cloud-scale networking portfolio, which includes software-defined networking (SDN), software-defined WAN (SD-WAN) and software-defined application delivery controller (SD-ADC).

Multi-tenancy and resource pooling can only be achieved with automation and security. Two items which are naturally integrated into Cloud Foundation. The SDDC management component of VCF also gives you the technical capability to create your regions and availability zones. Something a public cloud providers let’s you choose as well.

Rapid elasticity is provided with the hardware-agnostic (for the physical servers in your data centers) approach VMware offers to their customers. Besides that, all cloud computing components are software-defined, which can run on-premises, at the edge or in any public cloud, which allows you to quickly scale out and scale in according to your needs.

Service usage and resource usage (compute, storage, network) are automatically controlled and optimized by leveraging some level of abstraction of all different clouds. Resource usage can be monitored and reported in a transparent way for the service provider and the consumer.

VMware Multi-Cloud Services

In addition to that, VMware provides their customers the choice to consume the VMware operation tools on-premises or as a SaaS offering, which is then hosted in the cloud. With perpetual and subscription licenses you can define your own pay-per-use or pay-as-you-go pricing options and if you want to move from a CAPEX to a OPEX cost model. The same will be true somewhen for VCF and VCF in the public cloud as well. A single universal license which allows you to run the different components and tools everywhere.

Customers need the flexibility to build the applications in any environment, matching the needs of the application and the best infrastructure. They need to manage and operate different environments as one, as efficiently as possible, with common models of security and governance.

Customers need to shift workloads seamlessly between cloud providers (also known as cross-cloud workload mobility) without the cost, complexity or risk of rewriting applications, rebuilding process or retraining IT resources.

And that’s my simple explanation of VMware Cloud Foundation and why it so important and the core of the VMware (Multi-Cloud) strategy.

Let me know what you think! 🙂

A big thank you to my colleagues Christian Dudler, Gavin Egli and Danny Stettler who reviewed my content and illustrations.

Google Cloud VMware Engine (GCVE)

Google Cloud VMware Engine (GCVE)

Sep 17, 2020 | Citrix, Google Cloud, HCX, Horizon, NSX, VCPP, VMware Cloud, VMware Cloud Foundation, vSAN, vSphere

In June 2020 VMware and Google made the announcement that Google Cloud VMware Engine (GCVE) is generally available. Almost exactly one year ago, the market received the information that VMware’s Cloud Foundation (vSphere, vSAN and NSX) stack will come to Google Cloud.

With this milestone VMware is now present on top of all the so-called “big three” hyperscalers.

GCVE has the same goals like the other similar offerings like VMware Cloud on AWS or Azure VMware Solution and belongs to the VMware multi-cloud strategy – to seamlessly migrate and run applications in the public cloud. In this case in Google Cloud! Run your applications in the public cloud exactly the same way as you already do now withh your on-premises VMware environment. With the very important addition, that you have high speed access to Google Cloud services like Cloud SQL, Cloud Storage, big data or AI/ML services.

To be able to run VMware workloads on top of the Google Cloud global infrastructure, Google acquired CloudSimple (with which they partnered with already) last November 2019.

At the moment of writing, the VMware hybrid cloud experience on Google Cloud is sold, operated and supported by Google and their partners.

Many customers are already looking at this very interesting offer, which is going to be available in more regions until the end of 2020. But there are also already a few customers using the joint offering. Google just published a customer reference story about the “Deutsche Börse Group”, a large and international financial organization, which extended their on-premises environment to Google Cloud with Google Cloud VMware Engine. One of the reasons why Deutsche Börse went for this vSphere-based cloud approach, was, to keep migrations to the cloud easy. I expect we can hear more about this success story at VMworld 2020.

Cloud Migration and Workload Mobility

A lot of customers underestimate the amount of work, time and costs involved in refactoring or re-platforming applications and the overall challenges when it comes to migrations from on-prem to the cloud. To build this secure hybrid cloud extension with GCVE, you’ll need VMware HCX, which is included in the GCVE offering.

There are different options available to connect both worlds:

GCVE Connectivity Options

  • VPN Gateway for point-to-point connections, used for the secure admin access to vCenter. Useful for the initial setup of the GCVE environment.
  • Cloud VPN for site-to-site connections, a secure layer 3 connection over the internet. This is one of the lower cost options for use cases, that don’t require high bandwidth.
  • Dedicated Cloud Interconnect with a direct traffic flow to Google with 10Gbps or 100Gbps circuits with 50Mbps to 50 Gbps connection capacities. This direct connection is required for HCX and the preferable connectivity option for customers requiring high speed and low latency.
  • Partner (Cloud) Interconnect is another option of a Cloud Interconnect, where your traffic flows through one of the supported service providers (e.g. Colt, Equinix, BT, e-shelter, Verizon, InterCloud, Interxion, Megaport)

Note: One unique feature of GCVE is the ability to route between different GCVE environments in the same region, without the need for additional configuration. 

Use Cases

These use cases, if you made yourself already familiar with a hybrid cloud approach, shouldn’t be new to you.

Data Center extension or retirement. You can scale the data center capacity in the cloud on-demand, if you for example don’t want to invest anymore in your on-premises environment. In case you just refreshed your current hardware, another use case would be the extension of your on-premises vSphere cloud to Google Cloud.

Disaster Recovery and data protection. Here we’ll find different scenarios like recovery (replication) or backup/archive (data protection) use cases. You can also still use your existing 3rd party tools from Zerto or Veeam to replace or complement existing DR locations and leverage the Cloud Storage service. You can also use your GCVE private clouds as a disaster recovery (DR) site for your on-premises workloads. This DR solution would be based on VMware Site Recovery Manager (SRM) which can be also used together with HCX.

Cloud migrations or consolidation. If you want to start with a lift & shift approach to migrate specific applications to the cloud, then GCVE is definitely right for you. Maybe you want to refresh your current infrastructure and need to relocate or migrate your workloads in an easy and secure way? Another perfect scenario would be the consolidation of different vSphere-based clouds.

Application modernization. Re-architecting or refactoring applications is not that easy. Most customers start with a partial approach to modernize their applications and leverage cloud-native services (e.g. databases, AI/ML engines).

Interesting: Did you know that Google’s on-prem GKE (Google Anthos) is running on vSphere?

VMware Horizon on VMware Engine

The advantages of a public cloud like Google Cloud are the “endless” capacity, agility and high-bandwidth connections. These items are very important for a virtual desktop infrastructure (VDI) and specially during disaster scenarios, when onboardings have to happen fast or if you look for on-demand growth.

Another regular example could be a merger & acquisition use case, where we the main infrastructure doesn’t have the necessary physical resources to onboard to new company and their employees.

Because something like this has always happen as easy and fast as possible. Running virtual desktops in Google Cloud VMware Engine can help in such situations. Together with VMware Horizon, organizations could install a VDI environment in GCVE and connect it to their Horizon on-premises infrastructure using the Cloud Pod Architecture (CPA). 

Note: When migrating applications to the cloud (GCVE), it is a best practice to keep the virtual desktop close to the application, which is a general use case we see when talking about application locality.

Horizon Global Pod GCVE

With the release of Horizon 2006 (aka Horizon 8) it is also possible to choose “Google Cloud” as deployment option during the connection server installation.

C:\\Users\\mrebmann\\OneDrive - VMware, Inc\\cloud13\\2020 - Google Cloud VMware Engine\\Horizon on GCVE.png

In case you need a load balancer (for your Horizon components and in general) for your on-premises environment and the public cloud, have a look at NSX Advanced Load Balancer.

GCVE Node Specs

When planning your GCVE resource needs, be aware of the following specifications and limits:

CPU: Intel Xeon Gold 6240 (Cascade Lake) 2.6 GHz (x2), 36 Cores, 72 Hyper-Threads

Memory: 768 GB

Storage (vSAN): 2 × 1.6 TB (3.2 TB) NVMe (Cache), 6 × 3.2 TB (19.2 TB) NVMe (Data)

Number of nodes required to create a private cloud: 3 (up to 64 hosts per private cloud)

Number of nodes allowed in a cluster on a private cloud: 16

3rd party tools compatibility: Yes, you can use existing tools (elevated privileges allow you to install 3rd party software)

Interesting facts: It only takes about a half hour to spin up your private cloud with three nodes! The addition of a new node takes approximately 15 minutes.

GCVE Elevated Privileges

Software License and Versions

Please find the current software versions and licenses below used for the GCVE offering (purchased with a 1- or 3- year commitment). The listed software versions are fixed and all updates are managed by Google. Google is responsible for the lifecycle management of the VMware software, which includes ESXi, vCenter and NSX.

Component Version License
vCenter 6.7 U3 vCenter Standard
ESXi 6.7 U3 Enterprise Plus
vSAN 6.7 U3 Enterprise
NSX Data Center (NSX-T) 2.5.1 Advanced
HCX 3.5.3 Advanced

Shared Responsibilities

Google Cloud VMware Engine is coming with all components you need to securely run VMware natively in a dedicated private cloud. Google takes care of the infrastructure (service) and their native service integrations. As a customer you only need to take care of your virtual machines or containers with your applications and data. Besides that, you also need to make sure that your configurations, policies, network portgroups, authentication and capacity management are properly configured.

GCVE Shared Responsibilities

If you want to know and learn more about Google Cloud VMware Engine, have a look at the following resources: 

Multi-Cloud Load Balancing and Autoscaling with NSX Advanced Load Balancer (formerly Avi Networks)

Multi-Cloud Load Balancing and Autoscaling with NSX Advanced Load Balancer (formerly Avi Networks)

Sep 14, 2020 | AWS, Azure, Google Cloud, Horizon, Networking, NSX, VMC, VMware Cloud, VMware Cloud Foundation

Do you want to build your private cloud like a hyperscaler is doing it? You know that VMware Cloud Foundation is becoming the new vSphere, but still wonder how you can implement software-defined load balancing (LB) or application services and features like autoscaling or predictive scaling? Then this article about multi-cloud load balancing and autoscaling with NSX Advanced Load Balancer aka Avi Networks is for you!

My Experience with a Legacy ADC

A few years ago, I was working on the customer side for an insurance company in Switzerland as a Citrix System Engineer. My daily tasks included the maintenance and operation of the Citrix environment, which included physical and virtual Citrix NetScaler Application Delivery Controller (ADC) appliances. The networking team owned a few hardware-based appliances (NetScaler SDX) with integrated virtualization capability (XenServer as hypervisor) to host multiple virtual NetScaler (VPX) instances.

The networking team had their dedicated NetScaler VPX instances (for LDAP and HTTP load balancing mostly) and deployed my appliances after I filed a change request. Today, you would call this multi-tenancy. For a Citrix architecture is was best practices to have one high availability (HA) pair for the internal and one HA pair for the external (DMZ) network access. A HA pair was running in a active/passive mode and I had to maintain the same setup for the test environment as well.

Since my virtual VPX appliances were hosted on the physical SDX appliance, I always relied on the network engineers, if I needed more resources (CPU, RAM, SSL, throughput) chips allocated to my virtual instances. Before I could upgrade to a specific firmware version, I also had to wait until they upgraded the physical NetScaler appliances and approved my change request. This meant, we had to plan changes and maintenance windows together and had to cross fingers, that their upgrade went well, that we could upgrade all our appliances after.

NetScaler SDX

It was also possible to download a VPX appliance, which could run on top of VMware vSphere. To be more independent, I decided to install four new VPX appliances (for the production and test environment) on vSphere and migrated the configuration from the appliances running on the physical SDX appliance.

Another experience I had with load balancers was when I started to work for Citrix as a consultant in Central Europe and had to perform a migration of physical NetScaler MPX appliances, which had no integrated virtualization capability. I believe I had two sites with each two of these powerful MPX appliances for tens of thousands of users. Beside the regular load balancing configuration for some of the Citrix components, I also had to configure Global Server Load Balancing (GSLB) in active/passive mode for the two sites.

NetScaler GSLB Active Passive

There were so many more features available (e.g. Web Application Firewall, Content Switching, Caching, Intrusion Detection), but I never used anything else than the NetScaler Universal Gateway for the remote access to the virtual desktop infrastructure (VDI), load balancing, HTTP to HTTPS redirections and GSLB. In all scenarios I had a HA pair where one instance was idling and doing nothing. And the active unit was in average not utilized more than 15-20%. It was common to install/buy too large or powerful instances/licenses, because you wanted to be on the safe side and have enough capacity to terminate all your SSL sessions and so on.

It (load balancing) was about distributing network traffic across multiple servers by spreading the requests and work evenly, and do add some intelligence (health monitoring) in case an application server or a service would fail or be unavailable for any reason. If one more application server was needed, I ordered a new Windows Server, installed and configured the Citrix components and added the necessary load balancing configuration on the NetScaler. These were all manual tasks. The same work has been done by the network engineers when the application team requested a new application server, which then had to be added to the load balancing configuration on their NetScaler appliances.

This was my personal experience from 2017. Since then applications became more complex and distributed. The analysts and market are focusing on containerized and portable apps running and more and more in multiple clouds. The prediction is also that the future is multi-cloud.

Multiple Clouds vs. Multi-Cloud

There are different definitions and understandings out there what multi-cloud means. In my understanding, using a private cloud, AWS, Microsoft Office 365 and Azure are a typical setup with multiple clouds. There are simple scenarios where you migrate workloads from the private to the public cloud (e.g. Azure) or having applications with services lying on the private on the public cloud. The latter would be an example of a hybrid cloud architecture.

The reasons for which services and resources are needed or distributed on multiple clouds (on-prem, Azure, AWS, GCP etc.) are various:

  • Avoid dependence on only one cloud provider
  • Consume different specific services that are not provided elsewhere
  • Optimize costs for different workloads and services
  • React to price changes by the providers

That is why we are seeing also the trend to break up big legacy applications (monoliths) in smaller pieces (segments), which is a best practice and design principle today. The goal is to move to a loosely coupled and more service-oriented architecture. This provides greater agility, more flexibility and easier scalability, because of less inter-dependencies.

And, if we take the second example from the list before, a segmented application is much easier to run in different clouds (portability). Running one application over multiple clouds is in my understanding the right definition of multi-cloud.

Multiple Clouds versus Multi-Cloud

Let’s assume that most probably all the four reasons above apply to larger enterprises. If we take another angle, we can define some business and technical requirements for multi-cloud:

  • Application or services need to be cloud vendor-agnostic
  • Provide or abstract control and management interface of multiple clouds
  • Support application portability/relocation between clouds
  • Combine IaaS and services from different clouds
  • Possibility to deploy components of applications in multiple clouds
  • (Cloud) Broker service needed
  • Policy and governance over multiple clouds
  • Network connectivity for migration scenarios with partially modernized applications
  • Automated procedures for deployments
  • Application monitoring over different clouds
  • Costs management
  • Lifecycle management of deployed applications in multiple clouds
  • Self-adaption and auto scaling features
  • Large team with various expertises needed

How can you deliver and manage the different applications services like load balancing, web application firewalling, analytics, automation and security over multiple clouds?

Another important question would be, how you want to manage the deployment on the various clouds. But cloud management or a cloud management platform is something for another article. 🙂

The requirements for the developers, operations and the business are very complex and it’s a long list (see above).

It is important, that you understand, based on the requirements for multi-cloud, that it is mandatory to implement a modern solution for your modernized application architectures. Enterprises have become more application-centric and everyone is talking about continious integration, continuous delivery and DevOps practices to automate operation and deployment tasks. A modern solution implicits a software-defined approach. Otherwise you won’t be able to be agile, adapt to changes and meet future requirements.

My past experience with Citrix’ NetScaler is a typical example that “virtualized” and “software-defined” are not the same thing. And this is very important if we want to have a future-ready solution. If we look at VMware’s software-defined data center (SDDC), beside the virtual compute, also includes software-defined storage and networking. Part of the software-defined networking portfolio is “NSX Advanced Load Balancer“, the software-defined application services platform, which was also known as “Avi Networks” before VMware bought them in June 2019.

Unlike a virtualized load-balancing appliance, a software-defined
application services platform separates the data and control planes
to deliver application services beyond load balancing, real-time
application analytics, security and monitoring, predictive autoscaling,
and end-to-end automation for Transport (Layer 4) to
Application (Layer 7) layer services. The platform supports multicloud
environments and provides software-defined application
services with infrastructure-agnostic deployments on bare metal
servers, virtual machines (VMs), and containers, in on-premises
data centers and private/public clouds.

Autoscaling became famous with AWS as it monitors your applications and automatically adjusts capacity to maintain availability and performance at the lowest possible costs. It automatically adds or removes application servers (e.g. EC2 instances), load balancers, applies the right network configuration and so on.

Can you achieve the same for your on-premises infrastructure with VMware? Yes.

Is there even a solution which can serve both worlds – on-prem and cloud? Yes.

And what about predictive scaling with real-time insights? Yes.

NSX Advanced Load Balancer (NSX ALB)

Why did VMware buy Avi? Because it follows the same architecture principles like NSX: A distributed platform with a separate control and data plane built on software-defined principles for any cloud.

Avi High Level Architecture

Traditional ADCs or load balancers are mostly configured in active/standby pairs, no matter if physical or virtual. Typically you would see around 15% utilization on the active node where the secondary standby node is just idling and doing nothing. Each pair is its own island of static capacity which shares the management, control and data plane.

You have to decide where to place the virtual IP (VIP) and how much you want to overprovision the physical or virtual appliances, because there is no capacity pooling available. This leads to operational complexity, especially when you have hundreds of such HA pairs running in different clouds. Therefore, legacy and virtualized ADCs are not the ideal choice for a multi-cloud architecture. Let’s check NSX ALB’s architecture:

Control Plane – This is the brain (single point of management) of the whole platform that can be spun up in your on-prem environment or in the cloud (also available as a managed SaaS offering), typically as a three-node cluster. Within this cluster, all configuration is done, this also where the policies reside and the decisions are made. It is the controller’s duty to place virtual services on SEs to load balance new applications or increase the capacity of running applications.

The control plane comprises the three pillars that deliver the key capabilities of the Avi platform:

  • Multi-Cloud – Consistent experience for any cloud, no lock-in
  • Intelligence – The machine learning based analytics engine enables application performance monitoring, troubleshooting, and operational insights (gathered by the SEs)
  • Automation – Elastic and predictive auto scaling & self-service without over-provisioning through a complete set of REST APIs

Data Plane –  The Service Engines (SEs) handle all data plane operations by receiving and executing instructions from the controller. The SEs perform load balancing and all client- and server-facing network interactions. It collects real-time application telemetry from application traffic flows. 

As already mentioned, NSX ALB can be deployed in multiple cloud environments like VMware vCenter, Amazon Web Services, Microsoft Azure, Google Cloud Platform, Oracle Cloud, IBM Cloud, VMC on AWS, Nutanix, OpenStack or bare-metal.

Use Cases

Most customers deploy Avi because of:

  • Load Balancer refresh
  • Multi-Cloud initiatives
  • Security including WAF, DDoS attack mitigation, achieve compliance (GDPR, PCI, HIPAA)
  • Container ingress (integrates via REST APIs with K8s ecosystems like GKE, PKS (TKGI), OpenShift, EKS, AKS, TKG)

Advanced Kubernetes Ingress Controller Avi Networks

  • Virtual Desktop Infrastructure (Citrix, VMware Horizon)

Consistent Application Services Platform (Features)

Avi/NSX ALB is an enterpise-grade solution. So, everything you would expect from a traditional ADC (e.g. F5), layer 4 to layer 7 services, SSL, DDoS, WAF etc. is built-in without the need for a special license edition. There is also no NSX license requirement even the product name would suggest it. It can be deployed as a standalone load balancer or as an integrated solution with other VMware products (e.g. VCF, vRA/vRO, Horizon, Tanzu etc.).

Avi Networks Features

Below is a list with the core features:

  • Enterprise-class load balancing – SSL termination, default gateway, GSLB, DNS, and other L4-L7 services
  • Multi-cloud load balancing – Intelligent traffic routing across multiple sites and across private or public clouds
  • Application performance monitoring – Monitor performance and record and replay network events like a Network DVR
  • Predictive autoscaling Application and load balancer scaling based on real-time traffic patterns
  • Self-service – For app developers with REST APIs to build services into applications
  • Cloud connectors – VMware Cloud on AWS, SDN/NFV controllers, OpenStack, AWS, GCP, Azure, Linux Server Cloud, OpenShift/Kubernetes
  • Distributed application security fabric – Granular app insights from distributed service proxies to secure web apps in real time
  • SSO / Client Authentication – SAML 2.0 authentication for back-end HTTP applications
  • Automation and programmability – REST API based solution for accelerated application delivery; extending automation from networking to developers
  • Application Analytics – Real-time telemetry from a distributed load balancing fabric that delivers millions of data points in real time

Load Balancing for VMware Horizon

NSX ALB can be configured for load balancing in VMware Horizon deployments, where you place SEs in front of Unified Access Gateways (UAG) or Connection Servers (CS) as required.

Avi Horizon High Level Architecture

For a multi-site architecture you can also configure GSLB if needed. With GSLB, access to resources is controlled with DNS queries and health checking.

Note: If you are using the Horizon Universal Broker, the cloud-based brokering service, there is no need for GSLB, because the Universal Broker can orchestrate connections from a higher level based on different policies.

Automation

With NSX Advanced Load Balancer there are two parts when we talk about automation. One part is about infrastructure automation, where the controller talks to the ecosystem like a vCenter, AWS or Azure to orchestrate the Service Engine. So, when you configure a new VIP, the controller would talk to vCenter to spin up a VM, put it in the right portgroup, connect the front and the back-end, download the policy and service engine, and starts receiveing traffic.

The second piece of automation focuses more on the operational automation which is through the REST API (the UI and CLI don’t offer all the configuration, 100% can be done via REST API). But, on top of that you can also run Ansible playbooks, Terraform templates, Go and Python SDKs, have integrations with Splunk or other tools like vRealize Automation. This is the built-in automation in the product.

Avi Networks Automation

VMworld 2020 Sessions

This year VMworld is going to be for free and virtual. Take this chance and register yourself and learn more about Avi aka NSX ALB:

  1. Making Your Private Cloud Network Run Like a Public Cloud – Part 2 [VCNC2918]
  2. Modern Apps and Containers: Networking and Security [VCNC2920]
  3. Prepare for the New Normal of Work from Anywhere [VCNC2919]

Expectations and Current Approaches

There is the general understanding and need for hybrid or multi-cloud architectures. Different people will tell different stories and give different advices. The result are different architectures and different approaches. Some people will tell you, that you can use a cloud serially, so moving from one cloud to another. Or, simultaneously, when using different services from different clouds.

My last article focused on hybrid cloud, the architecture with some services lying on the private infrastructure, while other services are hosted on a public cloud. A public cloud providers tells you, that you can buy all services from them and tries to give you a better discount than the competition (to avoid multiple clouds). Enterprises see the need for multiple (public) clouds to avoid a vendor lock-in instead of going all-in with just one of them.

VMware is about multi-cloud and workload mobility, with the vision, that their VCF stack is running everywhere in the future. Now, some people would now say that this is also a vendor lock-in. Depending on your strategy and technology choices and preferences (e.g. databases, AI/ML services, virtual desktops), you have to decide somewhen which (cloud) vendor, approach and operation model is the right one for you.

It may not true for every large environment, but if you go for multiple clouds, multiple technologies, management and security consoles, architecture and so on, you’ll spend a lot of time and money on engineering and keeping your environment “integrated” and functional.

VMware offers you choice. The choice to run your workloads today and tomorrow wherever you want.

If you have the same vision and strategy like VMware, then you are looking for solutions which run in or on top of every cloud. Because of that it’s very important to understand the different between multiple clouds and multi-cloud.

In this case, NSX ALB brings you multi-cloud load balancing and auto scaling features for any cloud and for multi-cloud enabled applications and services.

Don’t forget: Some people are also saying,  that multi-cloud is not needed and doesn’t exist in reality. Nobody is saying multi-cloud is a piece of cake, but VMware can definitely help you to abstract this complexity. And part of this abstraction can be handled with vRealize Automation for example, which can act as a cloud broker to deploy your application and services.