Select Page

VMware’s Tanzu Kubernetes Grid

Since the announcement of Tanzu and Project Pacific at VMworld US 2019 a lot happened and people want to know more what VMware is doing with Kubernetes. This article is a summary about the past announcements in the cloud native space. As you already may know at this point, when we talk about Kubernetes, VMware made very important acquisitions regarding this open-source project.

VMware Kubernetes Acquisitions

It all started with the acquisition of Heptio, a leader in the open Kubernetes ecosystem. With two of the creators of Kubernetes (K8s), namely Joe Beda and Craig McLuckie, Heptio should help to drive the cloud native technologies within VMware forward and help customers and the open source community to accelerate the enterprise adoption of K8s on-premises and in multi-cloud environments.

The second important milestone was in May 2019, where the intent to acquire Bitnami, a leader in application packaging solutions for Kubernetes environments, has been made public. At VMworld US 2019 VMware announced Project Galleon to bring Bitnami capabilities to the enterprise to offer customized application stacks to their developers.

One week before VMworld US 2019 the third milestone has been communicated, the agreement to acquire Pivotal. The solutions from Pivotal have helped customers learn how to adopt modern techniques to build and run software and they are the provider of the most popular developer framework for Java, Spring and Spring Boot.

On the 26th August 2019, VMware gave those strategic acquisitions the name VMware Tanzu. Tanzu should help customers to BUILD modern applications, RUN Kubernetes consistently in any cloud and MANAGE all Kubernetes environments from a single point of control (single console).

VMware Tanzu

Tanzu Mission Control (Tanzu MC) is the cornerstone of the Tanzu portfolio and should help to relieve the problems we have or going to have with a lof of Kubernetes clusters (fragmentation) within organizations. Multiple teams in the same company are creating and deploying applications on their own K8s clusters – on-premises or in any cloud (e.g. AWS, Azure or GCP). There are many valid reasons why different teams choose different clouds for different applications, but is causing fragmentation and management overhead because you are faced with different management consoles and silo’d infrastructures. And what about visibility into app/cluster health, cost, security requirements, IAM, networking policies and so on? Tanzu MC let customers manage all their K8s clusters across vSphere, VMware PKS, public cloud, managed services or even DIY – from a single console.

Tanzu Mission Control

It lets you provision K8s clusters in any environment and configure policies which establish guardrails. Those guardrails are configured by IT operations and they will apply policies for access, security, backup or quotas.

Tanzu Mission Control

As you can see, Mission Control has a lot of capabilities. If you look at the last two images you can see that you not only can create clusters directly from Tanzu MC, but also have the ability to attach existing K8s clusters. This can be done by installing an agent in the remote K8s cluster, which then provides a secure connection back to Tanzu MC.

We focused on the BUILD and MANAGE layers now. Let’s take a look at the RUN layer which should help us to run Kubernetes consistently across clouds. Without consistency across cloud environments (this includes on-prem) enterprises will struggle to manage their hundred or even thousands of modern apps. It’s just getting too complex.

VMware’s goal in general is to abstract complexity and to make your life easier and for this case VMware has announced the so-called Tanzu Kubernetes Grid (TKG) to provide us a common Kubernetes distribution across all the different environments.

Tanzu Kubernetes Grid

In my understanding TKG means VMware’s Kubernetes distribution, will include Project Pacific as soon as it’s GA and is based on three principles:

  • Open Source Kubernetes – tested and secured
  • Cluster Lifecycle management – fully integrated
  • 24×7 support

Meaning, that TKG is based on open source technologies, packaged for enterprises and supported by VMware’s Global Support Services (GSS). Based on these facts we can say, that today your Kubernetes journey with VMware starts with VMware PKS. PKS is the way we deliver the principles of Tanzu today – across vSphere, VCF, VMC on AWS, public clouds and edge.

Project Pacific

Project Pacific, which has been announced at VMworld US 2019 as well, is a complement to VMware PKS and will be available in a future release. If you are not familiar with Pacific yet, then read the introduction of Project Pacific. Otherwise, it’s sufficient to say, that Project Pacific means the re-architecture of vSphere to natively integrate Kubernetes. There is no nesting or any kind of it and it’s not Kubernetes in vSphere. It’s more like vSphere on top of Kubernetes since the idea of this project is to use Kubernetes to steer vSphere.

Project Pacific

Pacific will embed Kubernetes into the control plane of vSphere and converge VMs and containers on the same underlying platform. This will give the IT operators the possibility to see and manage Kubernetes from the vSphere client and provide developers the interfaces and tools they are already familiar with.

Project Pacific Console

If you are interested in the Project Pacific Beta Program, you’ll find all information here.

I would have access to download the vSphere build which includes Project Pacific, but I haven’t got time at the moment and my home lab is also not ready yet. We hear customers asking about the requirements for Pacific. If you watch all the different recordings from the VMworld sessions about Project Pacific and the Supervisor Cluster, then we could predict, that only NSX-T is a prerequisite to deploy and enable Project Pacific. This slide shows why NSX-T is part of Pacific:

Project Pacific Supervisor ClusterFrom this slide (from session HBI1452BE) we learn that a load balancer built on NSX Edge is sitting in front of the three K8s Control Plane VMs and that we’ll find a Distributed Load Balancer spanned across all hosts to enable the pod-to-pod or east-west communication.

Nobody of the speakers ever mentioned vSAN as a requirement and I also doubt that vSAN is going to be a prerequisite for Pacific.

You may ask yourself now which Kubernetes version will be shipped with ESXi and how you upgrade your K8s distribution? And what about if this setup with Pacific is too “static” for you? Well, for the Supervisor Clusters VMware releases patches with vSphere and you apply them with the known tools like VUM. For your own built K8s clusters, or if you need to deploy Guest Clusters, then the upgrades are easy as well. You just have to download the new distribution and specify the new version/distribution in the (Guest Cluster Manager) YAML file.

Conclusion

We hear rumors that Pacific will be shipped with the upcoming vSphere 7.0 release, which even should include NSX-T 3.0. For now we don’t know when Pacific will be shipped with vSphere and if it really will be included with the next major version. I would be impressed if that would be the case, because you need a stable hypervisor version, then a new NSX-T version is also coming into play and in the end Pacific relies on these stable components. Our experience has shown that the first release normally is never perfect and stable and that we need to wait for the next cycle or quarter. With that in mind I would say that Pacific could be GA in Q3 2020 or Q4 2020. And beside that the beta program for Project Pacific just has started!

Nevertheless I think that Pacific and the whole Kubernetes Grid from VMware will help customers to run their (modern) apps on any Kubernetes infrastructure. We just need to be aware that there are some limitations when K8s is embedded in the hypervisor, but for these use cases we can deploy Guest Clusters anyway.

In my opinion Tanzu and Pacific alone don’t make “the” big difference. It’s getting more interesting if you talk about multi-cloud management with vRA 8.0 (or vRA Cloud), use Tanzu MC for the management of all your K8s clusters, networking with NSX-T (and NSX Cloud), create a container host with a container image (via vRA’s Service Broker) for AI- and ML-based workloads and provide the GPU over the network with Bitfusion.

Bitfusion Architecture

Looking forward to such conversations! 😀

VCAP7-DTM Design Exam, Part 10

In part 10 of my VCAP7-DTM Design exam series we take a look at the Horizon 7 Enterprise Reference Architecture.

To be honest, I didn’t study that much the last two weeks but I checked a few documents about App Volumes, Mirage, ThinApp and User Environment Manager.

This time I would like to summarize what I have learned from the reference architecture and the VMworld 2018 session called Architecting Horizon 7 Enterprise: The Official Reference Architecture (WIN3451BUR).

I only focus on the component design part since I already covered topics like use cases, business drivers, design methodology etc.

Horizon 7

A successful deployment depends on good planning and a very good understanding of the platform. The core elements include Connection Server, Composer, Horizon Agent and Horizon Client. Part 4 to part 9 cover the Horizon 7 component design and also provide more information on the following components.

Horizon 7 Logical Architecture

Identity Manager

VMware Identity Manager (VIDM) can be implemented on-premises or in the cloud, a SaaS-based implementation. If you decide to go with the SaaS implementation, a VIDM connector needs to be installed on-prem to synchronize accounts from Active Directory to the VIDM service in the cloud.

If cloud is no option for you, you still have the possibility for the on-prem deployment and use the Linux-based virtual appliance. There is also a Windows-based installer available which is included in the VMware Enterprise Systems Connector. VMware’s reference architecture is based on the Linux appliance.

VMware Identity Manager Architecture

Syncing resources such as Active Directory and Horizon 7 and can be done either by using a separate VMware Identity Manager Connector or by using the built-in connector of an on-premises VMware Identity Manager VM. The separate connector can run inside the LAN in outbound-only connection mode, meaning the connector receives no incoming connections from the DMZ.

VIDM comes with an embedded PostgreSQL database, but it’s recommended to use an external database server for production deployments.

For high availability, based on your requirements, at least two VIDM appliances should be deployed behind a load balancer. After you have deployed your first appliance, you simply clone it and assign a new hostname and a new IP address.

App Volumes

As you still may know from part 8, App Volumes has two functions. The first is the delivery of applications for VDI and RDSH. The second is the provision of writable volumes to capture user-installed applications and the user profile.

app volumes architecture

For high availability, always use at least two App Volumes Managers which are load-balanced.

AppStacks are very read intensive, hence, you should place AppStacks on storage that is optimized for read operations. Writable volumes should be placed on storage for random IOPS (50/50). There reference architecture uses vSAN to provide a single highly available datastore.

For the SQL database it is recommended using an AlwaysOn Availability Group.

User Environment Manager

When User Environment Manager design decisions need to be made, you have to think about user profiles (mandatory, roaming, local) and folder redirection. As already described in part 9, VMware recommendation is to use mandatory profiles and folder redirection. Use appendix B if you need help configuring the mandatory profile.

vmware user environment manager

The first key design consideration is using DFS-R to provide high availability for the configuration and user shares. Note: Connect the management console only to the hub member when making changes. DFS-R will replicated those changes to the spoke members.

The second consideration one is using GPO loopback processing.

Unified Access Gateway

In part 6 I mentioned that a UAG is typically deployed within the DMZ.

VMware Unified Access Gateway

UAG appliances are deployed in front of the Horizon 7 Connection Servers and sit behind a load balancer. The Unified Access Gateway also runs the Content Gateway as part the AirWatch (WorkspaceONE UEM) service.

You have two sizing options during the appliance deployment:

  • Standard (2 vCPU, 4GB RAM, 2’000 Horizon server connections, 10’000 AirWatch service connections)
  • Large (4 vCPU, 16GB RAM, 2’000 Horizon server connections, 50’000 AirWatch service connections)

As you can see, the big difference here are the estimated AirWatch service connections per appliance. In production you would deploy dedicated UAG appliances for each service. Example:

  • 2 standard size UAGs appliances for 2’000 Horizon 7 sessions (n+1)
  • 3 large size UAG appliances for 50’000 devices using Content Gateway and per-App Tunnel which gives us a total of 100’000 sessions. The third appliance is for high availability (n+1)

vSphere and Physical Environment

The software-defined data center (SDDC) is the foundation that runs all infrastructure servers and components. The products and the licensing for the foundation are outside of the Horizon 7 product (except vSAN), but are required to deliver a complete solution.

And in my opinion this is what makes the whole solution so brilliant. Even I work for VMware, I would never say from the beginning that Horizon is better than XA/XD. This was also the case when I worked as a consultant for Citrix before I joined VMware in May 2018.
It depends on the requirements and use cases which need to be satisfied. That are the most important things if you choose a vendor or a specific technology. Our goal is to make the customer happy! 🙂

But I would say that VMware Horizon including WorkspaceONE is very hard to beat if you use the complete stack! But that’s another topic.

The vSphere infrastructure in the reference architecture includes vSAN and NSX. In part 5 I covered the basics of vSAN, but I think I maybe need to write a short overview about NSX and how you can use it with Horizon.

vSAN provides a hyper-converged storage optimized for virtual machines without the need for an external SAN or NAS. This means that the physical server not only provides the compute and memory resources, but also storage in a modular fashion. You can use vSAN for the management and resource block  and follow a hybrid approach for the management resources and use all-flash vSAN for the Horizon resources.

VMware vSAN

I will not cover the vSphere design, but it’s important to understand that all components are operating redundantly and that you have enough physical resources to meet the requirements.

vSphere Networking

A general recommendation is to use at least 10 GbE connections, to separate each traffic (mgmt, VM traffic, vSAN, vMotion) and make sure that each of them has sufficient bandwidth.

NSX for vSphere

NSX provides several network-based services and performs several security functions within a Horizon 7 implementation:

  • Protects VDI infrastructure
  • Protects desktop pool VM communication with applications
  • Provides user-based access control (user-level identity-based micro-segmentation)

VMware NSX for vSphere

If you want to use NSX you have to think about a NSX infrastructure design as the NSX platform adds new components (e.g. NSX manager) and new possibilities (distributed firewall and identity firewall).

The most important design consideration for Horizon 7 is the concept of micro-segmentation. In the case of Horizon 7, NSX can block desktop-to-desktop communications, which are normally not needed or recommended. Each VM can now be its own perimeter and this desktop isolation prevents threats from spreading:

NSX isolation

The Horizon 7 reference architecture of probably the best document to prepare yourself for the VCAP7-DTM exam. What do the current VCAP7-DTM certified  people say? What else needs to be covered? Jump to part 11

VCAP7-DTM Design Exam, Part 6

This is the sixth part of my VCAP7-DTM Design exam series. In part 5 I covered the creation of a physical design for horizon storage. This time we take a look at section 5 of the blueprint, the creation of a physical network design for Horizon:

Section 5 – Create a Physical Design for Horizon Networking
Objective 5.1 – Plan and Design Network Requirements for Horizon solutions (including Mirage and Workspace One)
Objective 5.2 – Design Network and Security Components Based on Capacity and Availability Requirements
Objective 5.3 – Evaluate GPO and Display Protocol Tuning Options Based on Bandwidth and Connection Limits

Networking is also a very important and exciting when creating a Horizon architecture and a lot of questions are coming up when I think about Horizon and network access and devices:

  • How does the ISP infrastructure look like?
  • Do we have redundant internet uplinks?
  • Bandwidth in the data center?
  • Firewalls?
  • Remote connections?
  • How is the connection between Horizon client and agent?
  • ESXi host network interfaces?
  • Do we have mobile workers using WLAN?

I once had a customer who had a really nice and modern data center infrastructure, but their firewalls didn’t provide enough throughput. Make your homework and know how the routing and switching looks like and check every component’s limit.

Beside our VDI traffic, what about management, vMotion and vSAN traffic? Do we have enough network interfaces and bandwidth? If you think about management traffic, then 1Gbit interfaces are normally sufficient. But vMotion and vSAN traffic should have redundant 10Gbit connections and be on different subnets/VLANs.

Overview of the Network Architecture

In most network architectures two firewalls exist to create the DMZ.

The Unified Access Gateway (UAG) appliances are placed in the DMZ. UAG can perform authentication or pass a connection to the Connection Server for AD authentication.
Notauthenticated sessions are dropped at the Unified Access Gateway appliance and only authenticated sessions are allowed to connect to the internal resources.

UAG appliances in the DMZ communicate with the Connection Server instances inside the corporate firewalls and ensure that only the desired remote apps and desktop sessions can enter the corporate data center on behalf of this strongly authenticated user.

Inside the corporate firewall you install and configure at least two Connection Server instances. Their configuration data is stored in an embedded LDAP directory (AD LDS) and is replicated among all members of the group.

Firewall Ports

On March 22, 2016, an updated network ports diagram has been posted by VMware:

Horizon 7 Network Ports Diagram

On Tech Zone this diagram and all key firewall considerations are available for Horizon 7: https://techzone.vmware.com/resource/network-ports-vmware-horizon-7

Network Bandwidth Considerations

The used session bandwidth between the Horizon client and agent depends highly on the session configuration. For display traffic, many elements can affect network bandwidth, such as the used protocol, monitor resolution, frames per second, graphically intense applications or videos, image and video quality settings.

Because the effects of each configuration can vary widely, it’s recommended to monitor the session bandwidth consumption as part of a pilot. Try to figure out the bandwidth requirements for each use case.

Display Protocol

I would say that Blast Extreme is the way to go, because it has been optimized for mobile devices and can intelligently switch between UDP and TCP (Adaptive Transport). PCoIP has been developed by Teradici, but Blast is VMware’s own creation and that’s why I think that Blast will be “the future” and that RDP still can be used as fallback for some special scenarios.

Display Protocol Tuning Options

I will not cover this topic and explain you how you can configure the maximum bandwidth for PCoIP via GPO. There are several options to decrease and increase the used session bandwidth:

Configuring PCoIP session variables
VMware Blast Policy Settings

WAN Consideration

Nowadays, every client device is connected with 1Gbps. LAN connections and the user experience are most of the time perfect. How is it with WAN connections where you will have latencies that could be between 50 and 200ms? Do you apply Quality of Services (Qos) policies to prioritize Horizon traffic?

WAN optimization is one of the keywords when talking about WAN connections and is valuable for TCP-based protocols which require many handshakes between client and server, such as RDP.
PCoIP is  UDP-based and this was the reason why everyone in the past said, that you should prefer this protocol for connections with higher latencies and then no WAN optimization or acceleration would be needed.

Then inside the corporate network you would use RDP because your network is stable or did you leave this choice to the user?

With Blast Extreme, Adaptive Transport will automatically detect higher latencies and automatically switches between TCP and UDP if needed. Higher latencies could also occur with mobile devices working of WiFi networks.

In my opinion there are almost no reasons anymore to use anything else than Blast because it’s also more network efficient than PCoIP.

pcoip blast extreme comparison

Conclusion

Use separate networks for vSphere management, VM connectivity, vMotion and vSAN traffic. Make sure you have redundancy across different physical adapters (NIC, PCI slot) and devices (switches, router, firewall). Consider the use of a vSphere Distributed Switch (vDS) to reduce management overhead and provide a richer feature set. Maybe NSX could be interesting for micro segmentation.

Load balancing is a very important component of a Horizon architecture. The primary purpose of load balancing is to optimize performance by evenly distributing client sessions across all available Connection Server instances. The same is valid for UAG appliances, Identity Manager or App Volumes Manager. NSX comes with a virtual load balancer, but F5 and NetScaler are also fine.

Depending on your customer’s requirements and needs, the network design is another key part to remove single point of failures.

In part 7 we will figure out how we have to design Horizon desktops and pools.