Eric Sloof

The stability of the NSX Manager cluster is critical for VMware NSX operations. However, situations can occur where all NSX Managers crash simultaneously, leaving the control plane completely unavailable. Recently, such an event took place in my lab environment, and together with Broadcom Support, the exact consequences for the dataplane and operational processes were carefully analyzed.

Control Plane vs. Data Plane

It’s essential to understand the difference between the control plane and the data plane in NSX:

• Data plane continues to forward traffic between virtual machines (VMs). Existing VM-to-VM communication is preserved.

• Control plane, operated by the NSX Manager cluster, distributes ARP/MAC information, segment updates, and coordinates operations such as vMotion and VM power-on.

Immediate Impact of a Full Manager Cluster Outage

When the NSX Manager cluster goes down, the following consequences apply:

1. vMotion and VM Power-On Failures
   vMotion operations and new VM registrations depend on the control plane. Without it, workloads cannot be migrated or powered on, although existing VMs keep running.

2. Loss of ARP Suppression
   Normally, NSX suppresses ARP broadcasts by maintaining ARP/MAC tables. If the control plane fails, NSX falls back to “traditional ARP behavior”:
   → ARP requests are broadcast (flooded) via Geneve tunnels to all transport nodes (TEPs) in the segment.
   This preserves connectivity but reduces efficiency.

3. Aging Timers Trigger Broadcast Floods
   MAC, ARP, and VTEP entries in the dataplane age out after ~10 minutes by default. When they expire while the control plane is down, new ARP broadcasts are required to relearn them.

   • In small segments, this has limited impact.

   • In large segments, repeated flooding can introduce noticeable overhead and potential performance degradation.

4. Provisioning and Migration Issues
   If new VMs are provisioned or migrated during the outage, ESXi hosts may lack up-to-date forwarding information. This can cause temporary connectivity problems until the control plane is restored.

Root Cause and Risks

In this specific case, the outage was linked to a known JDK bug (JDK-8330017) in the Corfu/ForkJoinPool implementation, which caused NSX services to stop executing tasks. (NSX is Impacted by JDK-8330017: ForkJoinPool Stops Executing Tasks Due to ctl Field Release Count (RC) Overflow)

• The bug has been fixed in NSX versions 4.2.1.4 and 4.2.2.1 (and later) including VCF 9.0.

• For older versions, Broadcom engineering recommends a rolling reboot of the NSX Managers as a temporary workaround.

Key Takeaways

• Connectivity remains intact – the dataplane continues forwarding traffic, even if the control plane is fully unavailable.

• New operations fail – vMotion, VM power-on, and dynamic provisioning cannot function without the control plane.

• Fallback = ARP flooding – ensures connectivity but may introduce overhead in large-scale segments.

• Upgrade is essential – stability depends on running patched NSX versions.

Conclusion

A full NSX Manager cluster outage does not immediately disrupt existing workloads but does highlight the risks of control plane dependency. In dynamic or large-scale environments, the operational impact can be significant, ranging from reduced efficiency to failed provisioning.

To mitigate these risks:

• Upgrade to the latest patched NSX release (4.2.1.4 or higher).

• Plan preventive rolling reboots where necessary.

• Incorporate failure behavior into your NSX design and operational procedures.

The world of cloud computing is constantly evolving, and staying ahead of the curve is essential for any IT professional. We are excited to announce a brand-new lineup of training courses from VMware, designed to build expertise for the latest iteration of their powerful hybrid cloud platform: VMware Cloud Foundation 9.0.

These courses are tailored to different roles and responsibilities within the VCF ecosystem, ensuring that whether you are building, managing, designing, or troubleshooting, there’s a path for you to master this technology. Let's take a look at what each course has to offer.

1. VMware Cloud Foundation: Build, Manage and Secure [9.0]

This foundational course is designed for administrators and engineers responsible for the day-to-day management of a VCF environment. It provides the essential skills needed to deploy, configure, and maintain the platform, ensuring a secure and stable private cloud.

This course will likely cover key areas such as:

• Deployment and Configuration: Learn to deploy VCF components like vCenter and ESX, configure vSphere clusters, and manage virtual machines.
• Storage Fundamentals: Get hands-on with vSphere and vSAN storage, including deploying clusters and configuring storage policies for resilience and efficiency.
• Fleet Management: Understand how to manage the entire VCF fleet, including identity and access management (RBAC), license and certificate management, and lifecycle updates.
• Basic Operations: Monitor the health and diagnostics of VCF components, manage log events, and ensure the security and compliance of your environment.

2. VMware Cloud Foundation: Automation and Operations [9.0]

This course is perfect for cloud administrators and automation specialists who want to leverage the full potential of VCF's automation and operational capabilities. The provided course materials for "Automate and Operate" show a deep dive into creating a self-service private cloud experience.

Key topics you can expect to master include:

• VCF Automation Architecture: Understand the core components and architecture that enable automation, including the VCF Management Services Platform and the role of identity providers for single sign-on.
• Building a Private Cloud: Learn to build a private cloud environment from the ground up, configuring provider networking, creating regions, and managing multi-organizational tenancy.
• Self-Service and Consumption: Enable a self-service consumption model by creating and managing content libraries, deploying modern applications, and using the VCF Operations Orchestrator to automate complex workflows.
• Operations and Monitoring: Dive deep into VCF Operations, creating custom views, reports, and dashboards. Learn to configure alerts, monitor logs, and use policies to govern your cloud environment.

3. VMware Cloud Foundation: Troubleshooting [9.0]

When things go wrong, you need the skills to fix them quickly and efficiently. This troubleshooting course is aimed at support staff, experienced administrators, and anyone who needs to diagnose and resolve issues within a VCF 9.0 environment.

This course will equip you with the methodologies to tackle problems across the entire stack:

• Troubleshooting the VCF Fleet: Learn to resolve issues related to the initial deployment, scaling, and upgrading of the VCF fleet and its components.
• Workload Domain Issues: Diagnose problems with creating and scaling workload domains, including adding and removing clusters and hosts.
• Component Troubleshooting: Gain expertise in troubleshooting specific VCF components, including vSAN, NSX, VCF Operations, and the Identity Broker.
• Advanced Diagnostics: Master the use of the Observability Workbench, generate log bundles with Log Assist, and create custom reports and alerts to pinpoint issues effectively.

4. VMware Cloud Foundation: Solution Architecture and Design [9.0]

For cloud architects and consultants, designing a solution that meets business requirements is paramount. This course focuses on the high-level planning and design principles needed to build robust, scalable, and resilient VCF solutions.

This course will cover the critical aspects of solution design:

• Architectural Principles: Differentiate between business requirements and technical constraints, and understand the distinctions between conceptual, logical, and physical designs.
• Designing for VCF: Make key design decisions for VCF fleet topology, network infrastructure, and management and workload domains.
• Core Design Qualities (AMPRS): Design for Availability, Manageability, Performance, Recoverability, and Security by creating solutions that meet specific business continuity and disaster recovery requirements.
• Consumption and Migration Strategy: Develop strategies for workload migration, design a consumption model with tenant automation and self-service, and create a comprehensive monitoring plan for the VCF environment.

These new courses represent a fantastic opportunity to enhance your skills and validate your expertise with VMware Cloud Foundation 9.0. Keep an eye on the official VMware Education Services website for release dates and get ready to take your cloud capabilities to the next level!

VMware by Broadcom has recently announced the release of five new professional certification exams based on version 9 of their Cloud Foundation Platform. These certifications represent a significant evolution in VMware's certification portfolio, reflecting the company's continued innovation in cloud foundation and virtualization technologies under Broadcom's stewardship.

The new certification lineup includes specialized tracks for both VMware Cloud Foundation (VCF) and VMware vSphere Foundation (VVF), covering essential roles from administration and support to architecture. Each certification is designed to validate the skills required for modern cloud infrastructure management, troubleshooting, and design in enterprise environments.

Overview of the New Certification Portfolio

The five new VMware Certified Professional (VCP) certifications are:

1. VMware Certified Professional - VMware vSphere Foundation Support (2V0-18.25)
2. VMware Certified Professional - VMware Cloud Foundation Administrator (2V0-17.25)
3. VMware Certified Professional - VMware Cloud Foundation Support (2V0-15.25)
4. VMware Certified Professional - VMware Cloud Foundation Architect (2V0-13.25)
5. VMware Certified Professional - VMware vSphere Foundation Administrator (2V0-16.25)

These certifications address the growing demand for skilled professionals who can effectively deploy, manage, support, and architect private cloud environments using VMware's latest technologies. Each exam follows a consistent format with 60 multiple-choice questions, a 135-minute duration, and a passing score of 300 points, priced at $250 per attempt.

Detailed Examination of Each Certification

1. VMware Certified Professional - VMware vSphere Foundation Support (2V0-18.25)

The VMware Certified Professional - VMware vSphere Foundation Support certification represents a crucial entry point for IT professionals seeking to specialize in troubleshooting and supporting VMware vSphere Foundation environments.

This certification validates a candidate's proficiency in standard troubleshooting techniques and methodologies specifically tailored to issues encountered during the deployment and operation of VMware vSphere Foundation (VVF). The exam confirms a comprehensive understanding of VVF's architecture, including its core features, functions, components, and operational models.

Key technical areas covered in this certification include VCF Operations, vCenter Server administration, VMware ESX hypervisor management, VMware vSAN storage solutions, and VMware Operations for centralized logging and monitoring. Candidates who pursue this certification typically work in support roles where they are responsible for maintaining system availability and resolving technical issues that may arise in virtualized environments.

The certification is particularly valuable for IT professionals who are expanding their skill set from traditional infrastructure support roles into cloud-based virtualization support. It provides the foundational knowledge necessary to effectively troubleshoot complex virtualized environments and ensure optimal performance of critical business applications.

For those interested in pursuing this certification, the Exam Study Guide provides comprehensive preparation materials, while exam registration can be completed through Broadcom's certification portal.

2. VMware Certified Professional - VMware Cloud Foundation Administrator (2V0-17.25)

The VMware Certified Professional - VMware Cloud Foundation Administrator certification is designed for IT professionals who are responsible for the day-to-day administration and management of VMware Cloud Foundation environments.

This certification validates the comprehensive skills required to deploy, manage, and support private cloud environments built on VMware Cloud Foundation (VCF). The certification is specifically designed for IT professionals who are transitioning from traditional infrastructure roles to cloud administration responsibilities, reflecting the industry's ongoing shift toward cloud-first architectures.

Candidates for this certification typically include system administrators, cloud administrators, and infrastructure specialists who are responsible for implementing and maintaining VCF infrastructure. The certification ensures that professionals can effectively manage VCF environments to meet organizational service level objectives for availability, performance, and security.

The curriculum covers essential aspects of VCF administration, including initial deployment procedures, ongoing management tasks, performance optimization, security configuration, and troubleshooting common issues. By achieving this certification, professionals demonstrate their ability to effectively operate complex VCF environments and contribute to their organization's cloud transformation initiatives.

The certification is supported by recommended training courses including "VMware Cloud Foundation: Build, Manage, and Secure" and "VMware Cloud Foundation: Automate and Operate," which provide hands-on experience with the platform's administrative capabilities.

Professionals can access the Exam Study Guide for detailed preparation information and complete their exam registration through the official Broadcom portal.

3. VMware Certified Professional - VMware Cloud Foundation Support (2V0-15.25)

The VMware Certified Professional - VMware Cloud Foundation Support certification addresses the critical need for specialized support professionals in VMware Cloud Foundation environments.

This certification validates the skills required to support and troubleshoot private cloud environments built on VMware Cloud Foundation (VCF). It is specifically designed for IT professionals who are expanding from existing support infrastructure roles into cloud administration and Site Reliability Engineer (SRE) positions, reflecting the evolving nature of IT support roles in modern organizations.

The certification targets professionals who are responsible for implementing, maintaining, and supporting VCF infrastructure in ways that contribute to organizational service level objectives for availability, performance, and security. This includes proactive monitoring, incident response, performance tuning, and preventive maintenance activities.

Candidates who achieve this certification demonstrate their ability to effectively troubleshoot complex issues within private cloud environments based on VCF. The certification covers advanced troubleshooting methodologies, diagnostic tools and techniques, performance analysis, and resolution strategies for common and complex issues that may arise in VCF deployments.

The recommended preparation includes the "VMware Cloud Foundation: Troubleshooting" course, which provides hands-on experience with diagnostic tools and troubleshooting scenarios commonly encountered in production environments.

The certification is supported by comprehensive study materials available through the Exam Study Guide, and candidates can complete their exam registration through Broadcom's official certification portal.

4. VMware Certified Professional - VMware Cloud Foundation Architect (2V0-13.25)

The VMware Certified Professional - VMware Cloud Foundation Architect certification represents the most advanced level of expertise in VMware Cloud Foundation design and architecture.

This certification validates a candidate's advanced skills in designing comprehensive VCF solutions that address complex organizational requirements. The certification covers critical design areas including manageability, availability, performance, recovery, and security, as well as advanced topics such as monitoring, capacity planning, and migration strategies.

The certification is specifically targeted at infrastructure architects and consultants who are responsible for designing scalable, resilient, and secure cloud foundation solutions. These professionals typically work with organizations to assess requirements, design appropriate solutions, and provide guidance on implementation best practices.

The curriculum encompasses advanced architectural concepts including solution design methodologies, capacity planning and sizing, high availability and disaster recovery design, security architecture, and integration with existing infrastructure. Candidates learn to balance competing requirements such as performance, cost, security, and operational complexity to deliver optimal solutions.

The certification validates expertise in creating comprehensive design documentation, conducting technical reviews, and providing architectural guidance throughout the implementation lifecycle. Professionals who achieve this certification are equipped to lead complex VCF implementation projects and serve as technical authorities within their organizations.

Preparation for this certification is supported by the "VMware Cloud Foundation: Solution Architecture and Design" course, which provides comprehensive coverage of architectural principles and design methodologies.

Detailed preparation materials are available through the Exam Study Guide, and exam registration can be completed through the official certification portal.

5. VMware Certified Professional - VMware vSphere Foundation Administrator (2V0-16.25)

The VMware Certified Professional - VMware vSphere Foundation Administrator certification provides essential validation for professionals responsible for administering VMware vSphere Foundation environments.

This certification validates the skills required to deploy, manage, and support private cloud environments built on VMware vSphere Foundation (VVF). Similar to its Cloud Foundation counterpart, this certification is designed for IT professionals who are expanding from traditional infrastructure roles to cloud administration, but with a specific focus on vSphere Foundation technologies.

The certification targets professionals who are responsible for implementing and maintaining VVF infrastructure while ensuring it meets organizational service level objectives for availability, performance, and security. This includes tasks such as virtual machine management, resource allocation, performance monitoring, and security configuration.

Candidates who pursue this certification typically work in roles where they are responsible for the day-to-day administration of virtualized environments, including virtual machine lifecycle management, storage and network configuration, and user access management. The certification ensures that professionals have the knowledge and skills necessary to effectively operate VVF environments in production settings.

The curriculum covers essential administrative tasks including initial deployment and configuration, ongoing management and maintenance, performance optimization, security implementation, and troubleshooting common issues. By achieving this certification, professionals demonstrate their competency in managing complex virtualized environments and contributing to their organization's infrastructure reliability and performance.

The certification is supported by the "vSphere Foundation: Build, Manage and Secure" course, which provides practical experience with the platform's administrative capabilities and best practices.

Comprehensive study materials are available through the Exam Study Guide, and candidates can complete their exam registration through the official Broadcom certification portal.

Strategic Importance of These Certifications

The introduction of these five new certifications reflects VMware by Broadcom's commitment to maintaining industry leadership in cloud infrastructure technologies. Each certification addresses specific market needs and career paths, providing professionals with clear progression opportunities within the VMware ecosystem.

The certifications are strategically designed to address the growing complexity of modern IT environments, where organizations require specialized expertise in areas such as cloud administration, support, and architecture. By offering distinct certification tracks for different roles and technologies, VMware enables professionals to develop focused expertise that aligns with their career objectives and organizational needs.

Furthermore, the version 9 foundation of these certifications ensures that certified professionals are equipped with knowledge of the latest features, capabilities, and best practices. This is particularly important in the rapidly evolving cloud infrastructure landscape, where staying current with technology developments is essential for professional success.

Conclusion

VMware by Broadcom's release of these five new certification exams represents a significant opportunity for IT professionals to validate their expertise in cutting-edge cloud infrastructure technologies. Each certification provides a distinct pathway for career development, whether in support, administration, or architecture roles.

The comprehensive nature of these certifications, combined with their focus on practical skills and real-world scenarios, ensures that certified professionals are well-prepared to contribute effectively to their organizations' cloud infrastructure initiatives. As organizations continue to adopt and expand their use of VMware technologies, the demand for certified professionals in these areas is expected to grow significantly.

For IT professionals considering these certifications, the investment in preparation and examination fees represents a valuable opportunity to advance their careers and contribute to their organizations' technological success. The certifications provide not only validation of technical skills but also demonstrate commitment to professional development and staying current with industry best practices.

References and Resources

1. VMware Certified Professional - VMware vSphere Foundation Support
2. VMware Certified Professional - VMware Cloud Foundation Administrator
3. VMware Certified Professional - VMware Cloud Foundation Support
4. VMware Certified Professional - VMware Cloud Foundation Architect
5. VMware Certified Professional - VMware vSphere Foundation Administrator

As organizations move toward deploying AI models in production, the demand for private, secure, and scalable inference infrastructure is growing. The NVIDIA “Reference Design for Private AI Inference Using HGX Servers” offers a comprehensive architecture blueprint for building performant inference clusters tailored to enterprise needs.

What’s Inside the Reference Design?

1. Purpose of the Architecture

• Deliver fast, private AI inference across diverse workloads.

• Provide a blueprint optimized for NVIDIA HGX systems using the latest H100 GPUs.

• Address challenges such as data privacy, latency, model scaling, and operational efficiency.

2. Hardware Foundation: NVIDIA HGX Platform

• Based on the NVIDIA HGX system with 8x H100 GPUs interconnected using NVLink.

• Supports PCIe Gen5, 4th Gen Intel Xeon Scalable CPUs, and high-throughput networking via NVIDIA ConnectX-7 and BlueField DPUs.

• Designed for high-density data centers with liquid cooling and multi-node scale-out.

3. Key Software Components

• NVIDIA Triton Inference Server for scalable deployment of AI models.

• Kubernetes-based orchestration using NVIDIA AI Enterprise and NGC Helm charts.

• MIG (Multi-Instance GPU) partitioning for workload isolation and density optimization.

• Monitoring and observability tools like DCGM and Prometheus.

4. Performance Benchmarks

• Up to 9x increase in inference throughput with H100 over A100 (using GPT-J, ResNet50, etc.).

• Reduced latency and improved GPU utilization through MIG and TensorRT optimizations.

5. Use Cases

• Generative AI (e.g., LLMs, diffusion models)

• Vision-based applications (e.g., OCR, object detection)

• Speech and audio AI

• Healthcare, financial services, retail, and government workloads that demand on-prem privacy

6. Deployment Best Practices

• Use Kubernetes node pools for GPU and non-GPU workloads.

• Enable NUMA-aware scheduling and GPU pinning.

• Integrate inference with enterprise monitoring and CI/CD pipelines.

Why It Matters

This reference architecture helps enterprises build AI inference infrastructure that:

• Keeps sensitive data on-prem.

• Reduces reliance on public clouds.

• Achieves predictable latency and throughput for mission-critical AI applications.

Get the Full PDF

You can download the full official NVIDIA reference design here:

👉 Download Reference Design for Private AI Inference (PDF)

Maximize Your VMware vSphere 9.0 Performance with These Best Practices. VMware vSphere 9.0 brings numerous performance enhancements, but realizing its full potential depends on proper configuration of your hardware, software, and virtual infrastructure. The “vSphere 9.0 Performance Best Practices” guide by VMware offers detailed, field-tested advice for system administrators who want to optimize every layer of their vSphere environment.

Key Areas Covered

1. Hardware Selection and Configuration

• Validate that all hardware is compatible and meets minimum requirements.

• Use CPUs with hardware virtualization support (Intel VT-x, AMD-V), AES-NI, and NUMA-aware architecture.

• Leverage high-end NVMe storage, RDMA-enabled NICs, and optimized BIOS settings (e.g., Turbo Boost, Hyper-Threading).

2. ESX and Virtual Machine Optimization

• Avoid over-provisioning vCPUs and memory.

• Use hardware version 22 for best performance (note compatibility limits).

• Understand hyper-threading and NUMA to improve CPU scheduling.

• Tune host power policies (Balanced, High Performance, Custom) based on workload.

3. Memory Management

• Avoid memory thrashing and carefully size VMs.

• Use memory overcommit techniques like ballooning, page sharing, and compression.

• Configure 2MB large pages for better performance where possible.

4. Storage Considerations

• Follow VAAI best practices to offload tasks to the array.

• Use VMFS alignment and NVMe/NVMe-oF for high-throughput workloads.

• Understand performance trade-offs between hybrid and all-flash vSAN.

5. Networking

• Choose NICs that support offloads (TSO, LRO, RSS, VXLAN, GENEVE).

• Consider SmartNICs (DPUs) for high-performance networking.

• Tune NetIOC and traffic shaping for latency-sensitive applications.

6. Infrastructure Services

• Use vMotion and Storage vMotion with proper planning for best efficiency.

• Tune DRS, DPM, and vSphere HA/FT for your workload type.

• Manage lifecycle operations via vSphere Lifecycle Manager.

Intended Audience

This guide is aimed at system administrators who are deploying or maintaining a vSphere 9.0 environment and want to ensure optimal performance at every layer, from hardware through to guest operating systems.

Get the Full PDF

You can download the full official VMware document here:

👉 Download vSphere 9.0 Performance Best Practices (PDF)