Eric Sloof

After delivering this session at both VMUG Connect Amsterdam and VMUG Connect Minneapolis, I can honestly say the response exceeded my expectations. The rooms were packed, the questions were sharp, and the feedback was overwhelmingly positive. It's clear that VCF troubleshooting is a topic that resonates deeply with the community right now.

What the session was all about

The goal was simple: move beyond theory and give attendees practical, real-world troubleshooting techniques they could take back to their environments the very next day. The session was built around four core areas.

Preventive Health Checks

Before you can troubleshoot, you need visibility. I walked through three essential tools for keeping your VCF environment healthy. The VCF Diagnostic Tool, a Python-based script that runs locally on your vCenter or SDDC Manager appliance and generates reports in TXT, LOG and JSON format. The VMware Health and Security Toolkit (HST), which gives you a comprehensive security assessment across your entire vSphere environment. And the VCF Build and Integrate Lifecycle Toolkit (BILT), which handles both greenfield validation and brownfield readiness checks before expansions or upgrades.

For post-deployment health monitoring, the SoS Utility remains one of the most powerful tools in the VCF admin's arsenal. Running individual checks like certificate health, connectivity health, NTP synchronization and services health gives you a clear picture of your environment's status at any point in time.

A Real-World Troubleshooting Example

This was the part of the session that generated the most discussion. I walked through an actual customer case — a shoutout to Alexander Bituev from ING Frankfurt for this one — where a workload domain creation failed with a frustratingly vague error in the Fleet Management UI.

The key lesson here was what I call the Golden Rule of VCF troubleshooting: the first error in the log is the root cause. Everything that follows is just noise. Using a simple grep command against the domainmanager.log file, or alternatively drilling into the SDDC Manager API via the built-in Developer Center Swagger UI, reveals the full error chain. In this case the nested error told the complete story: the ESXi host already had vSAN enabled, which blocked the workload domain deployment. The referenceToken shown in the Fleet Management UI is your bridge between the UI and the API — always use it.

New Troubleshooting Features in VCF Operations

VCF Operations has matured significantly and now offers a genuinely powerful set of troubleshooting capabilities. The Diagnostic Findings feature collects data from all VCF platform components every four hours and surfaces both active and historical findings. The Findings Catalog currently contains over 637 checks covering components like vCenter, ESXi, vSAN, NSX and VCF Automation.

The Troubleshooting Workbench is a personal favorite. It correlates events, property changes and anomalous metrics across your entire VCF stack in a single view, making it much faster to identify what changed and when. Combined with the Log Compare feature in VCF Operations for Logs 9.0, which lets you run multiple log queries side by side across different time periods, you have an incredibly powerful toolkit for root cause analysis.

The Storage Operations dashboard provides vSAN cluster health scores from 0 to 100, performance KPIs including IOPS, throughput and latency, and links directly to Broadcom KB articles for remediation. The Network Operations view gives you a holistic topology of your entire NSX environment including the new NSX VPC Dashboard with dropped packet monitoring.

Training and Certification

I also covered the VCF troubleshooting training landscape, which has been significantly expanded. The path runs from the foundational VCF Fundamentals for Technical Support course through the professional-level VCF Troubleshooting course with live lab options, all the way up to the advanced domain-specific playlists that lead to VCAP certifications in Storage, Automation, Operations, VKS and Networking. For those with their sights set even higher, the VCDX Distinguished Expert certification remains the ultimate destination.

The Community Response

Both in Amsterdam and Minneapolis the session sparked great conversations. Whether it was the Golden Rule for log analysis, the referenceToken trick for bridging the Fleet Management UI with the Developer Center API, or the new capabilities in VCF Operations, attendees left with concrete techniques they could apply immediately. That's exactly what VMUG sessions should deliver.

If you missed the session, the slides are available for download, and on SlideShare. 

What an incredible week it has been! This past Thursday, I had the pleasure of presenting at VMUG Connect Amsterdam, held at the iconic RAI Amsterdam. The energy in the room was fantastic, and it was truly invigorating to connect with so many engaged professionals from the VMware community.

My session — "VMware Cloud Foundation Troubleshooting: Real-World Scenarios and Solutions" — focused on practical approaches to maintaining and fixing VCF environments. We explored essential tools for performing preventive health checks on VCF 9, walked step-by-step through real-world troubleshooting examples, and discovered some of the latest troubleshooting features within VCF Operations.

The room was nearly full with around 60 attendees, and the feedback I received afterward was overwhelmingly positive. It is always rewarding to share lessons learned from the field — especially when the audience is so interactive and eager to discuss real-world applications. A big thank you to everyone who attended, asked questions, and shared their own experiences!

It was also wonderful to catch up with so many familiar faces, both from the Netherlands and abroad — including a good number of former students. To top off an already memorable week, I also passed my VCAP-Automation exam 🎉 And a comfortable stay at the Van der Valk Hotel at the Zuidas certainly didn't hurt either.

Next Stop: VMUG Connect Minneapolis 🇺🇸

If you missed the Amsterdam session, don't worry! In just three weeks, I will be delivering this presentation again at VMUG Connect Minneapolis. I am looking forward to bringing these real-world VCF troubleshooting scenarios and practical strategies to the US audience.

After the Minneapolis session, I will make the full presentation slide deck available for download right here on NTPRO.NL — so stay tuned if you want access to all the materials, tools, and references we covered.

Thank you again to the VMUG community for the continued support and engagement. See you in Minneapolis! 👊

VMware vSAN stands as a cornerstone of the modern Software-Defined Data Center (SDDC), offering robust, high-performance, and scalable storage solutions integrated directly into the hypervisor. As the technology evolves, keeping up with the latest advancements is crucial for architects, administrators, and IT professionals. This article distills the essential takeaways from the comprehensive VMware vSAN FAQ document, providing a clear overview of its core concepts, latest features, and deployment architectures. For a more interactive deep dive, be sure to check out my upcoming video walkthrough on this topic, created with NotebookLM.

Core Architectural Pillars: OSA vs. ESA

vSAN offers two distinct architectures: the Original Storage Architecture (OSA) and the newer, high-performance Express Storage Architecture (ESA). Understanding their differences is key to designing and deploying a modern vSAN cluster.

While the OSA remains a viable option for existing hardware, all new deployments should be designed for the ESA to leverage its superior performance, efficiency, and lower Total Cost of Ownership (TCO) .

Flexible Deployment Models

vSAN is not a one-size-fits-all solution. It provides several deployment models to cater to different infrastructure needs, from the data center core to the edge.

•vSAN HCI (Hyperconverged Infrastructure): This is the traditional, aggregated model where compute and storage resources reside within the same cluster. It simplifies management and is ideal for a wide range of workloads.

•vSAN Storage Clusters: Previously known as vSAN Max, this disaggregated model separates compute and storage into independent clusters. It allows you to scale storage and compute resources independently, providing a centralized, highly scalable storage platform for multiple vSphere clusters. This is particularly beneficial for optimizing licensing costs and managing large-scale environments .

•Stretched and 2-Node Clusters: These topologies provide high availability and site-level resilience. Stretched clusters span two geographic locations, while 2-Node clusters are designed for Remote Office/Branch Office (ROBO) and edge environments, using a third-site witness to maintain data quorum.

Key vSAN Capabilities

Beyond its architecture, vSAN is packed with features that ensure data availability, security, and operational simplicity.

Availability: vSAN protects against failures at multiple levels, including individual disks, entire hosts, and even network partitions. By setting Storage Policy-Based Management (SPBM) rules like "Failures to Tolerate" (FTT), administrators can define the desired level of data redundancy for each virtual machine.

Security: vSAN provides both Data-at-Rest and Data-in-Transit encryption using the AES-256 cipher. This can be managed through external Key Management Service (KMS) solutions or the vSphere Native Key Provider (NKP), without requiring specialized self-encrypting drives (SEDs) .

Performance and Operations: Tools like HCIBench allow for standardized performance testing, while features like adaptive resynchronization minimize the performance impact of maintenance operations. The Skyline Health service in vCenter provides comprehensive monitoring and proactive diagnostics to ensure your cluster runs optimally.

Explore Further

The world of vSAN is deep and constantly evolving. This article provides a high-level overview of the key concepts you need to know. To get the full details and answers to more specific questions, I highly recommend exploring the official documentation.

As you continue your learning journey, be sure to watch my upcoming video where I walk through these concepts in more detail using NotebookLM.

References

[1] VMware. (2025, December 12). vSAN Frequently Asked Questions.

#vExpert Eric Sloof takes us through the a new era for elite private cloud professionals with the VMware Certified Distinguished Expert (VCDX) offerings: the very best certification that marks you as a "Distinguished Private Cloud" Expert.

With the release of VMware Cloud Foundation 9.0, Broadcom has taken a significant step forward in simplifying log management and troubleshooting for private cloud environments. VCF Operations for Logs 9.0 introduces a deeply integrated logging solution that brings log analytics directly into the VCF Operations interface — making life easier for NOC teams, SREs, IT administrators, and application teams alike.

In this blog post, I'll walk you through the key new features, architecture improvements, and what this means for your day-to-day operations.

What's New in VCF Operations for Logs 9.0?

Integrated Log Analysis in VCF Operations

The biggest change in version 9.0 is the introduction of Operations-Logs, a new logging solution built on VMware Aria Operations for Logs. This means you no longer need to switch between separate interfaces to analyze your logs. Everything is now available directly within the VCF Operations UI.

With this integration, you can:

• Create log-based alerts without leaving VCF Operations
• Design custom dashboards based on log data
• Save and reuse log queries across your team
• Package alerts, dashboards, and queries into management packs

Important note: VMware Aria Operations for Logs content packs are still supported in 9.0, but Broadcom recommends starting the migration to log-based tools in management packs, as content packs will be phased out in future updates.

Deployment Options

How you deploy VCF Operations for Logs depends on your license type:

• VMware Cloud Foundation license: Activate Operations-Logs via VCF Management in VCF Operations Fleet Management — no separate appliance deployment required.
• VMware vSphere Foundation (VVF) license: Deploy the VCF Operations for Logs virtual appliance using vSphere.

Key Features

High-Performance Log Ingestion

VCF Operations for Logs is built to handle large volumes of log data at high throughput with low latency. It accepts data through two main channels:

• Syslog: ports 514/UDP, 514/TCP, and 1514/TCP (SSL)
• Ingestion API: ports 9000/TCP and 9543/TCP (SSL)

Any environment component — operating systems, applications, VMs, hosts, vCenter, firewalls, switches, and storage — can push syslog feeds to VCF Operations for Logs.

Scalable Architecture

VCF Operations for Logs supports both single-node and multi-node cluster deployments:

• Single node: Good for development and lab environments. Use the Integrated Load Balancer (ILB) even for single nodes to simplify future expansion.
• Cluster: Required for production environments. Clusters provide primary and worker nodes, enabling linear scaling of ingestion throughput and high availability.
• Cluster with Forwarders: Extend your deployment with forwarder clusters at remote sites, forwarding all logs to the main cluster. Ideal for multi-datacenter environments.
• Cross-Forwarding for Redundancy: Mirror two main clusters across datacenters, each front-ended with dedicated forwarder clusters for full redundancy.

Near Real-Time Search

Log data ingested by VCF Operations for Logs is available for search within seconds. Historical data can be queried from the same interface with equally low latency — no need to wait or switch tools.

Runtime Field Extraction

Raw log data is often difficult to parse visually. VCF Operations for Logs provides runtime field extraction, allowing you to dynamically extract any field from log data using regular expressions. These extracted fields can then be used for:

• Searching and filtering log events
• Aggregating events in the Explore Logs chart
• Building dashboard widgets

A handy one-click extract feature makes this even easier — no need to manually type complex regex patterns.

Explore Logs

The Explore Logs page is your primary workspace for log analysis. From here you can:

• Search and filter log events by timestamp, text, source, or field values
• Create and save custom queries
• Visualize query results as charts
• Pin charts to custom dashboards

Dashboards

Dashboards give you a real-time view of the metrics that matter most. You can:

• Create custom dashboards with widgets based on your own queries
• Use content pack dashboards for out-of-the-box visibility into VMware components
• Clone content pack dashboards and customize them for your needs
• Share dashboards with your team via shared dashboard URLs

Log Management

The Log Management section provides full control over how logs are handled:

• Log filtering: Reduce noise by filtering out irrelevant log data
• Log masking: Mask sensitive data before indexing
• Log forwarding: Forward logs to external destinations or other VCF Operations for Logs instances
• Index partitions: Control log retention and archiving policies

Integrations

VCF Operations for Logs integrates natively with key VMware and third-party products, including:

• vSphere (vCenter log collection via centralized configuration)
• NSX
• Identity Firewall
• Third-party syslog sources (rsyslog, syslog-ng, log4j)

The Life Cycle of a Log Event

Understanding how VCF Operations for Logs processes events helps you use the tool more effectively. Here's what happens from the moment a log event is generated:

1. Generated on a device outside VCF Operations for Logs
2. Collected via a VCF Operations for Logs agent, third-party agent, or direct API/syslog write
3. Received by VCF Operations for Logs — directed to the appropriate node via the ILB
4. Processed through the ingestion pipeline:
   • Keyword index created and stored on local disk
   • Machine learning applied to cluster events
   • Event stored in compressed format
5. Available for search within seconds
6. Archived or deleted based on retention policies (FIFO deletion when storage reaches 97% capacity)

What This Means for Your Teams

Getting Started

To start using VCF Operations for Logs 9.0:

1. If you have a VCF license, activate Operations-Logs via Fleet Management → VCF Management
2. Configure your environment components to forward syslog to VCF Operations for Logs
3. Explore the Explore Logs page and start building your first queries
4. Create dashboards to monitor your most important metrics
5. Set up log-based alerts to get notified proactively

Conclusion

VCF Operations for Logs 9.0 represents a major step forward in how VMware Cloud Foundation environments handle log management and troubleshooting. By integrating log analytics directly into VCF Operations, Broadcom has made it significantly easier for teams of all types to gain visibility into their infrastructure and applications — without additional tools or context switching.

Whether you're troubleshooting a failed deployment, investigating a performance issue, or simply keeping an eye on your environment's health, VCF Operations for Logs 9.0 has the tools you need.