Eric Sloof

During the NSX upgrade, I ran into an issue where the "repo_sync" failed on the NSX Managers, resulting in an incomplete repository synchronization. After consulting VMware’s official documentation, I decided to try my own solution.

I connected to the NSX Manager via SFTP and discovered that the repository contents on one of the NSX Managers were not correctly synced. I copied the repository contents from a working NSX Manager to the failing NSX Manager.

After copying, I noticed that the file permissions were not aligned with the expected defaults. To ensure proper access, I manually adjusted the permissions of the synchronized repository files and directories to match those on the working NSX Manager.

Example commands used:

# Compare permissions on the working NSX Manager

ls -l /repository

# Adjust ownership to match the working node (typically uuc:grepodir)

chown -R uuc:grepodir /repository/<version>

# Ensure directory permissions are set correctly

chmod -R 770 /repository/<version

Finally, I executed the "resolve" option on the failed NSX Manager, which successfully fixed the issue.

On the working NSX Manager, there were two directories in the /repository folder:

• 4.2.0.2.0.24278654
• 4.2.3.0.0.24866349

There was also a file called current_version, which contained the text specifying the current version, in this case:

4.2.0.2.0.24278654

VMware Solution: Repo_Sync Status Fails on NSX Managers During Upgrade Pre-checks

Article ID: 392736
Last Updated: 06-27-2025

The NSX Manager upgrade pre-check fails with the following error on the NSX GUI:

"Failed to execute check for the version sync status of UC on all the MP nodes."

Additionally, the "Repository sync is not complete on all 3 NSX Managers" error appears on the NSX GUI.

Log snippets in var/log/nsxapi.log show errors similar to the following:

####-##-##T##:##:##.### INFO http-nio-127.0.0.1-7440-exec-10 RepositoryServiceImpl 4598 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" reqId="########-####-####-####-###########" subcomp="manager" username="uproton"] Repo-sync status for <IP of node> node is FAILED.

• NSX Version: 4.1.0.2

When upgrading NSX Managers from one version to another, the /repository folder on the Orchestrator node should contain both the current (from) version and the target (to) version directories. For example, when upgrading from 4.1.0.2 to 4.2.1.3, the /repository folder should contain both 4.1.0.2 and 4.2.1.3 directories.

VMware recommends following the KB steps to resolve the issue:

1. After replacing NSX Managers or during upgrade pre-checks, if the repo_sync fails, deploy both MUB files into the /repository folder of the Orchestrator node.

2. Once the repo_sync is successful on the Orchestrator node, click Resolve for repo_sync on the other two NSX Managers in the NSX GUI to sync the /repository folder across all nodes.

• Related KBs:

  • Repo_Sync Fails After Replacing Managers or During Upgrade Pre-checks

  • NSX Manager Upgrade Failures and Related Errors

By following the official VMware solution or the manual synchronization method I discovered, you can resolve the "repo_sync" failure issue during the NSX Manager upgrade process.

AI and machine learning are no longer experimental technologies—they have become essential in today’s enterprise IT landscape. While the training of AI models often happens in specialized environments, inference—the actual application of trained models—is increasingly being integrated into existing infrastructure. VMware Cloud Foundation (VCF) enables this through powerful distributed inference capabilities.

What is distributed inference?

Distributed inference allows AI models to be executed in parallel and at scale across multiple GPUs and clusters within VCF. This means inference tasks are not limited to a single server or GPU but can be distributed across the entire infrastructure. The result is faster processing and better utilization of available resources.

Integration in VMware Cloud Foundation

Within VCF, distributed inference is powered by the combination of:

• vSphere for compute virtualization and GPU partitioning

• NSX for secure and efficient network connectivity

• vSAN for high-performance, shared storage

• Tanzu Kubernetes Grid (TKG) for hosting and scaling AI workloads in containers

This integration enables inference workloads to be deployed, scaled, and secured with flexibility—without the overhead of managing a separate AI infrastructure.

Benefits for organizations

• Scalability: inference workloads can be distributed and scaled automatically across multiple GPUs.

• Efficient resource utilization: GPU capacity is used to its fullest, lowering costs.

• Security and isolation: with NSX micro-segmentation, AI workloads remain securely separated.

• Consistent infrastructure: AI applications run within the same trusted VCF environment as traditional workloads.

Conclusion

Distributed inference with VMware Cloud Foundation demonstrates how AI and enterprise IT are converging. By making inference workloads scalable, secure, and efficient within the existing infrastructure, organizations can accelerate the time-to-value of their AI models.

Want to learn more?

For a deep dive into all the concepts, configuration tips, and implementation best practices for distributed inference with VMware Cloud Foundation, get a copy of the full whitepaper.

Data security is at the top of the agenda for nearly every organization. Encryption plays a crucial role here: it ensures that data is only accessible to systems and users with the correct keys. VMware vSAN provides integrated encryption services that can easily be enabled at the cluster level.

Two Forms of Encryption

vSAN offers two types of encryption that can be used independently or together:

• Data-at-Rest Encryption: encrypts all data as it lands on physical storage devices.

• Data-in-Transit Encryption: secures all vSAN traffic transmitted between hosts in the cluster.

Both methods use the built-in VMkernel cryptographic module in vSphere, which has achieved FIPS 140-2 and 140-3 validation. This ensures compliance with the strictest security standards.

ESA versus OSA

The implementation of encryption differs significantly between the Original Storage Architecture (OSA) and the Express Storage Architecture (ESA).

• In the OSA, data-at-rest encryption is applied as the final step in the I/O path. This preserves the ability to leverage deduplication and compression, but it comes with higher resource overhead.

• In the ESA, encryption occurs higher in the stack, immediately after compression. This means that data only needs to be encrypted once, which reduces CPU and network usage and improves efficiency across the cluster.

Key Management: KMS or Native Key Provider

There are two main options for managing encryption keys:

1. External Key Management Server (KMS) – based on the KMIP protocol and often deployed in a highly available cluster configuration.

2. vSphere Native Key Provider (NKP) – a built-in key management solution that operates entirely within vSphere, and can leverage TPM chips on hosts for additional resilience and secure key persistence.

VMware strongly recommends using TPM chips on all vSAN hosts to ensure keys remain securely stored and available even if the key provider is temporarily unreachable.

Conclusion

vSAN Encryption Services provide organizations with powerful tools to protect sensitive data. With the ability to secure data both at rest and in transit, and flexible options for key management, vSAN ensures compliance, resilience, and peace of mind. Especially with the ESA architecture, organizations benefit from stronger security with lower performance overhead.

Want to learn more?

For a deep dive into all the available techniques, trade-offs, and configuration details, get a copy of the full whitepaper.

Storage capacity remains a critical concern in modern datacenters. Organizations want to host more and more workloads, while physical storage growth is often expensive or limited. VMware vSAN addresses this challenge with a range of space efficiency technologies that help reduce costs and improve performance.

Opportunistic vs. Deterministic Techniques

The whitepaper distinguishes between two categories of techniques:

• Opportunistic: dependent on the characteristics of the data, which means savings are not guaranteed. Examples include deduplication & compression (OSA), compression-only, thin provisioning, and TRIM/UNMAP space reclamation.

• Deterministic: delivers predictable and guaranteed space savings. This is the case with RAID-5/6 erasure coding.

Both types can be used independently or combined to maximize efficiency.

ESA versus OSA

With the introduction of the vSAN Express Storage Architecture (ESA) in vSAN 8, the options have significantly improved. While the Original Storage Architecture (OSA) had limitations around performance and deduplication domains, ESA provides much more effective data reduction without noticeable performance trade-offs.

• Compression in ESA: Data is compressed as soon as it enters the vSAN stack, reducing CPU load and network traffic.

• Global Deduplication in ESA: The deduplication domain spans the entire cluster, making data reduction far more effective than OSA’s per–disk group approach.

For existing environments still running on OSA, deduplication & compression or compression-only remain valid options, but they come with more performance considerations.

Thin Provisioning and TRIM/UNMAP

With thin provisioning, storage is only allocated when actually needed. This allows oversubscription, a common practice to make better use of capacity.

A well-known drawback is that released space inside a VMDK is not automatically reclaimed. TRIM/UNMAP solves this issue by freeing unused blocks. This not only increases usable capacity but also speeds up recovery and rebalance operations.

Erasure Coding: RAID-5 and RAID-6

Where RAID-1 mirroring always creates a full copy of the data—consuming large amounts of storage—erasure coding provides a more efficient alternative.

• RAID-5 (FTT=1) consumes 1.33x the original data in OSA, or just 1.25x in ESA.

• RAID-6 (FTT=2) consumes 1.5x the original data, compared to 3x with mirroring.

In ESA, these techniques are available without any performance trade-offs, making them the logical standard for most environments.

Conclusion

By combining opportunistic and deterministic space efficiency techniques, vSAN allows organizations to use storage more intelligently and cost-effectively. Especially with ESA, both performance and efficiency are greatly enhanced. For environments still running OSA, migrating to ESA is a worthwhile step to optimize both capacity and performance.

Want to learn more?

For a deep dive into all the available techniques, trade-offs, and configuration details, get a copy of the full whitepaper.