NVIDIA Certified Professional - AI Operations practice questions

Question 1

An architect is designing the storage layout for a DGX SuperPOD high-availability configuration and must place user home directories and shared data so the head nodes can fail over cleanly. What does NVIDIA require for these directories?

Accepted Answer

They must sit on a shared NFS filesystem made available to all of the head nodes for HA.

Answer

They must live on local NVMe inside each head node and sync over the RDMA failover link.

Answer

They must be stored in GPU memory and mirrored by GPUDirect Storage between head nodes.

Answer

They must be packed into the DGX OS image and re-flashed onto the secondary head node.

Question 2

A production team is splitting workloads across MIG instances and needs to confirm what isolation guarantees they get. Which property does the documentation attribute to each MIG instance?

Accepted Answer

Each MIG instance provides hardware-level isolation with dedicated memory bandwidth and SM engines on the partitioned GPU

Answer

Each MIG instance provides software-level isolation only, but shares L2 cache and HBM with other instances on the same physical GPU

Answer

MIG isolation is enforced by an admission webhook rather than by GPU hardware, so workloads share L2 and HBM until the webhook migrates them

Answer

MIG instances share SM engines and rely on the driver to time-slice access between them, so two MIG instances behave like oversubscribed replicas

Question 3

An engineer is being deployed onto a debug rotation that filters NCCL logs by subsystem so they can isolate transport vs initialization issues. Which NCCL environment variable is documented for that filtering?

Accepted Answer

NCCL_DEBUG_SUBSYS filters output by subsystem (INIT, NET, P2P, GRAPH) per the NCCL environment-variable documentation, so engineers can scope output to one subsystem at a time

Answer

NCCL_DEBUG=NET, which acts as both the verbosity knob and the per-subsystem filter at the same time, since NCCL collapses the verbosity-level and subsystem-filter settings into a single environment variable at startup time

Answer

NCCL_TRACE_SUBSYS, which is the only subsystem filter that ships with NCCL on the cluster's worker nodes and accepts the same subsystem keywords as the documented NCCL_DEBUG_SUBSYS variable in the NCCL user guide

Answer

There is no subsystem-level filter on the NCCL command line; NCCL logs are emitted as a single combined stream that must be parsed externally with a separate log pipeline tool to isolate transport from initialization events

Question 4

A developer needs to launch a job directly into a container image using srun on the DGX SuperPOD instead of running on the bare host. Which tool extends Slurm so that this container launch is possible?

Accepted Answer

Pyxis, the SPANK plugin that lets srun launch jobs into containers

Answer

Enroot, the unprivileged runtime that builds the container images

Answer

Pyxis, the block storage layer that mounts the container image data

Answer

Enroot, the scheduler that queues the container jobs ahead of srun

Question 5

An architect is choosing a tool that lets Kubernetes administrators deploy NVIDIA NIM and NeMo inference microservices on a cluster. Which product is purpose-built for that task?

Accepted Answer

The NVIDIA NIM Operator, which deploys NIM and NeMo microservices on Kubernetes clusters.

Answer

The NVIDIA GPU Operator, which provisions GPU drivers and the device plugin on each node.

Answer

The NVIDIA Network Operator, which configures RDMA and GPUDirect networking for the cluster.

Answer

NVIDIA Base Command Manager, which provisions and manages bare-metal Slurm and HPC clusters.

Question 6

An MLOps team is reviewing how NCCL builds its communicator objects from a fixed rank-to-device mapping at job startup. According to the documentation, which functions create the communicator objects?

Accepted Answer

ncclCommInitRank(), ncclCommInitRankConfig(), and ncclCommInitAll() create the communicator objects.

Answer

ncclGroupStart(), ncclGroupEnd(), and ncclCommSplit() build the base communicator objects per rank.

Answer

ncclCommDestroy(), ncclCommAbort(), and ncclCommFinalize() create the communicator objects per device.

Answer

ncclCommRegister(), ncclCommUserRank(), and ncclCommCount() create the communicator objects at startup.

Question 7

A developer is profiling a single CUDA kernel in Nsight Compute and is confused that the kernel appears to execute several times for one launch. According to the Profiling Guide, why does Kernel Replay run the kernel in multiple passes and how is correctness preserved across them?

Accepted Answer

Requested metrics are grouped into one or more passes, and memory the kernel writes is saved after the first pass and restored before each later pass.

Answer

The kernel is recompiled at a different optimization level on each pass, and Nsight Compute averages the metrics to cancel out compiler-induced variance.

Answer

Each pass runs the kernel on a different SM partition, and the tool stitches the per-partition counters together into full-GPU utilization metrics.

Answer

The first pass runs at full clocks while later passes run at locked base clocks, letting the tool subtract boost effects from the timing metrics.

Question 8

A team has decided to use NGC as the launch pad for managed services such as DGX Cloud and Riva Studio. Which characterization of NGC matches the documentation?

Accepted Answer

NVIDIA NGC is a cloud platform providing fully managed services such as NVIDIA AI Enterprise, DGX Cloud, and Riva Studio for NLU and speech AI

Answer

NGC is a local install-only product without managed services; teams that want DGX Cloud or Riva Studio must integrate them through a separate portal

Answer

NGC is an open-source project owned by the CNCF rather than NVIDIA; the source code and catalog are maintained under CNCF governance

Answer

NGC is an alternative name for the NVIDIA GPU Operator and shares the same codebase under a different naming scheme used in older marketing materials

Question 9

A team is rolling out DCGM-based monitoring on its newly-installed GPU Operator cluster and wants to confirm where utilization metrics will land for the existing observability stack. According to the GPU Operator overview, how is GPU telemetry exposed?

Accepted Answer

DCGM-based monitoring with Prometheus metric export is part of the components managed by the GPU Operator

Answer

Telemetry is sampled once at install time and never re-emitted unless a driver upgrade runs on the node

Answer

Metrics are pushed only to a proprietary NVIDIA SaaS endpoint; on-prem Prometheus scraping of DCGM Exporter is not supported

Answer

Metrics are written to the kubelet's /metrics endpoint only and are not available outside the node for cluster-wide scraping

Question 10

A sysadmin is scripting routine driver maintenance and adds an unload and reload of the NVIDIA kernel modules on a MIG-enabled A100. Based on the MIG User Guide, what will this operation do to MIG mode?

Accepted Answer

It disables MIG mode, because the mode persists only while the driver stays resident in memory.

Answer

It preserves MIG mode because the enabled state is cached inside the kernel modules across reloads.

Answer

It preserves MIG mode but resets every existing GPU instance back to its default size profile.

Answer

It preserves MIG mode and only clears the compute instances created inside each GPU instance.

Question 11

An administrator is being deployed onto a DGX platform and runs NVSM checks after a software update; one check verifies that installed software packages are at mutually compatible versions. Which NVSM check category is that?

Accepted Answer

The Version Compatibility category, which verifies version compatibility between the different software packages.

Answer

The Service check category, which verifies the running state and status of the essential software services.

Answer

The Packages check category, which verifies the deployment state of the expected platform software packages.

Answer

The Firmware check category, which verifies that NVSwitch and GPU firmware images match the shipped DGX baseboard.

Question 12

A sysadmin is reviewing squeue output on a DGX SuperPOD and sees a queued job in PD state whose reason column reads Priority. According to the DGX SuperPOD Slurm documentation, what does this tell the sysadmin about why the job is not running?

Accepted Answer

It waits behind another job that holds higher scheduling priority

Answer

It is running but was deprioritized partway through by the scheduler

Answer

It failed to start because no nodes match its resource constraints

Answer

It is paused because its assigned node was placed into the drain state

Question 13

A platform team is designing how operators will interact with the GPU Operator on an existing cluster. Which configuration model does the GPU Operator follow to enforce desired state?

Accepted Answer

The declarative configuration model means administrators specify desired state through CustomResources, and the operator continuously reconciles actual state to match

Answer

Administrators run an imperative CLI on each node directly and the GPU Operator records the result back into etcd for the cluster-wide audit log after each command completes

Answer

Administrators submit Slurm jobs that scrub each node and re-apply driver state on a fixed cadence so drift is corrected only at the next scheduled job's submission window

Answer

Each pod includes an initContainer that overrides the GPU Operator's desired-state settings at runtime, bypassing the operator's reconciliation guarantees in production

Question 14

A platform team is troubleshooting why MIG mode is gone after every scheduled maintenance reboot of its A100 nodes. According to the MIG User Guide, what is the documented behavior of MIG mode across system reboots?

Accepted Answer

MIG mode is not persistent across reboots because no status bit is stored in the GPU InfoROM.

Answer

MIG mode persists across reboots because the enabled state is written to the GPU InfoROM at boot.

Answer

MIG mode persists across reboots only when persistence mode is also enabled through nvidia-smi.

Answer

MIG mode persists across reboots whenever the GPU Operator MIG Manager reconciles the node label.

Question 15

A production team is hitting a wall bringing up a multi-node NVLink (MNNVL) cluster: the nvidia-imex service on most compute trays never reaches the READY state, and /var/log/nvidia-imex.log shows State = TRANSIENT_FAILURE. What does NVIDIA document as the corrective action?

Accepted Answer

Restart the nvidia-imex service on all compute nodes to rebuild the cluster, since one missing tray blocks it.

Answer

Disable IMEX on the failing trays so the remaining trays can form a smaller IMEX cluster and reach READY.

Answer

Reflash the NVSwitch firmware on the affected trays, since TRANSIENT_FAILURE indicates a fabric hardware fault.

Answer

Re-run Fabric Manager in shared-NVSwitch mode, which bypasses IMEX and brings the trays to READY directly.

NVIDIA Certified Professional - AI Operations

Sample questions

Sources

Similar exams