This guide contains information on using DPC++ to run SYCL™ applications on NVIDIA® GPUs via the DPC++ CUDA® plugin.
For general information about DPC++, refer to the DPC++ Resources section.
Supported Platforms
This release has been tested on the following platforms:
GPU Hardware |
Architecture |
Operating System |
CUDA |
GPU Driver |
---|---|---|---|---|
NVIDIA A100-PCIE-40GB |
Ampere - sm_80 |
Ubuntu 22.04.2 LTS |
12.0 |
525.85.12 |
This release should work across a wide array of NVIDIA GPUs and CUDA versions, but Codeplay cannot guarantee correct operation on untested platforms.
The package has been tested on Ubuntu 22.04 only, but can be installed on any Linux systems.
This release of oneAPI for NVIDIA GPUs is not available for Windows, but a Windows package will be available in a future release.
The plugin relies on CUDA being installed on your system. As CUDA no longer supports macOS®, a oneAPI for NVIDIA GPUs package is not available for macOS.
Prerequisites
Install C++ development tools.
You will need the following C++ development tools installed in order to build and run oneAPI applications:
cmake
,gcc
,g++
,make
andpkg-config
.The following console commands will install the above tools on the most popular Linux distributions:
Ubuntu
sudo apt update sudo apt -y install cmake pkg-config build-essential
Red Hat and Fedora
sudo yum update sudo yum -y install cmake pkgconfig sudo yum groupinstall "Development Tools"
SUSE
sudo zypper update sudo zypper --non-interactive install cmake pkg-config sudo zypper --non-interactive install pattern devel_C_C++
Verify that the tools are installed by running:
which cmake pkg-config make gcc g++
You should see output similar to:
/usr/bin/cmake /usr/bin/pkg-config /usr/bin/make /usr/bin/gcc /usr/bin/g++
Install an Intel® oneAPI Toolkit version 2023.2.1 that contains the DPC++/C++ Compiler.
For example, the “Intel oneAPI Base Toolkit” should suit most use cases.
The Toolkit must be version 2023.2.1 - otherwise oneAPI for NVIDIA GPUs cannot be installed.
Install the GPU driver and CUDA software stack for the NVIDIA GPU by following the steps described in the NVIDIA CUDA Installation Guide for Linux.
Installation
Download the latest oneAPI for NVIDIA GPUs (beta) installer:
Using Download API with cURL or WGET (requires an account)
Run the downloaded self-extracting installer:
sh oneapi-for-nvidia-gpus-2023.2.1-cuda-12.0-linux.sh
The installer will search for an existing Intel oneAPI Toolkit version 2023.2.1 installation in common locations. If you have installed an Intel oneAPI Toolkit in a custom location, use
--install-dir /path/to/intel/oneapi
.If your Intel oneAPI Toolkit installation is outside your home directory, you may be required to run this command with elevated privileges, e.g.
sudo
.
Set Up Your Environment
To set up your oneAPI environment in your current session, source the Intel-provided
setvars.sh
script.For system-wide installations:
. /opt/intel/oneapi/setvars.sh --include-intel-llvm
For private installations (in the default location):
. ~/intel/oneapi/setvars.sh --include-intel-llvm
The
--include-intel-llvm
option is required in order to add LLVM tools such asclang++
to the path.Note that you will have to run this script in every new terminal session. For options to handle the setup automatically each session, see the relevant Intel oneAPI Toolkit documentation, such as Set Environment Variables for CLI Development
Ensure that the CUDA libraries and tools can be found in your environment.
Run
nvidia-smi
- if it runs without any obvious errors in the output then your environment should be set up correctly.Otherwise, set your environment variables manually:
export PATH=/PATH_TO_CUDA_ROOT/bin:$PATH export LD_LIBRARY_PATH=/PATH_TO_CUDA_ROOT/lib:$LD_LIBRARY_PATH
Verify Your installation
To verify the DPC++ CUDA plugin installation, the DPC++ sycl-ls
tool can be
used to make sure that SYCL now exposes the available NVIDIA GPUs. You should
see something similar to the following in the sycl-ls
output if NVIDIA GPUs
are found:
[ext_oneapi_cuda:gpu:0] NVIDIA CUDA BACKEND, TITAN RTX 0.0 [CUDA 11.0]
If the available NVIDIA GPUs are correctly listed, then the DPC++ CUDA plugin was correctly installed and set up.
Otherwise, see the “Missing devices in sycl-ls output” section of the Troubleshooting documentation.
Note that this command may also list other devices such as OpenCL™ devices, Intel GPUs, or AMD GPUs, based on the available hardware and DPC++ plugins installed.
Run a Sample Application
Create a file
simple-sycl-app.cpp
with the following C++/SYCL code:#include <sycl/sycl.hpp> int main() { // Creating buffer of 4 ints to be used inside the kernel code sycl::buffer<sycl::cl_int, 1> Buffer(4); // Creating SYCL queue sycl::queue Queue; // Size of index space for kernel sycl::range<1> NumOfWorkItems{Buffer.size()}; // Submitting command group(work) to queue Queue.submit([&](sycl::handler &cgh) { // Getting write only access to the buffer on a device auto Accessor = Buffer.get_access<sycl::access::mode::write>(cgh); // Executing kernel cgh.parallel_for<class FillBuffer>( NumOfWorkItems, [=](sycl::id<1> WIid) { // Fill buffer with indexes Accessor[WIid] = (sycl::cl_int)WIid.get(0); }); }); // Getting read only access to the buffer on the host. // Implicit barrier waiting for queue to complete the work. const auto HostAccessor = Buffer.get_access<sycl::access::mode::read>(); // Check the results bool MismatchFound = false; for (size_t I = 0; I < Buffer.size(); ++I) { if (HostAccessor[I] != I) { std::cout << "The result is incorrect for element: " << I << " , expected: " << I << " , got: " << HostAccessor[I] << std::endl; MismatchFound = true; } } if (!MismatchFound) { std::cout << "The results are correct!" << std::endl; } return MismatchFound; }
Compile the application with:
icpx -fsycl -fsycl-targets=nvptx64-nvidia-cuda simple-sycl-app.cpp -o simple-sycl-app
Depending on your CUDA version, you may see this warning, which can be safely ignored:
icpx: warning: CUDA version is newer than the latest supported version 12.1 [-Wunknown-cuda-version]
Run the application with:
ONEAPI_DEVICE_SELECTOR="cuda:*" SYCL_PI_TRACE=1 ./simple-sycl-app
You should see output like:
Warning: ONEAPI_DEVICE_SELECTOR environment variable is set to opencl:*. To see the correct device id, please unset ONEAPI_DEVICE_SELECTOR. SYCL_PI_TRACE[basic]: Plugin found and successfully loaded: libpi_cuda.so [ PluginVersion: 13.32.1 ] SYCL_PI_TRACE[all]: Selected device: -> final score = 1500 SYCL_PI_TRACE[all]: platform: NVIDIA CUDA BACKEND SYCL_PI_TRACE[all]: device: NVIDIA GeForce RTX 2060 The results are correct!
If so, you have successfully set up and verified your oneAPI for NVIDIA GPUs development environment, and you can begin developing oneAPI applications.
The rest of this document provides general information on compiling and running oneAPI applications on NVIDIA GPUs.
Use DPC++ to Target NVIDIA GPUs
Compile for NVIDIA GPUs
To compile a SYCL application for NVIDIA GPUs, use the icpx
compiler
provided with DPC++. For example:
icpx -fsycl -fsycl-targets=nvptx64-nvidia-cuda sycl-app.cpp -o sycl-app
The following flags are required:
-fsycl
: Instructs the compiler to build the C++ source file in SYCL mode. This flag will also implicitly enable C++17 and automatically link against the SYCL runtime library.-fsycl-targets=nvptx64-nvidia-cuda
: Instructs the compiler to build SYCL kernels for the NVIDIA GPU target.
It is also possible to build the SYCL kernels for a specific NVIDIA architecture using the following flags:
-Xsycl-target-backend=nvptx64-nvidia-cuda --cuda-gpu-arch=sm_80
Note that kernels are built for sm_50
by default, allowing them to work on a
wider range of architectures, but limiting the usage of more recent CUDA
features.
For more information on available SYCL compilation flags, see the DPC++ Compiler User’s Manual or for information on all DPC++ compiler options see the Compiler Options section of the Intel oneAPI DPC++/C++ Compiler Developer Guide and Reference.
Using the icpx
compiler
The icpx
compiler is by default a lot more aggressive with optimizations than
the regular clang++
driver, as it uses both -O2
and -ffast-math
. In many
cases this can lead to better performance but it can also lead to some issues
for certain applications. In such cases it is possible to disable -ffast-math
by using -fno-fast-math
and to change the optimization level by passing a
different -O
flag. It is also possible to directly use the clang++
driver
which can be found in $releasedir/compiler/latest/linux/bin-llvm/clang++
, to
get regular clang++
behavior.
There are currently a few known issues with the icpx
compiler which are
listed below. See the Troubleshooting Page for more
details:
Hangs with SYCL Group algorithms when using double precision floating points:
broadcast
,joint_exclusive_scan
,joint_inclusive_scan
,exclusive_scan_over_group
,inclusive_scan_over_group
Compile for Multiple Targets
In addition to targeting NVIDIA GPUs, you can build SYCL applications that can be compiled once and then run on a range of hardware. The following example shows how to output a single binary including device code that can run on NVIDIA GPUs, AMD GPUs, or any device that supports SPIR e.g. Intel GPUs.
icpx -fsycl -fsycl-targets=amdgcn-amd-amdhsa,nvptx64-nvidia-cuda,spir64 \
-Xsycl-target-backend=amdgcn-amd-amdhsa --offload-arch=gfx1030 \
-Xsycl-target-backend=nvptx64-nvidia-cuda --offload-arch=sm_80 \
-o sycl-app sycl-app.cpp
Run SYCL Applications on NVIDIA GPUs
After compiling your SYCL application for an NVIDIA target, you should also ensure that the correct SYCL device representing the NVIDIA GPU is selected at runtime.
In general, simply using the default device selector should select one of the available NVIDIA GPUs. However in some scenarios, users may want to change their SYCL application to use a more precise SYCL device selector, such as the GPU selector, or even a custom selector.
The environment variable ONEAPI_DEVICE_SELECTOR
may be used to help the SYCL
device selectors by restricting the set of devices that can be used. For
example, to only allow devices exposed by the DPC++ CUDA plugin:
export ONEAPI_DEVICE_SELECTOR="cuda:*"
For more details on this environment variable, see the Environment Variables section of the oneAPI DPC++ Compiler documentation. Note: this environment variable will be deprecated in a subsequent release.