• --gtest_list_tests - List the names of all tests instead of running them, The name of TEST(Foo, Bar) is Foo.Bar.

• --gtest_filter=POSITIVE_PATTERNS[-NEGATIVE_PATTERNS] - Run only the tests whose name matches one of the positive patterns but none of the negative patterns; ? matches any single character; * matches any substring; and : separates two patterns.

• --gtest_also_run_disabled_tests - Run all disabled tests too.

• --gtest_repeat=[COUNT] - Run the tests repeatedly; use a negative count to repeat forever.

• --gtest_shuffle - Randomize tests’ orders on every iteration.

• --gtest_random_seed=[NUMBER] - Random number seed to use for shuffling test orders (between 1 and 99999, or 0 to use a seed based on the current time).

• --gtest_color=(yes|no|auto) - Enable/disable colored output. The default is auto.

• --gtest_print_time=0 - Don’t print the elapsed time of each test.

• --gtest_output=xml[:DIRECTORY_PATH/|:FILE_PATH] - Generate an XML report in the given directory or with the given file name. FILE_PATH defaults to test_details.xml.

• --gtest_stream_result_to=HOST:PORT - Stream test results to the given server.

• --gtest_death_test_style=(fast|threadsafe) - Set the default death test style.

• --gtest_break_on_failure - Turn assertion failures into debugger break-points.

• --gtest_throw_on_failure - Turn assertion failures into C++ exceptions.

• --gtest_catch_exceptions=0 - Do not report exceptions as test failures. Instead, allow them to crash the program or throw a pop-up (on Windows).

Except for --gtest_list_tests, you can alternatively set the corresponding environment variable of a flag (all letters in upper-case). For example, to disable colored text output, you can either specify --gtest_color=no or set the GTEST_COLOR environment variable to no.

For more information, please read the Google Test documentation. If you find a bug in Google Test (not one in your own code or tests), please report it.

UnitCL also parses some extra command line options above and beyond what Google Test provides:

• --gtest_filter=-*UNSPECIFIED* - Disable all tests for behavior not mandated by the specification.

• --unitcl_platform=<vendor> - Provide an OpenCL platform vendor to use for testing.

• --unitcl_device=<device_name> - Provide an OpenCL device name to use for testing.

• --unitcl_test_include=<path> - Provide the path to the supplied test_include directory. Current setting: test/UnitCL/test_include

• --unitcl_kernel_directory=<device_name> - Provide the path to the supplied kernels directory

• --unitcl_build_options=<option_string> - Provide compilation options to pass to clBuildProgram when compiling kernels in the ‘kernels’ directory.

• --unitcl_seed=<unsigned> - Provide an unsigned integer to seed the random number generator with.

• --unitcl_math={quick, wimpy, full} - Run math builtins tests over an increasing data size, defaults to wimpy.

• --vecz-check - Mark tests as failed if the vectorizer did not vectorize them.

• --opencl_info - Print OpenCL platform and devices info

To write a Kernel Execution Test, you will need to write two things: a C++ part that will include the test in the test suite and define the execution environment of the kernel (inputs, expected outputs, work size etc.), as well as the actual OpenCL-C kernel that you want to test.

Files related to Kernel Execution Tests may show references to the name KTS. These are remnants from when it was an entirely separate test suite known as the Kernel Test Suite (KTS).

The C++ test and the OpenCL-C kernel are linked through their names only, the name of the file and of the kernel are inferred from the name of the test in the following way:

• Format for test names: TestSet_X_KernelName, where X is a two digit number and where _X_ is the last underscore two digit combination that occurs in the test name.

• Example name: Task_01_01_Copy.

• Kernel file to load: task_1.01_copy.cl.

• Kernel name: copy.

To write the C++ test and set up the execution environment of your kernel, KTS provides convenient utility functions:

• RunGeneric1D: Will launch the kernel, it takes the global size as a first parameter and the local size as an optional second parameter. The default local size is 0. KTS provide kts::N and kts::localN as defaults global and local size that can be used in tests.

• RunGenericND: Will launch the kernel, it takes the number of dimensions as a first parameter, a pointer to the global dimensions as the second and a pointer to the local dimensions as an third parameter.

• AddInputBuffer: Will add an input buffer to the kernel, its first parameter is the size of the buffer, and the second is a kts::Reference1D function that will be used to generate the content of the buffer.

• AddOutputBuffer: Will add an output buffer to the kernel, its first parameter is the size of the buffer, and the second is a kts::Reference1D function that should generate the expected output of the kernel, it will be used to check the actual output of the kernel.

• AddInOutBuffer: Will add an input and output buffer to the kernel. The first parameter is the size of the buffers. The second is a kts::Reference1D that will generate the content of the input buffer whilst the third is a kts::Reference1D that will be used to check the actual output of the kernel.

• AddPrimitive: Will add a by value parameter to the kernel, and it takes only one parameter, the actual value to pass to the kernel.

• AddMacro: Will add a macro in the kernel, the first parameter of this function is the name of the macro, and the second is an unsigned value representing the value of the macro.

To generate the input and output of a kernel, KTS requires the user to provide functions matching the type of kts::Reference1D, this is a templated type that matches a function taking an int and returning a value of the template parameter type. The int value passed to the function is the global id for which we want to generate a value.

kts::BuildVecXReference1D (with X a vector size in 2,3,4) is provided to allow extrapolation of a kts::Reference1D for a vector type from a kts::Reference1D for the matching scalar type.

A set of common kts::Reference1D functions is also provided for use with kernel execution tests, available in source/kts_reference_functions.h.

The TEST_P(Execution, TEST_NAME) macro is used to create execution tests. The Execution fixture is parameterized over the source type of the OpenCL program to test:

enum kts::ucl::SourceType

enumerator OPENCL_C

Loads an OpenCL C (.cl) source file from disk.

enumerator SPIR

Loads a SPIR (.bc32/.bc64) file from disk.

enumerator SPIRV

Loads a SPIR-V (.spv32/.spv64) file from disk.

enumerator OFFLINE

Loads a pre-compiled .cl binary (.bin) file from disk.

enumerator OFFLINESPIR

Loads a pre-compiled .bc32/.bc64 (.bin) file from disk.

enumerator OFFLINESPIRV

Loads a pre-compiled .spv32/.spv64 (.bin) file from disk.

Sometimes it may be necessary to skip one or more source types. For example, a test might trigger a known SPIR bug so the SPIR versions need to be skipped, or a test might only be valid when compiled just-in-time from OpenCL C. In these cases the kts::ucl::isSourceTypeIn() utility function should be used to determine if a test should be skipped:

TEST_P(Execution, Test) {
  if (!isSourceTypeIn({OPENCL_C, SPIRV, OFFLINE, OFFLINESPIRV}) {
    GTEST_SKIP();
  }
}

When a parameterized test is needed, the ExecutionWithParam fixture template should be used in conjunction with the UCL_EXECUTION_TEST_SUITE_P macro to instantiate the test suite.

using MyExecutionWithInt = ExecutionWithParam<int>;

// Instantiates the test suite over all source types and 3 int values.
UCL_EXECUTION_TEST_SUITE_P(MyExecutionWithInt,
                           testing::ValuesIn(kts::ucl::getSourceTypes()),
                           testing::Values(23, 42, 88));

TEST_P(MyExecutionWithInt, Test) {
  // Use getParam() instead of GetParam() to access the int parameter, this
  // is provided by the ExecutionWithParam fixture to make accessing the non
  // source type parameter easier.
  int i = getParam();
  // ...
}

As part of our offline testing, the UnitCL build target will compile OpenCL-C, SPIR, and SPIR-V kernels through clc to produce output binaries that are later loaded by UnitCL. To control how these are handled, you can add the following comments to the OpenCL-C kernel. These are parsed by CMake.

• // CLC OPTIONS: - A semicolon-separated list of values that are to be passed to clc as options. For example // CLC OPTIONS: -cl-mad-enable;-DDEF1=7;-DOPTION_2=ON

• // REQUIRES: - A semi-colon separated list of values representing the requirements of the kernel. For example // REQUIRES: double; images

Here is the full list of supported requirements for // REQUIRES::

• noclc - Skip all clc steps for this kernel. Use this when you have a kernel that is not supported by clc. OfflineExecution, OfflineSpirExecution, and OfflineSpirvExecution tests will not work for this kernel if this requirement is set.

• nospir - Skip all SPIR generation steps for this kernel. This will only have an effect if you are running the regenerate-spir target. SpirExecution and OfflineSpirExecution tests will not work for this kernel if this requirement is set.

• nospirv - Skip all SPIR-V generation steps for this kernel. This will only have an effect if you are running the regenerate-spirv target. SpirvExecution and OfflineSpirvExecution tests will not work for this kernel if this requirement is set.

• double - If your kernel requires doubles (i.e., the cl_khr_fp64 extension), then CMake will only compile the kernel for targets with the fp64 capability.

• half - If your kernel requires halfs (i.e., the cl_khr_fp16 extension), then CMake will only compile the kernel for targets with the fp16 capability.

• images - If your kernel requires image support, then CMake will only compile the kernel when host has images enabled.

• parameters - Currently tests that are parameterized using macros do not support any offline compilation (either to IR or to executable). This requirement disables all but the Execution test type.

When a requirement is not met, CMake will not run clc. Instead, it will create a stub file with the same name that clc would have created. The stub file contains the text Skipped due to <reason>. Stub files allow CMake to check for stale files; if stub files were not used, CMake would need to always re-parse skipped kernel files in case the requirements had been changed.

The install target looks for the Skipped text to determine if a kernel file is a stub.

When UnitCL is built, clc is called by the UnitCL-offline-execution-kernels target.

The SPIR and SPIR-V kernels passed to clc are generated by the regenerate-spir and regenerate-spirv targets , respectively.

When UnitCL is running against an offline OpenCL driver i.e. no compiler is available, then tests which require a compiler are skipped. There are two methods of test an offline version of CL:

• Build the oneAPI Construction Kit with the CMake option CA_RUNTIME_COMPILER_ENABLED set to OFF.

• Build the oneAPI Construction Kit with the CMake options CA_RUNTIME_COMPILER_ENABLED set to ON, and CA_COMPILER_ENABLE_DYNAMIC_LOADER set to ON, then set the environment variable CA_COMPILER_PATH to the empty string to disable loading the compiler at runtime.

The copy-kernels CMake target is built as part of UnitCL. Its purpose is to copy kernels from the source directory to the build directory so that they are available in a known location when UnitCL is run.

The following diagram shows how the copy-kernels target works:

• File paths are relative to the UnitCL/ directory. Exceptions are paths with <target>, which will be somewhere in the ComputeMux target’s directory tree, and paths with ${PROJECT_BINARY_DIR} or ${CMAKE_INSTALL_PREFIX}, which point to CMake’s build or install directories.

• There are currently no hand-written .bc32 or .bc64 default kernels in UnitCL (yellow dashed ellipse), but these are theoretically supported.

• All .cmake scripts are in UnitCL/cmake/.

• The install target is separate from the copy-kernels target, but it’s shown for completeness.

The UnitCL-offline-execution-kernels CMake target is built as part of UnitCL. It wraps clc and compiles the kernels used by OfflineExecution, OfflineSpirExecution, and OfflineSpirvExecution tests for each target device.

The following diagram shows how the UnitCL-offline-execution-kernels target works:

• See the diagram in copy-kernels Target for how the ${${MUX_TARGET}_UNITCL_KERNEL_FILES} list is populated.

• The install target is separate from the UnitCL-offline-execution-kernels target, but it’s shown for completeness.

There are two custom targets - regenerate-spir and regenerate-spirv - which can be used for generating the appropriate SPIR BC and SPIR-V ASM files from all the kernels we have in UnitCL.

To use the regenerate-spir target you will require Khronos’ modified version of Clang and opencl_spir.h.

To use the regenerate-spirv target you will require the llvm-spirv tool.

• Khronos Clang (See Compiling Khronos Clang for build instructions)

• llvm-spirv (See Compiling LLVM-SPIRV for build instructions)

• opencl_spir.h

At the time of writing (2019-11-14), the llvm-spirv tool used to generate SPIR-V from OpenCL C was built from the the llvm_release_80 branch of SPIRV-LLVM.

As part of your CMake command set the following values:

• CA_EXTERNAL_KHRONOS_CLANG the absolute path to Khronos clang.

• CA_EXTERNAL_LLVM_SPIRV the absolute path to the llvm-spirv.

• CA_EXTERNAL_OPENCL_SPIRH the absolute path to opencl_spir.h.

With these set, calling the regenerate-spir and regenerate-spirv targets will build SPIR and SPIR-V for all the kernels, respectively.

regenerate-spir and regenerate-spirv will regenerate all binaries for all kernels, even if those binaries are not required. If a binary is not tracked by git, then that is probably for a good reason. For example, tests for that binary might not yet exist. It is recommended not to add binaries to git unless that is specifically what you intend to do. Untracked binaries can be removed with

# In UnitCL
git clean -f source/cl/test/UnitCL/kernels/
# In host, if present
git clean -f modules/mux/targets/host/test/UnitCL/kernels/

The following diagram shows how the regenerate-spir and regenerate-spirv targets work:

• See the diagram in copy-kernels Target for how the ${${MUX_TARGET}_UNITCL_KERNEL_FILES} list is populated.

• All .cmake scripts are in UnitCL/cmake/.

The regenerate-spir/regenerate-spirv targets are most useful for automating generation of all the kernels. However, sometimes you might prefer to generate these yourself by hand.

clang -cc1 -emit-llvm-bc -triple spir-unknown-unknown      \
      -cl-spir-compile-options "<compile options>"         \
      -include path/to/opencl_spir.h                       \
      <compile options>                                    \
      -O0 -Werror                                          \
      -o kernel_name.bc32                                  \
         kernel_name.cl
clang -cc1 -emit-llvm-bc -triple spir64-unknown-unknown    \
      -cl-spir-compile-options "<compile options>"         \
      -include path/to/opencl_spir.h                       \
      <compile options>                                    \
      -O0 -Werror                                          \
      -o kernel_name.bc64                                  \
         kernel_name.cl

clang -c -emit-llvm -target spir-unknown-unknown -cl-std=CL1.2   \
      -Xclang -finclude-default-header                           \
      <compile options>                                          \
      -o kernel_name.bc32                                        \
         kernel_name.cl
clang -c -emit-llvm -target spir64-unknown-unknown -cl-std=CL1.2 \
      -Xclang -finclude-default-header                           \
      <compile options>                                          \
      -o kernel_name.bc64                                        \
         kernel_name.cl

llvm-spirv kernel_name.bc32  \
        -o kernel_name.spv32
llvm-spirv kernel_name.bc64  \
        -o kernel_name.spv64

spirv-dis kernel_name.spv32    \
       -o kernel_name.spvasm32
spirv-dis kernel_name.spv64    \
       -o kernel_name.spvasm64