The OpenCL Intercept Layer allows developers to do a number of things with existing OpenCL binary applications, largely focusing on working with OpenCL program objects. The primary use case in the oneAPI Construction Kit is to enable testing of OpenCL drivers which do not contain a runtime compiler by applications but assume a runtime compiler is available, such as the OpenCL Conformance Test Suite. For more information see Injection Testing below. The OpenCL Intercept Layer requires the use of the OpenCL ICD Loader.
Configuration
There are a large number of configuration controls
made available by the OpenCL Intercept Layer which can be specified either
in the configuration file or through environment variables. When working with
multiple concurrent build directories editing a centralized configuration file
can become tedious and error prone so use of environment variables, prefixed
with CLI_, is preferred. Below the configuration controls most relevant to
Injection Testing are described.
- CLI_OpenCLFileName
The
CLI_OpenCLFileNameenvironment variable specifies where to find the OpenCL ICD Loader, i.e. the absolute path to thelibOpenCL.soorOpenCL.dllwhich should be used to load a client driver.See OpenCLFileName for more information.
- CLI_DumpDir
The
CLI_DumpDirenvironment variable specifies the absolute path of the directory to dump program binaries whenCLI_DumpProgramBinariesis enabled, or to lookup program binaries for injection whenCLI_InjectProgramBinariesis enabled.See DumpDir for more information.
- CLI_DumpProgramBinaries
The
CLI_DumpProgramBinariesenvironment variable, when set to a non-zero value, enables dumping program binaries resulting from calls to clCreateProgramWithSource followed by clBuildProgram. Dumped program binary files are written to the directory specified byCLI_DumpDir. Generated file names include a hash of the build options passed to clBuildProgram.See DumpProgramBinaries for more information.
- CLI_OmitProgramNumber
The
CLI_OmitProgramNumberenvironment variable, when set to a non-zero value, disables inclusion of the unique program number in program binary filenames whenCLI_DumpProgramBinariesis enabled. This is especially useful for OpenCL applications which perform compilation in a multi-threaded context which can result in non-deterministic program numbering.See OmitProgramNumber for more information.
- CLI_InjectProgramBinaries
The
CLI_InjectProgramBinariesenvironment variable, when set to a non-zero value, enables injection of previously dumped program binaries when the application calls clCreateProgramWithSource followed by clBuildProgram. Program binaries are loaded from theInjectsubdirectory of the directory specified byCLI_DumpDir. The search pattern for program binary file names does not include the hash generated from the build options passed to clBuildProgram as this information is not available during the call to clCreateProgramWithSource where injection occurs, thus dumped program binary file names must have the hash removed before they can be injected.Since the OpenCL Intercept Layer only searches for program binaries to inject in the
Injectsubdirectory ofCLI_DumpDir, additional preparation must be done in order to use previously dumped program binaries. Theinject-prepare-bins.pyscript should be used to perform this preparation, this involves removing the build options hash from the file name and copying the file to theInjectsubdirectory.$ python scripts/testing/inject-prepare-bins.py $CLI_DumpDirSee InjectProgramBinaries for more information.
Injection Testing
Injection testing is a method of enabling testing of an OpenCL driver which does not contain a runtime compiler by applications which assume a runtime compiler is available.
This process has three main steps:
Run the OpenCL application through the OpenCL Intercept Layer with an OpenCL driver which contains a runtime compiler to dump all program object that are built from source.
Prepare the dumped program object binary files for injection.
Run the OpenCL application through the OpenCL Intercept Layer with an OpenCL driver which does not contain a runtime compiler, using the prepared program object binaries.
Caution
The OpenCL Intercept Layer has a number of limitations which should be considered if unexpected results are experienced:
The OpenCL Intercept Layer was not designed to operate in a multi-threaded context, doing so can cause unexpected behavior. This is a known upstream issue.
Program objects are created using clCreateProgramWithBinary rather than clCreateProgramWithSource. Any calls to clCompileProgram or clLinkProgram using this program object will fail returning
CL_INVALID_OPERATION, which is behavior required by the specificaion.Dumped program binaries with differing clBuildProgram/clCompileProgram options are not differentiated when used for injection due to the ordering of program object creation and compilation stages, this causes issues when a program object is compiled multiple times by the application with different options. For example, an application which compiles a program object with the same source multiple times, using compile options that can change behaviour like -D, only the first program binary will be injected when a subsequent version is expected by the application, this can cause:
Positive tests to erroneously fail.
Negative tests to erroneously succeed.
Test fixtures which share a program object to receive an incorrect program binary.
Injected program binaries pass to clGetKernelInfo to query for attributes will fail as there is not source code to query for the attribute values.
Injected program binaries result in program objects that will not trigger any build status changes which can cause unexpected queries using clGetProgramBuildInfo if the application expects compilation to occur and uses this information to trigger other work.
Prerequisites
The required preparation for each step is as follows.
Step 1. requires the oneAPI Construction Kit built with a runtime compiler, i.e. a build configured with
CA_RUNTIME_COMPILER_ENABLEDset toON.Step 2. requires the
scripts/testing/inject-prepare-bins.pyscript.Step 3. requires the oneAPI Construction Kit built without a runtime compiler, i.e. a second build configured with
CA_RUNTIME_COMPILER_ENABLEDset toOFF.
Dumping Program Binaries
The following environment variables, when used for dumping program binaries, have been found to provide the best results.
OCL_ICD_FILENAMESset to the absolute path of an OpenCL driver which has a runtime compiler available.CLI_OpenCLFileNameset to the absolute path to the OpenCL ICD Loader.CLI_DumpDirset to the absolute path of the desired program binary output directory.CLI_DumpProgramBinariesset to1to enable dumping of program binaries.CLI_OmitProgramNumberset to1to disable appending the program number to dumped program binary filenames.
Injecting Dumped Binaries
The following environment variables, when used for injecting program bianries, have been found to provide the best results.
OCL_ICD_FILENAMESset to the absolute path of an OpenCL driver which does not contain a runtime compiler.CLI_OpenCLFileNameset to the absolute path to the OpenCL ICD Loader.CLI_DumpDirset to the absolute path of the desired program binary output directory.CLI_DumpProgramBinariesset to1to enable dumping of program binaries.CLI_OmitProgramNumberset to1to disable appending the program number to dumped program binary filenames.
OpenCL CTS Examples
Note
While the examples below are using the OpenCL CTS the steps are transferable to any OpenCL application or test suite.
In the following examples, the steps to perform injection testing with the
OpenCL CTS are given. A modified list of tests is used as a number of tests
rely on calling clBuildProgram with different build options
multiple times or calling
clCompileProgram/clLinkProgram to create program
objects. Both methods of program creation are not supported for injection
testing due to limitations of the OpenCL Intercept Layer. The full list of
disabled tests can be found in the
opencl_conformance_tests_wimpy_offline.csv file which can be found in the
scripts/jenkins/cts_summary source directory or the share/OpenCL-CTS
install directory, lines which begin with # are disabled tests.
Important
During research of the injection testing technique, OpenCL image support was
disabled, thus any OpenCL CTS test failures when image support is enabled
due to the failure methods described above have not been disabled in
opencl_conformance_tests_wimpy_offline.csv.
OpenCL CTS Dual Build Example
Example on Linux using a prebuilt install from $PWD/build-online which
contains a runtime compiler enabled libCL.so and another prebuilt
install from $PWD/build-offline which contains a compiler-less
libCL.so, CMAKE_INSTALL_PREFIX is set to
$PWD/build-online/install and $PWD/build-offline/install respectively.
Testing the OpenCL CTS using City Runner being invoked from the root of the
oneAPI Construction Kit repository.
Firstly, dump the program bianries using the runing compiler in the
$build-online/install/lib/libCL.so driver.
$ python scripts/testing/run_cities.py \
-b $PWD/build-online/install/bin -L $PWD/build-online/install/lib \
-e OCL_ICD_FILENAMES=$PWD/build-online/install/lib/libCL.so \
-e CLI_OpenCLFileName=$PWD/build-online/install/lib/OpenCL/lib/libOpenCL.so \
-e CLI_DumpDir=$PWD/build-online/CTSDump \
-e CLI_DumpProgramBinaries=1 \
-e CLI_OmitProgramNumber=1 \
-s $PWD/build-online/install/share/cts_summary/opencl_conformance_tests_wimpy_offline.csv
Then, prepare the dumped program binaries for injection in the next step.
$ python scripts/testing/inject-prepare-bins.py $PWD/build-online/CTSDump
Finally, test using the prepared dumped program binaries using the
compiler-less $PWD/build-offline/install/lib/libCL.so.
$ python scripts/testing/run_cities.py \
-b $PWD/build-offline/install/bin -L $PWD/build-offline/install/lib \
-e OCL_ICD_FILENAMES=$PWD/build-offline/install/lib/libCL.so \
-e CLI_OpenCLFileName=$PWD/build-offline/install/lib/OpenCL/libOpenCL.so \
-e CLI_DumpDir=$PWD/build-online/CTSDump \
-e CLI_InjectProgramBinaries=1 \
-s $PWD/build-offline/install/share/cts_summary/opencl_conformance_tests_wimpy_offline.csv
