This is a library that can be used by applications to read and write metadata to and from a well-specified binary format. This is mainly used in the oneAPI Construction Kit to store kernel and runtime metadata for offline compiled kernels.
Design
Metadata is added by pushing values to a stack. Each stack must be registered to the metadata context with a unique name. When the context is finalized, all stack will be serialized into the binary format. A stack can accept a value of any supported type, and supports arbitrary nesting of structures of values in arrays or hash-tables.
Stack data is serialized using one of two formats: MD_RAW_BYTES & MD_MSG_PACK.
MD_RAW_BYTES will recursively serialize each value on the stack into an endian-appropriate byte encoding and
write it to the output stream. It will therefore not preserve any type or length knowledge of the stack
and will simply be interpreted as a single byte-array when loaded. It is then the callers responsibility to reinterpret
the raw bytes into the data they expect. For simple metadata this may be sufficient and will reduce the overall
size of the binary, but for more complex metadata structures, especially those that use variable-length arrays, we
recommend using MD_MSG_PACK format, which includes additional type and length information when serialized
and will restore the stack exactly back to its pre-serialized state.
Hooks
The metadata API can be used in a variety of contexts and therefore its behaviour depends on user-provide callback functions, which we refer to as hooks. These hooks inform the API where to read and write metadata and optionally provide a custom implementation for allocation and de-allocation, if using a custom allocator.
struct md_hooks {
void *(*map)(void *userdata, size_t *n);
md_err (*write)(void *userdata, const void *src, size_t n);
void (*finalize)(void *userdata);
void *(*allocate)(size_t size, size_t align, void *userdata);
void (*deallocate)(void *ptr, void *userdata);
};
map- get a pointer to the start of a previously serialized metadata binary. The length of the binary is stored into the pointern.write- writenbytes fromsrcto somewhere.finalize- perform any cleanup operations i.e. close the file handle.allocate- allocatenbytes of memory.deallocatede-allocate a pointer previously allocated withallocate.
Basic Usage
The metadata API can effectively be used in 2-modes: reading metadata from an existing metadata binary; or creating metadata and writing it out to a new binary.
Reading from an existing binary
Construct an
md_hooksstruct with the requiredmap()callback as to where to read the data from.Initialize a metadata context with
md_init(). Since the map hook was provided this step will attempt to deserialize the binary. If it fails an error will be returned.Get a handle to a stack using
md_get_block(). If no stack with that name existsnullptris returned.Read the values from the stack using the appropriate calls based on the data e.g.
md_get_uint()ormd_get_zstr().When finished, release the context with
md_release_ctx(), this will destroy and de-allocate the context.
Writing metadata to a binary
Construct an
md_hooksstruct with the requiredwrite()andfinalize()hooks.Initialize an empty metadata context with
md_init().Register a new stack with a unique name to the context with
md_create_block().Push values to the stack using the appropriate type-compatible function e.g. push an unsigned integer with
md_push_uint().Once you have pushed all values to the stack, finalize the stack with
md_finalize_block().Finalize the context with
md_finalize_ctx.Release and de-allocate the context with
md_release_ctx().
