Kdb/q supports calling functions written in C from q. Some example use cases for this would include creating missing functions in q (such as native bit wise operators) or allowing proprietary functions that exist in C to be directly used from q.

For a simple example, let’s write two functions in C, being sure to include k.h and use its data types for the arguments and output types of the function. The latest k.h header file can be downloaded from github here

These two functions, smin and smax, will return the first or last symbol (in alphabetical order) from a given list of q symbols. Here is the code, we will save this to a file named smm.c.

#include "k.h"
#include <string.h>   // strcmp

K1(smin)
{
    if(xt!=KS) R krr("type");
    if(xn)
    {
        S m = xS[0];
        int n = xn;
        for(int i=1;i<n;i++) if(0<strcmp(m,xS[i])) m=xS[i];
        R ks(m);
    }
    R ks("");       // what should we return for empty list?
}

K1(smax)
{
    if(xt!=KS) R krr("type");
    S m = "";
    DO(xn,if(0>strcmp(m,xS[i]))m=xS[i]);
    R ks(m);
}

A Makefile is required to build the shared library smm.o, here is its contents:

smm.so: smm.c
gcc -std=gnu99 -DKXVER=3 -fPIC -c smm.c
gcc -shared -fPIC smm.o -o smm.so

Use make to create the shared library smm.o

$ make smm.o

We now have a shared library, let’s deploy it to a Managed kdb cluster. You can deploy code to clusters by identifying a zip file on S3 thru the code argument of the create-kx-cluster API. For this example, we have a directory, mycode, with a directory lib, that we have copied the smm.o file into.

sh-4.2$ ls -l mycode/lib
total 4
-rw-rw-r-- 1 ec2-user ec2-user 2184 Jan 16 22:02 smm.o

Create a zip file of the mycode directory’s contents, and copy it to an S3 location (defined as $S3_BUCKET) that will be given as the code argument in the create-kx-cluster operation.

sh-4.2$ cd mycode
sh-4.2$ zip -r -X ../mycode.zip .
sh-4.2$ aws s3 cp mycode.zip s3://$S3_BUCKET/mycode.zip

Once the cluster is created test the function by connecting to the running cluster and assign the shared library functions to q functions. This example is from a Jupyter notebook cell using pykx as the q client, connecting to the managed cluster.

%%q --host $host --port $port --user $username --pass $password

/ Assign the functions to q functions
smin:(`$"/opt/kx/app/code/lib/smm") 2:`smin,1
smax:(`$"/opt/kx/app/code/lib/smm") 2:`smax,1

With the functions now defined as q functions, you can call them like any other function in q

%%q --host $host --port $port --user $username --pass $password

/ Call the functions
smin `1`2`3`0`4`5
smax `1`2`3`0`4`5
smin `aaa`bbb`ccc`ddd`eee
smax `aaa`bbb`ccc`ddd`eee

0 5 aaa eee

And there you have it, a custom C function, callable from q, deployed to a FinSpace managed kdb cluster!

Production Deployment

For production deployments its best to use a q script to link shared library functions to q functions. A simple q example of this is below; we will save it to a file (smm.q) and put it into the same mycode directory as the smm.o file was in mycode/lib and zip everything together as before.

smin:(`$"lib/smm") 2:`smin,1
smax:(`$"lib/smm") 2:`smax,1

You can use smm.q as the initialization script of the cluster, identify it with the initialization-script argument when creating the cluster with create-kx-cluster. When the cluster starts, the identified initialization script will be run, linking the C functions to q functions.

Note the path is not absolute, when the cluster starts and the initialization script is run, the current directory iwill already be /opt/kx/app/code, so you can use reference paths.

KX References

https://code.kx.com/q/interfaces/using-c-functions/ https://code.kx.com/q/wp/capi/