This will work regardless of whether you are using the webapi library from my
previous post (now available from conda: conda install -c jmcmurray webapi -
see this post
about q package management for more details on that) or another custom
.z.ph/.z.pp definition (or even the default q HTTP interface).
Step one is to generate your certificates; this is pretty simple using certbot from Let’s Encrypt. This is a fairly straightforward process, documented here [instructions for various OSes available]. For example on Ubuntu, once certbot is installed we simply do something like:
sudo certbot certonly --standalone -d jmcmurray.hopto.org
(obviously jmcmurray.hopto.org is a domain I control). Certs will be
generated in e.g. /etc/letsencrypt/live/jmcmurray.hopto.org
jonny@grizzly ~ $ sudo ls -lFh /etc/letsencrypt/live/jmcmurray.hopto.org
total 4.0K
lrwxrwxrwx 1 root root 43 Oct 25 15:08 cert.pem -> ../../archive/jmcmurray.hopto.org/cert1.pem
lrwxrwxrwx 1 root root 44 Oct 25 15:08 chain.pem -> ../../archive/jmcmurray.hopto.org/chain1.pem
lrwxrwxrwx 1 root root 48 Oct 25 15:08 fullchain.pem -> ../../archive/jmcmurray.hopto.org/fullchain1.pem
lrwxrwxrwx 1 root root 41 Dec 8 14:10 jmcmurray.hopto.org -> /etc/letsencrypt/live/jmcmurray.hopto.org/
lrwxrwxrwx 1 root root 46 Oct 25 15:08 privkey.pem -> ../../archive/jmcmurray.hopto.org/privkey1.pem
-rw-r--r-- 1 root root 682 Oct 25 15:08 README
As the certs are created as root, we have a couple of options. We could run our q server as root, allowing us to read the certs. Alternatively, we can copy (as root) to another readable location e.g.
jonny@grizzly ~ $ mkdir certs
jonny@grizzly ~ $ sudo su
root@grizzly:/home/jonny# cp /etc/letsencrypt/live/jmcmurray.hopto.org/* certs/
cp: -r not specified; omitting directory '/etc/letsencrypt/live/jmcmurray.hopto.org/jmcmurray.hopto.org'
root@grizzly:/home/jonny# exit
exit
jonny@grizzly ~ $ ll certs/
total 20K
-rw-r--r-- 1 root root 2.2K Dec 8 15:06 cert.pem
-rw-r--r-- 1 root root 1.7K Dec 8 15:06 chain.pem
-rw-r--r-- 1 root root 3.8K Dec 8 15:06 fullchain.pem
-rw-r--r-- 1 root root 1.7K Dec 8 15:06 privkey.pem
-rw-r--r-- 1 root root 682 Dec 8 15:06 README
Now we can set the necessary environment variables & load up our q session:
jonny@grizzly ~ $ export SSL_CERT_FILE=~/certs/fullchain.pem
jonny@grizzly ~ $ export SSL_KEY_FILE=~/certs/privkey.pem
jonny@grizzly ~ $ q -E 1 -p 8100
KDB+ 3.6 2018.05.17 Copyright (C) 1993-2018 Kx Systems
l32/ 2()core 1944MB jonny grizzly 127.0.1.1 NONEXPIRE
q).utl.require"webapi"
q)\l git/qwebapi/example.q
(Here I load my webapi module previously mentioned & the same example script
I used in my previous post on HTTP APIs)
Now, we can query the API over HTTPS:
All in all, the process is pretty simple & straightforward, and takes about 5 minutes. As a couple of closing notes:
- Using
-E 1as I showed here still allows HTTP access; with-E 2only HTTPS is allowed (however, HTTP requests will not be redirected to HTTPS - I’ll be looking into this in future) - As per the kx docs
-u 1should be used to prevent remote access to your server key file, & q should not be run from a directory where the keys can be accessed
UPDATE: After publishing this post, I realised that kdb+ seemingly does not publish the full certificate chain it is provided, only the server certificate.
This means that, when using Let’s Encrypt, the intermediate LE certificate is not provided to clients. Some clients (e.g. web browsers) will download the intermediate(s) as necessary & the user will likely not notice anything is “wrong”. Other clients (e.g. another q session) will not, and will therefore fail to verify the server certificate.
Report from SSL Labs
In order to send an HTTPS query from another q session, you will need to either
disable SSL server verification (set env var SSL_VERIFY_SERVER=NO before
starting q) or add your server cert to the CA bundle you are using.
