cyrusdb API

Intro

The cyrusdb API is a common interface to a key-value store, used throughout the Cyrus code. It allows a choice of different backends for different access patterns, while ensuring a consistent interface.

This document will describe the interface, and how to use the cyrusdb interface from within parts of Cyrus code, as well as how to implement your own backend

If you pass incorrect values to these APIs, you will get an assertion failure in most cases. That’s generally considered safer than silently breaking things. Exceptions are noted below.

Code Layout

The implementation of each interface is in lib/cyrusdb_NAME.c, for example lib/cyrusdb_flat.c. General functions are in lib/cyrusdb.c and the interface in lib/cyrusdb.h.

Configuration

The name of the backend for each of the main internal databases can be configured in imapd.conf, for example: annotation_db: skiplist. This is then read in imap/global.h and imap/global.c during startup, so that the global variable config_annotation_db gets set to the configured backend name:

config_annotation_db = config_getstring(IMAPOPT_ANNOTATION_DB);

(Beware the misleading naming here: config_annotation_db is a string describing the backend used by the database, not, say, its location on disk.)

Internally, the main module for each database sets up struct of pointers to the cyrusdb functions it implements, which is registered in lib/cyrusdb.c

lib/cyrusdb.c provides backend-agnostic wrapper functions for interacting with cyrusdb databases.

A full example

struct db *db = NULL;
struct txn *tid = NULL;
const char *filename = NULL;
int flags = CYRUSDB_CREATE;

cyrus_init(alt_config, "toolname", 0);

filename = config_getstring(IMAPOPT_ANNOTATION_DB_PATH);

r = cyrusdb_open(config_annotation_db, filename, flags, &db);

r = cyrusdb_fetch(db, key, keylen, &data, &datalen, &tid);

r = cyrusdb_commit(db, tid);

r = cyrusdb_close(db);

cyrus_done();

Note that you always open a database first, and close it at the end. You must always call cyrus_init() and cyrus_done() to properly initialize and clean up the cyrusdb environments.

This example also uses a transaction, meaning that the database is locked in exclusive mode between the ‘fetch’ (the first use of the transaction) and the commit.

About Transactions

The cyrusdb interface works in two modes - transactional and non-transactional. The value of the ‘tid’ parameter decides which mode is used. There are three possible values:

  • NULL - non-transactional. Will create a temporary lock for the duration of the current action - either a write lock for “store” or a read lock for “fetch”. If you call “foreach”, the lock will be dropped between each record fetched

  • Pointer to NULL - transactional, transaction not yet started. Will always take a write lock on the database, and update the pointer to point to the new transaction.

  • Pointer to a valid transaction. Will keep using this transaction

If you are currently in a transaction, you MUST pass the same transaction to every database call. It is not possible to mix or nest transactions. There is one exception in the skiplist backend: If you pass NULL to a fetch or foreach while the database is in a transaction, it will silently do the read in the current transaction rather than returning an error

API Reference

All functions follow the normal C API of returning ‘0’ on success, and an error code on failure

cyrusdb_init(void)

Is called once per process. Don’t call this yourself, use cyrus_init(). No other calls will be made until this is called.

cyrusdb_done(void)

The opposite of cyrusdb_init() - called once per process to do any cleaning up after all database usage is finished. Don’t call this yourself, use cyrus_done().

cyrusdb_sync(const char *backend)

Perform a checkpoint of the database environment. Used by berkeley backend. Is called by ctl_cyrusdb -c on a regular basis

cyrusdb_open(const char *backend, const char *fname, int flags, struct db **retdb)

Opens the database with the specified ‘file name’ (or other descriptor, for example the sql backend is not a filename), and if successful returns an opaque database structure

Flags:

  • CYRUSDB_CREATE - create the database if it doesn’t exist

  • CYRUSDB_MBOXSORT - sort ‘.’ first, so folder listing is correct

Errors:

  • CYRUSDB_IOERROR - if there is any error reading the file, or any corruption detected while loading the file

cyrusdb_close(struct db *db)

Close the named database. Will release any locks if they are still held, but it’s bad practice to close without committing or aborting, so the backend should log an error

Errors:

  • CYRUSDB_IOERROR - if there are any errors during close

cyrusdb_fetch(struct db *db, const char *key, size_t keylen, const char **data, size_t *datalen, struct txn **tidptr)

cyrusdb_fetchlock(struct db *db, const char *key, size_t keylen, const char **data, size_t *datalen, struct txn **tidptr)

Fetch the value for the exact key given by key and keylen. If data is not NULL, set datalen and return a valid pointer to the start of the value.

Fetchlock is identical to fetch, but gives a hint to the database that the record is likely to be modified soon.

NOTE: it is possible to store a key with a zero length data record, in which case *datalen will be set to zero, and *data will be set to a non-NULL value

It is an error to call fetch with a NULL key or a zero keylen

It is an error to call fetch with a NULL datalen and a non-NULL data, however it is acceptable to call with a NULL data and a non-NULL datalen if you are only interested in the length

Errors:

  • CYRUSDB_IOERROR - if any error occurs reading from the database

  • CYRUSDB_LOCKED - if tidptr is incorrect

  • CYRUSDB_NOTFOUND - if there is no record that matches the key

cyrusdb_foreach(struct db *db, const char *prefix, size_t prefixlen, foreach_p *goodp, foreach_p *procp, void *rock, struct txn **tidptr)

cyrusdb_forone(struct db *db, const char *key, size_t keylen, foreach_p *goodp, foreach_p *procp, void *rock, struct txn **tidptr)

cyrusdb_foreach() iterates over all records matching the given prefix, in database order (which may be MBOXLIST sort, depending on the parameters given to open

It is legal to give a NULL pointer as prefix if prefixlen is zero, in which case it will return all records in the database. It is an error to give a non-zero prefixlen with a NULL prefix.

cyrusdb_forone() “iterates” over the single record matched by the given key. If you’ve already built callbacks for processing each record from a foreach, this lets you use the same interface to process a single record.

goodp - this function is only used for deciding if the record needs to be further processed. It can be used for basic filtering, and returns true (non-zero) to process, or zero to skip and move straight to the next record. Because goodp can’t make any database changes, it doesn’t break the lock, so it’s faster to use goodp to filter records if you don’t need to process all of them. NULL is a legal value for goodp, and means that all records will be processed.

procp - procp is the main callback function. If you use foreach in non-transactional mode, the database is unlocked before calling procp, and locked again afterwards. You are allowed to add, delete or modify values in the same database from within procp. If procp returns non-zero, the foreach loop breaks at this point, and the return value of the foreach becomes the return value of procp. If procp returns zero, the foreach loop will continue at the NEXT record by sort order, regardless of whether the current record has changed or been removed. procp MUST NOT be NULL.

Errors:

  • procp_result - whatever your callback returns

  • CYRUSDB_IOERROR - if any error occurs while reading

  • CYRUSDB_LOCKED - if tidptr is incorrect

cyrusdb_create(struct db *db, const char *key, size_t keylen, const char *data, size_t datalen, struct txn **tidptr)

cyrusdb_store(struct db *db, const char *key, size_t keylen, const char *data, size_t datalen, struct txn **tidptr)

Create a new record or replace an existing one. The only difference between these two is that cyrusdb_create will return an error if the record already exists, while cyrusdb_store will replace it

If tidptr is NULL, create/store will take a write lock for the duration of the action.

Any failure during create/store will abort the current transaction as well as returning an error

It is legal to pass NULL for the data field ONLY if datalen is zero. It is not legal to pass NULL for key or zero for keylen

Errors:

  • CYRUSDB_IOERROR - any error to write to the database

  • CYRUSDB_LOCKED - if tidptr is incorrect

  • CYRUSDB_EXISTS - if cyrusdb_create is called on an existing key

  • CYRUSDB_AGAIN - if a deadlock is created. The current transaction has been aborted, but a retry may succeed

cyrusdb_delete(struct db *db, const char *key, size_t keylen, struct txn **tidptr, int force)

Delete the given record from the database. If force is true, then succeed even if the record doesn’t currently exist.

It is not legal to pass NULL for key or zero for keylen

Errors:

  • CYRUSDB_IOERROR - any error to write to the database

  • CYRUSDB_LOCKED - if tidptr is incorrect

  • CYRUSDB_NOTFOUND - if force is not set and the key doesn’t exist

  • CYRUSDB_AGAIN - if a deadlock is created. The current transaction has been aborted, but a retry may succeed

cyrusdb_commit(struct db *db, struct txn *tid)

Commit the current transaction. tid will not be valid after this call, regardless of success

If the commit fails, it will attempt to abort the transaction

Errors:

  • CYRUSDB_IOERROR - any error to write to the database

  • CYRUSDB_LOCKED - if tidptr is incorrect

  • CYRUSDB_AGAIN - if a deadlock is created. The current transaction has been aborted, but a retry may succeed

cyrusdb_abort(struct db *db, struct txn *tid)

Abort the current transaction. tid will not be valid after this call, regardless of success

Attempt to roll back all changes made in the current transaction.

Errors:

  • CYRUSDB_IOERROR - any error to write to the database

  • CYRUSDB_LOCKED - if tidptr is incorrect

cyrusdb_dump(struct db *db, int detail)

Optional function to dump the internal structure of the database to stdout for debugging purposes. Don’t use.

cyrusdb_consistent(struct db *db)

Check if the DB is internally consistent. Looks pretty bogus, and isn’t used anywhere. Don’t use.