Caching

[riastradh]

It might be nice if bulktracker supported downstream HTTP caches by:

1. providing ETag, Last-Modified, and Cache-Control header fields in responses to direct downstream caches, based, perhaps, on database mtimes; and
2. interpreting If-Match, If-None-Match, If-Unmodified-Since, and If-Modified-Since header fields in requests to bypass response generation as appropriate.

References:

• RFC 9111: HTTP Caching
• RFC 9110: HTTP Semantics, Sec. 13.2 "Evaluation of Preconditions"

[bsiegert]

This is a great suggestion. I thought this was a feature out of the box, but apparently it is not. I'll look into it.

[riastradh]

The tricky part will be a cheap but reliable way to compute ETag values. I looked around to see whether sqlite3 provides any kind of global modification counter for this, and it seems not. There are several candidates, but they don't work very well:

• The file change counter in the database header is not incremented for WAL databases.
• PRAGMA data_version is only changed for the current database connection; restart the process and it goes back to 1. (Also, it only changes if other database connections change the database concurrently while this one is open. Probably not relevant here but that rules it out for some applications.)
• SQLITE_FCNTL_DATA_VERSION is also only changed for the current database connection. (It does count the current database connection's changes, unlike PRAGMA data_version.)

So it might be necessary to hack it explicitly with another table and an ON UPDATE trigger, or by hashing a tuple of the highest rowids of all the tables (if the tables are all insert-only). Of course, the more work it takes to compute an ETag value, the less effective it is at saving resources by caching.

[bsiegert]

Most of the tables are append only. Occasionally, we remove older builds, so the MIN(id) increases. But I think that the MAX(id) could serve as a useful ETag for quite a few queries.