From sebastiano at tronto.net Tue Jul 7 16:23:15 2026 From: sebastiano at tronto.net (Sebastiano Tronto) Date: Tue, 7 Jul 2026 18:23:15 +0200 Subject: [PATCH] Add title to site readme page Message-ID: <20260707162314.15076-2-sebastiano@tronto.net> Hello, I am playing around with cgit and I noticed that the title was missing in my about page (site readme). I am not sure this is the correct way to fix it, but it works on my machine. Signed-off-by: Sebastiano Tronto --- ui-repolist.c | 1 + 1 file changed, 1 insertion(+) diff --git a/ui-repolist.c b/ui-repolist.c index 1b224cf..4a70917 100644 --- a/ui-repolist.c +++ b/ui-repolist.c @@ -368,6 +368,7 @@ void cgit_print_repolist(void) void cgit_print_site_readme(void) { + ctx.page.title = ctx.cfg.root_title; cgit_print_layout_start(); if (!ctx.cfg.root_readme) goto done; -- 2.54.0 From aiden at aidenw.net Wed Jul 8 13:38:21 2026 From: aiden at aidenw.net (Aiden Woodruff) Date: Wed, 08 Jul 2026 09:38:21 -0400 Subject: [PATCH] Add title to site readme page In-Reply-To: <20260707162314.15076-2-sebastiano@tronto.net> References: <20260707162314.15076-2-sebastiano@tronto.net> Message-ID: <7a3b70590115fd40bbe9478e02046a1b@aidenw.net> On 2026-07-07 12:23, Sebastiano Tronto wrote: > Hello, > > I am playing around with cgit and I noticed that the title was missing > in my about page (site readme). I am not sure this is the correct way > to fix it, but it works on my machine. Good catch Sebastiano! I had never even noticed that. -- Aiden Woodruff From sgothel at jausoft.com Sun Jul 19 15:34:04 2026 From: sgothel at jausoft.com (=?UTF-8?Q?Sven_G=C3=B6thel?=) Date: Sun, 19 Jul 2026 17:34:04 +0200 Subject: [PATCH 0/4] Patch update, mostly server DoS related Message-ID: <08852c71-94af-4c95-bf84-f9408f8d3fac@jausoft.com> I will send 4 current patched from my repo , namely - 1: cgit: add_repo: Assign default branch from git HEAD to repo->defbranch if undefined - 2: cgit: print_slot: Avoid Slow-Attack via `client-io-idle-timeout` and `client-io-min-rate` - 3: Resolve concurrent initial cache fill post lock. Release lock after fill. - 4: Add cfg cache-lock-fail behavior if failing to acquire cache-file lock last 2 are new. I welcome a merge and/or discussion with the maintainer. Motivation of this work was to fight DoS attack on two cgit server, reducing the load by using - send timeout and min-bandwidth to client - rely on cache-lock (actually a concurrency fix) - fail on cache-lock failure -> error page (new default) Cheers, ~Sven -- mailto:sgothel at jausoft.com ; http://jausoft.com land : +49 471 9008 7111 From sgothel at jausoft.com Sun Jul 19 17:03:45 2026 From: sgothel at jausoft.com (=?UTF-8?Q?Sven_G=C3=B6thel?=) Date: Sun, 19 Jul 2026 19:03:45 +0200 Subject: [PATCH 1/4] cgit: add_repo: Assign default branch from git HEAD to, repo->defbranch if undefined Message-ID: From 5ccffc927460e651da10a6b0be2e9786129355bb Mon Sep 17 00:00:00 2001 From: Sven G?thel Date: Sat, 30 May 2026 00:56:55 +0200 Subject: cgit: add_repo: Assign default branch from git HEAD to repo->defbranch if undefined MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Currently, if defbranch is not `master` and no manual config performed, repo age is undefined. This patch reads the default branch from git HEAD if not configured via cgitrc and uses it in all commands like repo-list age, selected at log etc. add_repo is called from scan_path early when gathering and caching server repositories. Signed-off-by: Sven G?thel diff --git a/scan-tree.c b/scan-tree.c index c120efe..b701d00 100644 --- a/scan-tree.c +++ b/scan-tree.c @@ -79,6 +79,32 @@ static char *xstrrchr(char *s, char *from, int c) return from < s ? NULL : from; } +static char *read_git_head_branch(const char *repo_path) { + struct strbuf path = STRBUF_INIT; + size_t size; + char *line = NULL; + + strbuf_addf(&path, "%s/HEAD", repo_path); + + if (read_first_line(path.buf, &line, &size)) { + strbuf_release(&path); + free(line); + return NULL; + } + strbuf_release(&path); + + const char *refname; + char *result = NULL; + + if (!line || !skip_prefix(line, "ref: refs/heads/", &refname)) { + free(line); + return NULL; + } + result = xstrdup(refname); + free(line); + return result; +} + static void add_repo(const char *base, struct strbuf *path) { struct stat st; @@ -180,6 +206,9 @@ static void add_repo(const char *base, struct strbuf *path) if (!stat(path->buf, &st)) parse_configfile(path->buf, &scan_tree_repo_config); + if (!repo->defbranch) + repo->defbranch = read_git_head_branch(repo->path); + strbuf_release(&rel); } From sgothel at jausoft.com Sun Jul 19 17:04:49 2026 From: sgothel at jausoft.com (=?UTF-8?Q?Sven_G=C3=B6thel?=) Date: Sun, 19 Jul 2026 19:04:49 +0200 Subject: [PATCH 2/4] cgit: print_slot: Avoid Slow-Attack via `client-io-idle-timeout` and, `client-io-min-rate` Message-ID: <2844b1be-d8c8-4bba-b1fd-0c1fdbc4061d@jausoft.com> >From 3612cb2a9fbec0052ae6d8f5b5a4a0bb0c3aed35 Mon Sep 17 00:00:00 2001 From: Sven G?thel Date: Tue, 2 Jun 2026 04:15:57 +0200 Subject: cgit: print_slot: Avoid Slow-Attack via `client-io-idle-timeout` and `client-io-min-rate` default `cgitrc` configurable values - `client-io-idle-timeout`: 20s - `client-io-min-rate`: 500 Bps - Note GSM 2G -- 9.6Kbps or ~1200 Bps maximum return ETIMEDOUT if either - idle time from last successfull sendfile/write > `client-io-idle-timeout` - total time spend exceeds max(`client-io-idle-timeout`, size/`client-io-min-rate`) seconds In case of timeout, the event will be logged with REMOTE_ADDR, a potential bad actor if repetitive. Further adds `cgitrc` config value `log-level`, enabling verbose logging if set above zero. diff --git a/cache.c b/cache.c index e70af13..28e7180 100644 --- a/cache.c +++ b/cache.c @@ -82,32 +82,145 @@ static int close_slot(struct cache_slot *slot) return err; } +#define MY_MIN(X, Y) (((X) < (Y)) ? (X) : (Y)) +#define MY_MAX(X, Y) (((X) > (Y)) ? (X) : (Y)) + +static int sendslot_to_idle(time_t tStart, time_t tLastSend, time_t tNow, + size_t off, size_t size, const char *cache_name) +{ + const time_t td_total = tNow - tStart; + const time_t td_idle = tNow - tLastSend; + const long rate = off / MY_MAX(1, td_total); + cache_log("[cgit] send_slot timeout idle %lds: sending cache " + "%s (%ld/%ld bytes) to client `%s` " + "within [total %lds, idle %lds, rate %ld Bps]\n", + td_idle, cache_name, off, size, ctx.env.remote_addr, + td_total, td_idle, rate); + return ETIMEDOUT; +} + +static int sendslot_to_minrate(time_t tStart, time_t tNow, size_t off, + size_t size, const char *cache_name) +{ + const time_t td_total = tNow - tStart; + const long rate = off / MY_MAX(1, td_total); + cache_log("[cgit] send_slot timeout rate-limit %ld Bps: sending " + "cache %s (%ld/%ld bytes) to client `%s` " + "within [total %lds, rate %ld Bps]\n", + ctx.cfg.client_io_min_rate, cache_name, off, size, ctx.env.remote_addr, + td_total, rate); + return ETIMEDOUT; +} + +static int sendslot_ok(time_t tStart, time_t tNow, size_t size, + const char *cache_name) +{ + if (ctx.cfg.log_level > 90) { + const time_t td_total = tNow - tStart; + const long rate = size / MY_MAX(1, td_total); + cache_log("[cgit] send_slot status: sent cache %s (%ld bytes) to " + "client `%s` " + "within [total %lds, rate %ld Bps]\n", + cache_name, size, ctx.env.remote_addr, td_total, rate); + } + return 0; +} + +static int sendslot_ok2(time_t tStart, size_t size, const char *cache_name) +{ + if (ctx.cfg.log_level > 90) { + return sendslot_ok(tStart, time(NULL), size, cache_name); + } + return 0; +} + +static ssize_t write_in_full_to(int fd, const void *buf, size_t count, off_t *total_out, + time_t tStart, time_t *tLastSend, time_t to_max) +{ + if (!count) { + return 0; + } + const char *p = buf; + ssize_t total = 0; + time_t tNow = *tLastSend; + + do { + if (tNow - *tLastSend >= ctx.cfg.client_io_idle_timeout) { + errno = ETIMEDOUT; + return -2; + } + if (tNow - tStart > to_max) { + errno = ETIMEDOUT; + return -3; + } + + ssize_t written = write(fd, p, MY_MIN(count, MAX_IO_SIZE)); + tNow = time(NULL); + if (written < 0) { + if (errno == EINTR) + continue; + if (errno == EAGAIN || errno == EWOULDBLOCK) { + struct pollfd pfd; + pfd.fd = fd; + pfd.events = POLLOUT; + // no need to check for errors, + // subsequent read/write will detect unrecoverable errors + poll(&pfd, 1, -1); + continue; + } + return -1; + } else if (written > 0) { + *total_out += written; + count -= written; + p += written; + total += written; + *tLastSend = tNow; + if (!count) + return total; + } + } while (1); +} + /* Print the content of the active cache slot (but skip the key). */ static int print_slot(struct cache_slot *slot) { - off_t off; -#ifdef HAVE_LINUX_SENDFILE - off_t size; -#endif + time_t tStart = time(NULL); + time_t tLastSend = tStart; + time_t tNow = tStart; - off = slot->keylen + 1; + off_t off = slot->keylen + 1; + off_t size = slot->cache_st.st_size; -#ifdef HAVE_LINUX_SENDFILE - size = slot->cache_st.st_size; + if (!size) { + return sendslot_ok(tStart, tNow, size, slot->cache_name); + } + const time_t to_min_rate = + MY_MAX(ctx.cfg.client_io_idle_timeout, size / ctx.cfg.client_io_min_rate); +#ifdef HAVE_LINUX_SENDFILE do { - ssize_t ret; - ret = sendfile(STDOUT_FILENO, slot->cache_fd, &off, size - off); - if (ret < 0) { + if (tNow - tLastSend >= ctx.cfg.client_io_idle_timeout) + return sendslot_to_idle(tStart, tLastSend, tNow, + off, size, slot->cache_name); + if (tNow - tStart > to_min_rate) + return sendslot_to_minrate(tStart, tNow, + off, size, slot->cache_name); + + ssize_t count = + sendfile(STDOUT_FILENO, slot->cache_fd, &off, size - off); + tNow = time(NULL); + if (count < 0) { if (errno == EAGAIN || errno == EINTR) continue; /* Fall back to read/write on EINVAL or ENOSYS */ if (errno == EINVAL || errno == ENOSYS) break; return errno; + } else if (count > 0) { + tLastSend = tNow; + if (off == size) + return sendslot_ok(tStart, tNow, size, slot->cache_name); } - if (off == size) - return 0; } while (1); #endif @@ -115,14 +228,26 @@ static int print_slot(struct cache_slot *slot) return errno; do { - ssize_t ret; - ret = xread(slot->cache_fd, slot->buf, sizeof(slot->buf)); - if (ret < 0) + ssize_t count = xread(slot->cache_fd, slot->buf, sizeof(slot->buf)); + if (count < 0) return errno; - if (ret == 0) - return 0; - if (write_in_full(STDOUT_FILENO, slot->buf, ret) < 0) + + ssize_t res; + if ((res = write_in_full_to(STDOUT_FILENO, slot->buf, count, &off, + tStart, &tLastSend, to_min_rate)) < 0) + { + if (ETIMEDOUT == errno) { + if (-2 == res) + return sendslot_to_idle(tStart, tLastSend, time(NULL), + off, size, slot->cache_name); + else if (-3 == res) + return sendslot_to_minrate(tStart, time(NULL), + off, size, slot->cache_name); + } return errno; + } + if (off == size || !count /* should be redundant */) + return sendslot_ok2(tStart, size, slot->cache_name); } while (1); } diff --git a/cgit.c b/cgit.c index ca318e8..26b4045 100644 --- a/cgit.c +++ b/cgit.c @@ -129,7 +129,9 @@ static void config_cb(const char *name, const char *value) { const char *arg; - if (!strcmp(name, "section")) + if (!strcmp(name, "log-level")) + ctx.cfg.log_level = atoi(value); + else if (!strcmp(name, "section")) ctx.cfg.section = strdup_first_line(value); else if (!strcmp(name, "repo.url")) ctx.repo = cgit_add_repo(value); @@ -215,6 +217,10 @@ static void config_cb(const char *name, const char *value) ctx.cfg.cache_scanrc_ttl = atoi(value); else if (!strcmp(name, "cache-static-ttl")) ctx.cfg.cache_static_ttl = atoi(value); + else if (!strcmp(name, "client-io-idle-timeout")) + ctx.cfg.client_io_idle_timeout = atoi(value); + else if (!strcmp(name, "client-io-min-rate")) + ctx.cfg.client_io_min_rate = atol(value); else if (!strcmp(name, "cache-dynamic-ttl")) ctx.cfg.cache_dynamic_ttl = atoi(value); else if (!strcmp(name, "cache-about-ttl")) @@ -251,15 +257,16 @@ static void config_cb(const char *name, const char *value) ctx.cfg.max_commit_count = atoi(value); else if (!strcmp(name, "project-list")) ctx.cfg.project_list = strdup_first_line(expand_macros(value)); - else if (!strcmp(name, "scan-path")) + else if (!strcmp(name, "scan-path")) { + ctx.cfg.scan_path = strdup_first_line(expand_macros(value)); if (ctx.cfg.cache_size) - process_cached_repolist(expand_macros(value)); + process_cached_repolist(ctx.cfg.scan_path); else if (ctx.cfg.project_list) - scan_projects(expand_macros(value), + scan_projects(ctx.cfg.scan_path, ctx.cfg.project_list); else - scan_tree(expand_macros(value)); - else if (!strcmp(name, "scan-hidden-path")) + scan_tree(ctx.cfg.scan_path); + } else if (!strcmp(name, "scan-hidden-path")) ctx.cfg.scan_hidden_path = atoi(value); else if (!strcmp(name, "section-from-path")) ctx.cfg.section_from_path = atoi(value); @@ -381,6 +388,8 @@ static void prepare_context(void) ctx.cfg.cache_scanrc_ttl = 15; ctx.cfg.cache_dynamic_ttl = 5; ctx.cfg.cache_static_ttl = -1; + ctx.cfg.client_io_idle_timeout = 20; + ctx.cfg.client_io_min_rate = 500; ctx.cfg.case_sensitive_sort = 1; ctx.cfg.branch_sort = 0; ctx.cfg.commit_sort = 0; @@ -426,6 +435,7 @@ static void prepare_context(void) ctx.env.server_port = getenv("SERVER_PORT"); ctx.env.http_cookie = getenv("HTTP_COOKIE"); ctx.env.http_referer = getenv("HTTP_REFERER"); + ctx.env.remote_addr = getenv("REMOTE_ADDR"); ctx.env.content_length = getenv("CONTENT_LENGTH") ? strtoul(getenv("CONTENT_LENGTH"), NULL, 10) : 0; ctx.env.authenticated = 0; ctx.page.mimetype = "text/html"; @@ -871,6 +881,15 @@ static void print_repolist(FILE *f, struct cgit_repolist *list, int start) for (i = start; i < list->count; i++) print_repo(f, &list->repos[i]); } +static void print_config(FILE *f, const char *prefix) +{ + // TODO: May need to be completed, if desired to be functional + fprintf(f, "%slog-level=%d\n", prefix, ctx.cfg.log_level); + fprintf(f, "%sproject-list=%s\n", prefix, ctx.cfg.project_list); + fprintf(f, "%sscan-path=%s\n", prefix, ctx.cfg.scan_path); + fprintf(f, "%sclient-io-idle-timeout=%d\n", prefix, ctx.cfg.client_io_idle_timeout); + fprintf(f, "%sclient-io-min-rate=%ld\n", prefix, ctx.cfg.client_io_min_rate); +} /* Scan 'path' for git repositories, save the resulting repolist in 'cached_rc' * and return 0 on success. @@ -1009,12 +1028,14 @@ static void cgit_parse_args(int argc, const char **argv) * NOTE: We assume that there aren't more than 8 * different snapshot formats supported by cgit... */ + ctx.cfg.scan_path = strdup_first_line(arg); ctx.cfg.snapshots = 0xFF; scan++; scan_tree(arg); } } if (scan) { + print_config(stdout, "[cgit] scan: "); qsort(cgit_repolist.repos, cgit_repolist.count, sizeof(struct cgit_repo), cmp_repos); print_repolist(stdout, &cgit_repolist, 0); @@ -1067,6 +1088,8 @@ int cmd_main(int argc, const char **argv) cgit_parse_args(argc, argv); parse_configfile(expand_macros(ctx.env.cgit_config), config_cb); + if (ctx.cfg.log_level) + print_config(stderr, "[cgit] init: "); ctx.repo = NULL; http_parse_querystring(ctx.qry.raw, querystring_cb); diff --git a/cgit.h b/cgit.h index 7d7ece7..f16e501 100644 --- a/cgit.h +++ b/cgit.h @@ -194,6 +194,7 @@ struct cgit_query { }; struct cgit_config { + int log_level; ///< defaults to zero char *agefile; char *cache_root; char *clone_prefix; @@ -207,6 +208,7 @@ struct cgit_config { char *mimetype_file; char *module_link; char *project_list; + char *scan_path; struct string_list readme; struct string_list css; char *robots; @@ -227,6 +229,10 @@ struct cgit_config { int cache_static_ttl; int cache_about_ttl; int cache_snapshot_ttl; + /* idle timeout in seconds between sending/receiving chunks of the cached body to/from the client. Defaults to 20s. */ + int client_io_idle_timeout; + /* minimum transfer rate in Bps for sending/receiving a full cached body to/from the client. Defaults to 500 Bps. */ + long client_io_min_rate; int case_sensitive_sort; int embedded; int enable_filter_overrides; @@ -302,6 +308,7 @@ struct cgit_environment { const char *server_port; const char *http_cookie; const char *http_referer; + const char *remote_addr; unsigned int content_length; int authenticated; }; diff --git a/cgitrc.5.txt b/cgitrc.5.txt index 7c39bf9..a6016ee 100644 --- a/cgitrc.5.txt +++ b/cgitrc.5.txt @@ -100,6 +100,14 @@ cache-static-ttl:: version of repository pages accessed with a fixed SHA1. See also: "CACHE". Default value: -1". +client-io-idle-timeout:: + IDLE timeout in seconds between sending/receiving chunks + of the cached body to/from the client. Default value: "20". + +client-io-min-rate:: + Minimum transfer rate in Bps for sending/receiving a full cached + body to/from the client. Default value "500". + clone-prefix:: Space-separated list of common prefixes which, when combined with a repository url, generates valid clone urls for the repository. This @@ -456,6 +464,9 @@ virtual-root:: NOTE: cgit has recently learned how to use PATH_INFO to achieve the same kind of virtual urls, so this option will probably be deprecated. +log-level:: + Specifies the logging level. Above zero adds verbose logging. + Default value: "0". REPOSITORY SETTINGS ------------------- From sgothel at jausoft.com Sun Jul 19 17:05:53 2026 From: sgothel at jausoft.com (=?UTF-8?Q?Sven_G=C3=B6thel?=) Date: Sun, 19 Jul 2026 19:05:53 +0200 Subject: [PATCH 3/4] Resolve concurrent initial cache fill post lock. Release lock after, fill. Message-ID: <937d0add-0304-45ac-bb4e-1a6b5656320b@jausoft.com> >From 0771f700a9788cbe7e9f923ebeb8c4b6bf1ab1fa Mon Sep 17 00:00:00 2001 From: Sven G?thel Date: Sun, 19 Jul 2026 06:17:56 +0200 Subject: Resolve concurrent initial cache fill post lock. Release lock after fill. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enhance verbose cache_log logging unlock_slot - add: unlock the cache-lock file's lock - this frees the lock right after successfull initial fill and allows concurrent usage lock_slot - lock: add retry on EAGAIN, waiting for concurrent process - uses `cache_lock_timeout` (default 10s) - post-lock: test if cache slot became valid from concurrent process - if successful, unlock_lock and proceed w/ cached_fd - otherwise close_slot (cached_fd) for new cache-slot - failure on truncate and write issues unlock_slot process_slot - remove complicated is_expired branch, simply utilize concurrent check in lock_slot. - If slot doesn't exist or is_expired, close_slot and lock_slot. If the latter produced a cached_fd, use it. Otherwise fill_slot and swap lock-file as new cache-file. Signed-off-by: Sven G?thel diff --git a/cache.c b/cache.c index 28e7180..ca22194 100644 --- a/cache.c +++ b/cache.c @@ -16,6 +16,10 @@ #include "cgit.h" #include "cache.h" #include "html.h" +#include +#include +#include +#include #ifdef HAVE_LINUX_SENDFILE #include #endif @@ -76,8 +80,7 @@ static int close_slot(struct cache_slot *slot) if (slot->cache_fd > 0) { if (close(slot->cache_fd)) err = errno; - else - slot->cache_fd = -1; + slot->cache_fd = -1; } return err; } @@ -86,50 +89,52 @@ static int close_slot(struct cache_slot *slot) #define MY_MAX(X, Y) (((X) > (Y)) ? (X) : (Y)) static int sendslot_to_idle(time_t tStart, time_t tLastSend, time_t tNow, - size_t off, size_t size, const char *cache_name) + size_t off, size_t size, struct cache_slot *slot) { const time_t td_total = tNow - tStart; const time_t td_idle = tNow - tLastSend; const long rate = off / MY_MAX(1, td_total); cache_log("[cgit] send_slot timeout idle %lds: sending cache " - "%s (%ld/%ld bytes) to client `%s` " + "%s (%s) (%ld/%ld bytes) to client `%s` " "within [total %lds, idle %lds, rate %ld Bps]\n", - td_idle, cache_name, off, size, ctx.env.remote_addr, + td_idle, slot->cache_name, slot->key, off, size, ctx.env.remote_addr, td_total, td_idle, rate); return ETIMEDOUT; } static int sendslot_to_minrate(time_t tStart, time_t tNow, size_t off, - size_t size, const char *cache_name) + size_t size, struct cache_slot *slot) { const time_t td_total = tNow - tStart; const long rate = off / MY_MAX(1, td_total); cache_log("[cgit] send_slot timeout rate-limit %ld Bps: sending " - "cache %s (%ld/%ld bytes) to client `%s` " + "cache %s (%s) (%ld/%ld bytes) to client `%s` " "within [total %lds, rate %ld Bps]\n", - ctx.cfg.client_io_min_rate, cache_name, off, size, ctx.env.remote_addr, + ctx.cfg.client_io_min_rate, slot->cache_name, slot->key, + off, size, ctx.env.remote_addr, td_total, rate); return ETIMEDOUT; } static int sendslot_ok(time_t tStart, time_t tNow, size_t size, - const char *cache_name) + struct cache_slot *slot) { if (ctx.cfg.log_level > 90) { const time_t td_total = tNow - tStart; const long rate = size / MY_MAX(1, td_total); - cache_log("[cgit] send_slot status: sent cache %s (%ld bytes) to " + cache_log("[cgit] send_slot status: sent cache %s (%s) %ld bytes) to " "client `%s` " "within [total %lds, rate %ld Bps]\n", - cache_name, size, ctx.env.remote_addr, td_total, rate); + slot->cache_name, slot->key, + size, ctx.env.remote_addr, td_total, rate); } return 0; } -static int sendslot_ok2(time_t tStart, size_t size, const char *cache_name) +static int sendslot_ok2(time_t tStart, size_t size, struct cache_slot *slot) { if (ctx.cfg.log_level > 90) { - return sendslot_ok(tStart, time(NULL), size, cache_name); + return sendslot_ok(tStart, time(NULL), size, slot); } return 0; } @@ -192,7 +197,7 @@ static int print_slot(struct cache_slot *slot) off_t size = slot->cache_st.st_size; if (!size) { - return sendslot_ok(tStart, tNow, size, slot->cache_name); + return sendslot_ok(tStart, tNow, size, slot); } const time_t to_min_rate = MY_MAX(ctx.cfg.client_io_idle_timeout, size / ctx.cfg.client_io_min_rate); @@ -200,11 +205,9 @@ static int print_slot(struct cache_slot *slot) #ifdef HAVE_LINUX_SENDFILE do { if (tNow - tLastSend >= ctx.cfg.client_io_idle_timeout) - return sendslot_to_idle(tStart, tLastSend, tNow, - off, size, slot->cache_name); + return sendslot_to_idle(tStart, tLastSend, tNow, off, size, slot); if (tNow - tStart > to_min_rate) - return sendslot_to_minrate(tStart, tNow, - off, size, slot->cache_name); + return sendslot_to_minrate(tStart, tNow, off, size, slot); ssize_t count = sendfile(STDOUT_FILENO, slot->cache_fd, &off, size - off); @@ -219,7 +222,7 @@ static int print_slot(struct cache_slot *slot) } else if (count > 0) { tLastSend = tNow; if (off == size) - return sendslot_ok(tStart, tNow, size, slot->cache_name); + return sendslot_ok(tStart, tNow, size, slot); } } while (1); #endif @@ -239,15 +242,14 @@ static int print_slot(struct cache_slot *slot) if (ETIMEDOUT == errno) { if (-2 == res) return sendslot_to_idle(tStart, tLastSend, time(NULL), - off, size, slot->cache_name); + off, size, slot); else if (-3 == res) - return sendslot_to_minrate(tStart, time(NULL), - off, size, slot->cache_name); + return sendslot_to_minrate(tStart, time(NULL), off, size, slot); } return errno; } if (off == size || !count /* should be redundant */) - return sendslot_ok2(tStart, size, slot->cache_name); + return sendslot_ok2(tStart, size, slot); } while (1); } @@ -260,20 +262,6 @@ static int is_expired(struct cache_slot *slot) return slot->cache_st.st_mtime + slot->ttl * 60 < time(NULL); } -/* Check if the slot has been modified since we opened it. - * NB: If stat() fails, we pretend the file is modified. - */ -static int is_modified(struct cache_slot *slot) -{ - struct stat st; - - if (stat(slot->cache_name, &st)) - return 1; - return (st.st_ino != slot->cache_st.st_ino || - st.st_mtime != slot->cache_st.st_mtime || - st.st_size != slot->cache_st.st_size); -} - /* Close an open lockfile */ static int close_lock(struct cache_slot *slot) { @@ -281,17 +269,18 @@ static int close_lock(struct cache_slot *slot) if (slot->lock_fd > 0) { if (close(slot->lock_fd)) err = errno; - else - slot->lock_fd = -1; + slot->lock_fd = -1; } return err; } +static int unlock_slot(struct cache_slot *slot, int replace_old_slot); + /* Create a lockfile used to store the generated content for a cache * slot, and write the slot key + \0 into it. * Returns 0 on success and errno otherwise. */ -static int lock_slot(struct cache_slot *slot) +static int lock_slot(struct cache_slot *slot, time_t tStart) { struct flock lock = { .l_type = F_WRLCK, @@ -299,21 +288,65 @@ static int lock_slot(struct cache_slot *slot) .l_start = 0, .l_len = 0, }; + size_t wait_count = 0; slot->lock_fd = open(slot->lock_name, O_RDWR | O_CREAT, S_IRUSR | S_IWUSR); if (slot->lock_fd == -1) return errno; - if (fcntl(slot->lock_fd, F_SETLK, &lock) < 0) { + while (fcntl(slot->lock_fd, F_SETLK, &lock) < 0) { int saved_errno = errno; - close(slot->lock_fd); - slot->lock_fd = -1; + time_t tNow = time(NULL); + if (EAGAIN != saved_errno || + tNow - tStart >= ctx.cfg.cache_lock_timeout) { + close(slot->lock_fd); + slot->lock_fd = -1; + cache_log("[cgit] Lock (%ds): Unable to lock slot %s (%s): %s (%d)\n", + (int)(tNow - tStart), slot->lock_name, + slot->key, strerror(saved_errno), saved_errno); + return saved_errno; + } + ++wait_count; + usleep(100000); // 100ms sleep instead of sched_yield() + } + if (wait_count && ctx.cfg.log_level > 90) { + cache_log("[cgit] Lock: Waited %ds (%zu tries, cache_fd %d) to lock slot %s (%s)\n", + (int)(time(NULL) - tStart), wait_count, slot->cache_fd, + slot->lock_name, slot->key); + } + if (slot->cache_fd <= 0) { + int err = open_slot(slot); + if (!err && slot->match) { + // concurrent process wrote the file + if (ctx.cfg.log_level > 90) { + cache_log("[cgit] Lock: Concurrent produced slot %s (%s)\n", + slot->lock_name, slot->key); + } + unlock_slot(slot, 0); + return 0; + } + close_slot(slot); + } + if (ftruncate(slot->lock_fd, 0) < 0) { + int saved_errno = errno; + cache_log("[cgit] Lock (%ds): Unable to truncate locked slot %s (%s): %s (%d)\n", + (int)(time(NULL) - tStart), slot->lock_name, + slot->key, strerror(saved_errno), saved_errno); + unlock_slot(slot, 0); return saved_errno; } - if (ftruncate(slot->lock_fd, 0) < 0) - return errno; - if (xwrite(slot->lock_fd, slot->key, slot->keylen + 1) < 0) - return errno; + if (xwrite(slot->lock_fd, slot->key, slot->keylen + 1) < 0) { + int saved_errno = errno; + cache_log("[cgit] Lock (%ds): Unable to write to locked slot %s (%s): %s (%d)\n", + (int)(time(NULL) - tStart), slot->lock_name, + slot->key, strerror(saved_errno), saved_errno); + unlock_slot(slot, 0); + return saved_errno; + } + if (ctx.cfg.log_level > 90) { + cache_log("[cgit] Lock (%ds): Successful locked slot %s (%s)\n", + (int)(time(NULL) - tStart), slot->lock_name, slot->key); + } return 0; } @@ -323,24 +356,58 @@ static int lock_slot(struct cache_slot *slot) */ static int unlock_slot(struct cache_slot *slot, int replace_old_slot) { - int err; - - if (replace_old_slot) - err = rename(slot->lock_name, slot->cache_name); - else - err = unlink(slot->lock_name); + struct flock lock = { + .l_type = F_UNLCK, + .l_whence = SEEK_SET, + .l_start = 0, + .l_len = 0, + }; + int err = 0; + if (replace_old_slot) { + if (rename(slot->lock_name, slot->cache_name)) { + err = errno; + } + } else { + if (unlink(slot->lock_name)) { + err = errno; + } + } + if (ctx.cfg.log_level < 90 && ENOENT == err) { + err = 0; // suppress ENOENT messages + } + if (err) { + cache_log("[cgit] Unlock: Failed to %s slot lock %s, cache %s, key %s: %s (%d)\n", + replace_old_slot ? "rename" : "unlink", + slot->lock_name, slot->cache_name, slot->key, strerror(err), err); + } + if (ENOENT == err) { // not an error + err = 0; + } /* Restore stdout and close the temporary FD. */ if (slot->stdout_fd >= 0) { dup2(slot->stdout_fd, STDOUT_FILENO); close(slot->stdout_fd); slot->stdout_fd = -1; } - - if (err) - return errno; - - return 0; + if (slot->lock_fd > 0) { + if (fcntl(slot->lock_fd, F_SETLK, &lock) < 0) { + int saved_errno = errno; + close(slot->lock_fd); + slot->lock_fd = -1; + cache_log("[cgit] Unlock: Unable to unlock slot %s (%s): %s (%d)\n", + slot->lock_name, slot->key, strerror(saved_errno), saved_errno); + if (!err) + err = saved_errno; + } + } + if (!err) { + if (ctx.cfg.log_level > 90) { + cache_log("[cgit] Unlock: Successful unlocked slot %s (%s)\n", + slot->lock_name, slot->key); + } + } + return err; } /* Generate the content for the current cache slot by redirecting @@ -396,42 +463,13 @@ unsigned long hash_str(const char *str) static int process_slot(struct cache_slot *slot) { int err; + time_t tStart = time(NULL); err = open_slot(slot); - if (!err && slot->match) { - if (is_expired(slot)) { - if (!lock_slot(slot)) { - /* If the cachefile has been replaced between - * `open_slot` and `lock_slot`, we'll just - * serve the stale content from the original - * cachefile. This way we avoid pruning the - * newly generated slot. The same code-path - * is chosen if fill_slot() fails for some - * reason. - * - * TODO? check if the new slot contains the - * same key as the old one, since we would - * prefer to serve the newest content. - * This will require us to open yet another - * file-descriptor and read and compare the - * key from the new file, so for now we're - * lazy and just ignore the new file. - */ - if (is_modified(slot) || fill_slot(slot)) { - unlock_slot(slot, 0); - close_lock(slot); - } else { - close_slot(slot); - unlock_slot(slot, 1); - slot->cache_fd = slot->lock_fd; - } - } - } - if ((err = print_slot(slot)) != 0) { - cache_log("[cgit] error printing cache %s: %s (%d)\n", - slot->cache_name, - strerror(err), - err); + if (!err && slot->match && !is_expired(slot)) { + if ((err = print_slot(slot)) != 0 && err != ETIMEDOUT) { + cache_log("[cgit] error printing cache %s (%s): %s (%d)\n", + slot->cache_name, slot->key, strerror(err), err); } close_slot(slot); return err; @@ -442,38 +480,45 @@ static int process_slot(struct cache_slot *slot) * request. If this fails (for whatever reason), lets just generate * the content without caching it and fool the caller to believe * everything worked out (but print a warning on stdout). + * + * If the cachefile has been created between + * above `open_slot` and within `lock_slot`, we'll just + * serve the new content from the new cachefile. */ close_slot(slot); - if ((err = lock_slot(slot)) != 0) { - cache_log("[cgit] Unable to lock slot %s: %s (%d)\n", - slot->lock_name, strerror(err), err); + if ((err = lock_slot(slot, tStart)) != 0) { + time_t tNow1 = time(NULL); slot->fn(); + cache_log("[cgit] Unable to lock new slot (%ds, %ds) %s (%s): %s (%d)\n", + (int)(tNow1 - tStart), (int)(time(NULL) - tStart), + slot->lock_name, slot->key, strerror(err), err); return 0; } - - if ((err = fill_slot(slot)) != 0) { - cache_log("[cgit] Unable to fill slot %s: %s (%d)\n", - slot->lock_name, strerror(err), err); - unlock_slot(slot, 0); - close_lock(slot); - slot->fn(); - return 0; - } - // We've got a valid cache slot in the lock file, which - // is about to replace the old cache slot. But if we - // release the lockfile and then try to open the new cache - // slot, we might get a race condition with a concurrent - // writer for the same cache slot (with a different key). - // Lets avoid such a race by just printing the content of - // the lock file. - slot->cache_fd = slot->lock_fd; - unlock_slot(slot, 1); - if ((err = print_slot(slot)) != 0) { - cache_log("[cgit] error printing cache %s: %s (%d)\n", - slot->cache_name, - strerror(err), - err); + if (slot->cache_fd <= 0) { + // first concurrent process lock + if ((err = fill_slot(slot)) != 0) { + cache_log("[cgit] Unable to fill slot %s (%s): %s (%d)\n", + slot->lock_name, slot->key, strerror(err), err); + unlock_slot(slot, 0); + close_lock(slot); + slot->fn(); + return 0; + } + // We've got a valid cache slot in the lock file, which + // is about to replace the old cache slot. But if we + // release the lockfile and then try to open the new cache + // slot, we might get a race condition with a concurrent + // writer for the same cache slot (with a different key). + // Lets avoid such a race by just printing the content of + // the lock file. + slot->cache_fd = slot->lock_fd; + unlock_slot(slot, 1); + } // else concurrent process produced slot (opened) + if ((err = print_slot(slot)) != 0 && err != ETIMEDOUT) { + cache_log("[cgit] error printing cache %s (%s): %s (%d)\n", + slot->cache_name, slot->key, + strerror(err), err); } close_slot(slot); return err; @@ -592,9 +637,22 @@ int cache_ls(const char *path) /* Print a message to stdout */ void cache_log(const char *format, ...) { + char buffer[400]; + char *end = buffer + sizeof(buffer); + char *out = buffer; + *(end - 1) = 0; + struct tm tNowLocal; + time_t tNow = time(NULL); + struct tm *tres = localtime_r(&tNow, &tNowLocal); + if (tres == &tNowLocal) { + // 'YYYY-mm-dd hh:mm:ss ' + out += strftime(out, 20 + 1, "%Y-%m-%d %H:%M:%S ", tres); + } + pid_t pid = getpid(); + out += snprintf(out, end - out, "%7d ", pid); // ' 38588' va_list args; va_start(args, format); - vfprintf(stderr, format, args); + vsnprintf(out, end - out, format, args); va_end(args); + fputs(buffer, stderr); } - diff --git a/cgit.c b/cgit.c index 26b4045..25e29f9 100644 --- a/cgit.c +++ b/cgit.c @@ -217,6 +217,8 @@ static void config_cb(const char *name, const char *value) ctx.cfg.cache_scanrc_ttl = atoi(value); else if (!strcmp(name, "cache-static-ttl")) ctx.cfg.cache_static_ttl = atoi(value); + else if (!strcmp(name, "cache-lock-timeout")) + ctx.cfg.cache_lock_timeout = atoi(value); else if (!strcmp(name, "client-io-idle-timeout")) ctx.cfg.client_io_idle_timeout = atoi(value); else if (!strcmp(name, "client-io-min-rate")) @@ -388,6 +390,7 @@ static void prepare_context(void) ctx.cfg.cache_scanrc_ttl = 15; ctx.cfg.cache_dynamic_ttl = 5; ctx.cfg.cache_static_ttl = -1; + ctx.cfg.cache_lock_timeout = 10; ctx.cfg.client_io_idle_timeout = 20; ctx.cfg.client_io_min_rate = 500; ctx.cfg.case_sensitive_sort = 1; @@ -887,6 +890,7 @@ static void print_config(FILE *f, const char *prefix) fprintf(f, "%slog-level=%d\n", prefix, ctx.cfg.log_level); fprintf(f, "%sproject-list=%s\n", prefix, ctx.cfg.project_list); fprintf(f, "%sscan-path=%s\n", prefix, ctx.cfg.scan_path); + fprintf(f, "%scache-lock-timeout=%d\n", prefix, ctx.cfg.cache_lock_timeout); fprintf(f, "%sclient-io-idle-timeout=%d\n", prefix, ctx.cfg.client_io_idle_timeout); fprintf(f, "%sclient-io-min-rate=%ld\n", prefix, ctx.cfg.client_io_min_rate); } diff --git a/cgit.h b/cgit.h index f16e501..4aed96c 100644 --- a/cgit.h +++ b/cgit.h @@ -229,6 +229,8 @@ struct cgit_config { int cache_static_ttl; int cache_about_ttl; int cache_snapshot_ttl; + /* cache lock timeout in seconds to acquire the cache lock-file against concurrent processes. Defaults to 10s. */ + int cache_lock_timeout; /* idle timeout in seconds between sending/receiving chunks of the cached body to/from the client. Defaults to 20s. */ int client_io_idle_timeout; /* minimum transfer rate in Bps for sending/receiving a full cached body to/from the client. Defaults to 500 Bps. */ diff --git a/cgitrc.5.txt b/cgitrc.5.txt index a6016ee..d3a8ae8 100644 --- a/cgitrc.5.txt +++ b/cgitrc.5.txt @@ -100,6 +100,10 @@ cache-static-ttl:: version of repository pages accessed with a fixed SHA1. See also: "CACHE". Default value: -1". +cache-lock-timeout:: + Timeout in seconds to acquire the cache lock-file + against concurrent processes. Default value: "10". + client-io-idle-timeout:: IDLE timeout in seconds between sending/receiving chunks of the cached body to/from the client. Default value: "20". From sgothel at jausoft.com Sun Jul 19 17:07:33 2026 From: sgothel at jausoft.com (=?UTF-8?Q?Sven_G=C3=B6thel?=) Date: Sun, 19 Jul 2026 19:07:33 +0200 Subject: [PATCH 4/4] Add cfg cache-lock-fail behavior if failing to acquire cache-file, lock Message-ID: <42bfebb9-8768-4366-b65f-d1bd09ae0c17@jausoft.com> >From 23472c578356f8ac5a2858ebaa7287bc24c3a5b0 Mon Sep 17 00:00:00 2001 From: Sven G?thel Date: Sun, 19 Jul 2026 16:47:26 +0200 Subject: Add cfg cache-lock-fail behavior if failing to acquire cache-file lock New default is http-response 503 and simply serve an error page to avoid server lockup. cache-lock-fail can be configured to 200 for old behavior, i.e. producing an off-cache content page. diff --git a/cache.c b/cache.c index ca22194..10e5db5 100644 --- a/cache.c +++ b/cache.c @@ -16,6 +16,7 @@ #include "cgit.h" #include "cache.h" #include "html.h" +#include "ui-shared.h" #include #include #include @@ -290,10 +291,20 @@ static int lock_slot(struct cache_slot *slot, time_t tStart) }; size_t wait_count = 0; - slot->lock_fd = open(slot->lock_name, O_RDWR | O_CREAT, - S_IRUSR | S_IWUSR); - if (slot->lock_fd == -1) - return errno; + if (0 == strncmp("cgit_test_key_no_lock", slot->key, 21)) { + cache_log("[cgit] Lock (%ds): Test-Key: %s -> forced fail\n", + (int)(time(NULL) - tStart), slot->key); + return ENOENT; + } + slot->lock_fd = + open(slot->lock_name, O_RDWR | O_CREAT, S_IRUSR | S_IWUSR); + if (slot->lock_fd == -1) { + int saved_errno = errno; + cache_log("[cgit] Lock (%ds): Unable to open/create lock slot %s (%s): %s (%d)\n", + (int)(time(NULL) - tStart), slot->lock_name, + slot->key, strerror(saved_errno), saved_errno); + return saved_errno; + } while (fcntl(slot->lock_fd, F_SETLK, &lock) < 0) { int saved_errno = errno; time_t tNow = time(NULL); @@ -489,20 +500,34 @@ static int process_slot(struct cache_slot *slot) close_slot(slot); if ((err = lock_slot(slot, tStart)) != 0) { time_t tNow1 = time(NULL); - slot->fn(); - cache_log("[cgit] Unable to lock new slot (%ds, %ds) %s (%s): %s (%d)\n", - (int)(tNow1 - tStart), (int)(time(NULL) - tStart), - slot->lock_name, slot->key, strerror(err), err); + if (ctx.cfg.cache_lock_fail != 200) { + ctx.page.title = "cgit error"; + cgit_print_error_page(503, "Service Unavailable", + "Service Unavailable. Cache: Could not lock new-slot within %ds.", + (int)(tNow1 - tStart)); + } else { + slot->fn(); + } + cache_log("[cgit] Unable to lock new slot (%ds, %ds) %s (%s): %d: %s (%d)\n", + (int)(tNow1 - tStart), (int)(time(NULL) - tStart), + slot->lock_name, slot->key, ctx.cfg.cache_lock_fail, strerror(err), err); return 0; } if (slot->cache_fd <= 0) { // first concurrent process lock if ((err = fill_slot(slot)) != 0) { - cache_log("[cgit] Unable to fill slot %s (%s): %s (%d)\n", - slot->lock_name, slot->key, strerror(err), err); + cache_log("[cgit] Unable to fill slot %s (%s): %d: %s (%d)\n", + slot->lock_name, slot->key, ctx.cfg.cache_lock_fail, strerror(err), err); unlock_slot(slot, 0); close_lock(slot); - slot->fn(); + if (ctx.cfg.cache_lock_fail != 200) { + ctx.page.title = "cgit error"; + cgit_print_error_page(503, "Service Unavailable", + "Service Unavailable. Cache: Could not fill slot within %ds.", + (int)(time(NULL) - tStart)); + } else { + slot->fn(); + } return 0; } // We've got a valid cache slot in the lock file, which diff --git a/cgit.c b/cgit.c index 25e29f9..83665ca 100644 --- a/cgit.c +++ b/cgit.c @@ -217,6 +217,8 @@ static void config_cb(const char *name, const char *value) ctx.cfg.cache_scanrc_ttl = atoi(value); else if (!strcmp(name, "cache-static-ttl")) ctx.cfg.cache_static_ttl = atoi(value); + else if (!strcmp(name, "cache-lock-fail")) + ctx.cfg.cache_lock_fail = atoi(value); else if (!strcmp(name, "cache-lock-timeout")) ctx.cfg.cache_lock_timeout = atoi(value); else if (!strcmp(name, "client-io-idle-timeout")) @@ -390,6 +392,7 @@ static void prepare_context(void) ctx.cfg.cache_scanrc_ttl = 15; ctx.cfg.cache_dynamic_ttl = 5; ctx.cfg.cache_static_ttl = -1; + ctx.cfg.cache_lock_fail = 503; ctx.cfg.cache_lock_timeout = 10; ctx.cfg.client_io_idle_timeout = 20; ctx.cfg.client_io_min_rate = 500; @@ -890,6 +893,7 @@ static void print_config(FILE *f, const char *prefix) fprintf(f, "%slog-level=%d\n", prefix, ctx.cfg.log_level); fprintf(f, "%sproject-list=%s\n", prefix, ctx.cfg.project_list); fprintf(f, "%sscan-path=%s\n", prefix, ctx.cfg.scan_path); + fprintf(f, "%scache-lock-fail=%d\n", prefix, ctx.cfg.cache_lock_fail); fprintf(f, "%scache-lock-timeout=%d\n", prefix, ctx.cfg.cache_lock_timeout); fprintf(f, "%sclient-io-idle-timeout=%d\n", prefix, ctx.cfg.client_io_idle_timeout); fprintf(f, "%sclient-io-min-rate=%ld\n", prefix, ctx.cfg.client_io_min_rate); diff --git a/cgit.h b/cgit.h index 4aed96c..df6a6c1 100644 --- a/cgit.h +++ b/cgit.h @@ -229,6 +229,8 @@ struct cgit_config { int cache_static_ttl; int cache_about_ttl; int cache_snapshot_ttl; + /* cache lock fail action as http-response code. 200 returns un-cached processed content (old behavior), otherwise an error page is served (new default). Defaults to 503. */ + int cache_lock_fail; /* cache lock timeout in seconds to acquire the cache lock-file against concurrent processes. Defaults to 10s. */ int cache_lock_timeout; /* idle timeout in seconds between sending/receiving chunks of the cached body to/from the client. Defaults to 20s. */ diff --git a/cgitrc.5.txt b/cgitrc.5.txt index d3a8ae8..de25244 100644 --- a/cgitrc.5.txt +++ b/cgitrc.5.txt @@ -100,6 +100,11 @@ cache-static-ttl:: version of repository pages accessed with a fixed SHA1. See also: "CACHE". Default value: -1". +cache-lock-fail:: + Cache lock fail action as http-response code. + 200 returns un-cached processed content (old behavior), + otherwise an error page is served (new default). Default value: "503". + cache-lock-timeout:: Timeout in seconds to acquire the cache lock-file against concurrent processes. Default value: "10". From sgothel at jausoft.com Mon Jul 20 00:41:01 2026 From: sgothel at jausoft.com (=?UTF-8?Q?Sven_G=C3=B6thel?=) Date: Mon, 20 Jul 2026 02:41:01 +0200 Subject: [PATCH 0/4] Patch update, mostly server DoS related In-Reply-To: <08852c71-94af-4c95-bf84-f9408f8d3fac@jausoft.com> References: <08852c71-94af-4c95-bf84-f9408f8d3fac@jausoft.com> Message-ID: <0f1eeb39-1799-4735-83ed-ad63fa2c19d8@jausoft.com> I have refined the patch-set via force-push, single patch updates will follow. Also added 2 more patches, see below. Feel free to pick and discuss. ~Sven On 7/19/26 5:34 PM, Sven G?thel wrote: > I will send 4 current patched from my repo > , namely > > - 1: cgit: add_repo: Assign default branch from git HEAD to repo->defbranch if undefined > ? > > > - 2: cgit: print_slot: Avoid Slow-Attack via `client-io-idle-timeout` and `client-io-min-rate` > ? > > > - 3: Resolve concurrent initial cache fill post lock. Release lock after fill. > ? > update > > - 4: Add cfg cache-lock-fail behavior if failing to acquire cache-file lock > ? > update - 5 Refine log-level usage, define values - 6 filters/syntax-highlighting.sh: Use highlight version 3 invocation > > last 2 are new. > I welcome a merge and/or discussion with the maintainer. > > Motivation of this work was to fight DoS attack > on two cgit server, reducing the load by using > - send timeout and min-bandwidth to client > - rely on cache-lock (actually a concurrency fix) > - fail on cache-lock failure -> error page (new default) > From sgothel at jausoft.com Mon Jul 20 00:43:08 2026 From: sgothel at jausoft.com (=?UTF-8?Q?Sven_G=C3=B6thel?=) Date: Mon, 20 Jul 2026 02:43:08 +0200 Subject: [PATCH 3/4] Resolve concurrent initial cache fill post lock. Release lock after, fill. In-Reply-To: <937d0add-0304-45ac-bb4e-1a6b5656320b@jausoft.com> References: <937d0add-0304-45ac-bb4e-1a6b5656320b@jausoft.com> Message-ID: <07299c06-834d-453d-a3b4-5c1cfd0f5a89@jausoft.com> force pushed Refined locking errors w/ close_lock. using enum type, removed redundant logging. +++ From c701a724807d980b7876c3a1c15ea747088ea4c7 Mon Sep 17 00:00:00 2001 From: Sven G?thel Date: Sun, 19 Jul 2026 06:17:56 +0200 Subject: Resolve concurrent initial cache fill post lock. Release lock after fill. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enhance verbose cache_log logging unlock_slot - add: unlock the cache-lock file's lock - this frees the lock right after successfull initial fill and allows concurrent usage lock_slot - lock: add retry on EAGAIN, waiting for concurrent process - uses `cache_lock_timeout` (default 10s) - post-lock: test if cache slot became valid from concurrent process - if successful, unlock_lock and proceed w/ cached_fd - otherwise close_slot (cached_fd) for new cache-slot - failure on truncate and write issues unlock_slot process_slot - remove complicated is_expired branch, simply utilize concurrent check in lock_slot. - If slot doesn't exist or is_expired, close_slot and lock_slot. If the latter produced a cached_fd, use it. Otherwise fill_slot and swap lock-file as new cache-file. Signed-off-by: Sven G?thel diff --git a/cache.c b/cache.c index 28e7180..3f93aa9 100644 --- a/cache.c +++ b/cache.c @@ -16,6 +16,10 @@ #include "cgit.h" #include "cache.h" #include "html.h" +#include +#include +#include +#include #ifdef HAVE_LINUX_SENDFILE #include #endif @@ -76,8 +80,7 @@ static int close_slot(struct cache_slot *slot) if (slot->cache_fd > 0) { if (close(slot->cache_fd)) err = errno; - else - slot->cache_fd = -1; + slot->cache_fd = -1; } return err; } @@ -86,50 +89,52 @@ static int close_slot(struct cache_slot *slot) #define MY_MAX(X, Y) (((X) > (Y)) ? (X) : (Y)) static int sendslot_to_idle(time_t tStart, time_t tLastSend, time_t tNow, - size_t off, size_t size, const char *cache_name) + size_t off, size_t size, struct cache_slot *slot) { const time_t td_total = tNow - tStart; const time_t td_idle = tNow - tLastSend; const long rate = off / MY_MAX(1, td_total); cache_log("[cgit] send_slot timeout idle %lds: sending cache " - "%s (%ld/%ld bytes) to client `%s` " + "%s (%s) (%ld/%ld bytes) to client `%s` " "within [total %lds, idle %lds, rate %ld Bps]\n", - td_idle, cache_name, off, size, ctx.env.remote_addr, + td_idle, slot->cache_name, slot->key, off, size, ctx.env.remote_addr, td_total, td_idle, rate); return ETIMEDOUT; } static int sendslot_to_minrate(time_t tStart, time_t tNow, size_t off, - size_t size, const char *cache_name) + size_t size, struct cache_slot *slot) { const time_t td_total = tNow - tStart; const long rate = off / MY_MAX(1, td_total); cache_log("[cgit] send_slot timeout rate-limit %ld Bps: sending " - "cache %s (%ld/%ld bytes) to client `%s` " + "cache %s (%s) (%ld/%ld bytes) to client `%s` " "within [total %lds, rate %ld Bps]\n", - ctx.cfg.client_io_min_rate, cache_name, off, size, ctx.env.remote_addr, + ctx.cfg.client_io_min_rate, slot->cache_name, slot->key, + off, size, ctx.env.remote_addr, td_total, rate); return ETIMEDOUT; } static int sendslot_ok(time_t tStart, time_t tNow, size_t size, - const char *cache_name) + struct cache_slot *slot) { if (ctx.cfg.log_level > 90) { const time_t td_total = tNow - tStart; const long rate = size / MY_MAX(1, td_total); - cache_log("[cgit] send_slot status: sent cache %s (%ld bytes) to " + cache_log("[cgit] send_slot status: sent cache %s (%s) %ld bytes) to " "client `%s` " "within [total %lds, rate %ld Bps]\n", - cache_name, size, ctx.env.remote_addr, td_total, rate); + slot->cache_name, slot->key, + size, ctx.env.remote_addr, td_total, rate); } return 0; } -static int sendslot_ok2(time_t tStart, size_t size, const char *cache_name) +static int sendslot_ok2(time_t tStart, size_t size, struct cache_slot *slot) { if (ctx.cfg.log_level > 90) { - return sendslot_ok(tStart, time(NULL), size, cache_name); + return sendslot_ok(tStart, time(NULL), size, slot); } return 0; } @@ -192,7 +197,7 @@ static int print_slot(struct cache_slot *slot) off_t size = slot->cache_st.st_size; if (!size) { - return sendslot_ok(tStart, tNow, size, slot->cache_name); + return sendslot_ok(tStart, tNow, size, slot); } const time_t to_min_rate = MY_MAX(ctx.cfg.client_io_idle_timeout, size / ctx.cfg.client_io_min_rate); @@ -200,11 +205,9 @@ static int print_slot(struct cache_slot *slot) #ifdef HAVE_LINUX_SENDFILE do { if (tNow - tLastSend >= ctx.cfg.client_io_idle_timeout) - return sendslot_to_idle(tStart, tLastSend, tNow, - off, size, slot->cache_name); + return sendslot_to_idle(tStart, tLastSend, tNow, off, size, slot); if (tNow - tStart > to_min_rate) - return sendslot_to_minrate(tStart, tNow, - off, size, slot->cache_name); + return sendslot_to_minrate(tStart, tNow, off, size, slot); ssize_t count = sendfile(STDOUT_FILENO, slot->cache_fd, &off, size - off); @@ -219,7 +222,7 @@ static int print_slot(struct cache_slot *slot) } else if (count > 0) { tLastSend = tNow; if (off == size) - return sendslot_ok(tStart, tNow, size, slot->cache_name); + return sendslot_ok(tStart, tNow, size, slot); } } while (1); #endif @@ -239,15 +242,14 @@ static int print_slot(struct cache_slot *slot) if (ETIMEDOUT == errno) { if (-2 == res) return sendslot_to_idle(tStart, tLastSend, time(NULL), - off, size, slot->cache_name); + off, size, slot); else if (-3 == res) - return sendslot_to_minrate(tStart, time(NULL), - off, size, slot->cache_name); + return sendslot_to_minrate(tStart, time(NULL), off, size, slot); } return errno; } if (off == size || !count /* should be redundant */) - return sendslot_ok2(tStart, size, slot->cache_name); + return sendslot_ok2(tStart, size, slot); } while (1); } @@ -260,20 +262,6 @@ static int is_expired(struct cache_slot *slot) return slot->cache_st.st_mtime + slot->ttl * 60 < time(NULL); } -/* Check if the slot has been modified since we opened it. - * NB: If stat() fails, we pretend the file is modified. - */ -static int is_modified(struct cache_slot *slot) -{ - struct stat st; - - if (stat(slot->cache_name, &st)) - return 1; - return (st.st_ino != slot->cache_st.st_ino || - st.st_mtime != slot->cache_st.st_mtime || - st.st_size != slot->cache_st.st_size); -} - /* Close an open lockfile */ static int close_lock(struct cache_slot *slot) { @@ -281,17 +269,28 @@ static int close_lock(struct cache_slot *slot) if (slot->lock_fd > 0) { if (close(slot->lock_fd)) err = errno; - else - slot->lock_fd = -1; + slot->lock_fd = -1; } return err; } +enum lock_file_op_t { UNLINK_LOCK_FILE=0, REPLACE_OLD_SLOT=1 }; + +static int unlock_slot(struct cache_slot *slot, enum lock_file_op_t lock_file_op); + +static const char *to_string(enum lock_file_op_t lock_file_op) { + switch (lock_file_op) { + case UNLINK_LOCK_FILE: return "unlink"; + case REPLACE_OLD_SLOT: return "replace"; + default: return "undef"; + } +} + /* Create a lockfile used to store the generated content for a cache * slot, and write the slot key + \0 into it. * Returns 0 on success and errno otherwise. */ -static int lock_slot(struct cache_slot *slot) +static int lock_slot(struct cache_slot *slot, time_t tStart) { struct flock lock = { .l_type = F_WRLCK, @@ -299,48 +298,129 @@ static int lock_slot(struct cache_slot *slot) .l_start = 0, .l_len = 0, }; + size_t wait_count = 0; slot->lock_fd = open(slot->lock_name, O_RDWR | O_CREAT, S_IRUSR | S_IWUSR); if (slot->lock_fd == -1) return errno; - if (fcntl(slot->lock_fd, F_SETLK, &lock) < 0) { + while (fcntl(slot->lock_fd, F_SETLK, &lock) < 0) { int saved_errno = errno; - close(slot->lock_fd); - slot->lock_fd = -1; + time_t tNow = time(NULL); + if (EAGAIN != saved_errno || + tNow - tStart >= ctx.cfg.cache_lock_timeout) { + close_lock(slot); + cache_log("[cgit] Lock (%ds): Unable to lock slot %s (%s): %s (%d)\n", + (int)(tNow - tStart), slot->lock_name, + slot->key, strerror(saved_errno), saved_errno); + return saved_errno; + } + ++wait_count; + usleep(100000); // 100ms sleep instead of sched_yield() + } + if (wait_count && ctx.cfg.log_level > 90) { + cache_log("[cgit] Lock: Waited %ds (%zu tries, cache_fd %d) to lock slot %s (%s)\n", + (int)(time(NULL) - tStart), wait_count, slot->cache_fd, + slot->lock_name, slot->key); + } + if (slot->cache_fd <= 0) { + int err = open_slot(slot); + if (!err && slot->match) { + // concurrent process wrote the file + if (ctx.cfg.log_level > 50) { + cache_log("[cgit] Lock: Concurrent produced slot %s (%s)\n", + slot->lock_name, slot->key); + } + unlock_slot(slot, UNLINK_LOCK_FILE); + close_lock(slot); + return 0; + } + close_slot(slot); + } + if (ftruncate(slot->lock_fd, 0) < 0) { + int saved_errno = errno; + cache_log("[cgit] Lock (%ds): Unable to truncate locked slot %s (%s): %s (%d)\n", + (int)(time(NULL) - tStart), slot->lock_name, + slot->key, strerror(saved_errno), saved_errno); + unlock_slot(slot, UNLINK_LOCK_FILE); + close_lock(slot); return saved_errno; } - if (ftruncate(slot->lock_fd, 0) < 0) - return errno; - if (xwrite(slot->lock_fd, slot->key, slot->keylen + 1) < 0) - return errno; + if (xwrite(slot->lock_fd, slot->key, slot->keylen + 1) < 0) { + int saved_errno = errno; + cache_log("[cgit] Lock (%ds): Unable to write to locked slot %s (%s): %s (%d)\n", + (int)(time(NULL) - tStart), slot->lock_name, + slot->key, strerror(saved_errno), saved_errno); + unlock_slot(slot, UNLINK_LOCK_FILE); + close_lock(slot); + return saved_errno; + } + if (ctx.cfg.log_level > 90) { + cache_log("[cgit] Lock (%ds): Successful locked slot %s (%s)\n", + (int)(time(NULL) - tStart), slot->lock_name, slot->key); + } return 0; } /* Release the current lockfile. If `replace_old_slot` is set the * lockfile replaces the old cache slot, otherwise the lockfile is * just deleted. + * @param lock_file_op UNLINK_LOCK_FILE unlink or UNLINK_LOCK_FILE replace old-slot w/ lock-file */ -static int unlock_slot(struct cache_slot *slot, int replace_old_slot) +static int unlock_slot(struct cache_slot *slot, enum lock_file_op_t lock_file_op) { - int err; - - if (replace_old_slot) - err = rename(slot->lock_name, slot->cache_name); - else - err = unlink(slot->lock_name); + struct flock lock = { + .l_type = F_UNLCK, + .l_whence = SEEK_SET, + .l_start = 0, + .l_len = 0, + }; + int err = 0; + if (REPLACE_OLD_SLOT == lock_file_op) { + if (rename(slot->lock_name, slot->cache_name)) { + err = errno; + } + } else if (UNLINK_LOCK_FILE == lock_file_op) { + if (unlink(slot->lock_name)) { + err = errno; + } + } + if (ctx.cfg.log_level < 90 && ENOENT == err) { + err = 0; // suppress ENOENT messages + } + if (err) { + cache_log("[cgit] Unlock: Failed to %s slot lock %s, cache %s, key %s: %s (%d)\n", + to_string(lock_file_op), + slot->lock_name, slot->cache_name, slot->key, strerror(err), err); + } + if (ENOENT == err) { // not an error + err = 0; + } /* Restore stdout and close the temporary FD. */ if (slot->stdout_fd >= 0) { dup2(slot->stdout_fd, STDOUT_FILENO); close(slot->stdout_fd); slot->stdout_fd = -1; } - - if (err) - return errno; - - return 0; + if (slot->lock_fd > 0) { + if (fcntl(slot->lock_fd, F_SETLK, &lock) < 0) { + int saved_errno = errno; + close(slot->lock_fd); + slot->lock_fd = -1; + cache_log("[cgit] Unlock: Unable to unlock slot %s (%s): %s (%d)\n", + slot->lock_name, slot->key, strerror(saved_errno), saved_errno); + if (!err) + err = saved_errno; + } + } + if (!err) { + if (ctx.cfg.log_level > 90) { + cache_log("[cgit] Unlock: Successful unlocked slot %s (%s)\n", + slot->lock_name, slot->key); + } + } + return err; } /* Generate the content for the current cache slot by redirecting @@ -396,42 +476,13 @@ unsigned long hash_str(const char *str) static int process_slot(struct cache_slot *slot) { int err; + time_t tStart = time(NULL); err = open_slot(slot); - if (!err && slot->match) { - if (is_expired(slot)) { - if (!lock_slot(slot)) { - /* If the cachefile has been replaced between - * `open_slot` and `lock_slot`, we'll just - * serve the stale content from the original - * cachefile. This way we avoid pruning the - * newly generated slot. The same code-path - * is chosen if fill_slot() fails for some - * reason. - * - * TODO? check if the new slot contains the - * same key as the old one, since we would - * prefer to serve the newest content. - * This will require us to open yet another - * file-descriptor and read and compare the - * key from the new file, so for now we're - * lazy and just ignore the new file. - */ - if (is_modified(slot) || fill_slot(slot)) { - unlock_slot(slot, 0); - close_lock(slot); - } else { - close_slot(slot); - unlock_slot(slot, 1); - slot->cache_fd = slot->lock_fd; - } - } - } - if ((err = print_slot(slot)) != 0) { - cache_log("[cgit] error printing cache %s: %s (%d)\n", - slot->cache_name, - strerror(err), - err); + if (!err && slot->match && !is_expired(slot)) { + if ((err = print_slot(slot)) != 0 && err != ETIMEDOUT) { + cache_log("[cgit] error printing cache %s (%s): %s (%d)\n", + slot->cache_name, slot->key, strerror(err), err); } close_slot(slot); return err; @@ -442,38 +493,42 @@ static int process_slot(struct cache_slot *slot) * request. If this fails (for whatever reason), lets just generate * the content without caching it and fool the caller to believe * everything worked out (but print a warning on stdout). + * + * If the cachefile has been created between + * above `open_slot` and within `lock_slot`, we'll just + * serve the new content from the new cachefile. */ close_slot(slot); - if ((err = lock_slot(slot)) != 0) { - cache_log("[cgit] Unable to lock slot %s: %s (%d)\n", - slot->lock_name, strerror(err), err); + if ((err = lock_slot(slot, tStart)) != 0) { + time_t tNow1 = time(NULL); slot->fn(); return 0; } - - if ((err = fill_slot(slot)) != 0) { - cache_log("[cgit] Unable to fill slot %s: %s (%d)\n", - slot->lock_name, strerror(err), err); - unlock_slot(slot, 0); - close_lock(slot); - slot->fn(); - return 0; - } - // We've got a valid cache slot in the lock file, which - // is about to replace the old cache slot. But if we - // release the lockfile and then try to open the new cache - // slot, we might get a race condition with a concurrent - // writer for the same cache slot (with a different key). - // Lets avoid such a race by just printing the content of - // the lock file. - slot->cache_fd = slot->lock_fd; - unlock_slot(slot, 1); - if ((err = print_slot(slot)) != 0) { - cache_log("[cgit] error printing cache %s: %s (%d)\n", - slot->cache_name, - strerror(err), - err); + if (slot->cache_fd <= 0) { + // first concurrent process lock + if ((err = fill_slot(slot)) != 0) { + cache_log("[cgit] Unable to fill slot %s (%s): %s (%d)\n", + slot->lock_name, slot->key, strerror(err), err); + unlock_slot(slot, UNLINK_LOCK_FILE); + close_lock(slot); + slot->fn(); + return 0; + } + // We've got a valid cache slot in the lock file, which + // is about to replace the old cache slot. But if we + // release the lockfile and then try to open the new cache + // slot, we might get a race condition with a concurrent + // writer for the same cache slot (with a different key). + // Lets avoid such a race by just printing the content of + // the lock file. + slot->cache_fd = slot->lock_fd; + unlock_slot(slot, REPLACE_OLD_SLOT); + } // else concurrent process produced slot (opened) + if ((err = print_slot(slot)) != 0 && err != ETIMEDOUT) { + cache_log("[cgit] error printing cache %s (%s): %s (%d)\n", + slot->cache_name, slot->key, + strerror(err), err); } close_slot(slot); return err; @@ -592,9 +647,22 @@ int cache_ls(const char *path) /* Print a message to stdout */ void cache_log(const char *format, ...) { + char buffer[400]; + char *end = buffer + sizeof(buffer); + char *out = buffer; + *(end - 1) = 0; + struct tm tNowLocal; + time_t tNow = time(NULL); + struct tm *tres = localtime_r(&tNow, &tNowLocal); + if (tres == &tNowLocal) { + // 'YYYY-mm-dd hh:mm:ss ' + out += strftime(out, 20 + 1, "%Y-%m-%d %H:%M:%S ", tres); + } + pid_t pid = getpid(); + out += snprintf(out, end - out, "%7d ", pid); // ' 38588' va_list args; va_start(args, format); - vfprintf(stderr, format, args); + vsnprintf(out, end - out, format, args); va_end(args); + fputs(buffer, stderr); } - diff --git a/cgit.c b/cgit.c index 26b4045..25e29f9 100644 --- a/cgit.c +++ b/cgit.c @@ -217,6 +217,8 @@ static void config_cb(const char *name, const char *value) ctx.cfg.cache_scanrc_ttl = atoi(value); else if (!strcmp(name, "cache-static-ttl")) ctx.cfg.cache_static_ttl = atoi(value); + else if (!strcmp(name, "cache-lock-timeout")) + ctx.cfg.cache_lock_timeout = atoi(value); else if (!strcmp(name, "client-io-idle-timeout")) ctx.cfg.client_io_idle_timeout = atoi(value); else if (!strcmp(name, "client-io-min-rate")) @@ -388,6 +390,7 @@ static void prepare_context(void) ctx.cfg.cache_scanrc_ttl = 15; ctx.cfg.cache_dynamic_ttl = 5; ctx.cfg.cache_static_ttl = -1; + ctx.cfg.cache_lock_timeout = 10; ctx.cfg.client_io_idle_timeout = 20; ctx.cfg.client_io_min_rate = 500; ctx.cfg.case_sensitive_sort = 1; @@ -887,6 +890,7 @@ static void print_config(FILE *f, const char *prefix) fprintf(f, "%slog-level=%d\n", prefix, ctx.cfg.log_level); fprintf(f, "%sproject-list=%s\n", prefix, ctx.cfg.project_list); fprintf(f, "%sscan-path=%s\n", prefix, ctx.cfg.scan_path); + fprintf(f, "%scache-lock-timeout=%d\n", prefix, ctx.cfg.cache_lock_timeout); fprintf(f, "%sclient-io-idle-timeout=%d\n", prefix, ctx.cfg.client_io_idle_timeout); fprintf(f, "%sclient-io-min-rate=%ld\n", prefix, ctx.cfg.client_io_min_rate); } diff --git a/cgit.h b/cgit.h index f16e501..4aed96c 100644 --- a/cgit.h +++ b/cgit.h @@ -229,6 +229,8 @@ struct cgit_config { int cache_static_ttl; int cache_about_ttl; int cache_snapshot_ttl; + /* cache lock timeout in seconds to acquire the cache lock-file against concurrent processes. Defaults to 10s. */ + int cache_lock_timeout; /* idle timeout in seconds between sending/receiving chunks of the cached body to/from the client. Defaults to 20s. */ int client_io_idle_timeout; /* minimum transfer rate in Bps for sending/receiving a full cached body to/from the client. Defaults to 500 Bps. */ diff --git a/cgitrc.5.txt b/cgitrc.5.txt index a6016ee..d3a8ae8 100644 --- a/cgitrc.5.txt +++ b/cgitrc.5.txt @@ -100,6 +100,10 @@ cache-static-ttl:: version of repository pages accessed with a fixed SHA1. See also: "CACHE". Default value: -1". +cache-lock-timeout:: + Timeout in seconds to acquire the cache lock-file + against concurrent processes. Default value: "10". + client-io-idle-timeout:: IDLE timeout in seconds between sending/receiving chunks of the cached body to/from the client. Default value: "20". From sgothel at jausoft.com Mon Jul 20 00:45:26 2026 From: sgothel at jausoft.com (=?UTF-8?Q?Sven_G=C3=B6thel?=) Date: Mon, 20 Jul 2026 02:45:26 +0200 Subject: [PATCH 4/4] Add cfg cache-lock-fail behavior if failing to acquire cache-file, lock In-Reply-To: <42bfebb9-8768-4366-b65f-d1bd09ae0c17@jausoft.com> References: <42bfebb9-8768-4366-b65f-d1bd09ae0c17@jausoft.com> Message-ID: <75733b6f-2f35-4ee1-93ec-a832cdf92cf2@jausoft.com> force pushed Removed redundant logging, add useful logging (un-cached fill). +++ >From 23bce8965897c21213da1a2882e0b04392f8428a Mon Sep 17 00:00:00 2001 From: Sven G?thel Date: Sun, 19 Jul 2026 16:47:26 +0200 Subject: Add cfg cache-lock-fail behavior if failing to acquire cache-file lock New default is http-response 503 and simply serve an error page to avoid server lockup. cache-lock-fail can be configured to 200 for old behavior, i.e. producing an off-cache content page. diff --git a/cache.c b/cache.c index 3f93aa9..7f11fa7 100644 --- a/cache.c +++ b/cache.c @@ -16,6 +16,7 @@ #include "cgit.h" #include "cache.h" #include "html.h" +#include "ui-shared.h" #include #include #include @@ -300,10 +301,20 @@ static int lock_slot(struct cache_slot *slot, time_t tStart) }; size_t wait_count = 0; - slot->lock_fd = open(slot->lock_name, O_RDWR | O_CREAT, - S_IRUSR | S_IWUSR); - if (slot->lock_fd == -1) - return errno; + if (0 == strncmp("cgit_test_key_no_lock", slot->key, 21)) { + cache_log("[cgit] Lock (%ds): Test-Key: %s -> forced fail\n", + (int)(time(NULL) - tStart), slot->key); + return ENOENT; + } + slot->lock_fd = + open(slot->lock_name, O_RDWR | O_CREAT, S_IRUSR | S_IWUSR); + if (slot->lock_fd == -1) { + int saved_errno = errno; + cache_log("[cgit] Lock (%ds): Unable to open/create lock slot %s (%s): %s (%d)\n", + (int)(time(NULL) - tStart), slot->lock_name, + slot->key, strerror(saved_errno), saved_errno); + return saved_errno; + } while (fcntl(slot->lock_fd, F_SETLK, &lock) < 0) { int saved_errno = errno; time_t tNow = time(NULL); @@ -502,17 +513,37 @@ static int process_slot(struct cache_slot *slot) close_slot(slot); if ((err = lock_slot(slot, tStart)) != 0) { time_t tNow1 = time(NULL); - slot->fn(); + if (ctx.cfg.cache_lock_fail != 200) { + ctx.page.title = "cgit error"; + cgit_print_error_page(503, "Service Unavailable", + "Service Unavailable. Cache: Could not lock new-slot within %ds.", + (int)(tNow1 - tStart)); + } else { + slot->fn(); + cache_log("[cgit] Uncached fill took %ds %s (failed lock %s)\n", + (int)(time(NULL) - tNow1), slot->key, slot->lock_name); + } return 0; } if (slot->cache_fd <= 0) { // first concurrent process lock if ((err = fill_slot(slot)) != 0) { - cache_log("[cgit] Unable to fill slot %s (%s): %s (%d)\n", - slot->lock_name, slot->key, strerror(err), err); + time_t tNow1 = time(NULL); + cache_log("[cgit] Unable to fill slot (%ds) %s (%s): %d: %s (%d)\n", + (int)(tNow1 - tStart), slot->lock_name, slot->key, + ctx.cfg.cache_lock_fail, strerror(err), err); unlock_slot(slot, UNLINK_LOCK_FILE); close_lock(slot); - slot->fn(); + if (ctx.cfg.cache_lock_fail != 200) { + ctx.page.title = "cgit error"; + cgit_print_error_page(503, "Service Unavailable", + "Service Unavailable. Cache: Could not fill slot within %ds.", + (int)(time(NULL) - tStart)); + } else { + slot->fn(); + cache_log("[cgit] Uncached fill took %ds %s (failed locked fill %s)\n", + (int)(time(NULL) - tNow1), slot->key, slot->lock_name); + } return 0; } // We've got a valid cache slot in the lock file, which diff --git a/cgit.c b/cgit.c index 25e29f9..83665ca 100644 --- a/cgit.c +++ b/cgit.c @@ -217,6 +217,8 @@ static void config_cb(const char *name, const char *value) ctx.cfg.cache_scanrc_ttl = atoi(value); else if (!strcmp(name, "cache-static-ttl")) ctx.cfg.cache_static_ttl = atoi(value); + else if (!strcmp(name, "cache-lock-fail")) + ctx.cfg.cache_lock_fail = atoi(value); else if (!strcmp(name, "cache-lock-timeout")) ctx.cfg.cache_lock_timeout = atoi(value); else if (!strcmp(name, "client-io-idle-timeout")) @@ -390,6 +392,7 @@ static void prepare_context(void) ctx.cfg.cache_scanrc_ttl = 15; ctx.cfg.cache_dynamic_ttl = 5; ctx.cfg.cache_static_ttl = -1; + ctx.cfg.cache_lock_fail = 503; ctx.cfg.cache_lock_timeout = 10; ctx.cfg.client_io_idle_timeout = 20; ctx.cfg.client_io_min_rate = 500; @@ -890,6 +893,7 @@ static void print_config(FILE *f, const char *prefix) fprintf(f, "%slog-level=%d\n", prefix, ctx.cfg.log_level); fprintf(f, "%sproject-list=%s\n", prefix, ctx.cfg.project_list); fprintf(f, "%sscan-path=%s\n", prefix, ctx.cfg.scan_path); + fprintf(f, "%scache-lock-fail=%d\n", prefix, ctx.cfg.cache_lock_fail); fprintf(f, "%scache-lock-timeout=%d\n", prefix, ctx.cfg.cache_lock_timeout); fprintf(f, "%sclient-io-idle-timeout=%d\n", prefix, ctx.cfg.client_io_idle_timeout); fprintf(f, "%sclient-io-min-rate=%ld\n", prefix, ctx.cfg.client_io_min_rate); diff --git a/cgit.h b/cgit.h index 4aed96c..df6a6c1 100644 --- a/cgit.h +++ b/cgit.h @@ -229,6 +229,8 @@ struct cgit_config { int cache_static_ttl; int cache_about_ttl; int cache_snapshot_ttl; + /* cache lock fail action as http-response code. 200 returns un-cached processed content (old behavior), otherwise an error page is served (new default). Defaults to 503. */ + int cache_lock_fail; /* cache lock timeout in seconds to acquire the cache lock-file against concurrent processes. Defaults to 10s. */ int cache_lock_timeout; /* idle timeout in seconds between sending/receiving chunks of the cached body to/from the client. Defaults to 20s. */ diff --git a/cgitrc.5.txt b/cgitrc.5.txt index d3a8ae8..de25244 100644 --- a/cgitrc.5.txt +++ b/cgitrc.5.txt @@ -100,6 +100,11 @@ cache-static-ttl:: version of repository pages accessed with a fixed SHA1. See also: "CACHE". Default value: -1". +cache-lock-fail:: + Cache lock fail action as http-response code. + 200 returns un-cached processed content (old behavior), + otherwise an error page is served (new default). Default value: "503". + cache-lock-timeout:: Timeout in seconds to acquire the cache lock-file against concurrent processes. Default value: "10". From sgothel at jausoft.com Mon Jul 20 00:47:01 2026 From: sgothel at jausoft.com (=?UTF-8?Q?Sven_G=C3=B6thel?=) Date: Mon, 20 Jul 2026 02:47:01 +0200 Subject: [PATCH 5/6] Refine log-level usage, define values Message-ID: <2204c629-8feb-4ca6-ad9a-d12b9b455b59@jausoft.com> >From 6ead0ef2bc361fa7e0a292709922133aeb3776b3 Mon Sep 17 00:00:00 2001 From: Sven G?thel Date: Mon, 20 Jul 2026 02:19:24 +0200 Subject: Refine log-level usage, define values diff --git a/cache.c b/cache.c index 7f11fa7..8165f48 100644 --- a/cache.c +++ b/cache.c @@ -120,7 +120,7 @@ static int sendslot_to_minrate(time_t tStart, time_t tNow, size_t off, static int sendslot_ok(time_t tStart, time_t tNow, size_t size, struct cache_slot *slot) { - if (ctx.cfg.log_level > 90) { + if (ctx.cfg.log_level >= LOG_LVL_DBG) { const time_t td_total = tNow - tStart; const long rate = size / MY_MAX(1, td_total); cache_log("[cgit] send_slot status: sent cache %s (%s) %ld bytes) to " @@ -134,7 +134,7 @@ static int sendslot_ok(time_t tStart, time_t tNow, size_t size, static int sendslot_ok2(time_t tStart, size_t size, struct cache_slot *slot) { - if (ctx.cfg.log_level > 90) { + if (ctx.cfg.log_level >= LOG_LVL_DBG) { return sendslot_ok(tStart, time(NULL), size, slot); } return 0; @@ -329,7 +329,7 @@ static int lock_slot(struct cache_slot *slot, time_t tStart) ++wait_count; usleep(100000); // 100ms sleep instead of sched_yield() } - if (wait_count && ctx.cfg.log_level > 90) { + if (wait_count && ctx.cfg.log_level >= LOG_LVL_DBG) { cache_log("[cgit] Lock: Waited %ds (%zu tries, cache_fd %d) to lock slot %s (%s)\n", (int)(time(NULL) - tStart), wait_count, slot->cache_fd, slot->lock_name, slot->key); @@ -338,7 +338,7 @@ static int lock_slot(struct cache_slot *slot, time_t tStart) int err = open_slot(slot); if (!err && slot->match) { // concurrent process wrote the file - if (ctx.cfg.log_level > 50) { + if (ctx.cfg.log_level >= LOG_LVL_WARN) { cache_log("[cgit] Lock: Concurrent produced slot %s (%s)\n", slot->lock_name, slot->key); } @@ -366,7 +366,7 @@ static int lock_slot(struct cache_slot *slot, time_t tStart) close_lock(slot); return saved_errno; } - if (ctx.cfg.log_level > 90) { + if (ctx.cfg.log_level >= LOG_LVL_DBG) { cache_log("[cgit] Lock (%ds): Successful locked slot %s (%s)\n", (int)(time(NULL) - tStart), slot->lock_name, slot->key); } @@ -397,7 +397,7 @@ static int unlock_slot(struct cache_slot *slot, enum lock_file_op_t lock_file_op err = errno; } } - if (ctx.cfg.log_level < 90 && ENOENT == err) { + if (ctx.cfg.log_level < LOG_LVL_DBG && ENOENT == err) { err = 0; // suppress ENOENT messages } if (err) { @@ -426,7 +426,7 @@ static int unlock_slot(struct cache_slot *slot, enum lock_file_op_t lock_file_op } } if (!err) { - if (ctx.cfg.log_level > 90) { + if (ctx.cfg.log_level >= LOG_LVL_DBG) { cache_log("[cgit] Unlock: Successful unlocked slot %s (%s)\n", slot->lock_name, slot->key); } diff --git a/cgit.c b/cgit.c index 83665ca..dc5e41c 100644 --- a/cgit.c +++ b/cgit.c @@ -1096,7 +1096,7 @@ int cmd_main(int argc, const char **argv) cgit_parse_args(argc, argv); parse_configfile(expand_macros(ctx.env.cgit_config), config_cb); - if (ctx.cfg.log_level) + if (ctx.cfg.log_level >= LOG_LVL_VERBOSE) print_config(stderr, "[cgit] init: "); ctx.repo = NULL; http_parse_querystring(ctx.qry.raw, querystring_cb); diff --git a/cgit.h b/cgit.h index df6a6c1..2691fb3 100644 --- a/cgit.h +++ b/cgit.h @@ -193,6 +193,10 @@ struct cgit_query { char *vpath; }; +typedef enum { + LOG_LVL_ERR=0, LOG_LVL_WARN=50, LOG_LVL_DBG=75, LOG_LVL_VERBOSE=100 +} log_level_t; + struct cgit_config { int log_level; ///< defaults to zero char *agefile; diff --git a/cgitrc.5.txt b/cgitrc.5.txt index de25244..0c40561 100644 --- a/cgitrc.5.txt +++ b/cgitrc.5.txt @@ -474,8 +474,11 @@ virtual-root:: same kind of virtual urls, so this option will probably be deprecated. log-level:: - Specifies the logging level. Above zero adds verbose logging. - Default value: "0". + Specifies the logging level. + Zero only logs errors (LOG_LVL_ERR), greater-equal 50 adds warnings (LOG_LVL_WARN), + greater-equal 75 adds debug messages + and greater-equal 100 adds verbose information (LOG_LVL_VERBOSE). + Default value: "0" (error logging only). REPOSITORY SETTINGS ------------------- From sgothel at jausoft.com Mon Jul 20 00:47:51 2026 From: sgothel at jausoft.com (=?UTF-8?Q?Sven_G=C3=B6thel?=) Date: Mon, 20 Jul 2026 02:47:51 +0200 Subject: [PATCH 6/6] filters/syntax-highlighting.sh: Use highlight version 3 invocation Message-ID: <0771f148-2466-408b-9387-11baac7fd9bc@jausoft.com> >From 9383c3d46a1ff40439f46f761323c3b63b62c93b Mon Sep 17 00:00:00 2001 From: Sven G?thel Date: Sun, 19 Jul 2026 20:36:52 +0200 Subject: filters/syntax-highlighting.sh: Use highlight version 3 invocation diff --git a/filters/syntax-highlighting.sh b/filters/syntax-highlighting.sh index 840bc34..5033f58 100755 --- a/filters/syntax-highlighting.sh +++ b/filters/syntax-highlighting.sh @@ -115,7 +115,7 @@ EXTENSION="${BASENAME##*.}" # found (for example) on EPEL 6. # # This is for version 2 -exec highlight --force -f -I -X -S "$EXTENSION" 2>/dev/null +#exec highlight --force -f -I -X -S "$EXTENSION" 2>/dev/null # This is for version 3 -#exec highlight --force -f -I -O xhtml -S "$EXTENSION" 2>/dev/null +exec highlight --force -f -I -O xhtml -S "$EXTENSION" 2>/dev/null From aiden at aidenw.net Mon Jul 20 04:22:41 2026 From: aiden at aidenw.net (Aiden Woodruff) Date: Mon, 20 Jul 2026 00:22:41 -0400 Subject: [PATCH 6/6] filters/syntax-highlighting.sh: Use highlight version 3 invocation In-Reply-To: <0771f148-2466-408b-9387-11baac7fd9bc@jausoft.com> References: <0771f148-2466-408b-9387-11baac7fd9bc@jausoft.com> Message-ID: <42aa9714bf8c80bda42ceab08e1462a9@aidenw.net> On 2026-07-19 20:47, Sven G?thel wrote: > # This is for version 2 > -exec highlight --force -f -I -X -S "$EXTENSION" 2>/dev/null > +#exec highlight --force -f -I -X -S "$EXTENSION" 2>/dev/null > > # This is for version 3 > -#exec highlight --force -f -I -O xhtml -S "$EXTENSION" 2>/dev/null > +exec highlight --force -f -I -O xhtml -S "$EXTENSION" 2>/dev/null Sure highlight >= 3 is readily available in most distributions, why break compatibility now? We could instead extract and check the version before highlighting: ``` highlight_version=$(highlight --version | \ sed 's/^ highlight version //;t m;d;b;:m;s/\.[0-9]*$//') if [ $highlight_version -lt 3 ]; then exec highlight --force -f -I -X -S "$EXTENSION" 2>/dev/null else exec highlight --force -f -I -O xhtml -S "$EXTENSION" 2>/dev/null fi ``` -- Aiden Woodruff From sgothel at jausoft.com Tue Jul 21 11:33:32 2026 From: sgothel at jausoft.com (=?UTF-8?Q?Sven_G=C3=B6thel?=) Date: Tue, 21 Jul 2026 13:33:32 +0200 Subject: [PATCH 6/6] filters/syntax-highlighting.sh: Use highlight version 3 invocation In-Reply-To: <42aa9714bf8c80bda42ceab08e1462a9@aidenw.net> References: <0771f148-2466-408b-9387-11baac7fd9bc@jausoft.com> <42aa9714bf8c80bda42ceab08e1462a9@aidenw.net> Message-ID: On 7/20/26 6:22 AM, Aiden Woodruff wrote: > Sure highlight >= 3 is readily available in most distributions, why break > compatibility now? Just a proposal, you are probably right here. > We could instead extract and check the version before > highlighting: > > ``` > highlight_version=$(highlight --version | \ > ? sed 's/^ highlight version //;t m;d;b;:m;s/\.[0-9]*$//') > if [ $highlight_version -lt 3 ]; then instead of these extra calls and work, we could also add a filters/syntax-highlighting3.sh file to be picked up by the cgitrc or not. ~Sven From sgothel at jausoft.com Wed Jul 22 09:21:19 2026 From: sgothel at jausoft.com (=?UTF-8?Q?Sven_G=C3=B6thel?=) Date: Wed, 22 Jul 2026 11:21:19 +0200 Subject: [PATCH 4/4] Add cfg cache-lock-fail behavior if failing to acquire cache-file, lock In-Reply-To: References: <42bfebb9-8768-4366-b65f-d1bd09ae0c17@jausoft.com>

Message-ID: <0ee2d3b8-9708-435b-ab56-cec6adcdb331@jausoft.com> On 7/22/26 5:57 AM, Sven G?thel wrote: > I have force-push the commits (also reduced in number) > as described below. > See if interested > Did a self-review and found one more issue, i.e. added is_expired check while in lock and having detected a valid file otherwise. If is_expired, we don't use the slot but delete it and continue filling the new slot. Additionally I replaced 2 log_level int-literal uses with the enum LOG_LVL_* symbolic value. (oversight) Hope that's it. Force pushed to above link.