Performance issue

[Miroslav Suchý]

Hi,
I use cgit as WebUI for dist-git of Copr [1].  There are 136000 git repositories (and growing).
My problem is that no matter how aggressive caching in /etc/cgitrc is used, it takes enormous time to generate initial
/var/cache/cgit/rc-* file where are those "repo.*" configurations. And by enormous I mean 30 minutes.

I came up  with one solution. Set TTL to 2 hours and regenerate the cgitrc from cron every hour. This way the cgitrc
will never be generated by user coming from httpd request.

I can generate that cgitrc in cron job manually by running:

QUERY_STRING="url=frostyx/new7/rubygem-active_null.git/commit/&id=b3ceddf17119bc4c9b249fe1b63659039e282c99"
CGIT_CONFIG="/etc/cgitrc" /var/www/cgi-bin/cgit >/tmp/x.html

The problem is that even with --nocache it does not refresh existing /var/cache/cgit/rc-* file. The only way to refresh
the cgitrc file is to wait till it become older than TTL or delete it. But until it is regenerated the users who access
my server, will take it down by filling all apache slots with running cgit (which will traverse all git repositories).

I am thinking about implementing new option. E.g. --update-scan-path, which will force cgit to scan 'scan-path', create
the include cgitrc file in tempfile and at the and it will remove original /var/cache/cgit/rc-* and rename the newly
created cgitrc to that rc-* file. So it will be nearly atomic operation.

If you agree, I can prepare patch next week.


[1] https://copr.fedorainfracloud.org/

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys