Very slow startup of web / http manager #1076

ieugen · 2016-05-16T16:38:58Z

Hello,

I have configured gitblit on my machine and all was well until now. It seems whenever I start /restart the service it takes very long time to start SSH Manger and the web gui.

As you can see, it takes Started @136214ms to start =136 seconds

May 16 19:32:39 netdava.com java[2863]: 2016-05-16 19:32:39 [INFO ] ----[com.gitblit.manager.IServicesManager]----
May 16 19:32:39 netdava.com java[2863]: 2016-05-16 19:32:39 [INFO ] Federation passphrase is blank! This server can not be PULLED from.
May 16 19:32:39 netdava.com java[2863]: 2016-05-16 19:32:39 [INFO ] Fanout PubSub service is disabled.
May 16 19:32:39 netdava.com java[2863]: 2016-05-16 19:32:39 [INFO ] Git Daemon is listening on 0.0.0.0:9418
^[^[^[





May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] SSH Daemon (NIO2) is listening on 0.0.0.0:29418
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ]
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] ----[com.gitblit.manager.IFilestoreManager]----
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] No filestore metadata file found
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ]
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] ----[com.gitblit.manager.IPluginManager]----
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] PF4J version 1.7.1 in 'deployment' mode
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] Enabled plugins: []
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] Disabled plugins: []
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] No plugins
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ]
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] All managers started.
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ]
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] [GitBlitWebApp] init: Wicket core library initializer
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] [GitBlitWebApp] init: Wicket extensions initializer
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] [GitBlitWebApp] Started Wicket version 1.4.22 in deployment mode
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] Started o.e.j.w.WebAppContext@47da3952{/,file:/opt/gitblit/data/temp/webapp/,AVAILABLE}{file:/opt/gitblit/gitblit-1.7.1/gitblit.jar}
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] Started ServerConnector@4a254d75{HTTP/1.1}{localhost:8197}
May 16 19:34:51 netdava.com java[2863]: 2016-05-16 19:34:51 [INFO ] Started @136214ms

May 16 19:35:25 netdava.com java[2863]: 2016-05-16 19:35:25 [INFO ] Loading properties files from jar:file:/opt/gitblit/gitblit-1.7.1/gitblit.jar!/com/gitblit/wicket/GitBlitWebApp_en.properties
May 16 19:35:25 netdava.com java[2863]: 2016-05-16 19:35:25 [INFO ] Loading properties files from jar:file:/opt/gitblit/gitblit-1.7.1/gitblit.jar!/com/gitblit/wicket/GitBlitWebApp.properties
May 16 19:35:25 netdava.com java[2863]: 2016-05-16 19:35:25 [INFO ] Loading properties files from jar:file:/opt/gitblit/gitblit-1.7.1/ext/wicket-1.4.22.jar!/org/apache/wicket/Application.properties
May 16 19:35:25 netdava.com java[2863]: 2016-05-16 19:35:25 [INFO ] 12 repository models loaded for ieugen in 65 msecs

The text was updated successfully, but these errors were encountered:

gitblit · 2016-05-16T16:51:22Z

I see you are loading 12 repos. How large are they?
Are you serving your repos from a network share?
What hardware are you running Gitblit on?
How much memory does your box have?
What is your Java heap setting?

ieugen · 2016-05-16T17:25:34Z

Hello,

Thank you for the quick response. The repositories are not large - but why should that count? Are you pre-loading the repos? This should be done lazy, no?

I'm running on Debian - Digital Ocean droplet - SSD with 1gb of RAM. Restarted gitblit to watch RAM usage - stayed at 392 MB. Files are local.

The strange thing is that now it works ok. I have no idea what changed. I did reboot the machine to fix it but it was still loading slow.

Now, I logged to get information for a response and decided to restart the service just because. Now it loads fast.

It might do some network checks when it starts and the network was running slow, and now it's ok?

$ cat /etc/default/gitblit 
MODE=service
#-Dlog4j.configuration=file:/opt/gitblit/data/log4j.properties  
JAVA_OPTS="-Xmx128M -Dcom.unboundid.ldap.sdk.debug.enabled=true -Dcom.unboundid.ldap.sdk.debug.level=DEBUG"

$ cat /etc/systemd/system/gitblit.service 
[Unit]
Description= Gitblit server
After=network.target remote-fs.target

[Service]
User=gitblit
WorkingDirectory=/opt/gitblit/current
EnvironmentFile=/etc/default/gitblit

ExecStart=/usr/bin/java $JAVA_OPTS -jar /opt/gitblit/current/gitblit.jar --baseFolder /opt/gitblit/data --shutdownPort=-1
ExecStop=/usr/bin/java $JAVA_OPTS -jar /opt/gitblit/current/gitblit.jar --baseFolder /opt/gitblit/data --stop 

[Install]
WantedBy=multi-user.target

p.s. You can use the files for systemd config examples.
p.s. The reason I'm restarting the service is that I'm working on LDAP auth and things don't seem to work ok so I'm updating configs and restart - to login.

gitblit · 2016-05-16T17:34:58Z

The repositories are not large - but why should that count? Are you pre-loading the repos?

Yes. Pre-loading & caching. This is to reduce request processing time.

ieugen · 2016-05-16T18:18:30Z

Could you do that in a worker thread or lazy during startup?

gitblit · 2016-05-16T18:30:50Z

Possibly, but we have not identified this as the source of your slowdown.

ieugen · 2016-05-17T06:53:46Z

True :). Thanks again for the support.

shokkas · 2016-05-23T15:15:06Z

Hi,
Thanks to this thread I now understand my server's slowness.
Preloading & caching >300 repo, some quite large can be VERY lengthy.

Any way to shorten the loading time?

gitblit · 2016-05-23T16:41:13Z

You could disable caching altogether (not recommended).
You could shorten the cache window (web.activityCacheDays read notes!)
You could submit a PR that preloads/caches in a worker thread

* Make the CommitCache fully thread-safe. It was using a ConcurrentHashMap containing lists, but then handed out these lists. It also did multiple operations on that map that as a whole should be atomic. * Use isEmpty() instead of size() == 0. * Run the loading of the commit cache in a background daemon thread

Issue #1076: load commit cache in a background thread

bviktor · 2017-09-14T15:02:30Z

I'm pretty sure something's messed up with caching. On a fresh install (with the existing several hundred repos) it's like half a minute. Then eventually it slows down. Now I clocked it, 18 minutes.

Then I changed web.activityCacheDays to 3 (from the default 14), and it went down to 16 seconds. From 1113 seconds. Speaking of nonlinear...

ieugen changed the title ~~Slow startup of web / http manager~~ Very slow startup of web / http manager May 16, 2016

tomaswolf mentioned this issue Aug 22, 2016

Gitblit takes too long to initialize tomaswolf/gerrit-gitblit-plugin#23

Closed

gitblit added a commit that referenced this issue Oct 27, 2016

Merge pull request #1140 from tomaswolf/issue_1076

bac1241

Issue #1076: load commit cache in a background thread

flaix added Priority-Medium Status-New 🐱 Performance labels Dec 17, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Very slow startup of web / http manager #1076

Very slow startup of web / http manager #1076

ieugen commented May 16, 2016

gitblit commented May 16, 2016

ieugen commented May 16, 2016 •

edited

Loading

gitblit commented May 16, 2016

ieugen commented May 16, 2016

gitblit commented May 16, 2016 via email

ieugen commented May 17, 2016

shokkas commented May 23, 2016

gitblit commented May 23, 2016

bviktor commented Sep 14, 2017

Very slow startup of web / http manager #1076

Very slow startup of web / http manager #1076

Comments

ieugen commented May 16, 2016

gitblit commented May 16, 2016

ieugen commented May 16, 2016 • edited Loading

gitblit commented May 16, 2016

ieugen commented May 16, 2016

gitblit commented May 16, 2016 via email

ieugen commented May 17, 2016

shokkas commented May 23, 2016

gitblit commented May 23, 2016

bviktor commented Sep 14, 2017

ieugen commented May 16, 2016 •

edited

Loading