Vault Slowdown Since v4.1.4 upgrade

[wkrzystek]

We've just recently upgraded from 3.5.2 to version 4.1.4. Its kind of hard to quantify but it appears that Vault is becoming much slower as time goes by. Is there anything in particular that we can look at to try to see what's going on? Just a for instance, from Visual Studio 2008, its taking about 30 seconds to display the repository list after completing the login screen. Basic checkin/checkout commands via the command line client are much slower than they ever were. Any ideas you can provide on things to try or information I could supply to help diagnose would be appreciated. Thanks.

[Beth]

Where I'd like you to start is with the following KB articles that help improve performance.

Recommendations for optimal Vault performance
Maintenance: The Vault Server database

It could be a case of where some information needs to be resolved in the cache. Try renaming your cache to see if it performs faster.
Vault Client-Side Cache Files

[wkrzystek]

Linda,

Sorry for the length of time since your reply, we've been busy on other projects and just dealing with the slowdown. We've gone through most of the posts you referenced and we haven't been able to find anything that makes much of an impact.

We have the vault server running on a 3Ghz Pentium D, with 4GB Ram, Win2K3, IIS 6. The SQL database is installed on a separate server: Dual 2.8 Ghz Xeon, 2GB Ram, Win2K3, SQL2005, SCSI Raid5, 800GB free. The vault server is running other IIS webapps but we're barely scratching its capability. The SQL server is our main SQL server and does have alot of SQL traffic its serving and will be upgraded to 4GB ram soon but the vault traffic to it is pretty minimal. I'd consider installing SQL on the main vault server just to see if isolating the SQL traffic for vault makes any difference if you thought it might be worth it.

What I think may be more of the problem is the number of folders/files that we have in vault. In our main repository, we have 1066 folders (2 for each of our clients) and a little over 12000 files total. We're using the command line client extensively and I built some instrumentation into the process so I can start watching average times for commands and have some sort of a baseline to tell if we can make anything better or not. One of the most frequent commands we're calling is the LISTFOLDERNR command. We are seeing this command take between 8 and 35 seconds to execute (same folder, same file content, just different calls during the day).

Is there anything in particular that you may be able to point out to us? There was a single reference that Vault works better with fewer folders but that isn't really possible for how we're using it. Let me know if there is more information I can share with you to help resolve this. Thanks

[Beth]

The command-line client does take a little longer than the GUI client. Do you get the same speeds if you perform the same actions with the Vault GUI client?

Also, I think it would be worth a try to isolate the SQL traffic. Two GB is certainly low for SQL server even when running just a few things.

from Visual Studio 2008, its taking about 30 seconds to display the repository list after completing the login screen.

Do you have many repositories?

Did you perform the suggested SQL maintenance?

Are you using folder security? If so, try turning it off and see if you get better speeds.

[wkrzystek]

>>> The command-line client does take a little longer than the GUI client. Do you get the same speeds if you perform the same actions with the Vault GUI client?

The GUI Client is almost as slow but harder to get good numbers from. I wish there was a log somewhere we could grab so that I didn't have to build the instrumentation myself.

>>> Also, I think it would be worth a try to isolate the SQL traffic. Two GB is certainly low for SQL server even when running just a few things.

OK. I'll try that probably tomorrow night and see if that helps any.

>>> from Visual Studio 2008, its taking abo ... in screen.Do you have many repositories?

The URL didn't come across but we have 9 repositories total - but 2 main ones that are heavily used.

>>> Did you perform the suggested SQL maintenance?

Yes - we did both the index defrag as well as the DBREINDEX with the update stats afterwards. DBCC CHECKDB reported 0 problems.

>>> Are you using folder security? If so, try turning it off and see if you get better speeds.

This was another item I was wondering - I had turned folder security on at one point soon after the upgrade but for performance, I wound up turning it back off. I'm wondering if there is something somewhere that didn't get disabled properly when I turned off the folder security.

Let me know if you think of anything else, otherwise, I'll plan on moving the SQL portion back to the vault server and see if that helps any. I'll let you know in a couple of days how that goes.

Thanks again for your help with this.

Walt

[Beth]

You can certainly use the Client Side Logging. If you turn on that logging, you should also place the Vault Server Log into debug mode (done in the Vault admin web client). Then make note of when you are performing your actions and what actions.

Sorry, that should have been a quote. Nine repositories isn't bad at all.

Folder security is set for each repository, so check the settings for each one in the Vault admin web page. Then it might be good to perform an iisreset just to clear out the server side caches and to force it to reload the settings. That is done by going to Start - Run and typing iisreset. It will disconnect everyone from Vault, so it's best to only do that at a convenient time.