Vault API is getting slower

[peterbrown05]

hello
we use the vault api extensively to manage our release procedure; developers use an in house tool which updates version files, labels the modules and checks that files are correctly checked in.
Recently we have seen the API starting to run really slowly; so with the same code on our end, and same vault version; sometimes it can take 15minutes for this to complete; whereas at other times it only takes 15 to 20 seconds.

Can you offer any advice how we can;
a.) improve the performance of the api
b.) monitor the performance of the vault api/vault server to identify if we need to upgrade the h/w.
c.) are there any logs generated that would help me prove that the vault system is struggling on the provided hardware (if an upgrade is needed, I need to be able to justify the expenditure and some proof is always helpful!).

thanks in advance,
cheers
peterB

[lbauer]

Is performance slowing for users as well?

Has your database grown and is your repository more complex (more folders, labels, branches)?

It's possible it's not specifically the API, but performance in general.

What version of Vault are you using? What operating system is it installed on?

Is SQL Server on the same machine, or different machine?

How many users do you have?

How often does the API or other automated process log in/log out?

[peterbrown05]

for sure the number of repositories has increased and the complexity within them has increased too (yes; more branches, labels etc).

Using Vault 5.0 (I cant remember the exact version but its not the absolute latest available). If this is helpful I can find this out and repost later).

Installed on OS Server2003 (running on a VM)
SQL Server is on a different VM (again Server 2003, but running SQLServer 2005).

Users: 25

I believe the api logins in once per "developer module release" but at times this can mean (eg at end of day) that we have 4 or 5 developers using this tool at the same time hitting the same repository.


I suspect that the machine vault is running on or the sql server instance should be upgraded; but would be nice to be able to show some metrics of it struggling. or, can you offer a suggested spec for these boxes with 25 users.?

[peterbrown05]

vault version update; we are actually running;

This Vault client / server version 5.0.3.18802

[lbauer]

As a first step, We'd like to see a copy of the Vault Server log in debug mode to see if there are any obvious errors or slowdowns.

You can enable debug server logging in the Admin Web Client under Server Settings->Logging->Log Level.
Once you've captured what you consider an "api slowdown," Send a copy of the log zipped up to support at sourcegear.com, Attn: Linda. Please include a link to this forum post.

Also, let me know what account the API is logging in as, and about what time in the log to look for the API action.

You can enable debug server logging in the Admin Web Client under Server Settings->Logging->Log Level. Note, these logs can get very large, so you might want to change the logging back to Quiet.

[peterbrown05]

Hi There,
Ive finally had chance to turn this logging on. I have extracted out from the log some activity around the time of the api calls;
the log shows;
eric ; making changes and checking in files (normal developer interaction using vault GUI)
this runs from 19/08/2011 00:30:11 till 19/08/2011 00:30:29

then
at 19/08/2011 00:31:18 eric stars to use our release tool to formalise this module;
the tool follows the following "high level steps":

1. It checks to see the current status of files in that folder and ensures non are renegade etc
2. then checks out the PBSCLog.ver.ver file
3. the app then updates the .ver file with new data
4. the .ver file is checked in
5. the folder is labelled

This whole process here then finishes at 19/08/2011 00:31:52 which means this took approximately 34 seconds to complete. I think we would class this as being a "normal time" for the operation to complete.

We will monitor and wait for one of the "long" release times (where Eric reports it taking 5minutes or more to complete) and post back with logs for that once it occurs so we can see the difference.

However, I wonder if you see any issues/problems with these logs in advance of this ?

Worth noting that the SQL database is on a different machine to the Vault service.

[lbauer]

However, I wonder if you see any issues/problems with these logs in advance of this ?

No errors in the log.

I ran it through our log file analyzer which flags time gaps of more than 2 seconds. The only time gaps in your log were between logouts and logins, and even those were only about 5 seconds. So this log looks good.

[peterbrown05]

could you run your analysis over the attached logs?
one of my developers reported a problem during this period; and maybe this will help us narrow it down.

Also; assuming no errors show up; does your analysis tool help to show if the performance is as you would expect?

thanks

peteB

[lbauer]

What Vault account is being used by the Vault API?

[peterbrown05]

it is the users local account. so in that log I believe there are several users using the api; however the user that told me he had a very long release process was "eric.openshaw"

[lbauer]

I've been looking at the log -- do you know what was involved in the "release process?"

I see log activity by eric.openshaw from 9:08 a.m. to about 11 a.m. on 9/19. Is this all one release process?

Is this a process that runs regularly? Is it always slow, or was it just slow on this occasion?

[peterbrown05]

he released a number of modules during that time. Each module that is released follows the same process;
1. It checks to see the current status of files in that folder and ensures non are renegade etc
2. then checks out the PBSCLog.ver file
3. the app then updates the .ver file with new data
4. the .ver file is checked in
5. the folder is labelled

I believe the one he reported to take a long time was PBSCUIComponents

It seems that the process is occasionally very slow, but normally takes in total around 30seconds or so. On this occasion he reported that it took much longer. Note that after step 5 a number of other updates are done in other databases (bugtracker, release notes etc) but our ui seems to suggest that the api itself is being slow.

It might not be, but I would be interested to understand if you think the performance here is slower than you would expect for such operations.

cheers
peteB

[lbauer]

I took a close look at your log, but didn't see anything that would help me diagnose slowdowns in the API. The Vault server seems to be responding appropriately to requests, but I need more info on what is being requested.

For example, it may have taken 11 seconds to apply a label. That might be appropriate for labeling a folder, but too long for labeling a single file.

I'd suggest you contact me offline and we can get more details or discuss this by phone. Email support at sourcegear.com, Attn: Linda. Please include a link to this forum post.