Implicit Checkout and file saving REALLY slow.

[gaffa]

I'm experiencing a long delay when checking out files (implicitly checking out rather than explicitly). It seems to be taking upwards of 9 seconds to check out a file, and then when I hit Ctrl-S (and I'm an inveterate Ctrl-S'er - muscle memory pretty much means I hit Ctrl-S every time I pause), it seems to take another 8-10 seconds to save the file (and about 8-10 seconds per file when I hit Shift-Ctrl-S and there are a couple of file that require saving). While checking out and saving, VS2008 goes into the (Not Responding) state for part of the time, although not the entire time.

The log files (attached) show that is seems to query the current repository and gets a list of EVERY checked out file for EVERY user for EVERY project EVERY damn time... and we've got 11 users, and a large number of projects across 4 repositories, which makes for a very big list.

It's beginning to give me the right $%&@#s... any thoughts about what's going on?

Some salient tech details:
Vault Client : 4.1.4.18402
Vault Server: 4.1.3.18336 (yeah, I know they are mismatched...)

Solution contains about 25 projects (mostly little ones, but a couple of large ones with > 500 files)

Client Environment: Vista, VS2008 SP1, 64bit Quad Core, 8Gb RAM, UAC turned off.

Vault Server Environment: Win Web Server 2008 64bit (as a virtual machine), IIS 7, single host processor, 4Gb RAM (host machine is Dual Xeon Quad, 18Gb RAM, SAS RAID 5 running VMWare ESXi)

Database Server Environment: Windows 2003 Enterprise 32bit , 4Gb RAM, dual Xeon single core, SQLServer 2000 fully patched, SCSI RAID 5 disks, Vault DB is about 14Gb

All machines are fully patched.

Network: 1Gb via a HP ProCurve switch

Access to the vault server is via SSL with a valid certificate

Thats about all I can think of in terms of tech details. Server log and client log is attached.

The client log was created by having my solution open then:
1. Enabling logging
2. Pressing enter in a file that wasn't checked out
3. Waiting
4. Adding a single line of text to the newly checked out file
5. Waiting 5 seconds to pop a decent gap in the log file timestamps
6. Saving just the active file
7. Waiting
8. Disabling logging


Matt

[gaffa]

Forgot to add:
* Working in exclusive checkout mode
* Running the enhanced client

Matt

[gaffa]

One more addendum:

If I right click of a folder in the Solution Explorer (say to add a new file to my project), it takes a couple of seconds to actually display. The log file (attached) shows a fair bit of random activity going on, with the delay appearing to be in the GetSolutionSpecificInternalChangeSet bit of Vault.

Log file attached (only action performed while logging was enabled was right clicking on the folder).

One last note: I've killed my virus scanner (in the vain hope that it was the on access scanner interfering), but it hasn't made a difference.

Matt

[gaffa]

I rolled my Vault client back from 4.1.4 to 4.1.3 so it matches the server version, but this hasn't fixed my dramas.

Matt

[lbauer]

When you press "enter" in a file, it becomes modified and Vault checks it out automatically. The checkout list is updated with every checkout and sent to the client. According to the client log, you have about 1000 checkouts in the repository, which is quite a few. As you have seen, this has an impact on performance.

The first step might be to just checkout the items you need to work on to minimize the size of the checkout list.

[gaffa]

Ahhh, so it scans my ENTIRE checkout list?

Yeah, I've got heaps of checkouts out at the moment - I'll push a few of them back up and see if that improves my life a little.

Matt

[gaffa]

That seems to have improved the speed (but I've only checked out one file so far in testing, having checked all my changes back in).

I only had 200 odd files checked out (there seemed to be about 947 files checked out by all users from the repository)

Thanks for that

Matt

[lbauer]

Ahhh, so it scans my ENTIRE checkout list?

Not only yours but all the checkouts from every user. If other users reduce the number of checkouts, that will help you as well.