Vault client hangs at startuup

[ww]

I have been having this problem frequently, though mostly on one machine, for a long time (through several Vault updates, anyway). I just upgraded to 6.0.1 on server and client, but that has not resolved it.

When I start the client it hangs, using 90-98% of CPU time, until I kill it (I've let it go as long as 10 minutes without it becoming responsive.

When it happens I can start the client on another computer, connecting to the same repository, and everything is fine.

I generally try resetting the Vault server by doing an IIS reset, but that doesn't help. If I kill the vaultguiclient process and start it again it will sometimes work, but sometimes I have to kill/restart a few times.

The client log only shows one entry when this happens:

Code: Select all

11/18/2012 8:50:19 PM <generic>: [GUIClientWorkerThread:4] After loading from disk on startup, _cmRepository.Repository.Root is null.
   at VaultClientOperationsLib.TreeCache.Load()
   at VaultClientOperationsLib.TreeCache..ctor(Int32 repID, String username, String uniqueRepositoryID, String localStoreBasePath, ClientInstance ci)
   at VaultClientOperationsLib.ClientInstance.SetActiveRepositoryID(Int32 id, String username, String uniqueRepositoryID, Boolean bUseTreeCache, Boolean doRefresh, Boolean updateKnownChangesAll)
   at VaultClientPresentationLib.GUIClientInstance.ChooseRepository(Boolean forceDialogShow, String inProfile)
   at VaultClientPresentationLib.GUIClientThread.ProcessCommand(GUIClientThreadCommand command, GUIClientThreadCommandResult& outputResult)
   at VaultClientPresentationLib.GUIClientThread.Start()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Presumably there is something wrong with the cache.

How do I get past the immediate problem and get my client to load? I have a large number of files pending checkin right now and don't want to lose their status.

[Beth]

We have a couple of KB articles that are going to help you here.

Client Side Cache
Clear Client Side Cache

What you will want to do is be selective about what pieces of the cache you clear, and rather than deleting, I'm going to have you rename parts. That way if you need to return to a previous state, it's easy to do so.

1) Close Vault and any IDEs that are using Vault integration, such as Visual Studio.
2) Go to the Vault client-side cache. (%USERPROFILE%\\AppData\Local\Sourcegear\Vault_1\Client\{repository-guid}\{vaultuser}\) In the article about clearing the cache is the SQL query for finding which GUID matches the repository you're working in.
3) Rename cachemember_Repository and cachemember_LastStructureGetTime.
4) Then open your Vault GUI client and see if it's better.

If that doesn't do it, then perform steps 1-4 again, but the second time, I'll have you rename cachemember_Repository, cachemember_LastStructureGetTime, CacheMember_SelectionString, CacheMember_ExpandedNodes, CacheMember_ChangeSetComment, and CacheMember_CheckOutList.

What I'm trying to do here is only move out of the way first the most likely culprit to the error, then other items that won't be a problem if they are out of the way. There's about 3 more levels of clearing things out we can do, but we'll get to that if these two don't work.

[ww]

Renaming those two files did not make a difference. I renamed all files except cachemember_WorkingFolderAssignments, did an iisreset, and started the client. Still hanging. Still reporting "After loading from disk on startup, _cmRepository.Repository.Root is null." in the client log.

[Beth]

This could be something in your baselines then. Try renaming the _sgvault folder instead.

[ww]

I don't have a single _sgvault folder. My baseline folders are all under the source folders.

[Beth]

Let's change that temporarily.

Since you can't seem to open your Vault GUI client, I'll have you change the setting for the cache in the registry. The location to look in is HKEY_CURRENT_USER\Software\SourceGear\VaultPro\Client\Settings. The setting is StoreDataInWorkingFolders. Change the data to False.

Then open your Vault GUI client. Since it will have more to rebuild, it may take a while. Allow it some additional time. Then let me know your results.

[ww]

Since my previous reply (without doing anything to the _sgvault folders or settings) I started up the client and this time it loaded. It has rebuilt the pending change set. I shut everything down and put back the old ChangeSetItems file so I could get those items back, and that seemed to do fine. So now the Vault GUI client is OK as far as I can tell.

In the meantime, however, starting sometime yesterday and continuing today my Visual Studio 2010 instance (with the Vault source control provider) has been frequently using 90%+ CPU time when it shouldn't be. This appears to be SourceGear's fault-I closed Visual Studio and shut down the Vault server so it couldn't connect, then restarted Visual Studio and chose to work offline. The intensive CPU usage went away. Reconnected to Vault and it came back.

At this point I think I want to just check everything in through the GUI client so I can start fresh with everything. So my questions:

1. What do I need to do to clean everything out and have the best chance of getting this all back to working?
2. Why is this so difficult? Why is there not a useful error message or log file pointing to the problem?

[ww]

Update: since I checked everything in, Visual Studio is performing OK, and there have been no further exceptions reported in the log for the vs addin, so perhaps everything is fixed.

[Beth]

I still think something in the cache was trying to resolve itself, and the error did indicate that. The error did also indicate to me the most likely culprit which I had you deal with right away.

You had expressed in interest in not clearing out your entire cache, so I presented a lot of step by step options to deal with as little information at a time each time. I probably made it seem very confusing that way.

The "fast and easy" way would have been to close both your Vault GUI client and VS, then rename the cache at the top level, open the project through Visual Studio, allow it to rebuild the cache and then check in from there.

[ww]

Thank you for the help. Everything has been OK since yesterday.

Can you explain why, originally, the Vault GUI client was hanging, but the VS plugin was able to continue working? Do they use different caches? Or is it because the VS plugin was looking only at a subset of the repository?

[Beth]

They don't use different caches, but if you don't have their working folders exactly lined up, there will be some conflicts.

[ww]

This problem has come back. Same as before: client hangs on startup. Same error message in log. I have renamed all files in cache folder and still get the same problem. At least I assume--eventually I kill the client since it shows no sign of doing anything other than eating CPU time. How long should I let it go if it's rebuilding something? I assume the fact that I still get the error message in the log tells me it's still broken and there's no point in waiting for hours for it to do something.

Tried changing StoreDataInWorkingFolders to False. No difference.

I have been mostly working disconnected from Vault in VS for the last week or two because I was again seeing nearly constant very high CPU usage by devenv.exe. Once I disconnected Vault, that problem went away.

So I don't know if those are both the same problem or two different problems.

[lbauer]

These may be two different problems.

The slowdown in Visual Studio 2010 is a known issue, and has been fixed in Vault 6.1, which was released last week. I would suggest upgrading and see if it helps your Visual Studio issue.

If you are having slow logins with the Vault client, try disabling the setting in the Vault Client under Tools->Options->Network Settings-> Request database delta on repository cache miss. . . .

Let us know how it goes.

[ww]

I will try 6.1 for the VS problem but that still leaves me with the hang problem, which I have been experiencing intermittently since 5.x.

I have tried disabling "Request database delta on repository cache miss" without luck in the past. Not sure what it's set to now, since I can't get Vault to load at all.

Define "slow login". I have let this thing sit there as long as 30 minutes, using most of the computer's CPU time before I killed it. That is not a "slow login"--that's an unusable client. The log file shows only the one exception at startup. Is there some sort of logging that will show whether it's actually doing anything?

[lbauer]

Is there something different about this machine? Does it have anti-virus software that could be interfering?

If you log into you machine as a different user, does the client hang?

Is there anything in the the Vault Server log at the time of your hanging client loging that might provide a clue?

Anything in the Windows Event Viewer on your client machine?

Does clearing the cache and baselines always solve the hang?