Upgrade to 3.1.1 API causes custom program to hang.

[dking513]

Hello,

I have written an application that uses the Vault API. My application is using version 3.0.1 of the following files: SGDiff, vaultclientnetlib, vaultclientoperationslib, vaultlib, vcdiff.

I upgraded the Vault Server to 3.1.1. The upgrade went fine, and my custom application (using the version 3.0.1 client files) works with the upgraded server.

I wanted my client and server to be on the same revision, so I updated a client with the 3.1.1 api files. (I had to recompile my custom .NET app - so much for Xcopy deployment!) The client now hangs when attempting to upload a file. I can login to the repository, refresh the cache, and check out files, and can create new folders, but can't upload. Looking at the log file on the server, it doesn't seem to see the upload request.

Specs for my client: Windows XP, .NET framework 1.1 sp1, MDAC 2.8.

Any ideas?

PS - I tried to configure the app.config file to enable debugging, but did not see a log file, or entries in the event log.

[dking513]

Closer inspection of the VaultServer log shows that the file IS being uploaded - but the client using the API still hangs. It seems to be waiting to commit the AddFile transaction.

[dan]

If you try to upload a file from the Vault GUI client, does it work? (I want to make sure the problem is with the custom app rather than some change in config on the machine since the upgrade).

You can also turn on client logging for you app, and see what is happening within the Vault client libraries when it hangs.

[dking513]

Hi Dan,
I am able to upload and check in files from the Vault GUI.
But I am not able to do either from my custom client.

I tried to turn on the debugging using the App.config file, but do not see a log file in my %temp% dir, nor entries in the System event log. (I set enable logging to true, and clases to all - am I missing something?)

I stepped through my code to see where the application was hanging and it gets stuck calling the .Commit(changesetcollection) method.

(On a personal note, I can relate to this. A number of girlfriends have accused me of getting hung up on this call as well... I digress...)

My parenthetical comment is not so off topic because my tests now lead me to believe that the issue is not related to any one function (uploads or check ins or whatever) but more generally an inability to .Commit.

Interestingly, when I look at the log file on the server it tells me:
BeginTx returned: Success
CheckIn: $/... returned:Success

Nothing after that. Should we be seeing a EndTx line here?

Any ideas? How can I get the logging working?
Thanks,
Dave

[dan]

Try renaming the .confg file to YourAppName.exe.config (where your app is called "YourAppName.exe") and put it in the same folder as the app, and see if that generates some logging output.

Some wild stabs:

Have you changed the location of your client cache files since upgrading to 3.1.1?

Are you logged on as the same user when using your custom app as when you use the GUI client?

If you connect to sourcegear's public demo server, does it hang there (vaultdemo.sourcegear.com, using guest/guest as user/pw).

[dking513]

I renamed the .config file, but still can't get the logging working.
The custom exe is a vb6 exe which uses COM interop to talk to my .NET dll which makes the calls to the SourceGear API. Still seems like I should be able to get that logging to work.

As for client cache - I did not make any intentional changes, but it does appear that the 3.1 client, by default, stores to a different location. 3.0 stored to user\Application Data\Source Gear while 3.1 stores to user\Local Settings\Application Data\Source Gear. I don't understand why this would have any effect though. I deleted the cache folder and ran my client, and verified that it pulled the repository information to the 3.1 cache folder.

I still need to do the test to your public test server.

[dking513]

I can get the logging to work for the VaultGUIClient, so it's not a problem with my machine - just something with my .Net assembly not finding the config file...

[dan]

Another change we made in 3.1 was that we changed the Vault DLLs to be strongly named. I'm not sure why this would affect file uploads, but it is possible. Do you reference the dlls by name anywhere?

[dking513]

I guess I don't totally understand the impact of strong names.
I have references set to your API assembilies, and I have functions that use syntax like 'VaultClientOperationsLib.ClientInstance'

I am still frustrated that I can't get the logging to work, since I think this would shed a lot of light on this problem. To rule out that it was a VB6/.NET interop issue, I created a new .NET exe and tried to run that. Still no log file.

This is getting quite frustrating. Do you think you could give me a call?
Thanks,
Dave

[mlippert]

Dave or Dan,
When you get this figured out, could you post a synopsis here?

Thanks,
Mike

[dking513]

I opened a support case with Microsoft, to see if they could find out if there was a problem in the VB6/.NET interop layer.

They had me run the custom app until it hung, and then got a "hang dump". Here's the information that the Microsoft tech extracted from the hang dump:

======================================
Number of exceptions of this type: 1
Exception 0x07b4d2f4 in MT 0x79b96aec: System.ArgumentNullException
_message: A Null IWebProxy is not allowed, please use GlobalSelectProxy.GetEmptyWebProxy() instead.
_stackTrace:
0x7b25959b [DEFAULT] [hasThis] Void System.Net.HttpWebRequest.set_Proxy(Class System.Net.IWebProxy)
0x0b23faac
0x7b345f28
0x0acef4b7 [DEFAULT] [hasThis] Class System.Net.HttpWebRequest VaultClientNetLib.VaultConnection.UploadFile(String,String)
0x0b23fac4
0x035990a8
=====================================

Does this shed any light on anything? Is it possible that my custom application isn't configuring a proxy setting correctly? Has there been any change to the default proxy values from version 3.0.7 of the api to 3.1?

Thanks,
Dave

[jeremy_sg]

Dave,

Nothing that I can think of changed in the proxy handling between 3.0 and 3.1, but there are other changes that may have impact here. My first general (and probably useless) question is, you don't call ResetProxy or manually set a proxy in your VB code, do you? Another potential help would be to try with 3.0.7, to see if there was an intermediate 3.0 change that broke something, or if it was 3.1.

[dking513]

I do not set or reset the proxy from my VB code.
With the 3.0.7 libraries, my client works fine.

[jeremy_sg]

Email me your phone number using the button below this post, and we'll try to track down the problem.

[dking513]

After a fair amount of effort, I was able to isolate the problem to the fact that I was handling the progress events being raised from the SourceVault API. After calling .Commit, any attempt to handle a progress or status message event (to update my own custom progress indicator) would cause the program to hang.

With the help of Jeremy and Microsoft, we got to the bottom of this.
Below is the analysis from Microsoft (after looking at trace information captured during a hang)

******* The following is an email for a support case from Microsoft Corp.
******* DO NOT REPLY TO THIS MESSAGE--your email will not be added to
******* the case if you do. Instead, FORWARD your response to the
******* email address COMPMAIL@MICROSOFT.COM and place your text after
******* the keyword 'MESSAGE:'. Also, delete all other text above
******* and below the keywords 'CASE_ID_NUM: SRnnn' and 'MESSAGE:'
******* to ensure proper delivery of your email. Thank you.

CASE_ID_NUM: SRX050831601838
MESSAGE:
********************** The message for you follows ************************
Hello David,

On reviewing the dumps that you provided we found that a deadlock appears to be the cause of the issue. Thread 0, this is our main thread and is an STA, starts off in DocToolSourceVaultLibrary.DocToolSourceVaultLibrary.svCheckInFile where it calls into the 3rd party component (VaultClientOperationsLib) to do a VaultClientOperationsLib.ClientInstance.Commit.

This Commit spins up a new thread (thread 16) which uploads the item and then goes into a loop of Sleeping for 100ms and trying to kill the worker upload thread. This kill loop spins until all worker threads are done working. As part of uploading the item from the worker thread VaultClientOperationsLib updates it's progress percentage. DocToolSourceVaultLibrary is setup to handle this progress change event and in turn calls Invoke to update the progress %. Invoke passes the call to the object's parent thread, in this case thread 0, to do the work. But thread 0 is busy doing its sleep/kill loop so it can't process the Invoke call: This means that thread 0 is stuck waiting on thread 16 to finish and thread 16 is stuck waiting to call Invoke on thread 0.

There are two solutions that should resolve the issue:

1. To fix this issue in your code you will want to modify DocToolSourceVaultLibrary.DocToolSourceVaultLibrary.svCheckInFile to asynchronously call the VaultClientOperationsLib.ClientInstance.Commit on another thread. This will free up thread 0 to do the Invoke updates so we won't get into the deadlock situation.

2. The 3rd party could also make a modification to their code so the above workaround is not necessary. If they call DoEvents in their Sleep/Kill loop it should allow the Invoke to process on thread 0 which will prevent the deadlock.


-----------------------------------------

This analysis also explains why this problem wasn't happening with the prior version of the API - the 3.0 API probably relinquished control back to my single threaded VB6 app during the sleep/kill loop thus allowing me to process the progress events, while the 3.1 API does not.

My workaround for right now is just to ignore the events raised during a .commit.

Just thought I would post this conclusion for anyone who might have been following it.

(PS - Jeremy, so your initial response was right on the money when you were surprised that my progress indicator was working at all...)