Solaris command line client: timeout

[James Jeffers]

I'm seeing this now, quite consistently:

Code: Select all

<quote>
Wait timeout for mutex after 30000 milliseconds.
</error>

This is occuring with every client operation.

[lbauer]

Do you see any failures in the Vault Server log (%windir%\temp\sgvault)?

Try recycling the Vault App Pool or restarting IIS (iisreset). Does that help?

[James Jeffers]

Here's the entire error message, for more information:

Code: Select all

<vault>
<error>
Wait timeout for mutex after 30000 milliseconds.
</error>
<exception>
System.Exception: Wait timeout for mutex after 30000 milliseconds.
in <0x0024c> VaultLib.SystemMutex:Take (UInt32 ms)
in <0x00018> VaultClientOperationsLib.CacheMember:TakeSystemMutex ()
in <0x00074> VaultClientOperationsLib.CacheMember_Repository:.ctor (System.String folder)
in <0x0013c> VaultClientOperationsLib.TreeCache:.ctor (Int32 repID, System.String username, System.String uniqueRepositoryID, System.String localStoreBasePath, VaultClientOperationsLib.ClientInstance ci)
in <0x001a0> VaultClientOperationsLib.ClientInstance:SetActiveRepositoryID (Int32 id, System.String username, System.String uniqueRepositoryID, Boolean doRefresh, Boolean updateKnownChangesAll)
in <0x0088c> VaultCmdLineClient.VaultCmdLineClient:Login (Boolean bAllowAuto, Boolean bSaveSession)
in <0x00010> VaultCmdLineClient.VaultCmdLineClient:Login ()
in <0x000e4> VaultCmdLineClient.VaultCmdLineClient:ProcessCommandGet (System.Collections.ArrayList strItemArray)
in <0x0219c> VaultCmdLineClient.VaultCmdLineClient:ProcessCommand (VaultCmdLineClient.Args curArg)
</exception>
<result success="no" />
</vault>

[James Jeffers]

Reset IIS vis "iisreset" command. That didn't resolve the issue.

Looks like there's a system lock or mutex that the client library cannot acquire. Perhaps a previous operation was aborted or uncleanly ended?

[James Jeffers]

This may or may not be related.

I see lots of files named "tmpfbc539d9" etc, piling up in /var/tmp.
Does the Vault client attempt to clean these files after each operation? Can I remove them safely (that is, without disturbing any current working folder settings and caches)?

[lbauer]

Send me a copy of your Vault server log file.

Are you seeing this error on your Solaris box?

Does the same error happen on other machines?

[James Jeffers]

Only seeing this on Solaris. Windows clients are humming along.

I will send you a copy of the server file momentarily.

[James Jeffers]

I tried the command line client with another user. The command worked, that is, the timeout error did not occur.

It looks as if the temporary files or locks used by the Vault user are not being freed. Can I check this manually? What sorts of files are associated with a Vault user, not the system user?

[lbauer]

On Windows, the Command Line Client creates client-side cache files with information about Known Changes, the Checkout list, Repository, etc. If you set a working folder (SETWORKINGFOLDER) it also creates the hidden state folder, _sgvault, which contains baselines of the files in the working directory. These cache files are unique to each Vault user account on the client machine.

This could be what you are seeing. I'm not sure in what way they might be locking the client. Could tmp directory space be an issue here?

[James Jeffers]

This particular machine is being used because disk space is in ample supply for all partitions.

I'll see if brutal removal of temporary working folder caches helps at all.

[James Jeffers]

I clobbered _sgvault for the failing user. No change, still fails on the timeout.

[lbauer]

Anything different about the user account that's getting the timeout and the one that isn't? Are they doing different operations?

Are you using folder security in Vault? Do the users have different security rights?

If the user getting the timeout connects from a different machine, do they get the timeout?

[James Jeffers]

I can use the failing Vault user, let's call it "build", on another Solaris machine that also has a Mono instance built from source, and the same Vault client .zip package.

As far as I know we are not using Vault folder security.

[PG]

I am seeing this same issue on Red Hat Enterprise Linux ES release 3
(Taroon Update 5). I have the Mono framework installed and different
Mono versions result in different errors. I have the command line
client from vault 3.1.2.

Mono 1.1.8.3 gives:

Code: Select all

<vault>
<error>
Wait timeout for mutex after 30000 milliseconds.
</error>
<exception>
System.Exception: Wait timeout for mutex after 30000 milliseconds.
in <0x00168> VaultLib.SystemMutex:Take (UInt32 ms)
in <0x00016> VaultClientOperationsLib.CacheMember:TakeSystemMutex ()
in <0x0004b> VaultClientOperationsLib.CacheMember_Repository:.ctor (System.String folder)
in <0x00120> VaultClientOperationsLib.TreeCache:.ctor (Int32 repID, System.String username, System.String uniqueRepositoryID, System.String localStoreBasePath, VaultClientOperationsLib.ClientInstance ci)
in <0x000f8> VaultClientOperationsLib.ClientInstance:SetActiveRepositoryID (Int32 id, System.String username, System.String uniqueRepositoryID, Boolean doRefresh, Boolean updateKnownChangesAll)
in <0x00650> VaultCmdLineClient.VaultCmdLineClient:Login (Boolean bAllowAuto, Boolean bSaveSession)
in <0x0000e> VaultCmdLineClient.VaultCmdLineClient:Login ()
in <0x000c7> VaultCmdLineClient.VaultCmdLineClient:ProcessCommandGet (System.Collections.ArrayList strItemArray)
in <0x016ac> VaultCmdLineClient.VaultCmdLineClient:ProcessCommand (VaultCmdLineClient.Args curArg)
</exception>
<result success="no" />
</vault>

Mono 1.1.9.2 gives:

Code: Select all

<vault>
<error>
Array index is out of range.
</error>
<exception>
System.IndexOutOfRangeException: Array index is out of range.
in <0x00027> VaultLib.VaultNameValueCollection:SetNameValuesByArray (System.String[][] arNameValues)
in <0x000a3> VaultClientNetLib.VaultConnection:ListCloakedObjects (Int32 nRepID, VaultLib.VaultNameValueCollection nvc)
in <0x0002c> VaultClientOperationsLib.ClientInstance:RefreshCloakedObjectsList ()
in <0x00380> VaultClientOperationsLib.ClientInstance:SetActiveRepositoryID (Int32 id, System.String username, System.String uniqueRepositoryID, Boolean doRefresh, Boolean updateKnownChangesAll)
in <0x006a6> VaultCmdLineClient.VaultCmdLineClient:Login (Boolean bAllowAuto, Boolean bSaveSession)
in <0x0000e> VaultCmdLineClient.VaultCmdLineClient:Login ()
in <0x000c7> VaultCmdLineClient.VaultCmdLineClient:ProcessCommandGet (System.Collections.ArrayList strItemArray)
in <0x01794> VaultCmdLineClient.VaultCmdLineClient:ProcessCommand (VaultCmdLineClient.Args curArg)
</exception>
<result success="no" />
</vault>

Is there any resolution for this issue or some work around maybe?
My work here is at a complete standstill so please help as soon as
you can. I have restarted IIS. The vault username I use on Linux
and which results in these errors works just fine from windows
client.

Summary:
Operating system: Red Hat Enterprise Linux ES release 3 (Taroon Update 5)
Vault: version 3.1.2 (3511)
Mono: 1.1.8.3 and 1.1.9.2

[lbauer]

We have a newer build that may help with PG's issue.

Contact me via email or private message and I'll send you the build.

Unfortunately, Jim, it won't address the Solaris problem you're seeing.