Wait timeout for mutex after 30000 milliseconds

[bastress]

Hi we have been experiencing a new issue over the last several weeks that appears to be getting worse.
During our build process we are seeing:

10:35:25 Exit code -1 unsuccessful; throwing IOException: <vault>
10:35:25 <error>
10:35:25 <exception>System.Exception: Wait timeout for mutex after 30000 milliseconds.
10:35:25 at VaultLib.SystemMutex.Take(UInt32 ms)
10:35:25 at VaultClientOperationsLib.CacheMemberBase.TakeSystemMutex()
10:35:25 at VaultClientOperationsLib.CacheMember_Repository..ctor(String folder)
10:35:25 at VaultClientOperationsLib.TreeCache.Load()
10:35:25 at VaultClientOperationsLib.ClientInstance.SetActiveRepositoryID(Int32 id, String username, String uniqueRepositoryID, String repositoryName, Boolean bUseTreeCache, Boolean bDoRefresh, Boolean updateKnownChangesAll, Boolean bNotifyRepoChgEvents)
10:35:25 at VaultClientOperationsLib.ClientInstance.SetActiveRepositoryID(Int32 id, String username, String uniqueRepositoryID, String repositoryName, Boolean doRefresh, Boolean updateKnownChangesAll)
10:35:25 at VaultClientIntegrationLib.ServerOperations.SetRepository(VaultRepositoryInfo repositoryInfo)
10:35:25 at VaultClientIntegrationLib.ServerOperations.SetRepository(String repositoryName)
10:35:25 at VaultClientIntegrationLib.ServerOperations.Login(AccessLevelType altCommand, Boolean bAllowAuto, Boolean bSaveSession)
10:35:25 at VaultCmdLineClient.VaultCmdLineClient.ProcessCommand(Args curArg)
10:35:25 at VaultCmdLineClient.VaultCmdLineClient.Main(String[] args)</exception>
10:35:25 </error>
10:35:25 <result>
10:35:25 <success>False</success>
10:35:25 </result>
10:35:25 </vault>

This is after executing commands such as the one below in Jenkins during a build...
"C:\Program Files (x86)\SourceGear\Vault Client\vault.exe" VERSIONHISTORY -host vault.creditinfonet.com -username ******* -password ********* -repository CIN_2008 -rowlimit 1 $/Rio/branches/QA/Application

Jenkins will attempt two tries then fail the job. we have been seeing a LOT more failures recently. All builds and SCM pulling happen using the same user. We currently have about 150 projects in Jenkins. We have Jenkins set with 3 executors and a max of 5 SCM pulls.

We are using Vault 10 and Jenkins 2.114.

[Beth]

What happens if you try the same command manually from the command line?

[bastress]

We did test this and had no issues. We tried opening more then one command window as well and executing several at once. All returned fine each time we tried. This test was done on the Jenkins server where the builds are processed using the same user.

[Beth]

What is the OS of the machine in question?

Are you using the Vault .Net command-line client or the Java command-line client?

Have any updates been installed right before the issue started or since then?

Do you have any instances of either Vault or Jenkins on the server running that haven't terminated on their own?
(viewtopic.php?f=22&t=6499)

How many builds are launched at once with Jenkins?

[bastress]

Windows Server 2012 R2
We are using Windows PowerShell in Jenkins
We are using the Vault command line client that comes with the GUI Client. 10.0.0.736
I am not entirely sure when the issue started. I see the problem in the earliest build we have saved, about a month ago.
We have three executors set up so a maximum of 3 builds at once.
The SCM pulling is set for a max of 5 at once.

I did noticed that we normally have 5 vault processes "Running" on the Jenkins box. As the day gets busier, the number will increase and have 5 to 10 extra vault process that have a status of "Terminated" they will hang around for 5 to 10 seconds then go away and be replaced by others who were recently terminated..

[bastress]

I have some additional information. The developer before me wrote a plugin for Jenkins to handle the vault interaction. It does not look like we are using PowerShell. Here is some of the relevant vault client code...

package hudson.plugins.vault;

import hudson.FilePath;
import hudson.Launcher;
import hudson.Proc;
import hudson.model.TaskListener;
import hudson.util.ArgumentListBuilder;

import java.io.File;
import java.io.IOException;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;

import org.apache.commons.io.output.ByteArrayOutputStream;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

class VaultClient {

private static final Log LOG = LogFactory.getLog(VaultClient.class);

private final Launcher launcher;
private final Map<String,String> environment = new HashMap<String,String>();
private final FilePath workingFolder;
private final ArgumentListBuilder args = new ArgumentListBuilder();
private final TaskListener listener;

public VaultClient(Launcher pLauncher, TaskListener pListener, Map<String,String> pEnvironment, VaultHome pVaultHome, FilePath pWorkingFolder) {
this.launcher = pLauncher;
this.listener = pListener;
if(pEnvironment != null) this.environment.putAll(pEnvironment);
this.workingFolder = pWorkingFolder;

args.add((new File(pVaultHome.asFile(), "vault.exe")).getPath());
}

.................................
.................................

public VaultResult execute() throws IOException, InterruptedException {

if(LOG.isDebugEnabled()) LOG.debug("Executing: " + this.args.toStringWithQuote() + " @ " + (new java.util.Date()) + " in " + this.workingFolder.getRemote());

ByteArrayOutputStream vaultOutputStream = new ByteArrayOutputStream();

Launcher.ProcStarter vaultStarter = this.launcher.new ProcStarter();
vaultStarter.cmds(this.args.toCommandArray());
vaultStarter.envs(this.environment);
vaultStarter.stdout(vaultOutputStream);
vaultStarter.pwd(this.workingFolder);
Proc process = this.launcher.launch(vaultStarter);
int exitCode = process.join();

if(LOG.isDebugEnabled()) LOG.debug("Exit code: " + exitCode);
listener.getLogger().println("Exit code: " + exitCode);

byte[] vaultOutputBytes = vaultOutputStream.toByteArray();
vaultOutputStream.close();
vaultOutputStream = null;

if(LOG.isTraceEnabled()) LOG.trace("Output: " + (new String(vaultOutputBytes)));

// Something went wrong, don't attempt to parse anything
if(exitCode != 0) {
String output = new String(vaultOutputBytes);
if(LOG.isWarnEnabled()) LOG.debug("Exit code " + exitCode + " unsuccessful; throwing IOException: " + output);
listener.getLogger().println("Exit code " + exitCode + " unsuccessful; throwing IOException: " + output);
throw new IOException("Exit code " + exitCode + " unsuccessful");
}

VaultResult result = (new VaultResultParser()).parse(vaultOutputBytes);
result.setExitCode(exitCode);

return result;
}

[Beth]

I'd like to get more detailed logging, but we should take that offline. Could you send an email to support at sourcegear.com (attn: Beth) with a link to this forum thread?

[bastress]

sent to support@
sourcegear.com

[Beth]

Got it.

HS: 250749