Vault CmdLine login Timeout

[robe070]

Hi,

We have an overnight build that you'll be surprised to know runs every night! Last night one of the gets failed. We have 3 separate ones. The first succeeded the second failed and the third succeeded. It was 4 minutes between executing the 2nd and 3rd gets.

When we executed the (failed) command manually, it worked.

Is there a timeout setting somewhere?

The commands are almost identical. Only the disk directory and project folder differred. The one that failed is this:

Code: Select all

vault GET -user devbuild -password lansa -repository lansa -host devsrv               -verbose -nonworkingfolder f:\l4wbuild\Trunk\work/install $/VL/Release/v11/L4W11500_vlf/work/install 

The error reported was this:
<vault>
<error>
<exception>System.Exception: The connection to the server failed: server cannot be contacted or uses a protocol that is not supported by this client. Unable to connect to http://devsrv/VaultService. No server was found at the specified URL. Please verify your network settings using the Options dialog under the Tools menu in the Vault GUI Client. Web Exception: The operation has timed out ---> System.Exception: Unable to connect to http://devsrv/VaultService. No server was found at the specified URL. Please verify your network settings using the Options dialog under the Tools menu in the Vault GUI Client. Web Exception: The operation has timed out
at VaultClientNetLib.VaultConnection.Login(String strURLBase, String strUserLogin, String strPassword)
at VaultClientOperationsLib.ClientInstance.Login(String urlbase, String username, String password)
at VaultClientIntegrationLib.ServerOperations.Login(AccessLevelType altCommand, Boolean bAllowAuto, Boolean bSaveSession)
--- End of inner exception stack trace ---
at VaultClientIntegrationLib.ServerOperations.Login(AccessLevelType altCommand, Boolean bAllowAuto, Boolean bSaveSession)
at VaultClientIntegrationLib.ServerOperations.Login()
at VaultCmdLineClient.VaultCmdLineClient.ProcessCommand(Args curArg)
at VaultCmdLineClient.VaultCmdLineClient.Main(String[] args)</exception>
</error>
<result>
<success>False</success>
</result>
</vault>


regards

Rob Goodridge
LANSA Pty Ltd

[lbauer]

We'd like to see a copy of the Vault Server log for that time period. The Vault server log is sgvault.log and is in %windir%\temp\sgvault on the server machine. Send the logs zipped up to support at sourcegear.com, Attn: Linda. Please include a link to this forum post.

Let us know what time the the commands failed so we can look for corresponding errors in the log.

Also, let us know what version of Vault you are using.

[robe070]

I'm pasting part of the log here rather than sending the whole lot. I've included entries either side of the failure time back to the last stack trace (maybe you could explain those whilst you are at it too ).

I don't think its very informative. I think there is NOTHING in the log for the login that failed. It occurred between 18/03/2009 1:20:15 AM and 18/03/2009 1:21:10 AM which were the 2 gets that succeeded either side of the one that failed:

Code: Select all

----17/03/2009 6:06:45 PM    ChiVinh--CHIVINHVISTA(10.2.1.9)--SSL Disabled	Rolling Back a transaction   at VaultServiceSQL.VaultSqlConn.RollbackTransaction()
   at VaultServiceAPILib.VaultServiceAPI.Login(VaultSqlConn conn, Boolean allowwebonly, String strLogin, String strDecryptedPassword, String strHostname, String strSessionID, String strDomain, String& strToken, VaultUser& vu, Int32& nValidUserAttempts)
   at VaultServiceAPILib.VaultServiceAPISystem.DoLogin(Boolean allowwebonlyusers, Boolean bAdminLogin, HttpSessionState hss, String strHostname, String strLogin, String strPassword, String strRMKey, String& strAuthToken, String strDomain)
   at VaultServiceAPILib.VaultServiceAPISystem.DoLogin(Boolean allowwebonlyusers, Boolean bAdminLogin, HttpSessionState hss, String strHostname, String strLogin, String strPassword, String strRMKey, String& strAuthToken)
   at VaultService.VaultService.Login(String strHostname, Boolean bUseFullFiles, String username, String strEncryptedPassword, String strRMKey, String& strAuthToken)
   at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
   at System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Web.Services.Protocols.LogicalMethodInfo.Invoke(Object target, Object[] values)
   at System.Web.Services.Protocols.WebServiceHandler.Invoke()
   at System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest()
   at System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   at System.Web.HttpApplication.ApplicationStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData)
   at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)
   at System.Web.HttpRuntime.ProcessRequestNoDemand(HttpWorkerRequest wr)
   at System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr ecb, Int32 iWRType)
 
----17/03/2009 6:06:45 PM    ChiVinh--CHIVINHVISTA(10.2.1.9)--SSL Disabled	Login failed:  FailInvalidPassword 
----17/03/2009 6:06:50 PM    chivinh--CHIVINHVISTA(10.2.1.9)--SSL Disabled	Login 
----17/03/2009 6:09:07 PM    anthony--AUPC(10.2.1.20)--SSL Disabled	Logout 
----17/03/2009 6:13:19 PM    kate--KATEVISTA(10.2.1.11)--SSL Disabled	Login 
----17/03/2009 6:24:15 PM    kate--KATEVISTA(10.2.1.11)--SSL Disabled	Logout 
----17/03/2009 6:43:21 PM    admin--devsrv.syd.lansa.com.au(10.2.1.21)--SSL Disabled	Login 
----17/03/2009 6:43:22 PM    admin--devsrv.syd.lansa.com.au(10.2.1.21)--SSL Disabled	Logout 
----17/03/2009 6:48:18 PM    devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Login 
----17/03/2009 6:48:22 PM    devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Logout 
----17/03/2009 6:48:25 PM    devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Login 
----17/03/2009 6:56:21 PM    devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Logout 
----17/03/2009 7:31:03 PM    chivinh--CHIVINHVISTA(10.2.1.9)--SSL Disabled	Logout 
----17/03/2009 11:11:51 PM   devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Login 
----17/03/2009 11:11:55 PM   devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Logout 
----17/03/2009 11:11:58 PM   devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Login 
----17/03/2009 11:16:19 PM   devbuild--LANSABUILDPC4(10.2.1.35)--SSL Disabled	Login 
----17/03/2009 11:16:24 PM   devbuild--LANSABUILDPC4(10.2.1.35)--SSL Disabled	Logout 
----17/03/2009 11:16:26 PM   devbuild--LANSABUILDPC4(10.2.1.35)--SSL Disabled	Login 
----17/03/2009 11:17:16 PM   devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Logout 
----17/03/2009 11:21:29 PM   devbuild--LANSABUILDPC4(10.2.1.35)--SSL Disabled	Logout 
----18/03/2009 1:20:15 AM    devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Login 
----18/03/2009 1:21:10 AM    devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Logout 
----18/03/2009 2:13:51 AM    devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Login 
----18/03/2009 2:13:53 AM    devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Logout 
----18/03/2009 2:13:56 AM    devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Login 
----18/03/2009 2:22:17 AM    devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Logout 
----18/03/2009 5:08:11 AM    devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Login 
----18/03/2009 5:08:49 AM    devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Logout 
----18/03/2009 6:30:15 AM    devbuild--LANSABUILDPC4(10.2.1.35)--SSL Disabled	Login 
----18/03/2009 6:30:52 AM    devbuild--LANSABUILDPC4(10.2.1.35)--SSL Disabled	Logout 
----18/03/2009 7:46:44 AM    anthony--AUPC(10.2.1.20)--SSL Disabled	Login 
----18/03/2009 8:15:43 AM    alick--ALICKVISTA(10.2.1.61)--SSL Disabled	Login 
----18/03/2009 8:18:26 AM    kevin--KEVINXP(10.2.1.125)--SSL Disabled	Login 
----18/03/2009 8:23:58 AM    admin--devsrv.syd.lansa.com.au(10.2.1.21)--SSL Disabled	Login 
----18/03/2009 8:24:00 AM    admin--devsrv.syd.lansa.com.au(10.2.1.21)--SSL Disabled	Logout 
----18/03/2009 8:25:45 AM    alick--ALICKVISTA(10.2.1.61)--SSL Disabled	Logout 
----18/03/2009 8:38:47 AM    rapin--JMDESK2(10.2.1.120)--SSL Disabled	Login 
----18/03/2009 8:43:16 AM    alick--ALICKVISTA(10.2.1.61)--SSL Disabled	Login 
----18/03/2009 9:06:25 AM    kelvin--kelvinpc.syd.lansa.com.au(10.2.0.149)--SSL Disabled	Login 
----18/03/2009 9:15:25 AM    alick--ALICKVISTA(10.2.1.61)--SSL Disabled	Logout 
----18/03/2009 9:36:22 AM    kevin--KEVINXP(10.2.1.125)--SSL Disabled	Login 
----18/03/2009 9:41:32 AM    anura--ANURAPC(10.2.1.91)--SSL Disabled	Login 
----18/03/2009 9:44:48 AM    donn--DONNLAPTOP(10.2.0.113)--SSL Disabled	Logout 
----18/03/2009 9:54:12 AM    phil--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Logout 
----18/03/2009 9:54:26 AM    devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Login 
----18/03/2009 9:54:56 AM    devbuild--LANSABUILDPC3(10.2.1.44)--SSL Disabled	Logout 
----18/03/2009 9:55:23 AM    andrews--ANDREWPC(10.2.1.26)--SSL Disabled	Rolling Back a transaction   at VaultServiceSQL.VaultSqlConn.RollbackTransaction()
   at VaultServiceAPILib.VaultServiceAPI.Login(VaultSqlConn conn, Boolean allowwebonly, String strLogin, String strDecryptedPassword, String strHostname, String strSessionID, String strDomain, String& strToken, VaultUser& vu, Int32& nValidUserAttempts)
   at VaultServiceAPILib.VaultServiceAPISystem.DoLogin(Boolean allowwebonlyusers, Boolean bAdminLogin, HttpSessionState hss, String strHostname, String strLogin, String strPassword, String strRMKey, String& strAuthToken, String strDomain)
   at VaultServiceAPILib.VaultServiceAPISystem.DoLogin(Boolean allowwebonlyusers, Boolean bAdminLogin, HttpSessionState hss, String strHostname, String strLogin, String strPassword, String strRMKey, String& strAuthToken)
   at VaultService.VaultService.Login(String strHostname, Boolean bUseFullFiles, String username, String strEncryptedPassword, String strRMKey, String& strAuthToken)
   at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
   at System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Web.Services.Protocols.LogicalMethodInfo.Invoke(Object target, Object[] values)
   at System.Web.Services.Protocols.WebServiceHandler.Invoke()
   at System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest()
   at System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   at System.Web.HttpApplication.ApplicationStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData)
   at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)
   at System.Web.HttpRuntime.ProcessRequestNoDemand(HttpWorkerRequest wr)
   at System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr ecb, Int32 iWRType)

[lbauer]

Nothing helpful in the log, other than the fact the server didn't see the login, which means a client-side or network problem. So it might have been a network glitch, slowdown, or the Vault server being temporarily inaccessible over the network for some reason.

If this keeps happening, we'd recommend enabling debug logging for the Vault server (in the Admin Web Client under Server Settings->Logging) so we can get more detail on what's happening with the server during the failure.


BTW this is just a transaction rollback message after a failed login. Scary looking, but benign:

Code: Select all

----17/03/2009 6:06:45 PM    ChiVinh--CHIVINHVISTA(10.2.1.9)--SSL Disabled   Rolling Back a transaction   at VaultServiceSQL.VaultSqlConn.RollbackTransaction()
   at VaultServiceAPILib.VaultServiceAPI.Login(VaultSqlConn conn, Boolean allowwebonly, String strLogin, String strDecryptedPassword, String strHostname, String strSessionID, String strDomain, String& strToken, VaultUser& vu, Int32& nValidUserAttempts)

[robe070]

thanks, that's the conclusion we had reached. Its also worked fine the last 2 nights.

Can the timeout setting be adjusted?

[lbauer]

You can adjust the client timeout in the registry, under HKEY_CURRENT_USER\Software\SourceGear\Client\Settings. Look for ServiceRequestTimeoutSeconds. The default is 7200 seconds; try increasing it.

[robe070]

7200 seconds is 2 hours. Is that really the login timeout? That's not going to make a difference to our issue as we know it definitely took less than 1 minute.

What effect does the registry entry LowLatencyNetworkConnection=FALSE have?

[lbauer]

This is a registry setting for the Vault option under Tools-> Options-> Network Settings->Use Expect 100-Continue Headers (for low latency networks)Use Expect: 100-Continue Headers.

From Client Help:

The inherent behavior of Vault, as an ASP.NET web application, is for the client to notify the server that a request is pending and for the server to OK the request with an HTTP Continue before the client can continue. By default this is enabled.

However, on low latency networks, such as satellite networks, this constant notification and verification can impede performance. If Use Expect 100-Continue Headers is disabled, the client sends the request and the transaction data at the same time, reducing the number of round trips to the server.

You can certainly try this. Let us know if it makes a difference.