Cruise Control and Draconet and Vault Version 3.0.7.2863

bryanandrews · Post by **bryanandrews** » Mon Nov 28, 2005 6:30 pm

When we have multiple projects accessing vault we almost always get periodic errors (i am assuming from multiple connections timing out).

I am not the Vault admin, but can you tell me if there are connection throttles on the server that would prevent an ip from making multple simultaneous connections to vault?

Error is always the same (unless it builds alone):

ThoughtWorks.CruiseControl.Core.CruiseControlException: Source control operation failed: . Process command: c:\program files\sourcegear\vault client\vault.exe history $/DesktopModules/Gallery -excludeactions label -rowlimit 0 -begindate 2005-11-28T18:58:25 -enddate 2005-11-28T19:28:26 -host source.modified.com -user modified -password modified -repository Mod_Gallery at ThoughtWorks.CruiseControl.Core.Sourcecontrol.ProcessSourceControl.Execute(ProcessInfo processInfo) at ThoughtWorks.CruiseControl.Core.Sourcecontrol.ProcessSourceControl.GetModifications(ProcessInfo info, DateTime from, DateTime to) at ThoughtWorks.CruiseControl.Core.Sourcecontrol.Vault.GetModifications(IIntegrationResult from, IIntegrationResult to) at ThoughtWorks.CruiseControl.Core.Sourcecontrol.QuietPeriod.GetModifications(ISourceControl sourceControl, IIntegrationResult lastBuild, IIntegrationResult thisBuild) at ThoughtWorks.CruiseControl.Core.IntegrationRunner.RunIntegration(BuildCondition buildCondition)

Thanks!

Post by **jclausius** » Mon Nov 28, 2005 9:52 pm

The Vault server has no such limitation, but IIS does. Is your Vault server installed on a Windows 2000 Pro or Windows XP based machine?

If so, those will Operating Systems will only allow for a limited number of connections.

bryanandrews · Post by **bryanandrews** » Mon Nov 28, 2005 10:41 pm

Yes its server...

bryanandrews · Post by **bryanandrews** » Mon Nov 28, 2005 10:44 pm

Here is another possibility... the build server is behind an ISAServer... not sure if this could have anything to do with it... just throwing it out there to see if it rings a bell.

Post by **jclausius** » Tue Nov 29, 2005 7:16 am

Sorry, I don't think I've seen anything related to this, unless your ISA server configuration limits the number of connection.

If you create a batch file with the Vault command line / start.cmd to start about 5-10 CLC's to do something, do you still see the same blocked behavior? What if you run this batch file / CLC launcher on the Vault server itself?

In case this is a Cruise Control issue, I'm going to ask someone more familiar w/ CC.Net to take a look as well.

ian_sg · Post by **ian_sg** » Tue Nov 29, 2005 8:41 am

Bryan,

If you set Cruise Control's logging to debug, we can get a better idea what's failing here. This can be accomplished by un-commenting this line:

Code: Select all

	<!--		<add name="CruiseControlSwitch" value="4" /> --> <!-- Debug: All messages -->

from ccservice.exe.config if you're running Cruise Control as a service or ccnet.exe.config if you're running the console server. If you can post the error and a few lines before and after it, we'll have a better idea.

bryanandrews · Post by **bryanandrews** » Tue Nov 29, 2005 9:02 am

11/29/2005 9:53:14 AM: [DotNetNuke.Forum:Info]: Checking for modifications in Vault from 11/29/2005 9:51:53 AM to 11/29/2005 9:53:14 AM
11/29/2005 9:53:14 AM: [DotNetNuke.Forum:Debug]: Vault History command: FileName: [c:\program files\sourcegear\vault client\vault.exe] -- Arguments: [history $/DesktopModules/Forum -excludeactions label -rowlimit 0 -begindate 2005-11-29T09:51:53 -enddate 2005-11-29T09:53:14 -host MODIFIED -user MODIFIEDUSER -password MODIFIED -repository Mod_Forums] -- WorkingDirectory: [C:\Program Files\CruiseControl.NET\server\DotNetNuke.Forum\WorkingDirectory] -- StandardInputContent: [] -- Timeout: [120000]
11/29/2005 9:53:14 AM: [DotNetNuke.Forum:Debug]: Executing process c:\program files\sourcegear\vault client\vault.exe history $/DesktopModules/Forum -excludeactions label -rowlimit 0 -begindate 2005-11-29T09:51:53 -enddate 2005-11-29T09:53:14 -host MODIFIED -user MODIFIEDUSER -password MODIFIED -repository Mod_Forums in C:\Program Files\CruiseControl.NET\server\DotNetNuke.Forum\WorkingDirectory
11/29/2005 9:53:14 AM: [DotNetNuke.Forum:Debug]: Attempting to start process [c:\program files\sourcegear\vault client\vault.exe] in working directory [C:\Program Files\CruiseControl.NET\server\DotNetNuke.Forum\WorkingDirectory] with arguments [history $/DesktopModules/Forum -excludeactions label -rowlimit 0 -begindate 2005-11-29T09:51:53 -enddate 2005-11-29T09:53:14 -host MODIFIED -user MODIFIEDUSER -password MODIFIED -repository Mod_Forums]

11/29/2005 9:53:52 AM: [CruiseControl Server:Debug]: <vault>
11/29/2005 9:53:52 AM: [CruiseControl Server:Debug]: <error>
11/29/2005 9:53:52 AM: [CruiseControl Server:Debug]: The connection to the server failed: server cannot be contacted or uses a protocol that is not supported by this client. 1109 : FailDBUpdate
11/29/2005 9:53:52 AM: [CruiseControl Server:Debug]: </error>
11/29/2005 9:53:52 AM: [CruiseControl Server:Debug]: <exception>
11/29/2005 9:53:52 AM: [CruiseControl Server:Debug]: System.Exception: The connection to the server failed: server cannot be contacted or uses a protocol that is not supported by this client. 1109 : FailDBUpdate
11/29/2005 9:53:52 AM: [CruiseControl Server:Debug]: at VaultCmdLineClient.VaultCmdLineClient.Login(Boolean bAllowAuto, Boolean bSaveSession)
11/29/2005 9:53:53 AM: [CruiseControl Server:Debug]: at VaultCmdLineClient.VaultCmdLineClient.ProcessCommandHistory(String strReposPath)
11/29/2005 9:53:53 AM: [CruiseControl Server:Debug]: at VaultCmdLineClient.VaultCmdLineClient.ProcessCommand(Args curArg)
11/29/2005 9:53:54 AM: [CruiseControl Server:Debug]: at VaultCmdLineClient.VaultCmdLineClient.Main(String[] args)
11/29/2005 9:53:54 AM: [CruiseControl Server:Debug]: </exception>
11/29/2005 9:53:54 AM: [CruiseControl Server:Debug]: <result success="no" />
11/29/2005 9:53:54 AM: [CruiseControl Server:Debug]: </vault>

ian_sg · Post by **ian_sg** » Tue Nov 29, 2005 10:16 am

Okay it looks to me like Jeff's original diagnosis was correct: something other than Vault is imposing a concurrent connection limit. The ISA server sounds like a good first thing to investigate. I'd also double-check the IIS configuration, as it is possible to impose a limit even on a server OS.

Post by **jclausius** » Tue Nov 29, 2005 10:18 am

You may also want to look at your Vault server log file to see if you can see the login attempt from the server's point of view.

bryanandrews · Post by **bryanandrews** » Tue Nov 29, 2005 11:04 am

I opened 5 command instances of vault.exe and did an update on different locations... and all works fine from the build server.

Very strange. I am seeing these issues from both draconet and ccnet - maybe it's a time out in the app?

ian_sg · Post by **ian_sg** » Tue Nov 29, 2005 11:23 am

Well Cruise Control logs a message when it kills a process due to timeout, which we're not seeing, so that's one less possibility.

When you tried the 5 simultaneous command-line clients, were they running roughly the same commands that CC.NET runs? Did the last one take longer than the 38 seconds we got from CC.NET?

bryanandrews · Post by **bryanandrews** » Wed Nov 30, 2005 10:54 am

No it all happens very quickly and i can reproduce it at startup every time.

ian_sg · Post by **ian_sg** » Wed Nov 30, 2005 11:09 am

With your command-line client test, did you run the same commands Cruise Control is running? With the logging at debug, you can see the exact commands, which would be a good test.

bryanandrews · Post by **bryanandrews** » Wed Nov 30, 2005 11:16 am

I am not the greatest at batch files so.... i created one for each and just execute them as quickly as i can. If you know a way that will allow me to execute them simultaneously i can try that. Yes they are the closest i could get to being the same (tell me if i am missing something):

"c:\program files\sourcegear\vault client\vault.exe" history $/DesktopModules/Gallery -excludeactions label
-rowlimit 0 -begindate 2005-11-29T09:51:53 -enddate 2005-11-29T09:53:14 -host source.MODIFIED.com -username
MODIFIED -password MODIFIED -repository "Mod_Gallery" -DESTPATH
C:\CCBuilds\DotNetNuke\DotNetNuke_3.2.0-Release\DesktopModules\Gallery

I have tried executing 5 of these with the HISTORY and GET commands.

bryanandrews · Post by **bryanandrews** » Thu Dec 01, 2005 5:07 pm

From the logs:

11/29/2005 4:41:00 PM MODIFIED--MODIFIED.org(xxx.xxx.xxx.xxx)--SSL Disabled Transaction (Process ID 82) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

11/29/2005 4:41:00 PM MODIFIED--MODIFIED.org(xxx.xxx.xxx.xxx)--SSL Disabled Login failed: FailDBUpdate

SourceGear Support

Cruise Control and Draconet and Vault Version 3.0.7.2863

Cruise Control and Draconet and Vault Version 3.0.7.2863

log entries - from one of the 6 vault processes that run