Cruise Control Integration Failure

jnapier · Post by **jnapier** » Mon Feb 13, 2006 10:53 am

For the past month I have been having intermittent problems when Cruise Control tries to check vault for modifications. I was hoping that the latest release would fix the problem but it did not. Usually the problem occurs when I am logged into the Cruise Control server with the same account that CC.Net service as running as, but this morning it occurred when nobody was logged into the server. Here is the error.

ThoughtWorks.CruiseControl.Core.CruiseControlException: Source control operation failed: . Process command: c:\program files\sourcegear\vault client\vault.exe history $/CIProjects/RemoteNet/RncCorporate/Working/Source -excludeactions label -rowlimit 0 -begindate 2006-02-10T18:51:29 -enddate 2006-02-13T06:44:39 -host rnc_offsite.remotenet.com -user builduser -password !builduser! -repository development -ssl at ThoughtWorks.CruiseControl.Core.Sourcecontrol.ProcessSourceControl.Execute(ProcessInfo processInfo) at ThoughtWorks.CruiseControl.Core.Sourcecontrol.ProcessSourceControl.GetModifications(ProcessInfo info, DateTime from, DateTime to) at ThoughtWorks.CruiseControl.Core.Sourcecontrol.Vault.GetModifications(IIntegrationResult from, IIntegrationResult to) at ThoughtWorks.CruiseControl.Core.Sourcecontrol.FilteredSourceControl.GetModifications(IIntegrationResult from, IIntegrationResult to) at ThoughtWorks.CruiseControl.Core.Sourcecontrol.QuietPeriod.GetModifications(ISourceControl sourceControl, IIntegrationResult lastBuild, IIntegrationResult thisBuild) at ThoughtWorks.CruiseControl.Core.IntegrationRunner.RunIntegration(BuildCondition buildCondition)

Generally as soon as one fails, the other projects will fail too. One by one they will all fail with the same error. All of them don’t always fail though. Sometimes just a few will fail with the same exception.

I have also been getting this error sometimes after a build has been launched because of modifications:

[<vault>
<error>
The process cannot access the file "C:\Documents and Settings\builduser\Local Settings\Application Data\SourceGear\Vault_1\Client\A8803CE4-FF15-4386-85E2-D4C23A437164\BuildUser\CacheMember_Repository" because it is being used by another process.
</error>
<exception>
System.IO.IOException: The process cannot access the file "C:\Documents and Settings\builduser\Local Settings\Application Data\SourceGear\Vault_1\Client\A8803CE4-FF15-4386-85E2-D4C23A437164\BuildUser\CacheMember_Repository" because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String str)
at System.IO.File.Delete(String path)
at VaultClientOperationsLib.CacheMember_Repository.DeserializeCacheMember(Object& o, DateTime& lastSyncTime)
at VaultClientOperationsLib.CacheMember.UpdateObjectFromDisk(Object& o)
at VaultClientOperationsLib.CacheMember_Repository.Update()
at VaultClientOperationsLib.CacheMember_Repository..ctor(String folder)
at VaultClientOperationsLib.TreeCache..ctor(Int32 repID, String username, String uniqueRepositoryID, String localStoreBasePath, ClientInstance ci)
at VaultClientOperationsLib.ClientInstance.SetActiveRepositoryID(Int32 id, String username, String uniqueRepositoryID, Boolean doRefresh, Boolean updateKnownChangesAll)
at VaultCmdLineClient.VaultCmdLineClient.Login(Boolean bAllowAuto, Boolean bSaveSession)
at VaultCmdLineClient.VaultCmdLineClient.ProcessCommandCheckout(ArrayList strItemArray)
at VaultCmdLineClient.VaultCmdLineClient.ProcessCommand(Args curArg)
at VaultCmdLineClient.VaultCmdLineClient.Main(String[] args)
</exception>
<result success="no" />
</vault>

Please let me know if I can provide any more information to help resolve this issue.

ian_sg · Post by **ian_sg** » Mon Feb 13, 2006 11:56 am

The error indicates there are multiple processes trying to access the files. This is frequenty caused by virus scanning software on the machine. Is that a possibility here?

jnapier · Post by **jnapier** » Mon Feb 13, 2006 12:23 pm

Our network admin has just informed me that there is no anti-virus software on that server.

ian_sg · Post by **ian_sg** » Mon Feb 13, 2006 12:31 pm

Okay after reading your initial post more carefully I see there are two different issues here. Regarding the first error, the Vault Server Log should tell us why the command failed.

Regarding the second error, it looks like multiple Vault clients are trying to retrieve the same source. Do you have multiple CC.NET projects set up that share some source? (Note that I'm not saying this is a bad thing, if you are, I'm just trying to narrow the possibilities.)

jnapier · Post by **jnapier** » Mon Feb 13, 2006 12:46 pm

For the 1st problem, this is the only exception I am seeing in the log. This error has occurred several times for the user that is running CC.Net service.

----2/13/2006 9:37:54 AM --BUILDSRV1(10.0.1.161)--SSL Enabled System.NullReferenceException: Object reference not set to an instance of an object.
at VaultService.VaultServiceSystem.ValidateToken(String strToken)
at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e) at VaultService.VaultServiceSystem.ValidateToken(String strToken)
at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e)

For the second problem, Yes, I have 2 vs.net solutions that have one project included in both solutions. The build for each solution does not get the source though. the project that is shared by both solutions has its own solution and it is on its own CC.Net build cycle. So actually vault is called only once to get the source for the project that is shared.

Please let me know if I ned to provide any more information. Should I put my Vault server into debug logging mode?

jnapier · Post by **jnapier** » Mon Feb 13, 2006 1:05 pm

These errors both just happened so they might be related. A build was kicked off for one project. It was running along fine and then another project tried to check for modifications. It failed with the 1st error I have reported. Then when the project that was building went to check in the assemblyinfo files that it checked out, it failed with the second problem I have reported.

I checked the Vault server log and there were no errors logged for either event which makes me think this is strictly a client thing.

These projects are not related in any way.

I forced a build of both projects and then they built fine.

ian_sg · Post by **ian_sg** » Mon Feb 13, 2006 1:06 pm

What you're seeing in the server log doesn't appear to be related to the error, as it's a failure when during a history, not when downloading changes. It wouldn't hurt to put the server log into debug mode, and see if that reveals more. My guess at this point is that the history is simply timing out if you've got a number of projects hitting at once. We've submitted a number of improvements to the CruiseControl.NET project, and history timeouts is one of the issues addressed. Unfortunately they have not yet released those changes, so that's no help to you yet, but there's hope in a forthcoming version of CruiseControl.NET. Increasing the timeout in your ccnet.config might help, in the mean time.

With respect to the file in use error, it's possible that multiple vault client processes are stepping on eachother's toes with respect to this file. This shouldn't be possible, but that's what the log points to, so I'm going to need to try to reproduce the problem here. Can you tell me how many projects you have setup in CC.NET that use this repository (regardless of what projects within the repository they use)? Also, how often do you see this? (Something like hourly, daily, weekly, monthly is probably detailed enough for now.)

Thanks.

jnapier · Post by **jnapier** » Mon Feb 13, 2006 1:16 pm

I have 7 CC.Net projects configured. I just started seeing the second error last week. Granted, I did add a few more projects to CC.Net last week. I'm not really seeing the error all that much. It happened 2 or 3 times last week and then once today. I will update you as I see things progress throughout the week.

Thanks for the help.

ian_sg · Post by **ian_sg** » Mon Feb 13, 2006 1:16 pm

It would also be helpful if you could enable client-side logging for the command-line client on your build machine. Note that this log gets big fast, so you might need to babysit it a bit until you see these errors again, at which point you'll want to turn it back off. I'm most interested in the client-side log for the file in use error.

Thanks,

jnapier · Post by **jnapier** » Mon Feb 13, 2006 1:24 pm

Ok, I will enable the client side logging. Are you sure the 1st problem is a timeout issue? The cc.net build logs show running times of the failed builds at 4 to 9 seconds. That doesnt seem like much of a timeout to me.

Hopefully the logging will provide some further insight.

ian_sg · Post by **ian_sg** » Mon Feb 13, 2006 1:29 pm

We're certainly not timing out in less than ten seconds, so it must be something else. So no, I'm not sure, it was just a best guess.

The additional logging ought to help.

jnapier · Post by **jnapier** » Mon Feb 13, 2006 2:38 pm

I have the log file. Its 2.6 mb though. Where should I send it?

There should be one instance of the 1st problem and 2 instances of the second problem. the second problem started happening like mad. It really only happened a few times with Vault 3.1.6. Since the install of Vault 3.1.7 its been non stop. I had to move all of my modification checks from 1 to 5 minutes. This seemed to help things out some, but not really. If one build is trying to perform an operation while another project is checking for modifications then Im screwed.

Here are the times that cc.net reported the builds failing. Hopefully this will make it easier for you to locate the errors.

problem 1 - 11:39:47 (Project name: Benefits Online)

problem 2; 1st occurrence - 11:52:27 (Project Name : Ibis Project)
problem 2; 2nd occurrence - 11:57:16 (Project Name: RemoteNet Framework)

Let me know if you need more info.

ian_sg · Post by **ian_sg** » Mon Feb 13, 2006 2:40 pm

You can either email me the file (using the button below) or ftp it to ftp.sourcegear.com/incoming.

ian_sg · Post by **ian_sg** » Mon Feb 13, 2006 2:41 pm

Actually the forum emailer doesn't allow attachements. So use the ftp site, if you would.

jnapier · Post by **jnapier** » Mon Feb 13, 2006 2:45 pm

Ok. Its there. VaultClc.txt