CCNET not retrieving latest version

[humphry]

We are using CCNET for continuous integration together with Vault for source control.
CCNET : 1.8.2
Vault : Standard 6.0.1

The plugin from Sourcegear is used in CCNET (ccnet.vault.plugin.dll).

We experience the following problem. After CCNET is running some time the plugin starts to generate errors (in log and eventviewer) that a connection to Vault failed three times:

=========================================
[ProjectA:INFO] Project: 'ProjectA' is first in queue: 'QueueA' and shall start integration.
[ProjectA:DEBUG] In VaultProVaultSourceControl.GetModifications()
[ProjectA:ERROR] Vault Standard Login Failed Once.:System.Exception: Internal error: An attempt was made to connect to a server when already connected to a server.
[ProjectA:WARN] Vault Standard Login trying again (2).
[ProjectA:ERROR] Vault Standard Login Failed Twice.:System.Exception: Internal error: An attempt was made to connect to a server when already connected to a server.
[ProjectA:WARN] Vault Standard Login trying again (3).
[ProjectA:ERROR] Vault Standard Login Failed Three times, giving up.:System.Exception: Internal error: An attempt was made to connect to a server when already connected to a server.
[ProjectA:INFO] No modifications detected.
=========================================

This error is logged for all projects in cruisecontrol which results in about 4 errors in eventviewer per second!

The worst part is that cruisecontrol builds with old source because the vault plugin says there are no modifications (but in fact it failed to connect).

The code for this (in the plugin) seems a little bit strange:

=========================================
Modification[] array = new Modification[0];
...
if (!this._client.LoggedIn && !this._client.Login(this.Host, this.Repository, this.Username, this.Password, this.SSL, this.ProxyServer, this.ProxyDomain, this.ProxyPort, this.ProxyUser, this.ProxyPassword, ref this._url))
{
return array;
}
=========================================

When login fails, just return no modifications!

So I think the plugin is incorrect (it should not return 'no modifications') when it is not able to login.
And can you explain where the exception comes from in the first place?

Thanks

[humphry]

It just happened again (plugin is spamming the eventviewer with errors).
The first error is however the following:

2012-11-28 11:53:43,908 [ProjectB] ERROR CruiseControl.NET [(null)] - An exception was encountered during the get latest operation. Exception: VaultServiceAPILib.VaultSoapException: 1011 : FailInvalidSessionToken
at VaultService.VaultServiceSystem.ValidateToken(VaultAuth va)
at VaultService.VaultService.BeginDownloadFiles(Int32 nRepID, VaultRequestGetFile[]& requests, String& strDownloadID)

So, there is an exception while retrieving the source.
The build process is started anyway (with partial source)!

After this error the other exception are showing up (maybe they are related?):

2012-11-28 11:53:45,452 [ProjectC] ERROR CruiseControl.NET [(null)] - VaultProVaultSourceControl.GetSource() failed to retrieve the files for $/Trunk/ProjectC/:VaultClientOperationsLib.GetLatestVersionFailedException: Exception of type 'VaultClientOperationsLib.GetLatestVersionFailedException' was thrown.
at VaultClientOperationsLib.ClientInstance.ProcessGetFileRequests(GetFileInfo[] infos, MakeWritableType makeWritable, SetFileTimeType setFileTime, MergeType merge, Boolean updateHiddenFilesOnly, String ancestorFullPath, Boolean flat, String ancestorDiskPath, OverwritePrompt PromptData, Boolean isLabelGet, String currentPathToLabelItem, Int64 labelID, Boolean isRetry, Boolean isGetByDisplayVersion)
at VaultClientOperationsLib.ClientInstance.GetByDisplayVersionToNonWorkingFolder(VaultClientFile[] files, MakeWritableType makeWritable, SetFileTimeType setFileTime, String ancestorFullPath, String ancestorDiskPath, OverwritePrompt PromptData)
at VaultClientOperationsLib.ClientInstance.GetByDisplayVersionToNonWorkingFolder(VaultClientFolder folder, Boolean recursive, MakeWritableType makeWritable, SetFileTimeType setFileTime, String ancestorDiskPath, OverwritePrompt PromptData)
at ThoughtWorks.CruiseControl.Core.Sourcecontrol.VaultProClient.GetVersion(String repositoryFolderPath, String workingDir, Int64 folderVersion, Boolean useWorkingDir, Boolean deleteItemsThatAreNotInTheRepository)
at ThoughtWorks.CruiseControl.Core.Sourcecontrol.VaultProVaultSourceControl.GetSource(IIntegrationResult result)

[lbauer]

When did this last work properly? Were you on an earlier version of Vault? Earlier version of Cruise Control when this worked?

We'd like to see a copy of the Vault Server log. It's called sgvault.log and is in %windir%\temp\sgvault on the server machine. Zip it up and email me at support at sourcegear.com, ATTN: Linda.

Also, what is the event viewer error?

Please include a link to this forum post.

[humphry]

We suspect the problem might be that we have two vault services running on the same database.
Seems like the service is restarting sometimes (we also get session invalid messages in vault client).
Multiple times the following can be found in the Vault log:

----11/29/2012 1:21:19 PM sgvaultsystem--()--
System Started
Version 6.0.1.598
Cache Level = 1
DataBase Buffer Size (KB) = 256
LogFile Path = C:\Windows\Temp\sgvault
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = True
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 10080
SGVault Working Directory = C:\Windows\Temp
Identity = IIS APPPOOL\VaultAppPool

We are running two services because we are migrating to a new server (so this is temporarily).
Might this be the cause of these issues?

Nevertheless, I would expect the plugin to quit instead of continuing when exceptions during login occur.

[lbauer]

>>We are running two services because we are migrating to a new server (so this is temporarily).>>

Although using one database for two Vault servers can technically work, it's not a supported configuration. We haven't done much testing with this, so it could give you unexpected results.

It's possible this is contributing to the problem. Did this start when you installed the second Vault server?

[humphry]

I think it started recently when we started running two services, but I'm not sure.

[humphry]

In a couple of days we will shut down one of the services. I'll let you know if that solves the errors.

What can you say about the fact that the plugin continues in case of a (login) exceptions?

[grandavenuesoftware]

What can you say about the fact that the plugin continues in case of a (login) exceptions?

We are seeing the same problem as humphry, and we're not running two services. It seems to happen intermittently with no identifiable root cause, and once it starts, CruiseControl.NET will just silently fail to download source changes until the CC.NET service is restarted.

We need a way to prevent this from happening, but more importantly we need a way to catch the problem when it happens and force the build to fail so that we don't end up with builds reporting test results against old source.

We are running the same versions as humphry (SourceGear Standard 6.0.1 and CruiseControl.NET 1.8.2).

Here is the error in the CC log file:

Code: Select all

2012-12-19 15:49:41,458 [GasSolutionDebug_vBuild2008R2:DEBUG] In VaultProVaultSourceControl.GetModifications()
2012-12-19 15:49:41,474 [20:DEBUG]  [] has been granted ViewProject permission on 'GasSolutionDebug_vBuild2008R2'
2012-12-19 15:49:41,520 [GasSolutionDebug_vBuild2008R2:ERROR] Vault Standard Login Failed Once.:System.Exception: Internal error: An attempt was made to connect to a server when already connected to a server.
   at VaultClientNetLib.VaultConnection.Login(String strURLBase, String strUserLogin, String strPassword)
   at VaultClientOperationsLib.ClientInstance.Login(String urlbase, String username, String password)
   at ThoughtWorks.CruiseControl.Core.Sourcecontrol.VaultProClient.Login(String vaultHost, String vaultRepositoryName, String vaultUser, String vaultPassword, Boolean useSSL, String proxyHost, String proxyDomain, Int32 proxyPort, String proxyUser, String proxyPassword, String& url)
2012-12-19 15:49:41,536 [GasSolutionDebug_vBuild2008R2:WARN] Vault Standard Login trying again (2).
2012-12-19 15:49:41,630 [GasSolutionDebug_vBuild2008R2:ERROR] Vault Standard Login Failed Twice.:System.Exception: Internal error: An attempt was made to connect to a server when already connected to a server.
   at VaultClientNetLib.VaultConnection.Login(String strURLBase, String strUserLogin, String strPassword)
   at VaultClientOperationsLib.ClientInstance.Login(String urlbase, String username, String password)
   at ThoughtWorks.CruiseControl.Core.Sourcecontrol.VaultProClient.Login(String vaultHost, String vaultRepositoryName, String vaultUser, String vaultPassword, Boolean useSSL, String proxyHost, String proxyDomain, Int32 proxyPort, String proxyUser, String proxyPassword, String& url)
2012-12-19 15:49:41,630 [GasSolutionDebug_vBuild2008R2:WARN] Vault Standard Login trying again (3).
2012-12-19 15:49:41,833 [GasSolutionDebug_vBuild2008R2:ERROR] Vault Standard Login Failed Three times, giving up.:System.Exception: Internal error: An attempt was made to connect to a server when already connected to a server.
   at VaultClientNetLib.VaultConnection.Login(String strURLBase, String strUserLogin, String strPassword)
   at VaultClientOperationsLib.ClientInstance.Login(String urlbase, String username, String password)
   at ThoughtWorks.CruiseControl.Core.Sourcecontrol.VaultProClient.Login(String vaultHost, String vaultRepositoryName, String vaultUser, String vaultPassword, Boolean useSSL, String proxyHost, String proxyDomain, Int32 proxyPort, String proxyUser, String proxyPassword, String& url)
2012-12-19 15:49:41,833 [GasSolutionDebug_vBuild2008R2:INFO] No modifications detected.

Thanks ahead of time.

[humphry]

FYI: the problem seems to be gone when we shut down one of the services.
Running without errors now for about a week. Before the error occurred a couple of times a day.

Still very curious why Vault is continuouing in case of errors (and certainly when this might be caused by other situations like grandavenuesoftware).

[grandavenuesoftware]

FYI: the problem seems to be gone when we shut down one of the services.

Thanks. We verified that we're only running a single service, so that's not the problem in our case. Good to know that it's possible to get rid of the problem. Again, like you said, the priority for us is to catch the error so that we can make the build fail, instead of getting a false positive from our tests running against a previous version of the source.

[humphry]

Can somebody from SourceGear respond?

[lbauer]

We'll need to do more troubleshooting to determing what the problem might be when running two services.

Email me at support at sourcegear.com, ATTN: Linda.

Please include a link to this forum post.

[humphry]

I'm more interested in the fact that the plugin continues in case of exceptions.
We don't need to run two services, so that is irrelevant.

[grandavenuesoftware]

I'm more interested in the fact that the plugin continues in case of exceptions.
We don't need to run two services, so that is irrelevant.

Ditto.

[lbauer]

CCNET communicates with the Vault Plugin and the plugin uses a history command to check the latest version in the repository. If the version has been updated, CCNET triggers a build. Then the plugin does a GET.

Currently, if the GET fails, we have no way to stop CCNET from continuing with the build.

I've logged work item 16696 to see if there's a way we can change this behavior. We'll also take a look at changing the behavior for a failed login.