Vault plugin for CCNet

[LennyT]

I have just updated our CCNet configuration to use the vault plugin, and am having trouble getting this running reliably.
The main problem is the memory usage the plugin causes. Without the plugin the CCNet service uses ~120Mb of memory, with the service it jumps up to 1.5 Gb - just on start up. The service then fails fairly quickly after this when it starts to do anything. I would like to change over to the plugin, as there are apparently performance improvements but clearly cant while the plugin is killing CCNet, so any help on this issue will be greatly appreciated.

We are using

Vault client and server version: 4.1.3 (18336)
CruiseControl.NET Server 1.4.0.3400 -- .NET Continuous Integration Server
Copyright © 2008 ThoughtWorks Inc. All Rights Reserved.
.NET Runtime Version: 2.0.50727.1433 Image Runtime Version: v2.0.50727
OS Version: Microsoft Windows NT 5.1.2600 Service Pack 2 Server locale: en-AU

Getting this sort of error (not surprisingly):

2008-11-10 09:26:10,885 [692:ERROR] Unable to set active repository: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at VaultLib.VaultDateTime.get_Now()
at VaultClientOperationsLib.VaultClientFile..ctor()
at VaultClientOperationsLib.VaultClientFile.Clone(VaultClientFolder parent)
at VaultClientOperationsLib.VaultClientFolder.Clone(VaultClientFolder parent)
at VaultClientOperationsLib.VaultClientFolder.Clone(VaultClientFolder parent)
at VaultClientOperationsLib.VaultClientFolder.Clone(VaultClientFolder parent)
at VaultClientOperationsLib.VaultClientFolder.Clone(VaultClientFolder parent)
at VaultClientOperationsLib.VaultClientFolder.Clone(VaultClientFolder parent)
at VaultClientOperationsLib.VaultClientFolder.Clone(VaultClientFolder parent)
at VaultClientOperationsLib.VaultClientFolder.Clone(VaultClientFolder parent)
at VaultClientOperationsLib.VaultClientFolder.Clone(VaultClientFolder parent)
at VaultClientOperationsLib.VaultClientFolder.Clone(VaultClientFolder parent)
at VaultClientOperationsLib.VaultClientFolder.Clone(VaultClientFolder parent)
at VaultClientOperationsLib.VaultClientFolder.Clone(VaultClientFolder parent)
at VaultClientOperationsLib.PendingChangeRepository..ctor(VaultClientRepository repository)
at VaultClientOperationsLib.TreeCache.GetPendingChangeRepository()
at VaultClientOperationsLib.TreeCache.get_Repository()
at VaultClientOperationsLib.ClientInstance.NotifyActiveRepositoryChanged()
at VaultClientOperationsLib.ClientInstance.SetActiveRepositoryID(Int32 id, String username, String uniqueRepositoryID, Boolean doRefresh, Boolean updateKnownChangesAll)
at ThoughtWorks.CruiseControl.Core.Sourcecontrol.FortressClient.Login(String vaultHost, String vaultRepositoryName, String vaultUser, String vaultPassword, Boolean useSSL, String proxyHost, String proxyDomain, Int32 proxyPort, String proxyUser, String proxyPassword, String& url)

[shannon]

There are two variables that are fighting.

1. Server Performance
2. CC.Net memory

The old CC.Net implementation used less memory, but was more taxing on the server, since it would relogin with the CLC on ever check.

The new implementation doesn't log in every time, so it keeps the repository cache in memory between checks. This is much less taxing on the server, but will consume memory using the following equation:

(tree size * number of CC.Net projects)

There are two options that would help you:

1. Reduce the tree size by deleting (but there's no need to obliterate) any unused branches.

2. Reduce the number of CC.Net projects that are being monitored.

[LennyT]

Well this is not very helpful as there are no unused branches to get rid of, and we are about greatly increase the number of projects being monitored.

It would be possible to reduce the number of projects if it were possible for vault to distinguish the CCNet trigger, that way we could combine cleanCopy="false" for integration builds and cleanCopy="true" for overnight builds.

Cheers
Marc Thompson

[shannon]

We are investigating ways to improve the memory usage of the plugin for a future release, but, for now, I'm sorry to say those are the only options I can give you.

F:13807

[avonwyss]

Just out of interest, does it respect cloaked folders of the Vault user used for CCNet in order to reduce the tree size remembered?

[shannon]

Cloaking won't reduce the memory usage, unfortunately, just the number of items that are fetched during a get.

[jwboer]

We had the same issue: large project tree, and growing. Equally growing memory consumption.
We solved this issue by using the builtin vault support from ccnet instead of using the Vault plugin.

[jaf1348]

I agree with jwboer. CCNet's built-in Vault support is the only way to go unless you have a small source tree or few projects; Sourcegear's CCNet plug-in isn't an option. Unfortunately, it took me a very long time to figure this out.

While I can understand the desire to improve performance, the fact that it brings CCNet to its knees is hardly an improvement. Retaining the repository cache in memory should at least be configurable (preferably on a per project basis) if this behavior is kept.

[davenovak]

After recently branching, I too am seeing this same issue -- memory going through the roof when using the Vault plug-in for CCNet.

What I don't get is why is memory calculated as tree size * number of CC.Net projects when one of the options within CCNet is to specify the SCC Folder name (which is never $/ in my configuration). Worst case I think this should be branch size * number of CC.Net projects (where branch is the folder name you have specified within your CCNet configuration file). Even so, it seems like a faulty design to require that much memory in the first place. Our Vault database just isn't that big (in my humble opinion). Something's just not right here and I hope that SourceGear will fix it soon (as more and more people nowadays are using CCNet).

What's worse is that if I switch back to using just command-line Vault, I randomly run into "The connection to the server failed" errors connecting to the Vault server. SourceGear has appareartly confirmed that this is a race condition when the same Vault user logs in at the same time, which is something that happens frequently in CCNet (see http://support.sourcegear.com/viewtopic.php?f=5&t=11612). I know SourceGear is working on a fix that will serialize logins for the CCNet vault plug-in, though I am not aware they are doing anything to address the memory problem cited here. And then there's the problem that inclusion / exclusion filters are ignored in the plug-in (see http://support.sourcegear.com/viewtopic.php?f=5&t=11585).

Best to stick with the built-in command-line Vault for CCNet, though you do still need to watch out for those "The connection to the server failed" errors.

SourceGear -- Please fix these issues!

[jeremy_sg]

Dave,

The build that I gave you last week already has a fix for the memory-usage problem. The 5.0 cc.net plugin will have the fix for this, and if anyone has the memory-usage problem in 4.1, email support at sourcegear dot com, and we will send a 4.1 build with the fix.

[davenovak]

Jeremy --

Happy to hear that you are also working on the memory problem. I didn’t realize that. I should let you know, however, that the memory problem apparently still exists in the updated plug-in that I just downloaded today. I'm still seeing a large memory allocation (up to 1.8 GB) when CCNet first starts up and loads its projects followed by numerous System.OutOfMemoryException errors. Having said that, it's only fair to also report that the memory drops back down to a reasonable level after the initial per-project login.

One potential workaround I've seen for the memory problem is organizing the CCNet projects into logical queues (i.e., using the queue="<queue_name>" attribute in CCNet project node). I was able to organize my 60 CCNet projects into 15 logical queues. And since these are in queues, CCNet will automatically serialize the startup of projects contained within each queue (as opposed to putting each project in its own queue, which is the default behavior). This definitely seems to help although I'd like to not be fighting this issue every time I branch and my tree grows. The tipping point right now for me is somewhere around 20 - 25 active queues. The size of my main repository (as reported by Admin Statistics) is Folders: 15495, Files: 167721, Tree Size: 5.27GB, Database Size: 3.25GB

I will follow up with you later this afternoon with a full report on the results of my testing.

Thanks,

--Dave

[ThomasC]

I've run into the same problem with the vaultplugin. (4.1.4). We have 14 projects, 9097 folders and a db that is 10 GB in size. The ccservice was eating up 1.8 GB of memory and dragging down our build server. I swapped everything to the CC built-in vault client and forced all the projects into the same CC queue which forced the system to build them sequentially (if they were forced to build at once).

Now the ccservice takes up about 40 MB of memory with the vault client taking another 63 MB. I seem to remember that the reason I switched to the vaultplugin was to deal with intermittent lost connections, but that was before I moved to cc 1.4 with the queues. I'm hoping that the built-in client combined with the new queuing will fix the problem.

[ThomasC]

well, I spoke too soon. I'm now getting errors along the lines of:

<exception>System.Exception: The connection to the server failed: server cannot be contacted or uses a protocol that is not supported by this client. The network path was not found.
---> System.Security.Cryptography.CryptographicException: The network path was not found.

at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
at System.Security.Cryptography.RSACryptoServiceProvider.get_PersistKeyInCsp()
at System.Security.Cryptography.RSACryptoServiceProvider.set_PersistKeyInCsp(Boolean value)
at VaultLib.VaultUserCrypt.EncryptPassword(String strUnencryptedPassword, Byte[] publicKey, Byte[] exponent, String& strEncryptedPassword, String& strRMKey)
at VaultClientNetLib.VaultConnection.Login(String strURLBase, String strUserLogin, String strPassword)
at VaultClientOperationsLib.ClientInstance.Login(String urlbase, String username, String password)
at VaultClientIntegrationLib.ServerOperations.Login(AccessLevelType altCommand, Boolean bAllowAuto, Boolean bSaveSession)
--- End of inner exception stack trace ---
at VaultClientIntegrationLib.ServerOperations.Login(AccessLevelType altCommand, Boolean bAllowAuto, Boolean bSaveSession)
at VaultClientIntegrationLib.ServerOperations.Login()
at VaultCmdLineClient.VaultCmdLineClient.ProcessCommand(Args curArg)
at VaultCmdLineClient.VaultCmdLineClient.Main(String[] args)</exception>
</error>

This project has about eight project library references to it. I can connect via a browser to the source site and other sites seem to build. I hear rumor that there was a new version of the vaultplugin for 4.x which fixes the memory hog issue. Is that correct? How can I get a copy?

[ThomasC]

I think I found the issue (well, it works now so I'm guessing this was the problem). The problem was that the system was trying to apply the same label twice to a given project. Unfortunately, the error told me nothing about the actual problem. In this case, it was a library project used across multiple applications each using a dateLabeller. I changed the labeller to one with a prefix and that corrected the issue.

[lbauer]

Thanks for the update. : )