Vault Pro & Teamcity Server Hang

[PeteM]

Hello,

We currently have Teamcity 7.1 & Vault Pro 6.0 integration. After Upgrading both to these versions we started to see a TeamCity Server hang with Out Of Memery errors. I logged a call with Teamcity & supplied them with several memory dumps. ( This is the call http://youtrack.jetbrains.com/issue/TW-23565 ) They have said that it appears to be the Java CLC that is used as part of the integration that is causing the issue.

Yesterday I noticed there was a version 6.0.1 available and installed this version of the Java CLC but we are still experiencing the problem.

Can I send some memory dumps to you to help solve this issue?? Do you have any advice regarding this problem?

Many thanks,

Pete.

[Beth]

We haven't tried integrating with Teamcity, and it's not a supported product. You can send information, but there's no guarantee that we'll have a solution.

There needs to be some background information about this situation.

Try some of the source management actions with either the Java CLC alone or a Vault GUI client alone. Do they behave any differently than when you have TeamCity and Vault integrated?

Do any other machines have problems performing similar actions as the machine that is hanging?

At what actions does it seem to hang?

What is the OS of the TeamCity server?

[PeteM]

Beth, thanks for the reply, here is some more detail.

We had Teamcity (TC) workinf well with version 5 of Vault Pro. No memory issues until the upgrade to version 6.

When I began seeing the Out of memory issue in TC, I began monitoring the problem there as it appeared the upgrade to 7.1 was the issue. It just so happened the I upgraded to Vault 6 at the same time.

So after issuing TC with the memory dumps, the have said the issue is with the Java CLC. Their Vault API did not change between the prior and upgraded version. I believe this to be a wrapper for the Java CLC.

When I was monitoring the memory usage, it crept up during the day at a steady rate, even when it was not building projects. After they said it was vault, it kinda made sense as TC checks for source changes every 60 seconds. This is the default. So the leak maybe small, but at 1440 time a day, it will hurt!

So for the time being, I have changed this period and given the TC Service some more memory, but the server still needs a restart every day to free up the allocated memory.

To answer your questions :-

Try some of the source management actions with either the Java CLC alone or a Vault GUI client alone. Do they behave any differently than when you have TeamCity and Vault integrated?

We will have to find out how the Java CLC works and write something to make calls several thousand times in order to get a true reflection.

Do any other machines have problems performing similar actions as the machine that is hanging?

No, we havent seen this anywhere else, but it's also unlikely to make the same volume of calls.

At what actions does it seem to hang?

Difficult to say as it all works fine until it runs out of memory, so not a single action that is causing the crash.

What is the OS of the TeamCity server?

The OS is Server 2008 R2

Hope this helps,

Pete.

[Beth]

Is the Vault server and the Teamcity server on the same machine?

Can you check your Vault server log on the Vault server at %windir%\temp\sgvault\sgvault.log? Are any errors showing up there?

[PeteM]

No, the Vault & TeamCity servers are on different machines.

No errors in the log, just a whole lot of login / logout.

[lbauer]

We're aware that Team City created a plugin for Vault a year or two ago. They contacted our developers. with a few questions. However, we've never tested Vault with Team City and are not familiar with its design.

Do you know whether TeamCity is using our Java CLC or our Java API?

For Vault 6, we updated our third party .NET to Java libraries to run with .NET 2.0. Initial testing showed that those Java commands still worked. It could be that the memory allocation tree is handled differently by the .NET to Java bridge for .NET 2.0 and that is causing TeamCity to run out of memory.

Do you know the memory configuration allocated by Team City during the run of the CLC?

Since you're using a Windows server, is it possible to configure TeamCity to run the .NET (Windows) version of the Vault Command Line Client instead of the Java CLC?

[PeteM]

Linda,

Thanks for the reply.

I believe its using the CLC according to thier install instructions here http://confluence.jetbrains.net/display/TW/Vault

In terms of allocating memory, we can change the settings for the TeamCity service, but not what is allocated to the calls to Vault.

I do not believe its possible to use the windows version rather than the Java one.

I still have the memory dumps that I can send if it helps.

Thanks again,

Peter.

[lbauer]

Go ahead and send the memory dumps. We can't provide support for third-party plugins like TeamCity, but we can at least take a look to see if there's a Vault issue that's easy to fix.

It would also be a good idea for Team City to test their plugin with Vault 6 to make sure any incompatibilties are resolved.

[lbauer]

Send the dumps zipped up to support at sourcegear.com, Attn: Linda. Please include a link to this forum post.

[lbauer]

It may be possible to change the memory allocation used by the Vault Java Command Line Client.

There are two places where memory can be set.

Using notepad, open the "vault" file in the Java CLC directory, and look for this text: -Xmx512M
This is the memory allocation, in this case 512 MB. Try changing it to something larger, like -Xmx1024M

There's a similar setting in the "vault.cmd" file (open in notepad) : java -Xmx512M

I would suggest changing both these settings and see if that helps with the TeamCity memory issues.

[PeteM]

Hello Linda,

Apologies for the delay, I have been on leave.

While I was away, the guys at Teamcity said this to my earlier post http://youtrack.jetbrains.com/issue/TW-23565

"In a while we'll install Vault 6 on out test server and investigate the issue. We'll contact guys from SourceGear if necessary."

So I guess they have taken this issue on and will be in contact with you if required.

Many thanks for your assistance,

Take care,

Pete.

[bjin]

Hi,

I've just updated to Vault 6 and use TeamCity 7.1.1 and I also suffer from these memory leaks.
The only change made to TeamCity installation is the change of Java CLC (Vault Pro 6.0.1) .

Our TC server runs for a just couple of hours before grinding to a halt. I've turned up the JVM memory to 1 Gb without any luck (-Xmx1024M). This is getting very serious indeed and I keep my fingers crossed you guys at SourceGear can resolve this one asap.

Our TC server runs on Win2K8 R2 with 6 Gb of RAM.

Regards
Bjarne

[lbauer]

Team City did contact us more than two weeks ago, but we haven't heard any more since then. You may want to follow up with them, since this involves their product.

[bjin]

I can happily confirm that this problem is resolved in JavaCLC 6.0.1 30613

[lbauer]

Thanks for the update. That's good news.

One of our developers worked closely with Team City to determine what changes needed to be made to the Java CLC. Your copy is a pre-release build, and the final fix will be in Vault 6.1, due out later this month.