CruiseControl 1.3 stopped working with Vault Client 4.1.2

[gsmalter]

Ever since upgrading our server and client to 4.1.2, CruiseControl does not work. After hours of debugging, it seems like the problem stems from a combination of the new Vault Client and the Service version of CruiseControl (the desktop app version of CruiseControl works okay).

We have tried both the new and old Vault plugin.

Here is what we found:

New Vault Plugin & CruiseControl as an app: Works
Old Vault Plugin & CruiseControl as an app: Works

New Vault Plugin & CruiseControl as a service: Service stops after a few minutes without explanation. Nothing meaningful in Windows Event Log. No errors in ccnet.log file.

Old Vault Plugin & CruiseControl as a service: Service continues to run, but Exception repeatedly shows up in the ccnet.log file:

ThoughtWorks.CruiseControl.Core.CruiseControlException: Source control operation failed:
Unhandled Exception: System.Security.Cryptography.CryptographicException: Keyset does not exist

We have no idea what to do other than downgrade the Vault client to 4.1.0. It's almost as if the Vault client has code in it that looks like this:

if( !Environment.UserInteractive )
throw new StopWorkingException();

I know it doesn't, but the symptoms fit perfectly.

Any other causes of this problem are welcome. Thanks.

[gsmalter]

I should add that the Cruise Control service is set up to run as the Administrator user, the same user that is running the app version of Cruise Control, which works.

[ian_sg]

We've had one other report of this issue and are investigating. Running ccnet from the console is the easiest workaround until we have more information.

[gsmalter]

We downgraded to the 4.1.0 client and the problem remains. Some assumption we made was wrong. So, this brings a few additional things into the mix:

Simultaneous to upgrading the client to 4.1.2, we upgraded the server to 4.1.2 (obviously), and the server moved onto the same machine as cruise control. This server is Windows 2008 x64 (the previous server was 2003 x64). Also, we installed the 64-bit version of the 4.1.2 server (the previous server had the only 4.1.0 server available, which was 32-bit).

[Beth]

Do you have the 3.5 .NET framework on this machine?

Could you perform a reinstall of your 2.0 .NET framework on your machine to see if that helps?

There was a suggestion in this thread to run it as a console app instead of as a service. You might want to give that a try as well.

Also, I'm thinking you might as well put your client back to Vault 4.1.2.

[gsmalter]

We've put the 4.1.2 client back on there. We are currently running it as an app, which as I've said does work.

The machine does have 3.5 on it. It also has 2.0 on it, but 2.0 has never been installed. It's a Windows 2008 machine, and 2.0 comes on it.

[shannon]

Just for some clarification: The crypto exception seems to be occurring only when you have both .net 2.0 and 3.5 installed. It seems that the install of 3.5 changes the permissions on some directories. Reinstalling 2.0 seems to fix this.

You can also try checking the permissions on this folder:
C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys

Make sure the windows user you're using has full control.

[gsmalter]

Okay. To be clear, no .net framework of any version was installed in between when everything was working fine and the problem we're having now.

I checked out the RSA/MachineKeys folder (on Windows 2008, this seems to be at C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys). Administrators have only special permissions. I changed it to full control and Cruise Control Service with the old style plugin seems to be working okay now.

I will post again when I see if it works with the new plugin as a service.

[shannon]

Thank you for the update, every piece of information helps here. As I said, we're still investigating what is happening with the folder permissions.

[gsmalter]

The new plugin appears to work with CCNET as a service now. So, all four combinations listed in the original post now work. The only change made was giving the running user more permissions to the MachineKeys folder.

So, there are really two remaining problems (and one puzzle).

First, the new plugin does a bad job of reporting errors when it fails. Whereas the old one logged the Crypto exception and allowed us to fix the problem, the new plugin just killed the service without explanation. No logging anywhere.

Second, the permissions on the MachineKeys folder got mucked up by something other than a .NET framework installer. Candidates are the Vault 64-bit Server installer or the Vault client installer.

The puzzle is why the heck the console app version of cruise control worked. Unless it was impersonating Network Service (which, FYI, did have full control permissions to the MachineKeys folder) without my knowledge, it shouldn't have had permission to work where the Service version of cruise control failed.

In any event, we are off and running again. Thanks.