Vault Server Restarts about every Minute

[markmiddleton]

I was having problems with the Clients and the IDE both timing out there connections. I followed the instructions on changing the settings for IIS. Then I restarted the IIS service. The Vault Server is now restarting about every minute. I am running the Vault Server on a Windows 2003 Enterprise Server.

A sample from the log file follows:

----8/5/2004 9:22:22 AM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 9:25:37 AM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 9:27:51 AM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 9:28:03 AM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 9:29:17 AM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 9:32:19 AM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 9:35:35 AM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 9:37:50 AM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 9:38:01 AM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 9:39:16 AM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 9:42:18 AM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 9:45:34 AM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 9:47:48 AM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 9:47:59 AM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Quiet
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp


How do I solve the issue of the Vault Server restating every minute? How do I solve the Client and IDE timeout issue?

Thanks,
Mark

[dan]

Take a look at the KB article at http://support.sourcegear.com/viewtopic.php?t=1014. Most likely one of the settings in IIS is causing the Vault server to be recycled once a minute.

[markmiddleton]

I did follow the instructions in the knowledge base article “Server sessions keeps timing out under Win2k3 and IIS 6”

Application Pools:

All of the Recycles are off, unchecked.

Health:
Ping Enabled at 30 seconds
Enable Rapid-fail Protection
Failures: 5
Time Period 5
Startup Time Limit 90 seconds
Shutdown Time Limit 90 seconds

Vaults settings are the same as above.

There are no errors or warnings in the system logs.

I turned debugging on and here is the output from Vault:


----8/5/2004 12:33:29 PM --10.10.100.61(10.10.100.61)--SSL Disabled
Getting repository Structure.
----8/5/2004 12:33:32 PM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Debug
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 12:33:34 PM --10.10.1.20(10.10.1.20)--SSL Disabled
Getting repository Structure.
----8/5/2004 12:34:29 PM --10.10.100.61(10.10.100.61)--SSL Disabled
Getting repository Structure.
----8/5/2004 12:35:30 PM --10.10.100.61(10.10.100.61)--SSL Disabled
Getting repository Structure.
----8/5/2004 12:35:51 PM sgvaultsystem--()--
System Started
Version 2.0.4.2190
Cache Level = 1
DataBase Buffer Size = 256
LogFile Path = d:\temp\logs
Log Level = Debug
Archive Log = Weekly
ReverseDNS Lookup = False
Maximum HTTP Request Length = 102400
Overwrite Log on Startup = False
Session Timeout = 4320
SGVault Working Directory = d:\temp
----8/5/2004 12:35:53 PM --10.10.100.69(10.10.100.69)--SSL Disabled
Getting repository Structure.


Thanks,
Mark

[dan]

A few more tidbits:

1. If sharepoint is installed, it may have stomped on the modules that keep session state, which Vault needs. Uninstalling and re-installing Vault might help this, since it will re-create the Vault virtual folder in IIS.

2. If you have multiple web sites in IIS, installing to a different one might help reset the IIS settings.

3. Check the event viewer, and see if there is anything there related to what might be killing the Vault process.

[markmiddleton]

There is no SharePoint software installed on this system. There are no other web sites on this system either. This system is only used for Vault and our source code control.

The system log shows the following:

Type Date Time Source Category Event User Computer
Information 8/5/2004 12:51:07 PM IISCTLS None 1 N/A SENFORCE-BUILD
Information 8/5/2004 12:51:06 PM Service Control Manager None 7036 N/A SENFORCE-BUILD
Information 8/5/2004 12:51:06 PM Service Control Manager None 7036 N/A SENFORCE-BUILD
Information 8/5/2004 12:51:06 PM Service Control Manager None 7035 builder SENFORCE-BUILD
Information 8/5/2004 12:51:06 PM Service Control Manager None 7035 builder SENFORCE-BUILD
Information 8/5/2004 12:51:06 PM Service Control Manager None 7036 N/A SENFORCE-BUILD
Information 8/5/2004 12:51:06 PM Service Control Manager None 7036 N/A SENFORCE-BUILD
Information 8/5/2004 12:51:06 PM Service Control Manager None 7035 builder SENFORCE-BUILD
Information 8/5/2004 12:51:06 PM IISCTLS None 2 N/A SENFORCE-BUILD
Information 8/5/2004 12:51:05 PM Service Control Manager None 7036 N/A SENFORCE-BUILD
Information 8/5/2004 12:51:05 PM Service Control Manager None 7035 builder SENFORCE-BUILD
Information 8/5/2004 12:51:04 PM Service Control Manager None 7036 N/A SENFORCE-BUILD
Information 8/5/2004 12:51:02 PM Service Control Manager None 7035 builder SENFORCE-BUILD
Information 8/5/2004 12:51:01 PM Service Control Manager None 7036 N/A SENFORCE-BUILD
Information 8/5/2004 12:50:59 PM Service Control Manager None 7036 N/A SENFORCE-BUILD
Information 8/5/2004 12:50:59 PM Service Control Manager None 7035 builder SENFORCE-BUILD
Information 8/5/2004 12:50:59 PM Service Control Manager None 7035 builder SENFORCE-BUILD
Information 8/5/2004 12:49:50 PM IISCTLS None 1 N/A SENFORCE-BUILD


Thanks,
Mark

[dan]

I followed the instructions on changing the settings for IIS. Then I restarted the IIS service.

Which IIS values did you change? Can you reset them to their previous values, when Vault wasn't restarting every minute?

[markmiddleton]

Now I am back to my first problem. That was the Clients and IDE are timing out anywhere from two minutes to 20 minutes. This is not much of an change. We are still unable to use the Vault Server.

Mark

[dan]

Check the session state timeout value in the IIS application configuration dialog (directions are in the KB article referenced above). If it is set to 20 minutes (default), set to something a lot higher.

Another workaround to try is to set the automatic refresh for each Vault user from 10 minutes to 2 or 3 minutes, to try to keep the session alive (see Tools->Options in the Vault Client).

Out of curiousity, which setting caused Vault to be recycled every minute?

[markmiddleton]

The Timeout setting is set to 4320 minutes. Changing the timeout on the clients does not make any difference either. I am not sure which setting changed the server back to not restarting every minute. I changed them all back at once. This is our live server, and I dont have the time to play with it. It now restarts any where from 15 minutes to 40 minutes. So it is better, but not solved. Clients and IDEs still timeout between 2 to 20 minutes.

Mark

[dan]

There is a setting in IIS somewhere that is causing this. Win2K3//II6 is very aggresive about killing off IIS apps. You might consider switching to Win2k server or XP Pro to run Vault on, if you can't find the setting, because they don't exhibit the same aggressiveness.

One other question though: Have these timeouts always happened with your Vault installation, or has it changed recently?

[markmiddleton]

As I see back through the logs this comes and goes. With making any changes I am now back to having the Vault Server restarted every minute. If the answer is to switch this to a different OS then I am sure that will mean that Vault will not be able to be used by us. There has been alot of time, money, and effort that has gone into building this server. To start over now is not an option. Your documentation says that you support Windows 2000 and newer. So how do I make this server work?

Mark

[dan]

Diagnosing this here probably isn't going to work very well. Linda says you are also pursuing this via SG support, so we'll contact you to set up a phone call...

[markmiddleton]

I look forward to receiving your phone call.

Mark