Fortress grinds to a halt

[ecobb]

For some reason, this morning when we all came in to work Fortress was virtually unresponsive. I'm not sure what's going on, but it's happening in the web client, the desktop client, and in Eclipse.

The web client works, it's just incredibly slow. When the site comes up and you log in, it takes several minutes for the login to process, and every link you click takes several minutes before the page loads. Restarting IIS Web Services didn't help.

The Desktop client almost immediately prompts you for a login when you open it, but once you log in it just hangs. After about 10-11 minutes, it will eventually prompt you select a repository, then after another 10 minutes it will show the files. It literally took me over 20 minutes to log in to the desktop client. Selecting "show differences" on a file that was recently modified takes over 3 minutes to render on a file that had only 2 small changes made to it. This was when checking against the current version in the repository. However, when I do a show diff and select "the last version retrieved from the repository" option, it works fine.

Eclipse is pretty much the same way, only a little more responsive. Once you log in it hangs at "Connecting to Fortress Server" for 3-4 minutes before it moves to "Loading the Repository", at which time it takes another 5-6 minutes before finishing the process. In all total, it took 9 minutes in Eclipse from the time I logged in to Fortess until it processed the login and I could access my files. Also, right clicking on a file and selecting "Team > Show History" took almost 4 minutes to render for a file that only had 5 items in its history.

The server that is running Fortress is running fine. Memory usage and CPU are running normal, and all other sites and applications hosted on that box are running fine with no delays. The SQL Server is running fine, and I can connect to and access all three of the "sg" database with no problems. (the response is almost instantaneous in SQL Server Management Studio) . A continuous ping to that server shows no drops, with all responses being <1ms.

I've confirmed that this is happening from 5 separate developer desktops, all running Windows 7 64 bit or Windows XP 32 bit. No updates were applied to the desktops or server over the weekend.

Short of restarting the server (again), are there any suggestions for something we can try? Are there any log files to check or settings to change? Any help or suggestions are greatly appreciated!

[lbauer]

If all clients are having problems it's either server-side or network issue.

If you do the operation from a Fortress Client on the FortressServer machine, and use "localhost" for the Fortress Server name, does the operation work faster?

This may rule out any network issues.

Are there any errors in the Fortress Server log? It's called sgvault.log and is in %windir%\temp\sgvault on the server machine.

[ecobb]

I logged in to the server and opened the Fortress client with localhost as the server name and it still takes a several minutes to load. Also, when trying to pull up the web client via localhost directly on the server (http://localhost/VaultService/admin), it takes several minutes for the login page to render.

The only thing in the sgvault.log file are our logins and a couple of "System Started" messages from our reboots.

[ecobb]

I'm not sure if this means anything or not, but I was looking through our IIS log files and noticed that every item being requested takes 15-16 seconds to get. Here's a snip from the log file, notice the timestamps:

2010-07-12 20:33:01 W3SVC1 160.129.40.243 GET /Fortress/Style/Fortress.css -
2010-07-12 20:33:17 W3SVC1 160.129.40.243 GET /Fortress/Style/iehacks.css -
2010-07-12 20:33:32 W3SVC1 160.129.40.243 GET /Fortress/Style/ie7hacks.css -
2010-07-12 20:33:46 W3SVC1 160.129.40.243 GET /Fortress/style/fortressprint.css -
2010-07-12 20:34:01 W3SVC1 160.129.40.243 GET /Fortress/style/fortressbranding.css -
2010-07-12 20:34:16 W3SVC1 160.129.40.243 GET /Fortress/style/images/fortress.png -
2010-07-12 20:34:32 W3SVC1 160.129.40.243 GET /Fortress/Style/images/redbar.png -
2010-07-12 20:34:47 W3SVC1 160.129.40.243 GET /Fortress/Style/images/blackgrad.png -
2010-07-12 20:35:02 W3SVC1 160.129.40.243 GET /Fortress/Style/images/greytab.png -
2010-07-12 20:35:17 W3SVC1 160.129.40.243 GET /Fortress/Style/Images/Delete.png -
2010-07-12 20:35:32 W3SVC1 160.129.40.243 GET /Fortress/Style/Images/Edit.png -
2010-07-12 20:35:47 W3SVC1 160.129.40.243 GET /Fortress/Style/images/medgreygrad.png -
2010-07-12 20:36:02 W3SVC1 160.129.40.243 GET /Fortress/Style/images/ltgreymenubar.png -

So, it took 3 minutes just to get those few CSS files and images. None of the log files for any of our other sites on this server are showing lag time like this. Also, Fortress is the only .Net application being hosted on this machine.

[lbauer]

Have you tried resetarting Fortress Server with an iisreset from the command line? Does this help at all?

Were there any changes to the server machine -- perhaps a Windows update that could have changed things? I know you said no changes were made, but just confirming.

Any errors in the Windows Event viewer that might be relevant?

We'd like to see some detailed logs. Enable debug server logging in the Admin Web Client under Server Settings->Logging->Log Level.

Also enable logging in the Fortress GUI Client. To enable client-side logging in the GUI Client open the Fortress GUI Client and hold down the following keys on your keyboard:

Control-Alt Shift F12

The Fortress GUI Client log is called VaultGUIClient.txt and would be in your user's temp directory on the client machine. Perform a login or other operations that are slow. Send the logs zipped up to support at sourcegear.com, Attn: Linda. Please include a link to this forum post.

[ecobb]

According to update manager the last server updates were applied 3 weeks ago, on June 23rd.

We may have just found something. There is a service on that server called ".NET Runtime Optimization Service" that was not running. We started that service, and all of the lag time went away. I'm not sure what that service does, but it seems to have fixed our problem for now.

The funny thing is, that service starts up and then immediately shuts right back down. I'm not sure what all that's about, but for now everything seems to be working. I'll post back if it breaks again.

[lbauer]

Thanks for the update.

Here's some info on the CLR Optimization Service:

http://blogs.msdn.com/b/davidnotario/ar ... 12838.aspx