w3wp.exe high cpu usage

This forum is now locked, since Gold Support is no longer offered.

Moderator: SourceGear

thomas62j5
Posts: 71
Joined: Tue Aug 17, 2004 7:59 pm
Location: West Coast, USA

Post by thomas62j5 » Tue Aug 22, 2006 6:24 pm

Also, if I am allowed to add a second row for a second shadowservice, what value should I use for the pluginurlhash column?

jclausius
Posts: 3706
Joined: Tue Dec 16, 2003 1:17 pm
Location: SourceGear
Contact:

Post by jclausius » Wed Aug 23, 2006 9:47 am

I've provided our Support Department with instructions on this configuration. Please contact the SourceGear Technical Support if you would like more information.
Jeff Clausius
SourceGear

thomas62j5
Posts: 71
Joined: Tue Aug 17, 2004 7:59 pm
Location: West Coast, USA

Post by thomas62j5 » Fri Sep 15, 2006 7:00 pm

Well, I have our two servers split now, each shadowing to the local file system from a single vault server.

It has been running "OK" but still I would see the occasional slow update and cpu hit. Today though we would see massive CPU hits for extended periods of time. I know it's vault because it happens after a check-in, and the cpu spike is synchronized on both of these servers at that time.

I think the shadowing service needs to be re-thought. I think it is doing more than it needs to do with respect to directory comparisons to what is on-disk. I hope in the future that SourceGear will offer an option to default to updating only the files that have just changed, with an option to "refresh mirror" any time the developer wants to make sure everything is in sync. Right now it seems to be doing that every time, which is a waste, IMO.

lbauer
Posts: 9736
Joined: Tue Dec 16, 2003 1:25 pm
Location: SourceGear

Post by lbauer » Mon Sep 18, 2006 9:35 am

I'll log a task/bug to take a look at Shadow Folder performance and how we can optimize it, based on the issues brought up in this thread.
Linda Bauer
SourceGear
Technical Support Manager

TeamWiSE
Posts: 31
Joined: Thu Aug 10, 2006 2:40 am
Location: Mönchengladbach, Germany
Contact:

Post by TeamWiSE » Tue Sep 19, 2006 3:24 am

We're experiencing exactly the same problems (top-CPU usage with shadowing enabled) here - independent of shadowing on a network drive or locally. CPU usage hits the fan mostly when the structure changes, i.e. when files or folders are being removed or added. As long as single files merely are being checked in, everything runs smoothly. Studying the process performance, what is interesting is, that the amount of I/O operations that are not "read" or "write" (e.g. "stat" and the like) are enormous (see picture). This count easily quadruples the similar mark of McAfee with on-access scanning enabled, which is monitoring the entire filesystem and all accesses. My guess ist that some optimization in comparing the folder/directory structures could be done.

Another symptom seems to be, that after a restart of the IIS-worker process for sgVault, the CPU-usage when changing folder structure tends to be not so high and not so lengthy. We do not have specific measurements and comparisons to this, that is currently just an impression.
Attachments
performance.jpg
performance.jpg (254.51 KiB) Viewed 34434 times

TeamWiSE
Posts: 31
Joined: Thu Aug 10, 2006 2:40 am
Location: Mönchengladbach, Germany
Contact:

Post by TeamWiSE » Tue Sep 19, 2006 4:04 am

Turning off shadow folders also stops the enormous amount of creating and deleting mutants/mutexes named WF_something. Possibly some internal cache created for shadow folder usage steadily grows and causes the increasíngly bad performance. Objective data/measurements for this increase in performance can be found in this topic, where the horrendous performance was also caused by shadow folders - not on a network drive, but locally!

thomas62j5
Posts: 71
Joined: Tue Aug 17, 2004 7:59 pm
Location: West Coast, USA

Post by thomas62j5 » Tue Sep 19, 2006 9:29 am

I concur with your observation about restarting IIS. It does seem to bring the cpu usage down for shadowing, sometimes for a long while, other times not (perhaps dependent on how many check-ins are performed, and how rapidly).

TeamWiSE
Posts: 31
Joined: Thu Aug 10, 2006 2:40 am
Location: Mönchengladbach, Germany
Contact:

Post by TeamWiSE » Tue Sep 19, 2006 1:53 pm

There seems to be a correlation not with the number of checkins, but with the number of structure changes, say: adds/removes of files and folders. This seems to trigger a different kind of scan/sync in the shadowfolder service. So, after a restart, things may go well for a long time despite a lot of checkins, until structure changes happen.

lbauer
Posts: 9736
Joined: Tue Dec 16, 2003 1:25 pm
Location: SourceGear

Post by lbauer » Wed Sep 20, 2006 10:19 am

Thanks for the additional details.
Linda Bauer
SourceGear
Technical Support Manager

awdavis
Posts: 38
Joined: Fri Jan 07, 2005 12:27 am
Location: Los Alamos, NM
Contact:

Post by awdavis » Tue Oct 10, 2006 9:02 pm

We are seeing the same issue on our server. When I shut down shadow folders, things run smoothly. When they are turned on, the server locks (sits there busy and doesn't respond to anyone for up to 45 minutes) about three or four times a day. We're working with the support team, which directed me to this thread, so I'm hoping that when an answer is available, it'll be posted here.

lbauer
Posts: 9736
Joined: Tue Dec 16, 2003 1:25 pm
Location: SourceGear

Post by lbauer » Wed Oct 11, 2006 12:22 pm

Shadow folders do put an additional load on the Vault server. The shadow folder service is a type of Vault client, which updates the shadow folder contents whenever the shadowed folders are updated.

It is possible to run the Vault Shadow Folder Service on a different machine from the Vault Server, and this can improve performance somewhat. If you're interested in trying this configuration, email linda at sourcegear.com.
Linda Bauer
SourceGear
Technical Support Manager

awdavis
Posts: 38
Joined: Fri Jan 07, 2005 12:27 am
Location: Los Alamos, NM
Contact:

Post by awdavis » Wed Oct 11, 2006 12:46 pm

I can do that, but I have a couple of observations:

The load on the server comes in bursts. It seems to be fine for a long time and then suddently lock up for a very long time. Our server isn't a slouch; it's a good dual processor machine with lots of ram. Also, the CPU use isn't really that high, so I don't understand why the server process is totally locked up.

We didn't have this problem until version 3.5. Then suddenly (that day) this became an issue. I suspect things. I sort of hate to mess with another server for shadow folders when they seemed to take so little load in past. I wonder if I'll work to get them moved somewhere and then you'll find a bug and fix it, making my work redundant. Are you sure this isn't a bug? It sure looks like it. I can patiently wait a few weeks to see if you get a fix, but if you really think that this is a bedrock performance issue, then I'll go ahead an move things around. So what do you think it is?

lbauer
Posts: 9736
Joined: Tue Dec 16, 2003 1:25 pm
Location: SourceGear

Post by lbauer » Wed Oct 11, 2006 4:59 pm

We're investigating this to see if any changes in Vault 3.5 could be responsible for this performance issue.
Linda Bauer
SourceGear
Technical Support Manager

awdavis
Posts: 38
Joined: Fri Jan 07, 2005 12:27 am
Location: Los Alamos, NM
Contact:

Post by awdavis » Tue Oct 24, 2006 4:32 pm

Have you had any insights? Just wondering.

jclausius
Posts: 3706
Joined: Tue Dec 16, 2003 1:17 pm
Location: SourceGear
Contact:

Post by jclausius » Wed Oct 25, 2006 10:17 am

Tony:

We're currently investigating, looking for something which might explain the unexpected behavior. Once we find the problem, expect a fix in a Vault 3.5 maintenance release.
Jeff Clausius
SourceGear

Locked