w3wp.exe high cpu usage

thomas62j5 · Post by **thomas62j5** » Tue Aug 22, 2006 6:24 pm

Also, if I am allowed to add a second row for a second shadowservice, what value should I use for the pluginurlhash column?

Post by **jclausius** » Wed Aug 23, 2006 9:47 am

I've provided our Support Department with instructions on this configuration. Please contact the SourceGear Technical Support if you would like more information.

thomas62j5 · Post by **thomas62j5** » Fri Sep 15, 2006 7:00 pm

Well, I have our two servers split now, each shadowing to the local file system from a single vault server.

It has been running "OK" but still I would see the occasional slow update and cpu hit. Today though we would see massive CPU hits for extended periods of time. I know it's vault because it happens after a check-in, and the cpu spike is synchronized on both of these servers at that time.

I think the shadowing service needs to be re-thought. I think it is doing more than it needs to do with respect to directory comparisons to what is on-disk. I hope in the future that SourceGear will offer an option to default to updating only the files that have just changed, with an option to "refresh mirror" any time the developer wants to make sure everything is in sync. Right now it seems to be doing that every time, which is a waste, IMO.

lbauer · Post by **lbauer** » Mon Sep 18, 2006 9:35 am

I'll log a task/bug to take a look at Shadow Folder performance and how we can optimize it, based on the issues brought up in this thread.

TeamWiSE · Post by **TeamWiSE** » Tue Sep 19, 2006 3:24 am

We're experiencing exactly the same problems (top-CPU usage with shadowing enabled) here - independent of shadowing on a network drive or locally. CPU usage hits the fan mostly when the structure changes, i.e. when files or folders are being removed or added. As long as single files merely are being checked in, everything runs smoothly. Studying the process performance, what is interesting is, that the amount of I/O operations that are not "read" or "write" (e.g. "stat" and the like) are enormous (see picture). This count easily quadruples the similar mark of McAfee with on-access scanning enabled, which is monitoring the entire filesystem and all accesses. My guess ist that some optimization in comparing the folder/directory structures could be done.

Another symptom seems to be, that after a restart of the IIS-worker process for sgVault, the CPU-usage when changing folder structure tends to be not so high and not so lengthy. We do not have specific measurements and comparisons to this, that is currently just an impression.

TeamWiSE · Post by **TeamWiSE** » Tue Sep 19, 2006 4:04 am

Turning off shadow folders also stops the enormous amount of creating and deleting mutants/mutexes named WF_something. Possibly some internal cache created for shadow folder usage steadily grows and causes the increasíngly bad performance. Objective data/measurements for this increase in performance can be found in this topic, where the horrendous performance was also caused by shadow folders - not on a network drive, but locally!

thomas62j5 · Post by **thomas62j5** » Tue Sep 19, 2006 9:29 am

I concur with your observation about restarting IIS. It does seem to bring the cpu usage down for shadowing, sometimes for a long while, other times not (perhaps dependent on how many check-ins are performed, and how rapidly).

TeamWiSE · Post by **TeamWiSE** » Tue Sep 19, 2006 1:53 pm

There seems to be a correlation not with the number of checkins, but with the number of structure changes, say: adds/removes of files and folders. This seems to trigger a different kind of scan/sync in the shadowfolder service. So, after a restart, things may go well for a long time despite a lot of checkins, until structure changes happen.

lbauer · Post by **lbauer** » Wed Sep 20, 2006 10:19 am

Thanks for the additional details.

awdavis · Post by **awdavis** » Tue Oct 10, 2006 9:02 pm

We are seeing the same issue on our server. When I shut down shadow folders, things run smoothly. When they are turned on, the server locks (sits there busy and doesn't respond to anyone for up to 45 minutes) about three or four times a day. We're working with the support team, which directed me to this thread, so I'm hoping that when an answer is available, it'll be posted here.

lbauer · Post by **lbauer** » Wed Oct 11, 2006 12:22 pm

Shadow folders do put an additional load on the Vault server. The shadow folder service is a type of Vault client, which updates the shadow folder contents whenever the shadowed folders are updated.

It is possible to run the Vault Shadow Folder Service on a different machine from the Vault Server, and this can improve performance somewhat. If you're interested in trying this configuration, email linda at sourcegear.com.

awdavis · Post by **awdavis** » Wed Oct 11, 2006 12:46 pm

I can do that, but I have a couple of observations:

The load on the server comes in bursts. It seems to be fine for a long time and then suddently lock up for a very long time. Our server isn't a slouch; it's a good dual processor machine with lots of ram. Also, the CPU use isn't really that high, so I don't understand why the server process is totally locked up.

We didn't have this problem until version 3.5. Then suddenly (that day) this became an issue. I suspect things. I sort of hate to mess with another server for shadow folders when they seemed to take so little load in past. I wonder if I'll work to get them moved somewhere and then you'll find a bug and fix it, making my work redundant. Are you sure this isn't a bug? It sure looks like it. I can patiently wait a few weeks to see if you get a fix, but if you really think that this is a bedrock performance issue, then I'll go ahead an move things around. So what do you think it is?

lbauer · Post by **lbauer** » Wed Oct 11, 2006 4:59 pm

We're investigating this to see if any changes in Vault 3.5 could be responsible for this performance issue.

awdavis · Post by **awdavis** » Tue Oct 24, 2006 4:32 pm

Have you had any insights? Just wondering.

Post by **jclausius** » Wed Oct 25, 2006 10:17 am

Tony:

We're currently investigating, looking for something which might explain the unexpected behavior. Once we find the problem, expect a fix in a Vault 3.5 maintenance release.