w3wp.exe high cpu usage

[thomas62j5]

Whenever I check things in on our new server, I see extremely high CPU usage (across all four CPUs (well, 2 HT CPUs)). I have shadowing configured at a high level (i.e. shadowing a folder with a lot of subfolders). And the shadowing happens to both a local drive and a network drive.

Is Vault copying everything from vault to both drives, even if nothing has changed?

Are there some ideas on how to decrease CPU?

Thanks!

[thomas62j5]

BTW, I'm positive it's due to the shadowing, since nothing else is going on on this test server but check-ins to vault, and while the CPU usage is high, the network load is 10% on our 100mbps card.

[lbauer]

Is Vault copying everything from vault to both drives, even if nothing has changed?

It shouldn't be. The account used by the Shadow Folder Service should maintain a cache with baselines and only update files that have changed.

Note the timestamps of the files in your shadow folder and then change one file and check the timestamps again. Did more than just the one file get updated?

[thomas62j5]

I can verify that only the file I changed in vault is having the last modified time changed, although the very high CPU usage on w3wp.exe continues to be a problem immediately after check-in of a file.

[lbauer]

High CPU usage could be due to other work the Vault server is doing.

If you run this SQL query against the sgvault database, how many rows are returned?

select * from sgvault.dbo.tblplugins

How many users do you have? Do they receive email notifications when folders are modified?

Do you have debug logging enabled for the Vault Server? This can increase CPU usage.

Is Shadow Folder logging set to "true" in the VaultShadowFolder.config file? If you have many shadow folders, writing to these logs could consume additional CPU .

Are you experiencing problems as a result of the increased CPU useage?

[thomas62j5]

High CPU usage could be due to other work the Vault server is doing.

If you run this SQL query against the sgvault database, how many rows are returned?

Just one row, for the shadow service.

How many users do you have? Do they receive email notifications when folders are modified?

Only 3 active users, including myself. No emails are being generated.

Do you have debug logging enabled for the Vault Server? This can increase CPU usage.

Is Shadow Folder logging set to "true" in the VaultShadowFolder.config file? If you have many shadow folders, writing to these logs could consume additional CPU .

Logging is set to true, but up until I started trying to figure out this issue, I had the classesToLog string set to nothing, so it wasn't really logging anything. Not sure if this could still cause high cpu usage? I will set it to false, restart, and see what happens.

Are you experiencing problems as a result of the increased CPU useage?

Well we're talking ~95% cpu usage on w3wp.exe across all processors for several minutes. This is server will be going into production soon (primarily as a web/mail server), so this cpu usage issue will cause performance problems. In addition to the high cpu usage, it takes several minutes for the updates files to be shadowed, whereas our old server (with no network shadowing) it usually was updated within a matter of 15 seconds or less. With a 100mbit network card, it seems strange that the network aspect could be slowing it down this much. Even the local propagation of the shadowed folders is slow, too.

[lbauer]

Logging is set to true

This sounds like client logging or shadow folder logging.

Vault Server logging level is set in the Admin Tool under Server Options->Logging.

Is the log in debug mode? Does putting it quiet mode make a difference?

If it's already in quiet mode, put it in debug mode, then after a few checkins, send me a copy of the log (sgvault.log in %windir%\temp\sgvault). We'll see if anything unusual is going on that could cause the high CPU usage.

Also send me a copy of the web.config file from the VaultShadowFolder directory.

[thomas62j5]

This sounds like client logging or shadow folder logging.

Vault Server logging level is set in the Admin Tool under Server Options->Logging.

Is the log in debug mode? Does putting it quiet mode make a difference?

It is already in quiet mode.

If it's already in quiet mode, put it in debug mode, then after a few checkins, send me a copy of the log (sgvault.log in %windir%\temp\sgvault). We'll see if anything unusual is going on that could cause the high CPU usage.

Also send me a copy of the web.config file from the VaultShadowFolder directory.

OK, will do, thanks!

[cityone]

My company is experiencing the same problem. w3wp.exe runs at very high CPU usage whenever there is a vault client request. We have a 4-processor machine, 4GB RAM and running 3.1.7.3719 on Win2003 SP1. Also, the response time from Vault Server has been getting worse and worse lately. A single check-in can take up to 10 mins to complete.

Can you let me know what was the resolution to the problem in the post?

Thanks!

[thomas62j5]

My company is experiencing the same problem. w3wp.exe runs at very high CPU usage whenever there is a vault client request. We have a 4-processor machine, 4GB RAM and running 3.1.7.3719 on Win2003 SP1. Also, the response time from Vault Server has been getting worse and worse lately. A single check-in can take up to 10 mins to complete.

Can you let me know what was the resolution to the problem in the post?

Thanks!

Hi, I think you should start a different thread about this, since this does not seem to be what I am experiencing. My problem is with Vault 3.5 and shadowing.

[jclausius]

My company is experiencing the same problem. w3wp.exe runs at very high CPU usage whenever there is a vault client request. We have a 4-processor machine, 4GB RAM and running 3.1.7.3719 on Win2003 SP1. Also, the response time from Vault Server has been getting worse and worse lately. A single check-in can take up to 10 mins to complete.

Can you let me know what was the resolution to the problem in the post?

Thanks!

Sounds like a different problem. If you start a new thread we can explore this more there.

But off the top of my head here are some quick suggestions:

1) Upgrade the Vault Server to at least Vault 3.1.9.

2) Run any maintenance on the database. See Maintenance: The Vault Server database for more info.

3) When you get some downtime, shut down SQL Server, and defrag the Vault Server as well as the machine running SQL Server.

[lbauer]

thomas62j5: did you send me your Vault server log and VaultShadowFolder web.config file? I didn't see anything from someone named Thomas. If you do send them, provide a link to this post. Email linda at sourcegear.com.

Have you tried temporarily deleting shadow folder associations to see if CPU usage is still high?

[thomas62j5]

Hi Linda,

Sorry, been fixing some server issues unrelated to this.

It is absolutely related to the size of the folders being shadowed, in my opinion. I say this because I have one repository that is pretty small, with the top level folder shadowed locally and over the network. There is a CPU hit, but it's fairly brief.

Then I check in something to the larger repository with a top-level folder shadowed that is much deeper, and it's a 90% cpu hit for like a minute solid.

I will zip up my debug log and send it to you in a few minutes, thanks!

[thomas62j5]

FYI I did send those files over yesterday, in case you didn't see them.

[thomas62j5]

This issue is basically holding up our server rollout, since management is concerned that a checkin pegs all the cpus out for minutes at a time.