w3wp.exe high cpu usage

[jclausius]

One other test you may want to do is to temporarily remove all shadow folders, and try the check in again.

If the CPU usage stays within acceptable limits, then you would know that it is the Shadow Folders startup / get using up the CPU time.

[thomas62j5]

One other test you may want to do is to temporarily remove all shadow folders, and try the check in again.

If the CPU usage stays within acceptable limits, then you would know that it is the Shadow Folders startup / get using up the CPU time.

I took out the shadow folders, restarted IIS using iisreset /restart. Performed a check-in, no cpu spiking.

I put just the shadow folders in that were shadowing locally. IIS reset, performed checkin, very slight cpu spiking and for a very brief period.

I put the rest of the shadow folders back (network shadow), iis reset, performed checkin, and heavier cpu spiking for longer period of time.

However, so far I have not seen the extremely high spiking and long times that we were seeing last week. Only one thing was changed, while I was analyzing the shadow folder definitions in the config file, I noticed one of our repositories had the same definition twice, rather than having one of those defined for the network shadow. I fixed that when I put the shadows back.

However... that repository is not one of the ones being checked into lately, so I am guessing that is just a coincidence.

I will post again here if I see the really heavy CPU and long shadow times return. For now I would say, it's best if you don't do network shadowing. Unfortunately, for the time being we have to do that.

[jclausius]

Vault's Shadow Folders is implemented as a stand-alone Vault client which runs and does the gets on your files. The larger the tree and more shadow folders configured , the more work the Shadow folder does to retrieve those files.

Note, there is nothing that couples the Vault server to Shadow Folders. You could set up an environment where the Shadow Folders Web app is configured on a different machine than the Vault server.

[thomas62j5]

Vault's Shadow Folders is implemented as a stand-alone Vault client which runs and does the gets on your files. The larger the tree and more shadow folders configured , the more work the Shadow folder does to retrieve those files.

I think performance of shadow folder service could be improved. Without knowing the details of how it works currently, and only going by what appears to be worse performance the deeper a shadow folder is, I have to wonder if it could be made to work like this:

- Table is filled with any vault-stored files that have been changed that need to be shadowed. Shadow service periodically polls this table, and overwrites those files that have been changed, deleting the records once it has done so. This way it is only worrying about files that have changed, and the depth of the shadow folder doesn't matter.

- A certain "special row" could be added to trigger the shadow service to refresh the entire repository, or the shadowed folder's entire tree. These could then be initiated with new shadow folder options made available to users (Refresh Repository Shadow Folders, Refresh This Shadow Folder).

If SourceGear is not interested in an overhaul like this, could you tell me if there is a table that I could query on my own to determine recently changed files? I can imagine it not being a problem to create my own shadowing system if there is such a table to query.

Note, there is nothing that couples the Vault server to Shadow Folders. You could set up an environment where the Shadow Folders Web app is configured on a different machine than the Vault server.

Is there an existing document that details how this can be done, i.e. starting the vault shadow service on a separate server without starting vault on that server?

[thomas62j5]

I will post again here if I see the really heavy CPU and long shadow times return. For now I would say, it's best if you don't do network shadowing. Unfortunately, for the time being we have to do that.

After more use, the high CPU loads and lengthy shadowing times have returned.

[jclausius]

Is there an existing document that details how this can be done, i.e. starting the vault shadow service on a separate server without starting vault on that server?

I'll need to double check the instructions, but it should be somthing like:

1) Configure another server with a Vault Server. There are manual instructions, but perhaps the easiest way is to install Vault on the other machine when NO ONE is accessing Vault.

2) Vault notifies the Shadow Folder web service by a lookup in sgvault.dbo.tblplugins. You'll need to verify the tblplugins has a row to notify the other machine. Make sure there is only one row, and it is pointing to the different machine.

3) Shadow folder's setup is controlled by Shadow Folder's web.config file through the admin tool. You'll need to use the Admin Tool configured on the NEW machine, as it will use that to setup the shadowing.

4) Once you've set up a test Shadow, look at web.config for the new Shadow Folder. Make sure the Vault Server xml element points to the correct Vault server.

As I said, I haven't had a chance to try this out, so I'll need a couple of days to test it all out.

[jclausius]

I think performance of shadow folder service could be improved. Without knowing the details of how it works currently, and only going by what appears to be worse performance the deeper a shadow folder is, I have to wonder if it could be made to work like this:

- Table is filled with any vault-stored files that have been changed that need to be shadowed. Shadow service periodically polls this table, and overwrites those files that have been changed, deleting the records once it has done so. This way it is only worrying about files that have changed, and the depth of the shadow folder doesn't matter.

Here is how things currently work.

1) Change is made on the Vault server.
2) Vault server notifies all Plugins a transaction has changed.

On the Vault Shadow Folder
1) A thread is asleep w/ a stack of changes for all configured shadow folders.
2) When a transaction change comes in, the Shadow Folder clears the entire stack, and does a GET on all changed items found within folders it is configured to shadow. It also performs local deletions if items have been deleted from the repository. There's not much else going on here.

If you've set up Shadow to hit a UNC, it could be the scanning of working folders that is taking up that much time. Placing the Shadow Folder service on the machine of the destination UNC, might help alleviate the problem.

[thomas62j5]

Here is how things currently work.

1) Change is made on the Vault server.
2) Vault server notifies all Plugins a transaction has changed.

On the Vault Shadow Folder
1) A thread is asleep w/ a stack of changes for all configured shadow folders.
2) When a transaction change comes in, the Shadow Folder clears the entire stack, and does a GET on all changed items found within folders it is configured to shadow. It also performs local deletions if items have been deleted from the repository. There's not much else going on here.

OK, but if I'm not mistaken, Vault's shadow service doesn't just look at stuff that Vault says has been changed, it actually is examining all the files on disk to see if they are different, right? Because I know I have seen existing files that I've changed only on disk get overwritten by Vault after an unrelated checkin in the same shadow folder tree. Reality is that file shouldn't have been touched, since I didn't modify it in vault, only on disk.

I think that is where the inefficiency seems to come into play. I feel it should only try to update stuff that has happened in vault, by default, and then maybe have an option for a full refresh/mirror of a shadowed folder tree or repository for those rare times when things are totally off, or you are mirroring to a new server, etc.

If you've set up Shadow to hit a UNC, it could be the scanning of working folders that is taking up that much time. Placing the Shadow Folder service on the machine of the destination UNC, might help alleviate the problem.

Can you provide more details about what you mean by "scanning of working folders"? Do you mean what I refer to above, where I think vault shadow service is looking at the folders on disk to try and find changes, even if none are being described in vault itself?

Also, our problem is we currently need to shadow locally AND to a separate server. Are you saying we could have shadow service running on both servers, with each server configured only to shadow locally to itself?

[jclausius]

OK, but if I'm not mistaken, Vault's shadow service doesn't just look at stuff that Vault says has been changed, it actually is examining all the files on disk to see if they are different, right?

Correct. Shadow Folders is a service that makes an exact copy of your Vault repository. Any changes made to a Shadowed folder on disk are always over written by the files in the repository during a Shadow Folder update. The files are not meant to be edited or replaced.

Can you provide more details about what you mean by "scanning of working folders"? Do you mean what I refer to above, where I think vault shadow service is looking at the folders on disk to try and find changes, even if none are being described in vault itself?

Yes. The Shadow Folder service is scanning your configured drives looking for any changes you made on disk, which do not match the version in the Vault repository.

Also, our problem is we currently need to shadow locally AND to a separate server. Are you saying we could have shadow service running on both servers, with each server configured only to shadow locally to itself?

Yes. I believe that setup would work.

[thomas62j5]

Is it possible to write our own plugins for vault (I bring this up because I noticed the plugins table being used)? If so, are there docs on doing this?

At this point I feel our most efficient strategy for shadowing would be to write a plugin that shadows the way we need it to work.

[jclausius]

This is mostly undocumented, and we may not get anywhere. But if you want to go down this road, you would need to create a Web Service inherited from VaultPluginServiceBase.

With this web service, you register with the Vault Service with an array full of the VaultPluginEvent.* constants. Note - I don't think this is fully implemented, but VaultPluginEvent.EndTxEvent should work.

RegisterPluginNotification(string strPluginServerCallbackURL, string strName, int[] arTriggerEvents)

You then would override the OnXEvent(), and do the action you wish to take.

To remove a plugin, call UnregisterPluginNotification ( VaultSqlConn conn, string strPluginServerCallbackURL, string strName, int[] arTriggerEvents )

[thomas62j5]

Interesting. Thanks for the info.

Any pointers to info about VaultPluginServiceBase so that I can inherit from it?

Also, any table in particular that I might be able to query for info on recently updated/deleted files?

[jclausius]

Add VaultPluginLib.dll and VaultLib.dll to a project, and then you should be able to inherit from it.

VaultLib has the define for the plugin events, and the plugin base will have the overrides.

See the Vault Client API for other info related to integrating with Vault.

[jclausius]

As for changes to the repository, what a lot of people have been doing is using a history query for version changes on $/.

For example some people shell out to the Vault Command Line client, executing a history request for one row on $/, and parsing the results. Other people use the Client API for the history request.

When you notice a version change, you can use the GetTxInfo() API call to retrieve all info for a given transaction. See http://support.sourcegear.com/viewtopic.php?t=6546 for more info.

[thomas62j5]

1) Configure another server with a Vault Server. There are manual instructions, but perhaps the easiest way is to install Vault on the other machine when NO ONE is accessing Vault.

2) Vault notifies the Shadow Folder web service by a lookup in sgvault.dbo.tblplugins. You'll need to verify the tblplugins has a row to notify the other machine. Make sure there is only one row, and it is pointing to the different machine.

3) Shadow folder's setup is controlled by Shadow Folder's web.config file through the admin tool. You'll need to use the Admin Tool configured on the NEW machine, as it will use that to setup the shadowing.

4) Once you've set up a test Shadow, look at web.config for the new Shadow Folder. Make sure the Vault Server xml element points to the correct Vault server.

As I said, I haven't had a chance to try this out, so I'll need a couple of days to test it all out.

I am/was getting ready to try these instructions, since we haven't had the time necessary to write our own plugin, and I realized you said "Make sure there is only one row, and it is pointing to the different machine."

Are you saying only one server can be configured to shadow things at a time? Or can I have two plugin rows, one for each copy of vaultshadow service, each service shadowing locally. ? Also, have you had a chance to double-check these instructions as you mentioned? Thanks