ShadowFolder Client Slow

[Bill Bell]

Hi,

things have been going well with my Vault 2.05 application on Windows 2003 since Jeff sorted me out.

http://support.sourcegear.com/viewtopic.php?t=1704

However now that I have 10 developers hammering away on Vault concurrently, we are starting to see a behaviour with ShadowFolders i.e when something is moved into a folder it does not get moved into the Shadowed Folder right away. i.e Sometimes it take seconds, sometimes it takes minutes, the worst case was 30 minutes.

When I go and have a look at the shadow log, there's nothing there. Then all of a sudden it starts to process queries. I know this because the log file starts to fill again and I see hex.tmp files temporarly appear in the c:\windows\temp directory.

I am assuming from reading the documentation that the user gui uploads the file to the vault webservice which puts it into the sql database. The webservice also triggers some kind of process that gets the shadowfolder service running.

Is the above correct, how can I manually trigger the shadowfolder services?, Is there a queue to check to see if its Jammed up?

Users are losing faith in this process. They don't know if their changes will be deployed or they will need manual intervention.

Any help would be appreciated.

Regards,

Bill.

[jclausius]

Bill:

There isn't a queue per se, but rather a transaction id, in which the Shadow Folder client is retrieving.

In a nutshell, you correctly described the Shadow Folder architecture. Changes made to the repository are not automatically deployed in shadowed folders, but rather sent to a Shadow Folder Manager. This manager lives in its own process, and when it sees there is a new transaction, and it starts its work.

If the Shadow Folder is in the middle of processing some large request, say someone added 300 MB of files, and deleted 200 MB of files, the next repository change will be queued up until the previous work is completed. Now if a second change comes in, it over-rides the previous change, and a third change will over-ride the second change and so on and so forth until the original request has completed. Once the original request has been completed, if there is work to be done, the Shadow Folder service begins working on this as well.

Based on this architecture, the minute lag is understandable. However, I'm concerned about the 30 minute lag. Is it possible the 30 minute lag occurred when someone was executing a large obliterate / branch command?

There are some concurrency improvements that we are working to correct in Vault 2.1. Is it possible this is the cause?

One suggestion to gather more data. You could temporarily turn on the "shadowfolderdebug" class in your Vault Shadow Folder's web.config file. This would log more information, and you could compare the times here with the times found in Vault Server's Log.

Let us know what you find out.

[jclausius]

Is the above correct, how can I manually trigger the shadowfolder services?, Is there a queue to check to see if its Jammed up?

Shadow folder's won't really be jammed. The Shadow Folder client will be either working on synchronizing with the server, or it will be idle awaiting the next transaction.

A couple of other things to consider:

1) Are the shadowed folders on a local disk drive or across the network? If across the network, have you tried to set them up locally? If so, do you see the same delay?

2) Also, active virus scanners might slow down disk I/O for the Shadow Folder client. Are there any virus scanners watching / interpretting the Shadow Folder's writes to disk? Check both the server's working directory (as specified in vault.config) as well as the shadowed folders.

3) How many different shadow folders associations do you have configured?

[Bill Bell]

Hi Jeff,

have emailed you my shadowfolder log and some screen prints of my vault server temp directory and gui client with the program that started this request.

Regards,

Bill.

[Bill Bell]

Hi Jeff,

Here are some replies to your questions

1) Are the shadowed folders on a local disk drive or across the network? If across the network, have you tried to set them up locally? If so, do you see the same delay?

All the folders are across a network. It is a gigabit network I can measure the bandwidth usage but believe there is a low chance of network latency causing the problem. Network performance was once my bread and butter and in its worst days in running on an average of 23%.

We need to use network UNC's as part of our deployment strategy. For example if a frontline Web Server is compromised then Vault will not be compromised.

2) Also, active virus scanners might slow down disk I/O for the Shadow Folder client. Are there any virus scanners watching / interpretting the Shadow Folder's writes to disk? Check both the server's working directory (as specified in vault.config) as well as the shadowed folders.

There are Virus Scanners running on these servers I will exclude them from looking at the temp and vault directories. However these servers are fast 15K disks, Quad 2.8G Xeon's, 4Gb memory

3) How many different shadow folders associations do you have configured?

We currently have 5 Shadowed Folders however if Vault is the success I think it will be then I want convert all our systems to use it. This could end up being upwards of 50 shadowed folders.

Regards,

Bill.

[Bill Bell]

Hi Jeff,

I am also interested in the feature request you added in the topic

http://support.sourcegear.com/viewtopic.php?t=1742

Eric is correct. Vault's Shadow Folders is designed is to create an [bexact view[/b] of the latest files within the repository.

If someone makes changes to a file within a folder under Shadow Folder's control, the Shadow Folder Server will overwrite that file on the next change set transaction, thus creating an exact copy of the repository' files.

I've logged a feature request, in your name, to give Shadow Folders tje ability to use a flag controlling how files of unknown, edited, or renegade status are handled within the Shadow Folder directories.

Could you add me to the list to be notified if/when you need a beta tester.

We use Vault's shadowfolder component for more than just keeping a shadow folder of our environment. We use it as a deployment tool into test, qa, and production. For example we have Classic ASP websites, which we use Vault to deploy into production, Vault change controls .ASP, .JPG, .GIF files etc into the production site and all is well.

What Vault does not change control is the GLOBAL.ASA file. This file has environment specific information that developers cannot/should not have access to read, especially in production i.e Passwords.

A side affect of this setup is that if we make a change to the global.asa file directly, vault detects this and seems to reprocess all files within all repositories that have shadow folders. Thus we get time delays, sometimes big time delays. Does'nt overwrite the global.asa file as it was never in vault.

So it would be really cool if logic could be put in logic to ignore changes done to files behind the scene (directly), especially ones not actually in vault. This would come in handy for log files, attachments uploaded to the website. etc.

Regards,

Bill.


p.s In the future I hope that SourceGear expand their shadowfolder deployment methods to include FTP and Webdav to targets. This will allow us, as an organisation to impose stricter firewall rules on our websites.

[dan]

This seems like a different feature to me than Shadow Folders. A Shadow Folder is supposed to exactly match of what is in the database, and if we allowed cases where files were allowed to not match or be ignored, it would be inevitable that we would get angry calls from folks who had accidently configured it that way.

This seems more like a Deploy kind of feature. And, remember that you can always do a Get Latest to your deployment folder, which is probably a good safe choice anyway, since you probably don't want to automatically deploy a folder to a production server automatically on checkin, in case there is a bug introduced in the checkin.

[Bill Bell]

Hi Dan,

I believe you are right, I am asking for a deploy facility. But would live with the way shadow folders work at present minus the reprocessing it detects when something behind the the scenes changes. Why?

1. We used to use ftp in dev,test,qa,prod and things got overwritten, missed, even lost. Since implementing Vault in all these environments, rollbacks a winner, Business analysts can do there own change control in a limited fashion eg Helpscreens and I guess most importantly there is minimal requirement for human intervention to do the change i.e no copying of files to different directories just put it into vault and all is taken care of.

2. Version Control is a real winner, it keeps people accountable and lets admins like myself easly backout any changes if required. It allows developers to see the production code while not seeing configuration files.

3. Linking it in with Fogbugz has gotten the auditors off my back, anytime they want to see a change to production I can walk them through the Fogbugz Case, click on the modules and show the changes in Vault - Fantastic. I sell them on the fact that the change is automated right down to deployment in the target environment.

Shadow Folders allow me to do all the above. I guess all I'm asking for is what JeffC says in http://support.sourcegear.com/viewtopic.php?t=1742
put extra options into shadowfolders that will make them more configurable for people like myself. If people don't want to use them stay with the default configuration.

or

Change the way Shadow Folders reprocesses the files if it detects a change i.e Schedule - So I can set it up to run at night.

or (heres hoping)

SourceGear look at a "deploy" facility which is as stream lined as how shadowfolders work.

Ok I've had my rant - could you tell me if Jeff's request is being looked at and provide maybe a guestimate on when it would be implemented.

Regards,

Bill.

[dan]

Hi Bill,

Unfortunately, this won't make our upcoming 2.1 release. We haven't made firm plans on features for the release after that, but we'll certainly consider either a Deploy feature or a Shadow Folder enhancement.