"Optimized For Speed"

thomas62j5 · Post by **thomas62j5** » Mon May 14, 2007 6:31 pm

We just upgraded to 3.5.2 in order to fix the issues we've had in the past with shadow folder pegging out the CPU.

I was going through the shadow folder configuration and I see there is a new set of options labeled "Synchronization Options." Either Synchronized With Repository (default for existing shadow folders) or Optimized for Speed (default for new ones).

I've read the docs on the subject, but was just wondering if I could get clarification on "Optimized For Speed." It sounds like it does what I had been asking for a while back, where vault would only update/create/delete things on disk if you update/create/delete them in vault, rather than going through every folder in the tree and trying to keep it perfectly in sync.

Is this right? Are there any gotchas or nuances we should be aware of?

Also, in the web.config for the shadow service, in these repository elements, I see a few new ones.

synchepository appears to correlate with this new synch option above, false being "optimized for speed."

What is "initialget"?

Thanks,
Thomas

lbauer · Post by **lbauer** » Mon May 14, 2007 8:15 pm

It sounds like it does what I had been asking for a while back, where vault would only update/create/delete things on disk if you update/create/delete them in vault, rather than going through every folder in the tree and trying to keep it perfectly in sync.

That's correct. If you make changes to the shadow folder target, Vault won't overwrite or change them unless those items are modified in the repository.

What is "initialget"?

I think it's the initial population of the shadow folder, but I'll check in this.

thomas62j5 · Post by **thomas62j5** » Tue May 15, 2007 9:23 am

I think it's the initial population of the shadow folder, but I'll check in this.

So if we set this to true, would Vault detect the .config has been changed, see the true value, and refresh that shadow folder tree?

I'm trying to find the easiest way to refresh a shadow folder without having to checkout/checkin to trigger it.

Thanks,
Thomas[/quote]

thomas62j5 · Post by **thomas62j5** » Thu May 17, 2007 4:08 pm

thomas62j5 wrote:
I think it's the initial population of the shadow folder, but I'll check in this.
So if we set this to true, would Vault detect the .config has been changed, see the true value, and refresh that shadow folder tree?

I'm trying to find the easiest way to refresh a shadow folder without having to checkout/checkin to trigger it.

Thanks,
Thomas

[/quote]

Could someone let us know about these questions?

lbauer · Post by **lbauer** » Fri May 18, 2007 1:48 pm

Initialget is the first, synchronized get of the shadow folder, which means the contents of the shadow folder exactly match the contents of the folder being shadowed.

We don't recommend editing the web.config file to synchronize. A better way would be to edit the shadow folder association and click the Synchronize radio button in the Admin Tool shadow folder settings. Then after the shadow folder has synchronized, go back to the Optimize setting.

thomas62j5 · Post by **thomas62j5** » Fri May 18, 2007 2:41 pm

lbauer wrote:Initialget is the first, synchronized get of the shadow folder, which means the contents of the shadow folder exactly match the contents of the folder being shadowed.

We don't recommend editing the web.config file to synchronize. A better way would be to edit the shadow folder association and click the Synchronize radio button in the Admin Tool shadow folder settings. Then after the shadow folder has synchronized, go back to the Optimize setting.

We need an API or some way then to refresh shadowed folders in a programmatic fashion. If we can't do this with web.config, that is. We have dozens of shadowed folders, it's not practical to do this manually through a gui for each one.

M Wickardt · Post by **M Wickardt** » Sat May 19, 2007 3:54 pm

Although I absolutely agree on the need of a server-side API, especially with an ability to immediately react (event-driven) to all actions taken on the repositories (we now do this by emulating an smtp server and parsing the e-mail data - yikes), in the meantime you could use vaultagent for this (we do the same). Just send a few get commands to vaultagent's tcp port like:

(this is from unix btw, with a printer port routed to vaultagent on a windows server):

lp -dprt_vault < $SCRIPTS/VaultGetProd.txt

contents of VaultGetProd.txt:

Code: Select all

get -backup no -makereadonly -merge overwrite -setfiletime modification -destpath "P:\\srcs\\vault\\baspas\\prod\\" "$/prod/"
get -backup no -makereadonly -merge overwrite -setfiletime modification -destpath "P:\\srcs\\bertus\\s\\" "$/prod/s/"

Ofcourse, on windows this could be done in the same way very easily.

Tijs Wickardt, Bertus Distributie

lbauer · Post by **lbauer** » Sat May 19, 2007 9:03 pm

We need an API or some way then to refresh shadowed folders in a programmatic fashion. If we can't do this with web.config, that is. We have dozens of shadowed folders, it's not practical to do this manually through a gui for each one.

Then why not just keep the Synchronized option?

thomas62j5 · Post by **thomas62j5** » Mon May 21, 2007 9:43 am

lbauer wrote:Then why not just keep the Synchronized option?

Because the last version of Vault when we had this enabled, it caused extremely heavy CPU usage. Not sure if the algorithm has changed, but the way it was explained before, whenever we changed any file in an entire shadowed folder tree, the entire shadowed folder would need to be checked to see if it should be written to disk again.

For the most part, all we need is for a shadowed file to be updated on disk when its updated in the repository.

Occasionally with this latest version, we find the shadowing is not working. (I have logging enabled but do not see any logging happening on disk when I specify, so not sure what's up with that.)

In cases like this it would be nice to say, OK please refresh this tree now to disk.

thomas62j5 · Post by **thomas62j5** » Tue May 22, 2007 5:53 pm

We have re-done our shadowing setup since we put the latest version of Vault in place. We are now attempting to shadow each repository tree to two locations, one local, and one over the network. The one over the network is working OK, but some of the ones locally aren't. I turned on Process Monitor (from sysinternals) to monitor for a specific file that keeps failing, and noticed a weird attempt to open the file (Summaries.pm) at this location:

C:\windows\system32\inetsrv\$\c_perl\site_lib\Summaries.pm

The two shadow elements for this repository path are:

<sfa assoc_key="1:$/c_perl/site_lib-c:\perl\site\lib" reppath="$/c_perl/site_lib" localpath="C:\Perl\site\lib" readonly="False" repid="1" synchepository="False" filedateoption="0" initialget="False" webpath="D:\VaultService\VaultShadowFolder" />

<sfa assoc_key="1:$/c_perl/site_lib-\\120593-web\c\perl\site\lib" reppath="$/c_perl/site_lib" localpath="\\120593-WEB\c\Perl\site\lib" readonly="False" repid="1" synchepository="False" filedateoption="0" initialget="True" webpath="D:\VaultService\VaultShadowFolder" />

The actual local location for the file is c:\perl\site\lib\Summaries.pm. Any idea why it would be looking where it was?

lbauer · Post by **lbauer** » Tue May 22, 2007 7:58 pm

That's strange. I don't recall seeing something like that before.

What kinds of errors are you getting in the shadow folder log or in the Vault server log regarding that file?

Anything in the Windows Event viewer?

thomas62j5 · Post by **thomas62j5** » Wed May 23, 2007 10:11 am

Can you tell me what classes I should be logging? Right now I have it set to log "all" and there's too much to paste in here, not sure which parts are relevant.

Could there be a parsing issue due to our remote server name containing a dash in its name, referenced in the UNC paths?

I don't see anything in the event logs.

lbauer · Post by **lbauer** » Wed May 23, 2007 3:57 pm

Since we don't know what we're looking for, I'm not sure . . .

Here's a list of available classes:

http://support.sourcegear.com/viewtopic.php?p=7806

You might want the transaction items, like:

# createrequest - Begin and end of creation of request objects for change set items during a commit
# connection - Method name and parameters for functions called at the web services layer on the client
# get - The work the client does to request files from the server, download them, and send them to the update thread
# refresh - The work the client does to request, retrieve, apply a tree structure delta from the server, and save the results to disk
# upload - Begin and end of file upload through HTTP
# shadowfolderdebug - Shadow Folder operations for the Shadow Folder Service only.

SourceGear Support

"Optimized For Speed"

"Optimized For Speed"

Strange shadowing