Bandwidth, latency, and performance

[Entropy]

We're considering moving to Vault from an antiquated VSS/SoS solution. Most of the road blocks are out of the way, but we're having some issues with our remote users using Vault over average bandwidth but high latency links (think: satellite). Operations like viewing the history of a small folder or the details of a file listed in the history view take several minutes to complete. SoS, while somewhat slow, is much more responsive. We're using different servers for SoS/Vault, but they're situated on the same drop at the moment. I could drag out Ethereal and analyze a few Vault transactions to see how many back-and-forth messages took place (to see if it's really a link latency problem), but I figured it would be easier to come here and ask first. So...

Would anyone know how many messages exchanges occur in a typical transaction? in particular, a view history? How bandwidth-intensive are such operations? Is there any experience here using Vault over LFNs or simply pokey links like ISDN?

[jclausius]

Due to the stateless connection of web services, Vault communications are going to be "chattier" than that of SourceOffsite.

For instance, when doing a history query, the Vault client is going to first refresh its tree and checkout list. Next it will submit the history query to the server. Once the results have been retrieved, the client will then send a request for the results.

Also, do you know if the link between client/server supports Keep-Alives? If each web request from the Vault client has to open a new connection to the server, creating the association will slow things down.

[jclausius]

I'm going to log a feature request to see if we can "tone" down the communications on some operations.

[jclausius]

FWIW, we've tested some situations where bandwidth is the issue. That doesn't seem to be a problem in our testing scenarios.

When you say high latency, what are we talking about 3000 to 4000 ms? Higher?


[UPDATE]
I just checked some of the client code. I can't see too much traffic sent during a history query. I only see one call to query the history, and one call to get the first page of results (based on the client side option).

When the results have all been retrieved (or the user closes the window), the clients sends a request it is done with history.

For these calls, it doesn't seem like History should take minutes.

[Entropy]

Thanks for your detailed response.

We're talking about link delays over 1s, but typically less than 3s.

Hmm. I'd say that something odd is going on with our network, but the adjacent SoS box works well. More investigation is required here.

[jclausius]

I did a quick test using Packetyzer.


A history query resulted in the following action:

- HTTP POST sent (for the web service call to query history)
- HTTP CONTINUE recv'd from server

- HTTP SOAP sent (the data for the of the call)
- HTTP ACK recv'd from server

- HTTP POST sent (for the web service call to pick up history items)
- HTTP CONTINUE recv'd from server

- HTTP SOAP sent (the data for the of the call)
- HTTP ACK recv'd from server

-------------
After this, I get 5 packets of data (1514 bytes long) for the history results.


One other thing to note, SOS' uses its own proprietary binary protocol which is better suited for low bandwidth configurations, while Vault uses XML based SOAP Messages over HTTP for most of its protocol, except for file data. This difference may explain some of the differences have seen.

If you get a chance to look at things, I'd be interested in the results from your end of things.

[jclausius]

After talking to a colleague, he mentioned looking at the following changes to your client configurations:

In the Vault client directory, there will be a file named VaultGUIClient.exe.config. You will need to merge in this XML into the correct element within that file:

Code: Select all

<system.net>
   <settings>
      <servicePointManager expect100Continue="false"/>
   </settings>
</system.net>

This setting will instruct the client to send the Web Request / SOAP message at the same time. (Thus eliminating one of the calls). See http://msdn.microsoft.com/library/en-us ... etopic.asp for a more complete description of the setting.

[jclausius]

Entropy:

Just curious, did you ever get anywhere with this setting?

[Entropy]

With the setting, no. With the network, yes. (We discovered a possibly-flaky VPN box that was falling back to using the ISDN backup line. Ick.) Performance is now respectable (though still noticably sluggish) over the link.

I tried the above setting on two machines, but the Ethereal logs still show the client waiting for the HTTP 100 response. The top part of my VaultGUIClient.exe.config looks like this:

Code: Select all

<system.net>
<defaultProxy>
	<!-- 
            The following entry enables reading of the per user (LAN) Internet settings.
            Adding additional proxy settings, without first setting to "false",
            will individually override. Note that "Automatic configuration" and 
            "automatic configuration scripts" cannot be read.
            
            <proxy> settings:
                usesystemdefault="[true|false]" - Read settings from Internet Options (see above)
                proxyaddress="[string]" - A Uri string of the proxy server to use.
                bypassonlocal="[true|false]" - Enables bypassing of the proxy for local resources.
             -->
			<proxy usesystemdefault="true"/>
			<!--<proxy usesystemdefault="false" proxyaddress="http://proxyhost:proxyport" bypassonlocal="false"/> -->
		<!-- use this section to disable proxy use for matching servers
            example:
        <bypasslist>
            <add address="bypassRegexString" />
        </bypasslist> 
        -->
</defaultProxy>
	<settings>
		<servicePointManager expect100Continue="false"/>
	</settings> 
</system.net>

I've restarted the client, etc. I'm using Vault 3.0.7 (2863). I can send along the logs (or screenshots) if you wish.

[jclausius]

I haven't had time to fully investigate the settings. Here is some information, but it looks as if this applies to Win2003 Server. Hopefully it is useful. Just in case, I'll log a research item to see if the Vault client honors this setting on other platforms.


http://www.microsoft.com/technet/prodte ... 6perf.mspx

How to Change Expect100Continue and Nagling Behavior

Both the Expect100Continue behavior and nagling behavior can be changed on a per server or per application basis in Windows Server 2003. By default, both behaviors are on, that is, the value in the configuration system is true. To change the parameters for a system, you need to edit the following file:

%SYSTEMROOT%\Microsoft.NET\Framework\vVersion of the framework\CONFIG\machine.config

In the following section you need to add the properties and override them.

<system.net>
<settings>

<servicePointManager expect100Continue="true"|"false" useNagleAlgorithm="true"|"false"/>

</settings>
</system.net>

To change for a specific application, you modify the app.config file in the same directory as the application. The same change as above is made.

Note: Expect100Continue and useNagleAlgorithm are changed on the client side of the Web serviceor the consumer of the Web servicenot the server side. The server will behave accordingly to whatever behavior the client displays when interacting with the server.

[Entropy]

These settings, when applied to %SYSTEMROOT%\Microsoft.NET\Framework\v1.1.4322\CONFIG\machine.config on a Windows XP SP2 machine running the Vault client, seem to have no effect after a reboot.

[jclausius]

Sorry I cannot be of more help with configuration. The MS article mention thats you can implement this on the server side on Windows 2003 Server (although I'm not sure how this would affect the client). Is your test Vault server on this platform? I'm not sure it would help, but it is a possiblity.

Other than that, I'm not sure what other advice I can provide. It might turn out this cannot setting cannot be controlled in .Net 1.1 with a web.config setting, and must be done programatically. The feature request I logged will investigate if this behavior can be turned off.