FailDeltaApply

ICOM · Post by **ICOM** » Wed Mar 07, 2007 1:07 pm

We're evaluating SourceGear Vault for use with our system, however we have encountered some serious errors during normal use.

After some time working correctly, Vault has started throwing errors when attempting to check certain files in. Specifically:

Upload for item $/Code/login.php failed too many times, aborting transaction.
Please verify your network settings using the Options dialog under the Tools menu in the Vault GUI Client.
The specific error was "The server returned an unknown error header: VaultFileUpload.aspx encountered: FailDeltaApply"
An exception was encountered during the transaction. Exception: The server returned an unknown error header: VaultFileUpload.aspx encountered: FailDeltaApply at VaultClientOperationsLib.ClientInstance.UploadItem(ChangeSetItem item, String txID, Byte[]& streamBuffer, Int32& bytesWrittenThisFile, Boolean bIsImport)
at VaultClientOperationsLib.UploadThread.ProcessCommand(UploadThreadCommand command, UploadThreadCommandResult& outputResult)

With the following in the sgvault.log:

----3/7/2007 11:49:32 AM admin--ryan.icom.lan(192.168.0.44)--SSL Disabled VaultFileUpload.aspx encountered: FailDeltaApply

We have tried various things to fix this, including deleting the local cache (which occasionally works as a temporary measure to allow a single check-in operation to occur, but does not prevent the same file from breaking on the next check-in), turning on chunked encoding (which makes no difference), and even uninstalling and reinstalling the entire Vault Server application and starting over with a fresh database and repository structure (which lost all our previous versions, and provided no relief).

Other files of very similar structure and content will check in and out without issue. All the problem files so far have been PHP source files of under 5000 lines, some being less than 100.

What else can we try? If we can not resolve this, we will be forced to move to a different source control solution, which would be disappointing since we were quite happy with Vault, and on the verge of approving and purchasing it.

ICOM · Post by **ICOM** » Wed Mar 07, 2007 1:22 pm

Just for the heck of it, I also tried with the server set to force Unix EOL, None EOL, and with the client set to force Unix or Do not override. No combination of these helped.

lbauer · Post by **lbauer** » Wed Mar 07, 2007 2:20 pm

FailDeltaApply means the Vault server could not apply the delta (difference) to the file. The client doesn't send entire files, just the difference from the baseline.

This is often a network error. Could anti-virus, anti-hacking, or firewall software be interfereing with the upload of the PHP files?

Can you perform the same operation with a Vault Client on the Vault Server machine itself, logging in with "localhost" as the Vault server name?

Are there any corresponding error messages in the Vault Server log? It's sgvault.log and is in the %windir%\temp\sgvault directory on the Vault server machine.

ICOM · Post by **ICOM** » Wed Mar 07, 2007 2:46 pm

The server has no Firewalls, AV, or other things that interfere with network traffic. The client has no Firewall or other things, and still produces the error with the AV disabled.

Checking in the file locally on the server works, but is hardly a satisfactory workaround.

I pasted the appropriate line from server log in my first post.

ICOM · Post by **ICOM** » Thu Mar 08, 2007 11:24 am

This is still occurring, even on other workstations. We have completely uninstalled the AV just to test, and still get the error. The communication is over a straight forward LAN, with only a single switch between the client and the server, no other networking equipment to munge the packets.

Since this only happens on select files, while others will check in repeatedly just fine, we believe it's not a network problem, and can only conclude this is a problem with Vault.

ICOM · Post by **ICOM** » Thu Mar 08, 2007 12:26 pm

In an attempt to eliminate any possibility of network issues being to blame, we have installed a cert on our server, and tried the operation over SSL. The idea being that any network issues that affected port 80 traffic would not affect SSL traffic the same way.

Surprisingly, the server still reports a FailDeltaApply error, with the server log now containing:

Code: Select all

----3/8/2007 11:23:33 AM     admin--icom-ws17.icom.lan(192.168.0.39)--SSL Enabled	VaultFileUpload.aspx encountered:  FailDeltaApply

ICOM · Post by **ICOM** » Thu Mar 08, 2007 12:51 pm

Here's the one of the files in question, in case this helps. (note: I had to add the .txt to upload it)

lbauer · Post by **lbauer** » Thu Mar 08, 2007 12:59 pm

Checking in the file locally on the server works

This shows that the Vault server and client are working properly, when you take out the network factor. It also rules out a problem iwth the file itself.

Perhaps "network problem" isn't the best term to use. Something in the environment between the client and server is causing the communication to be altered.

Some other things to try:

Can you upload these same files to our demo server: vaultdemo.sourcegear.com? Use the account, vaulttester, password demo.

Try changing the duplex mode of the network card on the client and/or server machines. You could try half-duplex or full-duplex -- just so the setting is different from what you are using now. Sometimes the auto sensing features of some network adapters do not correctly identify the connection.

ICOM · Post by **ICOM** » Thu Mar 08, 2007 1:55 pm

A preliminary test using your server seems to work, though it often seems only to take effect after some number of successful check-ins happen first.

Changing the duplex of the server NIC made no difference, and since we are seeing 0 other network issues, including when checking in very similar files to the same repository, I didn't expect it to..

This also would not explain why clearing the local cache will occasionally cause the next checkin to succeed, would it?

ICOM · Post by **ICOM** » Thu Mar 08, 2007 2:16 pm

My suspicion is that something to do with regular maintenance of the server, such as a Windows or .NET hotfix has conflicted with Vault, causing this issue to start appearing suddenly, and that it's tied to the way the DeltaApply works in such a way that it only affects a portion of the files.

Of course, that's not very consistant with it working locally, and there's no real way for us to test that here...

ICOM · Post by **ICOM** » Fri Mar 09, 2007 11:43 am

I may have been premature in saying that similar files work. It's possible that the ones that "work" are in fact going through as "undo checkout" instead of having changes applied. I think I remember files checking in correctly with changes, but at this point all actual changes seem to be failing.

Is there any way to configure the server to avoid doing a Delta? I tried removing *.php from the Mergeable files list, but that does not appear to have helped.

ICOM · Post by **ICOM** » Mon Mar 12, 2007 10:23 am

Ok, we just installed the server component on a different server running Win2k instead of Win2003, and SQL 2000 instead of SQL 2005.

We still get the same error.

From 3 workstations to 2 servers.. I think we can rule out pretty much any hardware causes..

So what's left? I notice your test server is running an old version of Vault Server. Perhaps a bug was introduced in 3.5.1.4786?

Post by **jclausius** » Mon Mar 12, 2007 10:43 am

We need to eliminate what end is causing the problem.

Did you try the test at vaultdemo.sourcegear.com? I created $/ICOM under the "Second Repository". The file you uploaded works with the Vault GUI client 3.5.1. Can you try the same test with your Vault GUI client?

Also, is it possible the "baseline" files retrieved by the client into the _sgvault directory are somehow being edited? It is important that those files remain intact as originally downloaded. But sometimes people accidentally edit them with a massive Find / Replace or other operations. The end result is the deltas sent by the client may fail to apply on the server side.

ICOM · Post by **ICOM** » Mon Mar 12, 2007 11:01 am

No, I set up the clients to store the _sgvault files outside the working directory specifically to avoid such situations, and during testing have avoided all unnesessary actions. Just doing:

Check out file
Edit file (add a comment)
Save file
Check in file

All in the span of 20 seconds or so.
This was even done on a client with the AV removed, as stated above, so nothing should be touching those files.

Testing with your repository again results in no errors.

Immediately disconnecting, connecting to ours, doing a checkout and checkin of the same file (telling it not to overwrite, to preserve changes made in your repository)... Worked..?

Heh. Trying to check in a different problem file still gives us the error though.

So it's definately file or content specific in some way.

ICOM · Post by **ICOM** » Mon Mar 12, 2007 11:04 am

Tried the same with the other problem file, and it worked on your repository, but still failed on ours afterwards.