Vault Command Line Issues

[Aaron Young]

We have an automated build process which utilizes the Vault Command Line Client, this process has been in place for well over a decade and over the last several months we're getting more and more instances of the Vault Command Line Client hanging up, it's gotten to a point where it's very hit or miss as to whether the build will complete.

Some background on the process:

Each build the local source files/folders are deleted prior to getting the latest from Source Control.
The Solutions/Projects being downloaded are quite large.

What I've tried:

1. Breaking down the Get request into several requests along the directory structure.

2. Clearing the client-side cache as suggested in: viewtopic.php?f=13&t=11513

3. Restarting the Build host machine

4. Restarting the Vault Service

All these things have at one point appeared to resolve the issue for a build or two, but not consistently.

If I replicate the Get operation being performed by using the Vault GUI client, I see the following in the message window after several minutes:

An error occurred reading from the download stream ("Unable to read data from the transport connection:
An existing connection was forcibly closed by the remote host."), retrying...

The client then appears to get stuck in an endless loop of downloading the same files over and over without further errors reported which I assume is what's happening with the Command Line Client.

I've turned on Debug logging and it basically confirms the behavior I've described in the UI client (I can provide said log if needed.)

I'm at a loss and can think of nothing else to try, so I'm hoping for some insight.

Regards,

- Aaron.

[Beth]

The error

An error occurred reading from the download stream ("Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host."), retrying...

is usually something network related that is cutting off the communication.

The Vault client will try the download three times before giving up.

Do any other users have problems performing the same Get that the build machine performs?

Can you use the Vault CLC for the Get from other machines?

Try also using the Vault CLC on the Vault server, and use localhost as the server to connect to. Can it perform the same Get there?

[Aaron Young]

Beth,

Thank you for the response and suggestions.

I looked at the Vault log from yesterday's DEBUG test and picked a file at random and counted 20 attempts to download (apparently successfully - no errors logged), so quitting after 3 attempts didn't seem to kick in.

Today, I tested on a couple of other machines and ran into the same problem, I did not have a problem on my own machine however.

On the problem machines, I noticed that the network activity seemed pretty erratic and almost restricted with very low throughput. We tried removing some firewall/inspection rules to no avail.

Thinking we'd hit another dead-end, I decided to try disabling the Antivirus on the Build machine and presto, it worked! I verified the results several times, so this is definitely the culprit.

One difference between the various machines tried was the type of AV software used, for various reasons we use 3 different kinds; Microsoft Security Essentials/Windows Defender, WebRoot and Symantec. The problem machines were running one of the first 2 products and my own machine was running Symantec.

We installed Symantec on the Build machine and it still works.

So there you have it, AV software was causing enough interference with Vault to produce the problem.

Regards,

- Aaron.