Critical Error downloading file! The remote host closed the

[jvel]

Hi folks,

We have been happily using Vault 8.0.1.299 for many many months. But now one of the workstations is unable to use the Vault client (neither vaultguiclient.exe or vault.exe).

After logging in and choosing a repository, the GUI client shows "Retrieving repository structure information from the server" for about 20 seconds (this is normal) and then "An error has occurred. The error message is: The file exists.". (There is no mention of which file it refers to.)

The server's vault log shows this:

----07/08/2017 12:29:32 build--89.0.1.152(89.0.1.152)--SSL Disabled Login
----07/08/2017 12:29:53 build--2002:5900:195::5900:195(2002:5900:195::5900:195)--SSL Disabled Critical Error downloading file! The remote host closed the connection. The error code is 0x800704CD.
at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at VaultLib.VaultStreamCopy.CopyStream(Stream sIn, Stream sOut, Byte[] buf, Int32 nBufSize)
at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e)
----07/08/2017 12:29:53 build--2002:5900:195::5900:195(2002:5900:195::5900:195)--SSL Disabled System.Web.HttpException (0x800704CD): The remote host closed the connection. The error code is 0x800704CD.
at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at VaultLib.VaultStreamCopy.CopyStream(Stream sIn, Stream sOut, Byte[] buf, Int32 nBufSize)
at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e) at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at VaultLib.VaultStreamCopy.CopyStream(Stream sIn, Stream sOut, Byte[] buf, Int32 nBufSize)
at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e)
----07/08/2017 12:29:53 build--2002:5900:195::5900:195(2002:5900:195::5900:195)--SSL Disabled Uncaught Exception: Exception of type 'System.Web.HttpUnhandledException' was thrown.
Inner Exception: The remote host closed the connection. The error code is 0x800704CD.
----07/08/2017 12:29:53 build--2002:5900:195::5900:195(2002:5900:195::5900:195)--SSL Disabled at System.Web.UI.Page.HandleError(Exception e)
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
at System.Web.UI.Page.ProcessRequest()
at System.Web.UI.Page.ProcessRequest(HttpContext context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
Inner Exeption Stack Trace: at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at VaultLib.VaultStreamCopy.CopyStream(Stream sIn, Stream sOut, Byte[] buf, Int32 nBufSize)
at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e)
at System.Web.UI.Control.LoadRecursive()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
----07/08/2017 12:30:11 build--2002:5900:195::5900:195(2002:5900:195::5900:195)--SSL Disabled Login
----07/08/2017 12:30:11 build--89.0.1.152(89.0.1.152)--SSL Disabled Logout

and the Windows event log on the server shows this:

Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 07/08/2017 12:35:11
Event time (UTC): 07/08/2017 11:35:11
Event ID: e71ce8cd88584e459048ca4768ba4a81
Event sequence: 75836
Event occurrence: 61
Event detail code: 0

Application information:
Application domain: /LM/W3SVC/1/ROOT/VaultService-1-131461586253916970
Trust level: Full
Application Virtual Path: /VaultService
Application Path: C:\inetpub\wwwroot\VaultService\
Machine name: VAULT

Process information:
Process ID: 2896
Process name: w3wp.exe
Account name: IIS APPPOOL\VaultAppPool

Exception information:
Exception type: HttpException
Exception message: The remote host closed the connection. The error code is 0x800703E3.
at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at VaultLib.VaultStreamCopy.CopyStream(Stream sIn, Stream sOut, Byte[] buf, Int32 nBufSize)
at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e)
at System.Web.UI.Control.LoadRecursive()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)



Request information:
Request URL: http://vault.arden.group/VaultService/V ... 44217cde48
Request path: /VaultService/VaultFileDownload.aspx
User host address: 2002:5900:195::5900:195
User:
Is authenticated: False
Authentication Type:
Thread account name: IIS APPPOOL\VaultAppPool

Thread information:
Thread ID: 18
Thread account name: IIS APPPOOL\VaultAppPool
Is impersonating: False
Stack trace: at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at VaultLib.VaultStreamCopy.CopyStream(Stream sIn, Stream sOut, Byte[] buf, Int32 nBufSize)
at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e)
at System.Web.UI.Control.LoadRecursive()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)


Custom event details:

There is nothing suspicious in the event log of the client workstation. I have rebooted the client workstation. I have cleared the cache. I have reinstalled the Vault client. None of those actions helped.

I have confirmed that the problem is specific to the workstation, and not to the user. All other workstations (15 to 20 of them, as I recall) are fine.

Can you offer any guidance on what else I could try?

This may be relevant: I moved the Vault service to a new server last week, because the old server was coming out of warranty. The workstation in question was fine for around 2 days after the migration.

The workstation is running Windows 7 Professional SP1 (as are one or two of the good workstations) and the server is Windows Server 2012 Datacenter.

I appreciate any advice you can offer.

Regards,
John

[Beth]

The first step with the error "Critical Error downloading file!" is to try and IIS reset on the Vault server. That's because the most common cause is something with the server-side cache. To do this, run a command prompt as administrator and type iisreset. This will disconnect users, so you may need to warn them ahead of time.

If you have other websites on the the Vault server that can't go through a reset, you could alternatively try recycling each of the Vault App Pools.

[jvel]

Hi Beth,

Thanks for the quick response.

I have rebooted the Vault server (which I think does everything that an IISRESET would), and also cleared the cache on the workstation and rebooted the workstation. But the behaviour is unchanged - still the error "Critical Error downloading file! The remote host closed the connection. The error code is 0x800704CD."

One extra thing that I noticed: we have six Vault repositories and one of them is very small. I am able to connect to this repository without any error (and it takes less than 2 seconds to display the folder structure), but the error does occur for all the other repositories.

We have also disabled the firewall and the anti-virus on both server and workstation, with no beneficial effect.

I think my next task will be to check the network connection properties. Both server and workstation are virtual machines running under Hyper-V, on different servers. (The workstation has always been virtual, but until recently the Vault service was on a physical server.)

As I recall, there is some way to enable logging on the Vault client. Do you think this is worth trying, and if so please could you remind me how to do it?

Regards,
John

[Beth]

To turn on client logging, open Vault and login. Then press Ctrl - Alt - Shift - F12. Turning on Debug logging on the server might also help. That setting is in the Vault admin web page under the Logging link.

[jvel]

Since my last response, I have reverted a few Windows updates which appeared around the time of the first failure, but this did not help.

Also our IT guys have checked the network adaptor settings on the workstation, the server, and the Hyper-V servers, and declared them to be ok.

Also I have revived our old Vault server (using a copy of our Vault databases, not the live ones) and discovered that the same problem occurs when I direct Vault to this server (but again, only for this one workstation, so it seems).

I have attached the VaultGUIClient.txt in case it contains any clues, but I suspect not.

Regards,
John

[Beth]

Try running Disk Cleanup on your machine and allow it to clean up any unnecessary temp files. Then try Vault again. Any change?

Your client log file you sent still wasn't in debug level logging. When you successfully start debug logging on a client, you will see a notification in the bottom left corner of the Vault windows saying it started and where the log is found.

[jvel]

Disk Cleanup did not help. For good measure I also tried System File Checker (sfc) and Check Disk (chkdsk).

I have attached a couple more logs from the client, hopefully done properly this time.

Regards,
John

[jclausius]

On the client, can you check the logged in user's %TEMP% directory?

Usually the 'File Exists' exception occurs when attempting to get a temp file and the drive where the %TEMP% directory is either out of disk space or %TEMP% has 65K or more files.

[jvel]

Thank you Jeff, that's exactly what it was!

The %TEMP% folder contained 65,547 files - most of them created on the same day and they appear to be temporary binaries from our IDE.

I had thought that Disk Cleanup removed such files, but clearly not because I just tried it again. So I just deleted them manually.

Thanks to you and Beth for the good advice.

Regards,
John

[jclausius]

Great! We're glad you're back up and running.

Thank you for using SourceGear Vault.