Users are reporting multiple failures

[alan.bowens]

Hello,

Many of my colleagues have been reporting problems with using Vault today. They are reporting slow performance, being forcibly logged out, an inability to see the correct status of items in Vault, and being unable to log in. I have attached the server log file, please advise how we can fix this.

Thanks

Alan

[lbauer]

The server log you sent had a number of errors in a pattern we've seen before. Basically an interrupted download creates inconsistencies in the Vault Server in-memory cache, which causes more errors:

(username and IP addresses removed):

Code: Select all

----7/10/2012 7:40:00 AM     --SSL Disabled Critical Error downloading file! The remote host closed the connection. The error code is 0x80072746.

----7/10/2012 7:40:00 AM     SSL Disabled	System.Web.HttpException: The remote host closed the connection. The error code is 0x80072746.

----7/10/2012 7:41:19 AM    --SSL Disabled	System.Data.SqlClient.SqlException: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.

---7/10/2012 9:16:09 AM     SSL Disabled	GetRepositoryStructure returned: FailPermissionDenied 
----7/10/2012 9:16:09 AM     --SSL Disabled	System.InvalidCastException: Specified cast is not valid

AND

Code: Select all

----2/8/2012 11:51:06 AM     --SSL Disabled	Critical Error downloading file! The remote host closed the connection. The error code is 0x80072746.
   at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status, Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushResponse(Boolean finalFlush)
   at System.Web.HttpResponse.Flush(Boolean finalFlush)
   at System.Web.HttpResponse.Flush()
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
   at VaultServiceSQL.VaultSqlSCC.GetDeltaChainForFullFile(VaultSqlConn conn, Int64 nObjVerID, Int32 nBufSize, String strFileToken, Stream s, ListDictionary ldDataStreams, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultFileCacheManager.GetDeltaChainForFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nTargetObjVerID, Int32 nBufSize, String strFileToken, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.CreateFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nObjVerID, String strFileToken, Boolean& deltaIfAvailable, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.DeltaCreate(VaultSqlConn vsConn, Int32 nRepID, Int64 nBaseObjVerID, Int64 nTargetObjVerID, String strFileToken, Stream sDeltaStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultServiceAPI.GetVaultFile(Int32 nRepID, VaultIntDnldItem& vidi, String strFileToken, Stream s, Boolean bSeekToOrigin, Boolean bFolderExportCalcCRC, UInt32& nCRC)
   at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e) 
----2/8/2012 11:51:06 AM     --SSL Disabled	System.Web.HttpException: The remote host closed the connection. The error code is 0x80072746.
   at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status, Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushResponse(Boolean finalFlush)
   at System.Web.HttpResponse.Flush(Boolean finalFlush)
   at System.Web.HttpResponse.Flush()
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
   at VaultServiceSQL.VaultSqlSCC.GetDeltaChainForFullFile(VaultSqlConn conn, Int64 nObjVerID, Int32 nBufSize, String strFileToken, Stream s, ListDictionary ldDataStreams, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultFileCacheManager.GetDeltaChainForFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nTargetObjVerID, Int32 nBufSize, String strFileToken, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.CreateFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nObjVerID, String strFileToken, Boolean& deltaIfAvailable, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.DeltaCreate(VaultSqlConn vsConn, Int32 nRepID, Int64 nBaseObjVerID, Int64 nTargetObjVerID, String strFileToken, Stream sDeltaStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultServiceAPI.GetVaultFile(Int32 nRepID, VaultIntDnldItem& vidi, String strFileToken, Stream s, Boolean bSeekToOrigin, Boolean bFolderExportCalcCRC, UInt32& nCRC)
   at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e)    at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status, Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushResponse(Boolean finalFlush)
   at System.Web.HttpResponse.Flush(Boolean finalFlush)
   at System.Web.HttpResponse.Flush()
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
   at VaultServiceSQL.VaultSqlSCC.GetDeltaChainForFullFile(VaultSqlConn conn, Int64 nObjVerID, Int32 nBufSize, String strFileToken, Stream s, ListDictionary ldDataStreams, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultFileCacheManager.GetDeltaChainForFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nTargetObjVerID, Int32 nBufSize, String strFileToken, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.CreateFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nObjVerID, String strFileToken, Boolean& deltaIfAvailable, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.DeltaCreate(VaultSqlConn vsConn, Int32 nRepID, Int64 nBaseObjVerID, Int64 nTargetObjVerID, String strFileToken, Stream sDeltaStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultServiceAPI.GetVaultFile(Int32 nRepID, VaultIntDnldItem& vidi, String strFileToken, Stream s, Boolean bSeekToOrigin, Boolean bFolderExportCalcCRC, UInt32& nCRC)
   at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e)
----2/8/2012 11:52:19 AM     --SSL Disabled	Critical Error downloading file! The remote host closed the connection. The error code is 0x80072746.
   at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status, Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushResponse(Boolean finalFlush)
   at System.Web.HttpResponse.Flush(Boolean finalFlush)
   at System.Web.HttpResponse.Flush()
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
   at VaultServiceSQL.VaultSqlSCC.GetDeltaChainForFullFile(VaultSqlConn conn, Int64 nObjVerID, Int32 nBufSize, String strFileToken, Stream s, ListDictionary ldDataStreams, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultFileCacheManager.GetDeltaChainForFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nTargetObjVerID, Int32 nBufSize, String strFileToken, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.CreateFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nObjVerID, String strFileToken, Boolean& deltaIfAvailable, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.DeltaCreate(VaultSqlConn vsConn, Int32 nRepID, Int64 nBaseObjVerID, Int64 nTargetObjVerID, String strFileToken, Stream sDeltaStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultServiceAPI.GetVaultFile(Int32 nRepID, VaultIntDnldItem& vidi, String strFileToken, Stream s, Boolean bSeekToOrigin, Boolean bFolderExportCalcCRC, UInt32& nCRC)
   at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e) 
----2/8/2012 11:52:19 AM     --SSL Disabled	System.Web.HttpException: The remote host closed the connection. The error code is 0x80072746.
   at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status, Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushResponse(Boolean finalFlush)
   at System.Web.HttpResponse.Flush(Boolean finalFlush)
   at System.Web.HttpResponse.Flush()
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
   at VaultServiceSQL.VaultSqlSCC.GetDeltaChainForFullFile(VaultSqlConn conn, Int64 nObjVerID, Int32 nBufSize, String strFileToken, Stream s, ListDictionary ldDataStreams, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultFileCacheManager.GetDeltaChainForFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nTargetObjVerID, Int32 nBufSize, String strFileToken, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.CreateFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nObjVerID, String strFileToken, Boolean& deltaIfAvailable, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.DeltaCreate(VaultSqlConn vsConn, Int32 nRepID, Int64 nBaseObjVerID, Int64 nTargetObjVerID, String strFileToken, Stream sDeltaStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultServiceAPI.GetVaultFile(Int32 nRepID, VaultIntDnldItem& vidi, String strFileToken, Stream s, Boolean bSeekToOrigin, Boolean bFolderExportCalcCRC, UInt32& nCRC)
   at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e)    at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status, Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushResponse(Boolean finalFlush)
   at System.Web.HttpResponse.Flush(Boolean finalFlush)
   at System.Web.HttpResponse.Flush()
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
   at VaultServiceSQL.VaultSqlSCC.GetDeltaChainForFullFile(VaultSqlConn conn, Int64 nObjVerID, Int32 nBufSize, String strFileToken, Stream s, ListDictionary ldDataStreams, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultFileCacheManager.GetDeltaChainForFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nTargetObjVerID, Int32 nBufSize, String strFileToken, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.CreateFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nObjVerID, String strFileToken, Boolean& deltaIfAvailable, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.DeltaCreate(VaultSqlConn vsConn, Int32 nRepID, Int64 nBaseObjVerID, Int64 nTargetObjVerID, String strFileToken, Stream sDeltaStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultServiceAPI.GetVaultFile(Int32 nRepID, VaultIntDnldItem& vidi, String strFileToken, Stream s, Boolean bSeekToOrigin, Boolean bFolderExportCalcCRC, UInt32& nCRC)
   at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e)

The short-term solution is to restart the Vault service to clear the in-memory server cache. You can do this with an iisreset command or by recycling the Vault App Pool and restarting the Vault web site.

We have a special Vault 5.1.2. build that addresses some of the issues you are seeing with the "cascading errors." We'd like to see if the changes we applied improve your situation. Email support at sourcegear.com, Attn: Linda. Please include a link to this forum post. I'll provide you with the builds that may resolve this issue.

[Beth]

Update: The new solution for this issue is to upgrade to Vault 6.1.0 or higher. The special build is no longer available.