Repeated Vault application log errors restarting w3wp.exe

[Figbashing]

Hi there,

A few days ago we started getting repeated intermittent bogdowns on our webserver, causing very slow page loads--two minutes or more so we may as well be down. In troubleshooting, our team has found that the day before all this started, we began getting a ton of repeated Vault-related errors in Windows application event logs, a set of 5 all together (some of which may be repeated).

The errors appear to have something to do with a failed database login (we have made no changes along those lines) and end up causing a crash of the World Wide Web Publishing Service. One set of these errors is pasted below, with tech support info from the client under that.

I am seeing similar language to these errors in vaultnotifyservice.txt; nothing in vaultindexservice.txt or vaultshadowfolderservice.txt or sgvault.log that jumps out at me (though who knows what I am overlooking).

I am hoping Vault support can help us understand what's going on here...

Date and Time: 3/11/2016 1:02:41 PM [may repeat 2-4 times; also sometimes does not trigger the following errors but if it does, all of the following errors are present together]
Source: MSSQLServer
Level: Information
Event ID: 18456
--
Login failed for user 'sgvaultuser'. Reason: Failed to open the explicitly specified database. [CLIENT: <local machine>]

Date and Time: 3/11/2016 1:02:41 PM
Source: ASP.NET 4.0.30319.0
Level: Error
Event ID: 1325
--
An unhandled exception occurred and the process was terminated.
Application ID: DefaultDomain
Process ID: 9784
Exception: System.Runtime.Serialization.SerializationException
Message: Type 'VaultLib.VaultDbException' in Assembly 'VaultLib, Version=6.1.0.531, Culture=neutral, PublicKeyToken=a74d7badb9387c36' is not marked as serializable.
StackTrace: at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitSerialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter, SerializationBinder binder)
at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.Serialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter, SerializationBinder binder)
at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize(Object graph, Header[] inHeaders, __BinaryWriter serWriter, Boolean fCheck)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph, Header[] headers, Boolean fCheck)
at System.Runtime.Remoting.Channels.CrossAppDomainSerializer.SerializeObject(Object obj, MemoryStream stm)
at System.AppDomain.Serialize(Object o)
at System.AppDomain.MarshalObject(Object o)

Date and Time: 3/11/2016 1:02:41 PM
Source: .NET Runtime
Level: Error
Event ID: 1026
--
Application: w3wp.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: VaultLib.VaultDbException
Stack:
at VaultIndexService.VaultIndexServiceSQL.GetRepositoryIndexSettings(Int32, VaultLib.RepositoryIndexInfo ByRef)
at VaultIndexService.VaultIndexManager.MoveItemToProcessRequestList(VaultIndexService.RepositoryIndexStateChange)
at VaultIndexService.VaultIndexManager.RepositoryIndexStateChgProc()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart()

Date and Time: 3/11/2016 1:02:42 PM
Source: Application Error
Level: Error
Event ID: 1000
--
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: KERNELBASE.dll, version: 6.1.7601.19160, time stamp: 0x56bcd73c
Exception code: 0xe0434352
Fault offset: 0x000000000000965d
Faulting process id: 0x2638
Faulting application start time: 0x01d17bc034549332
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 7281a861-e7b3-11e5-a84a-001ec9b1ecbc

Date and Time: 3/11/2016 1:01:42 PM [repeats twice]
Source: Windows Error Reporting
Level: Error
Event ID: 1001
--
Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0
Problem signature:
P1: w3wp.exe
P2: 7.5.7601.17514
P3: 4ce7afa2
P4: VaultIndexService
P5: 6.1.0.531
P6: 50ac138e
P7: 4b
P8: 19e
P9: VaultLib.VaultDbException
P10:
Attached files:
These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_19bb2d2732a5a945f6202d96f402be4e997850_2aaae3d9
Analysis symbol:
Rechecking for solution: 0
Report Id: 7281a861-e7b3-11e5-a84a-001ec9b1ecbc
Report Status: 4

Tech support info:
Client Information
Vault Standard Client Version: 6.1.0.531
.NET Framework Version: 4.0.30319.34209
Operating System: Microsoft Windows Server 2008 R2 Standard
Service Pack: 1.0
OS Version: 6.1.7601
Total Physical Memory: 4 GB
Time Zone: (UTC-05:00) Eastern Time (US & Canada)
Server Information
Server Version: 6.1.0.531
.NET Framework Version: 4.0.30319.34209
Operating System: Microsoft Windows Server 2008 R2 Standard
Service Pack: 1.0
OS Version: 6.1.7601
Timezone: (UTC-05:00) Eastern Time (US & Canada)
SQL Version: Microsoft SQL Server 2008 R2 (SP1) - 10.50.2550.0 (X64)
License Information
3 serial number(s):
1 of 3: 10 full users, expiring 7/10/2012
2 of 3: 4 full users, permanent
3 of 3: 1 full users, permanent

[Beth]

Could you send me your Vault server log found on the Vault server at %windir%\temp\sgvault\sgvault.log, and the Vault Notify log from %windir%\temp\sgvaultnotify\vaultnotifyservice.txt? You can send send those in an an email to support at sourcegear.com. Please include a link to the forum thread.


HS: 240378