Add label times out when Database mirroring is enabled

[IanG]

Hi,
We having a problem with our Fortress setup where when we try to create a new label the operation times out.
The label appears to be there in the repositiory but having recived an error message we are not sure we can trust it.
The relevant part of the sgvault.log is included bellow:

Code: Select all

----4/24/2009 1:58:31 PM     admin--notebook32.paragon.pgn(192.168.97.134)--SSL Disabled	System.Data.SqlClient.SqlException: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.ReadBuffer()
   at System.Data.SqlClient.TdsParserStateObject.ReadByte()
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)
   at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
   at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransaction(TransactionRequest transactionRequest, String name, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
   at System.Data.SqlClient.SqlInternalTransaction.Commit()
   at System.Data.SqlClient.SqlTransaction.Commit()
   at VaultServiceSQL.VaultSqlConn.CommitTransaction()    at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.ReadBuffer()
   at System.Data.SqlClient.TdsParserStateObject.ReadByte()
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)
   at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransactionYukon(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
   at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransaction(TransactionRequest transactionRequest, String name, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
   at System.Data.SqlClient.SqlInternalTransaction.Commit()
   at System.Data.SqlClient.SqlTransaction.Commit()
   at VaultServiceSQL.VaultSqlConn.CommitTransaction()
----4/24/2009 1:58:31 PM     admin--notebook32.paragon.pgn(192.168.97.134)--SSL Disabled	Rolling Back a transaction   at VaultServiceSQL.VaultSqlConn.RollbackTransaction()
   at VaultServiceSQL.VaultSqlConn.CloseConn()
   at VaultServiceAPILib.VaultServiceAPI.AddLabel(Int32 nUserID, Int32 nRepID, VaultLabelItem& vLabel)
   at VaultService.VaultService.AddLabel(Int32 nRepID, VaultLabelItem& vLabel)
   at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
   at System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Web.Services.Protocols.LogicalMethodInfo.Invoke(Object target, Object[] values)
   at System.Web.Services.Protocols.WebServiceHandler.Invoke()
   at System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest()
   at System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   at System.Web.HttpApplication.ApplicationStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData)
   at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)
   at System.Web.HttpRuntime.ProcessRequestNoDemand(HttpWorkerRequest wr)
   at System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr ecb, Int32 iWRType)
 
----4/24/2009 1:58:31 PM     admin--notebook32.paragon.pgn(192.168.97.134)--SSL Disabled	System.InvalidOperationException: This SqlTransaction has completed; it is no longer usable.
   at System.Data.SqlClient.SqlTransaction.ZombieCheck()
   at System.Data.SqlClient.SqlTransaction.Rollback()
   at VaultServiceSQL.VaultSqlConn.RollbackTransaction()    at System.Data.SqlClient.SqlTransaction.ZombieCheck()
   at System.Data.SqlClient.SqlTransaction.Rollback()
   at VaultServiceSQL.VaultSqlConn.RollbackTransaction()
----4/24/2009 1:58:31 PM     admin--notebook32.paragon.pgn(192.168.97.134)--SSL Disabled	AddLabel returned: FailDBFault 
----4/24/2009 1:59:06 PM     colin--desktop4.paragon.pgn(192.168.97.156)--SSL Disabled	Logout 
----4/24/2009 1:59:12 PM     colin--desktop4.paragon.pgn(192.168.97.156)--SSL Disabled	Login 
----4/24/2009 2:00:53 PM     admin--notebook32.paragon.pgn(192.168.97.134)--SSL Disabled	Login 
----4/24/2009 2:01:11 PM     admin--notebook32.paragon.pgn(192.168.97.134)--SSL Disabled	Critical Error downloading file! The remote host closed the connection. The error code is 0x80072746.
   at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status, Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushResponse(Boolean finalFlush)
   at System.Web.HttpResponse.Flush(Boolean finalFlush)
   at System.Web.HttpResponse.Flush()
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
   at VaultServiceSQL.VaultSqlSCC.GetDeltaChainForFullFile(VaultSqlConn conn, Int64 nObjVerID, Int32 nBufSize, String strFileToken, Stream s, ListDictionary ldDataStreams, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultFileCacheManager.GetDeltaChainForFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nTargetObjVerID, Int32 nBufSize, String strFileToken, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.CreateFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nObjVerID, String strFileToken, Boolean& deltaIfAvailable, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.DeltaCreate(VaultSqlConn vsConn, Int32 nRepID, Int64 nBaseObjVerID, Int64 nTargetObjVerID, String strFileToken, Stream sDeltaStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultServiceAPI.GetVaultFile(Int32 nRepID, VaultIntDnldItem& vidi, String strFileToken, Stream s, Boolean bSeekToOrigin, Boolean bFolderExportCalcCRC, UInt32& nCRC)
   at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e) 
----4/24/2009 2:01:11 PM     admin--notebook32.paragon.pgn(192.168.97.134)--SSL Disabled	System.Web.HttpException: The remote host closed the connection. The error code is 0x80072746.
   at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status, Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushResponse(Boolean finalFlush)
   at System.Web.HttpResponse.Flush(Boolean finalFlush)
   at System.Web.HttpResponse.Flush()
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
   at VaultServiceSQL.VaultSqlSCC.GetDeltaChainForFullFile(VaultSqlConn conn, Int64 nObjVerID, Int32 nBufSize, String strFileToken, Stream s, ListDictionary ldDataStreams, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultFileCacheManager.GetDeltaChainForFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nTargetObjVerID, Int32 nBufSize, String strFileToken, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.CreateFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nObjVerID, String strFileToken, Boolean& deltaIfAvailable, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.DeltaCreate(VaultSqlConn vsConn, Int32 nRepID, Int64 nBaseObjVerID, Int64 nTargetObjVerID, String strFileToken, Stream sDeltaStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultServiceAPI.GetVaultFile(Int32 nRepID, VaultIntDnldItem& vidi, String strFileToken, Stream s, Boolean bSeekToOrigin, Boolean bFolderExportCalcCRC, UInt32& nCRC)
   at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e)    at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status, Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
   at System.Web.Hosting.ISAPIWorkerRequest.FlushResponse(Boolean finalFlush)
   at System.Web.HttpResponse.Flush(Boolean finalFlush)
   at System.Web.HttpResponse.Flush()
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
   at VaultServiceSQL.VaultSqlSCC.GetDeltaChainForFullFile(VaultSqlConn conn, Int64 nObjVerID, Int32 nBufSize, String strFileToken, Stream s, ListDictionary ldDataStreams, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultFileCacheManager.GetDeltaChainForFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nTargetObjVerID, Int32 nBufSize, String strFileToken, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.CreateFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nObjVerID, String strFileToken, Boolean& deltaIfAvailable, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultDelta.DeltaCreate(VaultSqlConn vsConn, Int32 nRepID, Int64 nBaseObjVerID, Int64 nTargetObjVerID, String strFileToken, Stream sDeltaStream, RunningCRC32 crc, UInt32& nCRC)
   at VaultServiceAPILib.VaultServiceAPI.GetVaultFile(Int32 nRepID, VaultIntDnldItem& vidi, String strFileToken, Stream s, Boolean bSeekToOrigin, Boolean bFolderExportCalcCRC, UInt32& nCRC)
   at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e)

I have increased the SQLCommandTimeout to 3600 seconds from 360 and restarted iis but this dosn't appear to have any effect as it still times out after about 6 mins.
The only way I have been able to get it to work is by turning off database mirroring which we use to provide a very upto date backup should the main server go down (we also to nightly backups).
We run periodic database maintenence, recompute stats every night, defrag indexes every two weeks and rebuild indexes every two months and we are due to defrag indexes this weekend.
I can understand why mirroring would slow the database down but why is the timeout setting aparently being ignored? is there anything else we could do to improve label performance? Would exporitng and removing unused branches (which get inclued in the label) help?
we are running fortress 1.1.4 and the database is just over 2GB in size.

Rgrds Ian

[Beth]

Mirroring isn't something we test with or support, so I don't have any known answer why there might be a problem.

Some things you might look at for a root cause would be your SQL server logs and Event Viewer logs. Make sure your mirror is successfully happening. I think there are instances where mirroring issues would make the SQL server unresponsive to other requests. Check how many server resources are being used when you have mirroring turned on and you are trying to make a label.

The way to check if your label was made on the things you want it on is to perform a Show Label, then select the label and click on Action - View to see what was all included in the label. If anything needs to be changed, you can use Label Promotion to change what is in a label. The Label Promotion feature is also in the Show Labels window under Label - Label Promotion.

[IanG]

Thanks for the reply, as for server resouces well when a label is being added one core on the serever is fully occupied this is the same whether mirroring is on or off, it just that it carries on for longer and then times out when mirroring is switched on. The web service and the database server are both on the same machine but the server is daul core and has 4GB of RAM so it doesn't appear to be starved for resources.
Any thoughts about why it times out at 6 mins when I have set the timeout in fortress to 1 hour? I don't mind it taking longer but timming out is a problem.
Rgrds Ian

[Beth]

Can you check your SQL log and Event Viewer logs? What I'm suspecting is maybe something else is overriding our setting.

[IanG]

There is nothing in the SQL logs but I am getting this error in the sever application log:

Code: Select all

Date,Source,Severity,Message,Category,Event,User,Computer
04/27/2009 16:51:05,ASP.NET 2.0.50727.0,Warning,Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 27/04/2009 16:51:05
Event time (UTC): 27/04/2009 15:51:05
Event ID: 0a4a9ebcc5194fcfad813234187406dd
Event sequence: 3208
Event occurrence: 60
Event detail code: 0

Application information:
    Application domain: /LM/W3SVC/1/Root/Fortress-2-128850452600558388
    Trust level: Full
    Application Virtual Path: /Fortress
    Application Path: C:\Inetpub\wwwroot\Fortress\
    Machine name: SERVER11

Process information:
    Process ID: 3772
    Process name: w3wp.exe
    Account name: NT AUTHORITY\NETWORK SERVICE

Exception information:
    Exception type: NullReferenceException
    Exception message: Object reference not set to an instance of an object.

Request information:
    Request URL: 
    Request path: 
    User host address: 
    User: 
    Is authenticated: False
    Authentication Type: 
    Thread account name: NT AUTHORITY\NETWORK SERVICE

Thread information:
    Thread ID: 1
    Thread account name: NT AUTHORITY\NETWORK SERVICE
    Is impersonating: False
    Stack trace:    at MantisServiceLib.Helpers.LogMessage(DNLogEntryType let<c/> String strMsg)   at MantisServiceLib.Helpers.DeleteSession(HttpSessionState hss<c/> Boolean signOut)   at MantisServiceLib.Helpers.DeleteSession(HttpSessionState hss)   at Mantis.Global.Session_End(Object sender<c/> EventArgs e)

It doesn;t appear to be linked with anything just getting one of these 20 mins or so.
Any ideas?
Rgrds Ian.

[Beth]

I agree that wouldn't be related. To get rid of the .NET errors, try first running the following command at a command prompt:
C:\Windows\Microsoft.NET\framework\v2.0.50727\aspnet_regiis -i

You will want to have your users not connect when you run that.

What about your SQL server logs located in your SQL Server Management Studio under maintenance?

[IanG]

Hi I'll run the reg_iis tomorrow and se if that helps.
As for the SQL logs there aren't any erorrs or anything in there, just messages about the backups having happened.

Rgrds Ian.