End transaction times out

[robe070]

The final error is:
GetRepositoryStructure returned: FailInvalidRevision

Full stack of error is:
----22/07/2014 4:01:43 PM michiyo--michiyow7.syd.lansa.com.au(10.2.1.48)--SSL Disabled System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at VaultServiceSQL.VaultSqlSCC.GetRepositoryTreeDelta(VaultSqlConn conn, Int32 nRepID, Hashtable htSharedItems, Int64 nBaseRevID, Int64 nTargetRevID, VaultRepositoryDelta& rep) at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at VaultServiceSQL.VaultSqlSCC.GetRepositoryTreeDelta(VaultSqlConn conn, Int32 nRepID, Hashtable htSharedItems, Int64 nBaseRevID, Int64 nTargetRevID, VaultRepositoryDelta& rep)
----22/07/2014 4:01:43 PM michiyo--michiyow7.syd.lansa.com.au(10.2.1.48)--SSL Disabled GetRepositoryStructure returned: FailInvalidRevision

If we try the same check in again it succeeds.

ps We defrag the index and update statistics as per best practice recommendation every week.

[Beth]

You can set the SQL timeout to a higher limit. Find the vault.config file in the vaultservice website directory. The default location is C:\inetpub\wwroot\vaultservice. In there is a SQL timeout set to 360. Add another 0 to the end.

Then perform an IIS reset by going to Start - Run and typing iisreset. This will kick your users out of Vault for a moment, so you will want to let them know ahead of time.

If you have mission-critical websites that can't be recycled, then you can instead recycle the vaultapppool.

[robe070]

The timeout is already set to 3600. Thats 1 hour I believe. Given that a txn that succeeds rarely takes more 20 seconds, something else is the cause.

[robe070]

markd.txt

The client side of the server timeout

(13.67 KiB) Downloaded 846 times

[Beth]

Do you get the error FailInvalidRevision only on check ins of edited files, or do you get it also on Gets, or adds of new files?

What version of Vault are you using?

What is the OS version for the machine the Vault server is on? What version of SQL Server are you running?

Would you be willing to either post or email me the entire Vault server log? If you post it, you'll be able to delete it when we're done. If you email it, send it to support at sourcegear.com with a link to this forum thread.

Can you check your Event Viewer logs on the Vault server around the same time that a transaction times out? Are there any errors there? Are any application pools recycling?

[robe070]

Error occurs on Check Ins only.

I've just noticed that the client-side log I sent you for this particular example does not have a corresponding server-side timeout. The user may just have not waited long enough (20 minutes), this time. There are plenty of actual timeouts in the server log.

SQL Server 2005
Vault 5.0.3
Windows Server 2003 R2 x64 SP2
Core 2 CPU 6600 " 2.40GHz, 4 GB RAM

Application Event Log
SQL Server has encountered 14 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [D:\Program Files (x86)\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\tempdb.mdf] in database [tempdb] (2). The OS file handle is 0x000009CC. The offset of the latest long I/O is: 0x000001cdfb0000

DBCC CHECKDB is fine, though "DBCC CHECKDB will not check SQL Server catalog or Service Broker consistency"

[Beth]

Do all check ins fail, or do just some files fail on check in?

Is it consistent on which files will fail to check in, or is there a particular file type that will fail to check in?

What happens if you perform an add of a very small 1 kb text file? Will that fully commit to the repository?

[robe070]

(Somehow I wasn't notified of your reply.)

Its completely random. Only a few files fail. Its not consistent. Usually a timeout will be followed by a successful check in. Many different size files successfully check in from very small to very large.

One that failed was small. Maybe 3 kb. Can't reproduce of course.

[Beth]

Can you turn off the setting "Request Database Delta on Repository Cache Miss" that is found in the Vault - Tools - Options - Networking?

Then try checking in again.

If the check in fails, then go to Vault - Tools - Options - Local Files - Cache Location and click the button that says "Reset Repository Cache." Then try your check in again.

If the check in still fails, then could you send an email to support at sourcegear.com (attn: Beth) with a link to this forum thread?

[robe070]

We run with "Request Database Delta on Repository Cache Miss" unchecked all the time.

We had already cleared out the cache and the error occurred which you have the logs for.

[Beth]

Could you send an email to support at sourcegear.com (attn: Beth) with a link to this forum thread?