Obliterate could not handle thousands of items

dewest225 · Post by **dewest225** » Tue May 27, 2008 6:13 pm

After our VSS to Vault migration failed, it left thousands of files in the Vault database. Since it was not a complete and error-free migration, we wanted to obliterate all these imported items. I deleted them in the Vault GUI client, then went to Vault Web to obliterate them. Vault chugged away for hours, then gave an error message about having lost the connection to the Vault server -- it was running on the same machine, so it was not a network connection problem!

I then tried using the Advanced Obliterate power toy. It chugged away for 24 hours, and never allowed me to confirm the list of files to obliterate. I finally gave up. It seems as if it shouldn't take more than 24 hours for this?

Since I was importing our main VSS database into a working Vault database, and we have now made changes to the project that already existed in Vault, we cannot go back and simply restore the Vault database from before the failed import attempt. Is there another strategy for obliterating that many items? Is there a way to do subsets at a time?

Beth · Post by **Beth** » Wed May 28, 2008 2:51 pm

Can you check in your Vault server log for the error that related to the end of your obliterate? If it's a timeout, then a timeout can be changed.

You don't have to select everything in the obliterate area. Can you confirm that it works ok for you with a small chunk of files?

In the address that the admin web page is on are you using a public IP, computer name, or 'localhost' for connecting?

dewest225 · Post by **dewest225** » Wed May 28, 2008 3:44 pm

Beth wrote:Can you check in your Vault server log for the error that related to the end of your obliterate? If it's a timeout, then a timeout can be changed.

You don't have to select everything in the obliterate area. Can you confirm that it works ok for you with a small chunk of files?

In the address that the admin web page is on are you using a public IP, computer name, or 'localhost' for connecting?

1) The excerpt below from our Vault server log indicates that SQL's temp DB is running out of space, which seems to be the problem.

2) Obliterate never gets to the point where I can select individual files. I only select the initial directory to obliterate, and that's when it chugs away and never returns.

3) I am running IE on the server itself, and the web address is that server's computer name.

======================================
----5/28/2008 11:20:51 AM admin--ENGR1.pathguide.local(192.168.20.9)--SSL Enabled System.Data.SqlClient.SqlException: Could not allocate space for object 'dbo.#tmprepfulltreefolders'.'PK__#tmprepfulltreef__5EF17FF2' in database 'tempdb' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at VaultServiceSQL.VaultSqlSCC.ObliterateFSObject(VaultSqlConn conn, Int64 deletionid, Int64 folderID, Int64 objid, Boolean reallyObliterate, Char separator, VaultStringColl& ObliteratedObjects, ArrayList& branchConflictArray) at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at VaultServiceSQL.VaultSqlSCC.ObliterateFSObject(VaultSqlConn conn, Int64 deletionid, Int64 folderID, Int64 objid, Boolean reallyObliterate, Char separator, VaultStringColl& ObliteratedObjects, ArrayList& branchConflictArray)
----5/28/2008 11:20:51 AM admin--ENGR1.pathguide.local(192.168.20.9)--SSL Enabled System.Data.SqlClient.SqlException: The INSERT statement conflicted with the FOREIGN KEY constraint "fk_tbltransactiondetails_tbltransactions". The conflict occurred in database "sgvault", table "dbo.tbltransactions", column 'txid'.
The statement has been terminated.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at VaultServiceSQL.VaultSqlSCC.CreateTransactionDetail(VaultSqlConn conn, Int64 nTxID, Int32 nTxItem, String strItemPath, Byte nTxType, Int64 nItemObjVerID, Int64 nOtherObjVerID, String strComment) at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at VaultServiceSQL.VaultSqlSCC.CreateTransactionDetail(VaultSqlConn conn, Int64 nTxID, Int32 nTxItem, String strItemPath, Byte nTxType, Int64 nItemObjVerID, Int64 nOtherObjVerID, String strComment)
----5/28/2008 11:20:51 AM admin--ENGR1.pathguide.local(192.168.20.9)--SSL Enabled FailCreateTxDetail
----5/28/2008 11:20:51 AM admin--ENGR1.pathguide.local(192.168.20.9)--SSL Enabled (a9869994-0f3f-4bbb-b5b9-dd55fb351f8d) EndTx (Revision - 0) returned: FailCreateTxDetail
======================================

Beth · Post by **Beth** » Wed May 28, 2008 4:08 pm

When you deleted, was it one huge directory that you deleted?

The two things I can think of trying.

1) Undelete the huge directory, then delete smaller chunks so that when you go to obliterate, you can obliterate in smaller chunks.

2) Check your disk space and your limits on your databases to make sure it can grow large enough for the entire obliterate. If you have any large files, the amount to grow by has to be bigger than the largest file.

Beth · Post by **Beth** » Wed May 28, 2008 4:20 pm

Do you only have one repository? Are each of these VSS imports dependent on what is already in Vault? If not, I'd recommend creating a new repository for the import. A repository can easily be deleted with no problems or remaining data in the database.

Once you manage to obliterate, we should discuss your VSS import in a new thread, or send an email to support at sourcegear.com so we can help you through that part.

dewest225 · Post by **dewest225** » Thu May 29, 2008 10:42 am

Beth wrote:Do you only have one repository? Are each of these VSS imports dependent on what is already in Vault? If not, I'd recommend creating a new repository for the import. A repository can easily be deleted with no problems or remaining data in the database.

We want everything in one repository, so we can share code between projects. Since we have one of our projects already in Vault, we had to import to this repository. Unless there is a way to move items between Vault repositories?

Beth wrote:When you deleted, was it one huge directory that you deleted?

Yes, it was one huge directory with lots of subdirectories. I think there were about 18,000 files.

Beth wrote:The two things I can think of trying.
1) Undelete the huge directory, then delete smaller chunks so that when you go to obliterate, you can obliterate in smaller chunks.

2) Check your disk space and your limits on your databases to make sure it can grow large enough for the entire obliterate. If you have any large files, the amount to grow by has to be bigger than the largest file.

I have tried undeleting everything, then deleting smaller chunks. Obliterating 10 files takes about 40 seconds. That's going to take a long time to delete and obliterate everything!

Also, I must use the Advanced Obliterate Client, because Vault Web's obliterate seems to balloon tempdb up over 2 GB, filling up drive C on our server (Vault's SQL database is on D, which has plenty of space).

Beth · Post by **Beth** » Tue Jun 03, 2008 12:42 pm

Vault Web's obliterate seems to balloon tempdb up over 2 GB,

You could move the tempdb by detaching it, moving it, and reattaching it to SQL if you want. That way, it can be on a disk that has room for it.