Obliterate 'waiting for response from server'

Tilman · Post by **Tilman** » Mon Aug 20, 2007 7:22 am

Hi,

I tried to obliterate a deleted folder with Vault 3.5.2. For 15 minutes it said 'Waiting for response from server' and the progress bar remained at '0', at which point i crashed the admin client, since there was no way of canceling at that point. (That also crashed the server.) So my question is, should it take so long to get at least a response? How do I know when it's working and when not?

Cheers,

Tilman

Beth · Post by **Beth** » Tue Aug 21, 2007 7:24 am

A good way to find out what happened is to take a look at the Vault Server Log. If you only see entries for login/logout, then it's in Quiet mode. You can switch it to debug mode in the Admin webpage (or Admin tool if before 4.x).

Check around the time you performed the obliterate for any errors. You can post those errors or attach the entire log and I'll take a look. If you don't want to post your log, you can send it to support at sourcegear.com (attn: Beth) with a link to this forum thread.

Also, so that I have more information about what could have caused a problem, can you open a client, connect to a repository, go to Help - Technical Support and post the environment information from there?

Tilman · Post by **Tilman** » Thu Oct 25, 2007 2:28 am

Sorry I took so long to reply...

I scheduled a command line obliterate to run during the night. (When I ran it during the day it made Vault unusable for everybody else.)

Here are the errors in the log (haven't set them to debug, though)

----10/25/2007 12:25:43 AM user1--server1.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled System.Data.SqlClient.SqlException: General network error. Check your network documentation.
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream)
at System.Data.SqlClient.SqlCommand.ExecuteScalar()
at VaultServiceSQL.VaultSqlSCC.GetUserRepositoryLastChange(VaultSqlConn conn, Int32 nRepID, VaultDateTime& dtLastChg) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream)
at System.Data.SqlClient.SqlCommand.ExecuteScalar()
at VaultServiceSQL.VaultSqlSCC.GetUserRepositoryLastChange(VaultSqlConn conn, Int32 nRepID, VaultDateTime& dtLastChg)
----10/25/2007 12:25:43 AM user1--server1.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled ListWatchers returned FailPermissionDenied
----10/25/2007 12:25:43 AM user1--server2.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled Login
----10/25/2007 12:25:58 AM user1--server1.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled System.Data.SqlClient.SqlException: General network error. Check your network documentation.
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at VaultServiceSQL.VaultSqlSystem.GetUserEmailOptions(VaultSqlConn conn, Int32 nUserID, String& emailAddress, Boolean& includeDiffs) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at VaultServiceSQL.VaultSqlSystem.GetUserEmailOptions(VaultSqlConn conn, Int32 nUserID, String& emailAddress, Boolean& includeDiffs)
----10/25/2007 12:25:58 AM user1--server1.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled GetUserEmailOptions returned FailDBLookup
----10/25/2007 12:25:58 AM user1--server2.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled System.InvalidOperationException: Invalid operation. The connection is closed.
at System.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso)
at VaultServiceSQL.VaultSqlConn.BeginTransaction(IsolationLevel isolationlevel) at System.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso)
at VaultServiceSQL.VaultSqlConn.BeginTransaction(IsolationLevel isolationlevel)
----10/25/2007 12:26:18 AM user1--server1.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled Logout
----10/25/2007 12:26:20 AM user1--server2.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled Login
----10/25/2007 12:26:20 AM user1--server1.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled Logout
----10/25/2007 12:26:20 AM user1--server1.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled Logout
----10/25/2007 12:26:20 AM user1--server1.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled Logout
----10/25/2007 12:26:58 AM user1--server2.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled Logout
----10/25/2007 12:27:14 AM user1--server2.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled Login
----10/25/2007 12:27:29 AM user1--server2.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled Login
----10/25/2007 12:27:29 AM user1--server2.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled Login
----10/25/2007 12:27:52 AM user1--server2.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled Logout
----10/25/2007 12:28:32 AM user1--server2.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled Logout
----10/25/2007 12:28:56 AM user1--server2.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled Login
----10/25/2007 12:28:56 AM user1--server1.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled Login
----10/25/2007 12:29:12 AM admin--server3.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
General network error. Check your network documentation.
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at VaultServiceSQL.VaultSqlSCC.ObliterateFSObject(VaultSqlConn conn, Int64 deletionid, Int64 folderID, Int64 objid, Boolean reallyObliterate, Char separator, VaultStringColl& ObliteratedObjects, ArrayList& branchConflictArray) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at VaultServiceSQL.VaultSqlSCC.ObliterateFSObject(VaultSqlConn conn, Int64 deletionid, Int64 folderID, Int64 objid, Boolean reallyObliterate, Char separator, VaultStringColl& ObliteratedObjects, ArrayList& branchConflictArray)
----10/25/2007 12:29:12 AM admin--server3.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled (b818daee-e63f-4cb7-9a89-7879399c7eb4) BeginTx returned: FailDBReader
----10/25/2007 12:29:12 AM admin--server3.red-gate.com(xxx.xxx.xxx.xxx)--SSL Disabled (b818daee-e63f-4cb7-9a89-7879399c7eb4) Obliterate: $/Red Gate/Temp returned: FailDBReader

After that it worked normal.

Here is the scheduled job log

"Obliterate.job" (vault)
Finished 10/25/2007 12:29:13 AM
Result: The task completed with an exit code of (ffffffff).

The odd thing is, that on a test system I managed to obliterate the same folder. And the test system was running on a virtual machine.

Client Information
Vault Client Version: 3.5.2.5087
.Net Framework Version: 1.1.4322.2032
Operating System: Microsoft Windows XP Professional
Service Pack: 2.0
OS Version: 5.1.2600
Total Physical Memory: 2 GB
Time Zone: (GMT) Greenwich Mean Time : Dublin, Edinburgh, Lisbon, London

Server Information
Vault Server Version: 3.5.2.5087
.Net Framework Version: 1.1.4322.2407
Operating System: Microsoft(R) Windows(R) Server 2003, Standard Edition
Service Pack: 2.0
OS Version: 5.2.3790
Timezone: (GMT) Greenwich Mean Time : Dublin, Edinburgh, Lisbon, London
SQL Version: Microsoft SQL Server 2005 - 9.00.3054.00 (Intel X86)
Mar 23 2007 16:28:52
Copyright (c) 1988-2005 Microsoft Corporation
Developer Edition on Windows NT 5.2 (Build 3790: Service Pack 2)

License Information
22 serial number(s):
1 of 22: 5 users, permanent
2 of 22: 1 users, permanent
3 of 22: 5 users, permanent
4 of 22: 5 users, permanent
5 of 22: 5 users, permanent
6 of 22: 5 users, permanent
7 of 22: 10 users, permanent
8 of 22: 1 users, permanent
9 of 22: 5 users, permanent
10 of 22: 1 users, permanent
11 of 22: 1 users, permanent
12 of 22: 2 users, permanent
13 of 22: 1 users, permanent
14 of 22: 1 users, permanent
15 of 22: 1 users, permanent
16 of 22: 1 users, permanent
17 of 22: 1 users, permanent
18 of 22: 1 users, permanent
19 of 22: 1 users, permanent
20 of 22: 5 users, permanent
21 of 22: 1 users, permanent
22 of 22: 1 users, permanent

(We do actually have 60 licenses...)

Thanks,

Tilman

Beth · Post by **Beth** » Thu Oct 25, 2007 9:08 am

Let's first try making the SQL timeout larger. In the Vault.config file on the SQL server, there is an entry for a SQL timeout. By default it's set to 360. Add another '0' after it and give the action a try again.

Tilman · Post by **Tilman** » Thu Oct 25, 2007 10:35 am

Will try that next week. As far as I can tell it failed after ~15 minutes, last time, though. What is that number, seconds, minutes?

Tilman · Post by **Tilman** » Wed Oct 31, 2007 2:56 am

Cool, that did it! Thanks for your help!

Tri · Post by **Tri** » Wed Oct 31, 2007 7:18 pm

Tilman wrote: I scheduled a command line obliterate to run during the night. (When I ran it during the day it made Vault unusable for everybody else.)

I would like to do the same.
May I know which Vault comand like you are using? And which scheduler do you use ?

Thanks in advance

Tilman · Post by **Tilman** » Thu Nov 01, 2007 4:12 am

Hi Tri,

May I know which Vault comand like you are using?

Code: Select all

vault obliterate -host "<server>" -user admin -password "<password>" -repository <repository> -yesiamsure "<repositorypath>"

I think you can use any user that has admin rights.

And which scheduler do you use ?

Just a normal Windows Scheduled Task.

Hope that helps,

Tilman

Tilman · Post by **Tilman** » Thu Nov 22, 2007 2:55 am

Hi,

me again. I managed to obliterate some smaller folders. But now i'm hitting a wall again. I've set the sql timeout to 10800 (which I presume means 3 hours), but the obliteration fails after approx. 2 hours.

Here's the relevant log section

----11/22/2007 1:57:34 AM admin--xxx.xxx.xxx(xxx.xxx.xxx.xxx)--SSL Disabled System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
General network error. Check your network documentation.
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at VaultServiceSQL.VaultSqlSCC.ObliterateFSObject(VaultSqlConn conn, Int64 deletionid, Int64 folderID, Int64 objid, Boolean reallyObliterate, Char separator, VaultStringColl& ObliteratedObjects, ArrayList& branchConflictArray) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at VaultServiceSQL.VaultSqlSCC.ObliterateFSObject(VaultSqlConn conn, Int64 deletionid, Int64 folderID, Int64 objid, Boolean reallyObliterate, Char separator, VaultStringColl& ObliteratedObjects, ArrayList& branchConflictArray)
----11/22/2007 1:57:34 AM admin--xxx.xxx.xxx(xxx.xxx.xxx.xxx)--SSL Disabled Beginning SQL transaction 64916
----11/22/2007 1:57:34 AM admin--xxx.xxx.xxx(xxx.xxx.xxx.xxx)--SSL Disabled SQL transaction 64916 successfully committed.
----11/22/2007 1:57:34 AM admin--xxx.xxx.xxx(xxx.xxx.xxx.xxx)--SSL Disabled (e27663f9-bd10-4de8-8455-cef6c4a4d6f2) BeginTx returned: FailDBReader
----11/22/2007 1:57:34 AM admin--xxx.xxx.xxx(xxx.xxx.xxx.xxx)--SSL Disabled (e27663f9-bd10-4de8-8455-cef6c4a4d6f2) Obliterate: $/Red Gate/Bits and bobs returned: FailDBReader

Tilman

Tilman · Post by **Tilman** » Mon Nov 26, 2007 5:15 am

Any ideas?

Beth · Post by **Beth** » Mon Nov 26, 2007 6:12 pm

Not yet. Do you have any errors in your server's Event Viewer logs that corresponds with the time of those errors?

Tilman · Post by **Tilman** » Tue Nov 27, 2007 3:49 am

No errors, but it says the database was backed up during that time.

Beth · Post by **Beth** » Tue Nov 27, 2007 8:36 am

Is it possible to try a large obliterate when a backup is not happening?