Vault 7.2.1: error when getting label of the $

[kha]

I am trying to view label at the $ inside a repository using Vault client on the Vault server itself and see below error (only with this repository, which is considerability big):

System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> System.NullReferenceException: Object reference not set to an instance of an object.
at VaultServiceBase.VaultLogManager.LogException(Exception e)
at VaultServiceSQL.VaultSqlSCC.BeginLabelQuery(VaultSqlConn conn, String strSessionID, Int32 nRepID, Int32 nUserID, Int16 nLabelTreeType, String strLCItemPath, Int64 nObjID, UInt16 nObjType, Boolean bGetRecursive, Boolean bGetInherited, UInt16 nObjTypeFlags, Int64 nLabelVersionFloor, Int64 nRowLimit, Int64& nRowsRetrieved, Int64& nRowsRetrievedRecursive, String& strQryToken)
at VaultServiceAPILib.VaultServiceAPI.BeginLabelQuery(String strSessionID, Int32 nRepID, Int32 nUserID, Int16 nLabelTreeType, String strItemPath, Int64 nObjID, Boolean bGetRecursive, Boolean bGetInherited, UInt16 nObjTypeFlags, Int64 nRowLimit, Int64& nRowsRetrieved, Int64& nRowsRetrievedRecursive, String& strQryToken)
at VaultService.VaultService.BeginLabelQuery(Int32 nRepID, String strItemPath, Int64 nObjID, Boolean bGetRecursive, Boolean bGetInherited, UInt16 nObjTypeFlags, Int64 nRowLimit, Int64& nRowsRetrieved, Int64& nRowsRetrievedRecursive, String& strQryToken)
--- End of inner exception stack trace ---

Is Vault 7.2.1 still supported? If so, any plan when it is end of support?

Thanks!

[Beth]

Can you check the Vault Server Log for errors happening around the same time? You might see a timeout or a warning on a specific file. Please post the errors you see. You can feel free to erase the user names and IP addresses in the error, or you can post the log and remove it later.

We don't currently have the end day for Vault 7.2.1 planned. We will send notices to customers when we have an end date planned,and it would come with enough forewarning for customers to plan appropriately. Support for 5.x and older has ended so far.

[kha]

Thanks, Beth. Here the error i copied from sgvault.log

SSL Disabled System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at VaultServiceSQL.VaultSqlSCC.BeginLabelQuery(VaultSqlConn conn, String strSessionID, Int32 nRepID, Int32 nUserID, Int16 nLabelTreeType, String strLCItemPath, Int64 nObjID, UInt16 nObjType, Boolean bGetRecursive, Boolean bGetInherited, UInt16 nObjTypeFlags, Int64 nLabelVersionFloor, Int64 nRowLimit, Int64& nRowsRetrieved, Int64& nRowsRetrievedRecursive, String& strQryToken)
ClientConnectionId:b6471329-56d6-4431-b3c5-5a6b658c8777
Error Number:-2,State:0,Class:11 at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at VaultServiceSQL.VaultSqlSCC.BeginLabelQuery(VaultSqlConn conn, String strSessionID, Int32 nRepID, Int32 nUserID, Int16 nLabelTreeType, String strLCItemPath, Int64 nObjID, UInt16 nObjType, Boolean bGetRecursive, Boolean bGetInherited, UInt16 nObjTypeFlags, Int64 nLabelVersionFloor, Int64 nRowLimit, Int64& nRowsRetrieved, Int64& nRowsRetrievedRecursive, String& strQryToken)

[Beth]

Let's increase the SQL timeout period then and see if it either completes or give a different error. Go to the location of the Vault website. By default that is in C:\inetpub\wwwroot\vaultservice. Find the file called vault.config, and open it in notepad. Edit the SQL timeout from 360 to 3600. Yes, it's a lot, but we want to make sure that if there are other hidden errors that we get to see those. Save the file, then perform an IIS reset (iisreset at an admin command prompt) or recycle the Vault application pools.

[kha]

Hi Beth,

Since the last time I reported the problem, I did up the number to 3600 in the vault.config file (2/3/17), rebooted the server (4/27/17), and still got the same error on Tuesday night 5/2/17 (very low traffic on network time)

----5/2/2017 11:09:46 PM admin--VaultServer(IP)--SSL Disabled System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at VaultServiceSQL.VaultSqlSCC.BeginLabelQuery(VaultSqlConn conn, String strSessionID, Int32 nRepID, Int32 nUserID, Int16 nLabelTreeType, String strLCItemPath, Int64 nObjID, UInt16 nObjType, Boolean bGetRecursive, Boolean bGetInherited, UInt16 nObjTypeFlags, Int64 nLabelVersionFloor, Int64 nRowLimit, Int64& nRowsRetrieved, Int64& nRowsRetrievedRecursive, String& strQryToken)
ClientConnectionId:be7418c5-0633-452d-9faa-1f016c63bc12
Error Number:-2,State:0,Class:11 at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at VaultServiceSQL.VaultSqlSCC.BeginLabelQuery(VaultSqlConn conn, String strSessionID, Int32 nRepID, Int32 nUserID, Int16 nLabelTreeType, String strLCItemPath, Int64 nObjID, UInt16 nObjType, Boolean bGetRecursive, Boolean bGetInherited, UInt16 nObjTypeFlags, Int64 nLabelVersionFloor, Int64 nRowLimit, Int64& nRowsRetrieved, Int64& nRowsRetrievedRecursive, String& strQryToken)

[jclausius]

Did it take an 1 hour for the label to timeout?

To confirm, you modified the Vault Server installation's vault.config file. And it looks like the following: <SqlCommandTimeout>3600</SqlCommandTimeout>

Also, retrieving a label shouldn't really take that long. Do you run a regular maintenance schedule in keeping the database running in an efficient manner.

[kha]

Did it take an 1 hour for the label to timeout?

To confirm, you modified the Vault Server installation's vault.config file. And it looks like the following: <SqlCommandTimeout>3600</SqlCommandTimeout>

Also, retrieving a label shouldn't really take that long. Do you run a regular maintenance schedule in keeping the database running in an efficient manner.

This is the copy and paste from my vault.config file:

<SqlCommandTimeout>3600</SqlCommandTimeout>

It took about 1 hour to timeout, i cannot tell exact because I just let it run and worked on something else then came back later seeing the error

I retrieved the label from $ and from this very big repository. Problem only happens with this big repository and loading this repository also takes longer time than the other repositories.

For the sql db, we run the sql maintenance job every weekend, which includes: shrink db limit 50%, free space 10%, rebuild index, Update Statistics. The sgvault db is now about 27GB in size.

Not sure if this info helps:

The SQL instance where the sgvault dbs are on has memory min and max usage. Min 2048 MB and Max 8196 MB.

[jclausius]

Thank you for that. I wanted to verify the setting was taking place. You can try setting the SQLCommandTimeout to a higher value, you just need to find one in that the data can all be retrieved from the database.

You could try 6 hours (43200) as the setting (same instructions as above).

Do you have a client installed on the Vault Server itself? If you do a GET of the same label, does it take as long as well? I'm curious what happens if the networking component is removed.

On a side note, when you say, loading the repository takes a long time. If you try loading the repository, closing Vault, and loading the repository a second time, does it take about the same time?

[kha]

You could try 6 hours (43200) as the setting (same instructions as above).

Cannot do this right now (during business hour as it requires IIS reset). Will try later tonight

Do you have a client installed on the Vault Server itself? If you do a GET of the same label, does it take as long as well? I'm curious what happens if the networking component is removed.

Yes, performed on the vault server itself.
Got error after 2 mins: "get latest version failed for some files. See the messages window for details"

Here the error messages from Messages Window (on the Vault client):

[5/4/2017 3:47:01 PM] Finished get latest of $
[5/4/2017 3:47:01 PM] An exception was encountered during the get latest operation. Exception: The Vault server could not be contacted to perform the operation. Your network connection to the server may have been interrupted. Please verify your network settings using the Options dialog under the Tools menu in the Vault GUI Client.The request failed with HTTP status 404: Not Found.

On a side note, when you say, loading the repository takes a long time. If you try loading the repository, closing Vault, and loading the repository a second time, does it take about the same time?

Loading 2nd time is quicker than the 1st time

More observation:
When I tried to view label at root on this repository, it shows "Not responding" in the title bar of the Client and also on the Task Manager. But eventually it will throw the timeout error.

[jclausius]

In regards to the test GET on the server itself, the message:

"The Vault server could not be contacted to perform the operation. Your network connection to the server may have been interrupted."

indicates something happening where the Vault Server is inoperable.

When using the client on the server itself, can you try http://127.0.0.1 ? I'd like to try to eliminate any possible networking issues.

If however, the Vault Server / IIS Server is being starved for CPU cycles, it might be worth looking at your setup and what is running on that server. Is the server being taxed by other processes? Is SQL Server located on this machine or another server?

[kha]

In regards to the test GET on the server itself, the message:

"The Vault server could not be contacted to perform the operation. Your network connection to the server may have been interrupted."

indicates something happening where the Vault Server is inoperable.

When using the client on the server itself, can you try http://127.0.0.1 ? I'd like to try to eliminate any possible networking issues.

I got this error when trying the http://127.0.0.1
Not Found

HTTP Error 404. The requested resource is not found.

If however, the Vault Server / IIS Server is being starved for CPU cycles, it might be worth looking at your setup and what is running on that server. Is the server being taxed by other processes? Is SQL Server located on this machine or another server?

During the whole time of the GET, VaultGUIClient.exe never took more than 50 CPU and the total processes CPU Usage (from Resource Monitor) spiked one time to 78% in just a second, mostly it stayed at 50-52%

DB is on a separate sql server which has 2 other sql instances running at the same time
Vault server is dedicated, and only Vault is installed on this server.

[jclausius]

If you're running this on the Vault Server, and http is enabled, then 127.0.0.1 is the loop back address, unless it's been disabled.

From a browser on the Vault Server itself, if you type http://127.0.0.1/VaultService/VaultService.asmx or http://localhost/VaultService/VaultService.asmx , do those give you any results? It should be similar to this browser link: http://vaultdemo.sourcegear.com/VaultSe ... rvice.asmx

[kha]

If you're running this on the Vault Server, and http is enabled, then 127.0.0.1 is the loop back address, unless it's been disabled.

From a browser on the Vault Server itself, if you type http://127.0.0.1/VaultService/VaultService.asmx or http://localhost/VaultService/VaultService.asmx , do those give you any results? It should be similar to this browser link: http://vaultdemo.sourcegear.com/VaultSe ... rvice.asmx

I got page not found error for first 2 URLs you provided, but I got actual VaultService page with formal definition of Vault options when I view these URLs (2 same result):

http://<Ip assigned to Vault website>/VaultService/VaultService.asmx

http://vaultdemo.sourcegear.com/VaultSe ... rvice.asmx

[jclausius]

I got page not found error for first 2 URLs you provided

That is done from within the Vault Server itself?

[kha]

I got page not found error for first 2 URLs you provided

That is done from within the Vault Server itself?

yes