GETLABEL fails with System.OutOfMemoryException

[achen]

This Vault client is version 4.1.4.18402
Your Vault server is version 4.1.4.18402

We are using the vault.exe CLC to perform GETLABEL.

This is the Vault.exe client exception:

<vault>
<error>
<exception>System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at VCDiff.VCDiffStreamAccessor.Read(Byte[]& dataBytes, UInt64 length)
at VCDiff.VCDiff.CopyStream(VCDiffAccessor sIn, VCDiffAccessor sOut, UInt64 nBuf)
at VCDiff.VCDiff.Apply()
at VaultClientOperationsLib.ClientInstance.ProcessFile_NonWorkingFolder(String tempFile, VaultClientFile file, VaultResponseGetFile response, String ancestorFullPath, String ancestorDiskPath, SetFileTimeType setFileTime, MakeWritableType makeWritable, MergeType merge, OverwritePrompt PromptData)
at VaultClientOperationsLib.ClientInstance.ProcessGetFileRequests(GetFileInfo[] infos, MakeWritableType makeWritable, SetFileTimeType setFileTime, MergeType merge, Boolean updateHiddenFilesOnly, String ancestorFullPath, Boolean flat, String ancestorDiskPath, OverwritePrompt PromptData, Boolean isLabelGet, String currentPathToLabelItem, Int64 labelID, Boolean isRetry, Boolean isGetByDisplayVersion)
at VaultClientOperationsLib.ClientInstance.GetByLabelToNonWorkingFolder_GetData(VaultClientFile[] files, Boolean overwrite, MakeWritableType makeWritable, SetFileTimeType setFileTime, String ancestorFullPath, String ancestorDiskPath, OverwritePrompt PromptData, Int64 labelID, String currentPathToLabelItem, String labelSubItemPath, Boolean isLabelGetForSingleFile)
at VaultClientOperationsLib.ClientInstance.GetByLabelToNonWorkingFolder_GetData(VaultClientFolder folder, Boolean recursive, Boolean overwrite, MakeWritableType makeWritable, SetFileTimeType setFileTime, String destinationDiskPath, OverwritePrompt PromptData, Int64 labelID, String currentPathToLabelItem, String labelSubItemPath)
at VaultClientIntegrationLib.GetOperations.performLabelGet(String objectPath, String label, String labelSubItem, String labelWorkingFolder, String destPath, GetOptions go)
at VaultClientIntegrationLib.GetOperations.ProcessCommandGetLabelToLocationOutsideWorkingFolder(String objectPath, String label, String labelSubItem, GetOptions getOptions, String destPath)
at VaultCmdLineClient.VaultCmdLineClient.ProcessCommand(Args curArg)
at VaultCmdLineClient.VaultCmdLineClient.Main(String[] args)</exception>
</error>
<result>
<success>False</success>
</result>
</vault>

This is the Vault server exception:

Critical Error downloading file! The remote host closed the connection. The error code is 0x80072746.
at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status, Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
at System.Web.Hosting.ISAPIWorkerRequest.FlushResponse(Boolean finalFlush)
at System.Web.HttpResponse.Flush(Boolean finalFlush)
at System.Web.HttpResponse.Flush()
at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at VaultServiceSQL.VaultSqlSCC.GetDeltaChainForFullFile(VaultSqlConn conn, Int64 nObjVerID, Int32 nBufSize, String strFileToken, Stream s, ListDictionary ldDataStreams, RunningCRC32 crc, UInt32& nCRC)
at VaultServiceAPILib.VaultFileCacheManager.GetDeltaChainForFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nTargetObjVerID, Int32 nBufSize, String strFileToken, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
at VaultServiceAPILib.VaultDelta.CreateFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nObjVerID, String strFileToken, Boolean& deltaIfAvailable, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
at VaultServiceAPILib.VaultDelta.DeltaCreate(VaultSqlConn vsConn, Int32 nRepID, Int64 nBaseObjVerID, Int64 nTargetObjVerID, String strFileToken, Stream sDeltaStream, RunningCRC32 crc, UInt32& nCRC)
at VaultServiceAPILib.VaultServiceAPI.GetVaultFile(Int32 nRepID, VaultIntDnldItem& vidi, String strFileToken, Stream s, Boolean bSeekToOrigin, Boolean bFolderExportCalcCRC, UInt32& nCRC)
at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e)

System.Web.HttpException: The remote host closed the connection. The error code is 0x80072746.
at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status, Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
at System.Web.Hosting.ISAPIWorkerRequest.FlushResponse(Boolean finalFlush)
at System.Web.HttpResponse.Flush(Boolean finalFlush)
at System.Web.HttpResponse.Flush()
at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at VaultServiceSQL.VaultSqlSCC.GetDeltaChainForFullFile(VaultSqlConn conn, Int64 nObjVerID, Int32 nBufSize, String strFileToken, Stream s, ListDictionary ldDataStreams, RunningCRC32 crc, UInt32& nCRC)
at VaultServiceAPILib.VaultFileCacheManager.GetDeltaChainForFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nTargetObjVerID, Int32 nBufSize, String strFileToken, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
at VaultServiceAPILib.VaultDelta.CreateFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nObjVerID, String strFileToken, Boolean& deltaIfAvailable, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
at VaultServiceAPILib.VaultDelta.DeltaCreate(VaultSqlConn vsConn, Int32 nRepID, Int64 nBaseObjVerID, Int64 nTargetObjVerID, String strFileToken, Stream sDeltaStream, RunningCRC32 crc, UInt32& nCRC)
at VaultServiceAPILib.VaultServiceAPI.GetVaultFile(Int32 nRepID, VaultIntDnldItem& vidi, String strFileToken, Stream s, Boolean bSeekToOrigin, Boolean bFolderExportCalcCRC, UInt32& nCRC)
at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e) at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status, Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
at System.Web.Hosting.ISAPIWorkerRequest.FlushResponse(Boolean finalFlush)
at System.Web.HttpResponse.Flush(Boolean finalFlush)
at System.Web.HttpResponse.Flush()
at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.BufferedStream.Write(Byte[] array, Int32 offset, Int32 count)
at VaultServiceSQL.VaultSqlSCC.GetDeltaChainForFullFile(VaultSqlConn conn, Int64 nObjVerID, Int32 nBufSize, String strFileToken, Stream s, ListDictionary ldDataStreams, RunningCRC32 crc, UInt32& nCRC)
at VaultServiceAPILib.VaultFileCacheManager.GetDeltaChainForFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nTargetObjVerID, Int32 nBufSize, String strFileToken, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
at VaultServiceAPILib.VaultDelta.CreateFullFile(VaultSqlConn vsConn, Int32 nRepID, Int64 nObjVerID, String strFileToken, Boolean& deltaIfAvailable, Stream sFullFileStream, RunningCRC32 crc, UInt32& nCRC)
at VaultServiceAPILib.VaultDelta.DeltaCreate(VaultSqlConn vsConn, Int32 nRepID, Int64 nBaseObjVerID, Int64 nTargetObjVerID, String strFileToken, Stream sDeltaStream, RunningCRC32 crc, UInt32& nCRC)
at VaultServiceAPILib.VaultServiceAPI.GetVaultFile(Int32 nRepID, VaultIntDnldItem& vidi, String strFileToken, Stream s, Boolean bSeekToOrigin, Boolean bFolderExportCalcCRC, UInt32& nCRC)
at VaultService.VaultFileDownload.Page_Load(Object sender, EventArgs e)


This just started occuring after we branched our code and needed to get files from a previous baseline label.
Any ideas on memory or timeout settings?

Andrew Chen

[achen]

Here is the Technical information:

Client Information
Vault Client Version: 4.1.4.18402
.Net Framework Version: 2.0.50727.3603
Operating System: Microsoft Windows XP Professional
Service Pack: 3.0
OS Version: 5.1.2600
Total Physical Memory: 3.23 GB
Time Zone: (GMT-08:00) Pacific Time (US & Canada)

Server Information
Vault Server Version: 4.1.4.18402
.Net Framework Version: 2.0.50727.3603
Operating System: Microsoft(R) Windows(R) Server 2003, Enterprise Edition
Service Pack: 2.0
OS Version: 5.2.3790
Timezone: (GMT-08:00) Pacific Time (US & Canada)
SQL Version: Microsoft SQL Server 2000 - 8.00.194 (Intel X86)
Aug 6 2000 00:57:48
Copyright (c) 1988-2000 Microsoft Corporation
Standard Edition on Windows NT 5.2 (Build 3790: Service Pack 2)

[Beth]

Before I go into the memory portion, I want to point out that your SQL Server is not on a supported service pack. When you can, I would recommend getting the rest of your SQL 2000 service packs in. I'm not sure if that's contributing to the issue, but I have seen not having the service packs in cause strange behavior that's inconsistent.

For the memory, start first by increasing the virtual RAM on your server. That is found in the Control Panel under System - Advanced - Performance Settings (click the settings 1st settings button) - Advanced and it's the last setting. If it's not set to server managed, you could try that as well. Make sure you have enough space on the drive you are setting the virtual RAM for. If you have a second drive available, you could set additional virtual RAM there as well.

When you are trying to perform the Get Label, are you getting it from the location it was actually applied at? For example, if you applied the label at $/FolderA, then you would get the label from $/FolderA. If you branch and can see the label in the history of the branch prior to the branch, I would recommend still retrieving it from $/FolderA because that's where it was applied at.

[achen]

I want to point out that your SQL Server is not on a supported service pack.

We will upgrade to SQL Server 2008 once our release is out the door.

For the memory, start first by increasing the virtual RAM on your server.

Virtual RAM could be an issue and I will increase it once we can reboot the server.

When you are trying to perform the Get Label, are you getting it from the location it was actually applied at?

We apply the label at $/TopFolder, and it has subfolders $/TopFolder/SubFolder. We then GETLABEL on $/TopFolder/SubFolder, since each subfolder pertains to a particular part of the build only.

The interesting thing is when we GETLABEL on a label that was applied on 7/21/2010 (Version 33711), the get is successful. However, when we GETLABEL on a label that was applied on 2/5/2010 (Version 32989), the get fails with the above exceptions.

It seems there is some threshold that we have exceeded (size/number of files, number of labels, etc.) when we get from the earlier label. Are there additional settings for the CLC or GUI client that can be set to increase or disable timeout?

[Beth]

The threshold you would be running up against is something either with your machine's memory or the server's memory. One way to check which it could be would be to load a client onto the server and try to GetLabel right on the server using localhost for the server name to connect to. Using localhost takes the network out of the communication.

Were any really large files added that would only be in the newer label? When I mean large, I'm talking almost a gigabyte or larger.

Let me know your results from the virtual memory change.

[achen]

I changed the server virtual memory to the Windows recommended size and performed the GET from localhost and it still failed with the same exception.

I did pinpoint the problem to 3 files that we have checked in that are 609MB each.

The interesting thing is I can sucessfully GET these files using the more recent label (7/20/2010), however, using a label that is older (6/4/2010) fails. Also, I can sucessfully GET using a label applied on 6/8/2010. The large files did not change during those labels.

Can the number of labels affect the memory used? We label the repository every time we perform our nightly build.

[Beth]

The number of labels shouldn't affect anything. Labels are small and don't take up much space in the database.

Are you able to diff the problem label against anything?

Do you know if you've performed label promotion on that label? If not, it would be possible to create a second label by a little different name at the exact same point and see if a Get works on that. To do this:
1) Find the folder that the label was applied to and the exact date and time.
2) Perform a Show History, but on the General tab, change the option to show the history by folder version.
3) Identify the folder version with the same time stamp as the label, or just right before the label.
4) Place a label on that folder version.
5) Try a Get on the new label. What are the results?