Vault Performance / Hardware

[hwapdc]

Hi
Due to increasing activity CPU usage sometimes skyrocks to 98% for several hours. Vault Server is running at Windows Server (VMWARE Guest with 4 vCPU). I'm considering adding 4 or 8 vCPU. Do I need to reconfigure IIS or reinstall Vault Server after adding more vCPU?

[Tonya]

Hi there,

Before you proceed with addition, we have some questions we'd like to ask. The problem may be resolved without adding more vCPU.

What are your maintenance plans for SQL Server? Are they working correctly? Sometimes tweaking a maintenance plan will improve overall performance.

How many repositories are active and how large (number of files/folders, versions are irrelevant) are in each repository? How many transactions do you commit a minute? We may be able to apply some changes in your vault.config that may help.

Do you know if ALL of your users have *unchecked* the "Request database delta on repository cache miss" on the "Network Settings" options?

Have you been able to determine the source of the CPU load? Vault Server (w3wp.exe) or SQL Server? Is SQL Server on the same virtual machine as Vault?

Thanks,

Tonya

[hwapdc]

Hi
The issue is not the SQL Server. The SQL server is running at dedicated Windows Server. We have carefully monitored the performance of the SQL server. The SQL is waiting to deliver the results to the Vault Server.
Vault server is running at dedikated Windows Server only running Vault Web Service. Server has 4vCPU and 32 GB RAM (only 10 GB is used). When Vault Service is busy proces w3wp.exe is consuming most of the CPUs.


I just need to know if I need to reconfigure IIS or other setting to improve the performance.

vault.config:
<?xml version="1.0" encoding="utf-8"?>
<!-- The path to this file must be set in web.config -->
<appSettings>
<WorkingDirectory>E:\SGVault\Temp</WorkingDirectory>
<CachingStrategy>1</CachingStrategy>
<FileCacheEnabled>False</FileCacheEnabled>
<FileCachePath>E:\SGVault\Temp\sgvault\filecache</FileCachePath>
<FileCacheDiskSizeMb>100</FileCacheDiskSizeMb>
<AuthHeader>_authHdr</AuthHeader>
<VaultContentType>application/octet-stream</VaultContentType>
<DownloadQryParam>DownloadToken</DownloadQryParam>
<TxQryParam>TxToken</TxQryParam>
<FileQryParam>FileToken</FileQryParam>
<ShelveQryParam>ShelveToken</ShelveQryParam>
<DBBufferSizeKB>512</DBBufferSizeKB>
<TreeManagerSize>-1</TreeManagerSize>
<WebBufferSizeBytes>32768</WebBufferSizeBytes>
<ChainLimit>20</ChainLimit>
<FullText>50</FullText>
<VaultContentLengthHeader>Content-Length</VaultContentLengthHeader>
<LogLevel>Quiet</LogLevel>
<OverWriteLogOnStartup>False</OverWriteLogOnStartup>
<ArchiveLog>Daily</ArchiveLog>
<LogFilePath>E:\sgvault\Log</LogFilePath>
<LogReverseDNSLookup>True</LogReverseDNSLookup>
<SqlCommandTimeout>3600</SqlCommandTimeout>
<SaveFullTreeInterval>2000</SaveFullTreeInterval>
<VerifyTreeIntegrity>False</VerifyTreeIntegrity>
<TreeDeltaCompresionThreshold>1000</TreeDeltaCompresionThreshold>
<MaxDownload>10240000</MaxDownload>
<GetFileCheckBatchSize>256</GetFileCheckBatchSize>
<DelayThreshold>10</DelayThreshold>
<DelayDurationSeconds>30</DelayDurationSeconds>
<CheckoutListDaysToExpire>5</CheckoutListDaysToExpire>
<AllowRepositories>*</AllowRepositories>
<FindinFilesWordChars>A-Za-z0-9_</FindinFilesWordChars>
<MaxFileSizeBytes>-1</MaxFileSizeBytes>
<ServerURL>https://Vault.pdc.xx:443/VaultService</ServerURL>
</appSettings>

[Tonya]

Thank you for the details. In order to provide you with further suggestions, we are going need answers to the following questions:

1) How many repositories are active and how large (number of files/folders, versions are irrelevant) are in each repository?
2) How many transactions are committed a minute?
3) Perhaps temporarily turning on debug logging on the server, and taking a look at the log file may give us an indication where Vault Server is spending its time.
4) Do you know if ALL of your users have *unchecked* the "Request database delta on repository cache miss" on the "Network Settings" options?

We're unsure that changing the number of virtualized CPUs won't require IIS re-configuration, this is sort of out of our expertise. Searching the internet may provide additional information to confirm any behavior.

Thanks,

Tonya

[hwapdc]

Yesterday I added more vCPU to the Windows Server running the Vault Web Server, so far it seems to be running OK. I'l look

[Tonya]

Glad to hear. Thank you for the update.

Tonya

[hwapdc]

Hi
SG Support did ask:
4) Do you know if ALL of your users have *unchecked* the "Request database delta on repository cache miss" on the "Network Settings" options?

At PDC some developers has this option checked some hasn't. What is the recommended setting of this option?

[Tonya]

Since you are experiencing performance issues, we recommend the option "Request database delta on repository cache miss" to be disabled.

Thanks,

Tonya

[Hans Olav Nymand]

Hi Tonya,

Below is some additional info in regard to the original post from Henrik:
"Due to increasing activity CPU usage sometimes skyrocks to 98% for several hours."

We seem to have tracked this - often or always - correspondinng to one or more clients "hanging" during get latest with a CPU utilisation well above 90% all the time (for that one CPU used by that client). I am working to get a specific example with client log and server log.

But - should I post this as an independent issue (an error) to avoid cluttering this specific discussion on performance analysis and hardware scoping. Or should I post my findings in this thread - what do you prefer ?

Best regards, Hans Olav

[Hans Olav Nymand]

Does the "Request database delta on repository cache miss" option in the UI also apply to the command line client (on the same machine)

/Hans Olav

[Tonya]

Hi Hans,

The answer to your most recent question, Does the "Request database delta on repository cache miss" option in the UI also apply to the command line client (on the same machine), is yes ONLY if the user is logged into their machine under a windows account.

Now on to the performance issue. Did adding the additional vCPU not resolve the problem? Why don't you email me directly with your new findings along with the log files. Please respond to support@sourcegear.com and reference this forum post.

Thanks,

Tonya