Page 1 of 1

GetLabel via Parallel.ForEach Loop Error

Posted: Thu May 26, 2016 8:42 pm
by ejhansen71
I have a process that does GetLabel to 5 different servers. To speed up the process, I used Async/Await for each of the server calls. I was told it would be better to use Parallel.ForEach. The problem is that I get the following error(see below). Is there a way to fix this problem? I use the code below the error to run the GetLabel.

vault>
<error>
<exception>System.IO.IOException: The process cannot access the file '\\ghprodetl4\src_trs\script\test.test' because it is being used by another process.
at VaultClientIntegrationLib.GetOperations.performLabelGet(String objectPath, String label, String labelSubItem, String labelWorkingFolder, String destPath, GetOptions go)
at VaultClientIntegrationLib.GetOperations.ProcessCommandGetLabelToLocationOutsideWorkingFolder(String objectPath, String label, String labelSubItem, GetOptions getOptions, String destPath)
at VaultCmdLineClient.VaultCmdLineClient.ProcessCommand(Args curArg)
at VaultCmdLineClient.VaultCmdLineClient.Main(String[] args)</exception>
</error>

var psi = new ProcessStartInfo
{
FileName = vaultRequest.PropertiesFile.Plink,
Arguments = string.Format("-P 44022 {0}@ghelsiapp \"GETLABEL -repository \"{3}\" \"{4}\" \"{5}\" -backup no -setfiletime modification -{1} \"{2}\" -verbose \""
, vaultRequest.Username, workingFolder, writeLocation, vaultRequest.Repository, ObjectPath, vaultRequest.ActualLabelName),
RedirectStandardError = true,
RedirectStandardOutput = true,
RedirectStandardInput = true,
UseShellExecute = false,
CreateNoWindow = true
};
var getLatestTask = new Task(() => ProcessDos(psi, vaultRequest));
getLatestTask.Start();
getLatestTask.Wait();

Re: GetLabel via Parallel.ForEach Loop Error

Posted: Fri May 27, 2016 7:50 am
by jclausius
Some questions:

a) Is the process that shells out to the command line physically running on the Vault Server, or is it on a different machine?

b) Is the destination location for each iteration of your loop located on the same machine as the process running the command (depends on A), on the Vault Server itself, or another different machine altogether?

Re: GetLabel via Parallel.ForEach Loop Error

Posted: Tue May 31, 2016 8:11 am
by ejhansen71
Machine 1: Personal PC
Machine 2: Application Server
Machine 3: Vault Server
Machine 4: Linux server via SAMBA

Answers:
A) Different Machine (#2)

B) Command is started on 1 then executed on 2. Machine 2 then executes the command and sends it to 3 which sends it back to 2 and finally to 4 to write.

Re: GetLabel via Parallel.ForEach Loop Error

Posted: Tue May 31, 2016 3:28 pm
by jclausius
Off Topic - Too bad you're not just using a part of the repository tree. You could do the same thing with Vault's Shadow Folders.

OK. Back to Business. The file lock is a bit of a nuisance, but perhaps this can be negated by looking at the process a bit closer. If I understood your description, here's what is happening:

For Loop {
a) Machine 2 issues a GETLABEL command line request to Machine 3 for label X.
a.1) Machine 3 authenticates, check permissions, etc. It then let's Machine 2 know the request has been granted.
a.2) Machine 2 requests files from Machine 3.
a.3) Machine 3 copies files from database to local temp space for retrieval
a.4) Machine 2 starts copying files from Machine 3 to Machine 2's temp space
a.5) Machine 2 expands files from Machine 2's temp space to Machine 4's location which is a network location
}

Since steps a thru e are static for each iteration, what if you modified this a bit:

a) Machine 2 issues a GETLABEL command line request to Machine 3 for label X.
a.1) Machine 3 authenticates, check permissions, etc. It then let's Machine 2 know the request has been granted.
a.2) Machine 2 requests files from Machine 3.
a.3) Machine 3 copies files from database to local temp space for retrieval
a.4) Machine 2 starts copying files from Machine 3 to Machine 2's temp space
a.5) Machine 2 expands files into a non-working folder which is a local location to Machine 2

For Loop {
b) Machine 2 next copies files from Machine 2's non-working folder from step f to Machine 4's location which is a network location.
}

This would eliminate some of the redundancy for static (non-changing) operations and may help with some parallelism as well if you place some of the copy within the loop into a thread. But be careful not to saturate your network.