Add Assignment Issue in Vault Admin

[jmercker]

We're running Vault 3.1.1.3506.

Other Vault operations seem to be working fine, but if I try to add an assignment to either a group or a specific user, I get one of two outcomes:
1) It appears to add (e.g. closes dialog), but the assignment doesn't show up in the list of assignments for the user or group.

2) Vault Admin hangs for a very long time when I click to add the assignment and never returns.

Some background...

We've been running Vault since 2.x and haven't hit this issue yet. We have on average 5-10 active users. Our installation was done as an upgrade from VSS using the VSSImport tool. The file size for the database is somewhere around 2.5GB. Hardware is more than sufficient (e.g. dedicated dual xenon servers for server and database, no resource shortages).

Thanks in advance for any help...

Jeramie

[jmercker]

This morning I uninstalled/reinstalled the Vault admin tool and got the same behavior. I also was able to add default rights to a user for the entire repository, but was still unable to add an assignment to either a user or a group. Still effectively no-ops with no exception being thrown to the client (admin tool) or logged to the Vault logs on the server.

User access to Vault is not interrupted nor is their ability to check-out, check-in, etc. Just appears to be isolated to adding assignments to users and groups.

[lbauer]

We had another report of this and did some extensive troubleshooting:

http://support.sourcegear.com/viewtopic.php?t=4362

You might try the suggestions in that thread. However, we may need a copy of your database to find a solution, as we were not able to duplicate the behavior on any of our own databases.

[jmercker]

I tried the suggestions in that thread and I have the same outcome as that individual. Also - I had the same perflib error that he mentioned showing up. I corrected the error by granting read access to the reg key noted in the dotnet247 article. That error no longer appears but the behavior of not being able to add assignments to a user or group remains. The thing that's troubling me in all of this is that no exception is being thrown. When I click the OK button to make the assignment, the dialog closes, but the user's rights are not updated, nor does the database get updated.

As far as a copy of the db goes, we'd likely have to get an NDA in place before that could happen.

Is there any additional information that I could get to help in diagnosing the problem?

[lbauer]

The log shows frequent system restarts -- seven within 4 hours -- plus a long failed obliterate operation. If you tried to use the Admin Tool during these times, you could very well get a hang.

Are you restarting Vault or is this happening on its own?

Do you know what time in the log you tried to add an assignment?

[jmercker]

The restart happens because we run IIS Crash/Hang Agent on the IIS server that the Vault web service is running on. The restart occurs because when trying to add the assignment to the user or group, the admin app hangs past the trigger point for a "Hang" in the IIS Crash Hang Agent (currently set at around 6 minutes) which causes the IIS services to reset.

The "add assignment" operation in Vault Admin should be near instantaneous from what I've seen in the past, so it feels like there is something else going on. Prior to the latest upgrade of Vault, things ran fine within the same environment (e.g. the IIS Crash/Hang agent has been in our environment longer than we've had Vault).

The obliterates in the log were me trying to clean up the repository given that there were a large number of items in the obliterate tab that we didn't need to be able to recover.

As far as the time that I tried to add the assignment, it would be towards the end of the log (e.g. the last day in the log).

I've checked error logs on all machines involved (e.g. Vault Admin, Vault Service, Vault database) and nothing sticks out at me...

Any other ideas?

Thanks in advance,
Jeramie

[lbauer]

I compared my debug log (after successfully changing a user assignment) with yours, and didn't see much difference.

Since the log reports no errors, it may be a client side problem.

If you use the Admin Tool from a different machine, or delete the client-side cache for the Admin, does the behavior change?

[jmercker]

I'll give it a try and get back to you. Thanks for being so responsive on this...

[jmercker]

On the original machine that I was running Vault Admin on, I deleted the cache and tried again with the same results.

I went ahead and installed the client on a new machine that had never had Vault Admin on it and achieved the same result (e.g. issue still exists).

As an additional data point, I tried to add the right to the user by using the folder security tab and the right could not be applied that way either.

At this point, I can grant the user default rights to the entire repository, but am unable to specifically grant them just the rights that they need at the file/folder level using either: Add assignment to group they belong to, Add assignment directly to user, or Add right to user through folder security tab.

I've also tried this with other users, so isn't likely an issue specific to the one user account.

Any suggestions are welcome...

Thanks.

[lbauer]

At this point, we need to take a look at your database. Contact me and we'll discuss details, NDA, etc. You can email linda at sourcegear.com.

[lbauer]

We got a copy of jmerker's database (sans sourcecode) and were able to determine the problem.

There was a scalability issue in that Vault was calculating inherited rights for every version in the path being assigned - not noticeable on folders with a few hundred or thousand versions, but it caused the Admin Tool to hang when assigning user rights on a database with 50,000 versions.

This will be fixed in the next version of Vault. Meanwhile, we have a patch to take carea of this.