3.5.3 major branching troubles

jdavidi · Post by **jdavidi** » Wed Jul 02, 2008 8:18 pm

Hi folks,

We'd been using 3.5.1 happily for a little over one and a half years, and decided to make the jump to 3.5.3 a few weeks ago to take advantage of the client improvements we saw in the release notes, of which we have seen the benefit on our build servers.

Tonight was our first major branch since the upgrade, and while I'm not quite at the point that I'm hosed, I've encountered what I believe another user did a little while back in that my branches are taking forever and all my other clients are getting locked out:

http://support.sourcegear.com/viewtopic ... ildbreader

We have branched this particular part of our tree 9 times now; it's a little less than 22,000 files, and the whole thing usually took about 4 minutes. I bumped the SQL timeout after getting the FailDBReader error to an hour as suggested in the post, but even THAT timed out. I've resorted to a chunking approach by branching the 8 subfolders separately, and for a metric one of the folders with about 8000 files in it took about 25 minutes.

The branching was attempted from 3 clients, so I'm pretty sure we're dealing with a server issue here.

I will continue to babysit this tonight until my chunking is complete so my team can use it tomorrow morning.

Has any headway/hot patches been distributed to other customers that have reported this? I don't see anything in my debug-level log file beyond what was in the other post. Would giving you our database help? We have another major branch about a week and a half away and I'm not looking forward to another 5 hour session as you might imagine.

Much obliged,

--jdavidi

Beth · Post by **Beth** » Thu Jul 03, 2008 7:48 am

Where I'll need you to start is with Vault database maintenance to see what kind of improvement that will give you.

After that, it will be good to be familiar with our KB article Recommendations for optimal Vault performance. Some of the items listed in that article are going to affect branching performance as well.

Next, can you tell me about your server? How large is your database? How much RAM is on the machine? What other types of programs is it running? Are you using virtual machines on that server?

While you are going through these suggestions and questions, I'll look into the results of the article you referred to.

Beth · Post by **Beth** » Thu Jul 03, 2008 10:30 am

After digging in on this, the time increase you are seeing has to be something else going on. There were speed improvements with that function in 3.5.2. Even on the one you referenced that said he had an increase in time, the leap in how much time it took was nothing like yours. Please start with the suggestions in my previous post, then let me know what kind of results it had.

The portion about being locked out isn't related to the speed of the operation, and we have a bug logged on that. We are working on a fix for this for one of our upcoming releases.

Tri · Post by **Tri** » Thu Jul 03, 2008 2:02 pm

Hi,

I have experienced the same branching issue a while ago:

Branch failure (3.52): http://support.sourcegear.com/viewtopic.php?t=8955
Please read toward the end of that thread for the solution.

I first uninstalled + Reinstall edVault Server 3.53. Branching issue was still there. Then I did the following:

1. Update Windows 2003 to SP2 + latest updates (was SP1) and apply SQL Server 2000 SP4 (was SP3).

Next I proceeded what SourceGear tech support had advised:

2. backup sgvault DB
3. uninstall Vault Server 3.52
4. restore DB
5. redownload Vault Server 3.53 and re-install

Intuitively, I think that Step 1 was what had really made the fix. But I don't know the exact technical reason.

jdavidi · Post by **jdavidi** » Thu Jul 03, 2008 2:10 pm

Hello Beth,

Thanks (as always) for the fast replies.

We'd made the suggested changes in the "optimal Vault performance article" shortly after implementing 3.5.1 in summer '06 after we'd encountered some VSS import trouble and IIS timeouts, so we shoud be up to speed (pun intended!

) there.

I emailed my DBA and I.S. team to confirm our maintenance plan matches the suggestions in the database article. I'm pretty sure we implemented those way back when as well to run nightly, but I'll have them double-check that this is [still] happening, especially since the upgrade.

All other operations have been normal since the global 3.5.3 upgrade though. The branch just brought it to its knees though...

Our server stats as requested:

standalone physical machine (not virutal nor running virtual machines)
no defragging recommended by Windows
OS: Windows Server 2003 SP1
Intel Xeon 3.8 GHz processor
3.5 GB RAM
1.36 TB hard drive with 1.06 TB free
sgvault database ~16 GB
Microsoft SQL Server 2005 (v9.00.3042.00)
Vault web service is the only persistent non-Windows process on the machine

Integration builds are promoted to a file share on the server throughout the day, though no file transfers were taking place during the branch.

No event log errors between 5:30 PM and 11:00 PM when branching was being attempted.

All I have for clues in the DEBUG level Vault log is the significant delay between the "transaction is pending" step and the EndTx() step. This particular "chunked" branch request would have affected 7000+ files.

----7/2/2008 9:50:58 PM Jdav--jdav.idi.com(196.168.68.108)--SSL Disabled (8bbb5712-d8a8-434a-abf2-5bb0a27c44c4) CopyBranch: $/9.0/9.02.maint_SP7/NetSource/Business to $/9.0/9.02.maint_SP8/NetSource/Business returned: Success
----7/2/2008 9:50:58 PM Jdav--jdav.idi.com(196.168.68.108)--SSL Disabled Ending transaction
----7/2/2008 9:50:58 PM Jdav--jdav.idi.com(196.168.68.108)--SSL Disabled Beginning SQL transaction 23419
----7/2/2008 9:50:58 PM Jdav--jdav.idi.com(196.168.68.108)--SSL Disabled Attempting to acquire repository lock.
----7/2/2008 9:50:58 PM Jdav--jdav.idi.com(196.168.68.108)--SSL Disabled Acquired repository lock.
----7/2/2008 9:50:58 PM Jdav--jdav.idi.com(196.168.68.108)--SSL Disabled TreeManager Signal - Tx Begin - CacheLockId:23420
----7/2/2008 9:50:58 PM Jdav--jdav.idi.com(196.168.68.108)--SSL Disabled TreeManager: A transaction is pending, but it belongs to the current user. Returning cached tree, revID 592397
----7/2/2008 10:05:59 PM Jdav--jdav.idi.com(196.168.68.108)--SSL Disabled EndTx(): Begin commit changes for TxID 592398, which will release repository lock.

I don't know if it matters, but when I did the branch through the GUI client, in a matter of seconds I saw "Ending Transaction..." in the status bar and that's when the wait began.

I'll keep you posted as to what the db guys find.

Thanks,

--jdavidi

jdavidi · Post by **jdavidi** » Thu Jul 03, 2008 2:23 pm

Thanks for the reply, Tri.

While our environment is slightly different than yours in that we're running against SQL 2005, I did notice when gathering some hardware stats for SG a few minutes ago that we are on Windows 2003 SP1 as opposed to 2. So I will certainly get our Vault server on the upgrade list for that SP when we can (hopefully before our next scheduled branch).

Uninstalling/reinstalling the Vault server certainly seems an easy enough task to try before our next branch as well if there's any hope of getting us back to normal branching speed.

Beth (or anyone else on the SG support team), do we know which Windows 2K3 service packs Vault 3.5.3 was tested against?

At this point, not sure I'll hear back from my I.S. group about the db server maintenance, so in case I don't get that answer till next week, I wish a happy long weekend to everyone perusing this thread.

We'll solve this together shortly, I know it!

--jdavidi

Beth · Post by **Beth** » Thu Jul 03, 2008 3:53 pm

I'm not exactly sure which service pack it was tested against.

Thanks Tri for pointing out your previous thread.

jdavidi: I would definitely agree with going the same route as Tri mentioned and then let me know the results.

jdavidi · Post by **jdavidi** » Tue Jul 08, 2008 10:30 am

Hi again,

My DBA has confirmed that "we have a nightly job which performs index defags and updating of statistics. The job has been running successfully every evening."

So we'll start with the Windows Server upgrade and go from there. Will let everyone know either way.

Thanks,

--jdavidi

Beth · Post by **Beth** » Tue Jul 08, 2008 12:42 pm

Thanks for the update.