Very slow import/export.

[cada0310]

Hi there,

We have a large Vault repository, and we're looking to export it and import into another Vault server as part of a consolidation effort.

It is taking *forever* to export this data...on the order of 4-5 seconds per transaction, and we have 368,000 transactions to export! At this rate, my kid will have to do the import when the export finishes because I'll be dead!

Any tips on improving the performance of this process? SQL tweaking? Anything?

Thanks,

Chris

[Beth]

Are you using the Export/Import tool on the server, on a machine on the same network, or from a remote machine? You might try using the Export/Import tool on the server and connect using localhost to cut the network out of the equation.

We have an article on SQL Server maintenance that you might try out as well: http://support.sourcegear.com/viewtopic.php?t=2924. SQL maintenance can speed up its response time.

Some of our recommendations for optimal performance may help as well, such as disk fragmentation on your server. The performance kb article is posted here: http://support.sourcegear.com/viewtopic.php?t=4206.

Other things that can affect this would include not having enough disk space or RAM.

[cada0310]

Hi Beth,

Yes, we're running the import/export tool directly on the vault server. We'll check out the SQL articles and such - the machine isn't really that powerful, so that might be part of the issue.

Is there a way to do this export without the transactions (i.e. ditching all the history and such)? Or would you recommend just doing a GET and importing the whole get into the new Vault?

Chris

[Beth]

If you only want the files and not the history, then a Get is the best way to go. Then you can add it back to Vault in a new location or repository.

[pogaz]

Hi Beth and/or Chris--

Were you guys (or just one of you) able to come to a resolution? I am having a similar problem. What's interesting is that I see that the export process "processes" transactions outside the directory that I am importing, which: 1) seems confusing on why it would even do that (shouldn't it just be able to know what transactions go with this directory, by use of joins/filters with its queries?), and 2) which, I would think, is why it takes so long (hours, but didn't complete because I canceled the process) if it's processing all transactions and not just those that are specific to that directory.

And yes - I did run the export on the Vault server and did use "localhost".

Thanks,
Paul

[Beth]

If you have moves, branches, or shares, then there are parts of the tree outside the folder that are referenced. Basically anything that might have some sort of reference outside that folder. You won't end up with additional folders in your export.

Have you performed database maintenance on your server?

What are the specs on your server?

Are users still making changes to the same repository during the export?

[pogaz]

First of all, thank you for you response. Here is my reponse:

The folder I was trying to export was a brand new folder that I had just created for the sole purpose of testing exporting.

By database maintenance, do you mean maintenance on the actual SQL Server, via SQL Server? If so, the database is not maintained by myself. Is there something specific would should be doing or not doing? If you mean something like was obliterate used on the repository, I just used history to find any and there was some instances of it being used back in 07/2008 and before.

Server specs are:
Windows Server 2003 Standard Edition
Service Pack 2
Intel Xeon 3.6GHz Processor
4GB RAM
PAE Enabled

Here are some details on the repository:
Revisions: 82705
Folders: 52511 (+10084 deleted)
Files: 363612 (+84738 deleted)
Tree Size: 16.37 GB
Disk Space Needed: 32.75 GB
Database Size: 3.43 GB

Here's the Vault server version:
SourceGear Vault
Version 4.1.0.16216

I don't know for sure if users were making changes to the repository during the export test; it's possible.

Please advise. Also, if you can advise me on how to reduce the size of our repository by archiving out old/unused directories to another repository, that would be great and that is the ultimate goal.

Thanks.

[Beth]

One thing that might help you is upgrading your Vault installation (server and clients) so that you have the latest fixes. I looked up what I think is your company account and it looks like your maintenance has remained current which would give you a free upgrade. If you want to find out what's involved in an upgrade first, you can find our Upgrade Guide here: http://support.sourcegear.com/viewtopic ... 13&t=11648&.

Obliterate does make it hard to export, because obliterate removes history. I would still recommend an upgrade before completely assuming that obliterate is the issue though. The one thing that could get in the way of an upgrade is if you can't run the 3.5 .NET framework.

Would you be willing to send me your logs? That would provide a good starting point for troubleshooting this and getting you pointed in the right direction. The logs that I'd need are VaultFolderExportImport.txt, which will be in the same directory as the Vault Export/Import executable, and the sgvault.log found on the Vault server at %windir%\temp\sgvault\sgvault.log. If you can send those, send an email to support at sourcegear.com (attn: Beth) with a link to this forum thread and the log files.

(Note to myself: Another thread here: http://support.sourcegear.com/viewtopic.php?f=5&t=15441.)

[pogaz]

I can send the logs the next time I am actively working on. Can't do right at this moment.

As far as upgrading, I don't want to add another variable to the scenario right now, unless you are positive that it would allow us to export out directories even though obliterate was used.

Questions:
1. Could we just use SQL Server to create a copy of the repo and give the repo a new name and then make the old repo read-only and delete out the directories that we don't want from the new repo?
2. Could we basically do the same as #1 above, but intead of using a copy of the repo, we just create a copy of the databases and put them on a new SQL Server instance?

Thanks.

[Beth]

What upgrading would do is rule out any bug we've fixed already from preventing the Export/Import.

On the SQL questions, you could restore a backup to a different SQL server and install a second instance of Vault to use that database. Then you would delete the repository (or repositories) from the first Vault install that you want available on the second install, and on the second install, you would delete all repositories except for the one (or more) you want to run on that server.

What is your purpose for Exporting/Importing? Is it to improve performance or to create more space on the server or some other reason?

[pogaz]

Beth, thanks for your reply.

I appreciate the suggestion that perhaps an upgrade would fix the issue with doing an export, but I really don't want to perform a relatively big step and the planning that would go with it (e.g. people's time, any affect on productivity) when we are not sure that it would fix anything. Is it possible that you can confirm that it will or will not fix the issue, perhaps by viewing your product's change/bug-fix list or perhaps discussing with the product manager or similar? Also, are there significant benefits (e.g. performance) in general with upgrading to the next version of Vault, or is it just some new features that may or may not be significantly beneficial?

Regarding the idea to use a second instance instance of Vault, would that be in conflict with licensing? If not, then that would seemingly be a good solution.

The purpose of exporting is to improve performance **and** to reduce the amount of space required in using the repository.

[Beth]

I couldn't guarantee that the issue you are having is related to any of the fixes without performing a full analysis on your database. I don't think we have any logs yet on this issue, or anything that exactly pinpoints the root cause yet for me.

There's a way to make this work without having a licensing conflict.

I think it might be good to take this conversation offline so we can get more detailed about what's happening. Could you send an email to support at sourcegear.com (attn: Beth) with a link to this forum thread?