SOS Client saturates CPU on VMware Workstation

[kingbolete]

I've installed the latest VMware Workstation (v 6.5)with XP SP3 as both host and guest operating systems and also installed the SOS client 4.2 (same problems with 4.1.2). The SOS repository is access across a CheckPoint VPN-1 link. When I connect to the repository it starts an "update file list operation"; there are a rather large number of files in the default project (around 4,000 files). The client takes a minute or two before it shows 100% on the progress bar; then it starts using CPU time like crazy (almost 100% of the CPU). When it gets into this mode it continues using most of the CPU for very, very long time (more than five minutes). Oddly, it stops using CPU time if it issues a stay-alive request but resumes gobbling up the CPU if I put the focus back on the SOS window. It goes to zero CPU load after it displays the "Received file list" status message and also shows the files on the screen (since this is a new client installation the status of the files are all "unknown"). When I click on the window again giving it focus it goes back to eating most of the CPU again (95%+). Sometimes it seems like it breaks free eventually but it takes a very long time (> 15 minutes or more). Another oddity is while SOS is hogging the CPU the TaskManager window shows two SOS applications, one bearing the circular SOS icon and the other using a generic window icon. When I have the task manager "goto process" on the circular icon application it goes to the explorer.exe process which is using some CPU time and is also doing some page-faulting though it's doing no I/O and it's memory usage is stable. When I goto the generic icon SOS process it switches to the sos.exe process which appears to be using lots of CPU time but not doing and page faulting or i/o. WinXP is also showing the SOS window on the screen as "not responding". Both applications are listed as "not responding" in the task manager window.

I wonder if there is some coordinate problem between the two processes and perhaps the SOS.exe is spinning while waiting?

[lbauer]

The first login when the SOS client is installed can take a long time, because it is building the client-side cache of the project tree. Subsequent logins should be much faster.

Do you have the client-side cache? The client-side cache is called databaseX.sos and is located in C:\Documents and settings\<username>\Application Data\SourceGear\SOS\Servers\<servername>?

I'm not sure why you're seeing two SOS processes. Do you see the same behavior with a SourceOffSite client when installed on the host machine or a different machine?

[kingbolete]

I'll try running it on a non VM system when I can get one set up (that looks like it may have to be the fall back).

I moved a copy of the database file from a computer that very recently had access to the repository (it's now only accessible via a VPN---CheckPoint's VPN-1 ). I thought maybe the one on the VM got corrupted somehow. SOS launched and came up fairly quickly, but indicated that all of the files were missing (this was correct since the working directly is different on the VM). When I changed the working directory so that it should see the source files (I selected a virtually empty ancestor projector when I did this) and then selected a project with a large number of files (4k) SOS went into it's cpu gobbling mode again. I took a walk and came back about 20 minutes later and it's still grinding away. It doesn't seem to be doing any disk or network I/O.

I killed it and restarted it. I clicked onto some fairly small project (< 100 files) and it behaved appropriately. But, when I went back to the big project again, it went back into it's compute-bound mode again. Does it spawn off another process to do things for it? That might be why there is a second app/process visible in the task manager window. The second process wasn't visible until it went compute-bound; maybe what ever thing it's outsourcing is normally over very quickly except when there are a large number of files?

[lbauer]

Thanks for the info about the project with 100 files behaving normally. This is a known issue. The more files a user has in a project or directory, the longer it takes the SOS client to refresh the information from the server. And since the SOS client does a "refresh" when it regains focus, it goes over the loop again.

This behavior will be addressed in SOS 5.0, currently under development.

In the meantime, you could try putting fewer files in more folders, if that's possible.

[kingbolete]

I don't think that asking the company I'm working with to completely revamp their file structure is a viable workaround. I'll probably have to revert back to VSS if that still works. When are they planning on having a fixed version released?

While the large folder may not be an ideal arrangement, the deficiencies in SOS's other features would be exacerbated by chopping the folder up:

1) There's not way to find a file of a given name in SOS if you don't know where it's located. A wildcard search feature analogous to Unix find is desperately needed.

2) The existing status search feature is too limited. It allows selection on the basis of only a single characteristic at a time. For example, if I'm tracing down a build problem, I want to be able to find any files that are old, missing or need merging. I have to use several queries and associated operations to safely update my source tree (see below).

3) The "get latest" feature is rather dangerous. Frequently I need to update my source tree while I have files checked out (which are modified); sometimes I even have renegade files in my source tree since I need to make a debugging hack that shouldn't go back into the saved configuration. Get latest stomps on all of those files. Since the source tree is huge, the "prompt for each" switch isn't going to help. I've had to go to my local backups to restore clobbered files.

[jclausius]

When are they planning on having a fixed version released?

Right now, I can go on the record to say it will be in the first half of 2009. There will also be a beta period for those wanting to get their hands on SOS 5 at an earlier date. Stay tuned for updated information about our SOS 5 Beta program.

Also, if anyone will be at PDC next week, stop by the booth to ask for a glimpse of the preview for SOS 5.

1) There's not way to find a file of a given name in SOS if you don't know where it's located. A wildcard search feature analogous to Unix find is desperately needed.

2) The existing status search feature is too limited. It allows selection on the basis of only a single characteristic at a time. For example, if I'm tracing down a build problem, I want to be able to find any files that are old, missing or need merging. I have to use several queries and associated operations to safely update my source tree (see below).

3) The "get latest" feature is rather dangerous. Frequently I need to update my source tree while I have files checked out (which are modified); sometimes I even have renegade files in my source tree since I need to make a debugging hack that shouldn't go back into the saved configuration. Get latest stomps on all of those files. Since the source tree is huge, the "prompt for each" switch isn't going to help. I've had to go to my local backups to restore clobbered files.

Some of these items might prove difficult as the client's design doesn't lend itself well to features 1 & 2 (as the file list is not found on the client until traversed). For issue #3, make sure you are not set to overwrite files on a GET, but rather MERGE in the changes.

In any case, we'll take a look at *all* these issues for SOS 5, and see if something can't be done.

Thanks for the feedback.

[kingbolete]

kingbolete wrote:1) There's not way to find a file of a given name in SOS if you don't know where it's located. A wildcard search feature analogous to Unix find is desperately needed.

2) The existing status search feature is too limited. It allows selection on the basis of only a single characteristic at a time. For example, if I'm tracing down a build problem, I want to be able to find any files that are old, missing or need merging. I have to use several queries and associated operations to safely update my source tree (see below).

3) The "get latest" feature is rather dangerous. Frequently I need to update my source tree while I have files checked out (which are modified); sometimes I even have renegade files in my source tree since I need to make a debugging hack that shouldn't go back into the saved configuration. Get latest stomps on all of those files. Since the source tree is huge, the "prompt for each" switch isn't going to help. I've had to go to my local backups to restore clobbered files.

Some of these items might prove difficult as the client's design doesn't lend itself well to features 1 & 2 (as the file list is not found on the client until traversed). For issue #3, make sure you are not set to overwrite files on a GET, but rather MERGE in the changes.

I've tried to use the merge option in the past, but that usually results in large number of stale and unmodified files failing merge. And I really don't want it to mess with my truly modified files (those without the readonly flag set) in any way, shape or form.

Actually, problem #2 is well within the capabilities of the existing client software. What it would require is simply a mod to the user interface to allow finer-grained control. The fact that I can accomplish the task by doing a sequence of status-search operations proves this out.

A lot of Item #1's functionality could be done with the existing code since it could simply search within the cached filenames. The same problem occurs in Visual Studio when files are grouped into a large number of folders; I wrote a VS macro to search through the files and open one that matches a pattern.

But the bigger problem is when a I need to look at a particular file that is causing a problem and is not on my local disk. As you point out, this sort of search needs to be carried out on the server side and unlike my first scenario there is no way to get around the problem by doing a search on my local disk since I don't have direct access to the repository files (which probably wouldn't help anyway given its structure). With large code bases a lot of things are best done on the server side rather than sending individual transfers about file information. I suspect that when just obtaining the list of files for a project the current design is involving a whole lot of tiny transfers (maybe one per file?) since an approach which ships across a zip of the folder information ought to take seconds rather than minutes (and I've got a moderate speed broadband connection). Now that I'm trying to use SOS on a VM and across some horrible VPN these performance issues are real problems.

[jclausius]

I've tried to use the merge option in the past, but that usually results in large number of stale and unmodified files failing merge. And I really don't want it to mess with my truly modified files (those without the readonly flag set) in any way, shape or form.

I haven't seen that kind of behavior before. During SOS 5's dev cycle, we'll re-visit this code to ensure correct behavior on GETs.

Actually, problem #2 is well within the capabilities of the existing client software. What it would require is simply a mod to the user interface to allow finer-grained control. The fact that I can accomplish the task by doing a sequence of status-search operations proves this out.

I think I misunderstood your request. You're asking for the search to work with a combination of different items, not just one of the exclusive options in the drop-down. Is that correct?

A lot of Item #1's functionality could be done with the existing code since it could simply search within the cached filenames.

Unfortunately, file names of the repository's entire, current folder structure are not necessarily cached client-side. As you pointed out, to be useful the search really needs to be done on ALL files in the repository, not just what has been retrieved/found on disk. Again, something we'll look at during SOS 5 development.

Now that I'm trying to use SOS on a VM and across some horrible VPN these performance issues are real problems.

Understood. We will be trying to come up with unique solutions to solve these kinds of problems (like re-retrieving the file list on each application activation) and the slow client side processing (100% CPU utilization) of large file lists.

Thanks.

[kingbolete]

What's the ETA of SOS 5? We were supposed to be migrating over to Vault so I stopped following SOS; however, IT tried to import the SOS codebase (it's huge) over to Vault and after a week or more of importation it still said it had 3+ weeks to go (and after all that time they would still have to go back and move the updates to the SOS codebase over somehow; our codebase is under heavy development). So, the migration to Vault appears to be dead and I need to push for some other solution.

[lbauer]

You have a couple of options. SOS 5 beta will be out in a few weeks. We don't have specific date as yet. There should be some performance improvements for your type of situation with lots of files in a project.

Another option is to try Vault 5.0 beta which was released a couple weeks ago. We have a new, streamlined VSS import which brings in just the latest VSS files and then maintains a connection to your VSS database for history. Details here:

http://www.sourcegear.com/vault/releases/5.0b1.html

http://support.sourcegear.com/viewtopic.php?p=48414

[jclausius]

What's the ETA of SOS 5?

SourceOffSite 5 will be released in the second half of this year. The actual release date will be more clear once it hits Beta.

Until then, we have some upcoming releases planned. First, there will be a Customer Technology Preview (CTP) due out in late May / early June. This release will be of the NEW SourceOffSite GUI client only of which the main purpose is to gather feedback about the product and some of its enhancements. After the CTP release, we will have an official SourceOffSite 5 Beta release to get the entire product out for general purpose use. This will include the SourceOffSite GUI, server as well as an updated Visual Studio IDE (MSSCCI based) client.

Anyone interested in assisting us with these releases should mail sosbeta@sourcegear.com.

Thanks,
Jeff Clausius
SourceOffSite Dev Team