Enhancement: option to ignore checked-out files on a Get

[christian]

Bit of background to what we're trying to do.

We have a team of developers all using a Check-out / edit / check-in approach to versioning.

We want a typical programming session for a single developer to go something like this:
1) Developer populates his working directory with the latest code in the repository by performing a recursive Get
2) Check-out files
2) Modify files
3) Unit test just those changes made
4) Run the build script on the developer machine against the developer's working directory to locally compile the entire system including his changes
5) Run the regression test suite against this build to verify that the changes made have not broken other parts of the system.
6) Perform a recursive Get to update the developers working directory with any changes checked-in by OTHER developers since step 1
7) Perform steps 4 and 5 against the updated working directory thereby verifying that his changes will integrate with rest of the codeline
8) Check-in

Now step 4 requires that ALL the vb project files, dll's and exe's to be made writeable including those that the developer did not check-out. So the build script makes these files writeable before doing the build.

Because these files have been made writeable, this introduces a problem in step 6. The Get has to ignore those files the developer has checked-out but replace those files that were made writeable in order to build the other parts of the system in his working directory.

You can make Vault replace local files that are writeable but this wipes out the changes the developer has made to the files he's got checked-out. Therein lies the problem!

Is there any way you guys can see how to configure Vault so that a Get ignores those files the developer requesting the Get has checked-out but that replaces all other writeable files in his working directory?

If Vault cannot be configured in this fashion can you add this as a feature request?

Thanks in advance
Christian

[dan]

I will go out on limb here and suggest that you really don't want what you are asking for

If you want to make sure a developer's changes are integrated with other developer's changes, you probably need to use the "automatically merge" option on the checked out files. You don't want those files overwritten, but you don't want them to stay unchanged either, because if someone else has made changes to the file, you need those changes in your copy so that the tests will run correctly.

If the files you are having troubles with are exes or dlls that are checked in, isn't it the case that you would need to recompile these anyway, so getting them from Vault won't really make a difference anyway?

One last thing to note: Vault doesn't get files because they are marked read-write - it gets them because there are newer versions on the server than are on the client. So, the read-write status isn't really the problem, unless I am misunderstanding the issue, which is always quite possible

[christian]

Thanks for the reply.

If you want to make sure a developer's changes are integrated with other developer's changes, you probably need to use the "automatically merge" option on the checked out files. You don't want those files overwritten, but you don't want them to stay unchanged either, because if someone else has made changes to the file, you need those changes in your copy so that the tests will run correctly.

As mentioned, us developers here follow a Check-out / Edit / Check-in approach to versioning. What I didn't say is that we make sure that an exclusive lock is acquired. This means that two developers cannot work on the same files concurrently, so we wouldn't be in a position to need to merge.

If the files you are having troubles with are exes or dlls that are checked in, isn't it the case that you would need to recompile these anyway, so getting them from Vault won't really make a difference anyway?

One last thing to note: Vault doesn't get files because they are marked read-write - it gets them because there are newer versions on the server than are on the client. So, the read-write status isn't really the problem, unless I am misunderstanding the issue, which is always quite possible :)

Right, I've just run through a little test example to prove to myself that Vault doesn't use the read-write status to determine whether it should do a Get on a file:

I checked-out a file and made some changes to the file in the working directory.
I performed a recursive get on an ancestor node specifying that local modified files should be overwritten.

I find that the changes I've made locally have been wiped out. This can only be because Vault has done a get on the file even though the modified file date in the repository is earlier than the modified file date in the working directory.

Now the reason why in step 6, I'm specifying overwrite for local modified files is because I want to remove those vb project files the developer hasn't himself had to change during the course of his programming session but that have had there version information updated in step 4. I want these locally modified project files to be replaced with whatever is the latest version from Vault. However, as demonstrated in my little test example, if I do this then the developer will wipe out changes he's made to the files he has checked-out.

I know it’s quite difficult to discuss a reasonably complex process in a dialog, however I need to get to a resolution to this so your help is most appreciated.

Christian

[dan]

Yes, modified files will be overwritten if you specify that in the Get - it has to do with their edited status though, rather than their read-write status.

Just to verify, the files you want to overwrite are text project files, and not generated binary files? I'm not understanding how those files are used - do your tests not run correctly unless you use the server version of the file? If so, what changes are made locally that conflict with the server version? Is it the case that these project files actually need merged rather than to be simply overwritten?

If you don't want to check the project files in, then I would suggest not checking them out in the first place - that way they will not show up in the pending change set, and they won't be accidently checked in. When you do a Get, choose to not overwrite modified files (or automerge them, which in your case should ever happen).

One idea is that the status search pane can be used to identify files with a certain status, so you could use that to identify files that need special processing.

[christian]

Just to verify, the files you want to overwrite are text project files, and not generated binary files? I'm not understanding how those files are used - do your tests not run correctly unless you use the server version of the file? If so, what changes are made locally that conflict with the server version? Is it the case that these project files actually need merged rather than to be simply overwritten?

Here's a silly example that hopefully will put some flesh on the bones of the workflow I outlined.

$/Mainline/Project1/ecicProject1.vbp
$/Mainline/Project1/class1.cls
$/Mainline/Project1/ecicProject1.dll

$/Mainline/Project2/ecicProject2.vbp
$/Mainline/Project2/class2.cls
$/Mainline/Project2/ecicProject2.dll

$/Mainline/Project3/ecicProject3.vbp
$/Mainline/Project3/class3.cls
$/Mainline/Project3/ecicProject3.dll

All files are at version 1

1) Developer performs a Get on $/Mainline to populate his working directory for the first time
2-3) Developer checks out Project2 and modifies class2.cls
4) Runs the build on his machine. This build makes the files ecicProject1.vbp, ecicProject1.dll, ecicProject3.vbp and ecicProject3.dll writeable (Project2 files are already writeable as a result of performing a check-out). The build updates these files even though the developer did not make any changes to these files himself.

In the meantime another developer has checked-in changes to ecicProject3.vbp, class3.cls and ecicProject3.dll.

6) Developer performs a Get on the $/Mainline. Developer specifies that the Get should not overwrite locally modified files. In this way the developer doesn't wipe out the changes to the files he has checked out. Looking at each project...
$/Mainline/Project1 files should be at vs 1. Yes, you're right the files don't strictly have to be the exact same files as in the repository but it would be nice if read-only files from the server replaced the local files.
$/Mainline/Project2 files would include the changes that the developer has made
$/Mainline/Project3 files would be at vs 2 ie the version that the other developer has checked in.

Now going on what you said the Get would retrieve Project3 files from the repository only if the modified date in the repository was later than the modified date of the files in the working directory. Well suppose our developer ran the build in step 4 AFTER the check-in of Project3 files by this other developer. The Project3 files in his working directory will be later than the repository and therefore will not be replaced by the files in the repository.

So this is why I think I need to be able to specify that the Get should overwrite modified files with the exception of those files the developer has checked out (a new Vault feature). This would give me the working directory that I outlined above.

One idea is that the status search pane can be used to identify files with a certain status, so you could use that to identify files that need special processing.

Had thought about this approach. My problem here is that I'm automating the process from a product called FinalBuilder2 (which uses the command-line) which means that I need to do things in a generic fashion without requiring input by the developer running the build.

So what do you think? Am I missing something simple here?
Thanks
Christian

[dan]

Thanks for the example. One question though - why does a compile on Project1 change the .vbp file? Nothing else in the project should have changed based on the example (no files added or removed, etc).

That the .dll files changed shouldn't be a problem - they probably should be recompiled anyway when running tests after the GET, so if they are not up to date, it isn't really a problem. One thing to note too, is that storing executable files in source control isn't really necessary, since it can always be built with the source code.

The Get logic is a little confusing: It determines whether the file is edited based on the datetime stamp of the file, and whether it is different from when it was retrieved. However, whether a file is retrieved from the server or not depends on the Vault version number, not the datetime. So if the server version is higher, it gets the file. regardless of the timestamp. Whether it overwrites, merges or does not overwrite
The Get determines whether to get a file or not based on the Vault depends on whether the local version has been modified.

[christian]

Thanks for the example. One question though - why does a compile on Project1 change the .vbp file? Nothing else in the project should have changed based on the example (no files added or removed, etc).

Two reasons why the vbp for Project1 may / will change, both as a consequence of running the build script:
1) Version number will be incremented and this new number written back to the vbp
2) FinalBuilder2 will "fix" any reference the vbp may have to a type library when it finds that the vbp is referencing an old typeLibID that is no longer in use. This fix is written back to the vbp

That the .dll files changed shouldn't be a problem - they probably should be recompiled anyway when running tests after the GET, so if they are not up to date, it isn't really a problem. One thing to note too, is that storing executable files in source control isn't really necessary, since it can always be built with the source code.

Agreed, however, we store these derived binary files in version control in a BIN directory structure that mirrors the deployment directory structure. Rather than hold the knowledge of how files are deployed in a deployment script, we find that version control in this way is "documenting" deployment.

The Get logic is a little confusing: It determines whether the file is edited based on the datetime stamp of the file, and whether it is different from when it was retrieved. However, whether a file is retrieved from the server or not depends on the Vault version number, not the datetime. So if the server version is higher, it gets the file. regardless of the timestamp. Whether it overwrites, merges or does not overwrite
The Get determines whether to get a file or not based on the Vault depends on whether the local version has been modified.

Yeah, I think you lost me all ready :-)
To relate this logic to the example - will a Get performed by our intrepid developer retrieve Project3? Remember that the vbp (and dll which we don't really care about) was retrieved at vs 1, changed and checked-in by another developer, and then modified by our developer in his working directory.

Thanks (again)
Christian

[dan]

To relate this logic to the example - will a Get performed by our intrepid developer retrieve Project3? Remember that the vbp (and dll which we don't really care about) was retrieved at vs 1, changed and checked-in by another developer, and then modified by our developer in his working directory.

Yes, it will recognize that the versions on the server are newer and the local files need updated, even if the local files were saved to disk AFTER the new server versions were uploaded to the server.

I suppose one way to go is on the Get, choose to automatically merge files that have changed in both the server and the client. Those files will then be in a Needs Merge state. Since you really want the server version and not the client version, you would then just reget those files in a Needs Merge state (through the pending change set or the status search pane).

[christian]

Yes, it will recognize that the versions on the server are newer and the local files need updated, even if the local files were saved to disk AFTER the new server versions were uploaded to the server.

Ok, that's good news.

I suppose one way to go is on the Get, choose to automatically merge files that have changed in both the server and the client. Those files will then be in a Needs Merge state. Since you really want the server version and not the client version, you would then just reget those files in a Needs Merge state (through the pending change set or the status search pane).

My aim here is to eliminate all developer intervention. If its not 100% automated then it increases the chance that this developer build will either get skipped by those developers of a lazy disposition or these "needs merge" files that are to be discarded will get checked in by mistake. Thanks for the suggestion anyway.

So I am still in a bit of a bind here. If this is a feature you think doesn't warrant adding to Vault then can you suggest how I can leverage the command-line interface to say remove the "needs merge" files from the developer's pending change set. I'll then invoke the appropriate command from the Build tool I'm using.

Thanks for the ongoing (and on, and on!) assistance
Christian

[dan]

To do this in an automated way, you would need to invoke a LISTFOLDER command on the command line client and parse out the XML returned to find all files with a status of Renegade, and then do a GET command that overwrites just those files with the latest version from the server. Then you can do a regular GET on the root folder choosing to not overwrite or automerge, and I think that will get you the local state you need.

[christian]

To do this in an automated way, you would need to invoke a LISTFOLDER command on the command line client and parse out the XML returned to find all files with a status of Renegade, and then do a GET command that overwrites just those files with the latest version from the server. Then you can do a regular GET on the root folder choosing to not overwrite or automerge, and I think that will get you the local state you need.

That sounds as though it will do what I need. If I get any problems I'll let you know.

Thanks for all your help.
Christian

PS. It would be nice to get this on the list of features for a new release 'cause I'm sure that developer builds like the one I mention is going to become more popular as the awareness of good software configuration practice grows. If this were a feature that came out of the box then it would mean that implementing a developer build would be easier. Developers’ running builds on their machine before check-in would increase the quality of the code in version control.

I suspect that this post won’t get picked up by others in the forum as the title does not read "Feature request: Support building system locally on developer machine”, and therefore the feature won't get much support :-(

Any way like I said, your help is all the same very much appreciated!