diffmerge doesn't show diff if file is large?

[bob@bos]

To verify testing results, I will take spool files created by the application I'm testing and compare the live production run against the test run. One of the files I try comparing contains 575,000+ lines (it is my customer statement print file, and I would be unable to share it with you because of the confidental information contained in it) and diffmerge reports no differences between the live and test runs, however, I saw that the file sizes were different and used the file compare feature of UltraEdit and it showed several differences. Does diffmerge have a limitation on number of records it will try to compare? Note that another file, which is larger in total size, but has fewer lines shows no differences in both diffmerge and UltraEdit, so I don't think it has anyuthing to do with file size per se, just number of records. Also looked through the past couple of years of posts and didn't see anything about this. I'm also using the batch command line compare for the check:

"\\s21shares\programmers\fiservftp\cbscompare\sourcegear\diffmerge\diffmerge -d=\\s21shares\programmers\fiservftp\cbscompare\diff-ftt90001i.txt -u \\s21shares\programmers\fiservftp\cbscompare\production\ftt90001i \\s21shares\programmers\fiservftp\cbscompare\release\ftt90001i "

[jeffhostetler]

We do have some performance problems for especially large files.
But no, there isn't any real limit.

You might try turning off "lines and characters" mode (on the Detail Level
page of the Options Dialog) so that we don't try to do the intra-line passes.

It's hard to guess why we wouldn't show a difference without a little
more info, but you might also check the Ignore/Fold case, eol, and
whitespace settings on the corresponding ruleset.

Do you have any of the "Hide Unimportant..." stuff turned on?

Could you characterize the differences that UltraEdit showed that DiffMerge
should have shown? Perhaps even cut out an example with a few lines
above and below the section in question and scrub/obscure the confidential
parts yet keeping the overall structure of the change so that both tools
show/don't show the change)?

thanks
j

[bob@bos]

Tried your suggestion, but the batch version still reports no differences, I'll attach a file showing the area of first differences between the two files. And here's a portion of the bat file I run (and I tried it both with and without the -u).

echo off

echo No differences for ftt90001i > \\s21shares\programmers\fiservftp\cbscompare\diff-ftt90001i.txt

\\s21shares\programmers\fiservftp\cbscompare\sourcegear\diffmerge\diffmerge -d=\\s21shares\programmers\fiservftp\cbscompare\diff-ftt90001i.txt -u \\s21shares\programmers\fiservftp\cbscompare\production\ftt90001i \\s21shares\programmers\fiservftp\cbscompare\release\ftt90001i

echo on

[jeffhostetler]

I cut the 2 examples from the attachment and loaded them up in DiffMerge
and didn't see any problems with them.

Could you extract the same ~20 line region of both input files into 2 files and
then launch diffmerge with them and see if the problem still happens (and
eliminate the huge file concerns). You might also turn on "Show Invisibles"
and make sure there aren't any whitespace oddities.

Then if you still see the problem, could you post a zip containing those 2 files.
And maybe a pair of screenshots.

thanks
jeff

[bob@bos]

Jeff,

So what I did was to take just the first 10,000 lines (give or take a few) and ran both the command line and GUI versions of DiffMerge against them.

Command line version reported no differences, GUI showed one (see attached screen scrape) - between you and me, this is getting very weird.

Thanks,
Bob

[bob@bos]

Jeff,

So I did some more playing aroung and was able to get a 590+ line file that shows a difference when using the GUI as opposed to using the command line and blanked out all the account names/numbers and will attach them to this post for your amusement - there has to be something in the data that confuses the command line processing that has nothing to do with the size of the files being compared.

[jeffhostetler]

I'm not seeing the problem here. I've tested on both a MAC and Windows XP machine.
In both cases, I'm seeing the "Interest" line delta and a blank line change at line 411.
These 2 changes appear in both the GUI window and the batch output.

I guess it is time to ask all of the stock questions:
What version of DiffMerge do you have? And what OS version? And etc...
Does it happen with local DOS paths in addition to when using share paths?

Also have you made any changes in your preferences? It is possible to get
DiffMerge to ignore some changes (using the "Lines to Omit" and other features).
If you go to the Rulesets page of the Options dialog and hit "Restore Defaults"
and then try it, does the output look any different?

jeff

[bob@bos]

Running on a Gateway PC WinXP Pro S/P 3 Pentium 4 3.8 GHz, 3.25 GB of RAM machine and the latest (as of a week or two ago) version of DiffMerge (3.3.0 (18513)) - that was in fact one of the first things I checked.

Having not caused enough confusion, let me throw this at you - I added errorlevel checking to my bat and see something most interesting, several of my diffmerge commands are telling me that I have syntax errors (I had, of course, duplicated the lines then just changed the file names) and one tells me that there's an I/O error (that file CAP00201, is not only the biggest file in total size, but it also has the largest record size - 3540 characters - it's a hunk of burning love my friend!).

I'll attach my bat file (renamed with a .txt extension, of course) and the diff-errors.txt showing the errorlevel returned from each compare I do (I identified each file by a different letter code so I'd know which one was throwing which error.)

And yes, this even happens when I run everything out of c:\download and I tried both with and without the -u and with both -d and -diff, before you ask also reverted back to defaults and even examined a hex display of the bat to check for nulls or something else silly like that to no avail.

[jeffhostetler]

Beats me! I'm not sure what would cause the various error codes.

You might try omitting the -d and -u args from your script and running
it and see if you get any error dialogs when each command starts.
(I can't print on stderr in batch mode because the OS closes them for
gui apps before we get started.)

At this point, seeing what you are trying to do, you might also try installing
cygwin (including the diffutils) and then point gnu diff at the release and
production directories.

jeff

[bob@bos]

You may, if you wish, just take me out back and shoot me.

Taking out the -d and -u did not fix any broblem, but did point out what I was doing stupid.

When I ran without those parameters, I keep getting a message that the files to compare wer missing - after the 3rd or 4th time I finally notices that in the bat I was misspelling the file names I was having themost trouble with they should have bee (for example) ftt900001i (4, that's 4 zeroes between the 9 and the 1) - after doing that, it worked like a charm - yes I am a dumb s**t!

However, it's still not too happy about trying to compare the cap00201 files (remember - those were the files with a 45xx long record size. Even the GUI isn't too happy with that file and I get a:

File has a NUL byte ar offset 636355; this is not valid for encoding 'Default encoding"

The file in the left panel P:\fiservftp\cbscompare\production\cap00201 cound not be imported because it contained a NUL character. THis may be a binary file.

The files are identical (using raw comparison).

Would you like to try another encoding?

Yes and No boxes

However, I took a look at that file (which should have all character data) with a hex text editor (UltraEdit, if you must know) and guess what I found at that location - yes you're right - a NUL!

So you're stuff works just great (thank you very much) and I have to figure out how I got a NUL in the middle of what's supposed to be character data.

Sorry for the confusion - and thanks for your help.

Bob

[jeffhostetler]

No problem. I've had my fair share of head-smacking gremlins too.

I'm just happy that you got everything figured out.

jeff