OS X FB & Toast COMPARE inconsistencies: DiffMerge reliable?

[sraf]

I just found and installed DiffMerge today in my search for a reliable file/folder compare utilty under OS X Tiger. What led me to this was a period of anomalous Finder behavior and frequent crashes, which I finally traced to a broken Spotlight function. I have disabled Spotlight (mds/mdimport), but now I am getting "Mismatch at byte X/sector X. Verification failed" error messages when burning data DVDs in Toast, when I have never had them before (strangely, the byte number in these error messages is always the same –– byte 560 in my case -- while the sector number varies from disc to disc, this also reported by other users with the same problem).

To make matters worse, the Compare functions in both Toast 9 and File Buddy are behaving inconsistently when I compare files or folders on my HD with those that I have burned onto DVD, sometimes reporting differences, other times saying the same compared files/folders are identical.

Here is some info from Macfixit regarding this issue:

QUOTE
Several users have experienced an issue where various Macs suddenly stop burning disks (CDs or DVDs) with or without giving errors. In the Finder, users will get error code "0x8002006e unable to verify" after writing the contents of a burn folder. The burned disc is sometimes readable and appears to be intact, but every subsequent burn attempt gives a similar error. This issue has affected burning activities outside the Finder: Toast users appear to have the same problem.

User Name Deleted writes:

"I'm having the same issue. Upgraded to OS 10.5.2 with my 20" Intel Core Duo iMac and I cannot burn discs. This is happening in the finder (burn folders) and with Toast 8 although Toast gives its own set of error codes."

Toast users will generally get an error stating: "Mismatch at byte 0/sector 0. Verification failed".

Fixes

Delete the Finder preferences file This disk verification error has been attributed to a corrupt Finder preferences file. Removing this file should clear the problem and resume normal disk verification. Open your ~/Library/Preferences/ folder, delete the com.apple.Finder.plist file, and then either restart the computer or logout and log back in (the Finder will recreate the file). When your user account has loaded up again, open Disk Utility and run a permissions verification/fix on the boot drive.

END QUOTE
*********************************************************************************************

I have trashed the Finder (and Toast) prefs, and repaired permissions, and while the first burn after doing this was successful, the following one again brought up the "Mismatch at byte X/sector X. Verification failed" error message.

Now comparing in DiffMerge the specific file on that "failed" disc with the original on my HD, which both File Buddy and Toast (Compare) reported as being different, I am told they are identical. Whom am I to trust? These are files I am archiving and deleting from my HD, and cannot afford to archive error-riddled files. I might be inclined to believe (especially given the fact that the byte number is always the same, while the sector number changes) that the errors are not actually on the burned discs themselves, but are simply symptoms of a malfunctioning Finder or operating system, except that one of these instances was actually audible as a glitch in a FLAC file on a burned DVD.

So what I am asking is whether there is anything about the DIffMerge routines that would make them completely reliable on what seems to be a somewhat compromised system such as my own, unlike those of Toast (Compare utility) and File Buddy?

BTW, all the original and burned files which are reported as having differences have an identical number of bytes when I do a Get Info on them.

Thank you for this free utilty and for your kind consideration of, and assistance with, this most vexing problem!

Raf

[jeffhostetler]

uh, wow. i've never seen anything like that before.

DiffMerge only uses the OS's file open() and read() routines, and so I have to trust
whatever the OS gives me. My reliability depends on the OS. If the OS is giving us
somewhat unpredictable data, there's nothing I can do about it.

Also, DiffMerge will read the files as and try to convert them from whatever character
encoding they happen to be in into Unicode. It does not handle binary files. Together
these 2 things mean that it will not load a file containing a NUL byte or it may have
issues if it can't determine the character encoding. But both of these events will
cause an error/warning dialog. So, if you're not seeing this, it's probably not an
issue.

Some things to try:
[1] copy the original data to a different directory on the hard disk (or an external usb drive)
and see how the copy compares to the original and/or the DVD.
[2] copy the DVD data back to the hard disk (in a different directory) and see how it compares
to the original and/or the DVD.

Either of these might or might not give an indication if you have a bad spot under the
original data and/or separate the DVD compare from the act of reading the DVD. (This
may or may not tell you anything, but it might be worth a try.)

[3] open a Terminal window and use /sbin/md5 to compute a checksum on each file in the
original directory and the DVD and/or on the copies and compare them. the checksums
will tell you if they are different or identical -- without any fluff.

[4] turn down the speed on the DVD burner and make sure that the machine if completely
idle while burning.

[5] maybe run the DiskUtility.app and do a Verify Disk on the hard disk. (do a full backup first)

Those are some things that I would try, but I'm just guessing here.

hope that helps and good luck,
jeff

[sraf]

Thanks, Jeff, for the prompt response!

I did try burning a disc at a lower speed after trashing the Finder prefs, but still got the dreaded "Verification failed" message. On the bright side, yesterday I tried burning a disc (wisely moving to DVD-RW now as I have enough coasters already) using Disk Utility instead of Toast, and that burn passed verification, so maybe that is the way to go from now on (assuming that was not a freak successfully-verified burn, as happens with Toast as well)! But it also occurs to me that burning via Disk Utility, I am essentially gathering all the files I am going to burn in one location (the .dmg file) on my boot drive (where the original files that are compared to in the verification process are those gathered-together ones), while in using Toast I usually just drag files from a number of different locations on both my internal HD and external FW HD (where, I assume, the files that are compared to in the verification process are the original ones which can be located on a number of different volumes or HDs). If there were some problem with the external FW HD, might this not account for anomalies involving the verification process when original files stored on the external HD were involved (especially if these anomalies were not present when burning files that were all gathered in one location on the internal HD)?

Another strange thing is that verification also failed when I burned a disc in Toast from the Tiger partition on my external FW HD. I haven't used that drive for anything but running Disk Utility to verify/fix my main (internal) drive in many months, and never had any problem burning/verifying discs when I was last using it regularly, so how likely is it that I would have corrupt Finder prefs (causing this problem) there? (BTW, I did trash the Finder prefs/check permissions on that drive as well and verification still failed).

I should mention that I did, prior to reading your suggestion, compare one original file with one burned file which failed verification which I then copied to my HD, and that verification failed as well.

Re encodings, etc., most of the files I am burning are audio formats such as APE, FLAC, and ALAC. One of the files, as I mentioned, had an audible glitch, but all the other "failed" files played fine with no audible glitches. How likely do you think it is that we are actually dealing with "bad burns" or errors actually written to disc, when:

a) the byte size of the original and the burned files is always identical

and

b) the mismatch is ALWAYS reported as occurring at byte 560, but of completely different sectors?

I haven''t checked out all the DiffMerge options yet, but is it possible to go in and inspect the data at specific bytes in specific sectors (or do you know of another utility that does that)? I used to use Quadrivio General Edit Lite for this, but I have not been able to get it to run under Tiger.

QUOTE:
[3] open a Terminal window and use /sbin/md5 to compute a checksum on each file in the
original directory and the DVD and/or on the copies and compare them. the checksums
will tell you if they are different or identical -- without any fluff.
END QUOTE

Would you have a link to, or info to post here with, precise instructions for performing this operation? I am a complete novice in the realm of Terminal (having only used it thus far to turn off Spotlight indexing), but if this method is absolutely reliable then it is obviously the way to go with questionable files.


Thanks again for your kind assistance!

Raf

[jeffhostetler]

If the files are all audio files, don't bother trying to get DiffMerge to compare them.
It doesn't handle binary files. It's designed for text files and is expecting them to
contain lines of text. It'll either fail immediately or go out to lunch on binary files.

I'm not sure I can comment on the various disk burning questions and/or hard disk
problems. I've not really used those tools on the Mac. You might post a question
on one of the Apple forums or visit one of the Genius Bars if there's an Apple store
near you. The byte 560 is probably significant, and someone over there might know
what it means. I'd do some more digging on Apple's support KB or on google.

As for the Terminal window suggestion: open a Terminal window and "cd" to one of
the directories that contain some of the files. Type "/sbin/md5 file" and it should
print a checksum. You can also type "/sbin/md5 *" and it will checksum them all.
Open another Terminal window and "cd" to the directory on the DVD and do the same.
You should be able to compare the checksums. This will tell you if they are the
same or different, but I'd wait on this until you've tracked down things on Apple's
KB.

Sorry I can't be more helpful,
jeff

[sraf]

So I guess that means the positive verification DM gave on several folders of audio files is not reliable/meaningful? Oh, well...

Re OS X forums/Genius Bar, I've gone through almost all the posts (of which there are VERY few) I could find on this issue in the Apple KB and in other OS X and Toast forums, and the best information was that which I pasted here in my fist post. I think the Genius Bar would be a waste of time/money: they would undoubtedly suggest that I do a system reinstall (after running a variety of disk utilities, which I have already done), which I agree would be a wise thing to do next, but I cannot do so until I have backed up a lot of data on my bloated HDDs to DVD.

I don't think the specific byte number that keeps popping up means anything: all users with this problem report failed verifications with "mismatch at byte X/sector Y," the byte number always being the same in every error message, though different for each user.

Thanks for the details on using Terminal to compute checksums. I also just found a more recent free hex editor with GUI for OS X (OxED) which I could use to look at the data at specific byte locations/sectors. Do you think if the system is reporting incorrectly when file comparison utilities are run that it could do so equally when showing data in a hex editor with GUI? Would you say that computing checksums using Terminal the only absolutely foolproof method for determining the identicalness or otherwise of two binary files in OS X (given the issue I have been dealing with)?

Thanks again, Jeff!

[jeffhostetler]

So I guess that means the positive verification DM gave on several folders of audio files is not reliable/meaningful? Oh, well...

well, if was able to read the files (and you didn't get any of the "binary file" or
"character encoding" complaints), then the results should be valid.

if you have 3.3.0, you can do a Folder Compare (and have "Disable File Equivalence -
Use Exact Match Only" set in the Options Dialog), you can point DiffMerge at the 2
directories of files and get a quick scan (without worrying about binary or encodings).

I really don't have any idea about the byte/sector thing. I was only going on what
you were describing. Again, this all sounds like an OS and/or disk problem.
And if that is the case, then *ANY* tool is going to give you questionable
results -- whether it is DiffMerge or Toast or a command line tool or whatever, they
all have to use the OS to read the files.... So, if the OS or disk is flaky, your
results may vary.

If I were you and had critical data, I'd go get a *big* external USB hard-drive
and drag everything over to it (maybe even make 2 copies on the external
drive if there's space). Then rebuild/repair the OS HD and then retry burning
the DVD see what happens.

Hope this helps,
jeff

[sraf]

Yes, I was actually gonna go get another 500 GB HD last weekend, only to discover that they are very difficult, if not impossible, to find now in the ATA/IDE version (which is all my machine and FW case take). I have an early G4 with upgraded processor and HDDs, and I cannot use SATA drives on my system. So I'm looking into used ATA ones on CL (though buying a used HD with a couple of years of use on it already is opening up a whole other can of worms, it seems to me...).

I haven't had the chance in the last day or two to do so, but at the earliest opportunity I'm going to go back and take another look at those supposedly bad burns using the variety of tools we've been discussing -- hopefully that will at least help me to determine one and for all if these are simply reporting errors or actually bad burns that I've been encountering. My greatest concern now is that, if the file comparisons are unreliable, then it may be the case that even successfully verified burns (as with my last one using Disk Utility) may have errors that are going unreported.

Once again, thanks for your suggestions and helpful attention to this!

[sraf]

If I were you and had critical data, I'd go get a *big* external USB hard-drive
and drag everything over to it (maybe even make 2 copies on the external
drive if there's space).

Unfortunately, I do not have USB 2 on my system, so external USB drives are also out!
I know... maybe time to think about getting a new machine!