I'm a bad librarian, I don't know how to do a search for this.

[snarp]

I need a piece of software that will thwart evil.

I have about a hundred text files, which contain lists of about eighty numbered questions, 4000 words in length. Most of them seem to be identical text-wise, but I've turned up one that has some very minor differences.

I need a way to search through all these files and figure out if there are any others with different text. I really don't want to have to skim 1600 pages of this stuff - I'm afraid I'll miss something, and I've got more important things to be doing. I guess conceivably I could put a script together myself, but I've only recently started playing with regular expressions, and it might take longer than doing the project by hand.

The thwarting evil part comes in here: We were sent these files by a horrible law firm representing a horrible corporation, who inserted these discrepancies precisely to waste our time. The horrible corporation needs representation because they recently -ed over several hundred desperately poor people, a number of whom are now homeless. There will come a reckoning. But first I need, like, a Python script or something.

Comments

[vector]

There should be a variety of things that let you diff two files, if the files are themselves almost identical. Let's see... try looking at diff? That wiki page has a section on "free file comparison tools" that might have what you want. You may or may not then still need a really basic script to iterate through the files and compare them all.

[azurelunatic]

The rather low-tech way one might start, without scripting, but assumes someone doing grunt work:

If they're numbered, and each number should be the same, one ouuuuught to be able to put them in a table, with one column for each file, each question or line being one cell. I believe that the spreadsheet should be able to evaluate equality of text strings, in addition to just numbers. Fill down. Set a conditional so that equal displays as green, unequal displays as red. Scan down for red.

[nicki]

could you feed them into one of those programs that checks for plagarism in term papers? My understanding is that they highlight identical text in some color and original text in another, so you could just identify the "original" text and discard the rest?

uncouth

[lacrimawanders.livejournal.com]

Step 3: Cockslap the asshats.* I have service called "Slaps Across America" that can help with that part.

* Kidding. Seriously.

[snarp]

Thanks! That looks like what I need. I'll play around with it.

Re: uncouth

[snarp]

I don't think you're actually kidding, and I'm very concerned.

Re: uncouth

[lacrimawanders.livejournal.com]

Well, I wouldn't actually /cockslap/ them, per se.

[alexey-rom.livejournal.com]

If identical files are exactly identical, calculate their hash with something like http://www.fourmilab.ch/md5/ Any difference will lead to very different hashes, so you'll see different files immediately. Then to find what actual differences are, use any diff software.