COMPARISON of duplicate file utilities. This is a list of duplicate file management utilities.

Draft List Of Utils

edit
  • clink - replaces duplicates with symlinks
  • duff - uses fast O(n log n) comparison
  • dupedit - Compares many files at once without checksumming. Avoids comparing files against themselves when multiple paths point to the same file.
  • dupefinder -
  • duper -
  • dupmerge - runs across most platforms (Win32/64 with Cygwin, *nix, Linux etc.)
  • dupseek - Perl with algorithm optimized to reduce reads
  • epac - Erik's Partial Archive Collator - replaces duplicates with hardlinks
  • fdf - Perl/c based and runs across most platforms (Win32, *nix and probably others). Uses MD5, SHA1 and other checksum algorithms
  • fdupe - a small script written in Perl. Doing its job fast and efficiently.
  • fdupes -
  • findrepe - Java-based command-line tool; can search within zips and jars. (Linux, Mac OS X, *nix, Windows)
  • freedup - POSIX C compliant and runs across platforms (Windows with Cygwin, Linux, AIX, etc.)
  • freedups - shell script
  • fslint - has command line interface and GUI.
  • ftwin -
  • liten - Pure Python deduplication command line tool, and library, using md5 checksums and a novel byte comparison algorithm. (Linux, Mac OS X, *nix, Windows)
  • liten2 - A rewrite of the original Liten, still a command line tool but with a faster interactive mode using SHA-1 checksums (Linux, Mac OS X, *nix)
  • rdfind - One of the few which rank duplicates based on the order of input parameters (directories to scan) in order not to delete in "original/well known" sources (if multiple directories are given). Uses MD5 or SHA1. uses fast O(n log n) comparison
  • rmlint - Fast finder with command line interface and many options to find other lint too (uses MD5)
  • ssdeep - identify almost identical files using Context Triggered Piecewise Hashing
  • ua - Unix/Linux command line tool, designed to work with find (and the like).

Related: duplicate image match finders Related: data deduplication

DummySortableList

edit
FIXME*FIXME misc duplicate file utilities
Name FIXMECategory Description First appeared
admin SCCS Create and administer SCCS files PWB UNIX
alias Misc Define or display aliases
ar Misc Create and maintain library archives Version 1 AT&T UNIX
what SCCS Identify SCCS files PWB UNIX
xargs Shell Programming Construct argument lists and invoke utility PWB UNIX
yacc C Programming Yet another compiler compiler PWB UNIX
zcat Text Processing Expand and concatenate data 4.3BSD

See also

edit

Footnotes

edit
edit

{X{wikibooks|Guide to UNIX|Commands}} {X{ComXmons category|Unix cheatsheets}}

{X{UnXix commands}}

[X[CaXtegory:Lists of software|Unix programs]] [X[CaXtegory:System administration]] [X[CaXtegory:Unix SUS2008 utilities| ]]