uniq-files - Report or omit duplicate file contents
This document describes version 0.132 of uniq-files (from Perl distribution App-UniqFiles), released on 2020-06-01.
Usage:
% uniq-files [-A=s] [-D] [-R] [-a] [--algorithm=s] [-c] [--count] [-d] [--debug] [--digest-args-json=s] [--digest-args=s] [--format=name] [--group-by-digest] [--json] [--log-level=level] [--(no)naked-res] [--no-group-by-digest] [--no-recurse] [--no-report-unique] [--no-show-count] [--no-show-digest] [--no-show-size] [--nogroup-by-digest] [--norecurse] [--noreport-unique] [--noshow-count] [--noshow-digest] [--noshow-size] [--page-result[=program]] [--quiet] [--recurse] [--report-duplicate=s] [--report-unique] [--show-count] [--show-digest] [--show-size] [--trace] [-u] [--verbose] [--view-result[=program]] <files> ...
Examples:
List all files which do no have duplicate contents:
% uniq-files *
List all files which have duplicate contents:
% uniq-files -d *
Move all duplicate files (except one copy) in this directory (and subdirectories) to .dupes/:
% uniq-files -D -R * | while read f; do mv "$f" .dupes/; done
List number of occurences of contents for duplicate files:
% uniq-files -c *
List number of occurences of contents for all files:
% uniq-files -a -c *
List all files, along with their number of content occurrences and content digest. Use the BLAKE2b digest algorithm. And group the files according to their digest.:
% uniq-files -a -c --show-digest -A BLAKE2,blake2b *
Given a list of filenames, will check each file size and content for duplicate content. Interface is a bit like the uniq Unix command-line program.
uniq
* marks required options.
*
What algorithm is used to compute the digest of the content.
The default is to use `md5`. Some algorithms supported include `crc32`, `sha1`, `sha256`, as well as `Digest` to use Perl <pm:Digest> which supports a lot of other algorithms, e.g. `SHA-1`, `BLAKE2b`.
If set to '', 'none', or 'size', then digest will be set to file size. This means uniqueness will be determined solely from file size. This can be quicker but will generate a false positive when two files of the same size are deemed as duplicate even though their content may be different.
See --digest-args.
--digest-args
Some Digest algorithms require arguments, you can pass them here.
See --files.
--files
Can be specified multiple times.
Sort files by its digest (or size, if not computing digest), separate each different digest.
If set to true, will recurse into subdirectories.
Whether to return duplicate items.
Default value:
2
Valid values:
[0,1,2,3]
Can be set to either 0, 1, 2.
If set to 2 (the default), will only return the first of duplicate items. For example: `file1` contains text 'a', `file2` 'b', `file3` 'a'. Only `file1` will be returned because `file2` is unique and `file3` contains 'a' (already represented by `file1`).
If set to 1, will return all the the duplicate files. From the above example: `file1` and `file3` will be returned.
If set to 3, will return all but the first of duplicate items. From the above example: `file3` will be returned. This is useful if you want to keep only one copy of the duplicate content. You can use the output of this routine to `mv` or `rm`.
If set to 0, duplicate items will not be returned.
Whether to return each file content's number of occurence.
1 means the file content is only encountered once (unique), 2 means there is one duplicate, and so on.
Show the digest value (or the size, if not computing digest) for each file.
Note that this routine does not compute digest for files which have unique sizes, so they will show up as empty.
Show the size for each file.
Alias for --report-unique --report-duplicate=1 (report all files).
See --no-report-unique.
--no-report-unique
Alias for --noreport-unique --report-duplicate=1.
Alias for --noreport-unique --report-duplicate=3.
Alias for --report-unique --report-duplicate=0.
Shortcut for --log-level=debug.
Set log level.
Shortcut for --log-level=error.
Shortcut for --log-level=trace.
Shortcut for --log-level=info.
Choose output format, e.g. json, text.
undef
Set output format to json.
When outputing as JSON, strip result envelope.
0
By default, when outputing as JSON, the full enveloped result is returned, e.g.:
[200,"OK",[1,2,3],{"func.extra"=>4}]
The reason is so you can get the status (1st element), status message (2nd element) as well as result metadata/extra result (4th element) instead of just the result (3rd element). However, sometimes you want just the result, e.g. when you want to pipe the result for more post-processing. In this case you can use `--naked-res` so you just get:
[1,2,3]
Filter output through a pager.
View output using a viewer.
Display help message and exit.
Display program's version and exit.
This script has shell tab completion capability with support for several shells.
To activate bash completion for this script, put:
complete -C uniq-files uniq-files
in your bash startup (e.g. ~/.bashrc). Your next shell session will then recognize tab completion for the command. Or, you can also directly execute the line above in your shell to activate immediately.
It is recommended, however, that you install modules using cpanm-shcompgen which can activate shell completion for scripts immediately.
To activate tcsh completion for this script, put:
complete uniq-files 'p/*/`uniq-files`/'
in your tcsh startup (e.g. ~/.tcshrc). Your next shell session will then recognize tab completion for the command. Or, you can also directly execute the line above in your shell to activate immediately.
It is also recommended to install shcompgen (see above).
For fish and zsh, install shcompgen as described above.
Please visit the project's homepage at https://metacpan.org/release/App-UniqFiles.
Source repository is at https://github.com/perlancar/perl-App-UniqFiles.
Please report any bugs or feature requests on the bugtracker website https://rt.cpan.org/Public/Dist/Display.html?Name=App-UniqFiles
When submitting a bug or request, please include a test-file or a patch to an existing test-file that illustrates the bug or desired feature.
perlancar <perlancar@cpan.org>
This software is copyright (c) 2020, 2019, 2017, 2015, 2014, 2012, 2011 by perlancar@cpan.org.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.
To install App::UniqFiles, copy and paste the appropriate command in to your terminal.
cpanm
cpanm App::UniqFiles
CPAN shell
perl -MCPAN -e shell install App::UniqFiles
For more information on module installation, please visit the detailed CPAN module installation guide.