06 Jun 2017 13:39:57 UTC
- Distribution: Linux-NFS-BigDir
- Module version: 0.004
- Source (raw)
- Browse (raw)
- How to Contribute
- Issues (0)
- Testers (152 / 0 / 62)
- KwaliteeBus factor: 1
- 90.91% Coverage
- License: gpl_3
- Perl: v5.14.4
- Download (31.93KB)
- MetaCPAN Explorer
- Subscribe to distribution
- This version
- Latest version
- TO DO
- SEE ALSO
- COPYRIGHT AND LICENSE
Linux::NFS::BigDir - use Linux getdents syscall to read large directories over NFS
use Linux::NFS::BigDir qw(getdents); # entries_ref is an array reference my $entries_ref = getdents($very_large_dir);
This module was created to solve a very specific problem: you have a directory over NFS, mounted by a Linux OS, and that directory has a very large number of items (files, directories, etc). The number of entries is so large that you have trouble to list the contents with
lsfrom the shell. In extreme cases, the operation just "hangs" and will provide a feedback hours later.
I observed this behavior only with NFS version 3 (and wasn't able to simulate it with local EXT3/EXT4): you might find in different situations, but in that case it migh be a wrong configuration regarding the filesystem. Ask your administrator first.
If you can't fix (or get fixed) the problem, then you might want to try to use this module. It will use the
getdentssyscall from Linux. You can check the documentation about this syscall with
man getdentsin a shell.
In short, this syscall will return a data structure, but you probably will want to use only the name of each entry in the directory.
How can this be useful? Here are some directions:
You want to remove all directory content.
You want to remove files from the directory with a pattern in their filename (using regular expressions, for example).
You want to select specific files by their filenames and then test something else (like atime).
These are examples, but it should cover the vast majority of what you want to do.
getdentssyscall will be more effective because it will not call
statof each of those files before returning the information to you. That means, you will have the opportunity to filter whatever you need and then call
statif you really need.
I came up at
getdentsafter researching about "how to remove million of files". After a while I reached an C program example that uses
getdentsto print the filenames under the directory. By using it, I was able to cleanup directories with thousands (or even millions) of files in a couple of minutes, instead of many hours.
This module is a Perl implementation of that.
getdents_safeare exported on demand.
Expects the complete path to the directory as a parameter.
Returns an array reference with all files inside that directory but the 'dot' files.
Meanwhile simple (and probably faster), you should be careful regarding memory restrictions when using this functions.
If you have too many files, you program may try to allocate too much memory, with all the undesired effects. See
"Safe" version of
getdentsbecause it will write each entry read to a text file instead of storing all the entries on memory.
Expects as parameters:
The complete path to the directory to be read.
The complete path to the file that will be used to print each entry, one per line. As convenience, all filenames will be prepended with the complete path to the directory given as parameter.
The filename given will be created. If it already exists, this function will
This function returns the number of files read from the given directory.
Create C versions of
getdents_safewith Inline::C to see if they get close to
readdirspeed when running over a local file system (currently they are slower).
Alceu Rodrigues de Freitas Junior, <email@example.com>
This software is copyright (c) 2016 of Alceu Rodrigues de Freitas Junior, <firstname.lastname@example.org>
This file is part of Linux-NFS-BigDir distribution.
Linux-NFS-BigDir is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
Linux-NFS-BigDir is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with Linux-NFS-BigDir. If not, see <http://www.gnu.org/licenses/>.
Module Install Instructions
To install Linux::NFS::BigDir, copy and paste the appropriate command in to your terminal.
perl -MCPAN -e shell install Linux::NFS::BigDir
For more information on module installation, please visit the detailed CPAN module installation guide.
|s||Focus search bar|
|?||Bring up this help dialog|
|g p||Go to pull requests|
|g i||go to github issues (only if github is preferred repository)|
|g a||Go to author|
|g c||Go to changes|
|g i||Go to issues|
|g d||Go to dist|
|g r||Go to repository/SCM|
|g s||Go to source|
|g b||Go to file browse|