NAME

makepp_scanning -- How makepp finds include files and other hidden dependencies

DESCRIPTION

Makepp can guess additional dependencies or targets for certain commands that it knows something about. This is especially important for C/C++ compilation, where it is too error-prone to list manually all of the include files that a given source file depends on. By looking at the compilation command and the source files themselves, makepp is able to determine accurately which object files need to be rebuilt when some include file changes.

Makepp looks at the first word of your command line. If it recognizes it, it uses the scanner corresponding to that first word. Specifically, it isolates the first word and looks it up in its table; if nothing is found, it strips off the directory information and looks it up again. (This means that you can specify the path to your compiler and makepp will still recognize it.) Currently, makepp recognizes most C/C++ compiler names. It also recognizes commands invoking sh and libtool; for these commands, it skips to the first word in the command that's not an option and looks it up in the table again.

If makepp thinks it's compiling a C/C++ program but can't find a scanner, it will give a warning message to let you know. This usually means that you buried your compiler command too deeply in the action for makepp to find it. For example, I have seen rules like this:

    %.o: %.c
        @echo Compiling $< now
        @gcc -c $< $(CFLAGS) -o $@

The first word of the action here is echo, for which therer is no scanner, so makepp will not scan for include files in this case.

C/C++ compilation

The C/C++ scanner is activated by a command beginning with a compile program that makepp knows about. (I've included every compiler I've ever heard of, but if I missed your compiler, you can tell makepp about it by adding an entry to the %Makesubs::scanners array (see Makesubs.pm in the distribution). Also send me email at holt-makepp@gholt.net so I can include it in subsequent releases.

It looks at the command for -Idir options specifying the include path or -Ldir options specifying the link path. It then scans any source files for #include directives, and also looks at the command line to see if there are any source files or libraries mentioned which are not listed as dependencies. It recognizes these by their extension.

This scanner gives a warning message if files included with #include "file.h" are not found in the include path, or in the directory containing the file which is #including, or in /usr/include. No warning is given if a file included with #include <file.h> is not found. makepp assumes it is in some system include directory that the compiler knows about, and that files in system include directories won't change.

In addition, files in /usr/include, /usr/local/include, /usr/X11R6/include, and any other directory which is not writable are not scanned to see what they include. Makepp assumes that these files won't change. (If you're running as root, the writability test is performed with the UID and GID of the directory you ran makepp from. This is so compiling a program as an ordinary user and then doing makepp install as root won't cause extra directories to be scanned.)

This is a fairly simple-minded scanner. It will get confused if you do things like this:

    #ifdef INCLUDE_THIS
    #include "this.h"
    #endif

because it doesn't know about preprocessor conditionals. This is usually harmless; it might cause additional extra files to be labelled as dependencies (occasionally causing unnecessary rebuilds), or else it might cause makepp to warn that the include file was not found. You can either ignore the warning messages, or put an empty file this.h out there to shut makepp up.

Libtool

Libtool is a very clever compilation system that greatly simplifies making shared libraries by hiding all the system-dependent details away in a shell script. The only difficulty is that the library binary files are not actually stored in the same directory as the output file--libtool actuall creates a subdirectory, .libs, which contains the real files. This is ordinarily not a problem, but makepp has to know where the real binaries are if it is to link them in from a repository. At the moment, libtool libraries (.la files) are not linked in from repositories; they are always rebuilt if needed. Also, makepp at the moment is not able to use the dependency information that is stored inside the .la file itself. This will hopefully change soon.

Swig

Swig (Simplified Wrapper and Interface Generator, http://www.swig.org) is a program that converts a C/C++ header file into the wrapper functions needed to make your code callable from a variety of other languages, such as perl, python, tcl, C#, ruby, ocaml, and probably some others that I don't know about.

Makepp understands and parses the swig command line, looking for -I and -l options. It also knows how to scan swig's interface definition files (.i files) looking for %include, %import, and also #include if -includeall is in effect.

Other special command words

Makepp recognizes the following command words and skips over them appropriately in in its search for the correct scanner: condor_compile, echo ignore_error libtool, noecho purify, sh.

Quickscan and smartscan

The :quickscan and :smartscan rule options, if applicable, affect the way that files are scanned.

In :quickscan mode (the default), all include directives are assumed active. This allows for very efficient scanning.

In :smartscan mode, an attempt is made to interpret macros and expressions so that inactive include directives are ignored. For example, the executable produced by compiling the following C program ought not to depend on foo.h:

    #if 0
    # include "foo.h"
    #endif
    int main() { return 0; }

In the future, :smartscan might become the default.

Custom scanners

Writing your own scanner is somewhat involved, but it is possible. There are 3 ways to hook into this:

1. If scanner foo is specified in a rule option, and there is a subroutine called parser_foo defined in the makefile's package, then that subroutine must return an object of type ActionParser, which will be used to determine a command parser for every command.

2. If scanner foo is specified in a rule option, and there is a subroutine called scanner_foo defined in the makefile's package, then that subroutine must either do all of the scanning work (which is deprecated), or return an object of type CommandParser which will be used to parse every command.

3. If no scanner is specified in a rule option, then an object of the ActionParser base class is used. That object determines the scanner based on the first word of each command. The base name is sought as a key in %scanners hash in the makefile's package, which is seeded with all of the default scanners. This hash can be extended using the register_scanner and register_command_parser statements. The value from this hash lookup is a coderef that either does the scanning work (which is deprecated), or returns an object of type CommandParser.

In most cases, objects of type CommandParser should instantiate at least one object of type Scanner. The Scanner base class takes care of the distinction between quickscan and smartscan. Note that the behavior of Scanner can be markedly affected by this distinction, but that should be transparent to the derived class if it is well-formed. New derived Scanner classes ought to be tested in both modes.

If you want to specify a particular CommandParser class for a rule, then you can specify a scanner with a rule option, and arrange for the parser_* subroutine to return an object of type ActionParser::Specific.

For more details, refer to the respective class documentation. For examples, see CommandParser::Gcc and CommandParser::Vcs.

Caching scanner info

If the all of the scanner's important side effects are effected through calls to methods of the CommandParser base class, then those side effects can be cached in the build info file, so that they can be played back by a subsequent invocation of makepp without doing all of the costly scanning work. This can save quite a bit of time, especially in smartscan mode.

If the scanner has other important side effects, then it should call the Rule object's mark_scaninfo_uncacheable method. Otherwise, the scanner info retrieved from the build info may be inaccurate, causing the build result possibly to be incorrect. This method is called automatically when a value from the %scanners hash does not return an object of type CommandParser, or when the scanner is specified with a rule option, no parser_* subroutine is defined and the scanner_* routine does not return an object of type CommandParser.

Cached scan info is invalidated using criteria similar to those used for determining when the target is out of date. Similarly, it can be retrieved from a repository using criteria similar to those used for determining when a target can be linked in from a repository.

You can force makepp to ignore the cached scanner info with the --force-rescan option. This is useful when a broken scanner may have caused incorrect scanner info to be cached.

Ad Hoc Scanner

Often you will have just one or few files which contain dependency information. You don't want to write this into a makefile reduntantly (since redundancy later often leads to inconsistencies when one update gets forgotten). But you also don't want to write a Scanner? As a workaround you can generate an include file on the fly. For example Qt has .qrc files which can look like:

    <RCC>
      <qresource prefix="...">
        <file>abc</file>
        <file>xyz</file>
        ...

If you adhere to the above layout, you can transform the relevant lines into a makepp include file, which gets automatically created by being included.

    %.qrc.makepp: %.qrc
        &grep 's!<RCC>\n!$(stem).cc:! || s! *<file>! ! && s!</file>\n!!' $(input) -o $(output)

    include $(wildcard *.qrc)   # .makepp is appended automatically

cpanm Rule

perl -MCPAN -e shell
install Rule