<?xml version='1.0'?>

<!--
SPDX-FileCopyrightText: Copyright The SCons Foundation (https://scons.org)
SPDX-License-Identifier: MIT
SPDX-FileType: DOCUMENTATION

This file is processed by the bin/SConsDoc.py module.
-->

<!DOCTYPE sconsdoc [
    <!ENTITY % scons SYSTEM "../scons.mod">
    %scons;

    <!ENTITY % builders-mod SYSTEM "../generated/builders.mod">
    %builders-mod;
    <!ENTITY % functions-mod SYSTEM "../generated/functions.mod">
    %functions-mod;
    <!ENTITY % tools-mod SYSTEM "../generated/tools.mod">
    %tools-mod;
    <!ENTITY % variables-mod SYSTEM "../generated/variables.mod">
    %variables-mod;
]>

<chapter id="chap-scanners"
         xmlns="http://www.scons.org/dbxsd/v1.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://www.scons.org/dbxsd/v1.0 http://www.scons.org/dbxsd/v1.0/scons.xsd">
<title>Extending &SCons;: Writing Your Own Scanners</title>

<!--

=head1 Using and writing dependency scanners

QuickScan allows simple target-independent scanners to be set up for
source files. Only one QuickScan scanner may be associated with any given
source file and environment, although the same scanner may (and should)
be used for multiple files of a given type.

A QuickScan scanner is only ever invoked once for a given source file,
and it is only invoked if the file is used by some target in the tree
(i.e., there is a dependency on the source file).

QuickScan is invoked as follows:

  QuickScan CONSENV CODEREF, FILENAME [, PATH]

The subroutine referenced by CODEREF is expected to return a list of
filenames included directly by FILE. These filenames will, in turn, be
scanned. The optional PATH argument supplies a lookup path for finding
FILENAME and/or files returned by the user-supplied subroutine.  The PATH
may be a reference to an array of lookup-directory names, or a string of
names separated by the system's separator character (':' on UNIX systems,
';' on Windows NT).

The subroutine is called once for each line in the file, with $_ set to the
current line. If the subroutine needs to look at additional lines, or, for
that matter, the entire file, then it may read them itself, from the
filehandle SCAN. It may also terminate the loop, if it knows that no further
include information is available, by closing the filehandle.

Whether or not a lookup path is provided, QuickScan first tries to lookup
the file relative to the current directory (for the top-level file
supplied directly to QuickScan), or from the directory containing the
file which referenced the file. This is not very general, but seems good
enough, especially if you have the luxury of writing your own utilities
and can control the use of the search path in a standard way.

Here's a real example, taken from a F<Construct> file here:

  sub cons::SMFgen {
      my($env, @tables) = @_;
      foreach $t (@tables) {
	  $env->QuickScan(sub { /\b\S*?\.smf\b/g }, "$t.smf",
			  $env->{SMF_INCLUDE_PATH});
	  $env->Command(["$t.smdb.cc","$t.smdb.h","$t.snmp.cc",
			 "$t.ami.cc", "$t.http.cc"], "$t.smf",
			q(smfgen %( %SMF_INCLUDE_OPT %) %<));
      }
  }

The subroutine above finds all names of the form <name>.smf in the
file. It will return the names even if they're found within comments,
but that's OK (the mechanism is forgiving of extra files; they're just
ignored on the assumption that the missing file will be noticed when
the program, in this example, smfgen, is actually invoked).

[NOTE that the form C<$env-E<gt>QuickScan ...>  and C<$env-E<gt>Command
...> should not be necessary, but, for some reason, is required
for this particular invocation. This appears to be a bug in Perl or
a misunderstanding on my part; this invocation style does not always
appear to be necessary.]

Here is another way to build the same scanner. This one uses an
explicit code reference, and also (unnecessarily, in this case) reads
the whole file itself:

  sub myscan {
      my(@includes);
      do {
	  push(@includes, /\b\S*?\.smf\b/g);
      } while <SCAN>;
      @includes
  }

Note that the order of the loop is reversed, with the loop test at the
end. This is because the first line is already read for you. This scanner
can be attached to a source file by:

  QuickScan $env \&myscan, "$_.smf";

This final example, which scans a different type of input file, takes
over the file scanning rather than being called for each input line:

  $env->QuickScan(
      sub { my(@includes) = ();
	  do {
	     push(@includes, $3)
		 if /^(#include|import)\s+(\")(.+)(\")/ && $3
	  } while <SCAN>;
	  @includes
      },
      "$idlFileName",
      "$env->{CPPPATH};$BUILD/ActiveContext/ACSCLientInterfaces"
  );

-->

  <para>

    &SCons; has built-in &Scanners; that know how to look in
    C/C++, Fortran, D, IDL, LaTeX, Python and SWIG source files
    for information about
    other files that targets built from those files depend on.

    For example, if you have a file format which uses <literal>#include</literal>
    to specify files which should be included into the source file
    when it is processed, you can use an existing scanner already
    included in &SCons;.

    You can use the same mechanisms that &SCons; uses to create
    its built-in Scanners to write Scanners of your own for file types
    that &SCons; does not know how to scan "out of the box."

  </para>

  <section id="sect-simple-scanner">
  <title>A Simple Scanner Example</title>

    <para>

      Suppose, for example, that we want to create a simple &Scanner;
      for <filename>.k</filename> files.
      A <filename>.k</filename> file contains some text that
      will be processed,
      and can include other files on lines that begin
      with <literal>include</literal>
      followed by a file name:

    </para>

    <programlisting>
include filename.k
    </programlisting>

    <para>

      Scanning a file will be handled by a Python function
      that you must supply.
      Here is a function that will use the Python
      <systemitem>re</systemitem> module
      to scan for the <literal>include</literal> lines in our example:

    </para>

    <programlisting>
import re

include_re = re.compile(r'^include\s+(\S+)$', re.M)

def kfile_scan(node, env, path, arg=None):
    contents = node.get_text_contents()
    return env.File(include_re.findall(contents))
    </programlisting>

    <para>

      It is important to note that you
      have to return a list of File nodes from the scanner function, simple
      strings for the file names won't do.
      As in the examples we are showing here,
      you can use the &f-link-File;
      function of your current &consenv;  in order to create nodes
      on the fly from a sequence of file names with relative paths.

    </para>

    <para>

      The scanner function must
      accept the four specified arguments
      and return a list of implicit dependencies.
      Presumably, these would be dependencies found
      from examining the contents of the file,
      although the function can perform any
      manipulation at all to generate the list of
      dependencies.

    </para>

    <variablelist>

      <varlistentry>
      <term><parameter>node</parameter></term>

      <listitem>
      <para>

      An &SCons; node object representing the file being scanned.
      The path name to the file can be
      used by converting the node to a string
      using the <function>str</function> function,
      or an internal &SCons; <methodname>get_text_contents</methodname>
      object method can be used to fetch the contents.

      </para>
      </listitem>
      </varlistentry>

      <varlistentry>
      <term><parameter>env</parameter></term>

      <listitem>
      <para>

      The &consenv; in effect for this scan.
      The scanner function may choose to use &consvars;
      from this environment to affect its behavior.

      </para>
      </listitem>
      </varlistentry>

      <varlistentry>
      <term><parameter>path</parameter></term>

      <listitem>
      <para>

      A list of directories that form the search path for included files
      for this Scanner.
      This is how &SCons; handles the &cv-link-CPPPATH; and &cv-link-LIBPATH;
      variables.

      </para>
      </listitem>
      </varlistentry>

      <varlistentry>
      <term><parameter>arg</parameter></term>

      <listitem>
      <para>

      An optional argument that can be passed
      to this scanner function when it is called from
      a  scanner instance. The argument is only supplied
      if it was given when the scanner instance is created
      (see the manpage section "Scanner Objects").
      This can be useful, for example, to distinguish which
      scanner type called us, if the function might be bound
      to several scanner objects.
      Since the argument is only supplied in the function
      call if it was defined for that scanner, the function
      needs to be prepared to possibly be called in different
      ways if multiple scanners are expected to use this
      function - giving the parameter a default value as
      shown above is a good way to do this.
      If the function to scanner relationship will be 1:1,
      just make sure they match.

      </para>
      </listitem>
      </varlistentry>

    </variablelist>

    <para>

    A scanner object is created using the &f-link-Scanner; function,
    which typically takes an <parameter>skeys</parameter> argument
    to associate a file suffix with this Scanner.
    The scanner object must then be associated with the
    &cv-link-SCANNERS; &consvar; in the current &consenv;,
    typically by using the &f-link-Append; method:

    </para>

    <programlisting>
kscan = Scanner(function=kfile_scan, skeys=['.k'])
env.Append(SCANNERS=kscan)
    </programlisting>

    <para>

    Let's put this all together.
    Our new file type, with the <filename>.k</filename> suffix,
    will be processed by a command named <command>kprocess</command>,
    which lives in non-standard location
    <filename>/usr/local/bin</filename>,
    so we add that path to the execution environment so &SCons;
    can find it. Here's what it looks like:

    </para>

    <scons_example name="scanners_scan">
      <file name="SConstruct" printme="1">
import re

include_re = re.compile(r'^include\s+(\S+)$', re.M)

def kfile_scan(node, env, path):
    contents = node.get_text_contents()
    includes = include_re.findall(contents)
    return env.File(includes)

kscan = Scanner(function=kfile_scan, skeys=['.k'])
env = Environment()
env.AppendENVPath('PATH', '__ROOT__/usr/local/bin')
env.Append(SCANNERS=kscan)

env.Command('foo', 'foo.k', 'kprocess &lt; $SOURCES &gt; $TARGET')
      </file>
      <file name="foo.k">
some initial text
include other_file
some other text
      </file>
      <!-- # leave dep file out to show scanner works via dep not found
      file name="other_file">
text to include
      </file>
      -->
      <directory name="__ROOT__/usr"></directory>
      <directory name="__ROOT__/usr/local"></directory>
      <directory name="__ROOT__/usr/local/bin"></directory>
      <file name="__ROOT_/usr/local/bin/kprocess" chmod="755">
cat
      </file>
    </scons_example>

    <para>

    Assume a <filename>foo.k</filename> file like this:

    </para>

    <scons_example_file example="scanners_scan"  name="foo.k">
    </scons_example_file>

    <para>

    Now if we run &scons; we can see that the scanner works -
    it identified the dependency
    <filename>other_file</filename> via the detected
    <literal>include</literal> line,
    although we get an error message because we
    forgot to create that file!

    </para>

    <scons_output example="scanners_scan" suffix="1">
      <scons_output_command>scons -Q</scons_output_command>
    </scons_output>

  </section>

  <section id="sect-scanner-search-paths">
  <title>Adding a search path to a Scanner: &FindPathDirs;</title>

    <para>

    If the build tool in question will use a path variable to search
    for included files or other dependencies, then the &Scanner; will
    need to take that path variable into account as well -
    the same way &cv-link-CPPPATH; is used for files processed
    by the C Preprocessor (used for C, C++, Fortran and others).
    Path variables may be lists of nodes or semicolon-separated strings
    (&SCons; uses a semicolon here irrespective of
    the pathlist separator used by the native operating system),
    and may contain &consvars; to be expanded.
    A Scanner can take a <parameter>path_function</parameter>
    to process such a path variable;
    the function produces a tuple of paths that is passed to the
    scanner function as its <parameter>path</parameter> parameter.

    </para>

    <para>

    To make this easy,
    &SCons; provides the premade  &f-link-FindPathDirs;
    function which returns a callable to expand a given path variable
    (given as an &SCons; &consvar; name)
    to a tuple of paths at the time the Scanner is called.
    Deferring evaluation until that point allows, for instance,
    the path to contain &cv-link-TARGET; references which differ for
    each file scanned.

    </para>

    <para>

    Using &FindPathDirs; is easy.  Continuing the above example,
    using <envar>$KPATH</envar> as the &consvar; to hold the paths
    (analogous to &cv-link-CPPPATH;), we just modify the call to
    the &f-link-Scanner; factory function to include a
    <parameter>path_function</parameter> keyword argument:

    </para>

    <scons_example name="scanners_findpathdirs">
      <file name="SConstruct" printme="1">
kscan = Scanner(
    function=kfile_scan,
    skeys=['.k'],
    path_function=FindPathDirs('KPATH'),
)
      </file>
    </scons_example>

    <para>

    &FindPathDirs; is called when the Scanner is created,
    and the callable object it returns is stored
    as an attribute in the scanner.
    When the scanner is invoked, it calls that object,
    which processes the <envar>$KPATH</envar> from the
    current &consenv;, doing necessary expansions and,
    if necessary, adds related repository and variant directories,
    producing a (possibly empty) tuple of paths
    that is passed on to the scanner function.
    The scanner function is then responsible for using that list
    of paths to locate the include files identified by the scan.
    The next section will show an example of that.

    </para>

    <para>

    As a side note,
    the returned method stores the path in an efficient way so
    lookups are fast even when variable substitutions may be needed.
    This is important since many files get scanned in a typical build.

    </para>

  </section>

  <section id="sect-scanner-with-builder">
  <title>Using scanners with Builders</title>

    <para>
    One approach for introducing a &Scanner; into the build is in
    conjunction with a &Builder;.  There are two relevant optional
    parameters we can use when creating a Builder:
    <parameter>source_scanner</parameter> and
    <parameter>target_scanner</parameter>.
    <parameter>source_scanner</parameter> is used for scanning
    source files, and <parameter>target_scanner</parameter>
    is used for scanning the target once it is generated.
    </para>

    <scons_example name="scanners_builders">
      <file name="SConstruct" printme="1">
import os, re

include_re = re.compile(r"^include\s+(\S+)$", re.M)

def kfile_scan(node, env, path, arg=None):
    includes = include_re.findall(node.get_text_contents())
    print(f"DEBUG: scan of {str(node)!r} found {includes}")
    deps = []
    for inc in includes:
        for dir in path:
            file = str(dir) + os.sep + inc
            if os.path.exists(file):
                deps.append(file)
                break
    print(f"DEBUG: scanned dependencies found: {deps}")
    return env.File(deps)

kscan = Scanner(
    function=kfile_scan,
    skeys=[".k"],
    path_function=FindPathDirs("KPATH"),
)

def build_function(target, source, env):
    # Code to build "target" from "source"
    return None

bld = Builder(
    action=build_function,
    suffix=".k",
    source_scanner=kscan,
    src_suffix=".input",
)

env = Environment(BUILDERS={"KFile": bld}, KPATH="inc")
env.KFile("file")
      </file>
      <file name="file.input">
some initial text
include other_file
some other text
      </file>
      <file name="inc/other_file">
text to include
      </file>
    </scons_example>

    <para>
    Running this example would only show that the stub
    <function>build_function</function> is getting called,
    so some debug prints were added to the scanner function,
    just to show the scanner is being invoked.
    </para>

    <scons_output example="scanners_builders" suffix="1">
      <scons_output_command>scons -Q</scons_output_command>
    </scons_output>

    <para>
    The path-search implementation in
    <function>kfile_scan</function> works,
    but is quite simple-minded - a production scanner
    will probably do something more sophisticated.
    </para>


    <para>

    An emitter function can modify the list of sources or targets
    passed to the action function when the Builder is triggered.

    </para>

    <para>

    A scanner function will not affect the list of sources or targets
    seen by the Builder during the build action. The scanner function
    will, however, affect if the Builder should rebuild (if any of
    the files sourced by the Scanner have changed for example).

    </para>
  </section>

</chapter>
