The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
TaskPipe-0.01
River stage zero No dependents
1 ++
/ TaskPipe::Task::Settings

NAME

TaskPipe::Task::Settings - Settings for TaskPipe::Task

METHODS

plan_mode

The format that taskpipe expects to find your plan in. There are 2 modes, tree and branch. If your tasks will always be executed in order (ie your plan is completely linear) then this is the mode to choose. This way you can write your plan thus:

    ---

    - name: Scrape_Example
      url: www.example.com

    - name: Record
      example_param: $this

tree format is slightly more complex, offering the ability to execute different tasks in parallel (ie the plan can have more than one branch). In tree format, tasks should be designated using the keyword task: and cascaded using pipe_to:. For example:

    ---
    task:
        _name: Scrape_Example
        url: www.example.com

    pipe_to:
        task:
            name: Record
            example_param: $this

An example of how to achieve branching in tree format is as follows:

    ---
    task:
        _name: Scrape_Example
        url: www.example.com
    
    pipe_to:

        - task:
            name: Record
            example_param: $this

        - task:
            name: Scrape_SomethingElse
            another_param: $this[1]

The tasks Record and Scrape_SomethingElse execute in parallel. See the general documentation for more information on plan modes and branching

persist_xbranch_record

Once an xbranch has been 'seen' - ie all the sub-xbranches have been executed, taskpipe can clear all records of those sub-xbranches in order to keep the database trim. The upside to persisting the xbranch cache is that any (sub) xbranch can be targetted in a rerun so that only that xbranch gets executed without executing the whole plan. The downside is a substantially larger database.

Persisting the xbranch cache may be a good idea during development when errors are expected frequently, but is probably less desirable in production with less frequent errors and a much larger amount of data.

persist_result_cache

Once an xbranch has been 'seen' without errors, taskpipe can clear all cached results associated with the xtasks that appear on the xbranch, in order to keep the database trim. The upside to persisting the result cache is that if the plan is changed, taskpipe will only execute any new or changed xtasks without needing to execute unchanged ones (since the unchanged xtasks are cached). The downside is considerable database overhead.

Like persisting the xbranch record, persisting the result cache may be a good idea during development when errors and changes are expected frequently, but is probably less desirable in production with a much larger amount of data.

threads

The maximum number of threads to use when running a plan. Taskpipe tries to adhere strictly to the number of threads you specify here - so parent threads are included in the value. You should experiment with your setup to determine the optimum value for your system

on_task_error

What to do if an error is encountered. Options are stop (ie attempt to stop all threads), and continue (which will log the error and continue)

AUTHOR

Tom Gracey <tomgracey@gmail.com>

COPYRIGHT AND LICENSE

Copyright (c) Tom Gracey 2018

TaskPipe is free software, licensed under

    The GNU Public License Version 3