NAME

TaskPipe::Task::Settings - Settings for TaskPipe::Task

OPTIONS

plan_mode

The format that taskpipe expects to find your plan in. There are 2 modes, tree and branch. If your tasks will always be executed in order (ie your plan is completely linear) then this is the mode to choose. This way you can write your plan thus:

    ---

    - name: Scrape_Example
      url: www.example.com

    - name: Record
      example_param: $this

tree format is slightly more complex, offering the ability to execute different tasks in parallel (ie the plan can have more than one branch). In tree format, tasks should be designated using the keyword task: and cascaded using pipe_to:. For example:

    ---
    task:
        _name: Scrape_Example
        url: www.example.com

    pipe_to:
        task:
            name: Record
            example_param: $this

An example of how to achieve branching in tree format is as follows:

    ---
    task:
        _name: Scrape_Example
        url: www.example.com
    
    pipe_to:

        - task:
            name: Record
            example_param: $this

        - task:
            name: Scrape_SomethingElse
            another_param: $this[1]

The tasks Record and Scrape_SomethingElse execute in parallel. See the general documentation for more information on plan modes and branching

cache_results

Cache xtask results. This prevents the same xtask (ie the same task with the same pinterp being executed again (the results will just be piped to the next task without the task being executed).

test_result_limit

The maximum number of results to output when testing a task

seen_xbranch_policy

Whether to remember xbranches that have been completed or not. TaskPipe can skip seen xbranches (ie prevent them from being executed more than once) or delete them (if they are not expected to be seen more than once, this option keeps the database trim)

xbranch_key_mode

Choose 'md5' to have taskpipe create xbranch identifiers using md5 hashes. Choose 'id' to use raw id values (less flexible, but also less CPU intensive)

resume_record_interval

The number of records to process between recording resume information. For example resume_record_interval=1 would mean recording resume information every time a new record is processed. If resume_record_interval=100, 100 records would be processed between recording resume information. This hits the database a lot less, but means after premature termination, the next run may repeat processing for up to 100 records

on_task_error

What to do if an error is encountered. Options are stop (ie attempt to stop all threads), and continue (which will log the error and continue)

AUTHOR

Tom Gracey <tomgracey@gmail.com>

COPYRIGHT AND LICENSE

Copyright (c) Tom Gracey 2018

TaskPipe is free software, licensed under

    The GNU Public License Version 3