TaskPipe::Task::Settings - Settings for TaskPipe::Task
The format that taskpipe expects to find your plan in. There are 2 modes,
branch. If your tasks will always be executed in order (ie your plan is completely linear) then this is the mode to choose. This way you can write your plan thus:
--- - name: Scrape_Example url: www.example.com - name: Record example_param: $this
treeformat is slightly more complex, offering the ability to execute different tasks in parallel (ie the plan can have more than one branch). In
treeformat, tasks should be designated using the keyword
task:and cascaded using
pipe_to:. For example:
--- task: _name: Scrape_Example url: www.example.com pipe_to: task: name: Record example_param: $this
An example of how to achieve branching in tree format is as follows:
--- task: _name: Scrape_Example url: www.example.com pipe_to: - task: name: Record example_param: $this - task: name: Scrape_SomethingElse another_param: $this
Scrape_SomethingElseexecute in parallel. See the general documentation for more information on plan modes and branching
Once an xbranch has been 'seen' - ie all the sub-xbranches have been executed, taskpipe can clear all records of those sub-xbranches in order to keep the database trim. The upside to persisting the xbranch cache is that any (sub) xbranch can be targetted in a rerun so that only that xbranch gets executed without executing the whole plan. The downside is a substantially larger database.
Persisting the xbranch cache may be a good idea during development when errors are expected frequently, but is probably less desirable in production with less frequent errors and a much larger amount of data.
Once an xbranch has been 'seen' without errors, taskpipe can clear all cached results associated with the xtasks that appear on the xbranch, in order to keep the database trim. The upside to persisting the result cache is that if the plan is changed, taskpipe will only execute any new or changed xtasks without needing to execute unchanged ones (since the unchanged xtasks are cached). The downside is considerable database overhead.
Like persisting the xbranch record, persisting the result cache may be a good idea during development when errors and changes are expected frequently, but is probably less desirable in production with a much larger amount of data.
The maximum number of threads to use when running a plan. Taskpipe tries to adhere strictly to the number of threads you specify here - so parent threads are included in the value. You should experiment with your setup to determine the optimum value for your system
What to do if an error is encountered. Options are
stop(ie attempt to stop all threads), and
continue(which will log the error and continue)
Tom Gracey <email@example.com>
Copyright (c) Tom Gracey 2018
TaskPipe is free software, licensed under
The GNU Public License Version 3