Marco Neves
and 1 contributors

NAME

Parallel::Dragons - Daemon are forever... Dragons lay eggs, grow fast and die in flames!

SYNOPSIS

  use parent qw'Parallel::Dragons';

  sub max_childs { 5 } # max 5 workers at a time

  sub wait_before_restart { 10 }

  sub main {
    my $i = 0;
    while ($i++ < 100) {
        print "$$: counted to $i\n";
        sleep 1;
    }

    print "counted to 100, going away\n";
  }

  main::->new()->run();

DESCRIPTION

Parallel::Dragons is a framework to create long lived processes that run the same code all the time, in some cases multiple times in parallel.

It doesn't know, and doesn't care about what the code does. Even when used in queue driven mode, it doesn't care about the queue, leaving the implementation of the queue access to the application itself.

USE

Parallel::Dragon was written to be the parent of you daemon class.

The minimal Daemon you can create implements only the sub main, which does the job you need done. In this case max_childs will default to the number of CPUs in your server and a new child is created every 3 seconds until that number is reached.

The method new doesn't take any parameters, but the class handles @ARGS to accept several command line commands and options.

Most of the time, you want to call $your_class->new()->run() and let Parallel::Dragons take care of everything for you.

Unlike what happens with most libraries, you don't call methods in Parallel::Dragons - Parallel::Dragons calls methods you write for it.

Parallel::Dragons was created with the idea that by defining specific methods in your class you can change the way your Dragon behaves. We call those methods hooks, and that's what the next section is about.

Parallel::Dragons works by having a main process that starts childs to do all the work - the main process only stops when a small set of hooks fails with error.

HOOKS

The hooks as listed in the order they are usually called. Some of them are called in the main process, some of them are called in the child process - this list tells you all you need to know.

max_childs

MAIN: Called during the init phase of the main process - it's assumed to be a constant - see needs_childs for a more dynamic hook. Your dragon will never create more than <max_childs> at the same time.

Params: $self

wait_before_restart

MAIN: called during the init phase of the main process - it's assumed to be a constant and represents the number of seconds that the main process waits between starting two childs and between the end of a child and starting a new one when the max_childs is reached.

See restart_next for a more dynamic way to decide when to start the next child.

Params: $self

restart_after

MAIN: if your Dragon uses a <queue> based model, this is the number of time a child will ask for tasks before ending. This is called during the init phase of the main process and is assumed to be a constant.

Params: $self

max_memory_per_child

MAIN: this is called during the init phase of the main process and is assumed to be a constant. It's used only in Dragons that use the <queue> model - when a child ends executing main for a task, it checks whether it is using more than this amount of memory and ends if it is.

Params: $self

pre_start

MAIN: called before the Dragon is started.

Note: pre_start can prevent the Dragon from starting by calling $self->stop

Params: $self

need_childs

MAIN[in the loop]: it's called every time it's time to start a new child to decide if a child needs to be created.

Params: $self

pre_fork

MAIN[in the loop]: the decision to create a new child was made, this allow you to set any data you need to pass to the child.

Params: $self

post_child_start

MAIN[in the loop]: called in the main process imediately after a new child is created.

Params: $self, $child

restart_next

MAIN[in the loop]: called after a child is started and is expected to return an unix timestamp for when the next child should be started. The child will only be started if there are less than max_childs running.

Params: $self

init_child

CHILD: called when a new child is created.

Params: $self

data_consumer

CHILD: expected to return a Data::Consumer compatible object. If this method is used, the Dragon will run in <DC> mode. In this mode the method main will be called as a (modified) consume call back. See main.

Params: $self

claim_task

CHILD: expected to return a <task> that main will be able to process. If claim_task is used, the Dragon will run in <queue> mode. In this mode main will be called with the task return by claim_tasks.

If claim_task returns false, the child will exit - and a new one will be created by the MAIN process when time for that comes.

Params: $self

Note on data_consumer and claim_task

If neither of those methods exists, main will run once for each child and the child will end after that. We call that <single run> mode.

main

CHILD: this is the body of the child - all the other methods are intented to setup the work to be done in this method.

Depending on which mode the Dragon is running in, the parameters will differ, but a single Dragon will always recieve the same type of parameters - meaning, all the childs of a given Dragon will always run in the same mode, as the mode depends on the existence of the methods data_consumer and claim_task.

Params:

DC mode: $self, $id

The $id is the $id claimed by Data::Consumer - your main method should be able to handle getting the task from that id.

queue mode: $self, $task

the $task returned by claim_task

single run more: $self

task_done

CHILD[queue mode]: called when the child finishes processing a task and the main method ended without errors. if the main method dies, task_failed will be called, instead of task_done.

task_done is only called in queue mode.

Params: $self, $task

task_failed

CHILD[queue mode]: called when the main method dies.

Params: $self, $task

post_child_exit

MAIN[in the loop]: when a child ends, post_child_exit is called. This allows you to clean up any data you may have in the main process specifically for that child.

Params: $self, $child

pre_exit

MAIN: Called just before the process ends - allows you to run any clean up you main want.

Params: $self

ADDITIONAL HOOKS

This two hooks are not related with the workflow of managing child processes. However, this hooks are still running in the loop of managing childs, so they should be as efficient as possible.

monitor

MAIN[in the loop]: monitor is called once a seconds in the main process.

Params: $self

idle

MAIN[in the loop]: idle is called in the main process when no child as ended, monitor didn't run and no child was created in that cycle.

Params: $self

COMMANDS

All the Dragons support several command line commands. You run a command by calling your dragon followed by that command. Example:

<dragon.pl> start

start

Starts the Dragon as a dameon - it detachs from the current terminal, create log files and socket file in /tmp. Before starting a new Dragon it checks if there is a Dragon running and if it is responsive.

A normal thing to do, if you want your Dragons to never stop running is to have a minutely cron starting then - if they are already running nothing will happen.

startfg

Runs the dragon in the foreground, as a single process. Useful to test and debug your process. It still calls all the relevant methods, except for post_child_start and post_child_exit.

stop

Stops a Dragon started with start. This uses the socket file to tell the dragon to stop. The stopping process is graceful, which means that this command tells the MAIN process of the running process that you want it to stop, and it tells all their childs to stop when they are done with their corrent task, which may take some time, depending on how long each of you tasks takes to finish.

ping

Send a ping command to the MAIN process of a running. If there is a Daemonized Dragon running, it will respond with pong.

info

Sends a info request to the MAIN process of a running Dragon and gets some information about it's status - this will probably be extended in the future.

ENVIRONMENT VARIABLES

This variables are useful when running start or startfg. The control the amount of information you will see from the Parallel::Dragons itself.

DRAGONS_DEBUG

Expected a values between 0 and 10. The relevant levels are:

0 - no debug.
1 - some information
3 - debug
10 - trace

DRAGONS_CRONJOB

Expected to be a true/false (1/0) value. It disables information about other or not a Dragon was already or a new one was started.

Useful for cronjobs - may be replaced in the future with a check of a TTY in STDIN/STDOUT

SEE ALSO

There are a lot of other perl packages to manage parallel execution of code - depending on your use case some of them may be better than Parallel::Dragons. This is just another option.

BUGS REPORTS and FEATURE REQUESTS

Please report any bugs , or request features in:

* github: https://github.com/themage/perl-parallel-dragons/ * Magick Source: http://magick-source.net/project/view/10/

AUTHOR

theMage <themage@magick-source.net>

COPYRIGHT AND LICENSE

Copyright (C) 2016 by theMage

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.22.1 or, at your option, any later version of Perl 5 you may have available.

Alternativally, you can also redistribute it and/or modify it under the terms of the GPL 2.0 licence (or any future version of it).