PDL::Parallel::MPI
Routines to allow PDL objects to be moved around on parallel systems using the MPI library.
use PDL; use PDL::Parallel::MPI; mpirun(2); MPI_Init(); $rank = get_rank(); $a=$rank * ones(2); print "my rank is $rank and \$a is $a\n"; $a->move( 1 => 0); print "my rank is $rank and \$a is $a\n"; MPI_Finalize();
Most of the functions from the MPI standard may be used from this module on regular perl data. This is functionallity inherited from the Parallel::MPI module. Read the documentation for Parallel::MPI to see how to use.
One may mix mpi calls on perl built-in-datatypes and mpi calls on piddles.
use PDL; use PDL::Parallel::MPI; mpirun(2); MPI_Init(); $rank = get_rank(); $pi = 3.1; if ($rank == 0) { MPI_Send(\$pi,1,MPI_DOUBLE,1,0,MPI_COMM_WORLD); } else { $message = zeroes(1); $message->receive(0); print "pi is $message\n"; } MPI_Finalize();
Call this before you pass around any piddles or make any mpi calls.
# usage: MPI_Init();
Call this before your mpi program exits or you may get zombies.
# usage: MPI_Finalize();
Returns an integer specifying who this process is. Starts at 0. Optional communicator argument.
# usage get_rank($comm); # comm is optional and defaults to MPI_COMM_WORLD
Returns an integer specifying how many processes there are. Optional communicator argument.
# usage comm_size($comm); # comm is optional and defaults to MPI_COMM_WORLD
Typically one would invoke a mpi program using mpirun, which comes with your mpi compiler. This function simply a dirty hack which makes a script to invoke itself using that program.
mpirun($number_of_processes);
move is a piddle method. It copies a piddle from one processes onto another processes. The first arguement is the rank of the source processes, and the second argument is the rank of the receiving processes. The piddle should be the allocated to be same size and datatype on both machines (this is not checked). The method does nothing if executed on a process which is neither the source or the destination. => may be used in place of "," for readability.
move
# usage $piddle->move($source_processor , $dest_processor);
# example $a = $rank * ones(4); $a->move( 0 => 1);
You can use send and receive to move piddles around by yourself, although I really recommend using move instead.
$piddle->send($dest,$tag,$comm); # $tag and $comm are optional $piddle->receive($source,$tag,$comm); # dido
Piddle method which copies the value at the root process to all of the other processes in the communicator. The root defaults to 0 if not specified and the communicator to MPI_COMM_WORLD. Piddles should be pre-allocated to be the same size and datatype on all processes.
# usage $piddle->broadcast($root,$comm); # $root and $comm optional
# example $a=$rank * ones(4); $a->broadcast(3);
These piddle methods initiate communication and return before that communication is completed. They return a request object which can be checked for completion or waited on. Data at source and dest should be pre-allocated to have the same size and datatype.
# $tag and $comm are optional arguments. $request = $piddle->send_nonblocking($dest_proc,$tag,$comm); $request = $piddle->receive_nonblocking($dest_proc,$tag,$comm); ... $request->wait(); # blocks until the communication is completed. or $request->test(); # returns true if the communication is completed. # $request is deallocated after a wait or test returns true.
# this example is similar to how mpi_rotate is implemented. $r_send = $source->send_nonblocking(($rank+1) % $population); $r_receive = $dest->receive_nonblocking(($rank-1) % $population); $r_receive->wait(); $r_send->wait();
get_status returns a hashref which contains the status of the last receive. The fields are count, source, tag, and error. print_status simply prints out the status nicely. Note that if there is an error in a receive and exception will be thrown.
count
source
tag
error
print_status(); print ${get_status()}{count};
mpi_rotate is a piddle method which should be executed at the same time on all processors. For each process, it moves the entire piddle to the next process. This movement is (inefficently) done in place by default, or you can specify a destination.
mpi_rotate
$piddle->mpi_rotate( dest => $dest_piddle, # optional offset => $offset, # optional, defaults to +1 );
Takes a piddle and splits its data onto all of the processors. This would take an n-dimensional piddle on the root and turn it into an n-1 dimensional piddle on all processors. It may be called as a piddle method, which is equivilant to simply specifing the 'source' argument. On the root, one must specify the source. On all other procs, one may also pass a 'source' argument to allow scatter to grok the size of the destination piddle to allocate. Alternatively on the non-root procs one may specify the dest piddle explicitly, or simply specify the dimensions of the destionation piddle.
# usage (all arguments are optional, but see above). # may be used as a piddle method, which simply sets the # source argument. $dest_piddle = scatter( source => $src_piddle, dest => $dest_piddle, dims => $array_ref, root => $root_proc, # root defaults to 0. comm => $comm, # defaults to MPI_COMM_WORLD );
# with 4 processes $a = sequence(4,4); $b = $a->scatter;
gather is the opposite of scatter. Using it as a piddle method simply specifies the source. If called on an n dimensional piddle on all procs, the root will contain an n+1 dimensional piddle on completion.
gather
scatter
memory => +------------+ +-------------+ ^ |a0 | -----> | a0 a1 a2 a3 | procs | |a1 | gather | | | |a2 | | | |a3 | <----- | | +------------+ scatter +-------------+
# usage gather( source => $src_piddle, dest => $dest_piddle, # only used at root, extrapolated from source if not specified. root => $root_proc, # defaults to 0 comm => $comm, # defaults to MPI_COMM_WORLD );
# example. assume nprocs == 4. $a = ones(4); $b = $a->gather; # $b->dims now is (4,4) on proc 0.
allgather does the same thing as gather except that the result is placed on all processors rather than just the root.
allgather
memory => +------------+ +-------------+ ^ |a0 | | a0 a1 a2 a3 | procs | |a1 | -----> | a0 a1 a2 a3 | | |a2 | allgather | a0 a1 a2 a3 | |a3 | | a0 a1 a2 a3 | +------------+ +-------------+
memory => +-------------+ +-------------+ ^ | a0 a1 a2 a3 | | a0 b0 c0 d0 | procs | | b0 b1 b2 b3 | -----> | a1 b1 c1 d1 | | | c0 c1 c2 c3 | alltoall | a2 b2 c2 d2 | | d0 d1 d2 d3 | | a3 b3 c3 d3 | +-------------+ +-------------+
# usage # calling as piddle method simply sets the source argument. $dest_piddle = alltoall( source => $src_piddle, dest => $dest_piddle, # created for you if not passed. comm => $comm, # defaults to MPI_COMM_WORLD. );
# example: assume comm_size is 4. $a = $rank * sequence(4); $b = $a->alltoall;
+-------------+ +----------------------------------+ | a0 a1 a2 a3 | reduce | a0+b0+c0+d0 , a1+b1+c1+d1, .... | | b0 b1 b2 b3 | -----> | | | c0 c1 c2 c3 | | | | d0 d1 d2 d3 | | | +-------------+ +----------------------------------+
Allowed operations are: + * max min & | ^ and or xor.
+ * max min & | ^ and or xor
# usage (also as piddle method; source is set) $dest_piddle = reduce( source => $src_piddle, dest => $dest_piddle, # signifigant only at root & created for you if not specified root => $root, # defaults to 0 op => $op, # defaults to '+' comm => $comm, # defaults to MPI_COMM_WORLD );
# example $a=$rank * (sequence(4)+1); $b=$a->reduce;
Just like reduce except that the result is put on all the processes.
# usage (also as piddle method; source is set) $dest_piddle = allreduce( source => $src_piddle, dest => $dest_piddle, # created for you if not specified root => $root, # defaults to 0 op => $op, # defaults to '+' comm => $comm, # defaults to MPI_COMM_WORLD );
# example $a=$rank * (sequence(4)+1); $b=$a->allreduce;
+-------------+ +----------------------------------+ | a0 a1 a2 a3 | scan | a0 , a1 , a2 , a3 | | b0 b1 b2 b3 | -----> | a0+b0 , a1+b1 , a2+b2, a3+b3 | | c0 c1 c2 c3 | | | | d0 d1 d2 d3 | | ... | +-------------+ +----------------------------------+
# usage (also as piddle method; source is set) $dest_piddle = scan( source => $src_piddle, dest => $dest_piddle, # created for you if not specified root => $root, # defaults to 0 op => $op, # defaults to '+' comm => $comm, # defaults to MPI_COMM_WORLD );
# example $a=$rank * (sequence(4)+1); $b=$a->scan;
Does a reduce followed by a scatter. A regular scatter distributes the data evenly over all processes, but with reduce_and_scatter you get to specify the distribution (if you want; defaults to uniform).
# usage (also as piddle method; source is set) $dest_piddle = reduce_and_scatter( source => $src_piddle, dest => $dest_piddle, # created for you if not specified. recv_count => $recv_count_piddle # 1D int piddle. put $r[$i] elements on proc $i. op => $op, # defaults to '+' comm => $comm, # defaults to MPI_COMM_WORLD );
# example, taken from t/20_reduce_and_scatter.t mpirun(4); MPI_Init(); $rank = get_rank(); $a=$rank * (sequence(4)+1); $b=$a->reduce_and_scatter; print "rank = $rank, b=$b\n" ;
This module is still under development. Signifigant changes are expected. As things are *expected* to break somehow or another, there is no waranty, and the author does not take responsibility for the damage/distruction of your data/computer/sanity.
Currently there is no support for any sort of indexing/dataflow/child piddles. Ideally one would like to say:
$piddle->diagonal(0,1)->move(0 => 1);
But currently one must say:
$tmp = $piddle->diagonal(0,1)->copy; $tmp->move( 0 => 1); $piddle->diagonal(0,1) .= $tmp;
I believe the former behavior to be possible to implement. I plan to do so once I reach a sufficent degree of enlightenment.
One might wish to use their own personal massively parallel supercomputer interactively with the pdl shell (perldl). This would require master/slave interactions and a distributed data model. Such a project won't be started until after I finish PDL::Parallel::OpenMP.
Darin McGill darin@ocf.berkeley.edu
If you find this module useful, please let me know. I very much appreciate bug reports, suggestions and other feedback.
This module is an extension of Parallel::MPI written by Josh Wilmes and Chris Stevens. Signifigant portions of code has been copied from Parallel::MPI verbatim. Used with permission. Josh and Chris did most of the work to make perl's built in datatypes (scalars, arrays) work with MPI. I rely heavily on their work for MPI intitialization and error handling.
The diagrams in this document were inspired by and are similar to diagrams found in MPI-The Complete Reference.
Sections of code from the main PDL distribution (such as header files, and code from PDL::CallExt.xs) were used extensively in development. Many thanks to the PDL developers for their help, and of course, for creating the PDL system in the first place.
The PDL::Parallel homepage: http://www.ocf.berkeley.edu/~darin/projects/superperl
The PDL home page: http://pdl.perl.org
The Perl module Parallel::MPI.
The Message Passing Interface: http://www.mpi-forum.org/
PDL::Parallel::OpenMP (under development).
This module is free software. It may be modified and/or redistributed under the same terms as perl itself.
To install PDL::Parallel::MPI, copy and paste the appropriate command in to your terminal.
cpanm
cpanm PDL::Parallel::MPI
CPAN shell
perl -MCPAN -e shell install PDL::Parallel::MPI
For more information on module installation, please visit the detailed CPAN module installation guide.