MPSH, MPSH_REGION, MPSH_SINGLE, MPSH_OUTPUT, MPSH_CLEAN
message passing shell toolbox
MPSH_REGION << 'MPSH_END'
echo part 1
MPSH_SECTION
echo another part
MPSH_SECTION
echo number $MPSH_SECTION
MPSH_END
...
MPSH_SINGLE echo "oups... forgot this one !"
...
MPSH
MPSH_OUTPUT
The definition of such sections is done through a here document given to MPSH_REGION or through a MPSH_SINGLE instruction.
The parallel execution itself is achived by calling MPSH.
Ordered outputs could be displayed through the MPSH_OUTPUT command.
Before calling MPSH, any number of MPSH_SINGLE or MPSH_REGION could occured, using the add mode of the MPSH_REGION command :
MPSH_REGION mode:add << MPSH_END
echo push this part to any existing job sequence if any
MPSH_END
The MPSH_SINGLE is always in the add mode.
If some chunks of shell absolutly need to run together (but this information is not available when writing the script for example in automatic shell generation), then one could used the tagging mechanism to achieve such behavior:
MPSH_REGION << MPSH_END
echo first part
MPSH_SECTION tag:coolpart
echo here is the cool part
MPSH_SECTION
echo something else to do ?
MPSH_SECTION tag:coolpart
echo pleeeease, I wanna be cool !
MPSH_END
In that case, only 3 jobs will be created (and not 4).
Options are cumulative and one could write :
MPSH_REGION mode:add tag:fun << MPSH_END ... MPSH_ENDFour environment variables are exported in each chunk of shell : MPSH_SECTION which is the number of the current job, MPSH_JOBS which is the total number of jobs to proceed, MPSH_NPES which is the total number of processing elements, MPSH_PEID which is the current processing element id (machine dependent).
Note that if MPSH_REGION command is used whith a single sequence of code ( no MPSH_SECTION inside the here document ), and all mode is activated, then this sequence of shell is duplicated for each processor, unless the add mode is specified :
MPSH_REGION mode:all << 'MPSH_END'
echo each proc should execute my shell task
my_task.sh $MPSH_SECTION
MPSH_END
Another feature of the MPSH_REGION or MPSH_SECTION
command is the loop mechanism :
MPSH_REGION loop:001-008-2,foo,bar << 'MPSH_END'
echo this is loop index $MPSH_INDEX
MPSH_END
... will produce 6 iterations, with associated indexes ``001'', ``003'',
``005'', ``007'', ``foo'' and ``bar''.
MPSH command are self-protected, ie : MPSH_REGION is ignored inside an other MPSH_REGION ( as well as matching MPSH_SECTIONs ).
The estimate overhead of using mpsh is 2 seconds...
/usr/local/mpsh/examples
Some examples for each functionality of the toolbox.
many thanks to :
pascal.lamboley@meteo.fr ( DP/PREVI/COMPAS )
ryad.elkhatib@meteo.fr( CNRM/GMAP/ALGO ).