SGE (Sun Grid Engine) – Cluster computation

Information

http://wikis.sun.com/display/GridEngine/Managing+Parallel+Environments

Python interfaces to the SGE

http://jeetworks.org/node/29

SGE and Openmpi

sh configure –prefix=/homeonthecluster –with-sge

http://www.open-mpi.org/faq/?category=running#run-n1ge-or-sge

Some hints how to setup:

http://blog.nguyenvq.com/2010/01/20/scheduled-parallel-computing-with-r-r-rmpi-openmpi-sun-grid-engine-sge/

http://idolinux.blogspot.com/2010/04/quick-install-of-open-mpi-with-grid.html

Number of Nodes

We can specify a variable number of nodes using #$ -pe openmpi min-max

Then the actual number of nodes assigned is given in $NSLOTS, see http://www.rocksclusters.org/roll-documentation/sge/4.2.1/submitting-batch-jobs.html

Queue overview

qstat -g c

Parallel environments

To list all installed parallel environments use qconf -spl

More infos: http://www.nottingham.ac.uk/hpc/html/docs/draftuserguide/user/node54.html

Installing MPI on the cluster

I downloaded the open-mpi implementation from http://www.open-mpi.org/ and installed it locally into my home folder using the –prefix option. Worked perfect. Then I installed http://mpi4py.scipy.org/ and configured it using python setup.py build and installed it using  python setup.py install –prefix=/myhome

Errors

Error: system limit exceeded on number of network connections that can be open
http://download.oracle.com/docs/cd/E19708-01/821-1319-10/appa_trouble.html

Sending Jobs to the Grid: Basic example using MPI

I  use this script to send jobs:

#!/bin/bash
#$ -S /bin/bash
#$ -q opt.q
#$ -N test
#$ -pe openmpi 10-10
#$ -cwd
#$ -e error.log
#$ -o stdout.log
PATH=$PATH:/home/theo/username/bin/
export PATH

PYTHONPATH=$PYTHONPATH:/home/theo/username/lib64/python2.6/site-packages/
export PYTHONPATH

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/theo/username/lib/
export LD_LIBRARY
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/theo/username/lib64/
export LD_LIBRARY

mpirun –prefix /home/theo/username -v -np 10 python test.py

The line #$ -pe openmpi 10-10  tells the grid that I want min 10 and max 10 processors. The line
mpirun –prefix /home/theo/username -v -np 10 python test.py runs python with the script test.py on 10 cores via mpi. The variables above are used to set some additional paths in my home on the cluster, where I installed openmpi and mpi4py and various python-related things. #$ -N test assigns a name to the job, #$ -e error.log and #$ -o stdout.log specify the path where to save error and stdout logs.

For a first test I used

#!/bin/env python
“””
Parallel Hello World !!!
“””
from mpi4py import MPI
import sys
hwmess = “Hello, World!! I am process %d of %d on %s.\n”
myrank = MPI.COMM_WORLD.Get_rank()
nprocs = MPI.COMM_WORLD.Get_size()
procnm = MPI.Get_processor_name()
sys.stdout.write(hwmess % (myrank, nprocs, procnm))

from http://www.siafoo.net/snippet/64 and submitted it via qsub script.sh. The result in stdout.log was

-catch_rsh /usr/local/SGE6/opti/spool/lfc142/active_jobs/5263627.1/pe_hostfile
lfc142
lfc128
lfc103
lfc145
lfc140
lfc154
lfc141
lfc134
lfc151
lfc139
====
/tmp/5263627.1.opt.q/machines
/tmp/5263627.1.opt.q:/usr/local/bin:/bin:/usr/bin
====
Hello, World!! I am process 0 of 10 on lfc142.
Hello, World!! I am process 1 of 10 on lfc128.
Hello, World!! I am process 6 of 10 on lfc141.
Hello, World!! I am process 9 of 10 on lfc139.
Hello, World!! I am process 8 of 10 on lfc151.
Hello, World!! I am process 2 of 10 on lfc103.
Hello, World!! I am process 5 of 10 on lfc154.
Hello, World!! I am process 3 of 10 on lfc145.
Hello, World!! I am process 4 of 10 on lfc140.
Hello, World!! I am process 7 of 10 on lfc134.

A more advanced example which sends and receives messages

#!/bin/env python
from mpi4py import MPI
import sys

def    server():
# Server node
clientlist    = range(1,comm.Get_size())
while True:
# Wait for a client
status    = MPI.Status()
data    = comm.recv(source=MPI.ANY_SOURCE, tag=MPI.ANY_TAG, status=status)
print    “server”,status.Get_source(), status.Get_tag()
sys.stdout.flush()
clientlist.remove(status.Get_source())
if len(clientlist) == 0:
break

def client():
# Client node
myrank    = comm.Get_rank()
data    = [myrank]
comm.send(data, dest=0, tag=20)

hwmess = “Hello, World!! I am process %d of %d on %s.\n”
myrank = MPI.COMM_WORLD.Get_rank()
nprocs = MPI.COMM_WORLD.Get_size()
procnm = MPI.Get_processor_name()
sys.stdout.write(hwmess % (myrank, nprocs, procnm))

# Setting up a client-server based job sheduler
comm        = MPI.COMM_WORLD    # Collection of all processes
# Retrieve information about this node
myrank        = comm.Get_rank()
nodecount    = comm.Get_size()

if nodecount == 1:
print “Only one node, abort”
sys.exit()

# Node 0 becomes the master
if myrank == 0:
server()
else:
client()

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: