PBS
Portable Batch System for Job Submission

PBS is a directory of examples which illustrate the use of the Portable Batch System for job submission, as implemented at the Pittsburgh Supercomputing Center.

A user typically logs into a special login node of the cluster, which is intended only for editing, file management, job submission, and other small interactive tasks.

The user wishes to run a parallel program on several processors of the cluster. To do so, the user must create an executable version of the program, write a suitable PBS batch job script describing the job limits, and listing the commands to be executed, and then submit the script to for processing using the qsub command.

The job script can be thought of as consisting of two parts:

• Commands to PBS, which must begin with the string #PBS, and which describe properties of the job;
• Ordinary operating system commands, which could essentially be typed interactively, which cause the program to execute, but which may also create or delete files, print the date, change directories, and so on.

The user has several separate issues when preparing a first job script:

PBS issues:
• understanding the local "dialect" of PBS, and the local job policies, such as the names of queues, the limits on the number of processors that can be requested, and so on. The exact form of the PBS commands is usually obscure, but you only have to figure it out once.

File and Directory issues:
• The first thing to realize is that the PBS script will essentially "log in" to your home directory, and thus, will be able to access files in that home directory easily. The job will also have executed your noninteractive customization commands (which might be in .bashrc, for instance), so that you may be able to take advantange of some of the shortcuts you have defined there. If that's not where you want to be, you can issue cd commands to move to another directory. But a very convenient command is

              cd $PBS_O_WORKDIR
            

  which changes to the exact directory from which you submitted the script. This is a very natural thing to do.

Program issues:
• For OpenMP, this primarily involves finding out the name of the compiler, and the switches required to compile with OpenMP;
• for MPI, it is often necessary to invoke a locally defined setup script, to know the names of the compiler scripts, and to know the syntax for the command to execute a program under MPI.
• for (nonparallel) MATLAB, you need to understand how to use MATLAB in batch mode; that means suppressing the exuberant display of the MATLAB logo and the desktop; it means saving graphics into files using MATLAB's print() command.
• for parallel MATLAB, you need to know how many parallel labs you can request, and any special commands that are necessary to get parallel MATLAB running. At the FSU HPC facility, for instance, there are a limited number of parallel MATLAB licenses, and parallel MATLAB may only be run by invoking a special script.

To log into the system, use ssh and the address of the component to which you have been assigned. For instance, you might log into the PSC Greenfield system by inserting your username in the following command:

        ssh USERNAME@greenfield.psc.xsede.org
      

To transfer files, you need the sftp command. I do this in a second window, so that I have interactive access in my ssh window, while file transfers occur in the sftp window. The command that makes the connection for me is:

        sftp USERNAME@greenfield.psc.xsede.org
      

        put fred.txt
      

        get jeff.txt
      

The main reason for using a cluster is to be able to compute in batch mode - one or many jobs, submitted to a queue, to run "eventually". You can log out after you submit jobs, and log in later at your convenience to collect the output from completed jobs. Parallel programs can be run on multiple processors this way. Matlab programs, whether parallel or sequential, can also be submitted to the batch queue.

The commands that make a job run in the batch queue form a PBS script. The first part of the script contains commands to the job scheduler. These commands begin with the string "#PBS" and specify the maximum time limit, the number of processes, the particular queue you will use, and so on. Most batch jobs should go into the "batch" queue.

The PBS command "#PBS -l nodes=1:ppn=15" specifies a limit of just 1 node, and all 15 cores on that node. In fact, on the PSC greenfield machine, the number of cores should always be 15. In fact, your job won't run with any other choice.

After the PBS commands come a sequence of commands that you might imagine typing in interactively; that is, these might be the normal sequence of UNIX commands you would issue to run a particular job.

Briefly, if you have a PBS script file called fred.sh you can submit it to the queue by a command like

        qsub fred.sh
      

        qstat
      

        qstat -uUSERNAME
      

Licensing:

The computer code and data files made available on this web page are distributed under the GNU LGPL license.

Related Data and Programs:

MPI, C programs which illustrate the use of the MPI application program interface for carrying out parallel computations in a distributed memory environment.

OPENMP, FORTRAN90 programs which illustrate the use of the OpenMP application program interface for carrying out parallel computations in a shared memory environment.

Source Code:

ENVIRON is a batch job script that simply queries the values of certain environment variables, in particular PBS_O_WORKDIR, which can be useful when trying to set up a program to run under the batch system.

• environ.sh, the job script.
• environ_output.txt, the output file.

FENICS_BVP_01 is a batch job script that invokes the FENICS program with the input file bvp_01.py, and an auxilliary program timestamp.py

• fenics_bvp_01.sh, the job script.
• bvp_01.py, the input file to Python FENICS.
• timestamp.py, an auxilliary file.
• fenics_bvp_01_output.txt, the output file.
• bvp_01_mesh.xml, an XML file, output by the script, containing information about the vertices (nodes) and cells (elements) that define the mesh.
• bvp_01_u.xml, an XML file, output by the script, containing "function data" (finite element coefficients associated with a mesh), including the index of the data item, the value of the data, the index of the cell (element), and the index of the cell's degree of freedom (internal node).

FENICS_POISSON is a batch job script that invokes the FENICS program with the input file poisson.py, and an auxilliary program timestamp.py.

• fenics_poisson.sh, the job script.
• poisson.py, the input file to Python.
• timestamp.py, an auxilliary file.
• fenics_poisson_output.txt, the output file.

FREEFEM_MPI is a batch job script that runs the parallel version of FreeFem++ with the input file schwarz_mpi.edp.

• freefem_mpi.sh, the job script.
• schwarz_mpi.edp, the input file.
• schwarz_mpi_mesh.png, a PNG file of the mesh.
• schwarz_mpi_u.png, a PNG file of the solution.
• freefem_mpi_output.txt, the output file.

FREEFEM_SEQUENTIAL is a batch job script that runs the sequential version of FreeFem++ with the input file membrane.edp.

• freefem_sequential.sh, the job script.
• membrane.edp, the input file.
• membrane_mesh.png, a PNG file of the mesh.
• membrane_u.png, a PNG file of the solution.
• freefem_sequential_output.txt, the output file.

HELLO is a batch job script that compiles and runs a "hello" program.

• hello.sh, the job script.
• hello_output.txt, the output file.

HELLO_MPI illustrates the compilation and execution of a program that includes MPI directives.

• hello_mpi.sh, the job script.
• hello_mpi_output.txt, the output file.

HELLO_OPENMP illustrates the compilation and execution of a program that includes OpenMP directives.

• hello_openmp.sh, the job script.
• hello_openmp_output.txt, the output file.

MATLAB_PARALLEL runs MATLAB with a parfor command for parallel execution.

• matlab_parallel.sh, the job script.
• matlab_parallel.m, a MATLAB M file to be executed.
• matlab_parallel_log.txt, the log file (MATLAB output).
• matlab_parallel_output.txt, the output file.

POWER_TABLE shows how a MATLAB program can be run through the batch system. We prepare a file of input commands, and invoke MATLAB on the commandline.

• power_table.sh, the job script.
• power_table.m, a MATLAB M file to be executed.
• power_table_log.txt, the log file (MATLAB output).
• power_table_output.txt, the output file.

You can go up one level to the EXAMPLES source code page.