Batch Queueing Systems

This page describes several batch queueing systems available for free, or nearly so, on the internet. In addition README documents and man pages are also served so that folks can take a look at how the individual systems operate. In the future I'm likely to add commentary on what a batch systems can and cannot do, and what administrators can expect out of the different sorts of packages.

This page does not intend to review messages passing/virtual machine technology at this point. When possible, I will try to include links to information and details.

Here is a quick index:

Basic Queueing Systems:

NQS and look alikes

NQS by Sterling Software.

Most versions of NQS runs on atleast:

NQS was architected and written with the following design goals in mind:

Full text of the READMEFIRST, README, USRQUICKREF, and qsub man page are also available.

NQS (modified by Monsanto). Check your archie server for a location near you. Academic Computing Services, University of Sheffield is one possibility.

This version of NQS runs on atleast:

This version of NQS is based on Version 1 of NQS from Cosmic. It includes enhancements from Cosmic Version 2 and from a number of other sources, especially Christian Boissat and Chuck Keagle of Boeing. This version is copyrighted by John Roman and is covered by the GNU General Public License. See the file info/COPYING for more information.

Monsanto NQS is similar to Cosmic Version 2, except for the following:

As a result, this version is mostly compatable with Cosmic V2, but not exactly. The functions and commands are somewhat modified.

I am making this software available in the hope that others will find it useful and that they will let me know of ways to improve it. This software is undergoing continual development, so if you let me know of ways to fix it I will be able to make future versions better. Please let me know if you try to build / try to use this software and the experiences you may have. I know the documentation is particularly feeble. If you have material you would like to contribute, I would be glad to accept it.

The main enhancement in this release is the ability to stage a release of NQS. A common problem is that long running NQS jobs prevent upgrading to a newer version easily. This feature allows one to place a new release of NQS in a staging area. NQS checks when a job completes to determine if there is no other activity, and if not, will shut itself down, install the new version and restart itself. This is useful when it is not urgent to install a new release of NQS, and one does not want to disable the queues.

Full text of the README, and qsub man page are also available.

NQS and NQS++ from CERN

This version of NQS runs on atleast:

NQS++is a CERN extension which allows users to submit batch jobs from remote non-NQS clients provided you have TCP/IP ( has been implemented under VM/CMS and VMS ). For information on NQS and NQS++ at CERN, contact Christian Boissat ( boissat@vxcern.cern.ch ).

Full text of the READMEFIRST, README, USRQUICKREF, and qsub man page are also available.

DQS from Supercomputer Computations Research Institute, Florida State University (current version 3.2.3)

The primary development platforms have been AIX and Linux though it has seen limited testing on the following:

See the DQS web page at SCRI

Full text of the readme documents are not yet on line, but the qsub man page is available.

MDQS from Ballistics Research Laboratory

Mdqs runs on atleast:

The Multiple Device Queueing System (MDQS) is designed to provide UNIX with a full function, modular, and consistent queueing system. The MDQS system has been designed with portability, expan dability, robustness, and data integrity as key goals. MDQS is designed around a central queue which is managed by a single privileged daemon. Requests, delayed or immediate, are queued by non-privileged programs. Once queued, requests can be listed, modified or deleted. When the requested device or job stream becomes available, the daemon executes an appropriate server process to handle the request. Once activated, the request can still be canceled or restarted if needed. MDQS can serve as a delayed- execution/batch subsystem and replaces internally the functions of lpr(I) and at(I) as a minimum. MDQS provides the system manager with a number of tools for managing the queueing system. Queues can be created, modified, or deleted without the loss of requests. MDQS recognizes and supports both multiple devices per queue and multiple queues per device by mapping input for a logical device to an appropriate physical output device. Anticipating the inevitable, MDQS also provides for crash recovery.

The MDQS system has been developed at the U.S. Army, Ballistics Research Laboratory by Doug Kingston and Michael Muuss to support the work of the laboratory and is available to other UNIX sites upon request. The MDQS system is designed to be compilable on a standard V7 UNIX system.

Full text of the Tour, USENIX paper, and batch man page are also available.

PBS: The Portable Batch System

The Portable Batch System (PBS) is a new suite of software tools that fulfills the growing need for a flexible, networked, multi-platform batch processing environment. The typical batch queuing system schedules jobs for execution using a set of queue controls. Within each queue, jobs are then selected in first-in, first-out (FIFO) order. This severely limits the set of scheduling policies available to a site adminstrator. The purpose of PBS is to provide additional controls over initiating or scheduling execution of batch jobs, and to allow routing of those jobs between different hosts. PBS's independent scheduling module allows the system administrator to define what types of resources, and how much of each resource, can be used by each job. The scheduling module has full knowledge of the available queued jobs, running jobs, and system resource usage. Using one of several procedural languages, the scheduling policies can easily be modified to suit the computing requirements and goals of any site. PBS also provides a mechanism which allows users to specify unique resources required for a job to complete.

Current Platforms:

Further documentation is available. There is a form to complete to become a beta test site for the PBS software.

Batch Queueing Systems

Batch (circa 1987) found at archive site gatekeeper.dec.com; by Allan Bricker

Gatekeeper Batch runs on:

Batch queues the command to be executed in order to reduce the system load average. Jobs are executed in first come first serve order. Batched jobs are given priority over normal background jobs, so it is to your advantage to use it. Note that any I/O redirection in the batch command line will only redirect output from the batch command.

Full text of the batch man page are also available.

Batch from University of Toronto

UT Batch runs on atleast:

The batch daemon can run several independent queues of batch jobs. A ``batch job'' is a sequence of shell commands. Users submit jobs with the "batch" command.

Each batch queue has a set of attributes (defined in a profile file) to control job processing, such as niceness, max concurrent jobs, etc. Jobs can be stopped when the load average is too high, and restarted when it falls. Job execution can be controlled by time of day, or by the exit status of a separate program. Jobs can also be restarted after a crash. See the man pages for details.

Originally by Alex White, modified by lots of people at the University of Waterloo. Thanks to Mike Peterson for the Apollo support. Thanks to Gordon Strachan for the HP-UX support. This code is freely redistributable.

Bug reports to: Ken Lalonde Computer Science Dept. University of Toronto ken@cs.toronto.edu

Full text of the README, and batch man page are also available.

Batch from U California San Francisco

UCSF Batch runs on atleast:

This software is designed to manage multiple batch job queues under the control of a daemon process. The daemon process controls the batch jobs through the use of the BSD job control signals, while client programs, batch, baq, and barm provide for submission, examination and removal of jobs, respectively.

The capabilities include:

This code was started when I was at Dept. Pharmaceutical Chemistry, University of California, San Francisco, CA.

But I am now at ZymoGenetics Inc., 1201 Eastlake Ave. E, Seattle, WA.
+1 206 442 6751
srp@zgi.com

Full text of the users guide, reference manaul, and batch man page are also available.

Qbatch from Alan Saunders

QBATCH runs on atleast:

QBATCH is a versatile queued batch processing system for *NIX.

Each queue consists of a file containing information about the queue itself, and about all jobs currently present in the queue. When the program 'qp' is run for a given queue, it will fork a child process for each job in the queue in turn, and wait for it to complete. If there are no jobs present in the queue, qp will wait for a signal from one of the support programs, which will 'tell' it that another job has joined the queue, or that it should terminate.

Features:

Facilities:

Of course, porting and installing QBATCH isn't the end. Queue discipline has to be imposed in order for it to be of any benefit. Scripts and commands triggering batch jobs will have to be changed to use QBATCH, and jobs normally run interactively which impose heavy system loads, will have to be reconfigured. (consider for example renaming 'make' as 'MAKE' and writing a 'make' script which submits 'MAKE' to a queue). Of course there will always be the awkward user who decides to submit a large job into a high priority queue. Analysing the queue monitors should show this up, they contain a certain amount of profile timing for jobs run in the queue, and the uid and gid of the submittor.

Full text of the js man page is also available.

Systems That Support Process Migration:

Condorfrom U Wisconson

Condor releases are machine specific. The most recent version is 5.62. See the ftp site for platforms.

Condor is a facility for executing UNIX jobs on a pool of cooperating workstations. Jobs are queued and executed remotely on workstations at times when those workstations would otherwise be idle. A transparent checkpointing mechanism is provided, and jobs migrate from workstation to workstation without user intervention. When the jobs complete, users are notified by mail.

No source code changes are required for use of Condor, but executables must be specially linked. Linking instructions vary slightly from language to language - see condor(1).

Due to the limitations of the remote execution and checkpointing mechanisms, there are several restrictions on the type of program which can be executed by the condor facility. Most importantly only single process jobs are supported, (i.e. the fork() and exec() system calls are not supported). Secondly, signal and IPC calls are not implemented, (e.g. signal(), kill(), and socket() calls will fail). Finally the checkpointing mechanism will only work correctly if all file operations are idempotent - read-only and write-only file access works correctly, but programs which both read and write the same file may become confused after a checkpoint.

There are also some practical limitations regarding use of disk space for execution of condor jobs. During the time when a job is waiting for a workstation to become available, it is stored as a "checkpoint file" on the machine from which the job was submitted. The file contains all the text, data, and stack space for the process, along with some additional control information. This means jobs which use a very large virtual address space will generate very large checkpoint files. Some advantage can be gained by submitting multiple jobs which share the same executable as a single unit or "job cluster". This is because all jobs in a cluster share a single copy of the checkpoint file until they begin execution. See condor(1) for details on how to submit multiple jobs as a cluster. Also the workstations on which the jobs will actually execute often have very little free disk. Thus it is not always possible to transfer a condor job to a machine, even though that machine is idle. Since large virtual memory jobs must wait for a machine that is both idle, and has a sufficient amount of free disk space, such jobs may suffer long turnaround times.

Full text of the Introduction, and submit man page are also available.

Commercial Systems:

For a review of some commercial systems, I have also included access to the paper: "A Comparason of Queueing, Cluster and Distributed Computing Systems."
(N.B. GhostScript has problems with this document: try xpsview instead.)
This paper compares the following systems.:

(Condor, and DQS are not commercial)

Thanks to Michael Nelson for his permission to serve the paper from this WWW page.

Other commercial systems include:

Key: messages passing support is indicated by a letter/number in [] brackets.

Please let me know of broken links, bad references, other queueing systems, better information, etc., via e-mail.

Copyright © 1993-8 by Scott Presnell. All Rights Reserved.


Scott Presnell <srp@zgi.com>
Last modified: Tue Aug 25 14:34:05 1998
Scott Presnell
Last modified: Wednesday, August 28, 2002 13:37 PDT