\begin{ManPage}{\label{man-condor-dbq.pl}\Condor{dbq.pl}}{1}
{Provides a relational database management system interface to Condor.
Condor submit file contents are inserted into a table, \Condor{dbq.pl}
submits the job to Condor, and updates a table with the status of the
work to allow the monitoring of its progress.}
\Synopsis

\SynProg{\Condor{dbq.pl}}
\oOptArg{{-}{-}dbinfofile}{dbinfofile}
\oOptArg{{-}{-}dbinfodir}{dbinfodir}
\oOptArg{{-}{-}logdir}{logfiledir}
\oOptArg{{-}{-}sleepamount}{seconds}
\oOptArg{{-}{-}maxwork}{numofwork}
\oOptArg{{-}{-}maxjobs}{numofjobs}
\oOptArg{{-}{-}grabamount}{numjobs}

\SynProg{\Condor{dbq.pl}}
\oOpt{{-}{-}initdb}
\oOpt{{-}{-}createlang}
\oOpt{{-}{-}revokepublic}
\oOptArg{{-}{-}createsubmituser}{[dbinfofile]}
\oOptArg{{-}{-}createworkuser}{[dbinfofile]}
\oOptArg{{-}{-}submit}{submitfile}
\oOpt{{-}{-}noexecute}

\SynProg{\Condor{dbq.pl}}
\oOpt{{-}{-}help}

\SynProg{\Condor{dbq.pl}}
\oOpt{{-}{-}version}


\Description
\Condor{dbq.pl} provides a relational database interface to submit Condor
jobs, and monitor their progress as they are executed.
The user submits work to the system by inserting a row into a database table.
The user can query the \Condor{dbq.pl} tables to monitor the progress of their
work.

\Condor{dbq.pl} is careful to submit each piece of work into Condor exactly
once.
\Condor{dbq.pl} can be stopped at any time.  Upon restart it will recover its
state and resume operation.


\Condor{dbq.pl} has two main modes of operation.  The first, its normal
mode, performs the role of taking jobs from the database, submitting them
to Condor and updating the results of their execution.
The second mode performs administrative tasks on the database including
creating the necessary tables and accounts to use \Condor{dbq.pl}.

In the normal mode of operation, \Condor{dbq.pl} alternates between polling
the database for new work to submit to Condor, and monitoring the submitted
work's user logs for new events.
The database is updated to reflect the current state of the system.
There are several options described in the Options Section that can be used to
throttle \Condor{dbq.pl}

The submission data for each work is in the same format allowed by
\Condor{submit}.
The only limitation is that the user can not specify a user log using the
\emph{log} attribute as \Condor{dbq.pl} will override this attribute with
its own value.
\Condor{dbq.pl} uses \Condor{submit} to perform the submission into Condor.
The only changes made to the submission are the user log is overridden, and
attributes with a prefix of \emph{CDBQ\_} are added.
For correct operation, work submitted to this \Condor{schedd} outside of
\Condor{dbq.pl} must not add attributes or modify atttibutes with a prefix of
\emph{CDBQ\_}.

\Condor{dbq.pl} operates by using two tables.
The \emph{work} table represents each piece of work.
The \emph{jobs} table represents the individual jobs once the work is submitted
to Condor.
Other than inserting the work initially, users should only query the tables,
and should not otherwise update values unless specified below.  A description
of the fields and types of each table are described next.

\begin{description}
\item[work table]
  Each row represents a piece of work to be executed by Condor.
  The table's attribute names, their SQL types and descriptions are as follows:

  \begin{description}
    \item[work\_data TEXT]
	Stores the Condor submit file data.
	This is the only attribute that needs to be set when inserting a
	new piece of work.
    \item[create\_ts TIMESTAMP]
	Database timestamp when record was created.
    \item[id BIGINT]
	Unique id for this record.
    \item[insert\_user TEXT]
	Database user that inserted this record.
    \item[state TEXT]
      The state of the piece of work in the system.
      The possible values are as follows:
      \begin{description}
	\item[initial]
	  Job has not yet been submitted to Condor.
	\item[chosen]
	  Job is about to be submitted to Condor.
	\item[in\_batch]
	  Job has successfully been submitted to Condor, but has not
	  yet completed.
	\item[complete]
	  All the jobs of this work have completed successfully or
	  have beeen removed.
	\item[failed]
	  The work has failed due to a permanent error.
      \end{description}
    \item[cdbq\_user TEXT]
	Database user that inserted this record into Condor.
	NULL if \Condor{submit} not executed.
    \item[cmd\_stdout TEXT]
	Standard out of \Condor{submit} after submitting this job.
	NULL if \Condor{submit} not executed.
    \item[cmd\_stderr TEXT]
	Standard error of \Condor{submit} after submitting this job.
	NULL if \Condor{submit} not executed.
    \item[cmd\_exit\_code INTEGER]
	Exit code of \Condor{submit}.
	0 is success, other values indicate failure.
	NULL if \Condor{submit} not executed.
    \item[cmd\_exit\_signal INTEGER]
	Signal causing this \Condor{submit} to fail.
	0 is non-signal exit, other values represent the signal number.
	NULL if \Condor{submit} not executed.
    \item[log\_file TEXT]
	Path to Condor's user log file for this work.
    \item[next\_pos INTEGER]
	Offset between last and next event record to read.
    \item[total\_jobs INTEGER]
	Total number of jobs that are represented by this work.
    \item[complete\_jobs INTEGER]
	Total number of jobs that have completed or been removed.
    \item[log\_err\_msg TEXT]
	Last error message caused by accessing the user log file.
    \item[log\_err\_num INTEGER]
	Last error number caused by accessing the user log file.
    \item[update\_ts TIMESTAMP]
	Database timestamp when record was last updated.
    \item[user\_id INTEGER]
	For use by the user to store an integer value.
    \item[user\_text TEXT]
	For use by the user to store a text value.
  \end{description}

\item[jobs table:]
  Each row repreents a job of a piece of work.
  These rows are inserted once the work has been submitted to the Condor system 
  and Condor has written a submitted event to the user log file.
  The table's attribute names, their SQL types and descriptions are as follows:

  \begin{description}
    \item[work\_id BIGINT]
	Id from the \emph{work} table.
    \item[cluster INTEGER]
	Condor cluster id of this job.
    \item[proc INTEGER]
	Condor process id of this job.
    \item[subproc INTEGER]
	Condor subprocess id of this job.
    \item[state INTEGER]
	Condor user log record event type of the last event seen for this job.
    \item[info TEXT]
	Additional information from the last log record seen for this job.
    \item[record\_ts TIMESTAMP]
	User log timestamp from the last log record event processed.
    \item[create\_ts TIMESTAMP]
	Database timestamp when job was first seen.
    \item[update\_ts TIMESTAMP]
	Database timestamp when record was last updated.
    \item[exit\_code INTEGER]
	Exit code of of this job.
	0 is success, other values indicate failure.
	NULL if job not yet complete.
    \item[exit\_signal INTEGER]
	Signal causing this job to fail.
	0 is non-signal exit, other values represent the signal number.
	NULL if job not yet complete.
  \end{description}
\end{description}

The other functions of \Condor{dbq.pl} perform administrative tasks on the
database.
These include inserting a new record into the work table given a Condor submit
file, and creating the tables and accounts required by \Condor{dbq.pl}.
If any of this functionality is requested, only the administrative tasks are
performed; the normal processing of work is not started.

If \Opt{{-}{-}noexecute} is used with any of the administrative functions, 
the SQL statements are printed to standard out instead of being executing.
This is useful to review the statements for security purposes, or to modify
them before creating the database objects.

The \OptArg{{-}{-}submit}{filename} option is used to insert a work record into
the system using the contents of \emph{filename} for the Condor submit file
data.  If \emph{filename} is `-', the program reads the Condor submit file from
standard in.

The \Opt{{-}{-}initdb} option causes the tables and other database objects to
be created.

The \Opt{{-}{-}createlang} option creates the plpgsql language to allow the
trigger functions created by \Opt{{-}{-}initdb} to be used.
This is not necessary if the database administrator created the language for
use when the database was created.

The \Opt{{-}{-}revokepublic} option prevents the database users from creating
object new tables in the database in the `public' schema that PostgreSQL grants
by default to all users.
If the user accounts do not need to create tables this option should be used to
revoke this unnecessary privilege.

The \OptArg{{-}{-}createsubmituser}{[dbinfofile]} option creates a database user
that can be used to submit jobs.
The username and password for this account is obtained from the db information
file specified, or using the default location for submit db information file
if none is specified.
This account has privileges to  insert work into the work table, and read
values from both tables.

The \OptArg{{-}{-}createworkuser}{[dbinfofile]} option creates a database user
that can be used by \Condor{dbq.pl} to perform its normal operation.
The username and password for this account is obtained from the db information
file specified, or using the default location for worker db information file
if none is specified.
This account has privileges to update existing records in the work table, and
to insert and update records in the jobs table.

There are three types of database accounts required by \Condor{dbq.pl}.
The first is an administrative database account that can create database
accounts, tables and other objects.
The administrative account requires database privileges to create tables and
users in the database.
This account is used by the administrative commands except the submit function.

The second account type is used to insert work records into the database by the
\Opt{{-}{-}submit} option.
This account requires privileges to insert records into the work table.
The account can be created using the \Opt{{-}{-}createsubmituser}, which also
grants access to select records from the two tables allowing this account to
used to both submit work and to monitor its progress.
Multiple acconts of this type may be used.

The final account type is used to perform the actual work of the system.
This account requires privileges to update records in the work table and to
both insert and update records in the jobs table.
The account can be created using the \Opt{{-}{-}createworkuser}.

The account information describing an account is stored in a database
information file that contains three lines:  the perl DBI string describing the
database, the username, and the password.
As these files contain sensitive information, only the operating system
accounts used to run the \Condor{dbq.pl} should be able to read the database
account information files.
By default, a separate db information file is used to store each of the three
accounts with the program selecting the credential information based on the
selected operation.
These files are stored in a directory specified by \Opt{{-}{-}dbinfodir}, and are
named db.admin.conf, db.submit.conf, and db.work.conf for the administrative,
submit and work accounts respectively.
If this scheme is used the correct account will be used without having to
specify the account information file.
If an account information file is missing \Condor{dbq.pl} will fail if an
operation is selected that requires the missing account information file.


\begin{Options}
  \OptItem{\OptArg{{-}{-}dbinfofile}{dbinfofile}}{
    Db information file containing the DBI connection string, username and
    password one to a line.
    If not specified this file defaults to files in the \Opt{{-}{-}dbinfodir}
    directory with the names db.work.conf, db.submit.conf, or db.admin.conf,
    based on the functionality requested being process the queue, submit a job,
    or perform administrative database functions respectively.
    The default value is the current directory.
    }
  \OptItem{\OptArg{{-}{-}dbinfodir}{dbinfodir}}{
    Directory to look for default database information files when
    \Opt{{-}{-}dbinfofile} is not specified.
    }
  \OptItem{\OptArg{{-}{-}logdir}{logfiledir}}{
    Directory name to place the Condor user log files generated by the Condor
    system.
    The default value is the current directory.
    }
  \OptItem{\OptArg{{-}{-}sleepamount}{seconds}}{
    Amount of time to sleep between each iteration of polling the database
    system for new work and processing new events in the Condor user logs.
    The default value is 10 seconds.
    }
  \OptItem{\OptArg{{-}{-}maxwork}{numofwork}}{
    Maximum number of inprogress work in the Condor system.
    If there is less work in the Condor system, new work will be submitted if
    available.
    }
  \OptItem{\OptArg{{-}{-}maxjobs}{numofjobs}}{
    Maximum number of incomplete jobs in the Condor system.
    If there are fewer jobs, new work will be submitted if available.
    }
  \OptItem{\OptArg{{-}{-}grabamount}{numofwork}}{
    Maximum amount of work to pull from the database system to insert into the
    Condor system per iteration.
    The default value is 10 work records.
    }

  \OptItem{\Opt{{-}{-}initdb}}{
    Create the tables and other associated database objects required by the
    condor\_dbq system.
    }
  \OptItem{\Opt{{-}{-}createlang}}{
    Causes the plpgsql language to created in the database.
    }
  \OptItem{\Opt{{-}{-}revokepublic}}{
    Revokes the default ability of database users to create new tables in the
    `public' schema in the database.
    }
  \OptItem{\OptArg{{-}{-}createsubmituser}{[dbinfofile]}}{
    Create a database account that has proper permissions to insert a work
    submission to the database.
    The username and paosword for the account is taken from the db information
    file specified or the default submit db information file is used if none is
    specified.
    }
  \OptItem{\OptArg{{-}{-}createworkuser}{[dbinfofile]}}{
    Create a database account that has proper permissions to allow condor\_dbq
    to process the queue.
    The username and paosword for the account is taken from the db information
    file specified or the default worker db information file is used if none is
    specified.
    }
  \OptItem{\OptArg{{-}{-}submit}{submitfile}}{
    Insert a row in the database containing the contents of \Arg{submitfile}.
    A \Arg{submitfile} value of `-' will read the submit file from standard
    in.
    }
  \OptItem{\Opt{{-}{-}noexecute}}{
    Do not execute any of the administrative database commands.
    Just print them out to standard out.
    }

  \OptItem{\Opt{{-}{-}help}}{
    Print a help message and exit.
    }

  \OptItem{\Opt{{-}{-}version}}{
    Print the version and exit.
    }
\end{Options}


\Examples

This section will present a complete introduction to using \Condor{dbq.pl}
including setup of the environment, creation of the database accounts and
tables, inserting work, and querying results.
This example assumes the database `condor\_dbq\_db' is already created, and
accessible on the same host via TCP/IP on the standard port.
It also assumes that the administrative database account,
`condor\_dbq\_admin\_user,' exists with the password
`condor\_dbq\_admin\_password,' and that this account has the ability to create
new database users and tables in the `condor\_dbq\_db' database.
The example code is written in the bash shell.

Create a secure directory to store the db information files.

\begin{verbatim}
mkdir condor_dbq_conf
chmod 0700 condor_dbq_conf
\end{verbatim}

Create the database account information files for a PostgreSQL database
named condor\_dbq\_db:

\begin{verbatim}
cat > condor_dbq_conf/db.admin.conf <<EOF
DBI:Pg:database=condor_dbq_db;host=localhost
condor_dbq_admin_user
condor_dbq_admin_password
EOF

cat > condor_dbq_conf/db.submit.conf <<EOF
DBI:Pg:database=condor_dbq_db;host=localhost
condor_dbq_submit_user
condor_dbq_submit_password
EOF

cat > condor_dbq_conf/db.worker.conf <<EOF
DBI:Pg:database=condor_dbq_db;host=localhost
condor_dbq_worker_user
condor_dbq_worker_password
EOF
\end{verbatim}

Create the plpgsql language used by objects created by \Opt{{-}{-}initdb}:

\begin{verbatim}
condor_dbq.pl --dbinfodir=condor_dbq_conf --createlang
\end{verbatim}

Create the database tables and objects:

\begin{verbatim}
condor_dbq.pl --dbinfodir=condor_dbq_conf --initdb
\end{verbatim}

Revoke the default ability of database users to create new tables:

\begin{verbatim}
condor_dbq.pl --dbinfodir=condor_dbq_conf --revokepublic
\end{verbatim}

Create the worker account that \Condor{dbq.pl} will use for its normal
operation using the default db worker information file for account information:

\begin{verbatim}
condor_dbq.pl --dbinfodir=condor_dbq_conf --createworkuser
\end{verbatim}

Create the submit account that users can use to insert work into the
system using the default submit db information file for account information:

\begin{verbatim}
condor_dbq.pl --dbinfodir=condor_dbq_conf --createsubmituser
\end{verbatim}

Create the directory to store the user logs:

\begin{verbatim}
mkdir condor_dbq_logs
chmod 0722 condor_dbq_logs
\end{verbatim}

Start the system:

\begin{verbatim}
condor_dbq.pl --dbinfodir=condor_dbq_conf --logdir=condor_dbq_logs
\end{verbatim}

Create a Condor submit file containing a simple job:

\begin{verbatim}
cat > my.submit <<EOF
Universe = vanilla
transfer_executable = false
notification = never
output = my.out
error = my.err
log = my.log
Executable = /bin/sh
Arguments = "-c 'exit 0'"
Queue
EOF
\end{verbatim}

Insert the Condor submit file into the work table:

\begin{verbatim}
condor_dbq.pl --dbinfodir=condor_dbq_conf --submit=my.submit
\end{verbatim}


\ExitStatus

\Condor{dbq.pl} will exit with a status value of 0 (zero) upon success,
and it will exit with a non-zero value upon failure.


\end{ManPage}
