gridtk.tools¶

Functions that replace shell-based utilities for grid submission and probing.

Functions

`get_array_job_slice`(total_length)	A helper function that let's you chunk a list in an SGE array job.
`make_shell`(shell, command)	Returns a single command given a shell and a command to be qsub'ed.
`qdel`(jobid[, context])	Halts a given job.
`qstat`(jobid[, context])	Queries status of a given job.
`qsub`(command[, queue, cwd, name, deps, ...])	Submits a shell job to a given grid queue.
`str_`(v)	Always returns the string representation of the given `name`.

gridtk.tools.str_(v)[source]¶

Always returns the string representation of the given name.

If it is a bytes object, it will be converted into str.

If it is a str object, it will simply be resurned.

Parameters:: v (str | bytes) – a bytes or str object
Return type:: str
Returns:: A str object

gridtk.tools.qsub(command, queue=None, cwd=True, name=None, deps=[], stdout='', stderr='', env=[], array=None, context='grid', hostname=None, memfree=None, hvmem=None, gpumem=None, pe_opt=None, io_big=False, sge_extra_args='')[source]¶

Submits a shell job to a given grid queue.

Keyword parameters:

command

The command to be submitted to the grid

queue

A valid queue name or None, to use the default queue

cwd

If the job should change to the current working directory before starting

name

An optional name to set for the job. If not given, defaults to the script name being launched.

deps

Job ids to which this job will be dependent on

stdout

The standard output directory. If not given, defaults to what qsub has as a default.

stderr

The standard error directory (if not given, defaults to the stdout directory).

env

This is a list of extra variables that will be set on the environment running the command of your choice.

array

If set should be either:

a string in the form m[-n[:s]] which indicates the starting range ‘m’, the closing range ‘n’ and the step ‘s’.
an integer value indicating the total number of jobs to be submitted. This is equivalent ot set the parameter to a string “1-k:1” where “k” is the passed integer value
a tuple that contains either 1, 2 or 3 elements indicating the start, end and step arguments (“m”, “n”, “s”).

The minimum value for “m” is 1. Giving “0” is an error.

If submitted with this option, the job to be created will be an SGE parametric job. In this mode SGE does not allow individual control of each job. The environment variable SGE_TASK_ID will be set on the executing process automatically by SGE and indicates the unique identifier in the range for which the current job instance is for.

context

The setshell context in which we should try a ‘qsub’. Normally you don’t need to change the default. This variable can also be set to a context dictionary in which case we just setup using that context instead of probing for a new one, what can be fast.

memfree

If set, it asks the queue for a node with a minimum amount of memory Used only if mem is not set (cf. qsub -l mem_free=<…>)

hvmem

If set, it asks the queue for a node with a minimum amount of memory Used only if mem is not set (cf. qsub -l h_vmem=<…>)

gpumem

Applicable only for GPU-based queues. If set, it asks for the GPU queue with a minimum amount of memory. The amount should not be more than 24. (cf. qsub -l gpumem=<…>)

hostname

If set, it asks the queue to use only a subset of the available nodes Symbols: | for OR, & for AND, ! for NOT, etc. (cf. qsub -l hostname=<…>)

pe_opt

If set, add a -pe option when launching a job (for instance pe_exclusive* 1-)

io_big

If set to true, the io_big flag will be set. Use this flag if your process will need a lot of Input/Output operations.

sge_extra_args

This is used to send extra argument to SGE. Note that all its arguments are directly used in qsub command. For example, jman submit -e “-P project_name -l pytorch=true” – … will be translated to qsub -P project_name -l pytorch=true – …

Returns the job id assigned to this job (integer)

gridtk.tools.make_shell(shell, command)[source]¶

Returns a single command given a shell and a command to be qsub’ed.

Keyword parameters:

shell The path to the shell to use when submitting the job.

command The script path to be submitted

Returns the command parameters to be supplied to qsub()

gridtk.tools.qstat(jobid, context='grid')[source]¶

Queries status of a given job.

Keyword parameters:

jobid The job identifier as returned by qsub()

context The setshell context in which we should try a ‘qsub’. Normally you don’t need to change the default. This variable can also be set to a context dictionary in which case we just setup using that context instead of probing for a new one, what can be fast.

Returns a dictionary with the specific job properties

gridtk.tools.qdel(jobid, context='grid')[source]¶

Halts a given job.

Parameters:

jobid (int) – The job identifier as returned by qsub()
context (str) – The setshell context in which we should try a ‘qsub’. Normally you do not need to change the default. This variable can also be set to a context dictionary in which case we just setup using that context instead of probing for a new one, what can be fast.

Return type:

None

gridtk.tools.get_array_job_slice(total_length)[source]¶

A helper function that let’s you chunk a list in an SGE array job.

Use this function like a = a[get_array_job_slice(len(a))] to only process a chunk of a.

Parameters:: total_length (int) – The length of the list that you are trying to slice
Return type:: slice
Returns:: A slice to be used.
Raises:: NotImplementedError – If “SGE_TASK_FIRST” and “SGE_TASK_STEPSIZE” are not 1.