1. Common problems
2. Retrieving Job Status Information
3. SGE Failure and Exit Codes

1. Common problems

1.1 Your job "starves" in the waiting queue

1.2 Only some of a set of identical jobs die

1.3 All your jobs die at once

1.4 qrsh fails with an error message complaining 'Your "qrsh" request could not be scheduled, try again later'

2. Retrieving Job Status Information

Shortly after jobs are finished the job status information is no longer accessible using normal SGE commands. The MACBAT web page does contain in the menu for a given farm the heading 'Reporting - Finished Jobs'. The related link gives access to the job information of your finished jobs. This URL will list an overview of finished jobs per day. From there listings of finished jobs on a given date can be retrieved. By further following the links every job detail can be displayed up to the single tasks in array jobs.
The same information can be obtained using command line tools on Linux. The command arcx sgejobs is provided for retrieving this info. A short usage is printed with

arcx sgejobs -h

The information is displayed for the default farm ('uge'), but another farm ('pax' or the old 'sge') can be chosen using the -f=<farm> switch. All information is displayed only belonging to authenticated users and only for own jobs. Group admins can be registered (please contact UCO) who then are able to view information on jobs belonging to other users of their group.
If arcx sgejobs called without further arguments a list of submission dates and number of jobs submitted that day is printed.

arcx sgejobs

If a submission date or submission interval is given (date format yyyy-mm-dd) then job data are printed in a tabular form.

arcx sgejobs 2015-09-01

Finally if a job number is given.

arcx sgejobs 946942

then the full information belonging to that job is displayed.

arcx sgejobs 946942

         qname = std.q
         hostname = bladecd.zeuthen.desy.de
         unixgroup = sysprog
         owner = ahaupt
         job_name = farmHEPSPEC.sh
         job_number = 946942
         submission_time = 1441134736
         start_time = 1441134747
         end_time = 1441144320
         failed = 0
         exit_status = 0
         ru_wallclock = 9572
         ru_utime = 9047
         ru_stime = 166
         ru_maxrss = 493216
         ru_minflt = 23156122
         ru_majflt = 483
         ru_inblock = 2699188
         ru_oublock = 7691376
         ru_nvcsw = 114531
         ru_nivcsw = 943807
         project = sysprog
         granted_pe = NONE
         slots = 1
         task_number = 0
         cpu = 9213
         mem = 1023.24
         category = -l h_cpu=32400,h_rss=2G,h_stack=10M,hostname=bladec*,m_mem_free=2.1G,s_rt=32700,tmpdir_size=5G -P sysprog -binding linear_automatic 1 0 0 0 no_explicit_binding
         pe_taskid = NONE
         maxvmem = 720318000

3. SGE Failure and Exit Codes

The exit code is the return value of the exiting program. It can be a user defined value if the job is finished with a call to 'exit(number)'. For abnormally terminated jobs it is the signal number + 128. If an SGE job is terminated because a limit was exceeded, SGE has sent a SIGUSR1 signal (10) to the job which results in an exit code of 138.
The SGE failure code indicates why a job was abnormally terminated. The following incomplete list mentions the most frequent failure codes: