| System Architecture |
| Architecture |
The main JobServer scheduling and processing engine runs in a dedicated Java VM(s). The supporting
GUI applications are web based and integrate closely with the main JobServer engine. See JobServer architecture diagrams:
Download diagrams of JobServer's architecture and deployment configuration options.
| Yes |
Yes |
| Database Transaction Reliability |
All critical operations, managed by JobServer, from the scheduling to the job processing
are managed and organized in a relational database. All major functions are backed by database transaction level reliability to insure
integrity and consistency. |
Yes |
Limited |
| Database Support |
JobServer Professional has support for Oracle, MYSQL, and PostgreSQL. JobServer Standard
edition comes bundled with its own built-in HSQL database. JobServer Pro offers greater scalability in processing and
management of jobs. Both Pro and Standard editions share many of the same set of features, but the Pro edition offers greater scalability, reliability and durability. |
Yes |
No |
| Mesos Cluster Processing |
JobServer integrates closely with Mesos clustering to allow jobs to be processed
across a Mesos cluster of any size and capacity. Partitions/Queues in JobServer can be associated to a Mesos cluster for highly
scalable distributed processing and leveraging the full power of Mesos. No JobServer
software has to be installed on the Mesos slave nodes. Connect JobServer to Mesos and
watch your jobs run with uber scalability! |
Yes |
Yes |
| Agent Clustering and Distributed Processing |
JobServer can be deployed on one or many servers. Job processing can be distributed across a cluster of servers allowing for highly scalable distributed processing and hot failover of job processing and job scheduling. You can easily scale JobServer to run on tens or hundreds of servers and spread your job processing load across a high capacity cluster. |
Yes |
No |
| Failover |
JobServer will detect failover on the primary scheduling engine and have the secondary server take over control and continue job scheduling and processing. |
Yes |
No |
| Web-based Interface |
All JobServer tools/applications are web based and also utilize the GWT (Google Web Toolkit) AJAX
framework for SDK plugin extensibility and customization. Sample JobServer GUI screen shots. |
Yes |
Yes |
| Security and Authentication |
JobServer supports SSL/TLS for web based authentication. All user passwords and configuration passwords are encrypted. |
Yes |
Yes |
| LDAP and Active Directory |
JobServer has support for LDAP version 3 allowing integration with LDAP and Active Directory servers for centralized authentication management. |
Yes |
Yes |
| Job Scheduling |
| Advanced Scheduling Rules |
The JobServer Scheduling Engine leaves no scheduling rule out
of your reach. Virtually, any repeating or one-off scheduling
pattern that you can imagine can be represented using the JobServer
Scheduling Engine.
As an example, here is a sample of scheduling rules that are supported:
Every 15 minutes starting on 1/1/2011 09:00AM and only between 10:00AM and 7:30PM and ending on 3/3/2012 5:00PM
Every other month starting on 1/1/2011 09:00AM and on the last, first and second Monday and last Friday of the Month
Every other Week on Monday and Wednesday starting on 7/7/2011 except for Wednesday of 7/30/2011
Many more...........
|
Yes |
Yes |
| Relative & Absolute Time Scheduling |
With this scheduling rule, a job can be run using "absolute scheduling" or "relative scheduling".
Absolute scheduling means the job will run relative to the original start time, so
if the job has a start time of 10:00 AM and the interval is one hour then the job will
run at the top of every hour. With this same example, if relative scheduling is used, the
job will still run every hour but it may or may not run at the
top of every hour as it will run one hour from the last time it ran last. |
Yes |
Yes |
| Skip Ahead Scheduling Rule |
This features defines what a job does if it falls behind in its schedule. Often times
this feature does not come into play. But for example, if a job is set to run at 6:00PM on a given day but JobServer
is down for 2 days due to say a hardware problem. When JobServer is started, the job will get kicked off, since its
next run date has long past. Now this is when the Skip-Ahead feature comes into play. If this option is set, then
after the initial run, the job will "skip" ahead to the next valid run date that is in the future. If it is not
set, then next scheduled date will be used even if it is not in the future. The job will
keep running until it encounters a next run date that is in the future again. |
Yes |
Yes |
| Schedule Jobs One-at-a-Time |
This scheduling feature permits only one instance of a job to be running at any one time.
This is useful in scenarios where you want to prevent more than one instance of a job from being scheduled at any
one time. For example, you may have a job that runs every two minutes and in some rare cases, when the job might take 3
minutes to complete its processing you do not want the same job to run again. This feature prevents any other instance
of the job from running until the current
instance has completed. This feature can also be used in scenarios where you want a job to be running for extended
periods of time and you only want to have one instance of the job to ever be running at any moment. So as an example,
you might setup a job to run every 2 seconds and this job might be polling indefinitely (until killed) looking for
certain events like a file being dropped in an FTP directory. In this scenario the job runs forever looking for
such events. If it ever dies, due to a failure for example, the scheduler will start another instance. |
Yes |
Yes |
| System Management |
| Controlled System Shutdown and Startup |
JobServer has a variety of controls available to administrators for managing and operating
the JobServer scheduler and job processing environment. Users can startup JobServer in several modes, such as "Gradual
Startup". This startup mode, for example, allows you to startup the JobServer engine but suspending all job processing
and job scheduling. You can then go into the JobServer system controls and turn on job scheduling and/or processing for the
entire system or for one Partition at a time - it is all under your control. There are similar controls for shutting down
the system. You can shutdown the system "immediately" (jobs that are running will be killed) or "gracefully" (all currently
running jobs will run to completion then JobServer will shut itself down once all jobs are done and during this time no new jobs will be allowed
to be run or be scheduled). There are other features similar to these as well that administrators will find useful. |
Yes |
Yes |
| Administer System Wide Scheduling and Queuing |
Administrators can control the scheduling, queuing and job processing of the JobServer
environment at a system wide level or at a Partition by Partition level. This allows users to shutdown, suspend and
throttle different groups of jobs in a variety of ways. |
Yes |
Yes |
| Event Triggers & File Transfer |
Jobs can be triggered when certain events happen like a file put into a directory or
a database record entered in a table. This feature allows you to create "monitoring jobs" that
when they detect an event (like a file being placed in a directory) can then invoke other
jobs to run, and also pass parameters to those jobs, based on the event that triggered them. Transferring
files using FTP, SFTP and FTPS are provided using the Java APIs and through the open source SDK. |
Yes |
Yes |
| Built-in Support for Batch/Shell Scripting |
JobServer comes with built-in support for executing Unix/Linux shell scripts and MS Windows
batch scripts. This is provided by a Tasklet that comes pre-packaged with JobServer. The Tasklet allows job designers
to configure arbitrary scripts to be run by JobServer. This scripting Tasklet uses JobServer's open source plugin SDK
for building custom Tasklets and Jobs. Source code is provided for developers to customize if they choose. The scripting
Tasklet provides many powerful features like the ability to store and save the script and output of the script for each
job run and comes with a GUI interface. |
Yes |
Yes |
| Variable Time Property (for ETL Jobs) |
This feature does not directly impact the scheduling rules of a job, but is instead passed
to a job when it is run. The start and end offsets defines the amount of days that will be added/subtracted to the
next run time and last run time, respectively. When a job is run, it has access to an effective date range
(start time and end time values). The
last run date is the end date and the next run date is the start date. Normally, this feature is used to make business
decisions and for ETL types of jobs that need to know when they ran last because the next ETL run depends on the date/time
the job ran last. For example, if you have a job that runs every day and it pulls data from a database, the start date
and end date can be used to access data from the database over that given date range. This can be a very useful
feature for ETL type processing applications. You also have the option of selecting a custom start and end dates
for each run time. |
Yes |
Yes |
| Job Management |
| Point and Click Workflow Builder |
Jobs within JobServer are composed of multiple Tasklets (and an optional customizable
GUI interface called a Weblet). A job can be composed of one or more Taskets that run as part of a workflow chain. Tasklets
are modular pieces of code that execute and log their output to JobServer. When you assign a Tasklet to a job you choose
the implementation of the Tasklet. You can use some of the built-in Tasklets or have a Java Developer write a custom one and
add it dynamically to the JobServer environment. Tasklets within a job, can also share and pass data to each other when they
run in a workflow chain. This
allows for very advanced capabilities. The property passing between Tasklets can be done all through a point and click GUI
interface. |
Yes |
Yes |
| Job Dependency Scheduling |
Dependency rules are a very powerful feature that can allow one Job to affect the
scheduling rules of other Jobs. The best way to describe it, is to use an example. Say you have Jobs C and B that
run every day at 8:00am, but they must not run until Job A runs before them. And Job A normally runs at 2:00am.
If for some reason Job A fails to run at its expected time, Jobs C and B must not run till Job A completes. This
is just one example of how to use Dependency Rules. |
Yes |
Yes |
| Job Property Time Versioning |
Versioning a Tasklet, allows you to specify changes to a Tasklet's input properties that will take effect at
predefined dates/times in the future. A Job that is executed by JobServer, will run with a specific time version of a Tasklet's properties that are effective
at the time the Job/Tasklet is run. The input properties of the Tasklet can change, with time, if you choose to define future versions to the properties of a Tasklet. This is a very powerful feature, as it allows you to make changes to a Tasklet's input properties ahead of time, and they only have them become "effective" at a set time in the future. For example, at the start of the new year, have the job (that runs weekly for example) start sending its summary report to "bob@aceme.com" instead of "mike@acme.com". You can do this by creating a future version of the Tasklet's properties that take effect at the start of the year, for example. |
Yes |
Yes |
| Importing/Exporting Jobs |
Export a job's state to an XML file format and import back into another JobServer environment. This feature can be useful when needing to move a job from say a staging environment to a production system. You can export/import one or many jobs at a time. All the state of a job from scheduling to Tasklet properties are persisted in an XML file format and can be later imported back into another JobServer system. Import/Export jobs using simple point and click GUI app or script it using easy to use command line utilities. |
Yes |
Yes |
| Job Processing |
| Job Grouping |
Job Groups are a way to logically organize jobs together. You can restrict user access
to groups of jobs and assign certain properties and behavior to groups of jobs (like error alerting for example). |
Yes |
Yes |
| Throttle Job Processing |
JobServers allows you to organize jobs into Partitions. Each Partition can be configured with its own resource allocation like how many jobs are allowed to run concurrently within it. There are other attributes you can assign, but one
of the main purposes of a Partition is to allow users to throttle job processing and job scheduling for a category or collection
of jobs. |
Yes |
Yes |
| Advanced Threading Design |
JobServer uses an advanced threading design throughout the system. This is used for such things as scheduling and queuing jobs as well as for managing job processing concurrency. JobServer is capable of taking advantage of high-end hardware with many CPUs and cores. For example, you can
configure JobServer (via the GUI tools) to create multiple job scheduling threads so as to improve scheduler response times (for
cases where you have hundreds or thousands of jobs running concurrently) by assigning more scan threads to the
JobServer schedulers. JobServer makes many of these features available in the GUI for the administrator to configure
if they need advanced tuning options. |
Yes |
Yes |
| Kill Running Jobs |
JobServer allows users to kill running jobs. Jobs that run in their own isolated JVM and jobs that run in shared JobServer JVM pools can be killed. |
Yes |
Yes |
| Delete and Edit Queued Jobs |
Jobs get put in a Queue when the maximum concurrent job threshold has been reached for
a Partition. JobServer offers end users tools for deleting and changing the order and priority of jobs in the Queue. |
Yes |
Yes |
| Isolated JVM Per Job |
When a job enters running state, it can be run in a dedicated thread within a shared JVM or
the user has the choice to have the job run in its own processes isolated JVM. Running jobs in isolated JVM's can be useful
in cases where process isolation is important and to reduce chances that a rogue or misbehaving job can bring down a partition or node. |
Yes |
Yes |
| Queue Management |
Monitor and track jobs as they enter the Queue and the Job Processor. Jobs can be organized
to use dedicated Queues and have varying priority levels. |
Yes |
Yes |
| Running Jobs Manually |
Jobs can be launched on demand from the GUI, invoked from the web service interface, or launched from the command line (using the jsmonitor tool). When you need to run jobs outside their normal scheduling times, running jobs manually on demand can be very useful. |
Yes |
Yes |
| Automatic Job Retry |
If there is an abrupt failure of a primary/secondary or remote job processing node (e.g. host computer crashes), JobServer will attempt to retry failed jobs. Jobs can be configured to allow this capability on a job per job basis. Jobs can be configured to retry from the start or they can be configured to retry from the last failed Tasklet. |
Yes |
Yes |
| Reporting and Monitoring |
| Detailed System Audit Trail |
Everything the main JobServer engine does is logged and can be monitored
using JobServer's reporting and monitor tools. You can track every startup and shutdown of JobServer and
every error that has occurred and even be alerted by email of problems. |
Yes |
Yes |
| Detailed Monitoring of Job Processing |
JobServer offers powerful tools for tracking every step and transition a jobs goes thru
from the time it is first scheduled to the last message it logs when the job finishes running. You can monitor and
query for jobs based on a variety of criteria, allowing users and administrators great control and visibility over what
is happening in the system as a whole and with individual jobs and groups of jobs. |
Yes |
Yes |
| Detailed Logging and Reporting |
JobServer's Tasklet SDK allows jobs and Tasklets to log messages when they run. The SDK
supports both the Java Logging API and Log4J. Every logging message made by a Tasklet is capture and stored and can
be reported on. |
Yes |
Yes |
| Future Schedule Reporting Calendar |
Would you like to know what jobs are about to run in the next hour or next month? JobServer offers
reporting tools to allow you to query for such reports at an individual job level and/or for a collection of jobs. You can see what jobs are expected to run across any future time window - a very powerful tool. |
Yes |
Yes |
| Performance Statistics Reports |
Curious how your job processing has been performing recently or in the past? Need to know how long it takes job or groups of
jobs to run or how often do jobs sit in the Queue? Wondering if there is a pattern over the past few weeks that jobs are taking
longer to run, wondering which jobs are the problem? Well, JobServer has reports you can run from the GUI to answer
all these questions and to help you quickly diagnose potential problem jobs and discover performance changes. |
Yes |
Yes |
| Security / Audit Trail / Alerting |
| Job Processing Audit History |
With this feature, JobServer offers the ability for every job to keep an audit trail of
every input and every output generated by the job. This is supported both by the SDK and by the JobServer applications.
This capability can allow a job to store all data used during the job run and to then report on this information. This
can be very critical for systems that need to keep strict accounting of what a job did and under what conditions it ran
for accounting, legal and other troubleshooting purposes. For example, if you use the Batch Script Tasklet, you can have it
store the exact script it used during every run and the exact output it generated during every run. All this information can
then be reported on using JobServer's standard report GUIs. |
Yes |
Yes |
| User Actions Audit Trail |
Keep track of every critical action users and administrators take. From who created a job,
to who edited it, to when the system was shutdown and by whom. This kind of accountability is critical in many organizations. |
Yes |
Yes |
| User Rights & Security Management |
Users can be granted access to applications/features and jobs using JobServer's detailed
rights management. For example, you can grant a user the right to view and edit the schedules of one job group but only the rights
view the schedules of another job group. You can grant users the rights to monitor and view job processing reports
but not to edit job attributes or properties. JobServer offers fine grained feature and resource access. |
Yes |
Yes |
| Cascading Email Alerts |
You can setup to receive alerts of system level errors/failures and/or individual job processing
errors. Alerts can be setup to cascade to users/administrators based on events/errors/alerts that occur on
groups of jobs, job partitions and for individual jobs. So you can
sign up to receive errors if any job in a Partition fails or just sign up to receive email alerts if one particular
job fails. |
Yes |
Yes |
| APIs, Hadoop, and SOA |
| Open Source Developer SDK |
The goal of the JobServer open source developer SDK, soafaces, is to allow developers to build their own jobs in a modular and reusable
fashion as they so desire. Many of the features are optional, so you may choose the level of functionality and
reusability that you wish to employ. To create a job that will run in JobServer, all it takes is implementing one
interface and putting the developed Java software into an appropriate JAR file. The job can be one line of code or
thousands of lines of code. JobServer also comes prepackaged with built-in components called Tasklets/Weblets that can, for
example, execute unix bash, ksh, and perl scripts and others that can send email alerts, FTP files, make Web
Services calls, or perform XML processing. |
Yes |
Yes |
| Hadoop Support |
JobServer comes with integrated support for Hadoop out of the box. You can use JobServer as your central platform for building, launching and monitoring your Hadoop job processing activity. JobServer's Java developer API makes building custom extensions to Hadoop easy. Use JobServer to build, schedule and track a single Hadoop task or to compose multiple Hadoop tasks into more complex jobs and workflows. For example, a job can be composed of multiple Hadoop tasks where one task directs its output to the input of other Hadoop tasks in the job chain. JobServer makes it easy to centrally schedule, track and monitor all your Hadoop tasks and jobs from one place for real-time monitoring as well as for historical reporting and auditing. JobServer's workflow orchestration allows users to manage and manipulate files and content between your local storage system and your Hadoop HDFS. You can also build custom web GUIs (using GWT) for your Hadoop tasks to visualize all input and output content and show real-time status as the tasks are running or to report on final status and results after a Hadoop task has finished processing. |
Yes |
Yes |
| Web Services API |
A web services API is available that allows control and managing various functions
within JobServer from a web services API. |
Yes |
Yes |
| SOA Integration |
JobServer comes with built-in support for SOA using the Mule platform. Both Tasklets
and Weblets have the option of using Mule to access SOAP, JMS and other services using the Mule Java API. |
Yes |
Yes |
