openQA Scheduler Design
When we say scheduling in openQA we are talking about the trigger which runs one or more tests in the framework.
Usually we use isos post to refer to the this - when we call them by terminal. the isos post in fact, is the API
which calls the endpoint which says to the according controller to perform the action. And it looks like this
❯ openqa-cli api --host http://localhost --apikey <apik> --apisecret <apis> -X POST isos
400 Bad Request
Error: missing parameters: DISTRI VERSION FLAVOR ARCH
This is one way, and as it is shown above any job requires some particular variables, DISTRI VERSION FLAVOR ARCH but it will also need ISO at the minimum or/and BUILD GROUP even.
So what’s the entry point? Confusing enough we find in the lib/OpenQA/WebAPI.pm
# api/v1/isos
$api_ro->post('/isos')->name('apiv1_create_iso')->to('iso#create');
The $api_ro name might suggest “read-only” but here it means “authenticated
with an API key, without needing operator or admin role”. So a regular user with
a valid API key can trigger scheduling.
Jumping into lib/OpenQA/WebAPI/Controller/API/V1/Iso.pm we can take a look
of what’s going on. The create function first validates the mandatory
parameters (DISTRI, VERSION, FLAVOR, ARCH), then creates a record in the
ScheduledProducts table via create_with_event, and only then proceeds to
schedule. The record exists in the database before any job is created.
At this point the path splits in two. If the caller passes async=1 the
controller enqueues a Minion background job and returns immediately:
return $self->render(json => $scheduled_product->enqueue_minion_job(\%params)) if $async;
The Minion task lives in lib/OpenQA/Task/Iso/Schedule.pm. Its main reason to
exist is that it can perform an HTTP fetch for SCENARIO_DEFINITIONS_YAML_FILE
before scheduling — something that would block the web process if done
synchronously. After the optional download it calls the same schedule_iso as
the synchronous path, so both paths converge there.
In the default (synchronous) case the controller calls directly:
my $scheduled_jobs = $scheduled_product->schedule_iso(\%params, undef);
Actually if we grep directly the function name create_from_settings we see
where the main operations eventually take place
❯ grep create_from_settings -rn lib
lib/OpenQA/Schema/Result/ScheduledProducts.pm:266: my $job = $jobs_resultset->create_from_settings($settings, $self->id);
lib/OpenQA/Schema/ResultSet/Jobs.pm:135:sub create_from_settings ($self, $settings, $scheduled_product_id = undef) {
lib/OpenQA/WebAPI/Controller/API/V1/Job.pm:328: my $job =
$self->schema->resultset('Jobs')->create_from_settings($job_settings);
So back to schedule_iso: before doing any real work it updates the
ScheduledProduct status to SCHEDULING and wraps everything in a
try/catch, calling set_done at the end to transition to SCHEDULED (or
CANCELLED if meanwhile a cancellation came in). Then it delegates to the
private _schedule_iso.
This is an intersection point where everything starts to get complicated. The
reason is that there are various approaches to configure a job, depending on
your workflow. If _OBSOLETE or _DEPRIORITIZEBUILD flags are set,
_schedule_iso will also cancel or deprioritize existing jobs for the same
DISTRI/VERSION/FLAVOR/ARCH before creating the new ones — that is what makes
re-triggering a build work cleanly in practice.
The following they will prepare the settings according to the workflow:
_schedule_from_yaml— used whenSCENARIO_DEFINITIONS_YAML(inline) orSCENARIO_DEFINITIONS_YAML_FILEis provided_generate_jobs— the classic path that reads job templates from the database
And later _create_jobs_in_database which will store the jobs in the
database (that’s why all the operations are in the Results/ dir). Looking at
the function signature we see what we expect: $jobs, \@failed_job_info, $skip_chained_deps, $include_children, \@successful_job_ids, \@minion_ids
Finally it will reach create_from_settings which for each job calls its
own create.
After all jobs are inserted, _schedule_iso pokes the scheduler daemon:
OpenQA::Scheduler::Client->singleton->wakeup;
The jobs are in the database at this point, but nothing runs until the scheduler daemon is notified. That wakeup call is the bridge between “ISO post” and actual test execution.
The architecture
What the pattern is called
The broad term for what happens when code accumulates concerns in one place over
time is the Big Ball of Mud 1 — a system with no clear architecture
that grew organically by adding code wherever it was convenient. The specific
manifestation here is the God Object 2 anti-pattern:
ScheduledProducts.pm is ~960 lines and owns too many responsibilities at once.
Where it is getting messy
A DB result class that runs the whole show
lib/OpenQA/Schema/Result/ScheduledProducts.pm lives in the Schema/Result/
namespace, which by DBIx::Class convention is supposed to be the database model
layer. But it does:
- YAML parsing and validation
- Settings inheritance and merging (product → machine → test suite → template)
- Dependency graph sorting and cycle detection
- Asset registration
- GitHub status reporting
- Minion job enqueueing
- Event emission (AMQP)
The controller (Iso.pm) is appropriately thin. The problem is everything was
pushed down into the model class as the feature grew. There is no service layer
in between.
Settings as an untyped, overloaded hash
The flat %settings hash is passed through the entire call chain carrying two
completely different kinds of data mixed together:
- Job configuration (what the test actually needs:
ISO,DESKTOP,ARCH, …) - Scheduling directives consumed and deleted before jobs are created:
_OBSOLETE,_DEPRIORITIZEBUILD,_SKIP_CHAINED_DEPS,_GROUP,_PRIORITY,__-prefixed product metadata
_schedule_iso opens by delete-ing the directive keys out of $args. That
mutation is the only thing separating “instructions for the scheduler” from
“settings for the job”. If a caller forgets a key or misspells the prefix, the
directive silently becomes a job setting. This is sometimes called
stringly-typed 3 code — a valid hash key is the only contract.
Duplicated settings-merging logic
_generate_jobs and _schedule_from_yaml both resolve the effective settings
for a job (product + machine + test suite + template + caller args), handle
WORKER_CLASS merging separately from other keys, and then call the same
filtering helpers. The merging logic at the top is written twice. A bug in how
precedence works between product and machine settings would need to be fixed in
both places.
Output parameters instead of return values
_create_jobs_in_database takes seven parameters, four of which are output
parameters passed as mutable array references:
sub _create_jobs_in_database ($self, $jobs, $failed_job_info, $skip_chained_deps,
$include_children, $successful_job_ids, $minion_ids = undef)
The caller (_schedule_iso) sets up empty arrays, passes refs down, then reads
the results back out after the call. This is a C-style pattern — you cannot look
at the call site and know what comes back without reading the callee. As more
edge cases were added (the deadlock retry handler also modifies those same
arrays), it becomes easy to miss that a reset is needed.
OpenQA::App->singleton inside a model class
_schedule_iso and _create_jobs_in_database both call
OpenQA::App->singleton->gru to enqueue Minion jobs. _schedule_from_yaml
calls $app->validate_yaml. A Schema::Result class that reaches for the
global application singleton is hard to test in isolation and creates an
invisible dependency on the full app stack being initialized.
GitHub CI reporting on the scheduling object
report_status_to_github is conceptually unrelated to scheduling — it is about
post-hoc CI status. It ended up on ScheduledProducts because that is the
object that has the job list and the status, but it means the scheduling model
now also knows about HTTP clients, GitHub API tokens, and webhook payloads. It
gets called from set_done, so every scheduling path implicitly triggers a
GitHub API call.
What is fine
- The public/private split (
schedule_iso/_schedule_iso) with the status machine (SCHEDULING → SCHEDULED, try/catch,set_done) is clean and correct. The state transitions are honest about failure modes. - The dependency resolution itself (
_sort_dep,_check_for_cycle,_create_dependencies_for_job) is well-contained. - The two scheduling paths converging at
_create_jobs_in_databaseis architecturally sound — the problem is the duplicated merging code above that point. - The savepoint pattern around each individual job creation
(
svp_begin/svp_release/svp_rollback) handles partial failures correctly.
What would be cleaner
The structural fix is a service layer 4 between the controller and the model:
Controller (Iso.pm)
→ SchedulingService (e.g. OpenQA::Scheduler::IsoScheduler)
→ ScheduledProducts (pure DB model: state machine only)
→ JobFactory (settings resolution + create_from_settings)
→ DependencyResolver (graph sort, cycle detection)
Concretely:
- Separate scheduling directives from job settings at the API boundary, not deep
inside
_schedule_isoviadelete. A typed params struct (even just a validated hashref with known keys) would make the contract explicit. _create_jobs_in_databaseshould return a result hashref, not accumulate into caller-owned arrays.- GitHub integration belongs in an event handler that reacts to the
openqa_job_createevent, not baked intoset_done. - The settings-merging logic shared between
_generate_jobsand_schedule_from_yamlshould live in one place — there is alreadyOpenQA::JobSettings, which is the right home.
References
Foote, B. & Yoder, J. (1997). Big Ball of Mud. Pattern Languages of Program Design 4. ↩︎
Riel, A. (1996). Object-Oriented Design Heuristics. Addison-Wesley. — The God Class heuristic (heuristic 3.1) states that a class should have a single, well-defined responsibility. ↩︎
Stringly-typed: a portmanteau of “string” and “strongly-typed”, used to describe designs that use strings where a more specific type would be safer. Popularised on Coding Horror (2012). ↩︎
Fowler, M. (2002). Service Layer. Patterns of Enterprise Application Architecture. Addison-Wesley. ↩︎