When we say scheduling in openQA we are talking about the trigger which runs one or more tests in the framework. Usually we use isos post to refer to the this - when we call them by terminal. the isos post in fact, is the API which calls the endpoint which says to the according controller to perform the action. And it looks like this

❯ openqa-cli api --host http://localhost --apikey <apik> --apisecret <apis> -X POST isos
400 Bad Request
Error: missing parameters: DISTRI VERSION FLAVOR ARCH

This is one way, and as it is shown above any job requires some particular variables, DISTRI VERSION FLAVOR ARCH but it will also need ISO at the minimum or/and BUILD GROUP even.

So what’s the entry point? Confusing enough we find in the lib/OpenQA/WebAPI.pm

    # api/v1/isos
    $api_ro->post('/isos')->name('apiv1_create_iso')->to('iso#create');

The $api_ro name might suggest “read-only” but here it means “authenticated with an API key, without needing operator or admin role”. So a regular user with a valid API key can trigger scheduling.

Jumping into lib/OpenQA/WebAPI/Controller/API/V1/Iso.pm we can take a look of what’s going on. The create function first validates the mandatory parameters (DISTRI, VERSION, FLAVOR, ARCH), then creates a record in the ScheduledProducts table via create_with_event, and only then proceeds to schedule. The record exists in the database before any job is created.

At this point the path splits in two. If the caller passes async=1 the controller enqueues a Minion background job and returns immediately:

return $self->render(json => $scheduled_product->enqueue_minion_job(\%params)) if $async;

The Minion task lives in lib/OpenQA/Task/Iso/Schedule.pm. Its main reason to exist is that it can perform an HTTP fetch for SCENARIO_DEFINITIONS_YAML_FILE before scheduling — something that would block the web process if done synchronously. After the optional download it calls the same schedule_iso as the synchronous path, so both paths converge there.

In the default (synchronous) case the controller calls directly:

my $scheduled_jobs = $scheduled_product->schedule_iso(\%params, undef);

Actually if we grep directly the function name create_from_settings we see where the main operations eventually take place

❯ grep create_from_settings -rn lib
lib/OpenQA/Schema/Result/ScheduledProducts.pm:266:            my $job = $jobs_resultset->create_from_settings($settings, $self->id);
lib/OpenQA/Schema/ResultSet/Jobs.pm:135:sub create_from_settings ($self, $settings, $scheduled_product_id = undef) {
lib/OpenQA/WebAPI/Controller/API/V1/Job.pm:328:    my $job =
$self->schema->resultset('Jobs')->create_from_settings($job_settings);

So back to schedule_iso: before doing any real work it updates the ScheduledProduct status to SCHEDULING and wraps everything in a try/catch, calling set_done at the end to transition to SCHEDULED (or CANCELLED if meanwhile a cancellation came in). Then it delegates to the private _schedule_iso.

This is an intersection point where everything starts to get complicated. The reason is that there are various approaches to configure a job, depending on your workflow. If _OBSOLETE or _DEPRIORITIZEBUILD flags are set, _schedule_iso will also cancel or deprioritize existing jobs for the same DISTRI/VERSION/FLAVOR/ARCH before creating the new ones — that is what makes re-triggering a build work cleanly in practice.

The following they will prepare the settings according to the workflow:

  • _schedule_from_yaml — used when SCENARIO_DEFINITIONS_YAML (inline) or SCENARIO_DEFINITIONS_YAML_FILE is provided
  • _generate_jobs — the classic path that reads job templates from the database

And later _create_jobs_in_database which will store the jobs in the database (that’s why all the operations are in the Results/ dir). Looking at the function signature we see what we expect: $jobs, \@failed_job_info, $skip_chained_deps, $include_children, \@successful_job_ids, \@minion_ids

Finally it will reach create_from_settings which for each job calls its own create.

After all jobs are inserted, _schedule_iso pokes the scheduler daemon:

OpenQA::Scheduler::Client->singleton->wakeup;

The jobs are in the database at this point, but nothing runs until the scheduler daemon is notified. That wakeup call is the bridge between “ISO post” and actual test execution.

The architecture

What the pattern is called

The broad term for what happens when code accumulates concerns in one place over time is the Big Ball of Mud 1 — a system with no clear architecture that grew organically by adding code wherever it was convenient. The specific manifestation here is the God Object 2 anti-pattern: ScheduledProducts.pm is ~960 lines and owns too many responsibilities at once.

Where it is getting messy

A DB result class that runs the whole show

lib/OpenQA/Schema/Result/ScheduledProducts.pm lives in the Schema/Result/ namespace, which by DBIx::Class convention is supposed to be the database model layer. But it does:

  • YAML parsing and validation
  • Settings inheritance and merging (product → machine → test suite → template)
  • Dependency graph sorting and cycle detection
  • Asset registration
  • GitHub status reporting
  • Minion job enqueueing
  • Event emission (AMQP)

The controller (Iso.pm) is appropriately thin. The problem is everything was pushed down into the model class as the feature grew. There is no service layer in between.

God Object diagram: ScheduledProducts.pm and its nine responsibilities

Settings as an untyped, overloaded hash

The flat %settings hash is passed through the entire call chain carrying two completely different kinds of data mixed together:

  • Job configuration (what the test actually needs: ISO, DESKTOP, ARCH, …)
  • Scheduling directives consumed and deleted before jobs are created: _OBSOLETE, _DEPRIORITIZEBUILD, _SKIP_CHAINED_DEPS, _GROUP, _PRIORITY, __-prefixed product metadata

_schedule_iso opens by delete-ing the directive keys out of $args. That mutation is the only thing separating “instructions for the scheduler” from “settings for the job”. If a caller forgets a key or misspells the prefix, the directive silently becomes a job setting. This is sometimes called stringly-typed 3 code — a valid hash key is the only contract.

Duplicated settings-merging logic

_generate_jobs and _schedule_from_yaml both resolve the effective settings for a job (product + machine + test suite + template + caller args), handle WORKER_CLASS merging separately from other keys, and then call the same filtering helpers. The merging logic at the top is written twice. A bug in how precedence works between product and machine settings would need to be fixed in both places.

Output parameters instead of return values

_create_jobs_in_database takes seven parameters, four of which are output parameters passed as mutable array references:

sub _create_jobs_in_database ($self, $jobs, $failed_job_info, $skip_chained_deps,
    $include_children, $successful_job_ids, $minion_ids = undef)

The caller (_schedule_iso) sets up empty arrays, passes refs down, then reads the results back out after the call. This is a C-style pattern — you cannot look at the call site and know what comes back without reading the callee. As more edge cases were added (the deadlock retry handler also modifies those same arrays), it becomes easy to miss that a reset is needed.

OpenQA::App->singleton inside a model class

_schedule_iso and _create_jobs_in_database both call OpenQA::App->singleton->gru to enqueue Minion jobs. _schedule_from_yaml calls $app->validate_yaml. A Schema::Result class that reaches for the global application singleton is hard to test in isolation and creates an invisible dependency on the full app stack being initialized.

GitHub CI reporting on the scheduling object

report_status_to_github is conceptually unrelated to scheduling — it is about post-hoc CI status. It ended up on ScheduledProducts because that is the object that has the job list and the status, but it means the scheduling model now also knows about HTTP clients, GitHub API tokens, and webhook payloads. It gets called from set_done, so every scheduling path implicitly triggers a GitHub API call.

What is fine

  • The public/private split (schedule_iso / _schedule_iso) with the status machine (SCHEDULING → SCHEDULED, try/catch, set_done) is clean and correct. The state transitions are honest about failure modes.
  • The dependency resolution itself (_sort_dep, _check_for_cycle, _create_dependencies_for_job) is well-contained.
  • The two scheduling paths converging at _create_jobs_in_database is architecturally sound — the problem is the duplicated merging code above that point.
  • The savepoint pattern around each individual job creation (svp_begin/svp_release/svp_rollback) handles partial failures correctly.

What would be cleaner

The structural fix is a service layer 4 between the controller and the model:

Controller (Iso.pm)
    → SchedulingService (e.g. OpenQA::Scheduler::IsoScheduler)
        → ScheduledProducts  (pure DB model: state machine only)
        → JobFactory         (settings resolution + create_from_settings)
        → DependencyResolver (graph sort, cycle detection)

Proposed service layer: Controller → SchedulingService → ScheduledProducts, JobFactory, DependencyResolver, GitHubReporter, EventEmitter

Concretely:

  • Separate scheduling directives from job settings at the API boundary, not deep inside _schedule_iso via delete. A typed params struct (even just a validated hashref with known keys) would make the contract explicit.
  • _create_jobs_in_database should return a result hashref, not accumulate into caller-owned arrays.
  • GitHub integration belongs in an event handler that reacts to the openqa_job_create event, not baked into set_done.
  • The settings-merging logic shared between _generate_jobs and _schedule_from_yaml should live in one place — there is already OpenQA::JobSettings, which is the right home.

References


  1. Foote, B. & Yoder, J. (1997). Big Ball of Mud. Pattern Languages of Program Design 4. ↩︎

  2. Riel, A. (1996). Object-Oriented Design Heuristics. Addison-Wesley. — The God Class heuristic (heuristic 3.1) states that a class should have a single, well-defined responsibility. ↩︎

  3. Stringly-typed: a portmanteau of “string” and “strongly-typed”, used to describe designs that use strings where a more specific type would be safer. Popularised on Coding Horror (2012). ↩︎

  4. Fowler, M. (2002). Service Layer. Patterns of Enterprise Application Architecture. Addison-Wesley. ↩︎