Thread Pool for Host Work​

jai::thread_pool is a persistent pool of worker threads that any host can use for its own work: asset decoding, pathfinding batches, a particle pass, blocking I/O. Each engine runs its parallel_for and parallel_transform regions on a pool of its own; a pool the host makes is a separate object, and an engine never touches it.

#include <jaiscript/thread_pool.hpp>

The header declares the class and nothing more (it includes <functional>, <memory>, <vector> and <exception>); the threads, locks and queues compile once, in the library. A pool is a plain object with no global state: a host may own as many as it likes, and each one owns its threads.

Construction and worker count​

thread_pool(n) starts n workers at once and keeps them for the pool's lifetime. n = 0, the default, means thread_pool::default_worker_count(): one less than the hardware threads, so the submitting thread has a core of its own, and never less than one. worker_count() reports the number. An engine's own pool is sized from thread_pool::performance_core_count() instead and built with thread_pool::fast_threads, which opts its workers out of the system's power throttling (Parallel worker count). A pool is neither copyable nor movable; keep it in the object that owns the work, or behind a std::unique_ptr.

The quick tour: submit_to hands each worker one stripe of a sum, and wait_idle joins them.

jai::thread_pool pool(4);
const size_t workers = pool.worker_count();
std::vector<int64_t> partial(workers);
for (size_t w = 0; w < workers; ++w) {
    pool.submit_to(w, [&partial, w, workers] {
        for (int64_t i = int64_t(w); i < 1000; i += int64_t(workers)) { partial[w] += i; }
    });
}
pool.wait_idle();
int64_t sum = 0;
for (int64_t p : partial) { sum += p; }
std::cout << sum << "\n";

499500

Submitting work​

A task is a thread_pool::task_type, a std::function<void()>. There are two queues to put it on:

• submit(task): the shared queue. The first free worker runs it.
• submit_to(worker, task): worker worker's own queue (0 to worker_count() - 1; another index throws std::out_of_range). Only that worker's thread runs it, in submission order, interleaved with the shared tasks it also picks up. Thread-affine state (a per-worker scratch buffer, a context that must stay on one thread) belongs here.

wait_idle() blocks until every task submitted so far, on either queue, has finished, including continuations that run on the pool; a finished task's captures are destroyed by then. It does not wait for posted work (below), which runs only when the host drains it.

Continuations: then and then_on​

submit(task, then, where) runs then exactly once, after task returns or throws. submit_batch(tasks, then, where) puts a whole std::vector of tasks on the shared queue under one lock and runs then once, after the last of them returns or throws, whichever worker that is; an empty batch continues at once. A batch whose queueing throws (memory runs out) still runs the tasks it queued, drops then and passes the exception to the caller. where is a thread_pool::then_on:

• then_on::pool (the default): then runs on the worker that finished the work, right after it.
• then_on::posted: then is posted, and runs on the thread that calls run_posted().

A task that throws still continues: its exception goes to the on_exception handler and then runs next, so a continuation can always close what its task opened.

jai::thread_pool pool(1);
pool.on_exception([](std::exception_ptr error) {
    try {
        std::rethrow_exception(error);
    } catch (const std::exception& e) {
        std::cout << "on_exception: " << e.what() << "\n";
    }
});
pool.submit([] { throw std::runtime_error("decode failed"); },
            [] { std::cout << "then runs anyway\n"; });
pool.wait_idle();

on_exception: decode failed
then runs anyway

The posted queue: post and run_posted​

Work that has to happen on the host's own thread (uploading a decoded texture, touching the scene, anything that is not thread-safe) goes through the posted queue. post(task) queues a task from any thread, a worker included; run_posted() runs the queue on the calling thread and returns how many tasks ran. A frame loop calls it once a frame.

• Tasks run in the order they were posted, and a task posted while the drain runs runs in the same drain.
• A posted task may call run_posted() itself; the nested drain continues the same queue in posting order and runs nothing twice.
• A std::exception a posted task throws goes to on_exception, and the drain goes on. Any other exception leaves run_posted(), with the tasks it had not run yet queued ahead of later posts.
• The posted queue has its own lock, so posting never contends with the workers. Drain it from one thread.

The shape a game loads with: decode on the workers, continue posted, and let the frame loop pick the result up. ready needs no atomic, since only the draining thread touches it.

jai::thread_pool pool;
std::vector<std::string> levels(3);
std::vector<jai::thread_pool::task_type> loads;
for (size_t i = 0; i < levels.size(); ++i) {
    loads.push_back([&levels, i] { levels[i] = "level_" + std::to_string(i); });
}
bool ready = false;
pool.submit_batch(std::move(loads), [&] { ready = true; }, jai::thread_pool::then_on::posted);

while (!ready) {
    // one frame: input, update, render, then the work the pool handed back
    pool.run_posted();
}
std::cout << levels[0] << " " << levels[1] << " " << levels[2] << "\n";

level_0 level_1 level_2

Exceptions: on_exception​

on_exception(handler) installs a std::function<void(std::exception_ptr)> that receives every exception a task lets escape: from a pool task, from a continuation, and the std::exceptions of posted tasks. It runs on the thread the task ran on. With no handler the exception is dropped; a handler that throws is dropped the same way. Either way the worker lives on and takes the next task. The handler can be replaced at any time, from any thread.

Spinning: spin_budget_ns​

A worker that runs out of work can spin briefly before it parks on a condition variable, so a dense run of small batches (fork-join work several times a frame) skips the kernel wake. It spins only when the pool's last two batches arrived within the budget; a pool fed once a frame or less often parks at once and burns nothing in between. wait_idle() spins on the same budget before it blocks.

spin_budget_ns(nanoseconds) sets the budget and spin_budget_ns() reads it; the default is 1 ms (1,000,000). 0 parks at once, the right setting for a pool whose work arrives irregularly: background loading, downloads, a database queue.

jai::thread_pool loader(2);
loader.spin_budget_ns(0);   // loads arrive a few times a second: never spin for the next one

Destruction​

Destroying a pool drops every queued task that has not started, on both queues, lets the tasks already running finish, and joins every worker before the destructor returns. A dropped task's continuation never runs, and posted tasks still waiting for a drain are destroyed unrun. So whatever a task references must outlive the pool: declare the pool after that state, and C++ destroys the pool first. A task still running while the pool is destroyed may submit or post; nothing it adds runs.

Thread safety​

• submit, submit_to, submit_batch, post, on_exception, spin_budget_ns and wait_idle may be called from any thread, concurrently, including from inside a task: a task can submit follow-up work.
• A task must not call wait_idle() on its own pool (it waits for itself) and must not destroy its own pool.
• The pool synchronizes its own queues, nothing else. Data two tasks share needs whatever synchronization it would need between any two threads; a submit_batch continuation sees every write its batch made.

A task may call into an engine only while no other thread uses that engine: engines share nothing and take no locks. Script work runs on many threads through the language's own parallel loops, on the engine's pool (see Parallel Loops).