jai_spider: a Web Application Host​

jai_spider serves a JaiScript program as a web application. It is a host like any other on these pages, built from the same engine calls: an HTTP/1.1 server, a request handed to the handler the program spins, the program's imports reached before every request, and SQLite bound as a class. Because the imports are reached per request, a saved edit applies to the next request with no restart, and an edit that fails its check is reported while the app keeps serving the code it had. The same program reaches other servers with the spider:: client, so one web serves and fetches.

The host lives in tools/jai_spider: the C++ (sockets, HTTP, SQLite) and the script library apps import (lib/spider.jai and lib/session.jai). Everything a script names lives under spider::; the one class the library itself provides is spider::Response, which a handler answers with and a client call returns.

Running an app​

The jai_spider CMake target builds bin/jai_spider beside the runner (option JAISCRIPT_BUILD_SPIDER, on by default). Point it at the app's entry file:

jai_spider tools/jai_spider/examples/guestbook/app.jai
/\(o.o)/\  jai_spider 0.5.0-beta
jai_spider: spinning tools/jai_spider/examples/guestbook/app.jai on http://127.0.0.1:8080/ (files from .../guestbook/public)
GET / 200 1.21 ms
GET /style.css 200 0.31 ms (file)
POST /sign 303 2.40 ms
jai_spider: reloaded views.jai
GET / 200 1.87 ms

Each request logs one line: method, path, status and time (a control byte or a byte past ASCII in the path is written as \xNN, so a request cannot write terminal escapes). Ctrl+C stops the server; the engine is destroyed with it, which closes every database.

A first app​

An app imports spider.jai, builds its routes and hands them to spider::spin. spider::serve is the same call by its plain name.

import "spider.jai";

Database db = open_database("notes.db");        // relative to the app's folder
db.execute("CREATE TABLE IF NOT EXISTS notes (id INTEGER PRIMARY KEY, text TEXT NOT NULL)");

int requests;                                   // no initializer: the count survives every reload

spider::Router routes = spider::Router();
routes.get("/", (spider::Request req) => {
    requests += 1;
    return spider::html(`<h1>${requests} requests</h1>`);
});
routes.get("/notes", (spider::Request req) => db.query("SELECT id, text FROM notes"));   // a value answers JSON
routes.post("/notes", (spider::Request req) => {
    db.execute("INSERT INTO notes (text) VALUES (?)", req.form()["text"]);
    return spider::redirect("/notes");
});
routes.get("/notes/:id", (spider::Request req) => {
    var note = db.query_one("SELECT id, text FROM notes WHERE id = ?", req.param("id"));
    return note == null ? spider::not_found() : spider::json(note);
});

spider::spin(routes);

spider::spin(handler[, options]) takes a spider::Router (add its routes first, then spin it) or any function of a spider::Request. A handler answers a spider::Response, a string (an HTML page), null (204 No Content) or any other value, which is sent as JSON. The app's top level runs again on every reload, so the spin that ends it hands the server the new routes. Until an app has spun, every request is a 500 naming the missing call. The options are budget (seconds a request may run, over --budget) and max_body (bytes, over --max-body), from the next request on; any other key raises.

The checker sees the whole program before anything runs, so a misspelled field or a string passed where an int is declared fails the load with a listing, exactly as in any other host.

Request and Response​

spider::Request is a script class in lib/spider.jai, so its source is the reference and an app can read or extend it.

spider::Response is the library's class (its members): status, headers (a map<string, string>), cookies (the Set-Cookie lines) and body, with header(name, value), set_cookie and clear_cookie for building one. lib/spider.jai adds the helpers that build the usual answers:

The host registers the web:: library (web::html_escape, web::url_encode, web::form_encode, web::parse_query, web::build_query, web::parse_url and the rest) and adds what only a web host needs: parse_cookies, parse_multipart, http_date and content_type_for; a session id is crypto::random_hex(32), from the operating system's secure random source (crypto::). Pages escape what visitors wrote with web::html_escape; the guestbook's views.jai writes its templates as functions returning backtick strings, with every visitor value passed through it.

Routing​

spider::Router has get, post, put, patch, delete, any and add(method, pattern, handler). Routes match in the order they were added. /posts/:id captures one path segment as req.params["id"], /files/*rest captures the rest of the path. A HEAD request takes the GET route and is answered without its body; a path that only other methods match answers 405 with an Allow header; no match is a 404. before(filter) runs a filter ahead of every route: null lets the request through, anything else is the answer, which is where a sign-in check goes.

Reaching other servers​

jai_spider installs the spider:: client and grants the network permission, so a handler can fetch from another service and answer with what came back. spider::catch(url) fetches (its plain name is spider::get), spider::sling(content, url) sends (its plain name is spider::post(url, content)), and both answer a spider::Response, the same class a handler returns:

routes.get("/weather/:city", (spider::Request req) => {
    spider::Response upstream = spider::catch("https://weather.example/v1/now?city=" + web::url_encode(req.param("city")), {"timeout": 5});
    if (!upstream.ok()) { return spider::json({"error": "the weather service answered " + to_string(upstream.status)}, 502); }
    return upstream;                              // the upstream answer, passed through as it came
});

The host writes its own framing: an answer passed through loses its X-Sendfile header and the headers that describe one connection (Connection, Keep-Alive, Proxy-Connection, TE, Trailer, Transfer-Encoding, Upgrade, Content-Length). Its cookies pass through as they came, so a handler that relays another site's answer clears cookies when they are not its own to set. A call blocks the one script thread until the answer arrives or its timeout passes, and the requests behind it wait too, so keep upstream timeouts short. On Windows the client goes through WinHTTP (the system's TLS and certificate store); elsewhere through sockets, with HTTPS when the build found OpenSSL (the HTTP transport section has the table).

SQLite, injection-proof by construction​

open_database(path) opens or creates a database file (relative to the app's folder; ":memory:" for a scratch one) and returns a Database, a C++ class bound with dynamic_binder. Copies of one share its connection, which closes with the last of them.

The SQL text of every statement is one of the program's own constants, and values only ever bind as parameters. execute, query, query_one and value take the SQL text first and the values after it, and the text is a constant argument: a string literal, a const holding one, a ternary choosing between them, or a + of them. Text built from data does not load: argument 1 of 'query' is not a constant: SQL text must be a constant; pass values as parameters (?). The guarantee does not rest on the checker: the host gathers every constant text the program passes to these methods as each unit loads, and a text that is not one of them is refused at run time with SQL text must be a constant; pass values as parameters (?) before SQLite sees it, however it arrived (a var, a [[warn_mode]] region, reflect::invoke). There is no escape or quote function, and nothing that splices a value into a statement.

const string FIND_POST = "SELECT * FROM posts WHERE id = ?";
var post = db.query_one(FIND_POST, req.param("id"));
int changed = db.execute("UPDATE posts SET title = ? WHERE id = ?", title, id);
array<map<string, var>> rows = db.query("SELECT id, title FROM posts WHERE author = ?", author);
var by_name = db.query_one("SELECT * FROM posts WHERE id = :id", {"id": id});   // one map binds by name
int count = db.value("SELECT COUNT(*) FROM posts");
var picked = db.query("SELECT * FROM posts WHERE id IN (?)", [3, 5, 8]);     // an array fills IN (?)
var sorted = db.query("SELECT * FROM posts ORDER BY {} LIMIT 20",           // {} takes one constant choice
                      sql_choose(req.query.get("sort", "created"), ["created", "title", "created DESC"]));
db.transaction(() => {
    db.execute("INSERT INTO posts (title) VALUES (?)", title);
    db.execute("INSERT INTO audit (what) VALUES (?)", "post");
});

Each ? takes the next value. An array value expands its ? to one placeholder per element, so IN (?) takes a list of any length (an empty one is IN (), which matches nothing). What a parameter cannot be, a column to sort by or a direction, goes in a {} slot, filled by sql_choose(value, choices): it answers the choice equal to value, and choices must be constants too (an array literal of strings, or a const of one), so the fragment it puts in the text is the program's own; a value that is none of them raises sql_choose: the value is not one of: created, title, created DESC. A {} takes nothing else, and a choice cannot fill a ?. One map value binds :name, @name and $name instead.

Values bind from int, float, bool, string and null; columns come back as int, float, string (TEXT, and a BLOB as its bytes) or null. Each statement is prepared once and kept until the database closes. A constant text of several statements given to execute without values runs them all, which suits a schema. transaction(fn) commits when fn returns and rolls back when it raises, then raises again; inside another transaction it is a savepoint. begin, commit, rollback, last_insert_id, changes, in_transaction, is_open and close are there too. Every SQLite failure is a catchable error carrying SQLite's message: SQLite error: no such table: posts. Files open in WAL mode.

Sessions​

lib/session.jai keeps sessions in a table of the app's database (spider_sessions). The cookie carries only a 256-bit random id; the data stays on the server as JSON.

import "session.jai";

spider::SessionStore sessions = spider::SessionStore(db);

routes.post("/sign-in", (spider::Request req) => {
    spider::Session s = sessions.load(req);      // the visitor's session, or a new empty one
    sessions.rotate(s);                          // a fresh id once the visitor signs in
    s.set("user", req.form()["name"]);
    s.set("flash", "Welcome back.");
    spider::Response r = spider::redirect("/");
    sessions.save(s, r);                         // the row and the cookie, when the session changed
    return r;
});

spider::Session has get(key, fallback), has, set, erase and take (read once, for a message shown after a redirect). spider::SessionStore has load, save, rotate, destroy and purge_expired; cookie_name, max_age and secure_cookie are fields.

Hot reload​

Before each request the host executes import("app.jai") with the engine's import check on every reach (Reloading from a running program). An unchanged program costs one stamp check per file. When a file has changed, the module is parsed and checked together with everything it imports, then runs as its own unit:

• its functions and classes replace the old ones in place, and live instances migrate (Hot Reload);
• its top-level statements run again, so a spider::Router built with an initializer is rebuilt from the new code and the closing spider::spin hands it to the server;
• a variable declared without an initializer (int requests;, map<string, var> cache;) keeps the value it holds.

An edit that fails to parse or check is logged once with its listing, and the app answers every request with the code it had until the file is saved again:

jai_spider: the edit was rejected; serving the code the app had:
static check failed:
views.jai:7:25: error: cannot initialize 'n' declared 'int' with 'string'
    string footer() { int n = "three"; return "..."; }
                            ^
GET / 200 0.84 ms
jai_spider: reloaded views.jai
GET / 200 1.66 ms

The host imports its library from the jai_spider_lib/ folder the build puts beside the executable. Under --dev it imports the source checkout's lib/ instead, and those files reload the same way: an edit to spider.jai applies to the next request.

Static files​

For GET and HEAD, a path that names a file under the static root is served before the app sees the request. The root is --root, or the app's public/ folder when it exists; a folder serves its index.html. A file is sent with a Content-Type from its extension, a weak ETag, Last-Modified and Cache-Control: no-cache, so a browser revalidates each time and sees an edited file on its next load (304 Not Modified otherwise). Files stream from disk in slices. A file or folder whose real name starts with a dot is never served, whatever name reaches it (a Windows 8.3 short name such as ENV~1 included), and neither is anything outside the root. A request path with a . or .. segment, or a segment that decodes to one holding /, \ or NUL, is answered 400 before the app sees it, so a *rest capture handed to spider::file never climbs out of the folder it names.

Errors, limits and the threading model​

One thread runs the network loop and the script. The loop keeps every connection moving (reading, writing, streaming files) and hands complete requests to the script one at a time, so a slow handler delays the requests behind it. HTTP/1.1 and 1.0 are served with keep-alive and pipelining, bodies by Content-Length or chunked, and Expect: 100-continue.

A connection holds one response at a time: the next request is read once the last response has left, so a client that never reads cannot pile responses up. The server speaks no TLS; a deployment puts a reverse proxy that terminates HTTPS (Caddy, nginx) in front and sets secure_cookie on the session store.

Options​

Every file function of the runner (read_file, write_file, list_dir and the rest) and the full standard library are registered too. tools/jai_spider/examples/guestbook is a complete app: templates, a form, a session, SQLite storage, a JSON API and a stylesheet. tools/jai_spider/tests/spider_test.py starts the host on a free port and drives every behavior on this page over HTTP, on both backends, then points the runner's spider:: client at it through each transport the build has.