Native module ABI

Lynxer's standard libraries are shared libraries (.so on Linux, .dylib on macOS, and PE DLLs on Windows) loaded at runtime through the host loader. This document is the contract a native module must implement. It applies to the bundled stdlib/*.so modules exactly as it applies to third-party modules.

A module may be written in C++ (stdlib/<name>.cpp) or in Rust. The Rust backends — compress, crypto, encoding, game, graphics, image, ini, json, lua, network, raylib, server, sound, sqldb, toml, tui, uuid, watch, xml and yaml — live under rust/ and are all cdylibs that export lynxer_module_init_v1, their ops, and (for game) lynxer_module_attach_v1 directly; there is no C++ shim. The lynxer_abi crate provides the shared FFI plumbing (packed-argument view, panic guards, string result buffer, host API, and the registration helper). The generated raylib backend registers its ops by hand instead of through lynxer_abi.

The workspace has one more member, rust/ffi, which is different in kind: a staticlib that is linked into the interpreter, not a cdylib module. It implements the shared native-call engine (see The native call engine) and is never installed into stdlib/; because the interpreter links it, cargo is a required build tool.

Entry point

Every module exports one C symbol:

int lynxer_module_init_v1(
    int (*register_function)(const char *name, const char *symbol,
                             const char *signature),
    int (*register_constant)(const char *name, int64_t value),
    int (*register_type)(const char *name, const char *layout)
);

Return 0 after registering everything. A non-zero return, an invalid identifier, a duplicate name, or a symbol that dlsym cannot resolve rejects the whole module: the load fails with a native module lifecycle failure, and the interpreter reports the short reason it recorded — registered symbol not found for an unresolved symbol, duplicate native registration for a repeated name — not the offending name.

register_function(name, symbol, signature) — name is the Lynxer-facing name under the module namespace; symbol is the exported C name; signature uses the grammar below.
register_constant(name, value) — exposed as a read-only integer field on the module namespace.
register_type(name, layout) — exposed as a string field holding a native memory layout (used with the memory* builtins).

On Windows, every registered symbol must also be exported from the DLL, not just lynxer_module_init_v1. The bundled MinGW/CLANGARM64 module rule passes -Wl,--export-all-symbols; an MSVC/clang-cl module must put LYNXER_MODULE_EXPORT on each op (or use a .def file). The examples/native_signatures.cpp module does this, and its MSVC CI job checks both the entry point and an op export.

Loading happens in lynxer/ast.cpp (ImportStatement::execute), where dlopen is called with RTLD_NOW | RTLD_LOCAL and the initializer is invoked with the three callbacks. Modules are kept loaded for the lifetime of their namespace so registered function pointers cannot dangle.

Importing

Import the library from setup() with any of the import forms:

global setup(){
    importAs("math.so", "nativeMath");
}

global main(){
    println(global.nativeMath.abs(-3));
}

A path ending in .so is treated as a native module; anything else is compiled as Lynxer source. The namespace is the import alias, or the file stem when no alias is given.

The native call engine

Every native call — an imported module's function and an explicit ffiCall alike — is performed by rust/ffi, a Rust staticlib built on libffi and linked into the interpreter. It is not a stdlib module: it exports no lynxer_module_init_v1, is never placed in stdlib/, and exists only inside the interpreter binary.

cargo is therefore a required build tool, for the engine and for every Rust backend alike: make buildLynxer builds them before linking, and a missing toolchain or a compile error fails the build rather than dropping a module. The C ABI is declared in lynxer/ffi_abi.h:

int         lynxer_ffi_call(void* address, const char* signature,
                            const LynxerFfiArg* args, int64_t nargs,
                            LynxerFfiResult* out);
const char* lynxer_ffi_last_error(void);

Each argument carries a type tag so the engine can apply the strict/promoting rules below, and a failed call returns non-zero with a message the interpreter wraps in a located error.

Signature grammar

cdecl:<return>(<arg>,<arg>,...)

Type tokens are int64 (Lynxer int), double/float64 (Lynxer float), cstring (Lynxer str) and bytes (Lynxer bytes). The value token is additionally accepted only with the v2: prefix. Argument lists may be empty.

Before looking a signature up, the dispatcher normalizes the tokens: every integer width is rewritten to int64 (int8, int16, int32, uint8, uint16, uint32, uint64 and uintptr), and double is rewritten to float64. So cdecl:int32(int32) and cdecl:int64(int64) resolve to the same shipped shape. The supported table is the normalized set; the tokens above are accepted spellings, not distinct shapes.

cdecl: is optional; the bare <return>(<args>) form is accepted too. The packed form is selected when the argument list is exactly the single token ... (types.size() == 1 && types[0] == "..."). For typed aggregate signatures, either cdecl:v2: or the bare v2: prefix is accepted.

Supported signature shapes

Any signature the grammar can describe is callable. The engine parses cdecl:<ret>(<args>), normalizes the tokens as above, and builds a libffi call description from them; there is no fixed table of shapes. Legacy int64, float64, cstring and bytes parameters may appear in any order and any number. The return type may be int64, float64, cstring, bytes or void (a void call yields 0); v2 signatures may also use value.

A bytes parameter is passed as two C arguments — const uint8_t* data, int64_t length — and a bytes return is a const uint8_t* to a buffer laid out as [int64 little-endian length][payload], which the engine reads and the interpreter copies. See Data conventions.

A token outside the grammar raises unsupported native signature '<sig>'; a fixed signature whose argument count differs from the call raises native call argument count does not match signature '<sig>'; and an argument that does not match its parameter type raises native call expected an integer argument, native call expected a string argument or numeric value required.

Numeric arguments are read leniently where the parameter is float64 — int, float and uint64 values are accepted — while integer and string parameters are strict and reject a mismatched value.

Both the interpreter's own ffiCall and every native-module invocation use this one engine, so a module call and an ffiCall follow identical rules.

Versioned typed aggregate values

The additive v2 signature cdecl:v2:value(value) passes common values as a recursively typed LynxerFfiValue tree rather than requiring JSON strings or integer handles. It supports none, signed and unsigned integers, floats, booleans, strings, chars, bytes, lists, tuples, named records (including structs and classes), and enum variants with named payloads. Aggregate children can themselves be any supported value. Objects that are opaque interpreter handles, codeblocks, and sentinels are not representable and are rejected.

The argument is a pointer to a borrowed value tree. The native function must not retain any pointer into that tree after returning. A returned value tree must remain valid until the interpreter has copied it, which is before the next native call on that thread. Record fields carry their names and the field's declared type and constant flag, and the record's type/display names and kind. Text and byte strings use explicit byte lengths and may contain NUL bytes.

The version prefix is intentional: existing cdecl: and packed signatures retain their original ABI and type checking. C++ modules include lynxer/ffi_abi.h; Rust modules use the matching LynxerFfiValue types from the lynxer_abi crate. A C++ implementation can echo a nested input tree:

extern "C" const LynxerFfiValue* echo_value(const LynxerFfiValue* value) {
    return value;
}

// Register with: cdecl:v2:value(value)

A Rust module can register the same direct pointer prototype:

use lynxer_abi::{lynxer_module, LynxerFfiValue};

const OPS: &[(&str, &str, &str)] =
    &[("echo", "echo_value", "cdecl:v2:value(value)")];
lynxer_module!(OPS);

#[no_mangle]
pub unsafe extern "C" fn echo_value(
    value: *const LynxerFfiValue,
) -> *const LynxerFfiValue {
    value
}

The executable nested round-trip fixture is lynxer/examples/native_aggregate.lynx.

Packed arguments (`...`)

An API with long or variadic argument lists can use the wildcard parameter token ... instead of a fixed shape:

cdecl:int64(...)
cdecl:float64(...)
cdecl:cstring(...)

These are the legacy packed shapes; the return type selects the prototype. This is the form the lynxer_abi macros generate. Version 2 may additionally use The typed v2 shape is currently supported in fixed signatures; it does not change the legacy packed argument convention.

A Rust cdylib must use a packed signature unless it uses the direct v2 typed-value prototype above. The export_int!, export_float! and export_string! macros (and their _buffers! counterparts) expand to the packed prototype below, so registering such an op with a fixed shape is a hard error that is not caught at build time: the interpreter calls the symbol through the fixed shape's C prototype (int64_t(*)(int64_t), const char*(*)(const char*), …), which does not match the real function, and the process segfaults on the first call to that op. dlopen and registration both succeed, so the mistake only shows up at run time. (The one benign exception is a shape that is not in the fixed-shape table at all, which fails fast with unsupported native signature '<sig>'.) Because the packed form accepts any number of arguments, a wrong signature is never an argument-count error — it is a call through the wrong ABI.

The function receives every argument as four scalars:

int64_t function(const double* nums, int64_t num_count,
                 const char* const* strs, int64_t str_count);
double  function(const double* nums, int64_t num_count,
                 const char* const* strs, int64_t str_count);
const char* function(const double* nums, int64_t num_count,
                     const char* const* strs, int64_t str_count);

Numbers (Lynxer int, float, and bool as 0/1) arrive in nums in their original order; strings arrive in strs in theirs. Either pointer is null when its count is zero, and at most 256 arguments in total are accepted across both lists (MAX_PACKED_ARGS in lynxer/rust/ffi/src/lib.rs); a call with more raises a located SourceError. Any other argument type raises a located SourceError. lynxer/stdlib/lynxer_native_abi.h documents the convention for module authors.

The buffered packed form

A packed op that also needs byte buffers registers cdecl:<ret>(...,bytes). The callee receives three more scalars:

const char* function(const double* nums, int64_t num_count,
                     const char* const* strs, int64_t str_count,
                     const uint8_t* const* bufs, const int64_t* lens,
                     int64_t buf_count);

Each bufs[i] is lens[i] bytes. The plain ... form is unchanged, so a module that needs no buffers keeps it. In lynxer_abi, register such an op with one of export_int_buffers! / export_float_buffers! / export_string_buffers! / export_bytes_buffers! and read the buffers with args.bytes(i); export_bytes_buffers! returns a bytes value via store_result_bytes.

Arguments are indexed per type, not by position. nums and strs are separate lists, so the n-th string argument is strs[n] and the n-th numeric argument is nums[n], regardless of how they were interleaved at the call site. The lynxer_abi Args view enforces this: args.int(i) / args.float(i) read nums[i], args.string(i) reads strs[i], and args.bytes(i) reads bufs[i]. For save(handle, path, quality) the correct reads are args.int(0) for the handle, args.string(0) for the path and args.int(1) for the quality — not args.int(0), args.string(1), args.int(2). Reading past the end of either list yields 0 / "" instead of failing, so a wrong index silently produces a zero value rather than an error.

Because that failure is silent, make test runs lynxer/scripts/check_module_contracts.py, which reads each stdlib/<name>.lynx wrapper and fails if a packed op reads an argument that the wrapper never passes — or if a wrapper calls an op the backend does not register. Write the reads the way the wrapper passes them, and the check stays quiet.

Calling back into Lynxer

A module may additionally export

int lynxer_module_attach_v1(const LynxerHostApi *host);

and the interpreter calls it (via dlsym) right after lynxer_module_init_v1 succeeds. Returning non-zero rejects the module:

typedef struct LynxerHostApi {
    int version; // 1
    void *context;
    int (*invoke)(void *context, const char *name, int has_arg, double arg);
    int (*interrupted)(void *context);
    const char *(*program_args)(void *context);
    int (*blocking)(void *context, void (*body)(void *user), void *user);
    int (*request_exit)(void *context, int64_t code);
    int (*invoke_threadsafe)(void *context, const char *name, int has_arg,
                             double arg);
} LynxerHostApi;

invoke runs the Lynxer function name with no argument or one numeric argument, and returns 0 on success.
invoke_threadsafe does the same after acquiring the interpreter lock. Use it only from native worker threads; callbacks already on the interpreter thread use invoke.
interrupted returns non-zero once the process has received SIGINT.
program_args returns a thread-local JSON array of process arguments.
blocking releases the interpreter lock while running a native blocking operation, then reacquires it.
request_exit requests orderly shutdown with the supplied process status.

The context pointer is opaque to the module. invoke resolves against the top-level program, so frame callbacks registered by a module imported from a source wrapper still reach the program's own functions. This entry point is optional; modules that do not export it behave as before.

Modules that need invoke_threadsafe use the separate lynxer_module_attach_v2(const LynxerHostApiV2 *) symbol. The v2 struct repeats the v1 fields and appends the thread-safe callback, and the interpreter negotiates it independently so neither an old module nor an old host reads beyond the struct size it supports.

Data conventions

The ABI has one aggregate carrier — bytes — so modules follow these conventions.

Booleans are int64 0/1; the .lynx wrapper converts with != 0.

Errors are in-band sentinels, because there is no error channel: -1 / -1.0 for indices and sizes, "" for strings, "ERROR: ..." for csv, and "Error: ..." for js.

Stateless structured data crosses as a JSON string. The native side encodes with stdlib/native_json.hpp (a header-only value/parser/serializer embedded into each .so, so no module depends on another module's symbols), and the wrapper either forwards the string or post-processes it with listJson* / splitStr.

The structured-value bridge. A document module (json, toml, ini, xml, yaml) represents its document as a JSON string and nothing else; serde_json::Value is the single in-Rust model every one of them parses to and serializes from. xml maps its document onto the same bridge (a JSON element tree), so a caller turns any of them into Lynxer values the same way, with jsonParse / jsonGet / listJson*.

Binary payloads cross as bytes when the module declares the bytes token. Modules that expose a text encoding continue to pass base64 or hex as strings; each module's page documents its payload representation.

Numeric list results cross as bytes holding little-endian f64 — the encoding the listToBytes / bytesToList builtins use; math's statistics and vector operations are the model. Other list results still use the older encodings:

values that cannot contain the separator: a joined cstring that the wrapper splits with splitStr (newline for paths);
arbitrary strings (command output): an int64 handle into a module-local registry, read back with resultCount/resultAt/codeAt and released with release — see stdlib/multiprocessing.cpp.

Cstring and bytes lifetime. Returned const char* and bytes values must stay valid until the interpreter copies them, which happens immediately after the call. Modules return a thread_local buffer — a stable() string, or lynxerBytes from lynxer/stdlib/lynxer_native_abi.h for bytes — and therefore support exactly one live result of each kind per call.

Worked example

#include <cstdint>
#include <string>

using RegisterFunction = int (*)(const char*, const char*, const char*);
using RegisterConstant = int (*)(const char*, std::int64_t);
using RegisterType = int (*)(const char*, const char*);

static const char* stable(std::string value) {
    thread_local std::string result;
    result = std::move(value);
    return result.c_str();
}

extern "C" std::int64_t pair_sum(std::int64_t left, std::int64_t right) {
    return left + right;
}

extern "C" const char* pair_label(const char* name, std::int64_t count) {
    return stable(std::string(name) + ":" + std::to_string(count));
}

extern "C" int lynxer_module_init_v1(RegisterFunction function,
                                     RegisterConstant,
                                     RegisterType) {
    return function("sum", "pair_sum", "cdecl:int64(int64,int64)") &&
                   function("label", "pair_label", "cdecl:cstring(cstring,int64)")
               ? 0
               : 1;
}

global setup(){ importAs("pair.so", "pair"); }
global main(){
    println(global.pair.sum(2, 3));
    println(global.pair.label("items", 4));
}

Extending the ABI

The ABI is deliberately small and changes only when a real module cannot be expressed with the conventions above. Before adding anything, a module must first try to fit one of them:

Convention	Use it for
Numbers (`int64`, `float64`, `bool` as `0`/`1`)	countable, measurable or flag-like values
Strings (`cstring`)	text, and paths
Integer handles	anything the module owns and must outlive a call
JSON strings	stateless structured data — lists, tuples and records
Named callbacks (`lynxer_module_attach_v1`)	the module driving Lynxer code

An additive ABI change is only merged with all four of:

a C example — lynxer/examples/native_signatures.cpp is the pattern, and make test compiles and runs it;
a Rust example — through lynxer_abi, because that is how the bundled backends are written;
compatibility coverage — the shape added to nativeCallTable() plus a fixture that exercises it, so a regression fails make test;
documentation — this page, and stdlib-contracts when the change alters what a module may rely on.

An additive change must not alter how existing shapes or signatures behave.

Platform support

Native modules are supported on Linux, macOS, and Windows. The bundled Lynxer wrappers keep their .so import names on Windows for source compatibility; those files contain PE DLLs and load through LoadLibrary. Third-party modules must be built for the host OS and toolchain.

On Windows, export the entry point explicitly when building with MSVC or clang-cl (MinGW exports it by default):

#include "lynxer_native_abi.h"

extern "C" LYNXER_MODULE_EXPORT int lynxer_module_init_v1(
    RegisterFunction function, RegisterConstant constant, RegisterType type) {
    return function("sum", "pair_sum", "cdecl:int64(int64,int64)") ? 0 : 1;
}

Build it as a DLL and load it by path, exactly like a .so:

cl /std:c++17 /LD /Ilynxer\stdlib pair.cpp /link /OUT:pair.dll
lynxer app.lynx

LYNXER_MODULE_EXPORT is __declspec(dllexport) on Windows and a default-visibility attribute elsewhere, so the same source works on every supported host.

The reverse direction: exporting Lynxer functions

The mirror of this ABI lets a Lynxer program expose functions to C instead of calling C. A program declares export "cdecl:<ret>(<args>)" name(...) { ... } and lynxer --emit-library builds a host shared library (.so on Linux, .dll on MinGW Windows) whose typed extern "C" wrappers marshall through the embedding C API in lynxer/lynxer.h (lynxer_embed_init / lynxer_embed_call / lynxer_embed_last_error). It reuses the same LynxerFfiArg/LynxerFfiResult wire types and the same cdecl: signature grammar and aliases described above. See embedding for the syntax, the C type mapping and the build flags.