nngn - lua & wasm

2021-03-23T10:10:25

This is a short interlude in my series of posts. The original post in the series briefly mentioned that one of my build targets is WebAssembly, and the last one was all about how Lua is a fundamental piece of the architecture. How that build process works did not seem particularly interesting, so I did not write about it.

I have since been involved in discussions and seen enough interest in blog posts — none of which I find satisfactory — that changed my mind, so here is an overview.

demo

You can see a basic demonstration on this page. A Lua script processes keyboard input (very roughly, I should add), evaluates it as Lua code when enter is pressed, and updates the text box with the result (I heard you like interpreters…). It is similar in function to the command-line lua interpreter that might be found in your operating system.

emscripten

Unsurprisingly, the WebAssembly build is based on the emscripten project. Tutorials for C/++ applications surely exist, so I will focus on the details pertinent to this post. emscripten supports several heavily used libraries, as described in their documentation. These include most of the libraries used in this code base, such as OpenGL/GLEW, GLFW, libpng, freetype2, etc., but Lua is not one of them. This means it has to be built from source, also using emscripten.

After it is built, the generated library can be included in the build process as one would do for any other library. As a bonus, a simple interface to the embedded Lua interpreter is also exposed to Javascript code in the web page which, combined with logging messages being displayed in the browser console, gives a nice REPL interface to it.

The final step in the process is to embed the Lua libraries and scripts present in the repository in the Javascript source, providing access to them by file path in the same way as a regular desktop build. This is a very simple process also described in the documentation.

building Lua

Building a scripting language's interpreter from scratch may seem intimidating, but two of the greatest appeals of Lua, as mentioned in the previous post, are its simplicity and focus on being an extension language. The papers section of the documentation pages is an excellent resource describing how these have been intentional and explicit goals of the language from its conception (prior to it, in fact, as Lua's predecessors shared those goals).

The source code can be easily obtained from the download page (also mirrored in Github), and it is very straightforward and portable C99 code. emscripten provides a make wrapper that exports standard environment variables that point to its tool chain, so the Makefile in the repository can be used almost unmodified.

A script is used in my builds, which is executed once, before the final make call. Other than downloading and extracting the source code, its only significant actions are:

patch -Np0 < "$PATCHES_DIR/lua.patch"
pushd src/
emmake make generic ALL=liblua.a

In order to be used with emmake, the original Makefile needs some trivial patches, all concerned with eliding or adjusting the environment variables that define the compiler and library tools. Further adjustments may be needed depending on the project, but those few are sufficient to build Lua as a static library.

linking

With the library built, all that is left is to include it in the linker calls. You will notice that the build script used in the previous section also sets up a few directories where Lua's headers and libraries are placed. That simplifies this step, which mainly consists of adding those directories to the build system's inclusion directories.

autotools and pkgconf are my preferred tools, so I include a set of files to be used with the PKG_CONFIG_PATH variable (e.g. for Lua). This also allows the PKG_CHECK_MODULES directives in configure.ac to work without modification (there were plans to add similar pkgconf files to the emscripten repository, but I have not checked if that materialized in a while). This process should be analogous for any other build system.

Because of how emscripten builds work, this is also the step where Lua libraries and scripts (all regular source code text files) are included:

nngn_js_LDFLAGS = \
	$(nngn_js_CXXFLAGS) -O3 --source-map-base / \
	--embed-file $(srcdir)/src/lua/@src/lua \
	--embed-file $(srcdir)/src/lson/@src/lson \
    # …

bonus: console REPL

As a fun coda to this post, here is how access to the Lua interpreter is also exposed to Javascript and, by extension, the browser console:

emscripten provides bidirectional communication between Javascript and the application. The simplest form is via ccall and cwrap, which directly expose C functions. A simple C entry point is included in the program:

#ifdef NNGN_PLATFORM_EMSCRIPTEN
extern "C" {
void lua(const char *s) { NNGN_LOG_CONTEXT_F(); p_nngn->lua.dostring(s); }
}
#endif

This function is then exposed as described in the documentation linked above:

nngn_js_CXXFLAGS = \
    $(AM_CXXFLAGS) \
	… \
	-s EXPORTED_FUNCTIONS='["_main","_lua"]' \
	-s EXTRA_EXPORTED_RUNTIME_METHODS='["ccall","cwrap"]' \
	…

And that is it! We now have an application compiled to WebAssembly that has a full Lua virtual machine (or several, go wild) and that can communicate with the web page and browser console. I hope you enjoyed this small detour, see you in the next post when we will be back to our regular (if that word can be used) program.