Pallene: A Programming Language for Lua Extension Modules

Pallene is a programming language with a Lua like syntax. It is a compiled statically typed language that ouputs a shared library that can be imported as an extension module in Lua 5.4. Here I discuss its merits and claim that while it is an interesting idea, there is no strong case as of now to switch from LuaJIT.

Lua is a programming language that is usually used as an embedded language inside another application. “Lua is designed, implemented, and maintained by a team at PUC-Rio, the Pontifical Catholic University of Rio de Janeiro in Brazil.” Lua 5.1 was forked into LuaJIT by Mike Pall. As the name implies LuaJIT includes a Just-In-Time compiler. In addition it provides a Foreign Function Interface (FFI) that is easy to use. There is no need for a wrapper that marshals and un-marshals data between Lua and C. PUC-Rio forked Lua 5.2 in a different way. It is now Lua 5.4. LuaJIT is usually faster than Lua 5.4 as the paper by Hugo Musso Gaulandi and Roberto Ierusalimschy (one of the original authors of Lua) indicates. See figure below for a comparative evaluation (taken from the original paper)

Relative Performance of Lua variants with C implementation for reference

To improve performance one could write the the extension module in C. Notice that this is still not as good as LuaJIT in many instances. Besides it is a laborious process that is prone to crashing the host application. Hence Pallene was developed as a language that produces C code that is then compiled to a shared object. This code is presumably faster than a hand generated version as it can exploit the low level machine characteristics.

A function to compute GCD in Pallene is shown below:

function M.gcd(n: integer, m: integer): integer
    if m == 0 then
        return n
    else
        return M.gcd(m,n%m)
    end
end

Notice the type annotations for the input parameters and the return value. The type of a local variable though, is inferred provided it is initialized. Thus variables are statically typed; in other words, the type of variable cannot be changed as compared to a dynamically type language where a numeric variable can be assigned a string.

Some of the key differences between Luajit and Pallene are listed below:
i. Luajit is Lua 5.1 with a few additions. A shared object for Lua 5.1 will work with Luajit. Pallene is not Lua except for a similarity in syntax
ii. Default parameters are not supported in Pallene.
iii. Functions in Pallene have a fixed arity; the number of parameters are fixed.
iv. Boolean operators in Pallene take only boolean operands; an object reference cannot be converted to a boolean by comparing with nil implicitly.
v. Pallene supports only single return values from functions.
vi. No implicit type conversion in Pallene.
vii. Lua has only one built-in data structure: table. Pallene has arrays and records but no tables.
viii. Pallene currently does not support extending operator as in metatable redefinition in Lua
viii. Luajit supports only one numeric format: 64 bit floating point. Hence integer arithmetic precision is just 51 bits. A higher precision is rarely required.
viii. Lua 5.4 and thus pallene uses 128 bits to represent a number and has two distinct types of numbers 64 bit integers and 64 bit floating point numbers

Many of the restrictions in Pallene are not too difficult to live with. Some of them like fixed typing, fixed arity and explicit type casting makes the code more robust. Extending the number representation to 16 bytes was not a great idea though. As the figure shows and Luajit outperforms Pallene in matrix multiplication because as Gaulandi and Ierusalimschy the lower storage requirements in LuaJIT improves cache hit ratio.

Did they make the right choices? Lua is not used as a run once and throw away scripting language like Perl or Python even if there are a large number of extension modules in those languages that are often reused. Consider a typical LuaJIT use case. Openresty allows one to extend NGINX by using LuaJIT rather than writing code in C. Thus the Lua code is anything but run once. The same goes for NeoVIM. The developers of Lua, Roberto Ierusalimschy et al., must adapt to how the language is currently being used rather than stick to the original goal whatever that was. Pallene is admiitedly a much safer and better performing language than Lua even if ease of programming had to be compromised. That is definitely welcome. The use of arrays rather than tables also reduces the amount of memory requirement and more importantly memory fragmentation.

It is not possible to automatically convert Lua code to Pallene. Manual labor is required. However improvement in performance by shifting from LuaJIT to Pallene is marginal at best. For LuaJIT users there is NO compelling case for shifting to Pallene. It would be nice though to port Pallene to LuaJIT. That way we could have best of both worlds.