I found my new programming language to focus on: Gleam1. I like to deep dive and learn new tools in depth, always looking for general purpose tools and ones scalable enough to be reasonably efficient to use for a wide range of applications. Gleam1 is my new language of choice for writing internal tools, daemons and backend::frontend code. You too can try it at tour.gleam.run.
Haskell2
I have been a student of Haskell1 for many years. Not professionally, but as a hobby. Whenever I needed a custom tool, I tried to use it as a vehicle for learning Haskell1 and a few tools were written :) I learned a lot and fell in love with declarative and functional programming. Haskell1 never became my default language, it almost never does that for anyone, but everyone learns a lot from it. I took these lessons and applied them in my professional “Big Data” work with MapReduce pipelines, SQL and Python3.
Gleam1
Gleam1 is a new functional language targeting Erlang’s Beam4 Virtual Machine and also JavaScript Runtimes (Node, Bun4, Browser JS engines). The feature-set and syntax is the best of Haskell1, with a sprinkle of Ocaml practical impurity and backed by power of entire Beam4 ecosystem. The language is immutable, impure, static, and eager - quite a shift from Haskell1’s immutable, pure, static and lazy!! It comes with an amazing toolchain which includes builder, very fast compiler with amazing error messages, LSP5 and great documentation.
Beam4 VM
Beam4 Virtual Machine was built for running high-availability massively multi-threaded (implicit mix of POSIX and green threads) functional software. Erlang was the original language for which Beam4 was build, but nowadays the other members of big-three are Elixir and Beam4. There is full module interoperability between libraries from each of these languages and a robust FFI for C/C++/others. Beam4 is behind highly scalable services such as WhatsApp, Discord, RabbitMQ and Pinterest.
Learning Gleam1 Basics
As a vehicle to learn the language basics, Beam4 features, standard library, build system and runtime characteristics, I wrote a micro-benchmark representative of workloads I tend to run. Nothing super complex, but exercising enough of the runtime to reveal it’s good and bad sides.
The micro-benchmark:
- streaming read of 16MiB chunks of 1GiB file to exercise system IO interface, memory allocation, GC
- hand-off chunks to worker-pool (size limited to number of hardware contexts) exercising IPC overhead
- each worker builds a histogram of values exercising (in)efficiency of the language for binary data handling
- workers send results back to main thread exercising IPC latency
Benchmarks
The Gleam1 benchmark implementation was structured as a shared library and I crafted two main() functions around it. One for running on Beam4 VM and utilizing a crew worker-pool, and another for running on Bun4 (JavaScript engine) single threaded. Additionally, I also ran the Beam4 variant with a single worker to measure single-threaded performance.
To give me a comparison against other languages of interest to me, I implemented the same micro-benchmark in Zig6, Haskell1 and two Python3 implementations: one with interpreter inner loop and another using str.translate() to produce histogram of the entire chunk (which Python3 implements in C).
Here are the results:
| Language: | seconds | peak_MiB | binary_MiB | runtime | features |
|---|---|---|---|---|---|
| zig | 0.7 | 17 | 3.72 | stand-alone | imperative, impure, static, mutable, eager |
| gleam-beam-ht | 6.1 | 425 | 0.45 | needs Beam4 VM | functional, impure, static, immutable, eager |
| py-c | 9.4 | 58 | 0.00 | needs python3 | imperative, impure, dynamic, mutable, eager |
| haskell | 25.3 | 60 | 14.07 | stand-alone | functional, pure, static, immutable, lazy |
| gleam-beam-1t | 25.9 | 176 | 0.45 | needs Beam4 VM | functional, impure, static, immutable, eager |
| gleam-bun | 111.9 | 168 | 75.96 | stand-alone | functional, impure, static, immutable, eager |
| py | 112.5 | 42 | 0.00 | needs python3 | imperative, impure, dynamic, mutable, eager |
Zig6 is a modern take on C with more safety and can be considered running on bare metal. The benchmark shows fastest execution and memory footprint barely bigger than chunk size buffer.
Multi-threaded Gleam1 is not too unreasonably behind, but it’s memory footprint reflects the overhead of Beam4 VM and parallel chunks per worker with GC overhead. Still, the footprint is « 1GiB, so GC was taking place. The gleam-beam-ht vs gleam-beam-1t shows the additional footprint due to the crew worker-pool memory cost.
Gleam’s1 single-threaded implementation using the Bun4 runtime, was completely non-competitive for this type of (server) workload. It excels at client-side in-browser workloads.
Haskell2 was surprisingly bad. I’ll admit to using AI to improve it from original implementation which suffered an even 5x slower performance due to inefficient immutable data handling.
Python3 interpreter based inner loop implementation was very slow, but using str.translate() to implement histogram processing on a chunk made it a top performer. The magic is in str.translate() being implemented in Python3 runtime as a C-function, so processing a chunk was done in bare-metal wrapped in py IO for reads. It still ended a bit slowed than gleam-beam-ht!! :)
For me, the features of Gleam1 language and Beam4 VM, combined with effortless and very safe worker pools (compared to other languages) allow for trivially simple and very linear speedups on SMP systems. When workloads are dominated by IO, latency, and blocking, Gleam1 is much simpler than async approaches and scales better. It’s raw data processing speed is impressive and when absolutely needed, inner loops can be implemented in C/Zig6 for the same kind of improvements as seen above in py-c vs py. TODO(self): try C-based histogram inner-loop in Gleam1! 😁
Language Features
My programming journey of over 30 years progressed through following languages: Basic, Pascal, Assembly, RPL, C, Java, C++, Perl, Python3, SQL7, Haskell1 and now Gleam1. I’m very excited about the future and feel we are programming in exciting times. Many languages are now adopting or making possible programming in functional style (tail recursion and functions being first-class values) even if they naturally skew towards imperative paradigms.
Different languages use different combinations of features to structure their execution. I mentioned follwing features in above benchmark results table. Here is a bit more context:
| Feature | Summary | Impact |
|---|---|---|
| Functional | describes how data is transformed | focuses on evaluating functions to produce values and passing functions as data |
| Imperative | describes how to perform a sequence of tasks | focuses on changing state |
| Pure | Results depend only on inputs | no side-effects and entire classes of complex bugs impossible, but requires use of more complex IO semantics |
| Impure | Allows side effects (global state mutation, IO) | pure vs impure is responsibility of dev, but allows normal IO semantics |
| Static | Types checked at compile time | entire classes of errors impossible, compiler enforces better safer code |
| Dynamic | Types resolved at runtime (strong or weak) | runtime errors and runtime cost, “but flexible” |
| Immutable | data cannot be changed after creation | changing data requires more memory and memory bandwidth, but sharing data between workers is easy and safe |
| Mutable | data can be changed in-place | lowest memory and memory bandwidth requirements, but sharing between workers requires complex locking and contention |
| Lazy | Expressions are evaluated as values are needed | values produced only when accessed, difficult to debug and can cause resource leaks, but simpler code in many cases |
| Eager | Expressions are evaluated immediately | more verbose code, but generators/IO_iterators can still be used to implement laziness where needed |
-
Gleam: a new functional programming language for Beam Virtual Machine. ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎
-
Haskell: an academic research programming language exploring frontiers of functional programming. ↩︎ ↩︎
-
Python: A very popular dynamic interpreted “glue” programming language. ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎
-
Beam VM: a virtual machine for efficiently running Erlang, Elixir and Gleam functional programs with high availability and massive multi-threading. ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎
-
LSP: Language Server Protocol - a mechanism for programming language toolchain to communicate auto-complete, syntax errors, diagnostic messages, lint, hover documentation, etc, in real-time to one’s editor of choice. ↩︎
-
Zig: A successor to C programming language with more safety and a great toolchain. ↩︎ ↩︎ ↩︎
-
SQL: Structured Query Language: declarative language for manipulating large data. ↩︎