So over the holidays I have had something of a little bug in my brain:
transducers. It started when I was thinking of working on
an unrelated side-project, but I more or less got frustrated and asked myself “why doesn’t Scheme
have a library that is as good as Rust’s Iterator
trait?” I am a strong proponent of Rust both at
work and outside of work; however, I don’t always want to use Rust. Sometimes I’m scripting
something fairly quick-and-dirty, but largely I’m part of the Lisp Cult1 and I have fun
writing Scheme. This isn’t about picking languages or favourites, it’s about finding out why, in
the case of my question above.
Needless to say, this eventually led me to writing my own egg (module) for transducers in Scheme. If all has gone well, it should already be available to download and install by running:
chicken-install -s transducers
I only note this because I think this blog post will serve as a good companion piece of documentation for the egg. I think the rationale is especially worth digging into, because this isn’t new ground and through the process of writing this egg I really felt like I was reinventing the wheel.
Likewise: before we start, I should note here that I’m going to mostly talk about CHICKEN Scheme
here since that’s the Scheme I’m most familiar with. I do think that almost everything I write here
is pretty applicable to R5RS / R7RS-small Scheme; however, if your specific Scheme system has some
(non-standard) way of handling anything I talk about here please do forgive me — I am very
likely not aware of it. I’ll also be talking heavily about concepts such as fold
, map
, etc. If
you don’t know what those are and haven’t done any kind of functional programming before you may
want to skip this post. I’d be happy to write a tutorial if needed, but I’ll avoid doing that here
for brevity.
Okay, so back to the bug in my brain: why transducers? Well, this journey for me actually started
with me thinking about Rust’s Iterator
trait. This trait is excellent and is
probably one of the traits I use the most when programming at work. Regardless of the type that I
start with, Iterator
allows me to write code using the same set of meta-functions (map
,
filter
, etc.) without having to write each implementation individually for each data structure.
Just by providing a next
implementation, I get access to all of the goodies provided by the
Iterator
trait for free relatively cheap.
But Scheme doesn’t really have anything like this! Rust can manage the above because traits are a
construct that happens at the type level. They’re a way of being able to abstract types in a
polymorphic way. You can write a map
such that it works for all T
that are Iterator
—
the equivalent in Scheme isn’t really possible. Scheme is a dynamic language, so this kind of
polymorphism isn’t gonna fly.
The idea of “iterate over an entire collection” is a pretty common one in programming. For the sake of brevity, I’m not going to justify why I think this is important. But let’s look at a couple examples of where Scheme fails because it doesn’t have some generic way to operate over different collections.
list?
of Scheme’s failures to abstract-map
/ -fold
/ -etc
for every data structure, except when there isn’tThe most recent SRFI (Scheme Request for Implementation)2 for working with Scheme vectors is
SRFI-133 - Vector Library (R7RS Compatible).
This library standardizes some of the API for working with vectors. Operations such as folding,
unfolding, copying, etc. are all made available by importing srfi-133
.
Except for one - vector-filter
. Unlike in other languages, vectors in Scheme are dynamically
allocated but fixed in size. This means that you can’t just push
or push_back
on a Scheme vector
like you can in e.g. Rust or C++. Because of this, SRFI 133 does not provide a procedure to filter
elements, because the final result of such a procedure could have an unknown size, and any
vector-filter
would need to know how many elements to allocate in the vector before it ran.
This is very very very annoying. It’s not that amortized O(1) pushing is hard to implement, but Scheme as a language tries as hard as it can to remain small. As a result, this kind of stuff is left to the last developer in the chain to work around. Now every time you want to do something like the following Rust code:
fn filter_odd_values(v: Vec<u8>) -> Vec<u8> {
v.into_iter()
.filter(|t| t % 2 != 0)
.collect()
}
You have to make the following choice:
vector-filter
, as well as add tests,
documentation, etc.filter
in SRFI-1), and then convert back to a vector.These kinds of operations (map, filter, etc.) are not universal across all Scheme types, usually because there’s a missing piece (cannot do amortized O(1) push to vector, has hidden internal state, etc). Every new collection has to define these operations every time a new type is added. This is a lot of work for very little gain!
Now let’s actually talk a little more in-depth about that second point, which is another failure of Scheme…
I’m going to quote here from the Google LISP guidelines:
You must select proper data representation. You must not abuse the LIST data structure.
Even though back in 1958, LISP was short for “LISt Processing”, its successor Common Lisp has been a modern programming language with modern data structures since the 1980s. You must use the proper data structures in your programs.
You must not abuse the builtin (single-linked) LIST data structure where it is not appropriate, even though Common Lisp makes it especially easy to use it.
You must only use lists when their performance characteristics is appropriate for the algorithm at hand: sequential iteration over the entire contents of the list.
Just because Scheme (and Lisps) are primarily based around lists does not mean that we should still be abusing the list data structure so. Let me clarify the actual point I am making here: I think library authors aren’t coordinating and as a result downstream users are forced to abuse the list data structure even when they don’t want to.
I am explicitly NOT saying that people writing Scheme today aren’t using appropriate data structures. There is certainly some of that happening in undergraduate classes somewhere, but that is irrelevant to my point. Instead, I want to point out how almost every SRFI or Scheme-specific module (such as CHICKEN eggs) provides the following two procedures:
(type->list t)
(list->type lst)
And yet, despite everything you almost never see type->vector
, type->u8vector
, type-1->type-2
,
etc. There are some rare exceptions to this (e.g. SRFI 158), but often even those exceptions are
providing type->vector
by first doing:
(define (type->vector t)
(list->vector
(type->list t)))
This… is exhausting. No matter what you do in Scheme you just can’t get away from this. We have a fragmented language, and that is very much by design. The fact that there are multiple implementations of the R5RS/R6RS/R7RS standards that can all be considered “Scheme” is not a problem. The problem is that when we build libraries to extend the core language we do so poorly and incentivize users to go through lists as an intermediate as much as possible.
To emphasize I want to point out what Rust does because the standard library and traits are cohesive in a way that Scheme isn’t. Look at the following snippet:
let new_collection = values
.into_iterator()
.map(/* ... */)
.enumerate()
.filter(/* ... */)
.collect();
This snippet of code shows so much about language design and the amount of thought that went into
developer ergonomics just to map, enumerate, and filter over a collection of items. You don’t even
know what values
is, but it doesn’t matter! You don’t know what new_collection
is, but it
doesn’t matter! That code will still work as long as the type of values
implements IntoIterator
and the type of new_collection
implements FromIterator
.
(f k x)
ing nightmareLet’s talk one more time about fold
, vector-fold
, et al. These functions provide a “folding”
given some function f
, a sentinel value k
, and a collection xs
. Usually the signature is:
(fold f k xs)
But what about that function f
? What is its signature?
(f x k)
Okay, that can sometimes look a lot like cons
, which takes in a value (x
) and a sentinel (k
)
and does something with it. You can then write:
(fold cons '() (list 1 2 3))
; => (3 2 1)
which will reverse a list. Okay, great! Let’s do the same thing for a vector of xs
and reverse it
into a list:
(import srfi-133)
(vector-fold cons '() (vector 1 2 3))
;=> (((() . 1) . 2) . 3)
What the hell? Why isn’t that the same? Aren’t these supposed to be abstractions over folding? Well,
it’s because fold
in SRFI-1 takes in an f
with the ordering (f x k)
and vector-fold
in
SRFI-133 takes in an f
with the ordering (f k x)
. SRFI-113 provides set-fold
that takes in the
ordering (f x k)
whereas SRFI-158 provides generator-fold
that takes in the ordering (f x k)
whereas SRFI-160 provides u8vector-fold
that takes in the ordering (f k x)
… I could keep
going.
This is the dumbest inconsistency in the language and we’re all losing brain cells thinking about
it. Universally I think cons
-like ordering is the wrong ordering, but here’s the thing: I wouldn’t
even care as long as we picked one and were consistent with it.
My point in all of this is to show that as it stands there’s a lot of complexity and a lot of
annoyances in trying to use the right data structure(s) in Scheme. Even if we dodge the whole
type->list
and list->type
anti-pattern, we still have to be vigilant about fold ordering, which
procedures are available (filter may not be!) and this all takes patience, time, experience, and for
a whole lot of nothing.
One last point I want to make is that often times libraries often come with a collection of their own map / fold / filter etc. procedures, but it is never really clear what the performance of these is. I mean, one can generally think of map as being O(n), but you can’t be sure that the person who implemented the standard was careful and wrote the best map possible.
A lot of SRFI’s provide a reference implementation that can work on just about any Scheme, but will aim to be portable before it aims to be fast or efficient. I think among the complaints I could have this is actually not a big deal. Benchmark and then make it faster based on what you measure, etc. But the key here is that we probably should be in a position where folding / mapping / filtering are optimized already, and we shouldn’t be suspecting those are our bottleneck. I think generally this can be assumed true because most SRFIs are pretty high quality, but everytime I switch between data structures I now have to ask myself:
I fully admit that you can’t abstract across different types without having to ask this in some
form. This is a lot to deal with when programming though. I don’t ask these kinds of questions in
Rust or C++ — Iterator::map
and std::transmute
operate the same as long as the type
implements the correct trait / interfaces (e.g. std::begin
and std::end
). Incrementing a pointer
or calling an expensive Iterator::next
are possible, but it’s a lot easier to know where to look
if you find iteration to be slow!
While I love what one can do with Scheme, I hate that this is the mind-space I have to put myself in every time I open my editor. We absolutely should strive for better, can do better, and we deserve better.
So I started seeking out if there was something in Scheme that allowed me to solve this problem. There were a lot of different ways to crack it, but I want to focus on two specifically that ended up being the closest to something that actually begins to touch on the issues I outlined.
SRFI-158 is the latest attempt by the SRFI committee3 to abstract over things that
“generate” and “accumulate” values. Let’s consider an example: a list can generate values by popping
off the head of the list (car
), and the rest of the values are held in the cdr
. Lists can also
be accumulated by cons
ing the different parts together. You often have to reverse a list after
cons
ing all the parts, but you can do that as a finalization step.
Generators and Accumulators abstract over the generation of values and the accumulation of values. This can be done for just about any type, because generators are just regular thunks, that is, functions that take in zero arguments. I suggest reading the SRFI, it is very instructive and the document explains its expected use-case very well. It expounds on the previous SRFI-121, which only outlined generators, but there’s notes about that too.
My main problem with generators and accumulators is that they basically replace all our existing
data types with a new type (generators) that can then be mapped / filtered over in a unified way.
After one is done with gmap
/ gfilter
/ etc. they can then convert this generator back into some
kind of type using an accumulator or one of the built in generator->type
procedures. This solves a
problem of abstraction by routing around it. Rather than worry about what operations are defined on
a type, we instead just create a type that has all operations work on it.
This kind of approach is a good first abstraction, but fails because it is generally impossible to
make this fast. It also doesn’t solve the fact that we have type->list
and list->type
proliferation. If anything, it makes it worse because most libraries are not generator-aware, and
writing generators correctly can be tricky. That’s not to say writing any code cannot be tricky, but
the obvious ways to write a generator are often using make-coroutine-generator
which uses
call/cc
4 and as a result is pretty slow on most Schemes.
Aha, so we finally get to transducers! Well, I think if you’re familiar with Clojure’s transducers, you’re already seeing where I was heading with this. But if SRFI-171 exists and works on CHICKEN Scheme, then why did I go ahead and make my own library? Why am I rambling about this here?
Well, I had some problems with SRFI-171. Mostly, I was unhappy with the interface and the sparsity
of the API. It provides some basic transduction operations like list-transduce
,
vector-transduce
, etc. as well as a bunch of different transducers (map / filter / etc). However,
it is missing some key components:
list-transduce
different from e.g.
vector-transduce
. I will get into this a bit more later.tdelete-neighbor-duplicates
and tdelete-duplicates
. This
is very much in the realm of my opinions but if you want to delete duplicates what you want is a
“set” data structure of some kind. SRFI-113 provides a data structure in that vein (based on
SRFI-69 hash-tables) but one could very well imagine a bunch of different set-like data structures
that one might use depending on the context of the problem.5SRFI-171 was actually not what I wanted to even use at first either, but I slowly realized that transducers were the only form that would tick all of the boxes for me. I tested it for performance and well… It remains faster than SRFI-158. There’s at least one benchmark example on the mailing list where SRFI-158 is slower. I’ve seen this across a lot of examples in my own testing, so I’m not convinced it’s a fluke. In fact, I actually think I know why SRFI-171 is faster, but let’s table that for now.
Needless to say, I felt that there was something lacking in SRFI-171 and that I could do better. Unfortunately the things I wanted to change weren’t really possible to retro-actively adapt into SRFI-171. Unlike a normal library, the whole point of SRFIs is that once they’re finalized, that SRFI is locked in stone. You can’t just add or remove or deprecate functions. You can change the default implementation if there’s a bug or add a post-finalization note about how the implementation should behave if there’s ambiguity, but you can’t just start picking and ripping at an SRFI. This kind of attitude towards software / APIs is anathemic to change. This can be seen as good or bad, but I think this kind of ideology is mostly dead outside of Scheme (and maybe parts of C++). I want to clarify that I don’t disparage the SRFI team, I think there’s a lot in the community who want to standardize across Scheme implementations, but this is not how any other language works with their libraries in the year 2023.
Okay, okay, let’s shift the tone a bit. I have been complaining a lot about Scheme and our libraries (with love), but let’s finally talk about what transducers actually are. Transducers were an “invention” of Rich Hickey, the creator of Clojure. I put “invention” in quotes here because I think that a lot of the groundwork and ideas somewhat predate Rich Hickey in the literature that he references in the original video I linked. But for the sake of making this easier to refer to, we’ll say “invented.”
Anyways, the main trick with transducers is that we can treat almost any map / filter / etc. operation as a left-fold over some data structure. That data structure must be some kind of “totally ordered multi-set,”7 that is:
Anyways, given some “multi-set” whose items “can be ordered” we can basically replace any map /
filter / etc. with an equivalent fold operation. For example one could take the fold
from SRFI-1
and implement map:
;; Assume some mapping function `f`
(fold cons
'()
(fold (lambda (x k) (cons (f x) k))
'()
(list 1 2 3 4 5))
I added in the reverse above as another fold as well. Anyways, the key is that “map” really only
depends on the reducing function (in the above example, cons
) to be implemented. So the difference
between e.g. SRFI-1 map
and SRFI-133 vector-map
is really just parameterizing on how we “reduce”
across the fold. If it’s cons
, then we get SRFI-1 map
. If it’s something more complicated (like
creating a vector and pushing items into it), then we could get a vector-map
instead.
So the first trick to transducers is that every “transducer” function (map / filter / etc.) is
parameterized based on the “reducer.” The video goes into good detail about this, but basically we
change map
from something that looks like fold into:
(define (map f)
(lambda (reducer)
(case-lambda
(() (reducer))
((result) (reducer result))
((result item)
(reducer result (f item))))))
Notice the ordering of our reducer is different than cons
ordering, but we’ll ignore that for
now. The second trick can also be seen in the above snippet as well: by using case-lambda
we can
have a few different forms for our transducer. If I were to give names to them I’d call them:
sentinel
): The sentinel value to use. As we can see here it depends on the
reducer to decide what that default value is.finalize / collect
): This is what to do once we have our final “almost reduced” value.
In the case of the example I gave earlier this was just returning the list, but one could imagine
reversing the list, shrinking a vector, etc. You can see map
depends on the reducer here too.reduce-step
): This is what we typically think of when thinking of the reduce
function. In the original example I gave for mapping-as-a-fold this was (lambda (x k) (cons (f x)
k))
.Okay, so that’s about the two tricks that one needs to know about transducers in order to understand how to use them in Clojure. Mostly, you can take any “collection” in Clojure and you can then use the following form:
(transduce (comp
(filter odd?)
(map (lambda (x) (* 3 x))))
+
(range 5))
or, you can provide a sentinel value (e.g. 100
) directly:
(transduce (comp
(filter odd?)
(map (lambda (x) (* 3 x))))
+
100
(range 5))
Okay, cool. This does seem to solve what I want in Scheme, at least in principle. Let’s tick off some boxes:
reduce-step
must always be in the same order. In the map
definition I gave
above it is always (result item)
not (item result)
like it is with cons
. This could go
either way but transducers only work if that is uniform!type-1->type-2
methods everywhere as we go.So transducers are really the solution I’ve been looking for. SRFI-171 suggests they are fast, and Clojure shows that they have most if not all of the properties I’m looking for. But SRFI-171 is limited and doesn’t provide everything I’m looking for. Additionally, Clojure seems to be able to do some magic. For example, why can I do the following in Clojure:
(transduce (map add1) + <list-or-vector-or-hashmap-or...>)
but in Scheme I seem to have to do:
(list-transduce (map add1) + <list>)
or
(vector-transduce (map add1) + <vector>)
???
It certainly seems like Clojure is doing some magic underneath the hood, but what is it?
Well, in short, Clojure can provide a form of parametric polymorphism via Clojure
Protocols. This allows them to dispatch the transduce
call
according to what type the argument in the last place of the call-site is. So if you passed a list,
you get effectively list-transduce
, and if you pass a hashmap, you’d get hashmap-transduce
.
This works well enough in Clojure because protocols leverage the JVM; specifically, they basically map 1:1 with a Java Interface on the backend. But Scheme isn’t on the JVM (at least, not all Schemes), and implementing some kind of dynamic-dispatch is going to introduce some overhead. Some Scheme systems have object systems like COOPS or GOOPS, but this is:
Portability wasn’t really my concern here. After all, I’m really only concerned with CHICKEN Scheme.
If I wanted to use Guile, I’d use it, but it is not the Scheme I reach for usually. Nevertheless, I
don’t want to dive into something so CHICKEN-specific here. The point I really want to make is one
that’s not too dissimilar to what Rich Hickey made about how map / filter et al. are just
parameterizations of a left-fold: Clojure’s polymorphic dispatch is just an assumed parameterization
of the transduce
form.
I wrote some of these thoughts on Mastodon but to
summarize here: I think the main difference here is that Scheme is aggressively monomorphic by
default. As a dynamic language that doesn’t have a single implementation or host platform /
environment, it is always going to be. So we have to think about transduce
not as it is in
Clojure, but as it should be if we parameterized the assumed traversal of the data structure.
Thinking this way gets us a new kind of transduce:
(transduce folder ; A procedure that knows how to fold across a data structure
transducer ; The xforms / transducers used in clojure
collector ; The reducer that "collects" the final result
sentinel ; A sentinel value (optional) to seed the collector
data) ; The data to fold over
This gives us a transduce
that shows us a pipeline of different pieces we can put together:
Folder, transducer, and collector all happen in that order, so I have placed the fold operation as the first argument. In my egg, one can then do:
(transduce list-fold
(compose
(filter odd?)
(map add1))
+
100
(list 1 2 3 4 5 6 7 8 9))
What does this do under the hood? Well, here’s the code:
(define transduce
(case-lambda
((folder xform collector iterable)
(transduce folder xform collector (collector) iterable))
((folder xform collector sentinel iterable)
(let* ((xf (xform collector))
(result (folder xf sentinel iterable)))
(xf result)))))
The important bit is in that second case-lambda
arm. We leverage the folder to be specific to the
iterable (list, vector, set, mapping, etc.). This maintains the monomorphic dispatch that Scheme
requires, without losing generality. We don’t have to proliferate multiple transduce
procedures
that then need to be tested / documented / remembered. Instead, we just need a special fold for each
type.
(define (list-fold f sentinel xs)
(if (null? xs)
sentinel
(let ((x (f sentinel (car xs))))
(if (reduced? x)
(unwrap x)
(list-fold f x (cdr xs))))))
In particular, this fold is a bit different than what is usually written. The trick here is that we
have one more branch operation for checking if a value is reduced?
or not. Some algorithms quit
early without traversing a whole list. For example, if we wanted to find
the first odd number in a
list we wouldn’t want to have to traverse the rest of the list after we find it. Likely, you’ve been
trained to use call/cc
for this kind of early exit, but transducers don’t need it. If any
transducer (or collector) returns (make-reduced x)
early, then the transduction will end at that
item.
By branching on reduced?
in the above list-fold
, we’re able to quit-early in a generic and
call/cc-free way. No continuations means that our code is very easy to convert into
continuation-passing-style (CPS) in a very straightforward-to-optimize way. In practice, there isn’t
really any material performance hit for doing this. It is required to make our transduce
behave
correctly, so we need new “fold” operations for every type in order to use transduction over that
type. This admittedly isn’t ideal, but I’ve reduced the problem space from “reinvent every map /
filter / fold / any / every / chain / flatten / zip” operation to “reinvent fold to allow early
termination.”
In truth, that’s not the whole story because flatten
/ chain
/ interleave
/ zip
operations
are type specific as well. However, these can be managed mostly through macros (which are publicly
exported and documented with clear examples). I would like to come up with a better solution here,
but I’m not sure that I will anytime soon. I’ve spent enough time faffing about on it already so I
think I’ll take a break and accept what I have.
I’m admittedly taking a deviation in normal Scheme naming in some places. More importantly though, I’ve chosen to not provide map / filter / etc. under prefixed names. I strongly believe we need to move out of the era of pretending that lists are the default for these. Transducers should be the default, and we should just train people to use them.
Are they confusing? Certainly at first they are very confusing. If you’ve made it this far into the post though I’m certain you (the reader) are not that far from understanding it completely.8
Performance is great. I’m going to mostly avoid benchmarks and let you do your own, because there’s no way I’ll be able to fairly represent whatever you’re trying to do. But just for fun run this:
$ cat transducer-bench.scm
(import transducers)
(time
(transduce
fixnum-range-fold
(compose
(filter odd?)
(map (lambda (x) (* 3 x))))
+
(iota 100000000)))
Note that iota
above is not from SRFI-1. On my machine I get the following output:
$ csc -O3 -static transducer-bench.scm
$ time ./transducer-bench
3.518s CPU time, 0.001s GC time (major), 1/108400 GCs (major/minor), maximum live heap: 877.83 KiB
real 0m3.540s
user 0m3.535s
sys 0m0.004s
That’s 3.540s to filter, multiply, and sum 100_000_000
numbers. That’s pretty good! Admittedly
this isn’t representative of a real workflow in any way, but the generator equivalent is much
slower:
$ cat gen-bench.scm
(import srfi-158)
(time
(generator-fold
+
0
(gmap (lambda (x) (* 3 x))
(gfilter odd?
(make-iota-generator 100000000)))))
$ csc -O3 -static gen-bench.scm
$ time ./gen-bench
9.666s CPU time, 0.001s GC time (major), 100000002/81404 mutations (total/tracked), 5/256433 GCs (major/minor), maximum live heap: 368.68 KiB
real 0m9.674s
user 0m9.607s
sys 0m0.064s
I mentioned earlier that I think I know the reason transducers are faster than generators (generally
speaking). In the definition of transduce
:
(define transduce
(case-lambda
((folder xform collector iterable)
(transduce folder xform collector (collector) iterable))
((folder xform collector sentinel iterable)
(let* ((xf (xform collector))
(result (folder xf sentinel iterable)))
(xf result)))))
We generate the procedure xf
by calling (xform collector)
. The final function that we call on
every item is the composition and execution of all the xform
s over the collector
. By the time we
start to fold over some data type by calling the folder
our function xf
is:
gmap
have to loop internally and manage internal state, which means that
they’re calling other generators, which might be calling other generators, and so on. Transducers
loop over data locally and call one function (which can be cached / inlined) that does
everything across the whole algorithm. Transducers’ speed is defined by the complexity of the
process, not by the number of intermediate layers taken to express that process.Now I might be wrong on one or both of those points (after all, I did not inspect the compiled output to really confirm this), but my intuition is telling me I can’t be that far off. Overall, I’m not sure that a well-written transducer is ever going to be slower than an equivalent generator. Many in the Scheme community won’t care about performance as much as I do here, so it may be immaterial. ¯\_(ツ)_/¯
Let’s say that the entire Scheme community looks at this project and thinks: yeah no thanks. If
you’re going to dismiss what is done here PLEASE PLEASE PLEASE just see
src/transducers.vectors.scm
and how I add vector collection. With this library you can do:
(import transducers)
(transduce range-fold
(compose
(filter odd?)
(map add1))
(collect-u8vector)
(iota 100))
The magic here is (collect-u8vector #!optional (size-hint 0))
, which will push all results to a
vector in amortized O(1) time, using a strategy reminiscent to what Rust’s Vec
and C++’s
std::vector
do. This operation (or at least one like it, named differently) is available for all:
vector
)c64
and c128
vectors)There is no intermediate list built, there is no conversion from another data structure. When the vector being collected runs out of size and needs to be extended, it uses 2× the capacity (so exponential growth) and will copy over the data using what effectively amounts to a memcpy. It is fast.
If nothing else these very same procedures work with SRFI-171 Transducers as well. I have released the egg under the MIT license so copy and paste them into your code if you must, but can we please incorporate this more wholly into the Scheme ecosystem?
Lastly - this can be optimized even further if one decides to add a size-hint to the collector:
(import transducers)
(transduce range-fold
(compose
(filter odd?)
(map add1))
(collect-u8vector 100)
(iota 100))
While the size may not be precise (filter could allow through 100% of the data or 0% of the data),
the size-hint can help you reduce the amount of re-allocation you’re doing if you think you have an
idea of the size of the final collection. Also notice that iota
is from the transducers
library.
It’s a custom range structure that should be faster to use with transducers than constructing a full
list with SRFI-1’s iota
procedure.
reader-fold
works with generators todayI probably won’t use SRFI-158 (generators and accumulators) ever again but you might already have
incorporated them into some of your code. The reader-fold
procedure works with generators as they
are today, as well as procedures like read
/ read-char
/ read-byte
/ etc.
Transducers are general enough that they can support generators without having to do anything special:
(import transducers srfi-158)
(transduce reader-fold
values
collect-list
(make-iota-generator 10))
So if you have a generator today I suspect that switching will be easy.
transducers
eggtransducers
isn’t really perfect — far from it. There are still a lot of things that I’m not
sure about. A short list of my gripes and open questions with the code I’ve written so far:
folder
procedure into transduce
ergonomic or should I be making
list-transduce
/ vector-transduce
/ reader-transduce
and such like SRFI-171 does? I like
how explicit the current form is so I am likely to leave it, but I’m open to hearing if the
Scheme community has strong dissent here.flatten
/ chain
/ interleave
/ zip
transducers. What to do with
these? Is the macro-based approach the most extensible for outside types? I feel like while the
macros I’ve written aren’t too hard to come to terms with it can be a bit dizzying to get started
with someone else’s macros vs. just implementing the procedure yourself.string-fold
/
collect-string
variants in the same way that I did for vectors. This is mostly because strings
suck, and I’m not sure how to best dance on the UTF-8 / non-UTF-8 minefield that CHICKEN has
going on. If one wants to the reader-fold
operations can be used with the corresponding
read-char
and with-input-from-string
procedures to get string support for transducers, but
that seems like it’s likely to encourage bad performance. Hopefully there’s a version of CHICKEN
someday where I can just ship UTF-8 string support only. That’s a possibility,
maybe?Before I’m done, I do want to clear some air on my thoughts on the SRFI and standardization efforts within the Scheme community. First and foremost, I want to be clear that I do not disparage those who work on SRFIs nor do I think the work they are doing is useless or otherwise harmful to Scheme as a community. I believe some of my post will come off as rather critical of the SRFI process, which I think is appropriate (in measures, at least).
I am not going to name or point to any particular member of the SRFI process, nor am I going to try and blame them all for my criticisms here. I think of a lot of this as evolutionary, not revolutionary. But to get to the point, my feedback for the SRFI process:
-map
/ -filter
/ etc. procedures as part of each SRFI
process. This is especially true when discussing data structure SRFIs. It’s a waste of time -
transducers have all but won (in my mind at least, and in the mind of the Clojure community). I
think given the level of abstraction allowed by transducers, some version of them (similar to my
egg or otherwise) should probably be the base standard to which we hold our collections.set-union
, sort!
, etc. are prime examples. For these kinds of algorithms, we should
absolutely spend more time fleshing out SRFIs to figure out how to make those work across
different Scheme implementations in an ergonomic way. But there’s a big difference between
operating on the whole collection vs. type-specific functionality.(f x k)
vs (f k x)
ordering all the dang time! I know
we’re all hacking for fun and education and to scratch our own itches but we really ought to get
the core bits right.Does this mean we need a regular Scheme conference where we can all get together and help guide this? Are we not inter-mingling ideas across different Schemes and Lisps and languages and programs enough? I won’t prescribe an answer but I can for sure say that I have hope that the SRFI process can adapt.
Overall I don’t even think most of the SRFIs are useless or anything; heck, I use a few of them in
some capacity as dependencies of transducers
. I don’t even think things like vector-copy!
or
reverse!
need to be in Scheme proper but ultimately we should make sure that we’re not directing
users to interfaces like vector-map
or vector-append
when in practice the code that folks want
to write is a transducer.9
In short: use transducers, whether it be the egg I just published or SRFI-171 or something you’ve built yourself. They’re performant, they’re composable, and they’re a hell of a lot easier to test. They stop us all from having to reinvent the wheel that is mapping / folding / filtering every time a new data type comes along. You and your code and your users will thank you.
I set out this blog post trying to understand why Scheme didn’t have anything as powerful or
flexible as Rust’s Iterator
trait. I don’t know if my transducers egg is as powerful or flexible,
but it certainly is a step towards it. Scheme is a great language regardless of the critcisms or
holes I’ve poked about different features of the language and community here, and it’s helped me
conceptualize a lot of important ideas that I’ve taken with me to other languages. My hope is that
by introducing a different kind of transducer into Scheme I’ll be helping someone else skip past a
lot of the churn that comes with learning and working with Scheme.
To end this, I want to provide a few helpful URLs:
I highly suggest giving those a read over and of course feel free to leave feedback (either on the
repo or the CHICKEN bug tracker) if you find anything good worth discussing. You can also find me in
#chicken
or #scheme
on LiberaChat.
I don’t know if I’m really a believer but I love the sense of community. ↩
Scheme-Requests-for-Implementation is a process in which libraries are “standardized” in the Scheme world. I’m not sure other language communities have anything like it. Most of the t ime it’s easiest to think about this as a process in which one can submit a library, discuss the API and how it would function across a number of different Scheme implementations, and eventually standardize the library by offering a reference implementation that works on one or more Scheme systems.
Ultimately the SRFI process is driven by the person who submits their library, so in many case the result is dependent on which authors present what work to the mailing list. You can see previous SRFIs and get a better understanding of the process at https://srfi.schemers.org. ↩
Again, this is a bit air-quotey. I don’t want to disparage individuals on this because I think generators are a bit of a clever solution to laziness but ultimately not what I was looking for. ↩
There’s a lot of take-downs of call/cc
, such as Oleg Kiselyov’s famous
writing on the subject. call/cc
is almost
always going to cause weirdness either with your program or with the garbage collector.
Delimited continuations are universally better on that front, but I think that’s mostly just my
opinion so I’ll avoid scrawling a diatribe against call/cc
across the entire article :) ↩
Yeah so this one is admittedly a flimsy point. Maybe there’s a joke in here about how I just
wanted to reinvent the wheel in my own image, but I will fully defend the point on
tdelete-duplicates
. If you want to do something like this you want a collector / reducer not a
transducer.
i.e. you should call:
(transduce list-fold
(map add1)
collect-set ; NOTE: doesn't exist in transducers as of 0.1.0
(list 1 1 2 3 4 2 1 3 4))
not
(transduce list-fold
(compose tdelete-duplicates (map add1))
rcons
(list 1 1 2 3 4 2 1 3 4))
Of course, most transducers in SRFI-171 are interchangeable with my own egg, so one could probably port it over if they really feel differently. ↩
I maintain several SRFI eggs for CHICKEN Scheme (including SRFIs 113, 116, 133, and more) and I have absolutely just copied the SRFI document over to the CHICKEN wiki, same as SRFI-171 here. So understand that when I levy the critcism of “poorly documented” at someone else’s library, I’m actually more or less thinking of my own crimes here.
I do really like Scheme and want it to be better and more accessible, so I’m going to leave the criticism in there even if I don’t have an immediately better solution in place today. ↩
Thanks to J✦rg Pre✦send✦rfer 🇪🇺🏳️🌈 for pointing me towards this series paper. It ultimately didn’t help what I was doing with transducers, but it did open my brain a bit. So maybe it did help, but not in the way that I was hoping / expecting! ↩
I generally think that people are pretty smart and that this isn’t that difficult of a concept to figure out. If you can work your way through the Little Schemer then you’re more than capable of figuring out transducers. Even more-so if you’re using them everywhere.
Scheme is often used as an instructional language, but there’s a good group of us who are trying
to use it as a general-purpose programming language. There’s no reason Scheme can’t be both, so
lets dispense with any ideas that transducers are “too hard” to add to Scheme or that providing
map
outside of SRFI-1 is blasphemy. I… am projecting some arguments I’ve seen hashed out
several times, sorry. ↩
You may think that I’m wrong here and that the code that users want to write shouldn’t be restricted to transducers (at least, in the case of map / filter / append / etc). However, even SRFI-171 represents a huge step towards the kind of future I’m hoping Scheme moves towards. One where we don’t have to import half a dozen different SRFIs in order to be able to map and convert across different types made available to a particular Scheme implementation. ↩
On November 10th 2021, I was involved in a pretty large car crash. I was rear-ended by a driver in a Dodge Ram1 in my Saturn ION. Given the size difference of the two vehicles and how fast I was hit, I didn’t have much of a back end afterwords, and even less of a functional car. Thankfully, I came out of it with a bit of whiplash in my shoulders, and that was that. I was down a car I had driven for over a decade, but I would be okay.
Vehicular violence is so common in our day-to-day life. This ranges from the small kinds of micro-aggression and road-rage we’re all familiar with, to the full-blown carnage and death that barely even makes the news cycle anymore. This violence is a huge problem, resulting in the worst case with a death toll of ~43K people a year in the United States alone.
This violence is perpetuated by a car-centric culture in North America, and is something that one has to contend with and fear on the roads of any city in North America. There’s a lot to lament — children dying, people injured, and least of all the property damage and an insurance industry that props itself upon and profits from this violence. Violence isn’t just an artefact of the system, it is baked in.
Needless to say, I’ve slowly come into the mindset of someone who hates cars. I think it’s pretty easy to become desensitized to it, after all, for a long time I think I was desensitized to it. Being involved in a violent crash opened my eyes in a number of ways, even if my friends tell me I was already pretty anti-driving / anti-car beforehand.
In part, my friends are correct in that I was never really bought into cars. To get to know me a bit:
So yeah, I’ll admit, I was already pretty far down the path of hating cars and car-centrism already; however, I’m not sure I was as far then as I am now.
So there I was, no car, and all the freedom in the world to choose what I wanted to do next2. So I chose to get a (cheap) e-bike and use that as my primary mode of transportation for as long as I could continue to do so. The photo above shows an important milestone for me - 1000 kilometres on the e-bike. I only recently hit this (almost a year later) which should tell you how far from my house I typically went and how much of a waste a car was for me.
I live in the Boulder-Denver area of Colorado, and while not perfect, there is a good amount of cycling infrastructure. Where that is lacking, RTD (the Denver area’s transit system) and ride-sharing can make up the gap. I’ll admit that while I haven’t fully weaned myself off of cars (others still do occasionally drive me), I don’t own a car and rarely feel the need to find one.
Switching to an e-bike has taught me a lot. Some of these lessons were harder pills to swallow than others. I think it’s important I share them though, because they permeate the discourse around cars and I wouldn’t be writing this at all if I didn’t find my perspective shift from this choice.
Firstly, I learned that my masculinity and acceptance as an adult in society is not defined by what car I own. This is something that we all kind of “know” in theory but isn’t always socially true. I’m reminded of a passage from an entirely different kind of article:
Taste in appearance is dressing for one’s peers, whomever we think they may be. My uniform of coat and tie had one meaning among the faculty and a completely different meaning to the students. The same outfit, which was a social asset to one group, became a social liability in another. One man’s meat, as they say. We feel that style is the image of character, that it reflects the person himself, while fashion gives us ideas the designer wants the clothes to convey. And, I don’t mean to get all philosophical, if we strip away the clothes looking for the person beneath, it might be rather like stripping away the leaves of an artichoke looking for the real vegetable underneath.
We latch our personalities to the things we own and the ways we use them to interact with the physical world. I think this is insightful for a couple of reasons: it certainly seems to me this is why it is so easy to become an asshole behind the wheel - the dominating, violent nature of driving multiple tonnes of steel and plastic becomes part of our personality. Those leaves means one thing to other drivers, and an entirely different thing to more vulnerable road users.
So why did I feel my masculinity and acceptance as an adult were challenged as a result of losing my car? Well, I didn’t get the e-bike right away. While I dealt with the insurance and recovered from my whiplash, I thought about it a lot. What kind of vehicle did I want? Would a normal, cheap bike suffice? Would an e-bike be an experiment I’d give up on quickly? Was it a huge mistake that I’d regret quickly once winter set in during December?
I spent a lot of time on these questions but the appeal of not paying for a new car definitely still drew me towards an e-bike. I had really started to dive into the technology at this point. I started to talk to others about the decision to avoid a car3. At this point my own father actually underscored why masculinity even comes into play here: “How will you drive your girlfriend places?” “How will you go on dates?” “What if work needs you to drive somewhere?”
While I’m sure my father believed he had great intentions, and while I can answer these questions pretty trivially (my girlfriend has her own car and is independent, I can take the bus or just bike, I work remote and Lyft exists, etc), I quickly realized that my dad was pestering me and being so insistent about getting a car because he grew up in an environment where not having a car meant that you weren’t an adult. You weren’t allowed to participate in society because you couldn’t — he was challenging my ability to provide as a man if I didn’t get a car4.
Honestly that may sound dramatic, but that’s the subtext. I think largely I realized I had also internalized this myself. I was worried that my girlfriend would see me as some kind of deadbeat if I couldn’t “just grab the car” and go. Internally, I knew that masculinity isn’t defined by the car you own (or owning a car at all), but that didn’t mean I didn’t act as if that wasn’t true. I didn’t want to be perceived as some Middle-Age-Man-In-Lycra, and I feared running afoul of not following the culture of owning a car and driving everywhere.
With reassurance of my girlfriend, and many of my friends, I pulled the trigger on a Heybike Cityscape, and the rest is history. Very quickly I came to realize that any concerns about masculinity or adulthood were put to rest. Car culture tells us that we cannot be adults without tonnes of steel and plastic. We should not let that steel become the leaves that wraps our personality, and should actively reject any message that we are lesser for not having them.
One surprise I had when starting out on the e-bike was that it was sometimes faster than taking a car. This seems counter-intuitive only if you’ve never used one. After all, cars can go upwards of 60mph / 100 km/h, so how can an e-bike limited to 33 km/h get somewhere faster?
Well, the answer is that roads are pretty inefficient due to their size. Parking lots also suck - more often than not I save time by being able to just roll up to wherever I’m going and park my bike at the entrance (or walk it inside). This is perhaps unique to where I live in Colorado - I can’t imagine every place in North America is like this but I can certainly say that parking one’s car at the local supermarket can be a much more frustrating endeavour than parking my bike often is.
I’ve mostly found that any “trip time” estimates on Google Maps or otherwise are kind of ridiculous. They are often based on a number of assumptions that rarely hold up, most importantly that the “cycling” time estimates assume that users are on a regular bike and go ~10 mph / 16km/h. Geometry hates cars and most mapping software isn’t very good at thinking outside of the big metal box.
Nevertheless, there’s a good number of ways in which I can’t get places as fast. Society’s focus on sprawl and creating ever more lanes and parking has to stop somewhere before Mad Max becomes a reality. I had an interesting conversation with an Airbnb host earlier this year when I visited Calgary. He had mentioned that he just absolutely adored the massive highways put in place in places like Houston, Texas. He wanted big, 16+ lane highways going through every city centre in both North-South and West-East directions. I now realize what an awful and shortsighted concept of beauty that is. Highways this large are terrible for everyone. The noise, the pollution, and most importantly the opportunity cost of that land-use for everyone else are all reasons that stand on their own to reject that kind of architecture in our lives. And in my experience and looking at the image above it often doesn’t save time unless you’re traveling really really far.
Unlike regular bikes e-bikes take away a lot of the strain of going up hills, gaining momentum after stopping, and pedaling in the most general sense.
This is kind of the whole point of e-bikes, but I will admit that I did believe that even with the
motor it’d be pretty hard to get back into biking again after not biking since my childhood.
Needless to say - they work. They’re great. Full stop send tweet toot post.
Honestly, I’m not sure I can describe how happy I’ve been with this vehicle. When I was driving, I used to loathe my trips to the grocery store, to go out to eat, or to have to drive and park and deal with traffic anywhere. Now… I kinda relish it? I don’t want people to come pick me up. Why would you take my e-bike ride away from me? It’s fun, and even if it takes me longer to get somewhere biking on a path or empty streets is just plain good fun and makes me enjoy the process of traveling.
I’ve already kind of hammered on this point but boy howdy have I saved a lot of money. No insurance, no gas, no maintenance, no depreciation, etc. All of that money in my transportation budget has now been freed up for a number of other things.
I highly suggest that if you are thinking of getting rid of a car for an e-bike, you do it. Whether it is your family’s second car, or you’re going car free entirely, you should get rid of it and just use an e-bike instead. There’s a ton of variety in e-bikes (from cargo bikes to city commuters like my own), so there’s probably something that’s right for you.
Conversely, how much has the bike cost me outside of the purchase cost itself? The answer is very little. Even after a pretty gnarly crash in August where I messed up the derailleur I’ve still spent less than $300 USD on maintaining the bike since November last year.
That doesn’t mean I haven’t spent money on it at all though. I’ve definitely bought panniers, a helmet, etc. for the purpose of using the e-bike. Still, my total costs including the above maintenance is still well below $500, and these costs are not recurring, but rather are fixed. Even if I spent as much as I did on the bike every year, I still would never even approach the cost of car ownership.
Finally, an excellent and timeless piece:
There’s honestly been a lot of mixed reception. I think part of that is due to my personality. When I get into something I’m very much the type of person who has to tell everyone about it in great detail. I’ve tried my best not to be so exhausting, but I’m sure some of my friends will attest that I’ve definitely had a “moment” or two with this whole experience. Nevertheless, not all the reactions are bad! I want to outline some of them:
I get these questions all the time. My dental hygienist feels the need to ask this every time I visit. It’s genuinely inquisitive, and she’s mentioned that she definitely has complained about finding parking at events and how if it weren’t for dealing with traffic and parking she’d easily be up to go to more venues.
Usually I try my best to respond positively. Sometimes it’s cold out, sure, but often I can also just take RTD and get where I’m going at my own pace. The bus is relaxing, and honestly if you’re in the Denver area and you haven’t taken the A-line train to the airport you’re adding too much stress to your life. Parking and paying for rideshares to the airport is a fool’s game and you always lose.
I also usually like to quip at people who say it’s too cold outside to bike. I grew up in the Calgary area in Canada — if there’s ever a place where it’s too cold, it’s out in the prairies when it’s -30°C. In contrast, the cold isn’t usually what stops me from getting on the bike, it’s more often than not some other constraint (often, time).
I don’t. I take the bus, rideshare, carpool with friends, and if I really need it I’ll rent a car. Here’s the thing you don’t really see from behind a windshield: once you own a car it’s pretty much your “transportation,” full stop. The marginal cost of driving your car almost always seems smaller than whatever other modes of transportation have to offer.
Contrast this with not having a car. I now make choices in how I get places based on what makes economic sense, as well as what makes sense from a logistics perspective. I’m going to downtown Denver and there’s no good way to park, but I can easily walk around? Cool, I’ll take the bus to Union Station and walk from there. Oh I want to visit the mountains? My girlfriend and I will take her car because there’s no buses going to the middle of nowhere.
Honestly, nobody I care about so far (sans my father, initially?) has made a big deal out of my choice to go car-lite / car-free as much as possible. I’ve received a lot of positive affirmations from friends, and even been told “yeah I didn’t think this was even a big step for you” by some. It’s so stupid that I have to repeat it: we should not let our ownership of specific kinds of property define our identity.
That said, there have been some awkward interactions. Weirdly, more than when I had a car others have offered to let me drive their car before they remembered that I don’t have insurance. This has happened with work as well as in my personal life. This never happened to me when I drove. It’s been a bit surprising since driving is so woven into the fabric of society and yet the fact I don’t drive is always a surprise. I’m fortunate and privileged enough that I don’t have to worry about social backlash due to not having a vehicle5. Either way, the “you can take my car” to “oh wait nevermind man I’d rather you didn’t” is some serious whiplash I didn’t expect to witness as much as I do.
Online there’s more trolls and bullshit artists out there who want to knock people down for not “staying in the lines” so to speak. Again, I am fortunate and privileged enough that I don’t have to worry about this at any worrying scale, but it’s out there.
Overall, I think the biggest reaction has been that I’m now the weird urbanist guy in my circles. I’ve always had a bit of a peek into urban planning from the surveying perspective, but I worry that layering activism and trying to convince others that living car-free is not only possible but desirable might stray too far beyond the overton window for many.
I do try to keep myself in check when I can, but I also increasingly find that harder and harder to do. Urbanism and our urban fabric are some of the most important issues we can focus on now. Between climate change, the housing crisis, all the way down to personal gripes like noise pollution and cars parked where they shouldn’t be — personal automobiles are ruining our cities and taking value out of our lives. We shouldn’t stand for this. I’m pretty bad at this activism thing, but if I can at least keep your attention, I’ll point you to folks who know better:
Perhaps the most annoying aspect of any person who gets on a bike is that they start to realize how absolutely fractured cars have left our streets, cities, and communities. They take up too much space and while we can’t get rid of all of them (ambulances and firetrucks can stay), that doesn’t mean they’re the default.
In conclusion, to hell with cars, and to hell with driving! There are better way to get around. My e-bike has been pretty life-changing and I’m very excited to get another 1M metres out of it. Not only is it more convenient, I actually enjoy riding it and save money while doing so.
There’s a funny thing that happens once you’ve gone car-free. Eventually, you start to realize that a lot of the “rules” you set for yourself (you need a car to be an adult, you are helpless to get places without a car, etc) are complete and utter bullshit. Moreover, you recognize that the trade-off you’re making is rooted in something. I know what I’m giving up by not having a car, but I don’t think that folks who have a car appreciate what they’re giving up (in both dollar value and lifestyle) by having a car. It may seem self-serving to say that; however, I’ve definitely seen it in action and don’t for a second believe I wasn’t the same a year and some time ago.
I don’t expect that posting this on the internet will change anyone’s mind, but I’ve been sitting on this article for a long while now and I wanted to get it out. I think it’s about time we stop sheltering ourselves and our egos inside giant metal boxes and we face the world directly. Don’t define yourself based on what you own, or how you get somewhere. Define yourself by what you value, and what you value by the happiness you want to inflict upon others.
My choice is easy: if I have to choose between an expensive and heavy metal box that inflicts unending violence as baggage, or a lifestyle that meets up with my values and makes me happier — I’m going to avoid that metal box as long as I damn well can. I hope everyone gets the opportunity to try as well.
You always have more of a choice than you’re probably letting yourself. I’ve yet to see a clear deconstruction of defeatism that I’d be willing to link here, but if you’re reading this and think “oh wow what incredible privilege to be able to choose a bike or transit” — I’m more than certain you can choose to avoid car trips more than you already are. Most cities in North America do suck for anything-that-is-not-cars, but that doesn’t mean it’s impossible to reduce your usage even if you can’t outright go car-free. ↩
Shoutout to Laura and Matt who joked I’d just get a bike and no new car, and then I did just that. Not quite a unicycle, but I think you both got a chuckle out of it. ↩
Looking back some of the desperation in the texts we exchanged during that period are just funny now. A lot of “Get a car!” and “u should really just get an SUV, they r cheap now like $35K” and the like. Look dad - your desperation for me to own a vehicle was a lot greater than mine, and probably will continue to be.
To others: if you want a more complete picture of this I’m totally comfortable talking about it over a beer :) ↩
I’ve seen more than a few Reddit posts on r/fuckcars where people have genuine stories of their bosses threatening their jobs because they didn’t own a car, or didn’t own a “good enough” car. I work remote and my boss is a really good guy, so I get the privilege to avoid this kind of garbage, but I do understand that this isn’t the norm in a lot of the States. ↩
The last time I did a book review, I lamented about how long it took me to get through the book. The topic of dependent types was both new and unfamiliar, and maneuvering through the exercises was long and required engaged thought. This time, I’ve approached “Bernoulli’s Fallacy: Statistical Illogic and the Crisis of Modern Science.” The book broaches subjects that are neither wholly new or unfamiliar to me, someone who practices engineering and science. It presents a fascinating perspective into the history of probability as well as a condemnation of many stastical norms, or orthodoxies. Controversial that may sound, and controversial it is! But I think it underscores some very important mistakes made in modern statistical practice, and on reflection of my own education, I think it’s worth discussing!
This book, authored by Aubrey Clayton, clocks in around ~300 or so pages of actual text, with notes / bibliography / index comprising about 50 pages thereafter. Unlike my last review, where I spent many months reading through the Little Typer, I blew through Bernoulli’s Fallacy in about 3 days. Needless to say, I quite enjoyed it. Which brings me to the point:
The short form of my opinion on the book is that you should definitely read this book. If you want to do that without being tainted by my review, this is the place to stop!1
My education in statistics was largely embroiled within the orthodoxy; that is, the frequentist version of statistics (I’ll get to what that means later). Out of all my courses during both my Bachelor’s and Master’s degrees I only ever took one “pure” engineering statistics course. This was a speciality biomedical engineering statistics course, and in hindsight it was somewhat lackluster. The course itself taught both Bayes theorem as well as many other orthodox or frequentist methodologies in statistics, but didn’t really raise any philosophical distinction nor point out much to do with what probability means. “Bernoulli’s Fallacy” pits the Bayesian philosophy against a frequentist interpretation, and lays out why the Bayesian approach is an extension of formal inductive logic, and how the frequentist interpretation side-steps this as an attempt to make probability “objective.”
Needless to say, I don’t think this impression was ever given to me in my formal education. The idea was always that Bayes’ Theorem was just something you do when you have to update a probability and you have known priors from some experiment. A “hack” to correct for the base-rate fallacy, but not fundamentally an extension of logic or some epistemic process unto itself. As I said, in hindsight my education was lackluster. Not terrible, I still had some foundation in the mathematics, but in the way that a robot can know the math and not understand the underlying principles for why that math is used.
Future courses in Geomatics and Surveying were not tailored to statistics, but required thinking statistics as one of the underlying tools. Eventually this culminated in a graduate level course on “robust” statistics in least-squares and Kalman filtering. I won’t criticize that course too harshly, but just like my statistics course I felt that there were many topics which seemed a little too specific or required some special interpretation of what “data” counted. I don’t think I could ever reconcile that or explain it well, but Bernoulli’s Fallacy did at least give me a good starting point to start thinking about it from an epistemic perspective (i.e. reasoning under uncertainty).2
So that’s fundamentally where I’m coming from when I read through Bernoulli’s Fallacy. While I’ve been mostly subject to the orthodoxy of present-day statistics classes, I also had an easier time bridging the gap to the Bayesian way of thinking. This is important context for later, and may drive some pieces of the book I didn’t fully understand or may wish to criticize.
The book could be fundamentally thought of as a mathematical text; However, is perhaps closer to the realm of philosophy (specifically, epistemology). Clayton argues that the orthodox methods of probability and statistics, namely the frequentist interpretation of probability, are fundamentally illogical and as a result are not useful when trying to learn something about the world (i.e. form inferences and weigh hypotheses from data).
Let’s get the definition of “frequentist” out of the way. The frequentist interpretation is the pre-cursor to the eponymous fallacy. Specifically, the “frequentist” interpretation of probability can be summed up as:
Probability is the frequency of occurance of an event in proportion to the total number of possible events that could have occured.
Put mathematically:
\[P(A) = \frac{\textsf{# of event A outcomes}}{\textsf{# of total possible outcomes}}\]Specifically, Jacob Bernoulli defined probability this way as a logical extension of an exercise in drawing coloured stones from an urn. But this is not the entire fallacy. Bernoulli then goes on to say that given a large enough sampling frequency defined as probability above, we can then make an inference as to the true probability. If we denote our sample size as \(P_{\textsf{sample size}}(A)\), then Bernoulli’s fallacy was assuming:
\[P_{\textsf{true size}}(H | D) = \lim_{n \to \infty} P_{n}(D | H)\]In short, that we could infer the frequency of the distribution of the true sample (i.e. our real world sample \(P_{\textsf{true size}}(H | D)\)) from the frequency of our collected sample (\(P_n(D | H)\)). In short, as that sampled frequency gets larger, we get closer to the “true” ratio of coloured pebbles in an urn.
The Bayesian alternative then, defines probability as a relative measure of knowledge (i.e. uncertainty) about some fact about the world. It does away with the notion of “sample sizes,” possible worlds, etc. and only consider actual data collected from an experiment. This would change the relationship of conditional probabilities above to:3
\[P(H_i | D \chi) = P(H_i | \chi) \frac{P(D | H_i \chi)}{P(D | \chi)}\]There’s a million articles about Bayesian probability on the internet, so I’ll try to summarize these terms briefly.
From a certain perspective this looks pretty close to the frequentist interpretation. In fact, you could even say this is a more general form of the frequentist interpretation, and that the frequentist interpretation falls out of it if \(P(H | \chi)\) and \(P(D | \chi)\) are both 1. Bernoulli’s fallacy then, is pretending that our base rate is always 1, and that data (\(P(D | \chi)\) is always objective (i.e. if the data was collected, it supports a hypothesis objectively and independently, not considering other hypotheses or interpretations).
The book makes a wonderful argument for why the Bayesian school of thought works, and more importantly how the above formula is all you really need to make inferences about the world. So why, then, does the frequentist interpretation persist today, and how deeply is that fallacy engrained in our present day practice?
Chapters 3 and 4 of Bernoulli’s Fallacy dive into the history of modern statistical practice, largely looking at three people: Galton, Pearson, and Fisher. These three men are some of the most prominent names in the practice, having invented many of the same techniques taught in university level classes even today.
Clayton reveals that these three men all had a vested interest in the ideas behind eugenics in the early 20th century. They clung to the frequentist interpretation for largely political reasons, namely racism. There was a vested interest in colonial ideologies of the day, and all three men vociferously rejected Bayes’ theorem because they felt that adding subjectivity to knowledge (as opposed to the alleged “objective” frequentist interpretation of probability) would invalidate their claims that certain races were better, or had “objective” measures that made them so.
Clayton calls this “The Frequentist Jihad,” and even names chapter 4 as such. Both in terms of the educational structures at the time as well as what was publishable in Pearson’s journal, Biometrika, which was considered the “gold standard” of publishing. The unfortunate echoes of this slant towards eugenics are heard today in the names we use for many statistical concepts, such as:
and so on. There’s a much larger list of phrases, techniques, etc. that were coined for explicitly loaded perogatives. The racist and eugenicist past of the field were certainly never taught when I learned statistics.4
Chapters 3 and 4 largely paint the background for why the social and political conditions of the time had an incentive to push the frequentist interpretation of probability. After reading these chapters an important take-away is to perhaps ask:
Clayton does a great job of citing his sources for these two chapters. It was perhaps shocking to learn of the degree in which early eugenicism in the colonial era influenced the direction of purported “objective” science. Recognizing the aforementioned rhetorical device used here does support one of Clayton’s later conclusions, however. Namely, that we should consider the hypothesis [we] didn’t assume. Just don’t walk away thinking that because you used Bayes’ theorem you’re not skewing your priors to support ideas on the wrong side of history, you still have to justify and write out those priors as well.
I have little to say about this other than Clayton does a wonderful job at portraying the Kafka-esque process that is trying to write a paper using orthodox probability and statistics. He examines several common issues that do not work under frequentist methods, but present no problem at all for Bayesian methods.
Chapters 5 and 6 were the point in the book where I realized that maybe much of what Clayton was arguing was not oriented for someone in photogrammetry and Geomatics. Earlier hints in the book suggested that Gauss and Laplace laid the future for the fields of surveying, astronomy, and physics; In contrast, Galton, Pearson, and Fisher laid the groundwork for biology, medicine, and many of the softer sciences.
Notably, many of the problems of base-rate neglect, extreme or unlikely data, and optional stopping aren’t really an issue in Geomatics problems. Although many in the field of Computer Vision will almost exclusively use a uniform set of priors (of which I could write a whole article about), it is rare in photogrammetry to ignore the prior and posterior uncertainties. My class on “robust” least-squraes came in handy here after all since I could imagine ways to estimate the “lost spinning robot” or unknowns with “extreme or unlikely data” with distributions that were not normal, or contained data with obvious outliers.
Unfortunately, I don’t necessarily think that the intuitions of Bayesian methods are evenly distributed across everyone working in my field. While we do start with the right tools, there is a tendency to assume uniform priors where there is real information at hand. Likewise, I rarely if ever see anyone solve the optional stopping problem in least-squares — few people will utilize what was taught to me as the “summation-of-normals” formulation of least-squares. Importantly, we shouldn’t treat several “sampled” experiements different than a single experiment where all the samples are part of the samples. Summing our inferences across many smaller experiments should be no different than putting all the data into the same inference problem.
The case is pretty strong for Bayesian methodologies to take over for our current orthodox methods. I agree with most of the messaging of the book. The history seems to check out, and the copious number of examples in the book strengthen the case that what we’re doing today is wrong.
Yet, I may still have misunderstood some parts of the book. I think that the book speaks with pretty broad strokes, and for the purpose of replacing our current illogical methods, I think that’s fine. But there are some nuances that I think are worth bringing up.
Clayton makes some claims in the preface of the book, that are labeled as being intentionally controversial at the outset. Further, he claims that by the end of the book we should be much more accepting of them, with a deeper perspective on the underlying logic. While I think the book does make good on attempts to support them for the most part, there is one which I’m not sure I can necessarily square up:
No special care is required to avoid “overfitting” a model to the data, and validating the model against a separate set of test data is generally a waste.
Most of what I work on with statistics is in the realm of optimization. Namely, I am often performing photogrammetric bundle adjustments to model optical sensors according to some mathematical model. A common practice is to leave out some points in a photogrammetric network and use these as test or check points. By using the final optimized parameters and “un-projecting” image points back into 3D space, we can use the coordinates of the check points (observed through some separate control survey) to evaluate the relative accuracy (not precision) of the final solution.
This is an important step, because many of the parameters we are optimizing for suffer from a form of projective compensation (more about this in a later criticism). It is entirely possible to be unable to observe (not directly observe, but in the sense of an observable parameter) many of the unknowns in the optimization. We don’t use check points or test points to validate over/under-fitting data, we use it to determine if our least-squares optimization resulted in a local optimum or if it converged to an answer that agrees with geometric reality!
Of course the above quote says that it is “generally a waste.” Perhaps this is meant to be indicative of the fact that methods resembling this are abused in softer sciences. Needless to say, I can tell I’m experiencing confusion here, because my intuition of how to compare my model of a camera to geometry in the real world seems to clashing with a broader condemnation of the method.
On the topic of overfitting, I would also (partially?) disagree. I’ve written numerous articles for my employer, Tangram Vision detailing ways in which classical computer vision models (such as using \(f_x\) and \(f_y\)) are incorrect models, precisely because they overfit observable quantities. More recently, I’ve gone into detail as to why those models are worse-off than the alternative, because they introduce more projective compensations throughout the final solution. It’s not that these models can’t be useful, but they aren’t nearly as repeatable, nor are they more indicative of the actual physics in practice. More on this in a later criticism, but I give Clayton the benefit of the doubt here. If I had zero prior knowledge of a better way to model some quantity, then worrying about overfitting before you have any indication that your model is even close to correct is premature.
The last chapter in the book is called “The Way Out,” and serves as a conclusion not just on what went wrong, but likewise what should be done to correct ourselves and move away from the errors of the past. In short, Clayton’s recommendations are:
I actually don’t have much criticism for many of these, and find myself in strong agreement. The first point is perhaps the only one I will question here, in the following sub-section.
Abandoning the frequentist interpretation is actually quite easy. Bayes’ theorem has not exactly evolved over time, and the case for a more unified practice of probability and statistics is made quite well.
Changing our language, however, I am much more skeptical of. Clayton actually enumerates what he finds to be the most important changes in our vocabulary. I’ve provided the following table to summarize these:
Orthodoxy says… | We should replace with |
---|---|
Random variable | Unknown |
Standard deviation | Uncertainty (or if using the inverse — precision) |
Variance / Covariance | Second central moment |
Correlation5 | N/A (don’t use this word ever) |
Linear regression | Linear modeling |
Significant difference | N/A (instead, report a probability distribution) |
This is pretty compact as far as these kinds of changes go and I have to wonder if he didn’t hold back in this part of the book somehow (I mean, there is a LOT of jargon in statistics). Nevertheless, I actually agree with the first and second (unknowns and uncertainty). In geomatics parlance, we actually already prefer unknown and uncertainty, although standard deviation will be used colloquially quite often. I’m not sure if that’s a problem in Canada / North America or if that’s very general across my own field, so perhaps I could be better informed here.
As for (co)variance, correlation, or regression — I’m seriously unconvinced this language will ever change. It may have a terrible past, but I genuinely think the field is damned to keep these forevermore. With respect to correlation, I prefer projective compensation. Mostly because in any optimization / machine learning context, the definition we apply to it is a bit better used:
Projective compensation is a relative measure of the degree in which residual errors in the modeling of two unknowns will correspond with one another. In the estimation of unknowns, it is an effect where the estimate of one parameter shifts as a result of compensating for residual error that has been projected into it due to its functional (model) or observed relationship (observed data) with another parameter.
This is the working definition that I use for projective compensation when dealing with it in the photogrammetry realm. If this doesn’t make a lot of sense, I’ve written a much better take at defining it on my employer’s blog.
As for (co)variance and regression — I’m not sure we can rightly replace these terms. There is
far too much use in the active machine-learning / deep-learning space that I would be quite
optimistic to believe it could be changed in short order. My main criticism of this here is mostly
that variance is pretty benign in my opinion. The variance across a selection of points in space
does not bring to mind the kind of work that Galton or Fisher were doing. Moreover, I can’t imagine
our primary APIs in the multitude of mathematical libraries / languages / etc. changing for this.
For example, swapping out
numpy.cov
with
numpy.second_central_moment
is unlikely to work out.
Perhaps my “criticism” is quite weak here though, after all I am advocating for doing nothing in the face of a book I largely agree with and am happy to have read. Maybe something a bit more succinct than “second central moment” is worth mulling over.
“Bernoulli’s Fallacy” is a wonderful introduction to the history of probability and statistics, and a scathing damnation of orthodox statistics as they exist today. While the book is not perfectly comprehensive, and is written in a tone towards those in the fields of medicine / biology / soft-sciences, it remains a good read with a lot of worthwhile food for thought for anyone who has ever touched statistics in their life.
The book discusses a racist past of statistics and how they were used to promote “eugenics” and any associated ideologies that endorse eugenics. Beyond pure history, it demonstrates the confusion of frequentist methodologies, and how they map onto reality. The book is eloquent, and kept me gripped to it from beginning to end.
Needless to say, I quite enjoyed it despite my criticisms, and suspect that many of my criticisms may be disingenuously nit-picky or may be misunderstanding something in particular. The topic is extremely relevant in my own day-to-day work, and I’ve done as much as I can to incorporate it to what degree I can. Not because it is morally right, but because Bayesian statistics are just more effective.
Lastly, if you’ve managed to tag along on this review this far, thank you. This has been one of my longer posts and on a topic few will engross themselves in for “fun.” As always, feel free to contact me if you have some comment on the review, or if you just wanted to share your insights from the book as well.
Also if you’re the author, Aubrey Clayton: hi! I hope I haven’t grossly misinterpreted anything you’ve said. I hope you enjoy the review. ↩
I’d be remiss if I didn’t also mention that in 2016 / 2017, near the end of my Master’s degree, I started reading the LessWrong sequences. This was perhaps the first experience I had where the Bayesian approach was not only heavily used in practical contexts, but generally admired and to some degree even worshipped. It was through this that I eventually tied together the concept that Least-Squares was fundamentally based on a Bayesian process.
I’m also happy to say that this was not a unique insight. Bernoulli’s fallacy does indeed go through the history of how Gauss and Laplace independently came up with the least-squares method using the Bayesian interpretation of statistics. It was refreshing having this intuition and then learning the history afterwards. I’m sure it should have perhaps been the other way around (history first, insights second), but it was somewhat validating to know that how I was applying my tools had the epistemic backing that the Bayesian approach argues for. ↩
There is also a sum-rule and product-rule for probabilities as well. These are:
\[P(A | \chi) + P(\neg A| \chi) = 1 \textsf{ ;; Sum Rule}\]and
\[P(A \land B | \chi) = P(A | \chi) \cdot P(B | A \land \chi) \textsf{ ;; Product Rule}\]Clayton copies these directly from Edwin Jaynes’ book, “Probability theory: the Logic of Science.” From these, we can derive Bayes’ theorem as well as pretty much any probabilistic logic. ↩
Where have I heard that before? Oh right, just about every history lesson. :(
It may be worth stepping back for a moment and recognizing both the rhetorical trick being done by Clayton here, and an evaluation of whether that changes how to perceive the content of the book. The rhetorical device in play here is “the orthodoxy is founded on racist/colonial/awful ideas and they were really against Bayes’ theorem.” What is left between the gaps is a sense that the Bayesian school of thought isn’t racist/colonial/awful. I think it’s important to not paint a clear canvas over history there either. Bayes’ theorem may be more mathematically correct, but coming away from the book with a notion that frequentist = racist, Bayesian = not racist would be a mistake.
Overall, I think a history mired in eugenics is probably bound to be problematic in a lot of ways, but in the spirit of critical thought it’s probably at least worth mentioning that Bayesian methods are not acquitted of any wrongdoing. You do have to at least be honest about your priors, so you may be more fault tolerant to bad hypotheses, but I have yet to be convinced that there’s any math that can change a man’s mind if he doesn’t want it changed. ↩
I added this in here myself, but given the text throughout the book Clayton is quite scathing of this term. ↩
It’s been about half a year since I wrote my original post comparing the Librem 5 and Pinephone. The original post saw some controversy as well as quite a bit of attention on Hacker News. Surprisingly, for a market dominated by new tech every year, there remains quite a bit of interest in these two devices.
Development on both devices continues day-by-day, in small and large parts. I wanted to revisit the devices as a lot of the ecosystem has changed. Further, I think they’ve definitely both evolved in terms of what’s possible, and more importantly what’s easy.
So with that said, let’s get into it.
Before being able to actually talk about the progress, I have to give the same disclaimer that I gave in the original article: I’m not testing voice, SMS, or MMS here. As far as I’ve heard from other sources, calls and SMS seem to be in working order. MMSd has (from what I’ve gathered, at least) been a large focus for many projects in the last six months. I try my best not to use MMS whenever possible, so I’m certainly a poor source of information if that’s a blocker for you to adopt one of these two devices.
Between the two phones, the Librem 5 has definitely surprised me the most in terms of actual development over the last half-year. This has been for both good and bad reasons, but overall I’ve definitely delved into the phone a lot more than I had the opportunity to last December.1
The Pinephone has been pretty stable for the most part, and hasn’t really changed. I am looking forward to some of the custom back cases for the phone itself, in particular the wireless charging case. It seems quite exciting to me that the hardware can continue to be hacked and improved. I don’t care much for a keyboard case with the Pinephone, but I am very excited to see how the wireless charging case performs. Wireless charging is something that I definitely miss with both the Pinephone and Librem 5.
I didn’t really have another spot to talk about this, but Ubuntu Touch has of late been maintaining
a more up-to-date kernel for the Pinephone on their kernelupgrade
channel. If you’re planning to
give Ubuntu Touch a try, do not be afraid to use this channel. Back in January it was much more
unstable, but I think now it is almost better than the official release on stable
. There is
probably some more work involved in getting it ready for stable, and I cannot comment on that here
as I am not an Ubuntu Touch developer. However, my day to day experience with this channel in Ubuntu
Touch was much vastly improved thanks to much of the hardware being more responsive. Some notable
things on this kernel that do not really work on stable right now:
stable
, you can plug in
headphones but the kernel fails to recognize that you’ve done anything.stable
if you turn the kill-switch to off
position (i.e. kill the modem) and you disconnect from WiFi because the phone went into sleep
mode, sometimes your WiFi and Bluetooth never come back.stable
I believe it is locked to 30FPS. It makes a huge
difference!No surprises here, I still vastly prefer Lomiri to Phosh. However, Phosh did get some nice-to-have upgrades. Auto rotation now works, which is very nice. Battery percentages in the Phosh drop-down menu from the top of the display also now correctly reports battery! These were minor issues in Phosh but it seems that since it is becoming more and more of a default for many projects outside PureOS that it is receiving consistent love.
Unfortunately, Phosh still isn’t very fluid, and suffers from many of the same downfalls I complained about last time. I don’t have a lot to say here, but I can definitely say that there’s a lot of room to grow.
I did get the opportunity to try out KDE Plasma via Manjaro on my Pinephone. The overall feel was much closer to older versions of Android (in particular, Android 4 and 5). Many of the gestures and interactions were much more fluid than say, Phosh, which doesn’t have gestures or animations to pair with those gestures. However, this is definitely the weakest UI / OS / environment out of the bunch. A few reasons in no particular order:
Overall, I felt like my experience with KDE Plasma was pretty bad. I hope to continue to keep testing it as it seems to be the default for new Pinephone units going forward. I am confident it will continue to get better over time, but my initial experience was definitely a step down.
I believe last time I complained that neither the Pinephone nor Librem had Bandcamp or Spotify apps available. This is still true.3 However, shortly after writing my previous review of the phones, quite a lot of progress on the sound front came about. For starters, an update was pushed quite rapidly to address the fact that microSD cards were not being automatically mounted. This makes the experience of just throwing all your music on a high-capacity microSD card more or less ideal.
Between running Lollypop on the Librem 5 and running the default Ubuntu touch music app or the default music app in KDE Plasma, Lollypop is by far the best out of all of them. I did bring up Lollypop before, but my experience so far is that the Librem 5 (using Lollypop, not sure if this varies based on software) is much, much better than that of the Pinephone.
To be fair here, the Librem 5 does actually include quite a fancy DAC in comparison to the Pinephone. When wired in, it does sound better. However, mainly I think the difference I’ve seen is that it manages to play FLAC and other lossless or high-fidelity formats without skips or interruptions. In contrast, the Pinephone does sometimes struggle with playback. Its music app should be able to run FLAC flawlessly, as I have definitely gotten FLAC to work. However, the music app does have some problems. Certain FLAC tracks for whatever reason seem to be unplayable, and I honestly can’t tell why. Other times, the app will skip like CDs used to, and you’ll miss ~1-2 seconds of a song. Lastly, the app really struggles when trying to shuffle through big collections on a microSD card. I have ~60GiB of music on a microSD, and occasionally the music app will just die when trying to go to the next song.
Overall, Lollypop on the Librem 5 is the clear winner. I took a long drive to Tulsa earlier this year (~10 hours), and I chose to bring the Librem 5 with me to play music during the drive. I had to keep it plugged into an external battery pack, but overall the sound quality and performance of the app itself were what you would hope for.
One thing I discovered which changed a lot of my experience with Phosh (Librem 5 / PureOS and any Phosh-based distro for the Pinephone) was that flatpak was so easy to install and set up. For PureOS or any Debian-based distro I believe it is as simple as:
sudo apt install flatpak gnome-software-plugin-flatpak
After adding flathub as a remote and rebooting, that unlocked quite a number of apps that I think change the experience significantly. For a quick overview, I ended up installing:
It seems that many of the apps on flathub are starting to get mobile or reactive-style application support. The clear winner here is Fluffychat, because hot-damn this app feels polished. It’s a full Matrix client with end-to-end encryption support, and can work alongside Element if you use that for cross-signing / messaging on desktop or other platforms. It may seem as if a Matrix client isn’t a game-changer, but for me this is honestly one of the main selling points of a phone like this: being able to use completely secure and open-source communication platforms (that are federated, to boot!).4
Needless to say, having more apps at my disposal meant a lot. While the default GNOME Software / PureOS Software stores do seem to lack a lot of apps, the above list was quite complete for me. I still do wish Signal was available on either of these phones, but I’m not holding tons of hope out that it’ll happen in any official capacity.
This perhaps might be the biggest reason that I found that the Librem 5 surprised me more than the Pinephone. While Ubuntu Touch on the Pinephone still has “more apps,” the short list above covers a vast majority of my needs (along with a web browser).
Speaking of apps that exist in Android but not on any Linux distros, Rudi Timmermans had a
GoFundMe opened up for bringing Anbox to Ubuntu Touch. It seems
like it’s closed now, but I’m fairly hopeful the work here will translate to being able to get apps
like Signal on other operating systems in the future. Of course, Anbox isn’t a perfect solution, and
lacks Google’s spyware security software called Secure Boot. Many apps for regular users (like
banking apps), or critically important apps for power users (like Snapchat) use secure boot to
ensure your phone is authentic Android (and hasn’t been rooted). I am willing to be corrected on
this, but I don’t think that Anbox includes Secure Boot in any way, so you are limited in what is
possible with it.
That said, while this is still in its infancy, this is a great example of how all these FOSS ecosystems are making mobile Linux a third option to Android and iOS.
So here’s where I start being a little more negative on the phones overall. First, the Librem 5. Its battery life is pretty abysmal, and it can get incredibly hot. I suspect these end up being related issues in some capacity, but first the battery. I unplugged my Librem 5 at ~9:30AM in the morning after charging to 100% battery, and opened zero apps (so doing nothing). The kill-switch for the modem, was turned off (so modem was disconnected), as was the mic and camera. WiFi and Bluetooth remained on, although I turned Bluetooth off in the software settings. By ~12:30PM or so, my Librem, which had been doing nothing and has no services running in the background, was at ~60% battery. That’s three hours of screen off time (and no, I didn’t have the lockscreen with the clock on. The display was blank). It’s pretty bad. I’ve repeated this a few times and its repeatable within a couple percentage points.
Even if I turn the phone on and enable an alarm, and keep it next to my bedside for the entire night, I cannot rely on the alarm actually firing, because the battery will run out and the device will die before morning comes. This phone needs to be plugged in frequently. Fortunately, updates have made it such that charging the phone is significantly easier and faster than it was back in December. I don’t think it’s perfect yet (some AC adapters like my Pixel 3 AC adapter still don’t seem to work that great), but the charging ports on my desk work great. Cable support has also improved, and more cables than just the one that came with the phone can be used now.5
Thermals are also a major concern. During my long road trip with the Librem 5, it performed quite well at playing music while I was driving. However, even with just a little sunlight in my car (and the AC going), it still got so hot I thought I had burned my hand touching the metal sides. Given that I have an Evergreen unit, I think this is something that isn’t going to be perfectly solved until a new Librem 5 with a different design exists. Most of the time I work remotely from my home, so it’s not really the end of the world for me, but the bad thermals plus a weak battery story means that you can’t rely on this thing lasting all day, and you’ll need to be careful with where you leave it (certainly not in the sun!!!).
In contrast, the battery life on the Pinephone seems much better across the board, despite being smaller in terms of overall capacity. Ubuntu Touch has the best battery life I’ve seen out of the distros. In fact, my Pinephone on Ubuntu Touch can easily last a day and a half (~35+ hours) if it’s doing nothing and the screen is off. That goes down with usage (screen-on time is still only ~4ish hours depending on brightness and other configurations), but overall I’ve been more impressed with my Pinephone’s battery than almost any other Android device I’ve had in a long time. Other distros seem to be pretty close in terms of battery life, but Phosh seems to be a little worse in most of my tests. It’s not very scientific, for sure, but I’d be skeptical if someone said Phosh outlives Lomiri in terms of battery life by some wide margin.
Thermals are much better on the Pinephone too, but I don’t think they’ve actually improved much since the last time I wrote about it. Overall, I’d love to see the phone run cooler, but I also think that the design of the Pinephone weighs into this quite a bit.
To start, the best browser out of all the options I’ve tried remains Morph Browser on Ubuntu Touch. It’s smooth and responsive even on the Pinephone, and feels pretty snappy. I don’t need apps for Twitter or Mastodon on Ubuntu touch for this reason – Morph is just better.
In contrast, GNOME Web feels like it’s gotten worse since I used it in December. I’m not 100% convinced this isn’t because I’m getting more frustrated with it over time or not, but it definitely can take over 3 minutes to load Hacker WebApp, which I consider to be fairly lightweight. I know that it’s the browser that’s the issue, because while most apps I use on my Librem 5 (GTK apps, to be specific) are smooth, GNOME Web just consistently sucks. I’m not sure what’s going on here, as it was definitely better than the mobile-skinned Firefox when I first got my Librem, but now it seems to be the opposite.
Speaking of mobile-skinned Firefox: it’s pretty decent now. Startup times for the app are still abysmal (15-20 seconds on Mobian on the Pinephone installed on an A1 microSD card). However, once it’s been loaded it doesn’t have any of the problems GNOME Web does. It’s definitely not as snappy or responsive as Morph Browser on Ubuntu Touch. However, you can install uBlock Origin so it’s really a bit of a trade-off in what you might be hoping to do.
Well, the Librem 5 still doesn’t have full-disk encryption. Apparently Byzantium, the next major PureOS release, will bring this with it, but the release date doesn’t seem locked in yet. Moreover, I have yet to be able to tell by trawling through the Purism forums if enabling encryption is going to require a full re-install. I suspect it will, which means more work for users, but that’s what you get when you play with Linux as a hobby.
I’m also somewhat excited to see how Purism utilizes the smart card reader in the Librem 5 along with full-disk encryption. This is one of those things that I’m not entirely sure is worth it for the cost (I mean, for most folks just having a separate lock screen / encryption password is more than enough), but from a technological perspective is pretty cool and unique. Nonetheless, they put a smart card reader in there, and that definitely helped inflate the phone price, so I’m hoping it wasn’t all for naught.
As for encryption on the Pinephone, the only OS that currently makes it easy to enable is
PostmarketOS. PostmarketOS is based off of Alpine Linux, and among the
Phosh-based distros is definitely my favourite. The installer image allows you to set a separate
lock-screen PIN, disk-encryption key, and SSH login. This is definitely the most polished experience
on any mobile Linux today for getting disk encryption. And since PostmarketOS also supports flatpak
/ flathub and traditional means of installing software (through apk
, not pacman
or apt
) the
end experience from a phone user isn’t much different than e.g. PureOS.
I am currently thinking that I’m going to abandon Ubuntu Touch for a while on my Pinephone in favour of keeping PostmarketOS installed on my eMMC because full-disk encryption is so important for mobile devices. Plus, while I dislike Phosh, the app experience of having Fluffychat available means a lot more to me while the UBPorts team works towards updating and polishing Ubuntu Touch. That’s not to say that I’ll abandon other distros entirely, but given the current state mobile Linux, PostmarketOS definitely is the distro to watch out for right now (even if Manjaro is the official release partner for Pine64).
This mostly applies to the Pinephone, since I have mostly stuck with PureOS on the Librem 5 at this point in time.6 I seem to have had considerable trouble with installing many of the Pinephone distros to a microSD card in the past month. I had thought it to be a case of a flaky microSD, but after rotating through a few it seems that mostly I was having trouble with Mobian nightly builds, the Mobian May-17 build (official stable at time of writing), and Manjaro Plasma builds.
I don’t think it was the microSD card at the end of the day, because PostmarketOS installed with their installer painlessly. I didn’t dig into it too much because by trying other builds or attempting the same build again at different times, I managed to get these distros working. I was using one of the following commands for all this, so perhaps this is on me?
sudo dd if=distro.img of=/dev/sdd status=progress
# OR
pv distro.img | sudo dd of=/dev/sdd
This was all done with a USB → microSD adapter, but I always had consistently good luck with PostmarketOS compared to others, so I’m not sure.
Lastly, I will note that if you want to install to an eMMC instead of the microSD card, Jumpdrive is excellent software and you should use it. You do have to flash it to a microSD to use it with the Pinephone, but it makes flashing images so much easier. Props to the team of folks who put that together, it’s a godsend.
Overall, I think if you look on a day-by-day or week-by-week scale you’re not going to see significant differences in either of these phones. However, having had my Pinephone for about a year now, and my Librem 5 Evergreen for about half a year, I’m seeing significant strides in a lot of different areas. I regrettably cannot cover everything that’s gone on in the realms of both of these phones, but there’s always a lot to talk about because of how much community force is behind both projects.
I still don’t think either of these phones are really 100% daily-driver ready yet.7 In fact, while I’ve been generally impressed with the pace of improvements from all over I think that there are still some critical areas where the phones fall short. For the Librem 5, the battery life and thermals are definitely this device’s Achilles heel. For the Pinephone, I think the weakness of some of the hardware is problematic and isn’t something that most people would want to put up with. Audio skipping and bugs related to the lack of power in the hardware are definitely noticeable. That said, this is all somewhat biased since if this was 2009 both of these phones would seem as if they’re the future. On the other hand it isn’t 2009 anymore, and these phones are here today.
Feel free to hit me up on Twitter with questions, and let me know if there’s something else I didn’t cover here that’s of interest.
Keep in mind, I had the device for a very short period of time by then. Evergreen started shipping late November, so I had only had a week or two with the device for that first review, which was very much a matter of “first impressions over the ~3+ year period since the crowd-funding campaign.” ↩
I also realize OpenStore doesn’t have flatpak at all, so this might be a bit of a dumb complaint to place solely on KDE Plasma. However, I think there’s something to say about being able to install / run software all from a single app, and flatpak support on the Librem 5 was one of the best things I got working in the last half-year. ↩
For fairness, if you purchase from Bandcamp you can definitely just download the songs locally, provided you have a microSD card for expanded storage. ↩
Fluffychat does have one very obnoxious bug that sometimes pops up, namely that text input will start entering from right-to-left instead of left-to-right. This results in everything you type coming out “backwards” since if you typed “abc” it would fill into the text-entry box as “|cba” with | being the cursor position. Annoying, and hopefully something that gets wrapped up soon, because otherwise the app feels as polished as many apps on iOS or Android do. ↩
Although, as with any USB-C product, USB-C cables are spotty as all hell :( ↩
I had grand plans to install KDE Plasma over PureOS and see if it ran better on my Librem 5, but I had such a bad experience with it on the Pinephone that I’m probably just going to wait until it’s more polished and I am ready to wipe my Librem 5 to install PureOS Byzantium with full-disk encryption instead. At that point, there will be far less regret in having to do install / wipe / install cycles. ↩
It seems that the (semi-?)reasonably priced Librem 5 will be affected by the global silicon shortage and will continue to have a long (6-month) lead time as a result. So you can’t really daily-drive this anyways, because you probably can’t purchase one. And no, I’m not suggesting the version that costs more than most rent payments. :-) ↩
I finally managed to finish working through “The Little Typer”. The “Little” series is a series of (semi-)introductory books published by MIT press, which typically use Scheme or Lisp as a vehicle to teach some interesting aspect of programming. In this case, “The Little Typer” aims to teach the most interesting aspects of dependently typed programming. The “Little” series has been one of my favourite series of programming books, and the books have always been a delight to work through. Having spent quite a considerable time working through the book (several weekends since around sometime last November / December or so), I figured I would write up a review, since I have a lot to say about the book!
The book clocks in at around 400 pages, but it’s not the length that made me spend so much time on it. I’ve read a considerable number of programming books, and I’ve worked in several languages, ranging from C, to Rust, to Scheme, and even dabbled in Haskell back in grad school once-upon-a-time. A large percentage of what I know about programming is self-study, but I like to believe that I’m relatively well informed. Needless to say, this book was dense, and it was considerably harder to read for me than any other books in the series have been in the past.
For this review, I wanted to go through the parts of the book I liked, some of what I didn’t, and advice to people who might try to work through the book themselves. If you just want to skip to the end, see my overall thoughts.
To be entirely honest, I did not pick up this book for the topic itself. Mostly , I read this because of how much I enjoyed the rest of the “Little” series, and I had some high expectations. In fact, when I had picked this book up, I hadn’t really known what dependent types were, or why one might be interested in them. I had known the book was about types, but I had actually thought it was going to be more inline with something like Shen, and discussing some of the more interesting points behind Hindley-Milner type systems.
To put it bluntly, this is not at all what the book is about. It certainly touches the boundary between strongly-typed languages and Scheme, but dependent types are different than just stapling Haskell and Scheme together1. Instead, the book opened my eyes to a very different concept, in which types can be formed around something that is not a type (usually, a value). I’m not wholly disappointed, but I think not having formal education in programming langugage theory certainly didn’t help my confusion around the topic of the book.
Anyways, now that I’ve read the book, I think I can say with confidence that I now understand that dependent types are cool. However, they also appear to be a lot of work. I can categorize the main things I learned from this book in the following sections.
Using types to prove something is equivalent to producing a function that determines that proof. This is a bit abstract,
but I think is part of the main thesis of the text. There’s a particular moment in the book where you define a function
called even-or-odd
, which not only proves that every Nat
(natural number) is either even or it is odd, but can also
given any number can tell you if that specific number is even or odd. Same code, but there’s two different ways to think
about it.
Using dependent types, you can implement a different program by first proving that two smaller programs are the same,
and then replace
-ing an easy-to-write program with a harder-to-write program in a correctness-preserving way. This was
the main thesis of chapter 9, which I will probably speak more on later.
In any case, the interesting bit here is that by writing a proof that two programs are the same, we’re able to not only
guarantee that a transformation preserves the semantics of the program but also use that proof to do the transformation
itself. This is… admittedly still very abstract. The book spends time in Chapter 9 to show off how one might do this
with two procedures, twice
and double
, and a third procedure twice=double
that relates the two.
Given that “[o]ptimization is always just a few correctness-preserving transformations away”, this is very interesting! The idea that you can write programs this way is something I haven’t done before, and I don’t think is possible if you’re not deeply embedded into the Idris or Coq ecosystems.
The main bridge between types and values today is a lack of induction on types. Especially when working in C or Rust, we tend to distinguish between “runtime” and “compile time” concerns. One of the chief advantages of Rust over C in this respect, is that the more powerful type system can sometimes help us push many of our errors from runtime to compile time. This helps us specify what an “incorrect” program is, since an incorrect program will fail. Rust does this in a lot of ways2, but there’s no way for the compiler to know about user-defined runtime values.
Values have types, and we can make judgements about those values (and their types). Together, this is how the book uses induction to effectively say: well, I don’t know what that value is yet, but I can break this type down and try to reason about the possibilities. This is akin to what is done in first-order logic, and I noticed a lot of similarities to relational (i.e. mini-Kanren) and logic programming styles, although the types do add a bit more theatre.
In any case, dependent types are all about bridging run and compile-time together. Why make assertions about just your types when you can make assertions about values as well? Or rather, assertions about every value of a possible type, or every type of a possible value? Or assert that a value must exist that has a property, etc.
One thing I really enjoyed about the book is that it provides clear and concise names for many of the underlying concepts in type-theory. The first few that it throws out are constructor and eliminator, which actually helped me formulate a lot of thoughts around structuring types in other languages. In particular, languages often try to separate different kinds of functions as one of the following:
…and so on. Being able to say a function is either a constructor (creates a value of a given type), or an eliminator (picks a new expression based on a value of a given type) seems semantically useful. I no longer need to think in terms of getters, setters, accessors, properties, etc.: only eliminators. Of course, for the sake of my peers who have not yet had this revelation, I will probably still maintain this vernacular.3
Normal and Neutral forms of an expression are also very useful! This is effectively a distinction between known values (normal) and runtime-specific values (neutral). However, being able to identify when a value is neutral has at the very least given me a clear and concise way to express that I can’t use types as a solution in a non-dependently typed language.
There’s a wealth of small bits of natural language scattered throughout the book that invoke similar feelings. While not strictly about dependent typing (the above terms could all be used to describe Rust or C code, for example), I thought it was a valuable aspect of the experience.
There was a lot of good insight into the book. It challenged me a lot. However, there are still things with which I remain unsatisfied by.
Chapters 8 and 9 were the most frustrating chapters for me. They were probably also some of the more important chapters of the book, in that they went very deep into introducing same-ness, equality, etc. In fact, one of my most important insights from the book was about how dependent types can be used to make correctness-preserving transformations! That said, I feel like these chapters were the weakest out of the whole book.
Notably, the textual descriptions of what was going on seemed to be lacking. It was very unclear early on why replace
was needed, and the textual description is pretty bad. It would have been helpful had there been more small-scale
examples of using replace
, or just not having cong
as a distraction at all, perhaps.
I think this distracted me pretty hard when I was going through these chapters, and I felt that the final implications
of what replace
can do given a proof that two values / types are the same was not made clear enough. These are easily
the weakest chapters of the book, and the worst part is that they’re right smack in the middle. If you can grasp
ind-Nat
and ind-List
you can probably work through every other inductive eliminator. However, I think the book
definitely shows some weakness in trying to introduce same-ness, equality, and replacing types in expressions.
I would have loved to see this done more completely, but it really isn’t until later chapters that types like Absurd
and Trivial
are introduced, and you don’t even really get to the point regarding same-ness vs. equality until page
323, which is in Chapter 15.
This is going to be very subjective, but by the end of the book I felt that the whole thing could have been ordered
differently. In particular it felt weird that the entire introduction chapters discussed Pie the language, then moved to
induction over numbers / lists / vectors, then went into types like Either
or Trivial
or Absurd
.
I think making small assertions using induction on Either
types would have been a bit more friendly early on,
especially with regards to induction. Starting with natural numbers seems small enough to be presentable while also
interesting, but by the time I got to the section of the book that dealt with Either
/ Trivial
/ Absurd
, I felt
like there wasn’t nearly as much to say.
I would grant that perhaps Absurd
belongs later in the book, since making negative assertions (i.e. not-X) gets into
some pretty weird territory. That said, if your goal is to teach induction, doing so with Either
is certainly easier
than doing so with Nat
. I found the later chapters easier than chapters 8 and 9, so perhaps I was expecting the
difficulty curve to be a bit more linear.4
Touching on this point and the next one a little bit together: parts of Pie like which-Nat
are effectively just a
reduction into ind-Either
. Perhaps the author doesn’t have to focus on every relationship between concepts in the
book, but I certainly feel that it would have made the concepts a lot less abstract.
This is mostly a complaint about symm
, more than anything. symm
is introduced as a way to invoke a “symmetry”
relationship over an equality type. Basically: (= T from to)
is the same as (= T to from)
. This is usually a pretty
useful feature in logic systems that is often taken for granted.
symm
is introduced near the end of chapter 9 to be able to describe to Pie that twice=double
is the same thing as
double=twice
. They’re equal, they’re symmetric, right? Well, guess what, that’s the last time you see symm
get used.
No, seriously, it’s only a brief mention at the end of chapter 9 and then it disappears! It is literally only on page
217, and it’s gone. I get that it exists because otherwise going through every motion to re-define double=twice
when
you already have defined twice=double
would be a lot of work; yet, it seems a bit of a distraction that it is included
in Pie the language by default.
I had expected the idea of symmetry to show up again later in the text. In a later chapter, there was a need to define
zero-not-add1
, which is an assertion that zero is distinct from any number that has had 1 added to it. Even later, the
book needs us to define add1-not-zero
. The minute this came up, before even reading the next dialog, I was convinced
this was a problem with symmetry, and the book was going to demonstrate how to define symmetries over absurdities.
Unfortunately, the book just redefined the same function (albeit it’s a 4 line definition) with the add1
and zero
in
reverse order.
Needless to say, I was a bit disappointed because when I took first-order logic it was fairly natural to abuse notions of symmetry, commutativeness, etc. Here, we just took the easy route. I’m sure there’s an explanation for why this isn’t brought up at all, but even a footnote would have been satisfactory to describe why we couldn’t apply some form of symmetry to this problem.
Early in the book, you’ll encounter the phrase “Recursion is not an option.” This is asserted over and over again, as if it is a cute joke about how the argument recurs onto itself. I found this very annoying, because the book doesn’t really ever give a proper explanation as to why recursion is hard when working with types in this way. There’s the introduction of “primitive recursion,” and induction as it is used in this book is a form of recursive reasoning (rather, recursion is a type of inductive problem solving), but alas, you only get “recursion is not an option.”
Eventually you might read beyond the book and discover why other languages like Idris or Coq or whatever have limited recursion, and need to make very specific guarantees about recursion when it occurs. This is distinctly due to decidability / completeness problems that have yet to be solved. The Little Typer, however, doesn’t even attempt to explain that this a challenge, it just meaninglessly asserts that we can’t use recursion. I hardly feel as if a Y-combinator is out of reach in this situation, and I’m also certain it would break some of the guarantees made by Pie.
Again, even a note at the end of the book that explains this directly would have been welcome, but there doesn’t appear to be such a thing. Maybe this expectation is too harsh to lay on the authors, but it did stand out to me at least as an annoying omission.
Many of the types you implement are probably not going to be very efficient in the name of pedagogy. If a function asks for a natural number, you may be inclined to put a very large number in. It is probably best not to use any number over 1000 if you don’t want Pie to ravage your CPU.5
The preface for this book asserts that all you really need to write and understand the code in the book is the first four chapters of The Little Schemer. It is highly probable that this is insufficient if this is your only programming experience. It is certainly all you need from a “how does this evaluate” perspective, but you’ll be missing out on a lot of context if you’ve never used a statically typed language before, or never structured a proof before.
Many of the insights I gained from this book were a direct result of having worked with many languages in the past, specifically in the context of having thought about types and type systems. To be entirely fair, you don’t need my experience to enjoy this book, and I probably have some level of assimilation bias due to the languages and tools I work with every day. However, if you’re coming at this and you’ve literally only ever used Scheme, and only worked through The Little Typer, you’re going to have a rough run of it. I also think you’ll be missing out on some of the key parts of the book that I enjoyed.
If I wanted to prescribe a minimum set of “what you need to know to get the most out of this book,” I’d probably recommend:
Without all the above, I don’t think I would have enjoyed this book or found enough merit in it to continue past the first couple chapters. Dependent types are cool, but the reasons I think they are cool depend 😉 on the context I had built up surrounding proofs, type systems, etc.
The book aims to be an introduction into dependent typing and into structuring your types as proofs. The book does a really good job at a lot of this, and is very good at providing names and natural language for describing the process as you learn. In genral, I would say I learned a lot from this book, and while I will probably take a break from dependent types for a while, I can see why dependently typed languages are pretty cool, and learned some of the unique aspects that make them interesting.
The book does have a fairly steep learning curve if you’re not familiar with proof systems, first-order logic, or typed
programming languages in general (more specifically, typed programming languages that use types in a more rigorous
fashion than say, C). If you’re looking for the book to connect every dot and line in the realm of dependent types, I
think you’ll be sorely disappointed. There are some concessions the book makes for brevity (symm
, not explaining why
recursion is not allowed), and you’ll have to live with those. Overall, what annoyed me the most were chapters 8 and 9,
which seemed to be far less direct than many of the previous chapters. There’s a lot to learn there, but I retain my
belief that they are the weakest chapters of the book (despite being very critical to later sections).
If you’re looking for a programming book to challenge you, or you just have a natural curiousity, I highly recommend this book. The whole “Little” series is pretty incredible, and this book doesn’t disappoint with both small insights you can inject into your current programming practice, or big shifts in how you think about types vs. values. Be warned, it is not something you’ll consume in a weekend or two. If you do though, feel free to @ me on twitter and brag, I will not be upset.
Having gone into this book expecting something completely different, I’m still happy I stuck through the book. I learned a lot, and despite my grievances, I feel like I still understood the central theme. While I don’t see any immediate practical first-order effects from the book (I am not, for example, going to try and convince my team at work to use Idris), I feel that some of the intuitions have already helped me in thinking about some of the problems I encounter.
In reality, the Pie language used throughout the book is more like stapling together Idris and Scheme. Idris 2, as I understand, is actually implemented in terms of Chez Scheme. They may not be so different after all! ↩
Rust does this in a lot of ways. The borrow-checker is one, in that null pointer checks and lifetime semantics are
represented as polymorphic types (i.e. 'a
or 'static
or whatever) on references / pointers. Strongly typed
enums are another way, if you want to guarantee that you
always check every possibility of a condition.
However, Rust still cannot perform induction on types and thus cannot fully check every part of your program. As an example: Rust cannot at compile time check if a number is zero or not-zero, and then branch on different types as a result of the value of that number. Dependently typed languages with induction are needed for this in a general form. ↩
I do understand why languages feel the need to be more specific about the difference between a function and a procedure, or a method vs. a constructor, but I also feel like this is one of those distinctions that nobody gets correct. When I talk to people who write primarily in C++, everything is a function. In Java, everything is a method. In Scheme, a procedure. Semantically there are differences but I have found pushing the discussion more towards the constructor / eliminator dichotomy has helped in some cases.
It is much clearer to say “we need to write a new constructor from this type” or “we need an eliminator to access the internal resource X” than to say “write a function doing X.” One problem that I see a lot with junior programmers is writing methods instead of constructors, resulting in types that can contain a lot of invalid states. This isn’t advice per sé, but I thought the distinction was a cool and useful insight in the book that was never directly pointed out to me before. ↩
There’s a lot of interesting “either-or” logic that could have been done by compounding the Either
type over
Atom
s. Had I written the book, I probably would have had at least a chapter early on to demonstrate what induction
is by limiting the scope to a set of binary choices, and then expanding this to numbers.
After all, if you wanted to test “either-or” relationships on numerical values, you’d need a nested Either
type
for every value, sequentially. Of course, this would get burdensome quickly, which leads you to introducing
ind-Nat
as a more natural (hah, get it) way to do induction on numbers. As I mentioned later, which-Nat
could be
more or less implemented in terms of Either
.
Perhaps the error in my thinking here is that I’ve been pre-disposed to a lot of programming and logic over the years, and I have some kind of assimilation bias. ↩
This is particularly important for the even-or-odd
function. I put a number over a few billion into it and very
quickly realized I had made an error in judgement. ↩
Recently I fixed a bug in librealsense21. The core of the bug was that the developers used a C++ functional cast expression in a C header (C does not have functional cast expressions!), which broke the realsense-rust wrapper we’re developing at work. Worse yet, this was shipped as part of the official 2.42.0 release of librealsense2. Oof.
The fix is pretty clear: use a C-style cast instead. Better yet, follow the advice in Modern C and don’t use casts at all. We can’t always avoid them though, and the compiler can make the whole process annoying. At least, it becomes annoying and difficult to detect such issues when you use a C++ compiler.
I’ve shipped an SDK before myself, and that ended up getting me thinking: I reject the premise of the fix in the first place. This code shouldn’t be broken, and it is frustrating that it is. The real sin here was shipping inline functions in a public header for C code.
If that’s confusing, let me expand briefly. Inlining code is a common trick for trying to improve the performance of code by having the compiler copy the implementation of a function and apply it in place (with no additional stack frame). This can improve performance of small code sections, and is critical in optimizing tight loops that centre around small functions. I’d go into extreme depth, but Modern C2 actually has an entire section on this, and it explains it a lot better (look for section 15.1 on inlining functions).
Anyways, the trade-off of inlining functions is that you typically have to place them in the header; alternatively, in
the same source file they’re used in if you’re not exposing the inline
‘d function publicly. Note that this is entirely
about “performance” and not “ergonomics” or “making it easier to code.”
In the scenario where I had to fix the bug above though, there’s a few differences. Rather than directly inlining the
functions, the author of the code chose to make them static
instead. This behaves somewhat similarly to inlining the
code directly but it isn’t exact. In this case, the static
functions in that rsutil.h
header:
I suppose the first problem above is a small trade-off as long as the code works the same, but the second one is definitely not something you expect. Particularly, this header is shipped as a C header to librealsense2’s C-API. If you understand C by reading the header, you’ll probably notice the subtle difference, but even still it’s not something that sits at the front of your mind, and may surprise you if you rely on this header in many places and decide to use the function pointer.
Eventually C introduced the inline
keyword to specifically handle these trade-offs when defining functions this way.
Thus, you can get the performance improvement of inlining small functions without the above problems with static
functions. However, even if we swap out the static
keyword for inline
we aren’t really solving the underlying
problem at all.
inline
keywordThe real issue is having those definitions in the header in the first place. By having all that code in your header, you’re shipping a dependency that is basically promising “this code will not break on your compiler.” You don’t know what compiler you’re shipping to; even if your ABI is stable you still have to worry about what standard the language supports, what extensions are available, etc. Inlined definitions are often done best if your code isn’t likely to change ever if at all, but here you can’t be sure.
If you take away anything from this, it should be the importance of not breaking downstream code because of inline definitions. By defining another C file and moving these definitions out of the header, you would probably lose very little, while making it easier to generate FFI bindings in other languages, as well as allow your end users using C to well… use C.
This ended up breaking the FFI generation by bindgen in Rust for us, because the inlined definition changed (and changed
to something that wasn’t standard C). That said, it’s just a bad idea overall. Most of these functions are quite large
already, so chances that the compiler inlines them directly (especially without the inline
) keyword are small.
Moreover, most people doing performance critical code with functions like rs2_project_to_pixel
or
rs2_project_color_pixel_to_depth_pixel
are unlikely to be banking on the fact that your utility functions are getting
inlined and operating at max efficiency. Just put the definitions in their own file, and only inline the code when you
can clearly benchmark and measure a strong use-case for doing so. Above all else, make sure you understand the language
you’re shipping to when you make that change.
I certainly disagree with the way the code is structured here, but its not an easy problem. It also isn’t something that stems from bad developers, so I can’t accuse Intel of brash incompetence. What likely happened here is that the engineer who made the original change works in C++ for 99% of their work day, and only touches the C headers occasionally when they get tired of compiler warnings or if something significant changes at a lower layer. It’s really easy to make this mistake, even if you do commit to best practices (thanks C!).
At the heart of the issue, working on an SDK is hard and often thankless. I suggest any team that might be shipping C headers take the above advice, and take care to watch that:
inline
is an effective tool for performance, not an easy way to avoid committing another .c
file to your repo.static
storage class on functions means something specific, and has downsides when used the same way as inline
.We do use Rust at Tangram Visions, but sometimes we still have to understand C and C++ to be able to interface with existing code. ↩
I can’t say enough good things about this book. C is complicated in a lot of ways, but if you break things down piece-by-piece you’ll find it’s actually way less bad than many make it out to be. I still love Rust & its type system significantly more, but I’ve worked in C / C++ in the past, so I know from experience where things can start to go awry.
Anyways, I highly suggest giving the book a read, you’ll probably learn something! Jens Gustedt is very well informed and while some opinions (e.g. don’t use casts) are likely to start a flame war, I tend to find myself agreeing with the final assessment. ↩
The (current) extended public health crisis has given me a gratuitous amount of free time. At least, comparatively to before the public health crisis. This past weekend, bored out of my mind, I came across a post from the K-9 developers on Hacker News. This got me thinking a lot more about email and how my use has changed over time.
I use K-9 on Android, and if you do too, you should definitely help fund the app. It’s one of the few choices on Android for email that doesn’t suck. However, this isn’t really about K-9 or their funding. This article kind of got me thinking about email and how I use it, and with an idle mind that had no other immediate priorities I started re-evaluating how I use & interact with email.
I’m writing this mostly because my blog is a blank canvas for my thoughts, and because I can.
In recent years I’ve been taking this attitude more and more. Email certainly is the universal platform (everyone has at least one email address), but I don’t send an awful amount of email myself. Back when I was in grad school email was literally the only way you could talk to some people. I met professors, other students, and faculty from all branches of the university that would put almost their entire lives into email. Nowadays, I suppose this is more likely to happen over Slack, or even Element?
I send a lot less email than I used to. I also receive a lot fewer emails than I used to, or at least, it feels like I do1. All my work communication is handled through Slack, and most of my personal communication will be over Signal or Element. Despite this, email is still pretty good at a number of things, and its not like I could see myself getting rid of it. Ignoring “you need an email address to sign up for X service,” email is still king at:
There’s good reason to think email isn’t going away anytime soon, so I’m fairly confident that lots of people do care about email, and will continue to do so.
Thinking about email and how I still use it, I was reminded that Hey exists. Hey is a subscription email service (think, paying for Gmail) that aims to do email radically differently than everyone else. There is a limited amount you can change about email from the get-go, but I went back and watched an entire 37-minute video of their CEO explaining how Hey is different than regular email.
The video is long, and touches on a lot of features. I don’t personally see myself ponying up ~$99 USD a year for email, but I think there’s both some good and bad ideas in there. Some of that probably depends on the volume of email you receive, and like I said, I don’t receive lots. Nonetheless, Hey looks to me more like they changed the frontend for a web-email client more than they changed anything about email itself. This is completely fine, and is not really to detract from their product (a lot of the choices are good!), but don’t go in expecting the behaviour of the protocol to vastly change.
Hey gets a lot of things right. I really like the idea of “surfacing” attachments and details from emails (i.e. displaying them as a separate category and making them much more searchable). If I ever go neck deep into using email to manage my life, this is something that would be a game changer.
The idea behind “the feed” is also really good. I’ve signed up for a few substack lists as well as some great columns like Matt Levine’s Money Stuff. The video demo that Hey shows is a bit wrong, as I don’t expect that most people would want to use the feed for marketing and advertisements, but maybe the way I use email is weird. Nonetheless, the concept of having a folder / dedicated filter for feed items seems both plausible and pragmatic to me. This also made me feel a little dumb because I had always separated all feed items / mailing lists into separate folders. For open-source mailing lists I still do this (don’t cross your chickens with your guile users, you don’t want anyone getting salmonella!); however most of the “feed” columns I subscribe to don’t post nearly often enough to make my feeds folder explode. And to the point Hey makes, you shouldn’t have to stress out about this folder or care.
Similar logic applies to their concept of a “paper trail,” but I’m also convinced that’s something that has to be mostly curated manually since automatic filters for sorting receipts & invoices are… ripe for error on that one.
Lastly, the idea of being able to permanently ignore threads is 💯 (the best). There are so many instances of people not knowing the difference between “Reply” and “Reply All” that I can veritably say that the lack of this feature in every other email client has driven someone mad and / or driven someone to death. Hard to quantify, but the probability seems high enough at scale. Anyone on the Thunderbird team willing to integrate this without having to make thread-specific filters? Anyone on the Dovecot team willing to make this a plugin?
Probably? Email obviously isn’t a secure communication medium, and the protocols are pretty old school. JMAP seems like it has it’s heart set in the right place, but I am skeptical of most of the internet moving in that direction for… oh the next 25 years maybe? Years may be a bad prediction metric, so maybe we say instead that Google will have launched and killed at least 10 more chat applications by that time?
I think something more interesting surrounds the fact that most people suck at email. I cringe whenever I see an unread count in the tens of thousands. Some people believe that to be a badge of honour, but I’m fairly certain that just describes you as someone who doesn’t care, and I generally take the position that it’s not a virtue to be proud of how much I don’t care about things.
For a lot of people who are hopeless with email, an offering like Hey might be good. On the other hand, I think if people grew up learning even the base level of email skills (etiquette?) then a significant part of the world would be a lot better off.
Knowing how to use email effectively is probably a superpower for some folks in some organizations.
These are just some of my tips, but they certainly helped me get more proficient at email over time.
Upfront: Maybe this isn’t the most universal advice. I got a lot farther with email when I started leaning heavily into using Thunderbird rather than trying to keep up with every change to the Gmail interface. What makes a good client is somewhat subjective, but I’d say that working with web clients has overall been a poor experience for me.
For every feature a web client has, it also has a million ways to be distracting or use 90% of your CPU in a single tab of your browser. “Hey” might be different in some regards here, as their UI seems quite intuitive at a glance, but you’ll pry my desktop native (read: not Electron) client from my cold, dead hands.
This is just good writing advice but: focus on content over form. The number of random business emails I get with fancy HTML signatures, tons of markup, etc. are ridiculous. Stop wasting yours and everyone else’s time, please, and just write the email. If your email requires more formatting to be readable, then you’ve already failed on the content front.
Being polite is a skill unto itself, but your email is (usually) not a thinkpiece. Just write the text and focus on the message. The people writing fancy substack newsletters and such obviously don’t need to follow this advice, but most people seem to think that plain text emails are somehow… bad?
This goes back to the idea of “feeds” or a “paper trail.” Message filters and folders are a great way to get unimportant stuff out of your inbox. It’s not even something that should take you a full afternoon to learn.
The biggest downfall of this is that message filters can’t be easily translated from one client to another. Likewise, server-side options with LMTP / Sieve or whatever are still complex as all hell to set up, and then you still need to understand how to write the scripts to do the email filtering. This is more a complaint for someone who has their own mail server and uses multiple clients (K-9 on Android, Thunderbird on Linux, Geary on Phosh-based distros).
Most people who want to set up filters in Gmail should absolutely just go do that via the web interface, since that will happen before mail reaches your local client. The filters you choose to set up or how you organize also doesn’t matter. I think the feeds and paper trail ideas have some merit, but the main point is stop pretending your inbox is the only folder that can ever exist.
And on that note: please for the love of god filter and silence automated emails (think: a CI pipeline), or at least funnel them to a different folder. Your inbox is not a notification bar, and on pretty much every mobile client ever you can have alerts get sent to you either 1) not through email or 2) with a filter that weeds out noise. So many people hate email for this reason alone!
I mean, this speaks for itself. Use Org mode, use one of literally hundreds of TODO apps, use a paper and pencil! But don’t use email2.
People always tell me inbox zero is hard. It’s not hard, just right click your inbox, mark all as read, and archive all3.
🔥 🔥 🔥 🔥 🔥 🔥 🔥 🔥 🔥
Exchanging my inbox as a-place-where-email-goes-to-die with my archive isn’t immediately the best choice, but my inbox is a working directory and I have filters for things that shouldn’t be lumped into the archive, so the difference comes out to be a positive impact at the end of the day.
This is maybe something I’ve picked up on since I work from home now, but most email isn’t urgent. If you need to urgently contact me in a personal manner… there are ways and that is kind of why the telephone network exists. If you need to urgently contact me professionally, my work email is open when I work but you’re still better off bugging me elsewhere.
While I do adore K-9 for being the best mobile client for Android out there, I actually think mobile clients for email pretty much universally suck (sorry if your business is mobile email clients, but I’m probably not going to be your customer). Email is best experienced with a physical keyboard, and most of the time I won’t respond to your email if I don’t have a keyboard. I used to do a lot of email via mobile, but I’ve since moved away from doing so because I’ll almost always have either my Pinebook Pro with me or I’ll be at home with my desktop.
I’d be interested to see if there’s others out there using email in a more professional or at least more sane way than most people do. I’m also interested to come back to this article in another five years and see if my current choices have held up or if the way I work with email now is just a function of the volume of email I don’t receive.
Give me a shout via any of the usual modes listed, and let me know what you think.
I should probably measure that, seeing the stats would be cool! At least, they’d probably more useful than my end of year Spotify stats, and I care enough about those to get a little interested at the end of the year, so it wouldn’t be entirely wasted effort. If you’re someone looking for a project, make this! This is absolutely something that would help me figure out how to optimize (read: stop reading) my emails. ↩
Cue the guy who has an org-mode TODO -> email integration set up in Emacs to tell me how he automated his TODO.org to get sent to his email every morning so emacs can parse that email and automatically do every task for him. All written in emacs-lisp and some C, naturally. 😂 ↩
Obviously, when I say this to most people they act like this is the up-front “nuclear” option. Look, I don’t need to tell you how to live your life. However, if you’re going to let email pile up in your inbox to the point you can’t find anything and then you’ll feel bad about the fact you can’t find or reply to anything, then those emails are no worse off in the Archive, and at least the next emails that come in can be dealt with properly and you won’t have to feel bad about those. ↩
Since starting my new role at Tangram Vision, I’ve been doing a lot of programming in Rust. This is a bit of a change for me having mostly worked in C / C++ / C# over the past 6 years, but my impressions overall are quite positive.
One of my favourite features of Rust are the enum types. They don’t stick out among Rust’s set of language features compared to some of the more novel aspects of the language (borrow checker, lifetimes, safe-multi-threading); however, Rust’s enum types drive some of the coolest parts of the language, and make modeling data in terms of types a pleasure.
So I wanted to write something to point out some of the reasons I like Rust enums so much. There’s gonna be a lot of “compared to C / C++” in here, so do be warned.
First and foremost, an “enumeration” or enum in Rust is a kind of sum-type. Rust utilizes an algebraic type system, so there’s a few different ways to specify what the “type” of some data is.
Category | In Rust | Definition |
---|---|---|
Atoms | i32 , u64 , f32 , etc. |
Atoms are types whose values evaluate to themselves. These are more or less the smallest types you’ll encounter, but they are the building blocks for everything else. |
Sum types | enum |
Sum types are types where you can choose one of a set of possible states. The number of possible states a variable can have is the sum of all options. |
Product types | struct |
Product types are defined similarly to sum types. The number of possible states a variable can have is the product of all combinations of data in the struct. |
Generic types | SomeType<T> |
A type that is defined in terms of some other type (or types) T . |
This may be a bit hand-wavy, and I certainly didn’t cover all possible types (I ignored traits and functions, for example), but it’s a good start to getting to know about enums.
Rust enums, or sum-types in general, make the most sense when you want to
represent a “choice” as a type somehow. The easiest example to represent this
in Rust is the Option<T>
type, which is a generic enum roughly of the form:
pub enum Option<T> {
Some(T),
None,
}
This is an enum that represents whether or not you have Some
of some T
, or
None
of it. Other enums are a “choice” in the same way. Every variable of
your enum type can only be one entry in the enum type at a time.
let x: Option<f32> = Some(42.0);
let y: Option<f32> = None;
While x
and y
above are the same type, they have to be either something
(Some(42.0)
) or nothing (None
). They can’t be both.
Based on what we saw above, there’s a couple of interesting ways we can write an enum type.
This is closest to what C++-style enum classes are like, although unlike in C or C++, these aren’t automatically mapped to a set of integers.
enum Color {
Red,
Green,
Blue,
}
If you’re coming from C you might think: well, why aren’t these mapped to
integers? Shouldn’t Red
be zero, Green
be one, Blue
as two, etc? Unlike
C, Rust’s type system isn’t entirely based on integers. Types have
significantly more meaning, and it’s difficult to explain this succinctly.
Instead, Color
is a type in your program where Color::Red
is a distinct
value from Color::Green
is a distinct value from Color::Blue
, in the same
way that 0 is distinct from 1 or 2.
If one wants to, there are crates for mapping integers to enum values. This can be useful if you’re serializing to some low-level protocol, but isn’t very interesting otherwise.
One thing I’ve avoided until now is talking about union
types in C. C has
enums and unions, which are two separate kinds of abstractions. C-enums give
names to a list of integer values, while C-unions provide a weak kind of
sum-type.
The weakness of C-unions is because while you may have a union which can map over one option among a set of types, you don’t know which type the union was actually mapped to. Example:
union my_union_t {
int a;
float b;
const char* c;
};
In the above example, you can’t know when you receive a my_union_t
type
whether it was created by setting some value a
, b
, or c
. You know it’s
one of the options, but not which one. A typical way to work around this is to
type-tag your union with an enum, and pass that around too.
enum my_union_type_t {
A,
B,
C
};
This means you’re passing around two values just to be able to understand what your compiler should already know for you! This “idiom” in C is also error-prone: what if you send the wrong enum value alongside your union? Undefined behaviour, that’s what. Rust’s enums are much more convenient, and allow for combining data alongside the enum options themselves:
enum MyType {
A(i32),
B(f32),
C(String)
}
Because of this, you never need to worry about mismatching your enum / union types. Likewise, Rust doesn’t need a keyword for unions since the above example does exactly the same thing as the example in C. I think this was the first thing that really struck me about how cool enums in Rust are, because manually tagging and passing around unions and enums in C / C++ is a huge pain and source of bugs.
Similar to the above, we can give the data we pass around with our enum types names:
enum MyTypeWithNames {
A { value: i32 },
B { quantity: f32 },
C { message: String },
}
Giving names to things can be a good idea, especially if you’re passing around a lot of data inside your enum, or passing around multiple values of the same type. However, sometimes the best name is no name, so don’t overdo it!
One last fun fact: all of these can all be interchanged. Sometimes, depending on what you’re building, you might even be able to build an enum like this:
enum PixelFormat {
Yuv {
y: u8,
u: u8,
v: u8,
},
Rgb {
r: u8,
g: u8,
b: u8,
},
Greyscale(u16),
Unknown,
}
It might seem like this complicates your code significantly, but we have to choose to live with complexity at some level. A bit of extra complexity in the definition drops a significant amount of complexity in how we pass around and use these types.
Without trying to (entirely) pitch you on Haskell strong type systems and
pure functional programming, enums in Rust are exciting because of what they
make easy. Rust uses enum types throughout the core language, and many of the
common idioms utilize enums in some form.
And if I’m being honest, equivalents can be made in C or C++ but the ergonomics and language integration just aren’t there. Passing C-enums or C++-enum classes alongside unions is vastly more error-prone.
Option<T>
Rust doesn’t have “optional” or “defaulted” arguments in functions like C does.
But we don’t need it! Instead, we can use the generic Option<T>
, to represent
whether or not something is required. In function parameters:
fn add_three_numbers(a: i32, b: i32, c: Option<i32>) -> i32 {
if let Some(value) = c {
a + b + value
} else {
a + b
}
}
While a bit contrived, when calling this you might do:
let x = add_three_numbers(13, 12, None); // => 25
let y = add_three_numbers(13, 12, Some(4)); // => 29
Similarly, if you avoid using unsafe
Rust and raw pointers, the default Rust
types for pointers (Box
, Arc
, etc) cannot ever exist as a “null” pointer.
Instead, if you want to represent a pointer that can be null, you use
Option<Box<T>>
, and null pointers are represented as None
! Unlike in C or
C++, this makes the type of pointers very explicit. You know that you have to
handle the enum differently than you would the pointer itself, and once you get
a Box<T>
you know for certain that it points to something. This is one way
safe Rust avoids null de-references!
Result<T, E>
Similar to Option
, Result
is Rust’s standard error type. It is a fairly
straightforward enum that roughly takes the form:
enum Result<T, E> {
Ok(T),
Err(E),
}
In this case, Result
is usually used as the return value from a function,
which tells you if it succeeded and you get an Ok(T)
back, or if it failed
and you get some error Err(E)
back. In practice, it’s pretty cool how
flexible this is for error handling. It also avoids all the problems with
C++-style exceptions.
By default Rust has built in support for deconstruction / pattern-matching enum types (even ones you define!). From our earlier example we can do:
let x: PixelFormat; // Assume this is given a value from something
match x {
PixelFormat::Yuv{ y, u, v } => {
println!("Pixel is y: {}, u: {}, v: {}", y, u, v);
}
PixelFormat::Rgb{ r, g, b } => {
println!("Pixel is r: {}, g: {}, b: {}", r, g, b);
}
PixelFormat::Greyscale(g) => {
println!("Greyscale value is: {}", g);
}
PixelFormat::Unknown => {
println!("Oh no, we don't understand this pixel format!");
}
}
While Rust doesn’t have switch statements that operate over integers like C does, this is a very close equivalent that semantically allows for some much more powerful abstractions.
As I said originally, enum types represent a “choice” of some kind when
programming. The match
expression above demonstrates how we dispatch a
“choice” in our program according to our type.
Enums in Rust unify the concept of enums and unions in C-family languages. They are more expressive and reduce a lot of complexity in managing state or tracking a union through a program. They also power some of the core types that are used in almost all Rust code.
Enums are one of my favourite parts of moving to Rust from C++. They are used in error management, optional arguments, and enable a whole host of different representations for data that aren’t ergonomic enough to see common use in C++.
]]>Huzzah! I recently received my Librem 5 (Evergreen) from Purism. The Librem 5 is a smartphone that runs an otherwise standard linux kernel. However, unlike Android which also relies on the linux kernel under the hood, the Librem 5 uses a GNU userspace, adapted for mobile. This makes it more akin to your typical laptop in some ways, although the form factor still resembles a modern smartphone (at least, mostly). Here are some preliminary thoughts about the phone and how it compares to Pine64’s Pinephone, which is another phone that uses neither Android nor iOS, and relies on a GNU / Linux based OS.
This is a bit of a long post, and I’m sorry for that. Brevity is certainly a hard-earned skill, but going forward without comparing the Librem 5 to the Pinephone seemed like the wrong choice ;)
Purism touts the Librem 5 as a “Security and Privacy Focused Phone.” This is not wholly untrue, and there are some features that are unique to the Librem 5 in this respect. For example, there are “kill-switches” on one side of the phone that allow you to disable:
There’s an additional feature where if you disable all three switches, then other sensors on the phone such as GPS are all disabled as well. You’d be better served reading through the Librem 5 product page to learn more.
In contrast to the Librem 5, which touts itself as being privacy and security focused, the Pinephone is touted as “An Open Source Smart Phone Supported by All Major Linux Phone Projects.” It is considerably different from the Librem 5 in many ways, from the hardware all the way to the final software. It too has kill-switches, but these are not easy to access on the side of the phone and instead are inside the case of the phone. You can access these kill-switches by removing the back plate.
One of the neat things about the Pinephone is the ability to boot and use a variety of Linux distributions on the hardware itself. It’s as easy as loading the distro onto an SD card, and then booting the phone. You don’t even have to flash anything to the internal eMMC in order to run it!
The Librem 5 starts at $799.00 USD, or can be purchased at $1999.00 USD if you want your Librem 5 assembled in the good ol’ US of A before they ship it to you. There’s a lot of talk about anti-interdiction or controlling the supply chain, but trust me in saying that you’re definitely not in the market for a $2k phone like the Librem 5. I don’t know anyone who is.
I could probably write an entire post on the economics of this, but quite frankly, it’s kind of appalling that literally anyone would pay this much. It’s kind of appalling that Purism would charge this much. I can imagine a scheme where the parts are shipped from China, and verified and assembled by a part-time college student in the States and they’d likely still be able to price it cheaper than $1999.00.
Anyways this post isn’t about Purism’s business practices, of which I will comment on at some point, but regardless I will drop this matter here. Please just don’t spend $1999.00 on a Librem 5.
And for that one person who will eventually claim they did purchase the $1999.00 USA version and it was very necessary for their use case: look buddy, I get it, you’ve drank the Kool-Aid on this one but I’m sincerely doubtful that your very specific concerns and use-case apply to even the general nerd who gets excited about the intersection of FOSS and mobile hardware. Moreover, I’m doubtful that even if they did apply to such a niche group that $2k is a price point upon which you still don’t balk at and go look for literally anything else.
I bought my Librem 5 back during the original crowdfunding campaign, so I ended up spending a lot less than even the $799 USD. Surprisingly, I’ve been waiting on this phone since around the end of 2017, and here we are in 2020 and I finally have it. Purism has been sending out what are effectively DVT and EVT runs of the phone that have been going through several phases named Aspen, Birch, Chestnut, Dogwood, and finally Evergreen. Evergreen is supposed to be the final version of the hardware, which I opted for. I wanted to get the final, true Librem 5, not one that could be radically different. While my unit does not have FCC certification (it is technically what you might consider a PVT unit), it is representative of the final hardware that Purism will land on for this phone, as there is ostensibly no more hardware revisions.
The Pinephone starts at $149.99 USD, sold in small batches surrounding a
particular distro / community that is building software for the device (called
<distro-name>
Community Edition). Recent batches have made a “convergence
edition” available, for $199.99 USD. The convergence editions have an extra
gigabyte of RAM and a larger eMMC flash storage, and are touted as being useful
for connecting the Pinephone to a full-size monitor and running it as if you
had a full desktop available.
I originally purchased a Pinephone from the UBPorts CE batch. Eventually, I upgraded my mainboard so that my specs match that of the convergence editions from later batches. All in all, I’ve managed to spend more than the original $200, but I’m not disappointed, since I have had a lot of fun with the Pinephone.
For the vast majority of people, the Librem 5 and Pinephone are probably not worth even considering. The GNU userspace and associated “mobile” paradigms within it are very much not ready for daily use.
For the Librem 5, there are some security aspects that this “security and privacy focused phone” don’t even hit (e.g. hardware backed security / secure enclave, device encryption, etc.). That’s not to say that these can’t be made possible in future updates, but as far as the software story today goes, they do not yet exist.
Anyways, so why do I, the author, care? I will admit, after my first bout of usage, neither phone will be a daily driver for me. I will probably continue to stick to my Pixel 3, and it is very likely that I will eventually get another Android device at some point in the next two years. Linux phones are definitely not yet ready to be daily drivers; however, they are interesting for a few key reasons.
Digital waste is a blight on our society. I can admit that the year-over-year hardware cycle provides a vast set of improvements that can change the smartphone landscape. However, I also acknowledge that the number of phones, tablets, and other electronics that find their way into landfills or are otherwise rendered into trash is a growing concern. Linux phones and mobile hardware present an opportunity to move away from that, and it’s important to support this in some way.
I’m a fan of free software! While my day job isn’t working on FOSS, I think that there’s a lot of benefits to having better FOSS software available. The least of which is because leaps in the mobile space have also attracted others to work on Linux in the desktop space as well.
I care about Linux phones because they present a third option from the Android / iOS duopoly. Even if they suck now in (a lot of) ways, they will improve over time. I want to move towards a future where I don’t have to choose between an OS that’s built towards data collection and advertising, and another that’s built on locking you into an ecosystem.
First, before I get into my impressions comparing the two, I should at the very least mention what I am / am not testing. I did not / will not test:
In general, I find Phosh to be awkward to use. I don’t like it. It feels stilted and static. This is most easily seen in how one launches the app drawer or notification tray. Rather than swiping these (which feels exceedingly natural), you merely tap them. This feels as if it was done this way because button-like behaviour is easier than handling accelerations and swiping. This is probably largely personal preference, but I really, really dislike Phosh as a base. The Librem 5 doesn’t have as many UI alternatives as the Pinephone, largely in part due to the facts that:
So unfortunately for me, I’ll have to stick with Phosh on the Librem 5 for the time being. This is very much a preference problem, but it is a problem.
Lomiri is the name of the UI used by UBPorts / Ubuntu touch (and hopefully one day the Manjaro!!!) distro for the Pinephone. This UI feels much, much more natural, and I really appreciate the swipe-based behaviour of everything. Even compared to modern Android and all its quirks and features I think Lomiri has the right idea for how one might interact with their device.
Despite my preference for Lomiri over Phosh as a phone interface, I do have to say that it is very apparent the difference in processing power between the two devices. This is expected, since the Pinephone is vastly cheaper, but the Librem 5 is solid with regards to how it handles touch inputs, scrolling speed, delays, etc. Lomiri does a good job of not feeling slow on the Pinephone; however, the Librem 5 just feels consistently faster. Better hardware and all, but the software story doesn’t seem like it’s starting from a bad place (granted, this could be very different from anything you read pre-Evergreen, but forward progress is being made here).
One can certainly hope that one day there will be a Lomiri frontend for the Librem 5. I am not holding my breath though, since it doesn’t seem as if there are a lot of distros that are advertising that they want to support the Librem 5 (again, probably mostly due to price). I find it a bit of a shame that they’ve stuck with Phosh, but hey, Purism went ahead and wrote that themselves, so why wouldn’t they? I do wish something like Ubuntu Touch was available though, I could see the software story being very different if that was the case.
The on-screen keyboard on the Librem 5 is okay and gets the job done, but leaves something to be desired. Most Pinephone distros also use the same keyboard, so there isn’t a big winner here. However, Ubuntu Touch’s keyboard definitely comes out on top. The fact that it has spelling corrections, word suggestions, etc. help make it that much better.
Again, this is very much a preference thing, and I hope that the keyboard receives more updates in the future. I don’t see any fundamental reason why it can’t, or why more advanced features won’t be made possible.
It may also be worth noting that if you know nothing of these devices: swiping to type is not a feature of any keyboard on Linux phones. Maybe one day, but no, not now.
One thing that has frustrated me about the Pinephone is that the kill switches seem to be a secondary feature of the phone. What does that mean? Basically: these kill switches aren’t really as nicely integrated in the software as they could be. I don’t actually care that you have to remove the backplate to get to them, but I’ve sometimes noticed a tendency for the software to not gracefully handle the fact that one of them could be switched off. I’ve said a lot of good things about Ubuntu Touch until now, but it really doesn’t like if you disable the mobile data / baseband switch. Other distros handle this in various ways, but especially with the original mainboard my UBPorts CE Pinephone had I noticed that disabling the baseband could sometimes render wifi from never re-connecting after the device has gone into a deep sleep (i.e. screen is off for a few hours).
I’m not in the business of debugging this or ascertaining why, but it seems that the kill-switches on this phone have not been a day-1 feature. It’s not the end of the world given that I now work from home every day (so like, what’s a phone for anyways?), however I would appreciate if the Pinephone worked as well as the Librem 5 in this regard.
And on that note, the Librem 5 kill switches appear to work fantastically. I say “appear” because it is basically impossible to test the camera and mic kill switch (mostly due to the lack of a functioning camera). I have not tested any communication apps yet, as I don’t believe there are any that I currently use. When there’s a fully functional Matrix client on both phones, I’ll be happy to go through and test both :-)
Nonetheless, the baseband and wifi kill switches do exactly what you expect them to do on the Librem 5. Further, when turning them off and on again, the device and software recover fantastically. Big win here for the Purism folks, as this can’t have been a small feat.
Not much to say here. A data-only SIM from Google Fi seems to work pretty well in either phone. I haven’t had any major issues with connecting to a network with either the Pinephone or Librem 5.
There is not a vast ecosystem for apps on Linux phones yet. Ubuntu Touch boasts the largest app store, but quite frankly a lot of those apps are not really in my wheelhouse. However, there are some aspects of apps from my first impressions that stand out.
I know I said that I wasn’t gonna discuss GPS stuff, since I’ve barely used it, but my basic first impressions are:
The Librem 5 impresses me a lot with it’s web app support. While I initially had some issues with the browser, an update did manage to correct it (seems to be recent, so I don’t think it’s the norm?). Anyways, the GNOME Web browser does a good job of being able to “pin” web pages to your launcher. This performs a very lightweight sandboxing as well, limiting that “app” on your launcher to only a few sites. I really like this approach as it enables quick access to mobile websites like Lyft, Twitter, Mastodon, etc that I might have just used the browser for anyways.
This also works if you use GNOME Web on any Phosh-based distro for the Pinephone, but doing this on the Librem 5 seemed to be much more pleasant, if only because of the increase in computing power. The general gist is that very few native apps currently exist for either phone (or distro, really) but using mobile websites seems to work pretty well.
Alas, neither phones have a good Bandcamp or Spotify equivalent yet. UBPorts has a Spotify client called Futify, which is still in beta and not on the Open Store (you have to get the click package manually). However, Ubuntu touch also doesn’t handle switching to headphones when a pair of headphones are plugged into the 3.5mm jack on the phone.
The Librem 5 does appear to handle switching to headphones, but the lack of streaming apps is a bit of a pain. Likewise, it seems that my FAT-formatted SD card is not automatically mounted / detected by the Librem 5.
Needless to say, you can listen to music on these phones, and the headphone jack is a very, very welcome addition, but you’ll need to do some work to get going here.
Also, for both the Pinephone and Librem 5 (Phosh based distros), I highly suggest using Lollypop. It is heavily focused on sorting / playing via “albums” as opposed to individual songs, but it’s a very pleasant app to use. A lot of thought has clearly been put into this app.
Charging the Librem 5 seems to take forever. No, really. It seems after asking
in the official Matrix room (#community-librem-5:talk.puri.sm
) this is
something that is being worked on. General conclusion: USB-C is the worst, and
power is hard. I’m inclined to agree on both those points but it is a little
disappointing that charging using anything other than the charger + cable that
came with the phone is a bit of a toss-up right now. Future software updates
should hopefully fix this.
On the Pinephone side I’ve read that charging the battery can be dangerous. This is more than a little concerning. However, it’s also pretty clear these devices aren’t being marketed as an Android replacement just yet. I haven’t had anything bad happen to my Pinephone, but it’s worth keeping an eye out for.
Both phones can run fairly hot. I think given how I hold the phone (closer to the base of the phone) I tend to notice the Librem 5 more often than I notice the Pinephone getting hotter. This is mostly because the frame / chassis for the Librem 5 is metal, and conducts the heat around the sides of the phone. So you’ll notice it getting hot after you’ve used your phone for a good bit.
All in all though, if you feel where the CPU is on the Pinephone, it’s not vastly different in terms of how the temperature feels while using the phone on high load. That doesn’t mean the temperatures aren’t different, but I haven’t measured enough to say either way.
Aha! You thought I wasn’t going to talk about the shape and size of the phone. Well, here it is:
The Pinephone is shaped like a phone. It is a little wide and tall for my tastes, especially given the screen resolution is much lower than e.g. my Pixel 3, but it is phone shaped. It fits in my pocket well, and otherwise feels like a smartphone does in 2020.
The Librem 5, however. Well, it’s a chonker, that’s for sure. This thing is massively thick. It’s about twice the thickness of my Pixel 3, and it’s not all battery. This is shaped more like an external hard drive than a phone. This definitely bulges out of any pocket and is somewhat comically large. It’s also heavy, and I find it much more tiresome to hold for longer periods than say, my Pixel or the Pinephone. From an aesthetic point of view, it just looks like a giant brick.
I know I’m ragging on the Librem 5 a lot here, however don’t expect that the iPhone-level price tag means you’re getting an iPhone-level device. Be warned: if you are expecting this phone to be “thinner than it seems” you will be disappointed.
Pine64 (for all their devices, not just the Pinephone) is really a vibrant and positive community in my experience. The leadership that Pine64 shows in their community updates, their trasnparency, and how they collaborate and give back to the projects powering their phone are very strong reasons for supporting the project. I always love reading the monthly updates, even if I know I’m not buying any more Pine devices anytime soon.
I also especially appreciate how they have been donating money from every Pinephone Community Edition run to the projects behind those community editions. It’s a great way to help sponsor and foster FOSS, and I am so happy that this is made possible.
I really don’t know if there’s anything I can say about Pine64 that is inherently awful. They’ve done a fantastic job doing what they’re doing.
P.S. Their web store used to suck but its gotten a lot better. Making a web store is insanely hard though, so props to them for being able to manage it as they have been. Where’s the FOSS Shopify when you need it?
Purism on the other hand… A lot can be said about their interactions with their community, or the community at large. In fact, a lot has been said.
Extending from this discussion, there’s a lot of people who wanted a refund on the Librem 5 after doling out large sums of money to pay for it. They may want a refund for a number of reasons, whether due to the economic downturn of COVID, the delays in getting this phone out, or just general distaste for Purism and how they have behaved regarding community updates and the lack of ownership over the mistakes made. However, you can find a couple of posts on Reddit from users who can’t seem to get a refund:
Purism claims that they provide refunds when they get to your slot and ask where to ship the Librem 5. I think this is dodgy as all hell. Refunds need to be provided in a timely manner, as per a matter of law. For folks like me who got in during crowd-funding, I knew that money was gone. Getting a Librem 5 was more or less a crapshoot.
I am, however, very upset with the way that the business has conducted itself. This really isn’t something that should be happening at all. Rather than shipping out EVT / DVT units in the form of Aspen, Birch, etc. the company should have admitted they did not have a clue what they were doing, and extended their timeline. Nothing they’ve done since then has made me believe that this company is behaving any better. Yes, that includes the fact that they shipped me a phone.
I also notice that there’s a lot of folks who are very ripe to jump in and defend Purism at a moment’s notice. Saying things like: “mobile linux wouldn’t exist without them,” or “I am doing this for <some-principle> despite the fact I disagree with their lack of transparency.” I think this is a bit of a toxic way to approach it. Look, I have the phone, I left my money with Purism and still want mobile linux to succeed. But the community at large has to call this sort of behaviour out if they don’t want it to continue happening.
Lastly, and this is very much a distraction from the rest of this article, but I notice that the Librem 5 as a phone tends to draw a particular crowd. People who label themselves as “freedom thumpers”, or folks who seem to believe that “security and privacy” as concepts are above reproach or scrutiny. It’s not everyone, and many people are pleasant to talk to especially on some of the Matrix channels, but there’s an oddly fierce following of folks who aren’t willing to look outside the bubble on this.
I can sympathize with the intent of these folks to some degree. I mean hell, I bought into the Librem 5 hoping that someone would start working on a third option that wasn’t going to be run by a mega-corporation intent on monetizing every aspect of your life. If I’m being honest though, I dislike how much the community jumps on these memes and touts a $1999 USD (unfinished) phone as the only path towards true security and privacy.
Purism has a lot to do to step up their own transparency, and parts of the community has a lot of work to do to stop sounding like some crazy uncle at Thanskgiving; or at the very least, show the more positive aspects rather than permit lengthy diatribes around how we should support them despite Purism’s horrible communication, and think strategically about what we really want.
I didn’t want to make an article about this all on its own, but I definitely needed to say something here. In the future, I want to avoid having to have this same discussion again, so I’m going to try to leave it at that.
It’s obviously not really appropriate to compare the Librem 5 and the Pinephone spec for spec. They come in at two very different price points and are marketed as two very different devices. However, comparisons can’t help but be drawn between the two, as these are really the only two “no Android, no iOS” phones out there. For me, these devices represent the beginning of a world in which we can avoid Android and have devices that are truly under our control. Moreover, I hope that the lifespan of both my Pinephone and Librem 5 outlasts my current Pixel 3 and whatever other Android phone I may buy in the interim.
Overall the experience on both is fairly positive. Both phones are clearly not daily driver ready, but are a lot farther along than anyone might expect. I don’t much like Phosh, perhaps for superficial reasons, but I have a pretty positive view towards the future of software on both the Pinephone and Librem 5. Perhaps once I get a little more reign on my free time, I will try to see how hard it is to write an app that can be used on either device.
Outside of maybe reddit, both communitites have been fairly eager to jump in and help when questions are asked, and Pine64 is definitely amazing in this regard.
This will largely depend on whether or not you want to help fund a community edition version of the Pinephone, and what your goals for such a device are. I think in general I would lean towards yes, you should, but I wouldn’t use it as a daily driver. Too few apps and so many weird corner cases in everything, this phone is more for hackers than it is for consumers (and I mean, they say that on the box). For now, keeping this as a secondary phone in data-only / WiFi mode seems to be the sweet spot. The software continues to improve over time, so if you want this device to help test that on a single or even multiple distros, then definitely get one!
Pine64 sports a great community as do the various distros for the Pinephone. There’s a lot of goodwill in this community to make mobile linux be even more of a thing.
If you’re considering spending $1999 USD on one, don’t. :-)
Additionally, you have to consider if you’re willing to invest in Purism despite their lack of transparency and past mistakes. I don’t think that Purism is immutable in this regard, they certainly do have a very vibrant community on Matrix that doesn’t suck, but it’s very hard to get anything official from the company with the level of transparency and accountability that e.g. Pine64 provides. When Purism makes mistakes, they don’t seem to like to admit to them. I am not so cynical as to believe that they are all bad actors, merely that the organization has a problem with taking credit for their failures.
From a purely technical perspective, the Librem 5 is pleasant to use, and is surprisingly very well integrated (despite how massive the actual device is). The kill switches are an excellent feature and they work incredibly well on PureOS. Additionally, I can see many of the technical problems I’ve listed above being fixed in future updates, and will continue to stay up to date on those as they come.
Overall, I do not regret buying into the Librem 5 campaign. The phone itself is a marvel considering how long it took to come to market (at least, compared to my expectations). I have been disappointed many times along the way, and I do find some of the community behaviour a bit toxic and / or predatory, especially when it comes to refunds or putting up with poor behaviour on Purism’s part. Despite this, I have been having fun with both the Librem 5 and Pinephone, and am looking forward to a bright FOSS future for my choices in mobile OS.
]]>iOS 14 upgrades LLVM from the 9.x line to the 10.x line. This jump in versions was not well communicated to users, and ended up producing a challenging and upsetting bug on the platform breaking a code within Occipital’s Structure SDK. Apple, a trillion-dollar company, needs to do better. The change made by the LLVM team is easily justified (read on), but the way that it was rolled out was not as clear cut.
How did this happen? Well…
Maintaining an SDK can be tiresome, thankless work on the best of days. There are a considerable number of platform issues to worry about, even after ignoring the litany of API issues, sample app snafus, out-of-date paradigms, etc. I could go on.
I have for some time been employed by Occipital, and work on some components of their Structure SDK. This SDK isn’t super typical by 2020 standards, in that most of the SDK code runs on device, is made available in Objective-C (not Swift, although interoperability isn’t too challenging), and is largely available to provide on-device SLAM and computer-vision adjacent code to work with Occipital’s Structure Sensors. Needless to say, this isn’t akin to Facebook’s or Twitter’s SDK, which are largely a set of web APIs.
The challenge with this is that we end up having to deal with a lot of issues at different levels of the stack, with different expertise required to make them all work together in harmony. Some examples of disparate work streams that go into making this work:
While somewhat of a limited view of what goes into the SDK itself, this should give an idea of all the different ways things can go wrong. This is somewhat burdened by the fact that we are producing our own hardware and partaking in Apple’s MFi program, and building an ecosystem around that. Naturally, there is a considerable amount of platform work in ensuring we’re up-to-date with regards to which Apple APIs we use and keeping our SDK running on the latest releases that Apple puts out.
Needless to say, when something breaks, it can take a lot of (disparate) expertise to get to the bottom of the issue!
Fast-forward to mid-September and iOS 14 was released. There was some talk internally of our SDK not working well on the iOS 14 beta release. However, the company was amidst other priorities at the time, working to release Calibrator 4.0. We admitted that it probably wasn’t worth testing the beta release extensively, or spending time investigating these issues, since it was pretty common for Apple betas to be buggy and we weren’t ready to invest a bunch of time investigating issues that could easily never make it into the final release of iOS 14.
Quite frankly, many of our engineers aren’t excited when Apple announces a new release either. For some time now Apple has pushed the burden of QA/QC onto developers’ lap, or just outright shipped buggy code. Does anyone remember last year, when Catalina was released? How about the iOS 13 release?
Developing on this platform is stressful. New releases from Apple being buggy isn’t some strange 2020 affliction. These aren’t unprecedented times, this is the norm now. Working in this space can be awesome, when you see customers launching products built on top of your work that are changing the world. The platform, though, is constantly shifting ground.
So what broke? It actually took a couple weeks to really trace down, but the
crux of the bug was that our frame synchronization (between iOS color frames
and Structure Sensor depth frames) broke due to a change in LLVM with regards
to the default std::chrono::steady_clock
implementation. This meant that
while no change on our SDK actually broke anything, merely upgrading to iOS 14
would cause apps that otherwise functioned well on iOS 13 and earlier to
immediately and irrevocably break, because apps link to libc++
on iOS
dynamically.
Rolling this back a bit, what actually happened? Well, turns out that part of
our code for performing frame synchronization relied on
std::chrono::steady_clock
, to timestamp frames upon arrival. These arrival
timestamps are a small part of the information used by the Structure SDK to
enable our system to synchronize Structure Sensor events with iOS sensor events
(camera, IMU). In iOS 13 and earlier, this clock matched the same time scale of
timestamps coming from CMSampleBufferRef
/ CVPixelBufferRef
produced by
AVFoundation. On iOS 14, because std::chrono::steady_clock
was changed as a
result of moving from LLVM 9 ➔ 10, these clocks no longer matched the same time
scale.
How do I know this? Well, mostly empirical testing, but we can roughly identify
which clock was being used by running a simple test. AVFoundation timestamps
come in as if they are acquired by
mach_absolute_time()
,
which uses the underlying kernel clock, CLOCK_UPTIME_RAW
.
This is something that you can’t really change via the API, so we were kind of stuck with it. However, as most of our frame code was written in C++, and our drivers work across multiple platforms, we weren’t exactly stoked to mix this Objective-C API within our pure-C++ code.
If we look hard enough, we find that std::chrono::steady_clock
, which we used
to time the arrival of sensor events from Structure Sensor devices used to be
based on the same CLOCK_UPTIME_RAW
kernel
clock prior to LLVM 10. On iOS 14, with the introduction of LLVM 10, this was
changed, to now use CLOCK_MONOTONIC_RAW
.
Initially, I hadn’t discovered the above LLVM threads, but I could tell something was wrong with the timestamps I was getting. To verify that there was a change in behaviour, I used the following code to get timestamps on both iOS 13 and iOS 14:
double now_nanoseconds_machclock(void)
{
mach_timebase_info_data_t timebase;
kern_return_t status = mach_timebase_info(&timebase);
const double machToNanoseconds = (status == 0)
? static_cast<double>(timebase.numer) / static_cast<double>(timebase.denom)
: 0.0;
const double machTime = mach_absolute_time();
return machToNanoseconds * machTime;
}
double now_nanoseconds_steadyclock(void)
{
auto now = std::chrono::steady_clock::now().time_since_epoch();
return std::chrono::duration_cast<std::chrono::nanoseconds>(now).count();
}
If you ran this on iOS 13, these clocks roughly matched (ignoring small
differences since I didn’t call both functions in parallel). On iOS 14, if you
had immediately rebooted your device and ran this, then the clocks matched.
However, if you had at any point put your device to sleep, these clocks quickly
become very, very different. This was particularly hard to reproduce, because
it wasn’t immediately obvious from the std::chrono::steady_clock
documentation that the device going to sleep was going to affect our time
count!
This is because the difference between CLOCK_UPTIME_RAW
and
CLOCK_MONOTONIC_RAW
is that CLOCK_UPTIME_RAW
does not increment when the
devices’ screen is off and the device is sent to sleep. This change isn’t
something we could hold off on, or even avoid without re-implementing that part
of libc++
within our SDK. If you upgraded to iOS 14, this broke your app. For
that, I’m sorry.
It’s hard to argue that the changes to std::chrono::steady_clock
are wrong,
per sé. It makes sense that those adhering to the standard and expecting code
to behave the same across platforms want a steady clock that is, well, steady.
However, the end consequences of fixing this bug with an OTA update and with very little insight into the exact changes made are pretty dire. Like I said, this affected the Structure SDK itself, and by extension, every app built with the SDK that supports Structure Sensor devices. This broke hundreds of apps powered by Structure Sensor, and the fix was for every developer to recompile their app with our latest SDK, since there was no switch I could hit that would restore the iOS 13 behaviour. With many of our customers in the medical 3D-scanning space, this caused a lot of anxiety. Some customers had to delay or even cancel appointments with patients because of this bug. We fixed it, and we did so as fast as we could, but there’s a real impact to human life when this kind of thing happens.
This shows the dangers of AppStore-like models in some ways: you can only have one version of an application live at any time, and you can’t choose the environment it is getting run in. This is exactly the reason that many are flocking to Snap, Flatpack, AppImage, and more on Linux today.
But author, you say, wasn’t this change good and justified? How is this a failure of the AppStore? Doesn’t this mean that apps that were broken subtly by this behaviour are now automatically fixed? Not so fast.
Look, this causes a lot of strain and chaos, even ignoring the struggle of our engineers to get to the bottom of this. We want to be able to provide an SDK that people find reliable, and want to build their applications on. But it is getting harder and harder to do when Apple, a trillion-dollar company, can drop an iOS update that changes a very low-level and core aspect of the platform on a whim.
Mostly, I wrote this because I’m upset at all the stress and chaos that something like this generated. It shouldn’t have to be like this. Apple absolutely has the resources to do better here. They absolutely have the talent to produce solutions to this problem that don’t involve breaking hundreds of apps on an OS update. They are choosing not to.
Again, the trend of Apple releasing buggy release after buggy release is concerning. In this case, the bug was a result of a change that was probably quid-pro-quo positive across the board. I don’t fault the LLVM team for making the decision they did. How are they supposed to know about an SDK from a fairly small fish in Apple’s ocean? But when you run a platform, take a cut of every purchase on that platform, force developers to cater to a growing list of rules and restrictions, you need to be open and transparent, and be able to provide developers on that platform a way to help it grow rather than expecting them to perform QA/QC cycles for free.
]]>