joe.neeman

Linesweeper: a new library for boolean path ops

2025-10-19T00:00:00+00:00

Linesweeper</a> is a library that takes shapes (defined by Bézier curves) as input, and computes their union, intersection, or other related operation (let's call these operations "boolean ops").

No big deal, really: this is a super common operation in vector graphics, and so programs like Illustrator and Inkscape have been able to do it for decades. There are multiple open-source implementations available, for example in CGAL</a>, Paper.js</a>, lib2geom</a> (as used by Inkscape), or Geom2D.CubicBezier</a> (as used by Diagrams</a>).

But I had some reasons to try my own thing:

None of the existing implementations were easy to use in Rust, and I think there's some demand for a good Rust implementation; it's a popular feature request for kurbo, for example.</li>

I wanted access to certain low-level operations that aren't exposed by existing packages.</li>

It's surprisingly hard to do correctly and robustly, and I wanted to try a new approach.</li>

Sometimes it's just fun to do something yourself.</li> </ul>
I think these are all valid reasons, but I want to really focus on the second-last one.
What makes it hard?</h1>
The basic ideas aren't too hard to explain: as an example, suppose we want the union of two shapes.

The first step is to compute all the intersection points of the shapes' boundaries.

Then we'll use these intersection points to determine the boundary of the union. Starting from the top, we'll "trace" the boundary, keeping the union to our left as we go (either way would work, as long as we're consistent).

Whenever we get to an intersection point, we turn right. There could be lots of choices; we take the right-most one.

And that's all! Once we get back to our starting segment, we've traced out the union. (Ok, if the original sets had holes then there's some more work to be done to find holes in the union. But that's pretty similar to the steps we've already done.)

The exact details of the tracing algorithm aren't that important here, but the key takeaway is that it's pretty simple once you know two important things: the locations of all the intersections of all the boundary segments, and "how" those boundary segments meet there (like, what's the clockwise order of the segments at an intersection point).
Numerical issues</h2>
When you actually try to implement this, finding the intersections already presents some problems. We like to use floating point numbers for problems like this because they're pretty accurate and very fast on modern computers. But floating-point calculations invariably involve approximations and they can sometimes be wrong. For example: does the line segment from $(0, 0)$ to $(1, 3)$ intersect the line segment from $(2, 2)$ to $(0.5, 1.5) + (3, 10) \times 2^{-53}$? It does, in fact, but a numerical implementation might claim that it doesn't. (This isn't just hypothetical: I came up with this example by exhaustively searching for line segment intersections that kurbo</a> gets wrong.)
If you're only comparing individual segments, you might not care: you're doing approximations and you got an approximate answer. But when you have a path made up with multiple segments, small errors on individual segments can conspire to mess things up on a larger scale. In the picture below, if we fail to detect that segment A intersects with segment B (which is defensible) and we also fail to detect that it intersects with segment C (also reasonable), we might fail to notice that these two paths intersect, and that's definitely wrong.

You can try to fix this by tweaking your intersection-finding code to also report almost-intersections, but it can be tricky to consistently turn almost-intersections into intersections across different path segments. For example, in the picture above, it doesn't matter exactly which intersections we detect, but it's important that we see the B-C path as crossing the A segment exactly once.
Existing approaches</h2>
There are a few approaches I know for handling these numerical issues.

Use exact math. This is relatively easy if you only support paths made out of straight lines, because straight lines between integer endpoints can only intersect at rational coordinates. It's more challenging -- but still possible -- to support paths made out of Bézier curves: you "just" need to compute with algebraic numbers</a>. This is the approach used by CGAL. The main disadvantage of exact math is that it's slow. But also, if you want to get the answer out in floating point then you need a rounding step at the end anyway.</li>
Use exact math, but only for detecting intersections. This is commonly used for handling paths made out of straight lines, because there's an algorithm</a> for it that's much faster than generic exact arithmetic. I don't know of anyone doing this for paths with curved segments.</li>
Use approximate math, and try to handle all the corner cases. This is what most implementations do, but it feels a bit unsatisfying to me.</li> </ul>
Linesweeper's goals</h1>
So what do I want out of this new boolean ops algorithm? The big one is this: it should be correct by design, while avoiding exact math. I want to use inexact floating point arithmetic (for speed), and I want our algorithm to be robust to rounding errors.
I also want a reasonably strong definition of correctness. I'll allow the algorithm to perturb the input curves (this is unavoidable, as you can't represent the intersection points otherwise), but after that perturbation I want the output to be exactly correct: all the output curves should intersect one another only at endpoints:

In the picture above, I'm taking the union of the orange and blue shapes at the top left. The two green outcomes are acceptable: the two inputs can be perturbed to not touch, in which case the union has two disjoint parts. Or they can be perturbed to overlap, in which case the union gets fused into a single piece. The red outcome, however, is forbidden.
In addition to being correct, I'd like the algorithm to be fast. After all, that's the main point of avoiding exact arithmetic. At this stage of development, I'm not too worried about optimizing constant factors in the implementation, but I want it to scale well to higher accuracy and larger inputs.
The classical sweep-line algorithm</h1> The first point to address is the search for all intersections between paths. There's a classical, simple, and efficient algorithm for this: the Bentley-Ottmann</a> sweep line algorithm. This will be the basis for our robust intersection finder, so let's begin with a quick outline of the classical algorithm, which was originally described for straight line segments using exact arithmetic.
The Bentley-Ottmann algorithm sweeps a horizontal line (the "sweep line") from top to bottom, keeping a list (ordered from left to right) of the path segments that cross it. As the sweep line advances downwards, there are three interesting things that can happen: it can touch a new segment that wasn't already in the sweep line (we say that the segment "enters" the sweep line), it can move past the end of a segment (we say that the segment "exits" the sweep line), or it can pass the intersection point of two segments.
When a segment enters the sweep line, we insert it at the appropriate index 1</a>. Then we look at the neighboring segments to the left and right to see if they intersect with the new segment. If so, we make a note of the intersection point so we can handle it when the sweep line passes it.

For example, here the sweep line contains (A, B). Then C enters and we insert it at the appropriate place (after B), and we note that it will intersect with C pretty soon (circled in black). We don't yet compare C with A, though.
When two segments in the sweep line cross one another, we swap their positions and then re-check them for intersections with their new neighbors.

For example, here the sweep line is at the previously-recorded crossing of B and C. So we swap their order -- the sweep line goes from (A, B, C, D) to (A, C, B, D). Then we compare both B and C against their new neighbors, and make a note that C will cross A at some point (circled in black).
When a segment exits the sweep line, we remove it from the sweep line data structure, and then we check whether its two neighbors intersect each other (and make a note of the intersection point if they do).

For example, here both A and B exit at the sweep line. After that, C and D are neighbors so we compare them and note that they'll cross one another soon.
The key to this algorithm's efficiency is that for reasonable inputs, it avoids comparing most pairs of segments: each segment in the sweep line is only compared to its left and right neighbors.
A two-phase strategy</h1>
The first novel (to me, at least) part of our robust sweep line algorithm is that it comes in two parts: first, we run a sweep line algorithm to determine the ordering of all our segments. Only once the ordering is decided do we fix the positions. This two-phase split sounds minor, but it makes corner cases much simpler: whenever you assign positions to segments, you perturb them. And whenever you perturb them, you might affect the validity of things you've already calculated.2</a> By looking after the ordering first, and then later assigning the positions all at once, you can avoid this problem.
The ordering phase</h2>
Here's a little example of what the ordering phase does: we have four segments (A, B, C, D).

And as we sweep the sweep line down the picture, there are four heights where the ordering changes. Initially, the ordering is (A, B). At height a, it switches to (A, B, C, D). At height b, it switches to (A, C, B, D), and height c, it switches back to (A, B, C, D), and at height d it changes to (C, D). This sequence of orderings actually encodes all the information about entrances, exits, and intersections: at height a, the order changed from (A, B) to (A, B, C, D) and so we know that C and D entered. At height b, the order changed from (A, B, C, D) to (A, C, B, D), so we know that B and C crossed one another. Even though we haven't figured out horizontal positions for the various crossings, this is already enough information to do the topological part of boolean ops, by which I mean that you can use the algorithm at the beginning of this post to figure out that the union of these two shapes is traced out by following A down to height d, and then B up to height c, and then C down to the end, and then D up to height a, and so on.
They key primitive for the ordering phase is an algorithm</a> that takes two (monotonic in y) curves and figures out their horizontal order. For each vertical range, the algorithm is allowed to output "left", "right", or "close." So in the following example, it might say that A is to the left up until height a, and then they're close up until height b, and then A is to the right after that.

This pair-wise curve ordering primitive was proposed</a> by Raph Levien a couple years ago, and it's such an important part of our robustness story that I'll put this in bold for the skimmers to see: the key to robustness is to look for orderings instead of intersections. Unlike the intersection-finding approach that is usually used, this approach doesn't get confused by near-intersections near the endpoints of segments.

Like, in this example from before, the ordering approach just says that B starts out to the left of A, and then they get close, and then C is close to A, and then C is to A's right. It doesn't even try to figure out which of B or C intersects A, because we just don't care.
In order for this primitive to be a useful building block, we impose some requirements. Without getting too much into the technical details, if the curves are very close we require that the answer is "close," and if the curves are clearly ordered, we require the answer to be correct. In intermediate cases, one of two answers is allowed. For example:

Above height a, the comparison must say that A is on the left. Within the a region, it can say either "left" or "close" (and is even allowed to alternate between the two). Between heights a and b, it must say "close". Within the b region, it can say either "right" or "close", and below b it must say "right".
A brief interlude here, to elaborate on why I consider this algorithm correct "by design" and how that differs from being correct by trying really hard to handle all corner cases: once we have the comparison guarantees above, all of the fuzziness and rounding in the floating point calculations is gone from the algorithm. We have our ternary categorization ("left", "right", or "close") and the rest of the algorithm's logic is based on these discrete categories. I think this separation of concerns makes it easier to get things right, and also to test that they're right.
Back from the interlude, we use this approximate pair-wise comparison to build a sweep line algorithm that guarantees approximate ordering: whenever curve A is definitely to the left of curve B according to our approximate comparison then it should precede B in the sweep line. There are a few tricky parts to the sweep line algorithm, because

we need the ordering guarantees to hold for all pairs of curves in the sweep line, not just adjacent pairs of curves; and</li>
we want to avoid comparing all pairs of curves in the sweep line, like how the classical Bentley-Ottmann algorithm only ever compares adjacent curves.</li> </ul>
Here's an example of a tricky case:

Initially, (A, B, C) is the only valid sweep line order. At the marked height, our pairwise comparison is still allowed to say that A is close to B and B is close to C. If we only look at those adjacent comparisons, we might think that (A, B, C) is still a valid sweep line at that height. But clearly that isn't the case, since C is left of A.
This example shows that (unlike Bentley-Ottmann) our approximate sweep line needs more than just nearest-neighbor comparisons: when there are multiple nearby neighbors, it keeps scanning until it sees a gap large enough to know that it doesn't need to look any further. This "large enough gap" test is implemented with the same pair-wise curve comparison algorithm, just with a larger threshold for the meaning of "left" and "right".
The positioning phase</h2>
In the second phase, we physically lay out our segments, while respecting the orders that the first phase chose for us. It turns out (surprisingly to me, at first) that producing Bézier curves with a guaranteed horizontal order is easier than checking the order of some given curves: the latter I can only do approximately, but the former I can do exactly. The trick is that if you have two Bézier curves and their control points have the same $y$ coordinates and ordered $x$ coordinates, the curves are guaranteed to also be ordered. In the picture below, we can tell that curve A is to the right of curve B just by comparing their control points (the black control points are common to both curves).

Therefore our strategy for outputting correctly ordered segments is to simultaneously approximate all segments with Bézier curves that all share the same control point $y$ coordinates. If the approximation isn't good enough, subdivide and try again. If the approximation is good enough, apply some maxes and mins to the control point $x$ coordinates to ensure the correct order.
The most complex part of this scheme is the part that tries to avoid doing it, because we'd prefer not to approximate and subdivide if we don't have to. So we conservatively detect regions where the order needs to be enforced. Outside of those regions, we simply copy the input segments to the output.

In this example, the three curves are close to one another at the top, so in the top rectangular region we'll have to approximate and subdivide. But pretty soon C</code> moves far away, so in the second rectangular region we'll only approximate and subdivide A</code> and B</code>. Near the bottom, we'll approximate and subdivide A</code> and C</code> while leaving B</code> alone.
Where we're at</h1>The algorithm described above is implemented (in Rust) in the linesweeper</a> crate, which recently released verson 0.1.0</code>. That version number probably gives the right idea of the implementation's maturity: it should basically work, but there are still lots of improvements to be made to correctness, performance, and clarity of implementation. But you can give it a try and report issues! If you'd like it to run particularly slowly and crash a lot, you can use it with the slow-asserts</code> feature, which exhaustively checks all the ordering invariants that we're supposed to uphold. Acknowledgement</h1> None of this would have been done without the advice and encouragement of the linebender</a> community, especially that of Raph Levien. In particular, the implementation of curve comparison is heavily based on his code</a>, and I learned about the approach to guaranteed output ordering from conversations with him on the linebender zulip</a>. ^{1 using some kind of tree structure for fast searching and insertion </div>}^2For a cautionary example, the geo</a> was using a sweep line algorithm that had a bunch of bugs caused by perturbations unexpectedly breaking their invariants. These were eventually fixed by ditching their custom implementation and switching</a> to the i_overlay</code> crate, which uses a variant of the exact math approach (implemented in fixed precision for speed). </div>

The Prospero challenge 2025-06-09T00:00:00+00:00 Matt Keeter posted his Prospero challenge</a> in March, and for a couple of months I did an excellent job of focusing on my other free-time projects and resisting the urge to jump in. That ended last week. For a proper description of the problem, see the original post</a>. But the basic idea is that we have a giant list of almost 8000 instructions: _0 const 2.95 _1 var-x _2 const 8.13008 _3 mul _1 _2 _4 add _0 _3 ... and 7861 more </code></pre> and we want to execute them as fast as possible, many times in parallel. There are already a bunch of great write-ups linked from the main Prospero challenge page, and most of them fall into two categories: interpreters, programs that iterate over the instructions to compute the result, and JIT compilers, programs that iterate over the instructions and emit machine code, which then gets executed to produce the result. One interesting solution emitted CUDA source code instead of machine code. Then it compiled the CUDA source and executed it on a GPU. The execution time was super fast, but the compilation step was very slow. This post is about a sort of a JIT compiler, but one that outputs code for the GPU instead.1</a> The first challenge is to figure out what kind of code to output. Most CPUs have a standardized instruction set (either ARM or x86_64 for most desktop CPUs), but GPUs don't: every GPU generation from every manufacturer does its own thing. Nvidia has PTX</a>, an instruction set that's portable among their own GPUs, but I don't have an Nvidia GPU. On non-Nvidia GPUs, SPIR-V</a> appears to be the most useful compilation target. It's the intermediate representation used by Vulkan, and so any Vulkan video driver knows how to compile SPIR-V into the GPU's native instruction set. Hopefully those compilers are fast! 2</a> Emitting SPIR-V</h1> I was relieved to discover that there's a crate (rspirv</code></a>) for emitting binary SPIR-V code in Rust. They even have a builder</a> with a convenient interface for building SPIR-V modules! Well, sort of convenient: SPIR-V is quite low-level and explicit, and it demands that you bind variables to everything that you want to refer to. For example, you can't just say that you want to declare a void</code> function returning void</code>. Instead, you have to first bind the void</code> type to a variable, then bind the void-function-returning-void</code> type to a variable, and then finally you can declare your function. In rspirv</code> it looks like this: let mut b = rpsirv::dr::Builder::new(); let void = b.type_void(); // declare the void type let voidf = b.type_function(void, vec![void]); // declare the void -> void type let f = b.begin_function(void, None, FunctionControl::None, voidf); </code></pre> If you want to add 1.0</code> to a floating point number, first you need to declare the floating point type, then the floating point constant 1.0</code>, and only then can you create the addition instruction: let float = b.type_float(32); let float_1 = b.constant_bit32(float, bytemuck::cast(1.0f32)); let result = b.f_add(float, None, float_1, x).unwrap(); </code></pre> To make things more fun, void</code>, voidf</code>, f</code>, float</code>, and float_1</code> are represented in Rust by the same type</a>, so there's nothing to prevent you from accidentally trying to add some thing to the float type instead of the floating-point constant 1.0</code>. If you mess something up then the Vulkan runtime will helpfully segfault without telling you where the problem is. To actually get things working, I made heavy usage of the spirv-val</code></a> tool to debug problems with my generated code. To figure out how to express things in SPIR-V, I also looked at a bunch of generated disassembly from the reference glslang</code></a> compiler, without which I never would have figured out how to access the built-in gl_GlobalInvocationId</code> variables or how to write to the output buffer. Once I'd figured out the basics, translating instructions to SPIR-V was pretty straightforward. The main loop looks something like this: // Map the line names in the input file to their SPIR-V variables let mut vars = HashMap::<&str, Word>::new(); for line in insts { // Skip comments if line.starts_with('#') { continue; } let mut parts = line.split_whitespace(); let name = parts.next().unwrap(); let op = parts.next().unwrap(); let var = match op { "const" => { let val: f32 = parts.next().unwrap().parse().unwrap(); b.constant_bit32(ty_float, bytemuck::cast(val)) } "add" => { let op0 = vars[&parts.next().unwrap()]; let op1 = vars[&parts.next().unwrap()]; b.f_add(ty_float, None, op0, op1).unwrap() } // ... and the rest of the ops }; vars.insert(name, var); } </code></pre> Does it go fast?</h1> Sort of, yes! The binary as a whole isn't super quick, because initializing the GPU resources takes a little while. But after about 50ms of setup time, it took about 12ms to read the input data, create and assemble the binary SPIR-V code, load it on the GPU, and render out a 1024×1024 image. We can break down that 12ms further: stage</th> time (ms)</th></tr></thead> shader creation</td> 1.5</td></tr> shader loading</td> 3.7</td></tr> dispatch</td> 6.8</td></tr> </tbody></table> Here, "shader creation" includes the time to parse the input file and call b.begin_function</code> and so on. "Shader loading" is just the call to vulkano::shader::ShaderModule::new</code>, which I think is where shader compilation happens,^{3</a> and "dispatch" consists of running the shader and reading back the filled buffer. As a point of comparison, Ken Micklas's vectorized JIT</a> implementation took 38ms on my machine. So 12ms seems pretty good, especially if we can further reduce the shader creation and loading time by targeting a lower-level representation.} Increasing the problem size scales a little better than I'd expect: doubling the sizes quadruples the number of pixels, but the runtime seems to consistently rise by less than a factor of four. I suspect this is caused by the GPU clocking up for the longer computations. I tried forcing my GPU to max power4</a>, and it sped things up quite a lot -- 1024×1024 went from 6.8ms to 2.6ms -- but it also consistently crashed my machine at size 8192. So all these measurements are with the default power settings. The main take-away here is that 1024×1024 is about the size where setup time and execution time are roughly comparable. Much larger than that, and the execution time dominates. The source code of this implementation is here</a>. What next?</h1> We've only explored brute-force computations so far, but apparently the really fast</a> and scalable way to do this sort of thing is via interval arithmetic and expression simplification. I'd like to explore these techniques on the GPU, where we might face different tradeoffs with setup time and execution time: we'll have to compile a new shader for each simplified expression, so it probably makes sense to do bigger batches and less simplification. ^{1 I actually started out by trying to write a fast GPU interpreter, but it was hard. I may write about some of the issues that I had and the benchmarks that I run, but it's more fun to first write about the part that worked. </div>}^2If we're willing to give up platform portability, we could look into Mesa -- the Linux graphics framework -- more directly. For modern AMD cards, their shader compilation process</a> has two stages: compilation from SPIR-V to NIR (their native intermediate representation), and then compilation to GPU machine code. It would be interesting to see if we could hook in to Mesa at one of those two stages instead of going through SPIR-V. </div> ^{3 Apparently, Mesa sometimes caches shader compilation, so I ran this with MESA_SHADER_CACHE_DISABLE=true</code> but didn't notice much difference. Maybe our shader doesn't trigger Mesa's caching heuristics? </div>}^{4 echo high | sudo tee /sys/class/drm/card1/device/power_dpm_force_performance_level</code> </div>} Sleep and power usage 2023-11-17T00:00:00+00:00 So I have a bunch of these</a> dev boards and I'd like to try using some of them as battery-powered remote sensors (I'm not sure yet exactly what I want to sense; maybe temperature would be a good start). I'd like it them to last for at least a couple of weeks on a battery charge, so let's try to figure out what my options are. First up, roughly how much charge can we expect our battery to have? Adafruit sells rechargeable 3.7v batteries with capacities ranging from 350mAh (for $7) to 10Ah (for $30). That's a big range, so let's just say "within an order of magnitude of 1Ah". Next up, how much power does our chip use? We can get a very rough first guess from the datasheet</a>. The current consumption is listed in Section 4.6.1: while transmitting over wifi the current can go as high as 335 mA, which would drain our 1Ah battery in 3 hours. At the other extreme, there is a "deep sleep" mode that is supposed to use only 5µA. At this current, our 1Ah battery would last more than 22 years. Those initial estimates leave a big range, and they aren't likely to be accurate for my usage, because those figures are just for the chip itself. Since I'm not (yet) hardcore enough to design my own PCBs, my chips are embedded in a development board full of other components that also use power. Since I couldn't find anything about power consumption on the board manufacturer's page</a>, I guess I'll just have to measure it myself. At this point I need to confess that I was having so much fun with my Lolin C3 Mini that I went out and bought another couple of dev boards to see how they compare: I got some Lolin C3 Picos</a> -- which are slightly smaller than the minis and have some battery charging circuitry and a few Seeed Studio XIAOs</a>, which have two fewer pins and are noticeably smaller than the Lolins unless you count their largish (but detachable) external antenna. Here you can see the three of them side-by-side, with my hand for scale. Sleep modes</h2> The esp32c3 has two "sleep modes</a>," called "light sleep" and "deep sleep." Light sleep behaves like I would expect sleep to behave: it puts the device into a low-power state, and then after some wakeup-trigger event it wakes up in the same state it was before going to sleep. Deep sleep also puts the device into a low-power state and waits for some wakeup-trigger event, but it's a bit more like a "reboot" in that when it starts back up again, it starts back from the beginning. Both sleep modes can be configured to wake up after a certain amount of time, or when a specific GPIO pin gets driven high or low. So I wrote a quick test program that runs for a little while, turns out the wifi for a little while, light-sleeps for a little while, and deep-sleeps for a little while. I powered it from a 5v power source and measured the power draw in the various different modes. I also varied the CPU frequency and tried running a bit of math code in a loop to see if the processor consumed more power. Board</th> idle@80MHz</th> work@80MHz</th> idle@160MHz</th> work@160MHz</th> wifi</th> light sleep</th> deep sleep</th></tr></thead> lolin mini</td> 20mA</td> 21mA</td> 27mA</td> 31mA</td> 101mA</td> 2.4mA</td> 400µA</td></tr> lolin pico</td> 21mA</td> 22mA</td> 27mA</td> 31mA</td> 100mA</td> 2.6mA</td> 437µA</td></tr> seeed xiao</td> 21mA</td> 23mA</td> 29mA</td> 32mA</td> 102mA</td> 2.5mA</td> 175µA</td></tr> </tbody></table> Take-aways include: running at 160MHz takes roughly 50% more power than running at 80MHz</li> doing work on the CPU uses a little more power than idling it</li> the three boards use about the same power except during deep sleep, where the Seeed XIAO is the most efficient by a long way. Maybe this is because it lacks an RGB LED?</li> we aren't getting anywhere near the 5µA deep sleep current promised by the datasheet. But still, deep sleep takes about 500x less power than WiFi.</li> </ul> Rtt and defmt 2023-11-04T00:00:00+00:00 In the first post, we printed a "Hello world" message to our computer using the esp_println::println!</code> macro. But there's another (better? I'm not sure. But at least, different and therefore worth trying) way to make code running on our esp32c3</code> show up on our screen. The Real Time Transfer</a> (RTT) protocol is a simple method of transferring data from an embedded device to the host system. It works by sort of not transferring data at all: it just writes buffers in the embedded device's memory. So then how does that data make it to the host? Well, the host connects to the device using a debug probe that's capable of reading to and writing from the device's memory. As the embedded device is writing its data to memory, the debug probe is reading it out to the host and then modifying the device's memory to tell it that the data's been read. The advantage of this is that the data transfer code on the device is simple, fast, and small: just write a few bytes to memory. There are several device-side implementations of RTT in rust. defmt-rtt</code></a> and rtt-target</code></a> appear to be two of the more popular options, but the esp_println</code> crate we were using before also has support for RTT as long as you set its features correctly. probe-rs</h2> There are a few options for RTT on the device side, but on the host side the only implementation I could find was the one in the probe-rs</code></a> family of tools. When I first started writing this post a couple of months ago, the released version of probe-rs</code> didn't work out-of-the-box with the binaries that esp-hal</code> produces by default. (The esp32c3</code> supports two different binary formats; details here</a>. The latest release -- 0.21.1</code> as of me writing this -- of probe-rs</code> works, though, as long as you pass the --format idf</code> argument: here we can run one of the led-blinking examples from the previous post. ❯ probe-rs run --chip esp32c3 --format idf target/riscv32imc-unknown-none-elf/debug/rmt DEBUG probe_rs::architecture::riscv: Before requesting halt, the Dmcontrol register value was: Dmcontrol { .0: 1, hartreset: false, hasel: false, hartsello: 0, hartselhi: 0, ndmreset: false, dmactive: true } Erasing sectors ⠁ [00:00:00] [##################################################################] 0 B/ 0 B @ 0 B/s (eta 0s ) Programming pages ⠁ [00:00:00] [##################################################################] 0 B/ 0 B @ 0 B/s (eta 0s )DEBUG probe_rs::architecture::riscv: Before requesting halt, the Dmcontrol register value was: Dmcontrol { .0: 1, hartreset: false, hasel: fals Erasing sectors ✔ [00:00:01] [#############################################################] 116.00 KiB/116.00 KiB @ 58.25 KiB/s (eta 0s ) Programming pages ✔ [00:00:04] [#############################################################] 112.00 KiB/112.00 KiB @ 26.82 KiB/s (eta 0s ) Finished in 6.181s ERROR probe_rs::cmd::run: Failed to enable_vector_catch: NotImplemented("vector catch") </code></pre> Despite the scary-looking error message, it does run and blink the LED. (Regarding the spurious debug messages, I'll spare you the account of my late-night debugging and just link to the tracing_subscriber</code></a> issue report that came out of it.) RTT on the device using esp-println</h2> Since we've been using the esp-println</code> crate already, the easiest way to get RTT working on the device is just to twiddle some features: remove the default uart</code> feature from esp-println</code> and replace it with the rtt</code> feature (and if using esp-backtrace</code>, remove its print-uart</code> feature and replace it with print-rtt</code>). There's a full runnable example, complete with .cargo/config.toml</code> and everything, here</a>. RTT on the device using defmt</h2> One of the exciting parts about using RTT is that it unlocks using [defmt] (https://defmt.ferrous-systems.com/), the "deferred" formatting system that makes formatting small and fast by doing the formatting on the host instead of the device: when you run println!("{}, ah ha ha", x)</code> using, say, esp-println</code>, it compiles down to code that formats the integer x</code>, builds the string with that formatting in it, and sends that string back to the host somehow. When you run the same code using defmt::println!</code>, it sends back the integer x</code> without formatting it, and all the formatting and string concatenation is done on the host. This removes the formatting code from the embedded device, and it also reduces the amount of data that has to go from the device to the host. So let's give it a try! We need to add in the defmt linker script in .cargo/config.toml</code> # ... [build] rustflags = [ "-C", "link-arg=-Tdefmt.x", # ... whatever was in rustflags before ] </code></pre> Then we need to open our Cargo.toml</code> and replace the esp-println</code> dependency by defmt</code>, as they can't really coexist. More precisely, we can't use defmt</code> with rtt</code> support at the same time as we use esp-println</code> with rtt</code> support (the binary will fail to link, as both crates expect to own the rtt</code> block of memory). We can use defmt</code> with rtt</code> support and esp-println</code> with some other method of printing. But then we can only see one of the two outputs -- as far as I could tell, there's nothing on the host end that will print them both. We also need to remove esp-backtrace</code>, since it only supports printing things using esp-println</code>. This also means we'll have to add our own backtrace implementation. Fortunately, it isn't too difficult to just copy one of the implementations out there. The full runnable example is here</a>. Gripes with probe-rs</h2> It's great that probe-rs</code> and defmt</code> support the esp32c3</code> with minimal setup pain, but I did have a couple of gripes that will keep me using espflash</code> for now: probe-rs</code> is slower to flash: this rtt-defmt</code> binary in the previous example takes about 2 seconds to flash with espflash</code>, but about 6 seconds to flash with probe-rs</code>. Not such a big deal for this example, but it's more painful for larger programs. </li> there's one situation that I often run into where espflash</code> has a decent error message but probe-rs</code> doesn't: if I put my board into boot-from-flash mode by holding down the button on the right and the tapping the reset button (more on this in a future post) then on the next cargo run</code> it won't actually boot up. When using espflash</code> you get a somewhat cryptic error message in this situation: ESP-ROM:esp32c3-api1-20210207 Build:Feb 7 2021 rst:0x15 (USB_UART_CHIP_RESET),boot:0x7 (DOWNLOAD(USB/UART0/1)) Saved PC:0x40380836 0x40380836 - _start_trap_rust_hal at /home/me/.cargo/registry/src/index.crates.io-6f17d22bba15001f/esp-hal-common-0.13.1/src/interrupt/riscv.rs:433 waiting for download </code></pre> At least it's an error message, though. probe-rs</code> just sits there saying nothing. </li> </ul> Blinky 2023-09-03T00:00:00+00:00 Last time</a> we left off by failing to get the integrated LED to light up. We were following the "blinky" example in esp32c3-hal</code>, which is pretty simple: the main loop looks just toggles the pin's voltage from low to high and back again, one cycle per second. There's not much that can go wrong besides choosing the wrong pin, and we double-checked that already: pin 7 is the one hooked up to our LED. That pin diagram gives us our first hint for what's gone wrong: pin 7 is marked as an RGB_LED</code> instead of just a LED. (There's also a Lolin C3 mini v1.0.0</a> that has a normal LED, but we're using the v2.1.0.) What's an RGB LED? A google search turned up LEDs with four pins</a>: a common ground and then one pin for each color. But our RGB LED has only one pin besides the ground, so we must have something different. Our next hint comes from our board's schematics</a>, which shows IO 7 connected to something that's labelled WS2812B-3535</code>. A quick google turns up a datasheet</a>. Instead of a mere analog LED, it appears we have an "Intelligent control LED integrated light source," which we control using a simple communication protocol described in that linked datasheet. The main parts are this: which says we need to send 24 bits of color data by transmitting (in big-endian order) a byte of green, a byte of red, and then a byte of blue. (I'm pretty sure the diagram has three blocks of 24 bits each because it's showing how you would transfer data if you had three LEDs wired up in series. We only have one LED, so we just need one block of 24 bits, followed by the reset code.) And then this: tells us how to send each bit: a 1 bit is sent by setting the pin high for 850ns and then low for 400ns, and a 0 bit is sent by setting the pin high for 400ns and then high for 850ns. After sending all 24 bits, we finish up by setting the pin low for 50µs. Bit-banging the WS2812</h2> Since the procotol is simple, we can implement it bare-hands: // Transmit a 1 to the LED. let mut one = || { io7.set_high().unwrap(); delay.delay_ns(850); io7.set_low().unwrap(); delay.delay_ns(400); }; // Transmit a 0 to the LED. let mut zero = || { io7.set_high().unwrap(); delay.delay_ns(400); io7.set_low().unwrap(); delay.delay_ns(850); } // Send G = 0, R = 255, B = 0. zero(); zero(); zero(); zero(); zero(); zero(); zero(); zero(); one(); one(); one(); one(); one(); one(); one(); one(); zero(); zero(); zero(); zero(); zero(); zero(); zero(); zero(); // 40µs of "low" to finish. delay.delay_us(40); </code></pre> Unfortunately, we can't do this because there is no delay_ns</code> function: esp32c3</code>'s Delay</code></a> only supports microsecond resolution. I'm not completely sure why, because apparently</a> the underlying timer runs at 16MHz, but anyway there are less brute-force ways to speak to our WS2812B. The SPI protocol</h2> I learned about this first method from the ws2812-spi-rs</a> crate, which I found by searching for "ws2812" on crates.io. The idea is that the Serial Peripheral Interface</a> is close enough to what we want that we can manipulate it into talking to our LED. The SPI protocol has a fixed clock speed and it sends one bit per clock tick by setting a pin high for a 1 or low for a 0. In contrast, each bit in the WS2812's protocol needs the pin to be high and then low (for durations depending on the bit we're sending). For example, the nibble 1101</code> gets represented like this in our two protocols: Clearly they aren't the same protocol, but just from looking at the diagram you can see how the WS2812's protocol can be emulated using SPI: we set the SPI frequency higher; then we send a WS2812 1 by sending several SPI 1s followed by an SPI 0, and we send a WS2812 0 by sending an SPI 1 followed by several SPI 0s. We have to do a little math to get the timings to line up. First up, I lied a little bit earlier: the WS2812's timings don't need to be exactly 850ns for the long pulse and 400ns for the short pulse: you get ±150ns of slack on both of those timings. So if we set the SPI clock to 3.33 MHz then one clock cycle (300ns) is suitable for a short pulse and three clock cycles (900ns) is suitable for a long pulse. Then we can sent a WS2812 1 by sending 1110</code> on this faster SPI interface, and a WS2812 0 by sending 1000</code>. For example, to send the nibble 1101</code> as above, we replace each 1 by 1110</code> and each 0</code> by 1000</code> to get 1110111010001110</code> and then we send that along our faster SPI interface like this: And it works! I implemented it here</a>; running cargo run --bin spi</code> finally gives me a life sign from the onboard LED. The RMT peripheral</h2> I didn't realize when I started, but the esp32c3-hal</code> has an example for us after all: it's called hello_rgb</code></a>. It talks the WS2812's protocol using (by way of the esp_hal_smartled</code> crate) the "Remote Control Transceiver (RMT)" peripheral. According to Espressif's documentation</a>, the RMT was originally designed as an infrared transceiver (hence the name), but has grown to be "a versatile and general-purpose transceiver, transmitting or receiving many other types of signals." One nice thing about RMT is that it has built-in support for the sort of long-short/short-long pulses we want, without needing to fake them: the esp32c3_hal::rmt</code> module provides a PulseCode</code> struct that can express our protocol like this: // The pulse for a 1 bit in WS2812's protocol: let one = PulseCode { level1: true, // Start by setting the pin high... length1: 68, // for 68 cycles, which is 850ns at the default 80 MHz clock speed. level2: false, // Then set the pin low... length2: 32, // for 32 cycles, which is 400ns. }; let zero = PulseCode { /* similar */ }; // Just zero for 4000 cycles. let reset = PulseCode { level1: false, length1: 0, level2: false, length2: 4000, }; </code></pre> I wrote a small example here</a> that uses the RMT peripheral to talk to our onboard LED. It works just as well as the SPI example. Which peripheral should you use?</h2> You can use either the RMT or the SPI peripheral to communicate with the Lolin C3 Mini v2.1.0's onboard LED. But which way is better? The honest answer is "I don't know," but here are a few relevant factors: The ESP32-C3 has only one general-purpose SPI controller, according to this</a>. So if you need the SPI bus for a non-LED purpose, you'll need to use RMT for the LED. There are four</a> RMT channels, two for sending and two for receiving. So the SPI controller seems to be the more precious resource.</li> The in-memory encoding of the signals is more efficient for SPI: we used four bits in-memory to encode one bit of protocol (we could have made do with three bits but four was easier). The RMT encoding uses a u32</code> (four bytes) for each bit of protocol. Given that we only 24 bits to send, this discrepency probably doesn't matter. But if we had a lot of LEDs to control, maybe it would make a difference.</li> The SPI method is less "blocking:" there is support (which I might write about later) for async operations on the SPI bus, while the RMT code in esp-hal</code> does a busy loop</a> until sending is complete.</li> The esp-hal-smartled</code> crate uses RMT, so you'll need to use RMT also if you want to use their utilities.</li> </ul> About 2023-08-19T00:00:00+00:00 Hello. My name is Joe. I work in a SimpleButtonFactoryAwareAspectInstanceFactory. Ok, not really, but I do write software for a living. Once in a great while, I also write about software. If that's what you're here for, check out my blog</a>. In a previous career, I published papers on probability, geometry and computer science. Those papers are all available on the arXiv</a>, and there's also a summary of some of my research themes here</a>. Going small: my adventures with an esp32c3 2023-08-19T00:00:00+00:00 I'm a software developer most of the time, but recently -- originally inspired by the brachiograph</a>, which is amazing and you should make one -- I decided to dabble in embedded devices. This is a collection of various random things I'm learning as I go. Who is it written for? I'm not really sure. If you're also dabbling in the area, you might find some useful tidbits. If you already know what's going on, maybe you'll be amused by my struggles. The goal here is learning and fun; productivity, reliability, maturity of tooling are all optional. In particular: everything will be in rust: I'll be writing in rust and using rust-native tooling. I enjoy writing rust, so that's what it will be.</li> RISC-V will be the target platform. People seem excited about it, so let's see what it's about.</li> </ul> After a bit of googling, I landed on the ESP32-C3</a>. It has wifi and bluetooth built in, and there seems to be an active rust community in the esp-hal</a> project. Including, if I'm not mistaken, some employees of the manufacturer. I've chosen the Lolin C3 mini</a> as my dev board, mostly because it's cheaply available on AliExpress</a>. This is what it looks like: There are some schematics</a> available and a link to the ESP32-C3 datasheet</a>. At some point I'll need to figure out what those things mean. First look at the docs</h1> There are a number of good sources for documentation: The Rust on ESP</a> has some useful instructions on how to set up projects. I learned that we have two possibilities: we can use [ESP-IDF](https:// docs.espressif.com/projects/esp- idf/en/latest/esp32c3/get-started/index.html), Espressif's SDK, via the esp- idf-sys</code> crate. This gives us support for rust's std</code> library. Or we can go bare-metal, using the esp32c3-hal</code> crate. I couldn't find any analysis of the advantages or disadvantages of these two choices, so let's start with the bare metal version.</li> There's some tutorial-style documentation in the Embedded Rust (no_std) on Espressif</a>. book. It assumes you're using a specific board, though. Maybe I should have bought that one instead?</li> The Embedonomicon</a> seems to be all about ARM processors, but there are a bunch of low-level details there that might be important to know. Like linker scripts. I know nothing about linker scripts.</li> </ul> I've had less success with the [API documentation](https://docs.rs/esp32c3- hal/0.12.0/esp32c3_hal/index.html) of the esp32c3_hal</code> crate. The module documentation is good, but I'm a bit stumped by pages like [this](https:// docs.rs/esp32c3-hal/0.12.0/esp32c3_hal/struct.Rmt.html) one. (What can I do with the channels? Channel0Creator<0></code> doesn't seem to be public...) The day is saved, however, by the [examples](https://github.com/esp-rs/esp-hal/tree/ main/esp32c3-hal/examples): there's a ton of them, covering a lot of different stuff. And they're kept up-to-date, which is amazing because this stuff is under heavy development. First look at the board</h1> The biggest thing on this small board is the USB-C connector in the middle of the bottom edge. To its right is an LED, marked with a 7</code>. The two bottom corners are taken up by two small buttons; the one on the left is marked RST</code> and the one on the right is marked 9</code>. More on those later. The ESP32-C3 comes with integrated USB-to-JTAG support, meaning that you can flash and debug it without any extra hardware. In this board, this functionality is hooked up directly to the USB port, so you can get started with an ordinary USB-C cable. (We'll see later what implications this has if you want to use the USB port for other stuff.) The board is recognized automatically when I hook it up to my computer with a USB-C cable: ❯ dmesg <...> usb 1-2.1: New USB device found, idVendor=303a, idProduct=1001, bcdDevice= 1.01 usb 1-2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 usb 1-2.1: Product: USB JTAG/serial debug unit usb 1-2.1: Manufacturer: Espressif <...> </code></pre> Permission to flash</h1> Before we can actually write to the flash chip, we'll need to convince Linux to give us the necessary permissions on the USB device. probe-rs</code> has some docs</a> for setting this up on "normal" Linux distributions. I'm using NixOS, though, which always has to be a little special. To configure the flasher's permissions on NixOS, add the following to your system configuration: # Note that I'm not using `services.udev.extraRules`, because those rules go # in `99-local.rules` and `uaccess` needs to be set before that. I learned this # from https://github.com/NixOS/nixpkgs/issues/210856; see also # https://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/ for # more about how the uaccess tag works. services.udev.packages = [ (pkgs.writeTextFile { name = "embedded-udev-rules"; text = '' ATTRS{idVendor}=="303a", ATTRS{idProduct}=="1001", TAG+="uaccess" ''; destination = "/etc/udev/rules.d/69-probe.rules"; }) ]; </code></pre> I made a NixOS module</a> for adding these permissions. Hello, world!</h1> If you're running Nix, you can get everything you need from this</a> template. It contains a flake for setting up a rust toolchain and espflash</code> (a tool for flashing stuff to our esp). It also has a .cargo/config.toml</code> that I lifted from here</a> and a .helix/languages.toml</code> that gets rid of some spurious diagnostics (if you use helix). Entering the Nix shell however you like to do that (I use direnv) and running cargo run</code> should be all you need to do. (The .cargo/config.toml</code> file has some configuration to make cargo run</code> flash the board using espflash</code>.) ❯ cargo run Serial port: /dev/ttyACM0 Connecting... Chip type: ESP32-C3 (revision 3) Crystal frequency: 40MHz Flash size: 4MB Features: WiFi MAC address: 68:67:25:b7:49:18 App/part. size: 185920/4128768 bytes, 4.50% <...> Hello ESP32-C3! </code></pre> Blinky</h1> Next up: make the little led flash. Flashing leds are great because unlike the "hello world" example, you really see that something is happening physically on the device. Fortunately, esp-hal</code> has an example</a> for this. So let's copy it over to our esp-template</code> and run it: ❯ cargo run --bin blinky Serial port: /dev/ttyACM0 Connecting... <...> </code></pre> Ok, it runs but I don't see blinking. Let's look at the file we copied over: //! Blinks an LED //! //! This assumes that a LED is connected to the pin assigned to `led`. (GPIO5) </code></pre> Ah, but our board's documentation</a> says that the LED is on GPIO7, not 5. Changing the 5 to a 7, and... ❯ cargo run --bin blinky Serial port: /dev/ttyACM0 Connecting... <...> </code></pre> Still nothing. To be continued... nnnoiseless: porting audio code from C to rust 2020-07-12T00:00:00+00:00 I ported a C library to rust last week, and it went pretty smoothly. This is the story, and here</a> is the repo. The library in question is RNNoise</a>, a library for removing noise from audio. It works well, it runs fast, and best of all it has no knobs that you need to tune. There's even a rust binding</a>. So why bother porting it? Well, I need to patch it so that it would compile with MSVC, but my PR went unnoticed for a month. I thought about maintaining my own fork, but it's been more than 10 years since I last wrote anything in C or C++. And that's how I ended up porting RNNoise to rust. It probably wasn't the most efficient use of my time, but I had fun and learned something. There's a lot of information out there about porting C to rust, but the most useful resource for me was the fantastic talk</a> by Carol (Nichols || Goulding). It lays out a simple process for porting one function at a time: first, you set up the cargo to compile as a static library and you set up the C build system to link that static library into the C library (see the slides for the relevant Makefile and Cargo.toml snippets). Then you can port one function at time: the C code goes like this: +extern void _celt_lpc(opus_val16 *_lpc, const opus_val32 *ac, int p); +void __celt_lpc(opus_val16 *_lpc, const opus_val32 *ac, int p) -void _celt_lpc(opus_val16 *_lpc, const opus_val32 *ac, int p) { /* C body of _celt_lpc */ } </code></pre> and the rust code goes like this: +#[no_mangle] +pub extern "C" fn _celt_lpc(lpc: *mut f32, ac: *const f32, p: c_int) { + unsafe { + let lpc_slice = std::slice::from_raw_parts_mut(lpc, p as usize); + let ac_slice = std::slice::from_raw_parts(ac, p as usize + 1); + rs_celt_lpc(lpc_slice, ac_slice); + } +} + +fn rs_celt_lpc(lpc: &mut [f32], ac: &[f32]) { +// rust body of celt_lpc +} </code></pre> If you've watched the talk (which you should), you might notice that this is a tiny bit different from what they recommend: I've renamed the original C function instead of deleting it. I found that this helped me narrow down porting mistakes, because it made it easy to switch back and forth between the C and rust implementations. Pain points</h1> Most of the porting process was mechanical and easy. One of the less fun parts was porting code involving C structs. RNNoise has structs that (when ported to rust) look like this: #[repr(C)] struct RnnState { model: *const RnnModel, // Various buffers, whose sizes are determined by some subfields of `model`. vad_gru_state: *mut f32, noise_gru_state: *mut f32, denoise_gru_state: *mut f32, } </code></pre> An idomatic rust version might look something like struct RnnState { model: &'static RnnModel, vad_gru_state: Vec<f32>, noise_gru_state: Vec<f32>, denoise_gru_state: Vec<f32>, } </code></pre> but this isn't layout-compatible with the original C version, and so I need to stick with the original struct for as long as RnnState</code> is being accessed by both C and rust code. This increases the amount of unsafe</code> sprinkled around the rust code, and it was also the source of an annoying bug of the sort that I thought I had left behind by moving to rust. A Heisenbug</h1> At some point during the porting process, my tests started failing in release mode, but not in debug mode. Most likely some undefined behavior triggered by my amateurish attempts at unsafe code, but I couldn't quickly spot the problem and the prospect of a more careful round of debugging didn't spark a whole lot of joy. So I did something that I never would have dared to do in my C/C++ days: I ignored the problem and kept porting; after all, the tests were still working in debug mode. And sure enough, a few more ported functions later and rustc</code> found the problem for me: in a function taking a &RnnState</code> parameter, I was modifying data in the vad_gru_state</code> buffer. Since I was using unsafe code, rustc</code> didn't complain at first. But once I ported the RnnState</code> struct to safe and idiomatic rust, the compiler flagged the problem immediately. Performance</h1> After getting everything to 100% safe (if not particulary idiomatic) rust, it was time to check whether performance had suffered. Yes, apparently, by about 50%. The most obvious culprit was bounds checking: there was a lot of indexing in the C code, and some of it wasn't trivial to convert to a more rust-friendly, iterator-based version. First priority was the neural network evaluation: let m = ...; // At most 114. let n = ...; // At most 96. for i in 0..n { let output[i] = layer.bias[i] as f32; for j in 0..m { output[i] += layer.input_weights[j * n + i] as f32 * input[j]; } } </code></pre> I can already see you shaking your head. I'm doing naive matrix-vector multiplication with a 100x100ish matrix in column-major format</a>? Not only is this costing me bounds checks, it's terrible for memory locality. Swapping the weights storage from column- to row-major order only made things about 1.5% faster, but more importantly it made the whole thing iterator-friendly. Converting to zips and sums bought another 15%, leaving me only about 25-30% slower than the C code. for i in 0..n { let output[i] = layer.bias[i] as f32 + layer.input_weights[(i * m)..((i + 1) * m)] .iter() .zip(input) .map(|(&x, &y)| x as f32 * y) .sum(); } </code></pre> For my next optimization opportunity, I moved on to the function that computes cross-correlations</a>. The un-optimized version of this function looks like fn pitch_xcorr(xs: &[f32], ys: &[f32], xcorr: &mut [f32]) { for i in 0..xcorr.len() { xcorr[i] = xs.iter().zip(&ys[i..]).map(|(&x, &y)| x * y).sum(); } } </code></pre> but the C code contained a massive, manually-unrolled version. I'd skipped it while porting, but maybe I'd gain something from porting it over. Here's an abbreviated version of the optimized function, assuming that all lengths are a multiple of 4 (the real code also handles the case that they aren't). for i in (0..xcorr.len()).step_by(4) { let mut c0 = 0.0; let mut c1 = 0.0; let mut c2 = 0.0; let mut c3 = 0.0; let mut y0 = ys[i + 0]; let mut y1 = ys[i + 1]; let mut y2 = ys[i + 2]; let mut y3 = ys[i + 3]; for (x, y) in xs.chunks_exact(4).zip(ys[(i + 4)..].chunks_exact(4)) { c0 += x[0] * y0; c1 += x[0] * y1; c2 += x[0] * y2; c3 += x[0] * y3; y0 = y[0]; c0 += x[1] * y1; c1 += x[1] * y2; c2 += x[1] * y3; c3 += x[1] * y0; y1 = y[1]; c0 += x[2] * y2; c1 += x[2] * y3; c2 += x[2] * y0; c3 += x[2] * y1; y2 = y[2]; c0 += x[3] * y3; c1 += x[3] * y0; c2 += x[3] * y1; c3 += x[3] * y2; y3 = y[3]; } } </code></pre> Basically, both inner and outer loops have been unrolled four times, and I've exploited the inner loop's unrolling to optimize the memory access pattern. Thanks to the amazing cargo asm</code></a>, I can happily report that there's no bounds-checking in the inner loop and that all the arithmetic has been auto-vectorized</a> to work four f32</code>s at a time. (Maybe it would get even faster if I unrolled 8 times and compiled with AVX enabled; I haven't tried that yet.) This change more than doubled the speed of pitch_xcorr</code>, and gained me about 10% overall. More importantly, it showed me how to coerce the compiler into auto-vectorizing something that it hadn't auto-vectorized before. I went back to the neural network code and replaced things like xs.iter().zip(ys).map(|(&x, &y)| x as f32 * y).sum() </code></pre> with things like { let mut sum0 = 0.0; let mut sum1 = 0.0; let mut sum2 = 0.0; let mut sum3 = 0.0; for (x, y) in xs.chunks_exact(4).zip(ys.chunks_exact(4)) { sum0 += x[0] * y[0]; sum1 += x[1] * y[1]; sum2 += x[2] * y[2]; sum3 += x[3] * y[3]; } sum0 + sum1 + sum2 + sum3 } </code></pre> for another 20% improvement. Current score: the rust version (still 100% safe) is about 15% faster, and there's probably plenty more still on the table. The performance lesson I learned from this is that bounds checking can be expensive in numerical code and iterator-style code can help a bit, but if you really want faster numerical code then you need to write in a style that the auto-vectorizer likes. (Or you could use the SIMD intrinsics</a> directly, but that's another story.) A huge thank you to cargo</code></h1> Like I wrote above, it's been a while since I did any C/C++, and because of that I've started to take tools like cargo for granted. This little porting project brought back some memories, mostly because about half of the code in RNNoise was actually "vendored" from opus</a>. I put "vendored" in quotes because I usually think of vendoring as involving a subdirectory (maybe even a git submodule if I'm lucky) with its own build artifacts. That's not what's going on here, though; I'm just talking about files that were copied from the source directory of one project to the source directory of another, complete with never-used functions and never-def'ed ifdefs. The thing is, though, that I understand exactly why they did it: it's by far the easiest way to share code between C projects. So I just want to finish by saying a big "thank you" to cargo</code> and crates.io</code> for making me not have to deal with C dependency management any more. Gaussian bubble clusters 2020-06-15T00:00:00+00:00 Suppose I ask you to divide $\mathbb{R}^n$ into two pieces of fixed Gaussian measure so that the surface area of the boundary is as small as possible. The Gaussian isoperimetric inequality states that the best way to do it is by cutting $\mathbb{R}^n$ with a hyperplane: Now what if I ask for three parts instead of two? Gaussian noise sensitivity 2020-06-15T00:00:00+00:00 Suppose that $X$ and $Y$ are positively correlated standard Gaussian vectors in $\mathbb{R}^n$. Define the noise sensitivity of $A \subset \mathbb{R}^n$ to be the probability that $X \in A$ and $Y \not \in A$. Borell proved that for any $a \in (0,1)$, half-spaces minimize the noise sensitivity subject to the constraint $\mathrm{Pr}(X \in A)=a$. This inequality can be seen as a strengthening of the Gaussian isoperimetric inequality: in the limit as the correlation goes to one the noise sensitivity is closely related to the surface area, because if $X \in A$ and $Y \not \in A$ are close together then they're probably both close to the boundary of $A$. From a more applied point of view, Borell's inequality and its discrete relatives played a surprising and crucial role in studying hardness of approximation in theoretical computer science. Patches, Part 5: Pseudo-edges 2019-05-07T00:00:00+00:00 This is the fifth (and final planned) post in a series on some new ideas in version control. To start at the beginning, go here</a>. The goal of this post is to describe pseudo-edges: what they are, how to compute them efficiently, and how to update them efficiently upon small changes. To recall the important points from the last post</a>: We (pretend, for now, that we) represent the state of the repository as a graph in memory: one node for every line, with a directed edges that enforce ordering constraints between two lines. Each line has a flag that says whether it is deleted or not.</li> The current output of the repository consists of just those nodes that are not deleted, and there is an ordering constraint between two nodes if there is a path in the graph between them, but note that the path is allowed to go through deleted nodes.</li> Applying a patch to the repository is very efficient: the complexity of applying a patch is proportional to the number of changes it makes.</li> Rendering a the current output to a file is potentially very expensive: its complexity requires traversing the entire graph, including nodes that are marked as deleted. To the extent we can, we'd like to reduce this complexity to the number of live nodes in the graph.</li> </ul> The main idea for solving this is to add "pseudo-edges" to the graph: for every path that connects two live nodes through a sequence of deleted nodes, add a corresponding edge to the graph. Once this is done, we can render the current output without traversing the deleted parts of the graph, because every ordering contraint that used to depend on some deleted parts is now represented by some pseudo-edge. Here's an example: the deleted nodes are in gray, and the pseudo-edge that they induce is the dashed arrow. We haven't really solved anything yet, though: once we have the pseudo-edges, we can efficiently render the output, but how do we compute the pseudo-edges? The naive algorithm (look at every pair of live nodes, and check if they're connected by a path of deleted nodes) still depends on the number of deleted nodes. Clearly, what we need is some sort of incremental way to update the pseudo-edges. Deferring pseudo-edges</h1> The easiest way that we can reduce the amount of time required for computing pseudo-edges is simply to do it rarely. Specifically, remember that applying a patch can be very fast, and that pseudo-edges only need to be computed when outputting a file. So, obviously, we should only update the pseudo-edges when it's time to actually output the file. This sounds trivial, but it can actually be significant. Imagine, for example, that you're cloning a repository that has a long history; let's say it has n</code> patches, each of which has a constant size, and let's assume that computing pseudo-edges takes time O(m)</code>, where m</code> is the size of the history. Cloning a repository involves downloading all of those patches, and then applying them one-by-one. If we recompute the pseudo-edges after every patch application, the total amount of time required to clone the repository is O(n^2)</code>; if we apply all the patches first and only compute the pseudo-edges at the end, the total time is O(n)</code>. You can see how ojo</code> implements this deferred pseudo-edge computation here</a>: first, it applies all of the patches; then it recomputes the pseudo-edges</a>. Connected deleted components</h1> Deferring the pseudo-edge computation certainly helps, but we'd also like to speed up the computation itself. The main idea is to avoid unnecessary recomputation by only examining parts of the graph that might have actually changed. At this point, I need to admit that I don't know whether what I'm about to propose is the best way of updating the pseudo-edges. In particular, its efficiency rests on a bunch of assumptions about what sort of graphs we're likely to encounter. I haven't made any attempt to test these assumptions on actual large repositories (although that's something I'd like to try in the future). The main assumption is that while there may be many deleted nodes, they tend to be collected into a large number of connected components, each of which tends to be small. What's more, each patch (I'll assume) tends to only affect a small number of these connected components. In other words, the plan will be: keep track (incrementally) of connected components made up of deleted nodes,</li> when applying or reverting a patch, figure out which connected components were touched, and only recompute paths among the live nodes that are on the boundary of one of the dirty connected components.</li> </ul> Before talking about algorithms, here are some pictures that should help unpack what it is that I actually mean. Here is a graph containing three connected components of deleted nodes (represented by the rounded rectangles): When I delete node h</code>, it gets added to one of the connected components, and I can update relevant pseudo-edges without looking at the other two connected components: If I delete node d</code> then it will cause all of the connected components to merge: This isn't hard to handle, it just means that we should run our pseudo-edge-checking algorithm on the merged component. Maintaining the components</h1> To maintain the partition of deleted nodes into connected components, we use a disjoint-set</a> data structure. This is very fast (pretty close to constant time) when applying patches, because applying patches can only enlarge deleted components. It's slower when reverting patches, because the disjoint-set algorithm doesn't allow splitting: when reverting patches, connected components could split into smaller ones. Our approach is to defer the splitting: we just mark the original connected component as dirty. When it comes time to compute the pseudo-edges, we explore the original component, and figure out what the new connected pieces are. The disjoint-set data structure is implemented in the ojo_partition</code></a> subcrate. It appears in the Graggle</code> struct</a>; note also the dirty_reps</code> member: that's for keeping track of which parts in the partition have been modified by a patch and require recomputing pseudo-edges. We recompute the components here</a>. Specifically, we consider the subgraph consisting only of nodes that belong to one of the dirty connected components. We run Tarjan's algorithm on that subgraph to find out what the new connected components are. On each of those components, we recompute the pseudo-edges. Recomputing the pseudo-edges</h1> The algorithm for this is: after deleting the node, look at the deleted connected component that it belongs to, including the "boundary" consisting of live nodes: Using depth-first search, check which of the live boundary nodes (in this case, just a</code> and i</code>) are connected by a path within that component (in this case, they are). If so, add a pseudo-edge. The complexity of this algorithm is O(nm)</code>, where n</code> is the number of boundary nodes, and m</code> is the total number of nodes in the component, including the boundary (because we need to run n</code> DFSes, and each one takes O(m)</code> time). The hope here is that m</code> and n</code> are small, even for large histories. For example, I hope that n</code> is almost always 2; at least, this is the case if the final live graph is totally ordered. This algorithm is implemented here</code></a>. Unapplying, and pseudo-edge reasons</h1> There's one more wrinkle in the pseudo-edge computation, and it has to do with reverting patches: if applying a patch created a pseudo-edge, removing a patch might cause that pseudo-edge to get deleted. But we have to be very careful when doing so, because a pseudo-edge might have multiple reasons for existing. You can see why in this example from before: The pseudo-edge from a</code> to d</code> is caused independently by the both the b -> c</code> component and the cy -> cl -> e</code> component. If by unapplying some patch we destroy the b -> c</code> component but leave the cy -> cl -> e</code> component untouched, we have to be sure not to delete the pseudo-edge from a</code> to d</code>. The solution to this is to track to "reasons" for pseudo-edges, where each "reason" is a deleted connected component. This is a many-to-many mapping between connected deleted components and pseudo-edges, and it's stored in the pseudo_edge_reasons</code> and reason_pseudo_edges</code> members of the GraggleData</code> struct</a>. Once we store pseudo-edge reasons, it's easy to figure out when a pseudo-edge needs deleting: whenever its last reason becomes obsolete</a>. Pseudo-edge spamming: an optimization</h1> We've finished describing ojo</code>'s algorithm for keeping pseudo-edges up to date, but there's stil room for improvement. Here, I'll describe a potential optimization that I haven't implemented yet. It's based on a simple, but non-quite-correct, algorithm for adding pseudo-edges incrementally: every time you mark a node as deleted, add a pseudo-edge from each of its in-neighbors to each of its out-neighbors. I call this "pseudo-edge spamming" because it just eagerly throws in as many pseudo-edges as needed. In pictures, if we have this graph and we delete the "deleted" line, then we'll add a pseudo-edge from the in-neighbor of "deleted" (namely, "first") to the out-neighbor of "deleted" (namely, "last"). This algorithm has two problems. The first is that it isn't complete: you might also need to add pseudo-edges when adding an edge where at least one end is deleted. Consider this example, where our graph consists of two disconnected parts. If we add an edge from "deleted 1" to "deleted 2", clearly we also need to add a pseudo-edge between each of the "first" nodes and each of the "last" nodes. In order to handle this case, we really do need to explore the deleted connected component (which could be slow). The second problem with our pseudo-edge spamming algorithm is that it doesn't handle reverting patches: it only describes how to add pseudo-edges, not delete them. The nice thing about pseudo-edge spamming is that even if it isn't completely correct, it can be used as a fast-path in the correct algorithm: when applying a patch, if it modifies the boundary of a deleted connected component that isn't already dirty, use pseudo-edge spamming to update the pseudo-edges (and don't mark the component as dirty). In every other case, fall back to the previous algorithm. Patches, Part 4: Line IDs 2019-02-25T00:00:00+00:00 I've written quite a bit about the theory of patches and merging, but nothing yet about how to actually implement anything efficiently. That will be the subject of this post, and probably some future posts too. Algorithms and efficiency are not really discussed in the original paper</a>, so most of this material I learned from reading the pijul</a> source code. Having said that, my main focus here is on broader ideas and algorithms, and so you shouldn't assume that anything written here is an accurate reflection of pijul (plus, my pijul knowledge is about 2 years out of date by now). Three sizes</h1> Before getting to the juicy details, we have to decide what it means for things to be fast. In a VCS, there are three different size scales that we need to think about. From smallest to biggest, we have: The size of the change. Like, if we're changing one line in a giant file, then the size of the change is just the length of the one line that we're changing.</li> The size of the current output. In the case of ojo (which just tracks a single file), this is just the size of the file.</li> The size of the history. This includes everything that has ever been in the repository, so if the repository has been active for years then the size of the history could be much bigger than the current size.</li> </ol> The first obvious requirement is that the size of a patch should be proportional to the size of the change. This sounds almost too obvious to mention, but remember the definition of a patch from here</a>: a patch consists of a source file, a target file, and a function from one to the other that has certain additional properties. If we were to naively translate this definition into code, the size of a patch would be proportional to the size of the entire file. Of course, this is a solved problem in the world of UNIX-style diffs (which I mentioned all the way back in the first post</a>). The problem is to adapt the diff approach to our mathematical patch framework; for example, the fact that our files need not even be ordered means that it doesn't make sense to talk about inserting a line "after line 62." The key to solving this turns out to be to unique IDs: give every line in the entire history of the repository a unique ID. This isn't even very difficult: we can give every patch in the history of the repository a unique ID by hashing its contents. For each patch, we can enumerate the lines that it adds and then for the rest of time, we can uniquely refer to those lines like "the third line added by patch Ar8f</code>." Representing patches</h1> Once we've added unique IDs to every line, it becomes pretty easy to encode patches compactly. For example, suppose we want to describe this patch: Here, dA</code> is the unique ID of the patch that introduced the to-do, shoes, and garbage lines, and x5</code> is the unique ID of the patch that we want to describe. Anyway, the patch is now easy to describe by using the unique IDs to specify what we want to do: delete the line with ID dA/1</code>, add the line with ID x5/0</code> and contents "work", and add an edge from the line dA/2</code> to the line x5/0</code>. Let's have a quick look at how this is implemented in ojo, by taking a peek at the API docs</a>. Patches, funnily enough, are represented by the Patch</code></a> struct, which basically consists of metadata (author, commit message, timestamp) and a list of Change</code></a>s. The Change</code>s are the most interesting part, and they look like this: pub enum Change { NewNode { id: NodeId, contents: Vec<u8> }, DeleteNode { id: NodeId }, NewEdge { src: NodeId, dest: NodeId }, } </code></pre> In other words, the example that we saw above is basically all there is to it, as far as patches go. If you want to see what actual patches look like in actual usage, you can do that too because ojo keeps all of its data in human-readable text. After installing ojo (with cargo install ojo</code>), you can create a new repository (with ojo init</code>), edit the file ojo_file.txt</code> with your favorite editor, and then: $ ojo patch create -m "Initial commit" -a Me Created patch PMyANESmvMQ8WR8ccSKpnH8pLc-uyt0jzGkauJBWeqx4= $ ojo patch export -o out.txt PSc97nCk9oRrRl-2IW3H8TYVtA0hArdVtj5F0f4YSqqs= Successfully wrote the file 'out.txt' </code></pre> Now look in out.txt</code> to see your NewNode</code>s and NewEdge</code>s in all their glory. Antiquing</h1> I introduced unique IDs as a way to achieve compact representations of patches, but it turns out that they also solve a problem that I promised to explain two years ago</a>: how do I compute the "most antique" version of a patch? Or equivalently, if I have some patch but I want to apply it to a slightly different repository, how do I know whether I can do that? With our description of patches above, this is completely trivial: a patch can only add lines, delete lines, or add edges. Adding lines is always valid, no matter what the repository contains. Deleting lines and adding edges can be done if and only if the lines to delete, or the lines to connect, exist in the repository. Since lines have unique IDs, checking this is unambiguous. Actually, it's really easy because the line IDs are tied to the patch that introduced them: a patch can be applied if and only if all the patch IDs that it refers to have already been applied. For obvious reasons, we refer to these as "dependencies": the dependencies of a patch are all the other patches that it refers to in DeleteNode</code> and NewEdge</code> commands. You can see this in action here</a>. By the way, this method will always give a minimal set of dependencies (in other words, the most antique version of a patch), but it isn't necessarily the right thing to do. For example, if a patch deletes a line then it seems reasonable for it to also depend on the lines adjacent to the deleted line. Ojo might do this in the future, but for now it sticks to the minimal dependencies. Applying patches</h1> Now that we know how to compactly represent patches, how quickly can we apply them? To get really into detail here, we'd need to talk about how the state of the repository is represented on disk (which is an interesting topic on its own, but a bit out of scope for this post). Let's just pretend for now that the current state of the repository is stored as a graph in memory, using some general-purpose crate (like, say, petgraph</code></a>). Each node in the graph needs to store the contents of the corresponding line, as well as a "tombstone" saying whether it has been deleted (see the first post</a>). Assuming we can add nodes and edges in constant time (like, say, in petgraph</code>), applying a single change is a constant time operation. That means the time it takes to apply the whole patch is proportional to the number of changes. That's the best we could hope for, so we're done, right? What was even the point of the part about three size scales? Revenge of the ghosts</h1> Imagine you have a file that contains three lines: first line second line last line </code></pre> but behind the scenes, there are a bunch of lines that used to be there. So ojo's representation of your file might look like: Now let's imagine that we delete "second line." The patch to do this consists of a single DeleteLine</code> command, and it takes almost no time to apply: Now that we have this internal representation, ojo needs to create a file on disk showing the new state. That is, we want to somehow go from the internal representation above to the file first line last line </code></pre> Do you see the problem? Even though the output file is only two lines long, in order to produce it we need to visit all of the lines that used to be there but have since been deleted. In other words, we can apply patches quickly (in timescale 1), but rendering the output file is slow (in timescale 3). For a real VCS that tracks decade-old repositories, that clearly isn't going to fly. Pseudo-edges</h1> There are several ingredients that go into supporting fast rendering of output files (fast here means "timescale 2, most of the time", which is the best that we can hope for). Those are going to be the subject of the next post. So that you have something to think about until then, let me get you started: the key idea is to introduce "pseudo-edges," which are edges that we insert on our own in order to allow us to "skip" large regions of deleted lines. In the example above, the goal is to actually generate this graph: These extra edges will allow us to quickly render output files, but they open up a new can of worms (and were the source of several subtle bugs in pijul</code> last time I used it (i.e. two years ago)): how do we know when to add (or remove, or update) pseudo-edges? Keep in mind that we aren't willing to traverse the entire graph to compute the pseudo-edges, because that would defeat the purpose. Patches, Part 3: Graggles can have cycles 2019-02-19T00:00:00+00:00 Almost two years ago, I promised a series of three posts about version control. The first two (here</a> and here</a>) introduced a new (at the time) framework for version control. The third post, which I never finished, was going to talk about the datastructures and algorithms used in pijul</a>, a version control system built around that new framework. The problem is that pijul is a complex piece of software, and so I had lots of trouble wrapping my head around it. Two years later, I'm finally ready to continue with this series of posts (but having learned from my earlier mistakes, I'm not going to predict the total number of posts ahead of time). In the meantime, I've written my own toy version control system (VCS) to help me understand what's going on. It's called ojo</code></a>, and it's extremely primitive: to start with, it can only track a single file. However, it is (just barely) sophisticated enough to demonstrate the important ideas. I'm also doing my best to make the code is clear and well-documented. Graggles can have cycles</h1> As I try and ease back into this whole blogging business, let me just start with a short answer for something that several people have asked me (and which also confused me at some point). Graggles (which, as described in the earlier posts are a kind of generalized file in which the lines are not necessarily ordered, but instead form a directed graph) are not DAGs</a>; that is, they can have cycles. To see why, suppose we start out with this graggle The reason this thing isn't a file is because there's no prescribed order between the "shoes" line and the "garbage" line. Now suppose that my wife and I independently flatten this graggle, but in different ways (because apparently she doesn't care if I get my feet wet). Merging these two flattenings will produce the following graggle: Notice the cycle between "shoes" and "garbage!" Although I was surprised when I first noticed that graggles could have cycles, if you think about it a bit more then it makes a lot of sense: one graggle put a "garbage" dependency on "shoes" and the other put a "shoe" dependency on "garbage," and so when you merge them a cycle naturally pops out. Patches, Part 2: Merging, patches, and pijul 2017-05-13T00:00:00+00:00 In the last post</a>, I talked about a mathematical framework for a version control system (VCS) without merge conflicts. In this post I'll explore pijul</a>, which is a VCS based on a similar system. Note that pijul is under heavy development; this post is based on a development snapshot (I almost called it a "git" snapshot by mistake), and might be out of date by the time you read it. The main goal of this post is to describe how pijul handles what other VCSes call conflicts. We'll see some examples where pijul's approach works better than git's, and I'll discuss why. Some basics</h1> I don't want to write a full pijul tutorial here, but I do need to mention the basic commands if you're to have any hope of understanding the rest of the post. Fortunately, pijul commands have pretty close analogues in other VCSes. pijul init</code> creates a pijul repository, much like git init</code> or hg init</code>.</li> pijul add</code> tells pijul that it should start tracking a file, much like git add</code> or hg add</code>.</li> pijul record</code> looks for changes in the working directory and records a patch with those changes, so it's similar to git commit</code> or hg commit</code>. Unlike those two (and much like darcs record</code>), pijul record</code> asks a million questions before doing anything; you probably want to use the -a</code> option to stop it.</li> pijul fork</code> creates a new branch, like git branch</code>. Unlike git branch</code>, which creates a copy of the current branch, pijul fork</code> defaults to creating a copy of the master branch. (This is a bug, apparently.)</li> pijul apply</code> adds a patch to the current branch, like git cherry-pick</code>.</li> pijul pull</code> fetches and merges another branch into your current branch. The other branch could be a remote branch, but it could also just be a branch in the local repository.</li> </ul> Dealing with conflicts</h1> As I explained in the last post, pijul differs from other VCSes by not having merge conflicts. Instead, it has (what I call) graggles, which are different from files in that their lines form a directed acyclic graph instead of a totally ordered list. The thing about graggles is that you can't really work with them (for example, by opening them in an editor), so pijul doesn't let you actually see the graggles: it stores them as graggles internally, but renders them as files for you to edit. As an example, we'll create a graggle by asking pijul to perform the following merge: Here are the pijul commands to do this: $ pijul init # Create the initial file and record it. $ cat > todo.txt << EOF > to-do > * work > EOF $ pijul add todo.txt $ pijul record -a -m todo # Switch to a new branch and add the shoes line. $ pijul fork --branch=master shoes $ sed -i '2i* shoes' todo.txt $ pijul record -a -m shoes # Switch to a third branch and add the garbage line. $ pijul fork --branch=master garbage $ sed -i '2i* garbage' todo.txt $ pijul record -a -m garbage # Now merge in the "shoes" change to the "garbage" branch. $ pijul pull . --from-branch shoes </code></pre> The first thing to notice after running those commands is that pijul doesn't complain about any conflicts (this is not intentional; it's a known issue). Anyway, if you run the above commands then the final, merged version of todo.txt</code> will look like this: That's... a little disappointing, maybe, especially since pijul was supposed to free us from merge conflicts, and this looks a lot like a merge conflict. The point, though, is that pijul has to somehow produce a file -- one that the operating system and your editor can understand -- from the graggle that it maintains internally. The output format just happens to look a bit like what other VCSes output when they need you to resolve a merge conflict. As it stands, pijul doesn't have a very user-friendly way to actually see its internal graggles. But with a little effort, you can figure it out. The secret is the command RUST_LOG="libpijul::backend=debug" pijul info --debug </code></pre> For every branch, this will create a file named debug_<branchname></code> which describes, in graphviz's dot</code> format, the graggles contained in that branch. That file's a bit hard to read since it doesn't directly tell you the actual contents of any line; in place of, for example, "to-do", it just has a giant hex string corresponding to pijul's internal identifiers for that line. To decode everything, you'll need to look at the terminal output of that pijul command above. Part of it should look like this: DEBUG:libpijul::backend::dump: ============= dumping Contents DEBUG:libpijul::backend::dump: > Key { patch: PatchId 0x0414005c0c2122ca, line: LineId(0x0200000000000000) } Value (0) { value: [Ok("")] } DEBUG:libpijul::backend::dump: > Key { patch: PatchId 0x0414005c0c2122ca, line: LineId(0x0300000000000000) } Value (12) { value: [Ok("to-do\n")] } </code></pre> By cross-referencing that output with the contents of debug_<branchname></code>, you can reconstruct pijul's internal graggles. Just this once, I've done it for you, and the result is exactly as it should be: What should I do with a conflict?</h2> Since pijul will happily work with graggles internally, you could in principle ignore a conflict and work on other things. That's probably a bad idea for several reasons (for starters, there are no good tools for working with graggles, and their presence will probably break your build). So here's my unsolicited opinion: when you have a conflict, you should resolve it ASAP. In the example above, all we need to do is remove the >>></code> and <<<</code> lines and then record the changes: $ sed -i 3D;5D todo.txt $ pijul record -a -m resolve </code></pre> To back up my recommendation for immediate flattening, I'll give an example where pijul's graggle-to-file rendering is lossy. Here are two different graggles: But pijul renders both in the same way: This is a perfectly good representation of the graggle on the right, but it loses information from the one on the left (such as the fact that both "home" lines are the same, and the fact that "shop" and "home" don't have a prescribed order). The good news here is that as long as your graggle came from merging two files, then pijul's rendering is lossless. That means you can avoid the problem by flattening your graggles to files after every merge (i.e., by resolving your merge conflicts immediately). Like cockroaches</a>, graggles are important for the ecosystem as a whole, but you should still flatten them as soon as they appear. Case study 1: reverting an old commit</h2> It's (unfortunately) common to discover that an old commit introduced a show-stopper bug. On the bright side, every VCS worth its salt has some way of undoing the problematic commit without throwing away everything else you've written since then. But if the problematic commit predates a merge conflict, undoing it can be painful. As an illustration of what pijul brings to the table, we'll look at a situation where pijul's conflict-avoidance saves the day (at least, compared to git; darcs also does ok here). We'll start with the example merge from before, including our manual graggle resolution: Then we'll ask pijul to revert the "shoes" patch: $ pijul unrecord --patch=<hash-of-shoes-patch> $ pijul revert </code></pre> The result? We didn't have any conflicts while reverting the old patch, and the final file is exactly what we expected: Let's try the same thing with git: $ git init # Create the initial file and record it. $ cat > todo.txt << EOF > to-do > * work > EOF $ git add todo.txt $ git commit -a -m todo # Switch to a new branch and add the shoes line. $ git checkout -b shoes $ sed -i '2i* shoes' todo.txt $ git commit -a -m shoes # Switch to a third branch and add the garbage line. $ git checkout -b garbage master $ sed -i '2i* garbage' todo.txt $ git commit -a -m garbage # Now merge in the "shoes" change to the "garbage" branch. $ git merge shoes Auto-merging todo.txt CONFLICT (content): Merge conflict in todo.txt Automatic merge failed; fix conflicts and then commit the result. </code></pre> That was expected: there's a conflict, so we have to resolve it. So I edited todo.txt</code> and manually resolved the conflict. Then, # Commit the manual resolution. $ git commit -a -m merge # Try to revert the shoes patch. $ git revert <hash-of-shoes-patch> error: could not revert 4dcf1ae... shoes hint: after resolving the conflicts, mark the corrected paths hint: with 'git add <paths>' or 'git rm <paths>' hint: and commit the result with 'git commit' </code></pre> Since git can't "see through" my manual merge resolution, it can't handle reverting the patch by itself. I have to manually resolve the conflicting patches both when applying and reverting. I won't bore you with long command listings for other VCSes, but you can test them out yourself! I've tried mercurial (which does about the same as git in this example) and darcs (which does about the same as pijul in this example). A little warning about pijul unrecord</code></h2> I'm doing my best to present roughly equivalent command sequences for pijul and git, but there's something important you should know about the difference between pijul unrecord</code> and git revert</code>: pijul unrecord</code> modifies the history of the repository, as though the unrecorded patch never existed. In this way, pijul unrecord</code> is a bit like a selective version of git reset</code>. This is probably not the functionality that you want, especially if you're working on a public repository. Pijul actually does have the internal capability to do something closer to git revert</code> (i.e., undo a patch while keeping it in the history), but it isn't yet user-accessible. Sets of patches</h1> The time has come again to throw around some fancy math words. First, associativity. As you might remember, a binary operator (call it +</code>) is associative if (x + y) + z = x + (y + z)</code> for any x</code>, y</code>, and z</code>. The great thing about associative operators is that you never need parentheses: you can just write x + y + z</code> and there's no ambiguity. Associativity automatically extends to more than three things: there's also no ambiguity with w + x + y + z</code>. The previous paragraph is relevant to patches because perfect merging is associative, in the following sense: if I have multiple patches (let's say three to keep the diagrams manageable) then there's a unique way to perfectly merge them all together. That three-way merge can be written as combinations of two-way merges in multiple different ways, but every way that I write it gives the same result. Let's have some pictures. Here are my three patches: And here's one way I could merge them all together: first, merge patches p</code> and q</code>: Then, merge patches pm</code> (remember, that's the patch I get from applying p</code> and then m</code>, which in the diagram above is the same as qn</code>) and r</code>: Another way would be to first merge q</code> and r</code>, and then merge p</code> in to the result: Yet a third way would be to merge p</code> and q</code>, then merge q</code> and r</code>, and finally merge the results of those merges. This one gives a nice, symmetric picture: The great thing about our mathematical foundation from the previous post is that all these merges produce the same result. And I don't just mean that they give the same final file: they also result in the same patches, meaning that everyone will always agree on which lines in the final file came from where. There isn't even anything special about the initial configuration (three patches coming out of a single file). I could start with an arbitrarily complex history, and there would be an unambiguous way to merge together all of the patches that it contains. In this sense, we can say that the current state of a pijul branch is determined by a set of patches; this is in contrast to most existing VCSes, where the order in which patches are merged also matters. Reordering and antiquing patches</h2> One of the things you might have heard about pijul is that it can reorder patches (i.e. that they are commutative). This is not 100% accurate, and it might also be a bit confusing if you paid attention in my last post. That's because a patch, according to the definition I gave before, includes its input file. So if you have a patch p</code> that turns file A</code> into file B</code> and a patch q</code> that turns file B</code> into file C</code>, then it makes sense to apply p</code> and then r</code> but not the other way around. It turns out that pijul has a nice trick up its sleeve, which allows you to reorder patches as long as they don't "depend" (and I'll explain what that means precisely) on each other. The key idea behind reordering patches is something I call "antiquing." Consider the following sequenced patches: According to how we defined patches, the second patch (let's call it the garbage patch) has to be applied after the first one (the shoes patch). On the other hand, it's pretty obvious just by staring at them that the garbage patch doesn't depend on the shoes patch. In particular, the following parallel patches convey exactly the same information, without the dependencies: How do I know for sure that they convey the same information? Because if we take the perfect merge of the diagram above then we get back the original sequenced diagram by following the top path in the merge! This example motivates the following definition: given a pair of patches p</code> and q</code> in sequence: we say that q</code> can be antiqued if there exists some patch a(q)</code> starting at O</code> such that the perfect merge between p</code> and a(q)</code> involves q</code>: In a case like this, we can just forget about q</code> entirely, since a(q)</code> carries the same information. I call it antiquing because it's like making q</code> look older than it really is. One great thing about the "sets of patches" thing above is that it let us easily generalize antiquing from pairs of patches to arbitrarily complicated histories. I'll skip the details, but the idea is that you keep antiquing a patch -- moving it back and back in the history -- until you can't any more. The fact that perfect merges are associative implies, as it turns out, that every patch has a unique "most antique" version. The set of patches leading into the most antique version of q</code> are called q</code>'s dependencies. For example, here is a pair of patches where the second one cannot be antiqued (as an exercise, try to explain why not): Since the second patch can't be made any more antique, the first patch above is a dependency of the second one. In my next post, I'll come back to antiquing (and specifically, the question of how to efficiently find the most antique version of a patch). I promised to talk about reordering patches, so why did I spend paragraphs going on about antiques? The point is that (again, because of the associative property of perfect merges) patches in "parallel" can be applied in any order. The point of antiquing is to make patches as parallel as possible, and so then we can be maximally flexible about ordering them. That last bit is important, so it's worth saying again (and with a picture): patches in sequence cannot be re-ordered; the same information represented in parallel using an antique of q</code> is much more flexible. Case study 2: parallel development</h2> Since I've gone on for so long about reordering patches, let's have an example showing what it's good for. Let me start with some good news: you don't need to know about antiquing to use pijul, because pijul does it all for you: whenever pijul records a patch, it automatically records the most antique version of that patch. All you'll notice is the extra flexibility it brings. We'll simulate (a toy example of) a common scenario: you're maintaining a long-running branch of a project that's under active development (maybe you're working on a large experimental feature). Occasionally, you need to exchange some changes with the master branch. Finally (maybe your experimental feature was a huge success) you want to merge everything back into master. Specifically, we're going to do the following experiment in both pijul and git. The master branch will evolve in the following sequence: On our private branch, we'll begin from the same initial file. We'll start by applying the urgent fix from the master branch (it fixed a critical bug, so we can't wait): Then we'll get to implementing our fancy experimental features: I'll leave out the (long) command listings needed to implement the steps above in pijul and git, but let me mention the one step that we didn't cover before: in order to apply the urgent fix from master, we say $ pijul changes --branch master # Look for the patch you want $ pijul apply <hash-of-the-patch> </code></pre> In git, of course, we'll use cherry-pick. Now for the results. In pijul, merging our branch with the master branch gives no surprises: In git, we get a conflict: There's something else a bit funny with git's behavior here: if we resolve the conflict and look at the history, there are two copies of the urgent fix, with two different hashes. Since git doesn't understand patch reordering like pijul does, git cherry-pick</code> and pijul apply</code> work in slightly different ways: pijul apply</code> just adds another patch into your set of patches, while git cherry-pick</code> actually creates a new patch that looks a bit like the original. From then on, git sees the original patch and its cherry-picked one as two different patches, which (as we've seen) creates problems from merging down the line. And it gets worse: reverting one of the copies of the urgent fix (try it!) gives pretty strange results. By playing around with this example, you can get git to do some slightly surprising things. (For example, by inserting an extra merge in the right place, you can get the conflict to go away. That's because git has a heuristic where if it sees two different patches doing the same thing, it suppresses the conflict.) Pijul, on the other hand, understood that the urgent fix could be incorporated into my private branch with no lossy modifications. That's because pijul silently antiqued the urgent fix, so that the divergence between the master branch and my own branch became irrelevant. Conclusion</h1> So hopefully you have some idea now of what pijul can and can't do for you. It's an actively developed implementation of an exciting (for me, at least) new way of looking at patches and merges, and it has a simple, fast, and totally lossless merge algorithm with nice properties. Will it dethrone git? Certainly not yet. For a start, it's still alpha-quality and under heavy development; not only should you be worried about your data, it has several UI warts as well. Looking toward the future, I can see reasonable arguments in both directions. Arguing against pijul's future world domination, you could question the relevance of the examples I've shown. How often do you really end up tripping on git's little corner cases? Would the time saved from pijul's improvements actually justify the cost of switching? Those are totally reasonable questions, and I don't know the answer. But here's a more optimistic point of view: pijul's effortless merging and reordering might really lead to new and productive workflows. Are you old enough to remember when git was new and most people were still on SVN (or even CVS)? Lots of people were (quite reasonably) skeptical. "Who cares about easy branching? It's better to merge changes immediately anyway." Or, "who cares about distributed repositories? We have a central server, so we may as well use it." Those arguments sound silly now that we're all used to DVCSes and the workflow improvements that they bring, but it took time and experimentation to develop those workflows, and the gains weren't always obvious beforehand. Could the same progression happen with pijul? In the next post, I'll take a look at pijul's innards, focussing particularly on how it represents your precious data. Acknowledgement</h1> I'd like to thank Pierre-Étienne Meunier for his comments and corrections on a draft of this post. Of course, any errors that remain are my own responsibility. Patches, Part 1: Merging and patches 2017-05-08T00:00:00+00:00 A recent paper</a> suggested a new mathematical point of view on version control. I first found out about it from pijul</code>, a new version control system (VCS) that is loosely inspired by that paper. But if you poke around the pijul</code> home page</a>, you won't find many details about what makes it different from existing VCSes. So I did a bit of digging, and this series of blog posts is the result. In the first part (i.e. this one), I'll go over some of the theory developed in the paper. In particular, I'll describe a way to think about patches and merging that is guaranteed to never, ever have a merge conflict. In the second part, I'll show how pijul</code> puts that theory into action, and in the third part I'll dig into pijul</code>'s implementation. Before getting into some patch theory, a quick caveat: any real VCS needs to deal with a lot of tedious details (directories, binary files, file renaming, etc.). In order to get straight to the interesting new ideas, I'll be skipping all that. For the purposes of these posts, a VCS only needs to keep track of a single file, which you should think of as a list of lines. Patches</h1> A patch is the difference between two files. Later in this series we'll be looking at some wild new ideas, so let's start with something familiar and comforting. The kind of patches we'll discuss here go back to the early days of Unix: a patch works line-by-line (as opposed to, for example, word-by-word); and</li> a patch can add new lines, but not modify existing lines.</li> </ul> In order to actually have a useful VCS, you need to be able to delete lines also. But deleting lines turns out to add some complications, so we'll deal with them later. For an example, let's start with a simple file: my to-do list for this morning. Looking back at the list, I realize that I forgot something important. Here's the new one: To go from the original to-do list to the new one, I added the line with the socks. In the format of the original Unix "diff" utility, the patch would look like this: The "1a2" line is a code saying that we're going to add something after line 1 of the input file, and the next bit is obviously telling us what to insert. Since this blog isn't a command line tool, we'll represent patches with pretty diagrams instead of flat files. Here's how we'll draw the patch above: Hopefully it's self-explanatory, but just in case: an arrow goes from left to right to indicate that the line on the right is the same as the one on the left. Lines on the right with no arrow coming in are the ones that got added. Since patches aren't allowed to re-order the lines, the lines are guaranteed not to cross. There's something implicit in our notation that really needs to be said out loud: for us, a patch is tied to a specific input file. This is the first point where we diverge from the classic Unix ways: the classic Unix patch that we produced using "diff" could in principle be applied to any input file, and it would still insert "* put on socks" after the first line. In many cases that wouldn't be what you want, but sometimes it is. Merging</h1> The best thing about patches is that they can enable multiple people to edit the same file and then merge their changes afterwards. Let's suppose that my wife also decides to put things on my to-do list: she takes the original file and adds a line: Now there are two new versions of my to-do list: mine with the socks, and my wife's with the garbage. Let's draw them all together: This brings us to merging: since I'd prefer to have my to-do list as a single file, I want to merge my wife's changes and my own. In this example, it's pretty obvious what the result should be, but let's look at the general problem of merging. We'll do this slowly and carefully, and our endpoint might be different from what you're used to. Patch composition</h2> First, I need to introduce some notation for an obvious concept: the composition of two patches is the patch that you would get by applying one patch and then applying the other. Since a "patch" for us also includes the original file, you can't just compose any two old patches. If p</code> is a patch taking the file O</code> to the file A</code> and r</code> is a patch taking A</code> to B</code>, then you can compose the two (but only in one order!) to obtain a patch from O</code> to B</code>. I'll write this composition as pr</code>: first apply p</code>, then r</code>. It's pretty easy to visualize patch composition using our diagrams: to compute the composition of two paths, just "follow the arrows" to get the (dotted red) patch going from O</code> to B</code>. Merging as composition</h2> I'm going to define carefully what a merge is in terms of patch composition. I'll do this in a very math-professor kind of way: I'll give a precise definition, followed by some examples, and only afterwards will I explain why the definition makes sense. So here's the definition: if p</code> and q</code> are two different patches taking the file O</code> to the files A</code> and B</code> respectively, a merge of p</code> and q</code> is a pair of patches r</code> and s</code> such that r</code> and s</code> take A</code> and B</code> respectively to a common output file M</code>, and</li> pr = qs</code>.</li> </ul> We can illustrate this definition with a simple diagram, where the capital letters denote files, and the lower-case letters are patches going between them: Instead of saying that pr = qs</code>, a mathematician (or anyone who wants to sound fancy) would say that the diagram above commutes. Here is an example of a merge: And here is an example of something that is not a merge: This is not a merge because it fails the condition pr = qs</code>: composing the patches along the top path gives but composing them along the bottom path gives Specifically, the two patches disagree on which of the shoes in the final list came from the original file. This is the real meaning underlying the condition pr = qs</code>: it means that there will never be any ambiguity about which lines came from where. If you're used to using blame</code> or annotate</code> commands with your favorite VCS, you can probably imagine why this sort of ambiguity would be bad. A historical note</h2> Merging patches is an old idea, of course, and so I just want to briefly explain how the presentation above differs from "traditional" merging: traditionally, merging was defined by algorithms (of which there are many</a>). These algorithms would try to automatically find a good merge; if they couldn't, you would be asked to supply one instead. We'll take a different approach: instead of starting with an algorithm, we'll start with a list of properties that we want a good merge to satisfy. At the end, we'll find that there's a unique merge that satisfies all these properties (and fortunately for us, there will also be an efficient algorithm to find it). Merges aren't unique</h1> The main problem with merges is that they aren't unique. This isn't a huge problem by itself: lots of great things aren't unique. The problem is that we usually want to merge automatically, and an automatic system needs an unambiguous answer. Eventually, we'll deal with this by defining a special class of merges (called perfect merges) which will be unique. Before that, we'll explore the problem with some examples. A silly example</h2> Let's start with a silly example, in which our merge tool decides to add some extra nonsense: No sane merge tool would ever do that, of course, but it's still a valid merge according to our rule in the last section. Clearly, we'll have to tighten up the rules to exclude this case. A serious example</h2> Here is a more difficult situation with two merges that are actually reasonable: Both of these merges are valid according to our rules above, but you need to actually know what the lines mean in order to decide that the first merge is better (especially if it's raining outside). Any reasonable automatic merging tool would refuse to choose, instead requiring its user to do the merge manually. The examples above are pretty simple, but how would you decide in general whether a merge is unambiguous and can be performed automatically? In existing tools, the details depend on the merging algorithm. Since we started off with a non-algorithmic approach, let's see where that leads: instead of specifying explicitly which merges we can do, we'll describe the properties that an ideal merge should have. Perfect merges</h1> The main idea behind the definition I'm about to give is that it will never cause any regrets. That is, no matter what happens in the future, we can always represent the history just as well through the merge as we could using the original branches. Obviously, that's a nice property to have; personally, I think it's non-obvious why it's a good choice as the defining property of the ideal merge, but we'll get to that later. Ok, here it comes. Consider a merge: And now suppose that the original creators of patches p</code> and q</code> continued working on their own personal branches, which merged sometime in the future at the file F</code>: We say that the merge (r, s)</code> is a perfect merge if for every possible choice of the merge (u, v)</code>, there is a unique patch w</code> so that u = rw</code> and v = sw</code>. (In math terms, the diagram commutes.) We're going to call w</code> a continuation, since it tells us how to continue working from the merged file. To repeat, a merge is perfect if for every possible future, there is a unique continuation. A perfect merge</h2> Let's do a few examples to explore the various corners of our definition. First, an example of a perfect merge: It takes a bit of effort to actually prove that this is a perfect merge; I'll leave that as an exercise. It's more interesting to see some examples that fail to be perfect. A silly example</h2> Let's start with the silly example of a merge that introduced an unnecessary line: This turns out (surprise, surprise) not to be a perfect merge. To understand how our definition of merge perfection excludes merges like this, here is an example of a possible future without a continuation: Since our patches can't delete lines, there's no way to get from merged</code> to future</code>. A serious example</h2> Here's another example, the case where there is an ambiguity in the order of two lines in the merged file: This one fails to be a perfect merge because there is a future with no valid continuation: imagine that my wife and I manually created the desired merge. Now what patch (call it w</code>) could be put between merged</code> and future</code> to make everything commute? The only possibility is which isn't a legal patch because patches aren't allowed to swap lines. Terminological remarks</h2> If you've been casually reading about pijul, you might have encountered the word "pushout." It turns out that the pattern we used for defining a perfect merge is very common in math. Specifically, in category theory, suppose you have the following diagram (in which capital letters are objects and lowercase letters are morphisms): If for every u</code> and v</code> there is a unique w</code> such that the diagram commutes, then (r, s)</code> is said to be the pushout of (p, q)</code>. In other words, what we called a "perfect merge" above could also be called a "pushout in the category with files as objects and patches as morphisms." For most of this article, we'll ignore the general math terminology in favor of language that's more intuitive and specific to files and patches. Conflicts and graggles</h1> The main problem with perfect merges is that they don't always exist. In fact, we already saw an example: The pair of patches above has no perfect merge. We haven't actually proved it, but intuitively it's pretty clear, and we also discussed earlier why one potential merge fails to be perfect. Ok, so not every pair of patches can be merged perfectly. You probably knew that already, since that's where merge conflicts come from: the VCS doesn't know how to merge patches on its own, so you need to manually resolve some conflicts. Now we come to the coolest part of the paper: a totally different idea for dealing with merge conflicts. The critical part is that instead of making do with an imperfect merge, we enlarge the set of objects that the merge can produce. That is, not every pair of patches can be perfectly merged to a file, but maybe they can be merged to something else. This idea is extremely common in math, and there's even some general abstract nonsense showing that it can always be done: there's an abstract way to generalize files so that every pair of patches of generalized files can be perfectly merged. The miraculous part here is that in this particular case, the abstract nonsense condenses into something completely explicit and manageable. Graggles</h2> A file is an ordered list of lines. A graggle^{1</a> (a mixture of "graph" and "file") is a directed graph of lines. (Yes, I know it's a terrible name, but it's better than "object in the free finite cocompletion of the category of files and patches," which is what the paper calls it.) In other words, whereas a file insists on having its lines in a strict linear order, a graggle allows them to be any directed graph. It's pretty easy to see how relaxing the strict ordering of lines solves our earlier merging issues. For example, here's a perfect merge of the sort that caused us problems before:} In retrospect, this is a pretty obvious solution: if we don't know what order shoes and garbage should go in, we should just produce an output that doesn't specify the order. What's a bit less obvious (but is proved in the paper) is that when we work in the world of graggles instead of the world of files, every pair of patches has a unique perfect merge. What's even cooler is that the perfect merge is easy to compute. I'll describe it in a second, but first I have to say how patches generalize to graggles. A patch between two graggles (say, A</code> and B</code>) is a function (call it p</code>) from the lines of A</code> to the lines of B</code> that respects the partial order, in the sense that if there is a path from x</code> to y</code> in A</code> then there is a path from p(x)</code> to p(y)</code> in B</code>. (This condition is an extension of the fact that a patch between two files isn't allowed to change the order.) Here's an example: The perfect merge</h2> And now for the merge algorithm: let's say we have a patch p</code> going from the graggle A</code> to the graggle B</code> and another patch q</code> going from A</code> to C</code>. To compute the perfect merge of p</code> and q</code>, write down the graggles B</code> and C</code> next to each other, and then</li> whenever a line in B</code> and a line in C</code> share a "parent" in A</code>, collapse them into a single line.</li> </ol> That's it: two steps. Here's the algorithm at work on our previous example: we want to merge these two patches: So first, we write down the two to-be-merged files next to each other: For the second step, we see that both of the "to-do" lines came from the same line in the original file, so we combine those two into one. After doing the same to the "work" lines, we get the desired output: Working with graggles.</h2> By generalizing files to graggles, we got a very nice benefit: every pair of patches has a (unique) perfect merge, and we can compute it easily. But there's an obvious flaw: all the tools that we use (editors, compilers, etc.) work on files, not graggles. This is where the paper stops providing guidance, but there is an easy solution: whenever a merge results in something that isn't a file, just make a new patch that turns it into a file. We'll call this flattening, and here's an example: That looks like a merge conflict!</h2> If your eyes haven't glazed over by now (sorry, it's been a long post), you might be feeling a bit cheated: I promised you a new framework that avoids the pitfalls of manual merge resolution, but flattening looks an awful lot like manual merge resolution. I'll answer this criticism in more detail in the next post, where I demonstrate the pijul</code> tool and how it differs from git</code>. But here's a little teaser: the difference between flattening and manual merge resolution is that flattening is completely transparent to the VCS: it's just a patch like any other. That means we can do fun things, like re-ordering or reverting patches, even in the presence of conflicting merges. More on that in the next post. Deleting lines</h1> It's time to finally address something I put off way at the beginning of the post: the system I described was based on patches that can't delete lines, and we obviously need to allow deletions in any practical system. Unfortunately, the paper doesn't help here: it claims that you can incorporate deletion into the system I described without really changing anything, but there's a bug in the paper. Specifically, if you tweak the definitions to allow deletion then the category of graggles turns out not to be closed under pushouts any more. Here's an example where the merge algorithm in the paper turns out not to be perfect: (Since this post has dragged on long enough, I'll leave it as an exercise to figure out what the problem is). Ghost lines</h2> Fortunately, there's a trick to emulate line deletion in our original patch system. I got this idea from pijul, but I'll present it in a slightly different way. The idea is to allow "ghost" lines instead of actually deleting them. That is, we mark every line in our graggle as either "live" or "ghost." Then we add one extra rule to our patches: a live line can turn into a ghost line, but not the other way around. We'll draw ghost lines in gray, and arrows pointing to ghost lines will be dashed. Here's a patch that deletes the "shoes" line. The last remaining piece is to extend the perfect merge algorithm to cover our new graggles with ghost lines. This turns out to be easy; here's the new algorithm: Write down side-by-side the two graggles to be merged.</li> For every pair of lines with a common parent, "collapse" them into a single line, and if one of them was a ghost, make the collapsed line a ghost.</li> </ol> The bit in italics is the only new part, and it barely adds any extra complexity. Conclusion</h1> I showed you (in great detail) a mathy way of thinking about patches in a VCS, although I haven't shown a whole lot of motivation for it yet. At the very least, though, next time someone starts droning on about "patch theory," you'll have some idea what they're talking about. In the next post, I'll talk about pijul</a>, a VCS that is loosely based around the algorithms I described in this post. There you'll get to see some (toy) examples where pijul's solid mathematical underpinnings help it to avoid corner cases that trip up some more established VCSes. Acknowledgement</h1> I'd like to thank Pierre-Étienne Meunier for his comments and corrections on a draft of this post. Of course, any errors that remain are my own responsibility. 1</a>: An earlier version of this post called them "digles" (for directed graph file), but a couple years later I decided that "graggles" sounds a bit better. Plus, if you mispronounce</a> it a little, it fits in the pijul's whole bird theme.

joe.neeman

Linesweeper: a new library for boolean path ops

The Prospero challenge

Sleep and power usage

Rtt and defmt

Blinky

About

Going small: my adventures with an esp32c3

nnnoiseless: porting audio code from C to rust

Pain points</h1> Most of the porting process was mechanical and easy. One of the less fun parts was porting code involving C structs. RNNoise has structs that (when ported to rust) look like this:</p>

Gaussian bubble clusters

Gaussian noise sensitivity

Patches, Part 5: Pseudo-edges

Patches, Part 4: Line IDs

Patches, Part 3: Graggles can have cycles

Patches, Part 2: Merging, patches, and pijul

Acknowledgement</h1> I'd like to thank Pierre-Étienne Meunier for his comments and corrections on a draft of this post. Of course, any errors that remain are my own responsibility.</p>

Patches, Part 1: Merging and patches

Acknowledgement</h1>
I'd like to thank Pierre-Étienne Meunier for his comments and corrections on a draft of this post. Of course, any errors that remain are my own responsibility.</p>