Micro optimizing F# code with a dash of imperative code

In addition to re-designing my Nelder-Mead solver to improve usability, I have also recently dedicated some time looking into performance improvements. This is usually not my primary concern: I tend to focus first on readability and correctness first, and address performance issues later.

However, in the case of a solver, performance matters. In my specific case, the solver works as a loop, iteratively updating a candidate solution until it is good enough. Anything that can speed up that loop will directly speed up the overall solver, so it is worth making sure the code is as efficient as can be.

One particular code area I worked on is the solver termination rule. The changes I made resulted in a significant speedup (roughly 10x), at the expense of style: the final version does not look like idiomatic F# at all. In this post, I will go over these changes.

Converting an F# pipeline into a C# fluent interface

In my previous post, I went over one of the changes I made to my library, Quipu, to make it more C# friendly. In this installment, I will go over another design change, turning the initial F# version, which used a classic pipeline, into a Fluent Interface.

For reference, here is how the original F# pipeline looks like:

let f (x, y) = pown (x - 1.0) 2 + pown (y - 2.0) 2 + 42.0
let solverResult =
    NelderMead.objective f
    |> NelderMead.withTolerance 0.000_0001
    |> NelderMead.startFrom (Start.around (100.0; 100.0))
    |> NelderMead.minimize

This looks pretty similar to a Fluent Interface. It is not, though: it is a classic F# pipeline, chaining functions using the pipe-forward operator. From the F# side, this feels like a fluent interface, but for a C# consumer, it is more or less unusable.

Can we turn this into an actual C# friendly Fluent Interface? Yes we can, and this is what I will go over in this post.

Making a F# library C# friendly

During December, I have been aggressively redesigning my library, Quipu. I initially wrote Quipu because I needed a Nelder-Mead solver in .NET, and could not find one ready to use. And, because I intended to use it from F#, I wrote Quipu in a style that wasn’t particularly C# friendly.

As I was going through the code base with my chainsaw, I thought it would be an interesting exercise to try and make it pleasant to use from C# as well. This post will go through some of the process.

First, what is Quipu about? Quipu is an implementation of the Nelder-Mead algorithm, and searches for arguments that minimize a function. As an example, suppose you were given the function f(x,y) = (x-1)^2 + (y-2)^2 + 42, and wanted to know what values of x and y give you the lowest possible value for f. With Quipu, now in C#, this is how you would go about it:

#r "nuget: Quipu, 0.5.2"
using Quipu.CSharp;
using System;

Func<Double,Double,Double> f =
    (x, y) => Math.Pow(x - 1.0, 2) + Math.Pow(y - 2.0, 2) + 42.0;

var solverResult =
    NelderMead
        .Objective(f)
        .Minimize();

if (solverResult.HasSolution)
{
    var solution = solverResult.Solution;
    Console.WriteLine($"Solution: {solution.Status}");
    var candidate = solution.Candidate;
    var args = candidate.Arguments;
    var value = candidate.Value;
    Console.WriteLine($"f({args[0]:N3}, {args[1]:N3}) = {value:N3}");
}

This produces the following result, which happens to be correct:

Solution: Optimal
f(1.000, 2.000) = 42.000

I would also like to think that this C# code looks reasonably pleasant, whereas the original version (pre version 0.5.*) definitely was not. Let’s go over some of the changes I made to the original code!

Side note: my C# is pretty rusty at that point, if you have any thoughts or feedback on how to make this better, I am all ears!

More SIMD vectors exploration

Since my earlier post looking into SIMD vectors in .NET, I attempted a few more experiments, trying to understand better where they might be a good fit, and where they would not.

The short version: at that point, my sense is that SIMD vectors can be very handy for some specific scenarios, but would require quite a bit of work to be usable in the way I was hoping to use them. This statement is by no means meant as a negative on SIMD; rather, it reflects my realization that a SIMD vector is quite different from a mathematical vector.

All that follows should also be taken with a big grain of salt. SIMD is entirely new to me, and I might not be using it right. While on that topic, I also want to thank @xoofx for his very helpful comments - much appreciated!

Anyways, with these caveats out of the way, let’s dive into it.

My premise approaching SIMD was along these lines: “I write a lot of code that involves vectors. Surely, the Vector class should be a good fit to speed up some of that code”.

To explore that idea, I attempted to write a few classic vector operations I often need, both in plain old F# and using SIMD vectors, trying to benchmark how much performance I would gain. In this post, I’ll share some of the results, and what I learnt in the process.

You can find the whole code here on GitHub.

Should I use dotnet SIMD Vectors?

Even though a lot of my work involves writing computation-heavy code, I have not been paying close attention to the System.Numerics namespace, mainly because I am lazy and working with plain old arrays of floats has been good enough for my purposes.

This post is intended as a first dive into the question “should I care about .NET SIMD-accelerated types”. More specifically, I am interested in understanding better Vector<T>, and in where I should use it instead of basic arrays for vector operations, a staple of machine learning.

Spoiler alert: some of my initial results surprised me, and I don’t understand yet what drives the speed differences between certain examples. My intent here is to share what I found so far, which I found interesting enough to warrant further exploration later on.

Anyways, let’s dive in. As a starting point, I decided to start with a very common operation in Machine Learning, computing the Euclidean distance between 2 vectors.

You can find the whole code here on GitHub.