Talking to Sia Hosts from…Ruby??

Sia’s native language is Go. Back in 2013, when the first few lines of Sia were being written, Go was very young: v1.1 of the language had been released just a few months prior. Perhaps it would have been more pragmatic to fork Bitcoin’s C++ codebase instead of starting from scratch in an unproven language, but hey, we were young and ambitious, and Go was exciting, productive, and easy to learn. (Personally, I’ve never regretted that decision; on the contrary, sometimes I gaze out the window and shed a tear for the parallel-universe-Luke who is currently waist-deep in C++ spaghetti.)

Anyway, our decision to use Go had some significant ramifications. In particular, Go has limited interoperability with other languages: Calling into C libraries from Go is possible (but painful), and calling Go from C was not possible at all until the release of Go 1.5 in late 2015. This meant that any sort of interop would have to happen at a higher level: our HTTP API. And while this works fairly well in practice – Sia-UI is just a client of the API, after all – it also precludes the possibility of low-level cross-language bindings. Speaking HTTP over a socket is vastly slower than directly invoking a function (especially if you need to convert to JSON and back), and our use of REST makes certain constructs awkward or impractical to express.

The real problem with the siad API, however, is that it is doing double-duty: it is both the backend for user-facing software like Sia-UI and siac, and the sole cross-language API available to developers who want to use the Sia network. This introduces some tension. siad has a sophisticated renter that picks good hosts, automatically renews your contracts and repairs your data, and presents a file-oriented interface, and so its HTTP API reflects those design choices. If a developer wants to color outside those lines – say, form a contract with a specific host, or work with block storage rather than files – they’re out of luck. Storing and manipulating a database on Sia, for example, is possible in principle, but cannot be accomplished with the siad API.

In practice, this isn’t a huge deal, because most developers want an automatically-managed, file-oriented interface anyway. But as the ecosystem matures, apps built on Sia will naturally seek greater control over how they interact with the network. When they’re ready to cross that bridge, I want to make sure we can provide them with powerful tools that will meet their needs. Building those tools has been the primary focus of the us project.

After developing the Sia mobile app proof-of-concept with @DangerCZ, I was inspired to revisit the possibility of low-level cross-language bindings in us. The mobile app was created with a program called gomobile that leverages quantum pixie dust (or some similar mechanism, not clear on the specifics) to generate Java and Obj-C bindings from Go code, allowing it to be run on iOS and Android devices. Well, if gomobile can do it, so we can we! After reading the relevant docs and some blog posts, I determined the secret of the quantum pixie dust: by means a special flag and some wrapper code, the Go compiler can be made to produce a shared library instead of a binary program.

This was tremendously exciting, because most languages support calling into shared libraries via their Foreign Function Interface. In other words, with a bit of wrapper code, it would be possible to talk directly to Sia hosts from every mainstream language under the sun: C, Java, Rust, Python, Node.js, Haskell, and yes, even Ruby!

And so, after some feverish hacking over the weekend, I’m happy to report that the Sia ecosystem is now a bit friendlier to non-Go programmers: us can now be compiled to a shared library, and soon it will have bindings for your favorite language.

For fun, I decided to try writing a set of bindings myself – for a language that I had never used before. I picked Ruby, mostly because it has a reputation for being slow and “magical,” whereas us is fast and (intentionally) non-magical. The result is, I’m sure, far from idiomatic Ruby, but it does work, and hopefully someone more experienced can make it pretty later. So, without further ado, here’s a snippet of real Ruby code that will upload and download a file, talking directly to Sia hosts:

hs = Us::HostSet.new("<shard server address>")
hs.add_host(Us::Contract.new("<hex contract string>"))
Us::FileSystem.new("meta", hs) do |fs|
    fs.create("foo.txt", minHosts: 1) do |f|
        str = "Hello from Ruby!"
        f.write(str)
        puts "Uploaded:   " + str
    end
    fs.open("foo.txt") do |f|
        puts "Downloaded: " + f.read(16)
    end
end

Some languages make these bindings easier to write than others, but there’s a decent chance that you'll be able to write bindings for your language of choice in an afternoon. It’s a fairly mechanical process: you just load the us.so library, wrap each library function in a native function/method, define a class or some helper types if you feel like it, and you’re done. For the Ruby bindings, it took me about two hours to write the requisite 100-ish lines of code, and I had never written Ruby before.

If you’re interested in contibuting bindings, check out the us-bindings repo. You can message me on Discord, reddit, email, etc. and I’ll help you get set up. Let’s make something cool! :)

Technical Details

For interested developers, here’s how the Ruby bindings work.

First, I wrote a wrapper in Go that uses cgo to compile a shared library. It consists of a bunch of small functions like this:

//export us_fs_init
func us_fs_init(root *C.char, hs unsafe.Pointer) unsafe.Pointer {
    pfs := renterutil.NewFileSystem(C.GoString(root), loadPtr(hs).(*renterutil.HostSet))
    return storePtr(pfs)
}

Cgo doesn’t allow you to pass Go memory to C, so you can’t return an object like *renterutil.FileSystem directly. Instead, I use a gross hack: the Go wrapper maintains a global table of Go objects, and wherever we would normally return a pointer, we return an index into this table instead. The C code then passes these indices back to the Go functions when it needs to reference a previously-created object. For example, the hs argument in the function above is an index corresponding to a *renterutil.HostSet, which we retrieve by calling loadPtr.

Errors are dealt with similarly, using a construct much like C’s global errno and perror(). Since C functions can only return a single value, it’s cumbersome to mimic the Go convention of returning (Thing, error). Instead, functions that can fail will return “falsey” values like 0, nil, etc., upon which your code can call us_error() to retrieve the associated error string. This string can then be converted to an exception (or whatever error-handling mechanism your language provides).

I compiled the wrapper with the -buildmode=c-shared flag, producing the us.so shared library. This is the file that the other language bindings will need in order to call us functions. Since it contains the Go runtime, it’s pretty hefty – 8MB! I hope to slim this down in the future, but it’s acceptable for now.

Next, I used the ffi Ruby gem to create the Us wrapper class. This required specifying the signatures of each function I needed to call:

module Us
    extend FFI::Library
    ffi_lib './us.so'
    attach_function :us_error, [], :string
    attach_function :us_contract_init, [:pointer, :pointer], :void
    attach_function :us_fs_init, [:string, :pointer], :pointer
    attach_function :us_fs_create, [:pointer, :string, :int], :pointer
    # ...etc.

Then I defined classes for each of the important types (Contract, HostSet, FileSystem, and File) and implemented methods on them. Each class constructor calls the corresponding us_xxx_init C function, which returns a pointer (actually an index, remember) that can be used in future calls:

class HostSet < FFI::Pointer
    def add_host(contract)
        ok = Us.us_hostset_add(self, contract)
        raise Us.us_error() if !ok
    end
    def initialize(shard_addr)
        hs = Us.us_hostset_init(shard_addr)
        raise Us.us_error() if hs.nil?
        super(hs)
    end
end

Lastly, I made the API a little nicer by automatically closing instances of File and FileSystem if their constructor is called with a block, as in the example code above.

Overall, the process went quite smoothly. I’m guessing that FFI is easier in Ruby than it is in most languages, but regardless, the experience has made me optimistic that we’ll be able to write bindings for other languages without pulling out too much hair. :)