Skynet represents a huge step forward for the Sia network. It both greatly simplifies the average user’s access to data, and provides decentralized file sharing via short links. At its heart, Skynet is a natural extension of the existing Sia platform. It consists of three main components:
In this post, I’ll show you how to use the us library to implement each of
these components in just a few dozen lines of code. When we’re done, you’ll have
a program that can upload files and give you a link, or take a link and
give you a file – just like a Skynet portal. The only prerequisite is
a muse server with some contracts. (If you need such a server, ping me on
reddit or Discord and I’ll give you one for free.)
Normally, files on Sia are encrypted and uploaded to a subset of the hosts on the network, and each file’s metadata – specifying its hosts, sectors, erasure-encoding settings, and decryption key – is stored on the local computer. If you want to download the file later, you need this metadata.
This approach works fine, but it doesn’t meet Skynet’s requirements. We want Skynet users to be able to share files using short links. If we kept all of the usual metadata, Skynet links would be hundreds of characters long! By contrast, links on Skynet today are just 46 characters. How is this possible?
The trick is that the link does not encode any information about which hosts the file is stored on. Instead, it encodes the Merkle root of a 4 MiB sector, and some positional information that denotes which part of the sector contains the file. In order to download the file, we simply ask each host whether they have this sector, and then use the first host that replies in the affirmative.
(Note that this strategy only works for files that are smaller than 4 MiB. For larger files, we need another indirection: instead of the 4 MiB sector storing the file data directly, it stores a list of other sectors that store the data. For the purposes of this article, we’ll focus on the simpler case of single-sector files.)
There are three steps involved in uploading a Skynet file. First, we need to construct the sector that contains the file data, prefixed by a header. We then upload this sector to a set of hosts. Finally, we encode a link using the Merkle root of the sector.
The Skynet sector header consists of two parts: a fixed-size “layout” section (encoded in binary), and a variable-size “metadata” section (encoded in JSON):
type linkfileLayout struct {
Version uint8
Filesize uint64
MetadataSize uint64
FanoutSize uint64
FanoutDataPieces uint8
FanoutParityPieces uint8
CipherType [8]byte
CipherKey [64]byte
}
type linkfileMetadata struct {
Filename string
Mode os.FileMode
}
For our purposes, we only care about unencrypted files that fit in a single
sector, so we can ignore the Fanout and Cipher fields. Let’s
write a simple function that constructs a sector from a file:
func encodeLinkFile(name string, data []byte) *[renterhost.SectorSize]byte {
metadata, _ := json.Marshal(linkfileMetadata{
Filename: name,
Mode: 0666,
})
layout := linkfileLayout{
Version: 1,
FileSize: uint64(len(data)),
MetadataSize: uint64(len(metadata)),
}
var sector [renterhost.SectorSize]byte
buf := bytes.NewBuffer(sector[:0])
binary.Write(buf, binary.LittleEndian, layout)
buf.Write(metadata)
buf.Write(data)
return §or
}
Once we have a sector, uploading it is easy. We’ll request the active contracts
from our muse server, open a Session for each contract, and use the Write
RPC to upload the data to each host. And since this is Go, we’ll of course
want to upload in parallel:
func uploadSector(mc *muse.Client, sector *[renterhost.SectorSize]byte) error {
currentHeight, err := mc.SHARD().ChainHeight()
if err != nil {
return err
}
contracts, err := mc.Contracts("default")
if err != nil {
return err
}
results := make(chan *renterutil.HostError)
for _, c := range contracts {
go func(c muse.Contract) {
s, err := proto.NewSession(c.HostAddress, c.HostKey,
c.ID, c.RenterKey, currentHeight)
if err == nil {
// Append is a convenient wrapper around the Write RPC
_, err = s.Append(sector)
s.Close()
}
results <- &renterutil.HostError{HostKey: c.HostKey, Err: err}
}(c)
}
var errs renterutil.HostErrorSet
for range contracts {
if err := <-results; err.Err != nil {
errs = append(errs, err)
}
}
if len(errs) == len(contracts) {
return errs // none of the uploads succeeded
}
return nil
}
Great – we constructed a Skynet sector and uploaded it to some hosts. Now we just need to encode it into a link. Skynet links use a fancy algorithm for cramming a version, offset, and length into 16 bits. (We’ll omit the implementation here, since it’s of lesser importance; if you were inventing Skynet from scratch, you could use a simpler approach at the cost of slightly longer links.)
func encodeLink(data []byte, sector *[renterhost.SectorSize]byte) string {
var linkdata [34]byte
binary.LittleEndian.PutUint16(linkdata[:2], computeLinkBits(uint32(len(data))))
root := merkle.SectorRoot(sector)
copy(linkdata[2:], root[:])
return "sia://" + base64.RawURLEncoding.EncodeToString(linkdata[:])
}
We now have all the pieces we need to pin a file, making it accessible from any Skynet portal. Let’s try it out on this blog post:
func main() {
mc := muse.NewClient("<my muse server URL>")
data, _ := ioutil.ReadFile("post.html")
sector := encodeLinkFile("post.html", data)
if err := uploadSector(mc, sector); err != nil {
log.Fatal(err)
}
log.Println(encodeLink(data, sector))
// prints "sia://KABaBLrJzYkQ5GYWvVpbw5zRhUnQL_Onv2JU4Y8-arC8Vw"
}
Neat! We can confirm that this worked by accessing the link via a Skynet portal.
Downloading is just the inverse of uploading. The main difference is
that we don’t know which hosts the file was stored on, so we need to try all of
them. That means our muse server needs to have contracts with every host.
(There’s a reason not everyone runs a portal!) For our purposes, we’ll assume
that our muse server has at least one contract with a host storing the file.
We’ll proceed in the opposite direction of uploading: first we’ll parse the
link, then we’ll download the sector, then we’ll extract the data. As before,
we’ll omit the fancy logic of decodeLinkBits:
func decodeLink(link string) (req renterhost.RPCReadRequestSection, err error) {
link = strings.TrimPrefix(link, "sia://")
data, err := base64.RawURLEncoding.DecodeString(link)
if err != nil {
return
} else if len(data) != 34 {
err = errors.New("invalid Link")
return
}
req.Length, err = decodeLinkBits(binary.LittleEndian.Uint16(data[:2]))
copy(req.MerkleRoot[:], data[2:])
return
}
Now we can call the Read RPC with this request object to retrieve the header
and file data. But if the data could be stored on any host, how do we decide
which host to try first? Well, for now, we’ll just do the simplest thing that
works, which is to try all of them, one at a time:
func downloadSector(mc *muse.Client, req renterhost.RPCReadRequestSection) ([]byte, error) {
contracts, err := mc.Contracts("default")
if err != nil {
return nil, err
}
var errs renterutil.HostErrorSet
for _, c := range contracts {
s, err := proto.NewSession(c.HostAddress, c.HostKey, c.ID, c.RenterKey, 0)
if err == nil {
var buf bytes.Buffer
err = s.Read(&buf, []renterhost.RPCReadRequestSection{req})
s.Close()
if err == nil {
return buf.Bytes(), nil
}
}
errs = append(errs, &renterutil.HostError{HostKey: c.HostKey, Err: err})
}
return nil, errs
}
(A real Skynet portal queries multiple hosts in parallel. Implementing this is left as an exercise for the reader.)
Okay, we now have the raw sector data. We just need to parse it and extract the actual file:
func decodeLinkFile(sector []byte) (md linkfileMetadata, data []byte, err error) {
buf := bytes.NewBuffer(sector)
var lfl linkfileLayout
if err = binary.Read(buf, binary.LittleEndian, &lfl); err != nil {
return
}
if lfl.FanoutSize > 0 {
err = errors.New("can only handle single-sector files")
return
}
if err = json.Unmarshal(buf.Next(int(lfl.MetadataSize)), &md); err != nil {
return
}
data = buf.Next(int(lfl.Filesize))
return
}
And that’s it! We have all the pieces we need to download the file we previously uploaded. Let’s try it:
func main() {
mc := muse.NewClient("<my muse server URL>")
req, err := decodeLink("sia://KABaBLrJzYkQ5GYWvVpbw5zRhUnQL_Onv2JU4Y8-arC8Vw")
if err != nil {
log.Fatal(err)
}
sector, err := downloadSector(mc, req)
if err != nil {
log.Fatal(err)
}
meta, data, err := decodeLinkFile(sector)
if err != nil {
log.Fatal(err)
}
ioutil.WriteFile(meta.Filename, data, meta.Mode)
}
It works! Even though our downloading strategy is extremely naive, it actually
performs quite well if you use the same muse server for both uploading
and downloading. You can view the full source code here.
Now that we have the basic functionality working, we could add fun features, like serving an HTTP API (like Skynet portals do), or adding an LRU cache to reduce latency and cost. At any rate, I hope I’ve demystified Skynet a little. You could have invented it!
Again, if you’re interested in trying out this code, but you don’t have a muse
server, DM me on reddit or Discord and I’ll give you one for free. If you’re
too shy to ask, don’t worry: I plan to release a muse "faucet" soon that will
give you free contracts at the push of a button. Stay tuned!