How to use Sia Irresponsibly

Recently I published us, a low-level interface to Sia. With us, you can interact directly with Sia hosts, allowing you to manage your own file contracts and control exactly what you upload and download. In this post, we’ll demonstrate this by using us to bypass the safety features of siad. Specifically, we’ll upload an unencrypted file to a single host, sacrificing all of the privacy and redundancy advantages of Sia. Of course, in a real app, you should be encrypting your files and uploading them redundantly! This example merely serves as an introduction to the us packages, and shows that us enables you to do things that are not possible with siad.

The following code snippets use these packages:

import (
    "lukechampine.com/us/ed25519"
    "lukechampine.com/us/renter"
    "lukechampine.com/us/renter/proto"
    "lukechampine.com/us/renter/renterutil"
    "lukechampine.com/us/renterhost"

    "lukechampine.com/frand"
    "gitlab.com/NebulousLabs/Sia/crypto"
)

Also, for brevity, all error handling is omitted. In real code, don’t do that!

Part 1: Contract Formation

Before we can do anything interesting, we need a contract. To get one, we need to pick a host, query its current prices, calculate how much money to spend, construct the contract transaction, and perform the contract negotiation protocol with the host. If all goes well, the result will be a transaction signed by both parties, which we'll want to write to disk somewhere for safekeeping.

The first step is to connect to siad and use it to “scan” the host in order to learn its prices. We don’t know the host’s IP, just its public key; fortunately, siad keeps track of all the host announcements in the blockchain, so it will be able to resolve that public key to the host’s most-recently announced IP address. The Scan method hides all these details from us:

const siadPassword = "f7d3aac1fad0e4af706ee3f648b73610" // found in ~/.sia/apipassword
siad := renterutil.NewSiadClient(":9980", siadPassword)
const hostPubKey = "ed25519:4a1df2dc1f0e5a6ad..." // truncated
hostIP, _ := siad.ResolveHostKey(hostPubKey)
host, _ := hostdb.Scan(context.Background(), hostIP, hostPubKey)

Using the retrieved host settings, we can calculate how many siacoins we would need to spend on a given contract. When you form a contract, you lock up some quantity of “funds” for the duration of the contract, which you can use to pay for uploading, downloading, and storage costs. Calculating this quantity thus requires estimating how much data we expect to store and how much we expect to download. In this example, let’s assume that we want to store 1 GB for 1000 blocks, and that we expect to download a total of 2 GB. The total amount of funds to allocate can then be calculated as:

const uploadedBytes = 1e9
const downloadedBytes = 2e9
const duration = 1000
uploadFunds := host.UploadBandwidthPrice.Mul64(uploadedBytes)
downloadFunds := host.DownloadBandwidthPrice.Mul64(downloadedBytes)
storageFunds := host.StoragePrice.Mul64(uploadedBytes).Mul64(duration)
totalFunds := uploadFunds.Add(downloadFunds).Add(storageFunds)

(Note that there is a distinction between the contract funds and the total contract cost. The funds are available for the renter to spend throughout the contract duration, while the cost is the total number of coins spent to form the contract. The latter is equal to the funds plus three fees: the contract formation fee imposed by the host, the 3.9% siafunds tax applied to all file contracts, and the transaction fee that ensures the contract transaction will be included in the blockchain. In other words, don’t be spooked if the transaction that appears in your wallet history spends more than totalFunds.)

Sia’s consensus rules require us to specify the exact block heights for the start and end of the contract, so the last thing we’ll need is the current block height, which we can get from siad. Finally, we can pass everything to the contract formation function:

currentHeight, _ := siad.ChainHeight()
start, end := currentHeight, currentHeight+duration
key := ed25519.NewKeyFromSeed(frand.Bytes(32))
rev, _ := proto.FormContract(siad, siad, key, host, totalFunds, start, end)
contract := renter.Contract{
    HostKey:   rev.HostKey(),
    ID:        rev.ID(),
    RenterKey: key,
}

That’s it! FormContract will connect to the host and execute the contract formation protocol, producing a signed contract. To finish up, we just need to save that contract to disk. us provides an efficient format for storing contracts, which is available in the renter package:

contractPath := "contracts/mycontract.contract"
_ = renter.SaveContract(contract, contractPath)

(In real code, you should use a better name for your contract files. A good approach is to concatenate the host’s public key with the contract ID.)

Part 2: Uploading

Now that we have a contract, we can use it to transfer data. We’ll connect to the host and enter a loop, where we repeatedly upload a sector of file data and then revise the contract to pay the host for the cost of uploading and storing the sector. For each sector we upload, we’ll record the Merkle root of the sector; when we want to download, we’ll request each sector (in order) using these roots. An important implementation detail here is that sectors have a fixed size of 4 MiB. So, unless our file is an exact multiple of 4 MiB, we’ll need to pad the final sector when uploading and trim the padding when downloading.

To begin, we’ll open the contract that we formed previously, as well as the file we want to upload. We’ll also want to record the size of the file so that we can calculate the padding later:

c, _ := renter.LoadContract(contractPath)
f, _ := os.Open("myfile.jpg")
defer f.Close()
stat, _ := f.Stat()
filesize := stat.Size()

To facilitate the upload, we’ll create a Session object. For this, we’ll need the host’s current IP, which we can get from siad:

hostIP, _ := siad.ResolveHostKey(c.HostKey)
session, _ := proto.NewSession(hostIP, c.HostKey, c.ID, c.RenterKey, currentHeight)

Now we’re ready to start uploading. On each iteration, we read the next sector, upload it, and record the sector’s Merkle root, stopping when we reach the end of the file:

var sector [renterhost.SectorSize]byte
var roots []crypto.Hash // that's "Sia/crypto", not "crypto" from the stdlib
for {
    if _, err := io.ReadFull(f, sector[:]); err == io.EOF {
        break
    }
    root, _ := session.Append(&sector)
    roots = append(roots, root)
}
session.Close()

We’re done! We’ve uploaded a plaintext file to a single Sia host. Next, we’ll see how to retrieve it.

Part 3: Downloading

As before, we’ll use the contract and host IP to create a Session that uses the contract revision protocol to facilitate downloads. We’ll also want to create a destination file to hold the downloaded file:

session, _ := proto.NewSession(hostIP, contract.HostKey, contract.ID, contract.RenterKey, currentHeight)
dst, _ := os.Create("myfile2.jpg")
defer dst.Close()

To retrieve the file, we’ll construct a Read request for each sector, writing the responses directly to our output file. To handle padding, we’ll just Truncate the file once we've downloaded all of the sectors. (This is a bit wasteful, but we're not aiming for maximum performance here.)

for _, sectorMerkleRoot := range roots {
    _ = session.Read(dst, []renterhost.RPCReadRequestSection{{
        MerkleRoot: sectorMerkleRoot,
        Offset:     0,
        Length:     renterhost.SectorSize,
    }})
}
session.Close()
_ = dst.Truncate(filesize)

And that’s all there is to it!

Conclusion

It’s worth stating again that this is not how Sia was intended to be used. First, without encryption, hosts can see what you’re uploading; besides the obvious privacy concerns, this could allow them to compress your data and thus reduce their storage costs while charging you full price. Second, without redundancy, your file is at the mercy of a single host. The host could refuse to let you download unless you paid a ridiculous price, or they could simply go offline. Even if they aren’t malicious, their hard drive could fail, rendering your data unrecoverable. If that wasn’t enough to convince you, redundancy can also speed up your downloads, since you can download in parallel and use only the fastest subset of hosts.

In short, there are many reasons to use proper encryption and redundancy when storing data on Sia! In a subsequent post, we’ll look at how us makes it easy to add these important properties to your files.