usToday marks one year since I introduced us, my “alternative
interface to Sia.” I figured I’d mark the occasion with a little retrospective:
what progress was made on us in 2019, and what’s next for 2020?
Let’s start from the beginning. In the two weeks after my initial
announcement, I published three blog posts.
How to use Sia
Irresponsibly came first, demonstrating the power and flexibility of
us by using it to upload files without encryption or redundancy.
In hindsight, I regret writing this post. It seems
to have made a strong first impression, and not a good one – many people (even
some Nebulous employees) came away thinking that us was
not capable of encryption or redundancy at all! What was
intended to be a cute demo ended up backfiring on me quite severely.
My next post, How to use
Sia Responsibly, showed how us allows developers to add custom
encryption and redundancy to their data. This post did not attract as much
interest as the first, and the primary topic of community discussion was
“please change the name.” It’s true, us is a somewhat
awkward name, but no one has suggested a better one. Personally, I like that
us can be used to form larger words (user,
walrus, muse, etc.), just as
us-the-library can be used to build larger pieces of software.
My third post, Files as
Capabilities in us, departed from the previous two by including no code
samples and instead focusing on the project’s design philosophy. It received
even less attention, except for one commenter who insisted that the
us metafile format would not scale to hundreds of thousands of
files. Not really the point of the post, but ok. ¯\_(ツ)_/¯
My intention with this series of posts was to excite and inspire developers by showing them what was possible with the new tools I had developed. Needless to say, reading the comments on these posts left me feeling a bit deflated. It took me a long time to realize what was, in retrospect, obvious: most of my audience is invisible! According to the 90-9-1 rule, for every 100 people who read a post, maybe 10 will upvote (or downvote), and maybe 1 will leave a comment. In other words, I was only seeing the tip of the iceberg.
To my relief, the project's reception among the "silent majority" was more
favorable than I initially thought. I repeatedly observed the community members
I respected the most – the people creating apps and services on top of Sia –
speaking highly of the
us project. And in private chats, nearly every developer I spoke to
told me that they were watching the project closely and were considering
building something with it. It was only a matter of time before one of them took
the plunge, and in February, Salva Herrera (aka @hakkane) became the first to do
so. Salva and I collaborated to develop a continuous host benchmarking tool that
measures the transfer speed of every host on Sia on a regular basis.
us was well-suited to the task: the tool only required ~200 lines
of code and took just a few hours to write. It now serves as the backbone of the
SiaStats Host Monitor.
In May, I added FUSE support to user,
enabling it to mount a virtual Sia filesystem that can be used with
any other program.
user is a CLI program that leverages the us libraries
to upload and download files, much like the siad renter. Although I
had released user alongside us back in January, it had
not seen much interest, so I was hoping to attract more attention to it by
leap-frogging the capabilities of siad. Unfortunately, FUSE didn’t
move the needle much. Despite its attractive features and performance,
user simply does not offer a good experience for the average Sia
fan. It requires you to take an active role in selecting hosts, renewing
contracts, and repairing files – responsibilities that siad handles
automatically. As long as this continues to be the case, user will
be restricted to a fairly small audience of enthusiasts.
The next month, I collaborated with another community member: Michal Šefl,
aka @Danger, who wanted to know if us could be used to build a Sia
mobile app. I had some inkling of how to do it, so down the rabbit hole I went.
When I emerged, I had implemented something pretty neat: cross-language bindings for
us. Michal used these bindings to call us code
from C#, releasing a
demo in which he uploads and downloads a file on an iPhone, connecting
directly to Sia hosts. His blog post was well-received, as many in the community
have long desired a mobile app for Sia. I followed up Michal’s post two weeks
later with one of my own, Talking to Sia Hosts
from…Ruby??, showing off the new cross-language bindings. Reception was
positive here as well, although no one took me up on my offer to help them port
us to additional languages.
In August, there was some internal discussion at Nebulous regarding “garbage
collection.” Simply put, Sia software has never really supported
deleting data that you no longer want to store. Historically, this
hasn’t been a problem, because storage is so cheap. But in the long-term, we
need a way to delete. This seemed like a good test of us, so I took
at stab at it. Within a day I had a working implementation, albeit one that is
unlikely to scale to massive amounts of storage. This is in line with my
previous experiences implementing features like HTTP streaming, FUSE, etc. with
us – I can achieve “minimum viability” extremely quickly, and
iterate from there. This helps me allocate my effort more effectively: If it
turns out that people don’t care about garbage collection, then I won’t have
wasted much time on it; conversely, if people use the feature a lot and start
hitting scaling limits, I can justify spending more time optimizing it.
Later that month, us turned an important corner by winning its
first big “customer:” StoreWise. The
StoreWise developers reached out to me in August to explore building their v2
product architecture on us. This architecture will allow
StoreWise’s users to connect directly to Sia hosts, bringing them the latency
and throughput benefits of a decentralized network while also massively
reducing the company’s bandwidth costs. I’m tremendously pleased that StoreWise
has chosen to bet on us – in fact, I had them in mind as a
potential partner long before I announced the project. Now it’s up to me to
ensure that their bet pays off! If the integration is a success, it
will bring real legitimacy to the us project, and will likely
inspire other developers to pursue integrations of their own.
With the year half over, I switched my focus from storing data to storing
siacoins. In September I released walrus and narwal,
two new wallet projects built from us libraries. walrus features
first-class support for the Ledger Nano S hardware wallet and a
developer-friendly API, while narwal offers a “lite wallet” version
of walrus that lowers the barrier of entry to the Sia ecosystem.
Specifically, narwal makes it possible to form and renew Sia file
contracts without having to run a full node or waiting hours to sync the
blockchain. My accompanying blog post, A Lightweight Wallet for Sia,
attempted to clarify the distinction between walrus and
narwal, with...dubious success. The post was received well on
reddit, and two community members offered to run narwal nodes to
reduce centralization. But real usage did not materialize: I watched the logs on
my narwal
server, and only a handful of people tried it out. I’m hoping that this will
improve with the introduction of new wallet frontends on desktop and mobile,
currently expected for release in 2020.
On a whim – probably inspired by Satoshi’s Treasure – I created a series
of three “treasure hunts” for the Sia community in October, each requiring the
use of various us tools. The clue for the first hunt was simply a
user
“metafile” encoded in base64; inspecting the metafile revealed that the file was
stored on a single host. After forming a contract with the host, anyone could
download the file, thus winning the “treasure” of this
meme. The clue for the second hunt was again a metafile, this time split
into its (rot13’d) index and single (base64) shard. Downloading it (using the
same process as before) yielded a file named “Rick.And.Morty.S04E01.mkv”, which
actually contained this
video. For the third challenge, I upped the stakes a bit, offering 100,000
siacoins instead of silly memes. I also increased the difficulty: this time, the
clue consisted of nothing but an Ed25519 public key. To my surprise (and slight
dismay), this hunt was solved
in just over an hour! I was very impressed by all the participants in these
hunts. Given that us has mostly "flown under the radar," I expected
that substantial hand-holding would be required, but no – the community hardly
had any more trouble solving my challenges than I would have myself! Overall,
the treasure hunts were both fun and gratifying (and perhaps drew more attention
to the project), so I plan to do more of them in 2020.
I capped off the year by creating muse, the first Sia contract
server.
muse integrates with other us tools like
walrus, shard, and user, making it
possible to upload and download files with nothing more than a user
binary and a muse server URL. This is something that I’ve wanted to
do for
ages – I remember discussing it in a Nebulous board meeting years ago,
and since then I’ve brought it up whenever someone asks about apps they could
build with us. Just stick a paywall in front of a muse
server, and you’ve got a viable business that sells contracts for fiat! I teased
some muse functionality back in November, but sadly I wasn't able
to get it to a release-ready state before the end of the year. It will be
released sometime in January, with an accompanying blog post describing the
important role it will play in the us
ecosystem.
My most important takeaway this year was the "90-9-1" rule from above: most
of the people using my software don’t open pull requests, or ask for new
features, or report bugs, or even hit the“Star” button on GitHub. And yet, in the
past two weeks alone, the various us repos have been cloned over
1000 times! Who are all these people? What feedback could they give me? It pains
me to know that some of them must have lost interest due to a bug that I could
have easily fixed – if only they had reported it. Even developers that I
know are using us seem hesitant to open issues or ping me
with questions, as if they’re afraid they’ll bother me. So if you’re reading
this now, and you want to report a bug/request a feature/ask a question,
please open an issue, email me, tag me on Discord,
whatever!
Regardless, the “invisibility” problem means that I have re-evaluate my
approach to releasing code. Previously, I assumed that I could release code as
soon as it was minimally useful, and my users would let me know what to improve.
This might work for a project that receives multiple bug reports and PRs per
week, but it’s not going to fly for us. Instead, I need to put
extra care into polishing my code before release – I need to make a good first
impression! Otherwise, people won't stick around long enough to help me make
things better.
A more personal takeaway for me is that I am much more productive when
working on projects that I have complete ownership of. Over the years,
siad has grown from an amateur two-man effort into a true flagship
product supported by a full complement of engineers. That's great, but I found
that the more sophisticated siad became, the more “burnt out” I
felt when working on it. It was nice to discover that I have not burnt out on
programming in general; hacking on us feels much like the early
days of hacking on siad. Accordingly, I am taking a step back from
siad development and refocusing on other ways that I can contribute
to the vision of Sia – including us – without burning out. So far,
so good. :)
I have a few ideas for 2020, but on the whole it’s rather open-ended. My top
priority will be working with StoreWise (and other “customers,” if they reach
out to me) to ensure that us has the features and performance they
need. After all, if Sia is going to succeed, it will be because of projects like
StoreWise – projects that bring decentralized storage to the masses.
I expect to put a lot of time into polishing muse, as it’s still
quite rough around the edges. I am very eager to see a site that sells contracts
for currencies other than siacoin, so if I can find someone willing to operate
such a site, muse will evolve as necessary to meet their needs. It
occurred to me that siad could implement the muse
API; that way, users could rely on siad to pick hosts and maintain
contracts, two areas where us is currently lacking. I’m also
interested in collaborating with a community member to develop a graphical
frontend for
muse, as the CLI frontend is not very user-friendly.
On that note, more work is needed on user to make it appeal to a
wider audience. Specifically, I need to quit waffling and add some form of
automatic migration (i.e. file repair). Historically, I’ve been uneasy about
doing this; I don’t like making decisions on the user's behalf, but maybe
it’s acceptable as long as its opt-in. On the other hand, I expect that
user will eventually be made obsolete by more professional apps, so
I’m not certain how much energy I ought to devote to it.
As for walrus, I hope to see one or two graphical clients
released in 2020. A mobile app has been in active development for the past few
months, which I’ve contributed to in the form of additional C# bindings. I have
also explored writing a desktop UI for walrus with first-class
support for hardware wallets. I don’t have the design sensibilities (or
JavaScript chops) to make this a reality by myself, but with a committed
frontend developer, I think we could put something together very quickly.
Lastly, I’m interested in devising some more treasure hunts. The hunts I organized in October went really well, and I’m sure I can come up with new ones. In particular, I’d like to design a treasure hunt that is more collaborative, so that participants are incentivized to share their knowledge rather than hoarding it for themselves to gain an advantage.
us improved by leaps and bounds in 2019. It did not gain mass
adoption, but that wasn’t my goal; I’m building us for Sia app
developers, and that’s a pretty small crowd. I don’t aspire to have millions of
users, just a handful who can appreciate what I’m doing. If I can build
something that they really like, something that empowers and inspires them, then
I’ll have succeeded.
