Skip to content

Commit e39fd2a

Browse files
robherleyrobherley
committed
09: snips retro
1 parent 359acc2 commit e39fd2a

File tree

15 files changed

+359
-3
lines changed

15 files changed

+359
-3
lines changed

‎assets/css/custom.css‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
img {
2+
border-radius: var(--border-radius);
3+
}
4+
5+
.twitter-tweet > iframe {
6+
border-radius: 13px;
7+
}
8+
9+
.counter {
10+
flex: 1;
11+
font-family: var(--font-mono);
12+
}
13+
14+
.counter .counter-count {
15+
font-size: 3rem;
16+
}
17+
18+
.counter .counter-label {
19+
font-size: 1.2rem;
20+
font-family: var(--font-mono);
21+
color: var(--color-gray-6);
22+
}

‎assets/js/main.js‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,38 @@
1-
console.log("👋");
1+
const observer = new IntersectionObserver((entries, observer) => {
2+
entries.forEach(entry => {
3+
if (entry.isIntersecting) {
4+
startCounter(entry.target);
5+
observer.unobserve(entry.target);
6+
}
7+
});
8+
}, {
9+
threshold: 0.5
10+
});
11+
12+
const startCounter = (element) => {
13+
const target = +element.getAttribute('data-count');
14+
let count = 0;
15+
let speed = target / 300;
16+
17+
const countElement = element.querySelector('.counter-count');
18+
19+
const update = () => {
20+
const jitter = Math.floor(Math.random() * 2);
21+
count += speed + jitter;
22+
countElement.textContent = Math.ceil(count);
23+
24+
if (count < target) {
25+
requestAnimationFrame(update);
26+
} else {
27+
countElement.textContent = target;
28+
}
29+
}
30+
31+
update();
32+
}
33+
34+
window.addEventListener("DOMContentLoaded", () => {
35+
document.querySelectorAll('.counter').forEach(counter => {
36+
observer.observe(counter);
37+
});
38+
});

‎config.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,9 @@ weight = 2
3838
[markup.highlight]
3939
style = 'catppuccin-frappe'
4040
tabWidth = 4
41+
[markup.goldmark]
42+
[markup.goldmark.parser]
43+
wrapStandAloneImageWithinParagraph = false
4144

4245
[outputs]
4346
home = ["HTML", "JSON"]

‎content/posts/09-snips-retro.md‎

Lines changed: 256 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,256 @@
1+
---
2+
title: "✂️ snips.sh retrospective: 1000+ stars later"
3+
date: 2024-10-04T13:50:16-04:00
4+
draft: false
5+
tags:
6+
- golang
7+
- terminal
8+
- ssh
9+
- sqlite
10+
- tensorflow
11+
---
12+
13+
{{< tweet "1657139515557920770" >}}
14+
15+
## What the snip?
16+
17+
A tad bit over a year ago, I released https://snips.sh, a passwordless, anonymous SSH-powered pastebin with a human-friendly TUI and web UI. No logins, no passwords, nothing to install. It's ready-to-go on any machine that has SSH installed.
18+
19+
It's a simple as:
20+
21+
```bash
22+
echo 'this is amazing!' | ssh snips.sh
23+
```
24+
25+
I wanted an easy utility to copy code snippets to/from machines, a dead-simple web UI to link to line numbers and something to just dump code snippets.
26+
27+
And the development community loved it. To my surprise, it rapidly gained popularity across social media. It even made the top of GitHub's [`/trending`](https://github.com/trending) under the Go language category for a couple days.
28+
29+
![stargazers](https://api.star-history.com/svg?repos=robherley/snips.sh&type=Date&theme=dark&size=mobile "Surpassed 1k stars just over a year!")
30+
31+
Given I procrastinated for over a year making an original "release" blog post for snips, I figured a retrospective would be just as good.
32+
33+
## The philosophy
34+
35+
When designing snips, I wanted it to be as _simple_ as possible. If I learned anything from maintaining open source libraries and supporting public APIs used by millions of people, it's important to not bloat with verbose functionality that becomes a maintenance/compatibility nightmare.
36+
37+
I manifested my inner Ken and Dennis and kept the [Unix Philosophy](https://en.wikipedia.org/wiki/Unix_philosophy) top of mind:
38+
39+
> Write programs that do one thing and do it well.
40+
> Write programs to work together.
41+
> Write programs to handle text streams, because that is a universal interface.
42+
43+
And that's exactly what snips is.
44+
45+
1. It's a snippet store (with a UI and TUI), nothing more.
46+
2. It works with other command line programs via pipes.
47+
3. The "API" is just text over stdin/stdout.
48+
49+
![pipe examples](https://vhs.charm.sh/vhs-7j0LzNCGaBjF6v91QkXJgr.gif "Pipe into whatever you want")
50+
51+
But, this isn't just a command line utility. While I love the Unix Philosophy, it is not my creed. Just as much as I believe simplicity is key in software development, the user experience is just as important. This is often a very hard balance.
52+
53+
Under the covers, snips.sh is a stateful remote resource that requires functionality beyond the simple input/output. And that's what the TUI is for. It's a shell into the user's snips. You can use the TUI to view snips syntax highlighted, edit attributes and delete them.
54+
55+
![tui](https://vhs.charm.sh/vhs-1MRS4DCN6XUpxzM2PrqCfL.gif "A user can `ssh` into the TUI to view/manage snips")
56+
57+
As a developer building tools for developers, I know how comfortable most are in the terminal, which is why I chose that as the entrypoint over a web UI. They don't even need to lift their fingers off the keyboard.
58+
59+
I also wanted the onboarding experience to be as smooth as possible. Here's how the upload works for a new user:
60+
61+
{{< mermaid >}}
62+
%%{init: {'theme': 'dark' } }%%
63+
flowchart TD
64+
ssh([ssh session])
65+
fail((fail))
66+
onboard((onboard))
67+
a@{ shape: diamond, label: "auth" }
68+
pk@{ shape: diamond, label: "exists?" }
69+
wf((write snip))
70+
71+
72+
ssh --> a
73+
a --"password"--> fail
74+
a --"pubkey"-->pk
75+
76+
pk --"no"--> onboard
77+
pk --"yes"--> wf
78+
onboard --> wf
79+
80+
{{< /mermaid >}}
81+
82+
The usage of snips.sh does require public key auth in order to identify users. If a connection attempt is made with a password, it fails and sends a message to stdout. This also helps prevent against bots and other things that like to poke at port 22.
83+
84+
For any new users, if their public key doesn't exist in the database, we'll "onboard" which will create a new user record and associate a public key with it. A terms of service message is also printed for new users. Since we're able to create a new identity and automatically onboard, there is literally zero friction to get started.
85+
86+
While it's nice to keep things as simple as can be, like anything with software engineering "it depends" on your use case. Personally, I'm a fan of making easy (and fun!) to use software, which might need a few engineering tradeoffs for a better user experience.
87+
88+
As it turns out, people like easy to use software:
89+
90+
{{< tweet "1657456167524917248" >}}
91+
92+
{{< tweet "1657477244158189570" >}}
93+
94+
## The technology
95+
96+
All of snips.sh is written in Go, from the SSH app to the web UI. While Go may not be as creative or fast as other languages, I do find beauty in the simplicity.
97+
98+
As luck might have it, there's also an organization called [Charm](https://charm.sh/) that builds amazing libraries for the command line, all built in Go. So surprise, snips.sh uses [plenty of Charm libraries](https://github.com/robherley/snips.sh/blob/b6c00d501f44ccddbbd323fb4cbaded1124aef5b/go.mod#L9-L13). They're a great group of people, you should totally check them out.
99+
100+
There's no real fancy frameworks in snips.sh. All the web-based routes use the standard library's [`net/http`](https://pkg.go.dev/net/http) server along with the [`html/template`](https://pkg.go.dev/html/template) package for server side rendering. There is about ~120 lines of JavaScript and some old fashioned hand-rolled CSS to keep things as tiny as can be.
101+
102+
As for the backing storage, I went with the most deployed database in the world, SQLite. Why SQLite you ask?
103+
1. It's _really_ fast.[^1]
104+
2. It's stupid simple to use. It's embedded and doesn't need extra resources/configuration.
105+
3. The database is all stored in a single file, making it easy to manage (and [backup](https://litestream.io/)).
106+
4. Some people much smarter than me have been scaling it like crazy.[^2]
107+
108+
While some S3-compatible storage might have been first choice for some, I considered it overkill. Having to run another program or worse (connect to a cloud provider!) I figured the 1MB file size limit would be absolutely fine in a blob column, especially since it's compressed with [zstd](https://facebook.github.io/zstd/) too.
109+
110+
For a lot of the fancy web UI rendering, I have to give credit to some amazing open source libraries:
111+
- [`alecthomas/chroma`](https://github.com/alecthomas/chroma): Syntax Highlighter
112+
- [`yuin/goldmark`](https://github.com/yuin/goldmark): Markdown Parser
113+
- [`microcosm-cc/bluemonday`](https://github.com/microcosm-cc/bluemonday): HTML Sanitizer
114+
- [`tdewolff/minify`](https://github.com/tdewolff/minify): Asset Minifier
115+
116+
And that's pretty much it! Keeping the technology simple means it's just as easy for someone else to run snips.sh on their own hardware. And that's exactly why we have a [self hosting guide](https://snips.sh/docs/self-hosting.md) and publish an multi-arch container image to GitHub Container Registry:
117+
118+
```
119+
ghcr.io/robherley/snips.sh
120+
```
121+
122+
Given the simple tech stack, it's pretty easy to get going after a couple volume mounts and environment variables.
123+
124+
## The tensorflow-sized elephant in the room
125+
126+
So, one _must have_ that I wanted for snips is to automatically detect the uploaded code language. To do this, I used a tensorflow model, [`yoeo/guesslang`](https://github.com/yoeo/guesslang). This is actually the same model that [Visual Studio Code](https://code.visualstudio.com/) uses, but they use [Tensorflow.js](https://www.tensorflow.org/js/), you can check it out at [`Microsoft/vscode-languagedetection`](https://github.com/Microsoft/vscode-languagedetection).
127+
128+
But we do not have server side JS here, we're in a compiled language. This was my first hurdle, and I ended up writing [`robherley/guesslang-go`](https://github.com/robherley/guesslang-go) which uses some wrappers around libtensorflow's C API.
129+
130+
Unfortunately, this means we lose the ability to make a static executable and need to sacrifice portability:
131+
132+
{{< terminal >}}
133+
you@local$ docker run -it --entrypoint=ldd ghcr.io/robherley/snips.sh /usr/bin/snips.sh
134+
linux-vdso.so.1 (0x00007ffd219a8000)
135+
libtensorflow.so.2 => /usr/local/lib/libtensorflow.so.2 (0x00007f2922485000)
136+
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f292225c000)
137+
libtensorflow_framework.so.2 => /usr/local/lib/libtensorflow_framework.so.2 (0x00007f292036d000)
138+
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f2920368000)
139+
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2920363000)
140+
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f292027a000)
141+
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2920275000)
142+
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f2920049000)
143+
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f2920029000)
144+
/lib64/ld-linux-x86-64.so.2 (0x00007f2933a32000)
145+
{{< /terminal >}}
146+
147+
Even worse, look how big this is!
148+
149+
{{< terminal >}}
150+
you@local$ docker run -it --entrypoint=ls ghcr.io/robherley/snips.sh -lah /usr/local/lib
151+
total 416M
152+
drwxr-xr-x 1 root root 4.0K Sep 13 15:53 .
153+
drwxr-xr-x 1 root root 4.0K Aug 8 14:03 ..
154+
-r-xr-xr-x 1 root root 375M Sep 13 15:51 libtensorflow.so.2
155+
-r-xr-xr-x 1 root root 42M Sep 13 15:51 libtensorflow_framework.so.2
156+
{{< /terminal >}}
157+
158+
Yikes. Not ideal.
159+
160+
I did search for alternatives, like relying on chroma's [built in lexers to identify the language](https://github.com/alecthomas/chroma#identifying-the-language) but it was not good enough for small snippets. Other language detection features of editors and other tools like GitHub's [linguist](https://github.com/github-linguist/linguist) rely on file extensions, which we don't have.
161+
162+
This is a prime example of making sacrifices for an extremely useful feature. It does put a smile on my face when I see the correct language detected on upload.
163+
164+
Another huge gotcha with libtensorflow is the lack of support for many architectures. Luckily this can be solved with some compiler flags (`-tags noguesser`) and a multiarch container image, but some users lose that critical functionality.
165+
166+
This is an area that I am not very strong in. I'd love any suggestions on this topic, feel free to [open an issue](https://github.com/robherley/snips.sh/issues).
167+
168+
## The ship
169+
170+
I started on this side project at the beginning of 2023[^3], and "released" it via social media in May of that same year. This was my first real side-hack that turned into a pretty useful tool, and I felt a warm welcome from the developer community.
171+
172+
My largest audience was Twitter, having over 120k views on [my tweet](https://x.com/robherley/status/1657139515557920770). Some retweets from folks like [@mxcl](https://twitter.com/mxcl) (creator of Homebrew) and [@charmcli](https://twitter.com/charmcli) really helped get it to the right audience.
173+
174+
Surprisingly, even folks on reddit took it pretty well!
175+
176+
{{< reddit "https://www.reddit.com/r/golang/comments/13fyp1k/snipssh_passwordless_anonymous_sshpowered_pastebin/" "snips.sh: passwordless, anonymous SSH-powered pastebin" >}}
177+
178+
I was delighted to see snips.sh in all difference communities:
179+
- [console.dev](https://console.dev) made a review: https://console.dev/tools/snips
180+
- [@JeremiahSecrist](https://github.com/JeremiahSecrist) published a nixpkg: https://mynixos.com/nixpkgs/package/snips-sh
181+
- [@Sanix-Darker](https://github.com/Sanix-Darker) made a nvim extension: https://github.com/Sanix-Darker/snips.nvim
182+
183+
Shortly after release, we already had issues and contributions coming in too!
184+
185+
It truly was a great ship! 🚢
186+
187+
## The numbers
188+
189+
### Connections
190+
191+
Users can reach snips.sh via HTTP or SSH. The request metrics are emitted to DataDog but unfortunately I only have up to a little over a year's worth of retention, so here's since July 2023:
192+
193+
{{< row >}}
194+
195+
{{< count n="130844" label="HTTP Requests" color="green-6" >}}
196+
197+
{{< count n="116889" label="SSH Sessions" color="amber-6" >}}
198+
199+
{{</ row >}}
200+
201+
Note: the above SSH sessions are for successfully authenticated users. If we include all non-authenticated (anything hitting port 22), snips.sh has seen **2.148 million** unique SSH sessions.
202+
203+
### App
204+
205+
This entire app is hosted on a [Digital Ocean](https://digitalocean.com) droplet to keep costs low. The database is still relatively small around **~24MB**, which is not including backups on [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) via [litestream](https://litestream.io).
206+
207+
{{< row >}}
208+
209+
{{< count n="1534" label="Users" color="purple-6" >}}
210+
211+
{{< count n="2486" label="Files" color="blue-9" >}}
212+
213+
{{< count n="57" label="Langs" color="teal-9" >}}
214+
215+
{{< /row >}}
216+
217+
After copying the sqlite database to my host machine and running aggregations, we can see a nice time series of usage:
218+
219+
![users created over time](/content/snips-retro/users-created.png "")
220+
221+
![files created over time](/content/snips-retro/files-created.png "")
222+
223+
Unsurprisingly, we had a huge burst of users during the "Twitter hype period" and gradually slowed down. While I would have loved to market this more, my goal wasn't to make a disrupting product, just a fun developer tool. Plus, over the lifecycle of this release, I was busy planning an engagement and then my wedding!
224+
225+
Back to the metrics, we had the usual suspects of popular files.
226+
227+
![files by type](/content/snips-retro/files-by-lang.png "Files by programming language")
228+
229+
You can find the full language list... [on snips](https://snips.sh/f/yfojYVMqSU)!
230+
231+
### Open Source
232+
233+
We've had some great contributions like [zstd compression](https://github.com/robherley/snips.sh/pull/46), [arm64 support](https://github.com/robherley/snips.sh/pull/42), bugfixes and more. Dependabot is also carrying the weight a bit with **~65** Pull Requests alone.
234+
235+
{{< row >}}
236+
237+
{{< count n="148" label="Commits" color="green-9" >}}
238+
239+
{{< count n="175" label="Pull Requests" color="purple-6" >}}
240+
241+
{{< count n="26" label="Issues" color="teal-9" >}}
242+
243+
{{< count n="1028" label="Stars" color="amber-9" >}}
244+
245+
{{< count n="13" label="Contributors" color="pink-9" >}}
246+
247+
{{</ row >}}
248+
249+
## Thanks!
250+
251+
Appreciate all the contributions and kind words people have given me throughout this project! It gives me the motivation to keep on building, and you should too 💪
252+
253+
[^1]: https://sqlite.org/fasterthanfs.html
254+
[^2]: https://fly.io/docs/litefs/
255+
[^3]: https://github.com/robherley/snips.sh/commit/4982dafd6204d56c7670aa2ef258638e318447f4
256+
[^4]: https://github.com/robherley/snips.sh/pkgs/container/snips.sh

‎layouts/shortcodes/count.html‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{{ $count := .Get "n" }}
2+
{{ $label := .Get "label" }}
3+
{{ $color := .Get "color" | default "primary" }}
4+
{{ $colorstyle := printf "color: var(--color-%s);" $color }}
5+
<div class="counter" data-count="{{ $count }}" style="{{ $colorstyle | safeCSS }}">
6+
<div class="counter-count">
7+
0
8+
</div>
9+
<div class="counter-label">{{ $label | markdownify }}</div>
10+
</div>

‎layouts/shortcodes/reddit.html‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
{{ $url := .Get 0 }}
2+
{{ $title := .Get 1 }}
3+
<blockquote
4+
class="reddit-embed-bq"
5+
style="height: 316px"
6+
data-embed-theme="dark"
7+
data-embed-height="316"
8+
>
9+
<a href="{{ $url }}">
10+
{{ $title }}
11+
</a>
12+
</blockquote>
13+
<script
14+
async=""
15+
src="https://embed.reddit.com/widgets.js"
16+
charset="UTF-8"
17+
></script>

‎layouts/shortcodes/row.html‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
<div style="display:flex;flex-wrap:wrap;gap:1rem;">
2+
{{ .Inner }}
3+
</div>

‎layouts/shortcodes/tweet.html‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{{ $id := .Get 0 }}
2+
{{ $data := dict }}
3+
{{ $url := printf "https://api.twitter.com/1.1/statuses/oembed.json?dnt=1&hide_media=1&theme=dark&align=center&hide_thread=1&id=%s" $id }}
4+
{{ with resources.GetRemote $url }}
5+
{{ $data = . | transform.Unmarshal }}
6+
{{ $data.html | safeHTML }}
7+
{{ end }}
8+

‎script/server‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,4 @@ if ! [ -x "$(command -v hugo)" ]; then
66
exit 1
77
fi
88

9-
hugo server -D
9+
hugo server --ignoreCache

‎static/content/snips-retro/files-by-lang.png‎

76.7 KB
Loading

0 commit comments

Comments
 (0)