Tomi Chen's BlogThoughts, projects, and more!Sometimes I build things, and sometimes I'll write about what I built. This blog aims to explain my through process behind decisions I make, as well as other things I find interesting.2022-06-03T02:52:09Zhttps://tomichen.com/blog/Tomi Chenhttps://tomichen.comtomichen33@gmail.comhttps://tomichen.com/assets/logo.pngResponsive Images in Markdown with Eleventy Image2022-04-16T00:00:00Z2022-04-18T21:47:31Zhttps://tomichen.com/blog/posts/20220416-responsive-images-in-markdown-with-eleventy-image/I love markdown's simplicity and convenience, but sometimes you need a little more than the default image tag. In this post, I walk through how responsive images (with the picture tag) are handled on this blog in a way that doesn't compromise the authoring experience.<p>I use markdown on this very blog to author content, and I love its simplicity and convenience. By default, images in markdown render to a standard <code><img></code> tag, which is sometimes all you need. However, for this blog, I wanted to generate responsive images, which serve different-sized images depending on screen size. Can we the authoring experience of markdown with the performance benefits of responsive images? Let’s find out!</p> <p><a href="https://tomichen.com/blog/posts/20220416-responsive-images-in-markdown-with-eleventy-image/#final-code">TL;DR: Skip to the final code</a>.</p> <h2 id="what-are-responsive-images%3F" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220416-responsive-images-in-markdown-with-eleventy-image/#what-are-responsive-images%3F"><span>What are responsive images?</span></a></h2> <p>In essence, responsive images are a way to let browsers pick the best image for the situation, depending on different conditions. MDN has a great <a href="https://developer.mozilla.org/en-US/docs/Learn/HTML/Multimedia_and_embedding/Responsive_images">article on responsive images</a>, which goes into a lot more detail, but to sum it up, we can use the <code><picture></code> tag to list different image formats and sizes that will respond to the size of the screen.</p> <pre><code class="language-html"><span class="hljs-tag"><<span class="hljs-name">picture</span>></span> <span class="hljs-tag"><<span class="hljs-name">source</span> <span class="hljs-attr">type</span>=<span class="hljs-string">"image/avif"</span> <span class="hljs-attr">srcset</span>=<span class="hljs-string">" /images/cat-340.avif 340w, /images/cat-768.avif 768w, /images/cat-1536.avif 1536w "</span> /></span> <span class="hljs-tag"><<span class="hljs-name">source</span> <span class="hljs-attr">type</span>=<span class="hljs-string">"image/webp"</span> <span class="hljs-attr">srcset</span>=<span class="hljs-string">" /images/cat-340.webp 340w, /images/cat-768.webp 768w, /images/cat-1536.webp 1536w "</span> /></span> <span class="hljs-tag"><<span class="hljs-name">img</span> <span class="hljs-attr">alt</span>=<span class="hljs-string">"a very cute cat"</span> <span class="hljs-attr">src</span>=<span class="hljs-string">"/images/cat.png"</span> /></span> <span class="hljs-tag"></<span class="hljs-name">picture</span>></span> </code></pre> <p>The <code><picture></code> tag is very powerful, with many different ways of using it. It can enable art direction (where the image content changes based on width, for example, to crop into a subject), new image formats (like AVIF or WebP), and resolution switching (where the same image content is served, just a different size).</p> <p>In our case, we’ll be adding new image formats and enabling resolution switching.</p> <h2 id="configuring-a-custom-markdown-it-renderer" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220416-responsive-images-in-markdown-with-eleventy-image/#configuring-a-custom-markdown-it-renderer"><span>Configuring a custom <code>markdown-it</code> renderer</span></a></h2> <p>Since I use <a href="https://www.11ty.dev/">Eleventy</a> as the static site generator for this website, the markdown renderer is <a href="https://github.com/markdown-it/markdown-it">markdown-it</a>, and that’s what this post will use. If you’re using another renderer such as <a href="https://github.com/remarkjs/remark">Remark</a>, some concepts might still apply and can be adapted to your needs.</p> <p>Anyway, we need to <a href="https://github.com/markdown-it/markdown-it/blob/master/docs/architecture.md#renderer">replace the default image renderer</a> with a custom one that will output our <code><picture></code> tag. Luckily, this isn’t too difficult.</p> <pre><code class="language-javascript"><span class="hljs-keyword">const</span> markdown = <span class="hljs-built_in">require</span>(<span class="hljs-string">'markdown-it'</span>)({ <span class="hljs-comment">/* options */</span> }) markdown.<span class="hljs-property">renderer</span>.<span class="hljs-property">rules</span>.<span class="hljs-property">image</span> = <span class="hljs-keyword">function</span> (<span class="hljs-params">tokens, idx, options, env, self</span>) { <span class="hljs-comment">// ...</span> <span class="hljs-keyword">return</span> html } </code></pre> <p>We can now access the markdown content and attributes, too. The <a href="https://github.com/markdown-it/markdown-it/blob/master/docs/architecture.md#renderer">architecture docs</a> can be helpful, as well as the <a href="https://markdown-it.github.io/markdown-it/">API docs</a>.</p> <pre><code class="language-javascript"><span class="hljs-keyword">const</span> markdown = <span class="hljs-built_in">require</span>(<span class="hljs-string">'markdown-it'</span>)() markdown.<span class="hljs-property">renderer</span>.<span class="hljs-property">rules</span>.<span class="hljs-property">image</span> = <span class="hljs-keyword">function</span> (<span class="hljs-params">tokens, idx, options, env, self</span>) { <span class="hljs-keyword">const</span> token = tokens[idx] <span class="hljs-keyword">let</span> imgSrc = token.<span class="hljs-title function_">attrGet</span>(<span class="hljs-string">'src'</span>) <span class="hljs-keyword">const</span> imgAlt = token.<span class="hljs-property">content</span> <span class="hljs-keyword">const</span> imgTitle = token.<span class="hljs-title function_">attrGet</span>(<span class="hljs-string">'title'</span>) <span class="hljs-comment">// ...</span> <span class="hljs-keyword">return</span> html } </code></pre> <h2 id="generating-image-sizes-and-formats" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220416-responsive-images-in-markdown-with-eleventy-image/#generating-image-sizes-and-formats"><span>Generating image sizes and formats</span></a></h2> <p>Now we need to generate all the images we’re using. I’m using <a href="https://www.11ty.dev/docs/plugins/image/"><code>eleventy-image</code></a>, but despite the name, it doesn’t need to be used with Eleventy.</p> <p>The main problem with using <code>eleventy-image</code> is that image generation is asynchronous, but <a href="https://github.com/markdown-it/markdown-it/blob/master/docs/development.md#faq">markdown-it doesn’t support asynchronous renderers</a>. Luckily, since we don’t care if the images have finished generating, we can use <a href="https://www.11ty.dev/docs/plugins/image/#synchronous-usage"><code>eleventy-image</code>’s synchronous usage</a>.</p> <p><code>eleventy-image</code> also provides a handy <code>generateHTML</code> function that builds our <code><picture></code> tag for us.</p> <pre><code class="language-javascript"><span class="hljs-keyword">const</span> markdown = <span class="hljs-built_in">require</span>(<span class="hljs-string">'markdown-it'</span>)() <span class="hljs-keyword">const</span> <span class="hljs-title class_">Image</span> = <span class="hljs-built_in">require</span>(<span class="hljs-string">'@11ty/eleventy-img'</span>) markdown.<span class="hljs-property">renderer</span>.<span class="hljs-property">rules</span>.<span class="hljs-property">image</span> = <span class="hljs-keyword">function</span> (<span class="hljs-params">tokens, idx, options, env, self</span>) { <span class="hljs-keyword">const</span> token = tokens[idx] <span class="hljs-keyword">let</span> imgSrc = token.<span class="hljs-title function_">attrGet</span>(<span class="hljs-string">'src'</span>) <span class="hljs-keyword">const</span> imgAlt = token.<span class="hljs-property">content</span> <span class="hljs-keyword">const</span> imgTitle = token.<span class="hljs-title function_">attrGet</span>(<span class="hljs-string">'title'</span>) <span class="hljs-keyword">const</span> htmlOpts = { <span class="hljs-attr">title</span>: imgTitle, <span class="hljs-attr">alt</span>: imgAlt, <span class="hljs-attr">loading</span>: <span class="hljs-string">'lazy'</span>, <span class="hljs-attr">decoding</span>: <span class="hljs-string">'async'</span> } <span class="hljs-keyword">const</span> widths = [<span class="hljs-number">250</span>, <span class="hljs-number">316</span>, <span class="hljs-number">426</span>, <span class="hljs-number">460</span>, <span class="hljs-number">580</span>, <span class="hljs-number">768</span>] <span class="hljs-keyword">const</span> imgOpts = { <span class="hljs-attr">widths</span>: widths .<span class="hljs-title function_">concat</span>(widths.<span class="hljs-title function_">map</span>(<span class="hljs-function">(<span class="hljs-params">w</span>) =></span> w * <span class="hljs-number">2</span>)) <span class="hljs-comment">// generate 2x sizes</span> .<span class="hljs-title function_">filter</span>(<span class="hljs-function">(<span class="hljs-params">v, i, s</span>) =></span> s.<span class="hljs-title function_">indexOf</span>(v) === i), <span class="hljs-comment">// dedupe</span> <span class="hljs-attr">formats</span>: [<span class="hljs-string">'avif'</span>, <span class="hljs-string">'webp'</span>, <span class="hljs-string">'jpeg'</span>], <span class="hljs-attr">urlPath</span>: <span class="hljs-string">'/assets/img/'</span>, <span class="hljs-attr">outputDir</span>: <span class="hljs-string">'./_site/assets/img/'</span> } <span class="hljs-title class_">Image</span>(imgSrc, imgOpts) <span class="hljs-keyword">const</span> metadata = <span class="hljs-title class_">Image</span>.<span class="hljs-title function_">statsSync</span>(imgSrc, imgOpts) <span class="hljs-keyword">const</span> generated = <span class="hljs-title class_">Image</span>.<span class="hljs-title function_">generateHTML</span>(metadata, { <span class="hljs-attr">sizes</span>: <span class="hljs-string">'(max-width: 768px) 100vw, 768px'</span>, ...htmlOpts }) <span class="hljs-keyword">return</span> generated } </code></pre> <p>There are a couple of things you can configure here:</p> <ul> <li><code>widths</code>: the widths in pixels of the images you want to generate. You’ll want a balance between enough to have beneficial size reductions, but not too many that it’ll take forever to generate. Honestly, I probably have too much. I’m also multiplying each size by 2 to get higher resolution images for high-dpi displays.</li> <li><code>formats</code>: the output image formats. JPEG is widely supported, but WebP and AVIF offer better file sizes. <a href="https://caniuse.com/avif">AVIF has very little browser support</a>, so you might not want to generate those to save time.</li> <li><code>sizes</code>: the <code>sizes</code> attribute on the <code><img></code> tag (inside <code><picture></code>). This hints to the browser what size the image will be. It doesn’t need to be precise since it (most likely) won’t affect the actual image size, but it should ideally be pretty close. On my site, the max-width is 768px but spans the full width on smaller screens.</li> </ul> <p>We’re also setting <code>loading=lazy</code> and <code>decoding=async</code>. Learn more about <a href="https://developer.mozilla.org/en-US/docs/Web/Performance/Lazy_loading#images_and_iframes">lazy loading</a> and the <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/img#attr-decoding">decoding attribute</a>.</p> <h2 id="adding-a-caption" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220416-responsive-images-in-markdown-with-eleventy-image/#adding-a-caption"><span>Adding a caption</span></a></h2> <p>Sometimes, you may want a caption to go with your image. The best way to represent this semantically is with <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/figcaption">a <code><figcaption></code> element</a>. We can hijack the title string to use as a caption instead.</p> <pre><code class="language-javascript"><span class="hljs-keyword">const</span> markdown = <span class="hljs-built_in">require</span>(<span class="hljs-string">'markdown-it'</span>)() <span class="hljs-keyword">const</span> <span class="hljs-title class_">Image</span> = <span class="hljs-built_in">require</span>(<span class="hljs-string">'@11ty/eleventy-img'</span>) markdown.<span class="hljs-property">renderer</span>.<span class="hljs-property">rules</span>.<span class="hljs-property">image</span> = <span class="hljs-keyword">function</span> (<span class="hljs-params">tokens, idx, options, env, self</span>) { <span class="hljs-keyword">function</span> <span class="hljs-title function_">figure</span>(<span class="hljs-params">html, caption</span>) { <span class="hljs-keyword">return</span> <span class="hljs-string">`<figure><span class="hljs-subst">${html}</span><figcaption><span class="hljs-subst">${caption}</span></figcaption></figure>`</span> } <span class="hljs-keyword">const</span> token = tokens[idx] <span class="hljs-keyword">let</span> imgSrc = token.<span class="hljs-title function_">attrGet</span>(<span class="hljs-string">'src'</span>) <span class="hljs-keyword">const</span> imgAlt = token.<span class="hljs-property">content</span> <span class="hljs-keyword">const</span> imgTitle = token.<span class="hljs-title function_">attrGet</span>(<span class="hljs-string">'title'</span>) <span class="hljs-keyword">const</span> htmlOpts = { <span class="hljs-attr">alt</span>: imgAlt, <span class="hljs-attr">loading</span>: <span class="hljs-string">'lazy'</span>, <span class="hljs-attr">decoding</span>: <span class="hljs-string">'async'</span> } <span class="hljs-keyword">const</span> widths = [<span class="hljs-number">250</span>, <span class="hljs-number">316</span>, <span class="hljs-number">426</span>, <span class="hljs-number">460</span>, <span class="hljs-number">580</span>, <span class="hljs-number">768</span>] <span class="hljs-keyword">const</span> imgOpts = { <span class="hljs-attr">widths</span>: widths .<span class="hljs-title function_">concat</span>(widths.<span class="hljs-title function_">map</span>(<span class="hljs-function">(<span class="hljs-params">w</span>) =></span> w * <span class="hljs-number">2</span>)) <span class="hljs-comment">// generate 2x sizes</span> .<span class="hljs-title function_">filter</span>(<span class="hljs-function">(<span class="hljs-params">v, i, s</span>) =></span> s.<span class="hljs-title function_">indexOf</span>(v) === i), <span class="hljs-comment">// dedupe</span> <span class="hljs-attr">formats</span>: [<span class="hljs-string">'avif'</span>, <span class="hljs-string">'webp'</span>, <span class="hljs-string">'jpeg'</span>], <span class="hljs-attr">urlPath</span>: <span class="hljs-string">'/assets/img/'</span>, <span class="hljs-attr">outputDir</span>: <span class="hljs-string">'./_site/assets/img/'</span> } <span class="hljs-title class_">Image</span>(imgSrc, imgOpts) <span class="hljs-keyword">const</span> metadata = <span class="hljs-title class_">Image</span>.<span class="hljs-title function_">statsSync</span>(imgSrc, imgOpts) <span class="hljs-keyword">const</span> generated = <span class="hljs-title class_">Image</span>.<span class="hljs-title function_">generateHTML</span>(metadata, { <span class="hljs-attr">sizes</span>: <span class="hljs-string">'(max-width: 768px) 100vw, 768px'</span>, ...htmlOpts }) <span class="hljs-keyword">if</span> (imgTitle) { <span class="hljs-keyword">return</span> <span class="hljs-title function_">figure</span>(generatd, imgTitle) } <span class="hljs-keyword">return</span> generated } </code></pre> <h2 id="minor-tweaks" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220416-responsive-images-in-markdown-with-eleventy-image/#minor-tweaks"><span>Minor tweaks</span></a></h2> <p>Finally, we can add some minor tweaks to improve the experience.</p> <h3 id="fixing-image-paths" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220416-responsive-images-in-markdown-with-eleventy-image/#fixing-image-paths"><span>Fixing image paths</span></a></h3> <p>When writing markdown, you may want to use the image source relative to the final URL, which might not be where the image exists on the filesystem. This makes sense when using the default renderer since the path is interpreted by the browser, but won’t work at build time, which is when the images are generated. We can fix this by patching the <code>imgSrc</code> before passing it to <code>eleventy-image</code>.</p> <pre><code class="language-javascript"><span class="hljs-comment">// ...</span> <span class="hljs-keyword">if</span> (imgSrc.<span class="hljs-title function_">startsWith</span>(<span class="hljs-string">'/assets'</span>)) { imgSrc = <span class="hljs-string">'src'</span> + imgSrc } <span class="hljs-comment">// ...</span> </code></pre> <h3 id="passing-more-options" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220416-responsive-images-in-markdown-with-eleventy-image/#passing-more-options"><span>Passing more options</span></a></h3> <p>There might be certain situations where you want to pass a custom <code>sizes</code> attribute or even skip image processing altogether. We can define a custom title string syntax for passing in some additional data and then parse it with a <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions">regular expression</a>. This is what I came up with:</p> <pre><code>@skip[widthxheight] ?[sizes] caption </code></pre> <p>This can be parsed with:</p> <pre><code>/^(?<skip>@skip(?:\[(?<width>\d+)x(?<height>\d+)\])? ?)?(?:\?\[(?<sizes>.*?)\] ?)?(?<caption>.*)/ </code></pre> <p>Don’t worry if that’s confusing, regexes aren’t very readable <img class="emoji" draggable="false" alt="💀" src="https://tomichen.com/twemoji/72x72/1f480.png" />. At least we have tools like <a href="https://regexr.com/">RegExr</a>!</p> <p>The code to handle these cases is below. When skipping image processing, you can manually pass in the image dimensions to prevent <a href="https://web.dev/cls/">layout shift</a>. I also decided to skip image processing when linking to an external image.</p> <pre><code class="language-javascript"><span class="hljs-comment">// ...</span> <span class="hljs-keyword">const</span> parsed = (imgTitle || <span class="hljs-string">''</span>).<span class="hljs-title function_">match</span>( <span class="hljs-regexp">/^(?<skip>@skip(?:\[(?<width>\d+)x(?<height>\d+)\])? ?)?(?:\?\[(?<sizes>.*?)\] ?)?(?<caption>.*)/</span> ).<span class="hljs-property">groups</span> <span class="hljs-keyword">if</span> (parsed.<span class="hljs-property">skip</span> || imgSrc.<span class="hljs-title function_">startsWith</span>(<span class="hljs-string">'http'</span>)) { <span class="hljs-keyword">const</span> options = { ...htmlOpts } <span class="hljs-keyword">if</span> (parsed.<span class="hljs-property">sizes</span>) { options.<span class="hljs-property">sizes</span> = parsed.<span class="hljs-property">sizes</span> } <span class="hljs-keyword">const</span> metadata = { <span class="hljs-attr">jpeg</span>: [{ <span class="hljs-attr">url</span>: imgSrc }] } <span class="hljs-keyword">if</span> (parsed.<span class="hljs-property">width</span> && parsed.<span class="hljs-property">height</span>) { metadata.<span class="hljs-property">jpeg</span>[<span class="hljs-number">0</span>].<span class="hljs-property">width</span> = parsed.<span class="hljs-property">width</span> metadata.<span class="hljs-property">jpeg</span>[<span class="hljs-number">0</span>].<span class="hljs-property">height</span> = parsed.<span class="hljs-property">height</span> } <span class="hljs-keyword">const</span> generated = <span class="hljs-title class_">Image</span>.<span class="hljs-title function_">generateHTML</span>(metadata, options) <span class="hljs-keyword">if</span> (parsed.<span class="hljs-property">caption</span>) { <span class="hljs-keyword">return</span> <span class="hljs-title function_">figure</span>(generated, parsed.<span class="hljs-property">caption</span>) } <span class="hljs-keyword">return</span> generated } <span class="hljs-comment">// ...</span> <span class="hljs-keyword">const</span> generated = <span class="hljs-title class_">Image</span>.<span class="hljs-title function_">generateHTML</span>(metadata, { <span class="hljs-attr">sizes</span>: parsed.<span class="hljs-property">sizes</span> || <span class="hljs-string">'(max-width: 768px) 100vw, 768px'</span>, ...htmlOpts }) <span class="hljs-comment">// ...</span> </code></pre> <h2 id="using-with-eleventy" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220416-responsive-images-in-markdown-with-eleventy-image/#using-with-eleventy"><span>Using with Eleventy</span></a></h2> <p>Now that we have a custom markdown renderer and <code>markdown-it</code> instance, we need to tell Eleventy to use it. If you’re not using Eleventy, you can use the markdown-it instance as a drop-in replacement.</p> <pre><code class="language-javascript"><span class="hljs-comment">// in .eleventy.js</span> <span class="hljs-variable language_">module</span>.<span class="hljs-property">exports</span> = <span class="hljs-keyword">function</span> (<span class="hljs-params">eleventyConfig</span>) { <span class="hljs-comment">// ...</span> <span class="hljs-comment">// either put the code here, or factor it out into another file</span> <span class="hljs-keyword">const</span> markdown = <span class="hljs-built_in">require</span>(<span class="hljs-string">'./utils/markdown'</span>) eleventyConfig.<span class="hljs-title function_">setLibrary</span>(<span class="hljs-string">'md'</span>, markdown) <span class="hljs-comment">// ...</span> } </code></pre> <h2 id="final-code" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220416-responsive-images-in-markdown-with-eleventy-image/#final-code"><span>Final Code</span></a></h2> <pre><code class="language-javascript"><span class="hljs-comment">// responsive images with 11ty image</span> <span class="hljs-comment">// this overrides the default image renderer</span> <span class="hljs-comment">// titles are also used for size setting</span> <span class="hljs-comment">// ![alt text](/assets/img/image.jpg "title text")</span> <span class="hljs-comment">// title text format:</span> <span class="hljs-comment">// @skip[widthxheight] ?[sizes] caption</span> <span class="hljs-keyword">const</span> <span class="hljs-title class_">Image</span> = <span class="hljs-built_in">require</span>(<span class="hljs-string">'@11ty/eleventy-img'</span>) markdown.<span class="hljs-property">renderer</span>.<span class="hljs-property">rules</span>.<span class="hljs-property">image</span> = <span class="hljs-keyword">function</span> (<span class="hljs-params">tokens, idx, options, env, self</span>) { <span class="hljs-keyword">function</span> <span class="hljs-title function_">figure</span>(<span class="hljs-params">html, caption</span>) { <span class="hljs-keyword">return</span> <span class="hljs-string">`<figure><span class="hljs-subst">${html}</span><figcaption><span class="hljs-subst">${caption}</span></figcaption></figure>`</span> } <span class="hljs-keyword">const</span> token = tokens[idx] <span class="hljs-keyword">let</span> imgSrc = token.<span class="hljs-title function_">attrGet</span>(<span class="hljs-string">'src'</span>) <span class="hljs-keyword">const</span> imgAlt = token.<span class="hljs-property">content</span> <span class="hljs-keyword">const</span> imgTitle = token.<span class="hljs-title function_">attrGet</span>(<span class="hljs-string">'title'</span>) <span class="hljs-keyword">const</span> htmlOpts = { <span class="hljs-attr">alt</span>: imgAlt, <span class="hljs-attr">loading</span>: <span class="hljs-string">'lazy'</span>, <span class="hljs-attr">decoding</span>: <span class="hljs-string">'async'</span> } <span class="hljs-keyword">if</span> (imgSrc.<span class="hljs-title function_">startsWith</span>(<span class="hljs-string">'/assets'</span>)) { imgSrc = <span class="hljs-string">'src'</span> + imgSrc } <span class="hljs-keyword">const</span> parsed = (imgTitle || <span class="hljs-string">''</span>).<span class="hljs-title function_">match</span>( <span class="hljs-regexp">/^(?<skip>@skip(?:\[(?<width>\d+)x(?<height>\d+)\])? ?)?(?:\?\[(?<sizes>.*?)\] ?)?(?<caption>.*)/</span> ).<span class="hljs-property">groups</span> <span class="hljs-keyword">if</span> (parsed.<span class="hljs-property">skip</span> || imgSrc.<span class="hljs-title function_">startsWith</span>(<span class="hljs-string">'http'</span>)) { <span class="hljs-keyword">const</span> options = { ...htmlOpts } <span class="hljs-keyword">if</span> (parsed.<span class="hljs-property">sizes</span>) { options.<span class="hljs-property">sizes</span> = parsed.<span class="hljs-property">sizes</span> } <span class="hljs-keyword">const</span> metadata = { <span class="hljs-attr">jpeg</span>: [{ <span class="hljs-attr">url</span>: imgSrc }] } <span class="hljs-keyword">if</span> (parsed.<span class="hljs-property">width</span> && parsed.<span class="hljs-property">height</span>) { metadata.<span class="hljs-property">jpeg</span>[<span class="hljs-number">0</span>].<span class="hljs-property">width</span> = parsed.<span class="hljs-property">width</span> metadata.<span class="hljs-property">jpeg</span>[<span class="hljs-number">0</span>].<span class="hljs-property">height</span> = parsed.<span class="hljs-property">height</span> } <span class="hljs-keyword">const</span> generated = <span class="hljs-title class_">Image</span>.<span class="hljs-title function_">generateHTML</span>(metadata, options) <span class="hljs-keyword">if</span> (parsed.<span class="hljs-property">caption</span>) { <span class="hljs-keyword">return</span> <span class="hljs-title function_">figure</span>(generated, parsed.<span class="hljs-property">caption</span>) } <span class="hljs-keyword">return</span> generated } <span class="hljs-keyword">const</span> widths = [<span class="hljs-number">250</span>, <span class="hljs-number">316</span>, <span class="hljs-number">426</span>, <span class="hljs-number">460</span>, <span class="hljs-number">580</span>, <span class="hljs-number">768</span>] <span class="hljs-keyword">const</span> imgOpts = { <span class="hljs-attr">widths</span>: widths .<span class="hljs-title function_">concat</span>(widths.<span class="hljs-title function_">map</span>(<span class="hljs-function">(<span class="hljs-params">w</span>) =></span> w * <span class="hljs-number">2</span>)) <span class="hljs-comment">// generate 2x sizes</span> .<span class="hljs-title function_">filter</span>(<span class="hljs-function">(<span class="hljs-params">v, i, s</span>) =></span> s.<span class="hljs-title function_">indexOf</span>(v) === i), <span class="hljs-comment">// dedupe</span> <span class="hljs-attr">formats</span>: [<span class="hljs-string">'webp'</span>, <span class="hljs-string">'jpeg'</span>], <span class="hljs-comment">// <span class="hljs-doctag">TODO:</span> add avif when support is good enough</span> <span class="hljs-attr">urlPath</span>: <span class="hljs-string">'/assets/img/'</span>, <span class="hljs-attr">outputDir</span>: <span class="hljs-string">'./_site/assets/img/'</span> } <span class="hljs-title class_">Image</span>(imgSrc, imgOpts) <span class="hljs-keyword">const</span> metadata = <span class="hljs-title class_">Image</span>.<span class="hljs-title function_">statsSync</span>(imgSrc, imgOpts) <span class="hljs-keyword">const</span> generated = <span class="hljs-title class_">Image</span>.<span class="hljs-title function_">generateHTML</span>(metadata, { <span class="hljs-attr">sizes</span>: parsed.<span class="hljs-property">sizes</span> || <span class="hljs-string">'(max-width: 768px) 100vw, 768px'</span>, ...htmlOpts }) <span class="hljs-keyword">if</span> (parsed.<span class="hljs-property">caption</span>) { <span class="hljs-keyword">return</span> <span class="hljs-title function_">figure</span>(generated, parsed.<span class="hljs-property">caption</span>) } <span class="hljs-keyword">return</span> generated } </code></pre> <h2 id="acknowledgments" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220416-responsive-images-in-markdown-with-eleventy-image/#acknowledgments"><span>Acknowledgments</span></a></h2> <p>I posted a shortened version of this post as a <a href="https://stackoverflow.com/questions/67429694/adding-a-fallback-images-in-markdown/67680156#67680156">StackOverflow answer</a> last year, but haven’t gotten around to writing this post until now.</p> <p><a href="https://twitter.com/stephenjbell/status/1340380364330708993">This Twitter thread</a> was the original inspiration for this method.</p>A Contrast-Focused Color Picker2022-04-10T00:00:00Z2022-04-18T21:47:31Zhttps://tomichen.com/blog/posts/20220410-a-contrast-focused-color-picker/Choosing colors with good contrast is a critical part of accessible design in all formats and mediums, but it can be difficult to pick these colors with tooling that wasn’t created with contrast in mind. I’ve always wanted to have a color picker that would visually display which colors had sufficient contrast against a base color, but couldn’t find any existing tools that had this feature. So I built it myself!<p>Choosing colors with good contrast is a critical part of accessible design in all formats and mediums. However, it can be difficult to pick these colors with tooling that wasn’t created with contrast in mind.</p> <p>I’ve always wanted to have a color picker that would visually display which colors had sufficient contrast against a base color, but couldn’t find any existing tools that had this feature. Ideally, this would be an otherwise normal HSV color picker, but with a line drawn at the contrast threshold. So I built it myself!</p> <p></p><figure><picture><source type="image/webp" srcset="https://tomichen.com/assets/img/TzxHZIsaZ5-250.webp 250w, https://tomichen.com/assets/img/TzxHZIsaZ5-316.webp 316w, https://tomichen.com/assets/img/TzxHZIsaZ5-426.webp 426w, https://tomichen.com/assets/img/TzxHZIsaZ5-460.webp 460w, https://tomichen.com/assets/img/TzxHZIsaZ5-500.webp 500w, https://tomichen.com/assets/img/TzxHZIsaZ5-580.webp 580w, https://tomichen.com/assets/img/TzxHZIsaZ5-632.webp 632w, https://tomichen.com/assets/img/TzxHZIsaZ5-768.webp 768w, https://tomichen.com/assets/img/TzxHZIsaZ5-852.webp 852w, https://tomichen.com/assets/img/TzxHZIsaZ5-920.webp 920w, https://tomichen.com/assets/img/TzxHZIsaZ5-1160.webp 1160w, https://tomichen.com/assets/img/TzxHZIsaZ5-1536.webp 1536w" sizes="(max-width: 1024px) 100vw, 1024px" /><source type="image/jpeg" srcset="https://tomichen.com/assets/img/TzxHZIsaZ5-250.jpeg 250w, https://tomichen.com/assets/img/TzxHZIsaZ5-316.jpeg 316w, https://tomichen.com/assets/img/TzxHZIsaZ5-426.jpeg 426w, https://tomichen.com/assets/img/TzxHZIsaZ5-460.jpeg 460w, https://tomichen.com/assets/img/TzxHZIsaZ5-500.jpeg 500w, https://tomichen.com/assets/img/TzxHZIsaZ5-580.jpeg 580w, https://tomichen.com/assets/img/TzxHZIsaZ5-632.jpeg 632w, https://tomichen.com/assets/img/TzxHZIsaZ5-768.jpeg 768w, https://tomichen.com/assets/img/TzxHZIsaZ5-852.jpeg 852w, https://tomichen.com/assets/img/TzxHZIsaZ5-920.jpeg 920w, https://tomichen.com/assets/img/TzxHZIsaZ5-1160.jpeg 1160w, https://tomichen.com/assets/img/TzxHZIsaZ5-1536.jpeg 1536w" sizes="(max-width: 1024px) 100vw, 1024px" /><img alt="a hue saturation brightness style color picker with a line across the picker area, representing the boundary between high contrast and low contrast colors against white" loading="lazy" decoding="async" class="ring-1 ring-neutral-900 dark:ring-neutral-50" src="https://tomichen.com/assets/img/TzxHZIsaZ5-250.jpeg" width="1536" height="1664" /></picture><figcaption>The final product</figcaption></figure><p></p> <h2 id="how-do-you-calculate-color-contrast%3F" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220410-a-contrast-focused-color-picker/#how-do-you-calculate-color-contrast%3F"><span>How do you calculate color contrast?</span></a></h2> <p>The standard for web content accessibility is the WCAG, or the <a href="https://www.w3.org/WAI/standards-guidelines/wcag/">Web Content Accessibility Guidelines</a>, which require a minimum contrast ratio of 4.5:1 with 7:1 required in certain situations.</p> <p>To compute color contrast, WCAG 2 uses a <a href="https://www.w3.org/TR/WCAG20-TECHS/G17.html#G17-tests">luminosity-based algorithm</a>. Where do these numbers come from? <a href="https://en.wikipedia.org/wiki/Relative_luminance">Who knows!</a> Color and color perception is a super weird and complex thing, and I have gone down Wikipedia rabbit holes and come out more confused than I started with.</p> <p><a href="https://xkcd.com/1882/"><figure><picture><source type="image/webp" srcset="https://tomichen.com/assets/img/bHIU1yStXB-250.webp 250w, https://tomichen.com/assets/img/bHIU1yStXB-316.webp 316w, https://tomichen.com/assets/img/bHIU1yStXB-426.webp 426w, https://tomichen.com/assets/img/bHIU1yStXB-460.webp 460w, https://tomichen.com/assets/img/bHIU1yStXB-500.webp 500w" sizes="(max-width: 1024px) 100vw, 1024px" /><source type="image/jpeg" srcset="https://tomichen.com/assets/img/bHIU1yStXB-250.jpeg 250w, https://tomichen.com/assets/img/bHIU1yStXB-316.jpeg 316w, https://tomichen.com/assets/img/bHIU1yStXB-426.jpeg 426w, https://tomichen.com/assets/img/bHIU1yStXB-460.jpeg 460w, https://tomichen.com/assets/img/bHIU1yStXB-500.jpeg 500w" sizes="(max-width: 1024px) 100vw, 1024px" /><img alt="xkcd #1882, Color Models" loading="lazy" decoding="async" class="ring-1 ring-neutral-900 dark:ring-neutral-50" src="https://tomichen.com/assets/img/bHIU1yStXB-250.jpeg" width="500" height="627" /></picture><figcaption>“Color Models” from xkcd / CC BY-NC 2.5</figcaption></figure></a></p> <p>Let’s all just thank the W3C for giving us this algorithm and move on with our lives.</p> <h2 id="color-formats" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220410-a-contrast-focused-color-picker/#color-formats"><span>Color Formats</span></a></h2> <p>But wait! We’re not finished with this color nonsense yet! The WCAG 2 algorithm uses RGB values, but our color picker uses HSV (aka HSB, not to be confused with HSL). We have to first figure out how to convert from HSV to RGB, which we can then plug into the contrast algorithm. Luckily for us, <a href="https://en.wikipedia.org/wiki/HSL_and_HSV#HSV_to_RGB_alternative">Wikipedia tells us exactly how to do that</a>!</p> <h2 id="prototyping-in-desmos" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220410-a-contrast-focused-color-picker/#prototyping-in-desmos"><span>Prototyping in Desmos</span></a></h2> <p>To figure out if this will even work, I decided to make a <a href="https://www.desmos.com/calculator/edil445dki">quick prototype</a> in <a href="https://www.desmos.com/">Desmos</a>, a free online graphing calculator.</p> <p><picture><source type="image/webp" srcset="https://tomichen.com/assets/img/DIuRiu-Zip-250.webp 250w, https://tomichen.com/assets/img/DIuRiu-Zip-316.webp 316w, https://tomichen.com/assets/img/DIuRiu-Zip-426.webp 426w, https://tomichen.com/assets/img/DIuRiu-Zip-460.webp 460w, https://tomichen.com/assets/img/DIuRiu-Zip-500.webp 500w, https://tomichen.com/assets/img/DIuRiu-Zip-580.webp 580w, https://tomichen.com/assets/img/DIuRiu-Zip-632.webp 632w, https://tomichen.com/assets/img/DIuRiu-Zip-768.webp 768w, https://tomichen.com/assets/img/DIuRiu-Zip-852.webp 852w, https://tomichen.com/assets/img/DIuRiu-Zip-920.webp 920w, https://tomichen.com/assets/img/DIuRiu-Zip-1160.webp 1160w, https://tomichen.com/assets/img/DIuRiu-Zip-1536.webp 1536w" sizes="(max-width: 1024px) 100vw, 1024px" /><source type="image/jpeg" srcset="https://tomichen.com/assets/img/DIuRiu-Zip-250.jpeg 250w, https://tomichen.com/assets/img/DIuRiu-Zip-316.jpeg 316w, https://tomichen.com/assets/img/DIuRiu-Zip-426.jpeg 426w, https://tomichen.com/assets/img/DIuRiu-Zip-460.jpeg 460w, https://tomichen.com/assets/img/DIuRiu-Zip-500.jpeg 500w, https://tomichen.com/assets/img/DIuRiu-Zip-580.jpeg 580w, https://tomichen.com/assets/img/DIuRiu-Zip-632.jpeg 632w, https://tomichen.com/assets/img/DIuRiu-Zip-768.jpeg 768w, https://tomichen.com/assets/img/DIuRiu-Zip-852.jpeg 852w, https://tomichen.com/assets/img/DIuRiu-Zip-920.jpeg 920w, https://tomichen.com/assets/img/DIuRiu-Zip-1160.jpeg 1160w, https://tomichen.com/assets/img/DIuRiu-Zip-1536.jpeg 1536w" sizes="(max-width: 1024px) 100vw, 1024px" /><img alt="a prototype of the boundary line in Desmos, an online graphing calculator. equations are on the left hand side, and a color gradient representing the picker area is on the right, with a line across it." loading="lazy" decoding="async" class="ring-1 ring-neutral-900 dark:ring-neutral-50" src="https://tomichen.com/assets/img/DIuRiu-Zip-250.jpeg" width="1536" height="716" /></picture></p> <p>The Desmos prototype just involved slapping all the algorithms in there and importing an image of a gradient. Hue is adjustable, as well as the color to contrast against. It worked!</p> <p>This prototype allowed me to check against it when building the actual thing, making sure things worked properly and letting me play with certain parameters.</p> <h2 id="actually-making-the-thing" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220410-a-contrast-focused-color-picker/#actually-making-the-thing"><span>Actually making the thing</span></a></h2> <p>That’s enough screwing around, it’s now time to put this plan into action. The first thing I did (tbh even before all that color research) was to try building my own vanilla color picker. While I could probably have used a pre-built component, I thought it would be easier to add the contrast boundary if I made it myself.</p> <p>I was initially a little daunted by the task but decided to see how other people have built color pickers for inspiration. I discovered that essentially, the picker is just a <code><div></code> with a gradient background and another <code><div></code> absolutely positioned inside as the cursor. Javascript is used to make things interactive and work properly.</p> <p>Since this was going to be a utility in my <a href="https://tools.tomichen.com/">personal tools site</a> built using <a href="https://kit.svelte.dev/">SvelteKit</a>, I could build the picker using <a href="https://svelte.dev/">Svelte</a>, which made things super easy. While I’m not sure how accessible my implementation is, I did make it keyboard usable. If you’re interested in the details, the <a href="https://github.com/tctree333/tools/blob/main/src/routes/color-contrast-picker.svelte">code is open-source on GitHub</a>.</p> <h3 id="computing-the-points" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220410-a-contrast-focused-color-picker/#computing-the-points"><span>Computing the points</span></a></h3> <p>While making a color picker is all good and fun, the interesting part about this project is computing and drawing the line on the color picker. My initial idea was to sort-of brute force search for valid points, then use an SVG to connect the dots. Using brute force didn’t seem very elegant to me, but after discussing it with some friends I decided to go with it anyway, since I couldn’t find any other alternatives. Luckily, the boundary lines are pretty smooth and don’t require too high of a resolution.</p> <p>The brute force method sweeps across the x-axis (saturation), going through the y-axis (brightness) for each x to find the two points where one point passes the contrast check and the other does not. I tried some other methods, but it turns out that this brute-force method was the most stable. I fine-tuned the search intervals to strike a balance between speed and accuracy.</p> <p>Additional logic was added later to handle the case with two boundary lines (where the contrasting color was not black or white). This was done by checking if two points have been found, and splitting the top point into one line and the bottom point into another. If two lines are being generated but only one point is found, we compute the distance between the found point and the end of the generated lines, placing the point into the closer line. This method worked surprisingly well.</p> <h3 id="rendering-the-line" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220410-a-contrast-focused-color-picker/#rendering-the-line"><span>Rendering the line</span></a></h3> <p>Now we have a list of points, it’s time to render the line. I wanted the line to be smooth, with as few bumps or corners as possible. SVG can render <a href="https://developer.mozilla.org/en-US/docs/Web/SVG/Tutorial/Paths#b%C3%A9zier_curves">cubic Béziers</a>, but how do you find the control points?</p> <p>A StackOverflow answer referenced <a href="https://en.wikipedia.org/wiki/Centripetal_Catmull%E2%80%93Rom_spline">Catmull-Rom splines</a>, but it felt a little overkill considering the lines should be fairly smooth. More searching revealed a Medium article on <a href="https://francoisromain.medium.com/smooth-a-svg-path-with-cubic-bezier-curves-e37b49d46c74">smoothing SVG paths</a> that looked promising, so I modified their code to fit my purpose.</p> <h2 id="conclusion" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220410-a-contrast-focused-color-picker/#conclusion"><span>Conclusion</span></a></h2> <p>Wow! Through this project, I learned a lot about color algorithms, unlocked Desmos as another tool for prototyping, and realized that keeping things simple might sometimes be the best solution.</p> <p>Feel free to check out <a href="https://tools.tomichen.com/color-contrast-picker/">the final product</a>, and let me know what you think! I’d love to hear any feedback you have.</p>Redesigning SciOlyID: My process for building websites2022-01-18T00:00:00Z2022-06-01T20:00:16Zhttps://tomichen.com/blog/posts/20220118-redesigning-sciolyid-my-process-for-building-websites/This post will outline my process for building and designing websites, from planning to production. I'll go into how I redesigned the SciOlyID website through structure, functionality, and design.<p>I recently did a redesign and rewrite of the SciOlyID website, available at <a href="https://sciolyid.org/">sciolyid.org</a>. SciOlyID is a group of Discord bots and other services to help students practice specimen identification. It started as just a summer project by my friend and me, and it has since grown into the largest project I’ve ever worked on.</p> <p>The previous website primarily provided an interface for uploading and verifying images for Fossil-ID and Star-ID (now discontinued). The website wasn’t designed to be informational, with a hastily thrown together homepage and a generic theme.</p> <p></p><figure><picture><source type="image/webp" srcset="https://tomichen.com/assets/img/A6hQjm3bCA-250.webp 250w, https://tomichen.com/assets/img/A6hQjm3bCA-316.webp 316w, https://tomichen.com/assets/img/A6hQjm3bCA-426.webp 426w, https://tomichen.com/assets/img/A6hQjm3bCA-460.webp 460w, https://tomichen.com/assets/img/A6hQjm3bCA-500.webp 500w, https://tomichen.com/assets/img/A6hQjm3bCA-580.webp 580w, https://tomichen.com/assets/img/A6hQjm3bCA-632.webp 632w, https://tomichen.com/assets/img/A6hQjm3bCA-768.webp 768w, https://tomichen.com/assets/img/A6hQjm3bCA-852.webp 852w, https://tomichen.com/assets/img/A6hQjm3bCA-920.webp 920w, https://tomichen.com/assets/img/A6hQjm3bCA-1160.webp 1160w" sizes="(max-width: 1024px) 100vw, 1024px" /><source type="image/jpeg" srcset="https://tomichen.com/assets/img/A6hQjm3bCA-250.jpeg 250w, https://tomichen.com/assets/img/A6hQjm3bCA-316.jpeg 316w, https://tomichen.com/assets/img/A6hQjm3bCA-426.jpeg 426w, https://tomichen.com/assets/img/A6hQjm3bCA-460.jpeg 460w, https://tomichen.com/assets/img/A6hQjm3bCA-500.jpeg 500w, https://tomichen.com/assets/img/A6hQjm3bCA-580.jpeg 580w, https://tomichen.com/assets/img/A6hQjm3bCA-632.jpeg 632w, https://tomichen.com/assets/img/A6hQjm3bCA-768.jpeg 768w, https://tomichen.com/assets/img/A6hQjm3bCA-852.jpeg 852w, https://tomichen.com/assets/img/A6hQjm3bCA-920.jpeg 920w, https://tomichen.com/assets/img/A6hQjm3bCA-1160.jpeg 1160w" sizes="(max-width: 1024px) 100vw, 1024px" /><img alt="a screenshot of the sciolyid site before redesign" loading="lazy" decoding="async" class="ring-1 ring-neutral-900 dark:ring-neutral-50" src="https://tomichen.com/assets/img/A6hQjm3bCA-250.jpeg" width="1160" height="1129" /></picture><figcaption>before the redesign</figcaption></figure><p></p> <p>I had always wanted to refresh the website with a sexy landing page and comprehensive documentation but wasn’t motivated to do so. However, after discussions about a possible partnership with the 2022 Science Olympiad National Tournament, I knew that the website would not cut it.</p> <p>This post will outline my process for building and designing websites, from planning to production. As always, I aim to build performant, accessible websites that fulfill a purpose to the best of my ability, so accessibility and performance are factors in every decision I make.</p> <h2 id="structure" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220118-redesigning-sciolyid-my-process-for-building-websites/#structure"><span>Structure</span></a></h2> <p>Whenever I work on a project, I need to understand the scope and requirements. After that’s established, I can start planning the site’s structure.</p> <p>When I say “structure”, I include the technical details and the content structure. In this case, the site would require static pages for documentation and guides, with interactive pages for online ID practice and image verification. These requirements made SvelteKit a clear choice for this project since I’ve used it in the past and enjoyed it quite a bit. SvelteKit allows me to prerender static pages for better performance and use Svelte to build interactive components, which is lightweight and easy to use.</p> <p>For content structure, I started by considering the landing page. Since the landing page has a unique design compared to other pages, I wrote out all the marketing copy beforehand. I brainstormed ideas by asking how SciOlyID is different (and better) than other options, listing the features we support and the various sections I wanted to add. I also brainstormed different hero title ideas. During this process, I looked at other landing page designs for inspiration.</p> <p><picture><source type="image/webp" srcset="https://tomichen.com/assets/img/i_CsBIXH7Q-250.webp 250w, https://tomichen.com/assets/img/i_CsBIXH7Q-316.webp 316w, https://tomichen.com/assets/img/i_CsBIXH7Q-426.webp 426w, https://tomichen.com/assets/img/i_CsBIXH7Q-460.webp 460w, https://tomichen.com/assets/img/i_CsBIXH7Q-500.webp 500w, https://tomichen.com/assets/img/i_CsBIXH7Q-580.webp 580w, https://tomichen.com/assets/img/i_CsBIXH7Q-632.webp 632w" sizes="(max-width: 1024px) 100vw, 1024px" /><source type="image/jpeg" srcset="https://tomichen.com/assets/img/i_CsBIXH7Q-250.jpeg 250w, https://tomichen.com/assets/img/i_CsBIXH7Q-316.jpeg 316w, https://tomichen.com/assets/img/i_CsBIXH7Q-426.jpeg 426w, https://tomichen.com/assets/img/i_CsBIXH7Q-460.jpeg 460w, https://tomichen.com/assets/img/i_CsBIXH7Q-500.jpeg 500w, https://tomichen.com/assets/img/i_CsBIXH7Q-580.jpeg 580w, https://tomichen.com/assets/img/i_CsBIXH7Q-632.jpeg 632w" sizes="(max-width: 1024px) 100vw, 1024px" /><img alt="a section of a Notion document that lists how SciOlyID is different from other study tools" loading="lazy" decoding="async" class="ring-1 ring-neutral-900 dark:ring-neutral-50" src="https://tomichen.com/assets/img/i_CsBIXH7Q-250.jpeg" width="632" height="293" /></picture></p> <p><picture><source type="image/webp" srcset="https://tomichen.com/assets/img/plsIRkGwIh-250.webp 250w, https://tomichen.com/assets/img/plsIRkGwIh-316.webp 316w, https://tomichen.com/assets/img/plsIRkGwIh-426.webp 426w, https://tomichen.com/assets/img/plsIRkGwIh-460.webp 460w, https://tomichen.com/assets/img/plsIRkGwIh-500.webp 500w" sizes="(max-width: 1024px) 100vw, 1024px" /><source type="image/jpeg" srcset="https://tomichen.com/assets/img/plsIRkGwIh-250.jpeg 250w, https://tomichen.com/assets/img/plsIRkGwIh-316.jpeg 316w, https://tomichen.com/assets/img/plsIRkGwIh-426.jpeg 426w, https://tomichen.com/assets/img/plsIRkGwIh-460.jpeg 460w, https://tomichen.com/assets/img/plsIRkGwIh-500.jpeg 500w" sizes="(max-width: 1024px) 100vw, 1024px" /><img alt="a section of a Notion document with many heading ideas and a list of landing page sections" loading="lazy" decoding="async" class="ring-1 ring-neutral-900 dark:ring-neutral-50" src="https://tomichen.com/assets/img/plsIRkGwIh-250.jpeg" width="500" height="446" /></picture></p> <p>Armed with the brainstorming session results, I could combine everything into a final document for the textual content of the landing page.</p> <p>After completing the landing page, I created a list of all the other pages that should exist. This included existing pages that needed to be ported over and new pages to write.</p> <p><picture><source type="image/webp" srcset="https://tomichen.com/assets/img/gbG7Aokw57-250.webp 250w, https://tomichen.com/assets/img/gbG7Aokw57-316.webp 316w, https://tomichen.com/assets/img/gbG7Aokw57-426.webp 426w, https://tomichen.com/assets/img/gbG7Aokw57-460.webp 460w" sizes="(max-width: 1024px) 100vw, 1024px" /><source type="image/jpeg" srcset="https://tomichen.com/assets/img/gbG7Aokw57-250.jpeg 250w, https://tomichen.com/assets/img/gbG7Aokw57-316.jpeg 316w, https://tomichen.com/assets/img/gbG7Aokw57-426.jpeg 426w, https://tomichen.com/assets/img/gbG7Aokw57-460.jpeg 460w" sizes="(max-width: 1024px) 100vw, 1024px" /><img alt="a section of a Notion document listing the pages on the website" loading="lazy" decoding="async" class="ring-1 ring-neutral-900 dark:ring-neutral-50" src="https://tomichen.com/assets/img/gbG7Aokw57-250.jpeg" width="460" height="444" /></picture></p> <h2 id="functionality" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220118-redesigning-sciolyid-my-process-for-building-websites/#functionality"><span>Functionality</span></a></h2> <p>With the structure planned out, it’s now time for the implementation. I laid out the page structure in this stage, creating empty routes to be filled in later. I then worked through page by page, adding the necessary content and functionality. I usually like to start with the more straightforward pages to get them out of the way, then move on to the more complex ones. This is up to personal preference, though.</p> <p>There’s not much else to say here except that Svelte is a joy to work with. Even if the compiled output is much larger than the vanilla counterpart, it’s still lightweight enough (especially compared to other frameworks). The developer-facing code is also smaller and easier to understand.</p> <p>For content-heavy pages, I decided to use <a href="https://mdsvex.pngwn.io/">mdsvex</a> to allow content to be written in markdown. Using mdsvex with SvelteKit is super easy: use the <a href="https://github.com/svelte-add/mdsvex">svelte-add adder for mdsvex</a>, and .md files in src/routes automatically become pages.</p> <h2 id="design" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220118-redesigning-sciolyid-my-process-for-building-websites/#design"><span>Design</span></a></h2> <p>While I could probably have done the styling in vanilla CSS, I decided to use Tailwind for their typography plugin and the color palette. The old SciOlyID site was also styled with Tailwind, allowing me to copy/paste some styles.</p> <p>For the navbar with mobile styles, I followed <a href="https://bholmes.dev/blog/building-a-sexy-mobile-ready-navbar-in-any-web-framework/">a guide from Ben Holmes</a> with some slight modifications. I disabled scrolling when the nav menu is open and modified the background opacity based on whether the browser supports backdrop blur. On wider screens, nested nav menus are <a href="https://css-tricks.com/in-praise-of-the-unambiguous-click-menu/">click menus instead of hover menus</a>.</p> <p>Since I wouldn’t consider myself a designer, I browsed sites like <a href="https://dribbble.com/">Dribbble</a> and <a href="https://tailwindui.com/">Tailwind UI</a> to inspire the landing page. For illustrations, I found some online that were free to use with attribution. I decided to go with a simple warm gray color scheme that matched the graphics pretty well.</p> <p>Even though design can be a rough spot at times, I’ve found that there can be actionable tips to keep in mind that can help make designs look that much better. The free resources from <a href="https://www.refactoringui.com/">Refactoring UI</a> are helpful tips, as well as general advice from <a href="https://www.joshwcomeau.com/">Josh W. Comeau’s blog</a> and his <a href="https://css-for-js.dev/">CSS course</a>.</p> <h2 id="tweaking" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220118-redesigning-sciolyid-my-process-for-building-websites/#tweaking"><span>Tweaking</span></a></h2> <p>Now that everything’s built, we’re done, right? Not quite. There are a lot of little things to tweak here and there. My checklist of things to do before production are:</p> <ul> <li>Fonts</li> <li>Error Pages</li> <li>Favicon</li> <li>Social Cards</li> <li>Lighthouse</li> </ul> <p>I drew the logo and favicon myself in the <a href="https://scratch.mit.edu/">Scratch vector editor</a> (lol). I want to eventually learn how to use a “proper” vector editor like Inkscape, but Scratch is straightforward and has (almost) everything I needed. The alignment is eyeballed, but it shouldn’t be too noticeable at the small sizes of a favicon. I referenced some other logo designs for inspiration.</p> <p><picture><source type="image/webp" srcset="https://tomichen.com/assets/img/aoEFlKVMK2-250.webp 250w, https://tomichen.com/assets/img/aoEFlKVMK2-316.webp 316w, https://tomichen.com/assets/img/aoEFlKVMK2-426.webp 426w, https://tomichen.com/assets/img/aoEFlKVMK2-460.webp 460w, https://tomichen.com/assets/img/aoEFlKVMK2-500.webp 500w, https://tomichen.com/assets/img/aoEFlKVMK2-580.webp 580w, https://tomichen.com/assets/img/aoEFlKVMK2-632.webp 632w, https://tomichen.com/assets/img/aoEFlKVMK2-768.webp 768w, https://tomichen.com/assets/img/aoEFlKVMK2-852.webp 852w, https://tomichen.com/assets/img/aoEFlKVMK2-920.webp 920w, https://tomichen.com/assets/img/aoEFlKVMK2-1160.webp 1160w, https://tomichen.com/assets/img/aoEFlKVMK2-1536.webp 1536w" sizes="(max-width: 1024px) 100vw, 1024px" /><source type="image/jpeg" srcset="https://tomichen.com/assets/img/aoEFlKVMK2-250.jpeg 250w, https://tomichen.com/assets/img/aoEFlKVMK2-316.jpeg 316w, https://tomichen.com/assets/img/aoEFlKVMK2-426.jpeg 426w, https://tomichen.com/assets/img/aoEFlKVMK2-460.jpeg 460w, https://tomichen.com/assets/img/aoEFlKVMK2-500.jpeg 500w, https://tomichen.com/assets/img/aoEFlKVMK2-580.jpeg 580w, https://tomichen.com/assets/img/aoEFlKVMK2-632.jpeg 632w, https://tomichen.com/assets/img/aoEFlKVMK2-768.jpeg 768w, https://tomichen.com/assets/img/aoEFlKVMK2-852.jpeg 852w, https://tomichen.com/assets/img/aoEFlKVMK2-920.jpeg 920w, https://tomichen.com/assets/img/aoEFlKVMK2-1160.jpeg 1160w, https://tomichen.com/assets/img/aoEFlKVMK2-1536.jpeg 1536w" sizes="(max-width: 1024px) 100vw, 1024px" /><img alt="a screenshot of the scratch vector editor" loading="lazy" decoding="async" class="ring-1 ring-neutral-900 dark:ring-neutral-50" src="https://tomichen.com/assets/img/aoEFlKVMK2-250.jpeg" width="1536" height="1236" /></picture></p> <p><picture><source type="image/webp" srcset="https://tomichen.com/assets/img/-MRBMAc3j8-250.webp 250w, https://tomichen.com/assets/img/-MRBMAc3j8-316.webp 316w, https://tomichen.com/assets/img/-MRBMAc3j8-426.webp 426w, https://tomichen.com/assets/img/-MRBMAc3j8-460.webp 460w, https://tomichen.com/assets/img/-MRBMAc3j8-500.webp 500w, https://tomichen.com/assets/img/-MRBMAc3j8-580.webp 580w" sizes="(max-width: 1024px) 100vw, 1024px" /><source type="image/jpeg" srcset="https://tomichen.com/assets/img/-MRBMAc3j8-250.jpeg 250w, https://tomichen.com/assets/img/-MRBMAc3j8-316.jpeg 316w, https://tomichen.com/assets/img/-MRBMAc3j8-426.jpeg 426w, https://tomichen.com/assets/img/-MRBMAc3j8-460.jpeg 460w, https://tomichen.com/assets/img/-MRBMAc3j8-500.jpeg 500w, https://tomichen.com/assets/img/-MRBMAc3j8-580.jpeg 580w" sizes="(max-width: 1024px) 100vw, 1024px" /><img alt="sciolyid logo" loading="lazy" decoding="async" class="ring-1 ring-neutral-900 dark:ring-neutral-50" src="https://tomichen.com/assets/img/-MRBMAc3j8-250.jpeg" width="580" height="219" /></picture></p> <p>You can use any of the many favicon generators online to create the appropriate files.</p> <p>Feedback from family and friends was also good, especially if you can watch someone try to use the website. This will help you figure out if the flow makes sense.</p> <h2 id="conclusion" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20220118-redesigning-sciolyid-my-process-for-building-websites/#conclusion"><span>Conclusion</span></a></h2> <p>I hope this peek into my process for frontend web development was helpful, or at least a little interesting. Remember that this is how I approach things, which may be different from others. I’m also constantly evolving my process to find what works and what doesn’t.</p> <p></p><figure><picture><source type="image/webp" srcset="https://tomichen.com/assets/img/IT8UynbMmE-250.webp 250w, https://tomichen.com/assets/img/IT8UynbMmE-316.webp 316w, https://tomichen.com/assets/img/IT8UynbMmE-426.webp 426w, https://tomichen.com/assets/img/IT8UynbMmE-460.webp 460w, https://tomichen.com/assets/img/IT8UynbMmE-500.webp 500w, https://tomichen.com/assets/img/IT8UynbMmE-580.webp 580w, https://tomichen.com/assets/img/IT8UynbMmE-632.webp 632w, https://tomichen.com/assets/img/IT8UynbMmE-768.webp 768w, https://tomichen.com/assets/img/IT8UynbMmE-852.webp 852w, https://tomichen.com/assets/img/IT8UynbMmE-920.webp 920w, https://tomichen.com/assets/img/IT8UynbMmE-1160.webp 1160w" sizes="(max-width: 1024px) 100vw, 1024px" /><source type="image/jpeg" srcset="https://tomichen.com/assets/img/IT8UynbMmE-250.jpeg 250w, https://tomichen.com/assets/img/IT8UynbMmE-316.jpeg 316w, https://tomichen.com/assets/img/IT8UynbMmE-426.jpeg 426w, https://tomichen.com/assets/img/IT8UynbMmE-460.jpeg 460w, https://tomichen.com/assets/img/IT8UynbMmE-500.jpeg 500w, https://tomichen.com/assets/img/IT8UynbMmE-580.jpeg 580w, https://tomichen.com/assets/img/IT8UynbMmE-632.jpeg 632w, https://tomichen.com/assets/img/IT8UynbMmE-768.jpeg 768w, https://tomichen.com/assets/img/IT8UynbMmE-852.jpeg 852w, https://tomichen.com/assets/img/IT8UynbMmE-920.jpeg 920w, https://tomichen.com/assets/img/IT8UynbMmE-1160.jpeg 1160w" sizes="(max-width: 1024px) 100vw, 1024px" /><img alt="a screenshot of the sciolyid website after redesign" loading="lazy" decoding="async" class="ring-1 ring-neutral-900 dark:ring-neutral-50" src="https://tomichen.com/assets/img/IT8UynbMmE-250.jpeg" width="1160" height="2618" /></picture><figcaption>after the redesign</figcaption></figure><p></p>Tracking Positive COVID-19 Cases in My School District2021-09-05T00:00:00Z2022-06-01T20:00:16Zhttps://tomichen.com/blog/posts/20210905-tracking-positive-covid-19-cases-in-my-school-district/Last month, school started in person, and while masks are required, students are still crowded into classrooms. Our school district has a Positive Case Dashboard that lists the number of students and staff on campuses that test positive for COVID-19, but no historical tracking or info on daily changes. I decided to use Simon Willison’s git scraping technique to store historical data about positive cases and track changes over time.<p>Last month, school started in person. I go to Castro Valley High School in Castro Valley, an unincorporated area in Alameda County, California. While there is a mask requirement, students are still crowded into classrooms with no social distancing.</p> <p>With the pandemic still happening, our school district has published a <a href="https://www.cv.k12.ca.us/apps/pages/index.jsp?uREC_ID=1728675&type=d&pREC_ID=2165165">Positive Case Dashboard</a> that lists the number of students and staff on campuses that test positive for COVID-19. However, it only lists totals for each month, without information about the number of new cases each day. I decided to use <a href="https://simonwillison.net/2020/Oct/9/git-scraping/">Simon Willison’s git scraping technique</a> to store historical data about positive cases and track changes over time.</p> <p>The <a href="https://github.com/tctree333/CVUSD-COVID19-Cases">repository for this project</a> is publicly available on GitHub. Check it out if you want to adapt it for yourself, or take a look at <a href="https://github.com/tctree333/CVUSD-COVID19-Cases/commits/main/data/daily.csv">how cases have changed over time</a>.</p> <p>We’ll also be logging the data to a Google Sheet, which allows us to do some calculations and visualizations. Look at my hawt <img class="emoji" draggable="false" alt="🥵" src="https://tomichen.com/twemoji/72x72/1f975.png" /><img class="emoji" draggable="false" alt="🤤" src="https://tomichen.com/twemoji/72x72/1f924.png" /> graphs (that are live updating too!)</p> <p><img alt="graph of total cases over time" loading="lazy" decoding="async" class="ring-1 ring-neutral-900 dark:ring-neutral-50" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vSQsh8AKab1supcISGvs753qjOEbB0MBbVS3ipsQIVtK6vIvXjxgTJW8QRddVJqQJOmHZ_wW-5Jhikj/pubchart?oid=426307024&format=image" width="806" height="498" /> <img alt="graph of daily case change" loading="lazy" decoding="async" class="ring-1 ring-neutral-900 dark:ring-neutral-50" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vSQsh8AKab1supcISGvs753qjOEbB0MBbVS3ipsQIVtK6vIvXjxgTJW8QRddVJqQJOmHZ_wW-5Jhikj/pubchart?oid=1445148168&format=image" width="806" height="498" /> <img alt="graph of estimated active cases" loading="lazy" decoding="async" class="ring-1 ring-neutral-900 dark:ring-neutral-50" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vSQsh8AKab1supcISGvs753qjOEbB0MBbVS3ipsQIVtK6vIvXjxgTJW8QRddVJqQJOmHZ_wW-5Jhikj/pubchart?oid=903770645&format=image" width="806" height="498" /></p> <h2 id="fetching-the-data" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210905-tracking-positive-covid-19-cases-in-my-school-district/#fetching-the-data"><span>Fetching the data</span></a></h2> <p>To get the data from the district’s spreadsheet, convert it to CSV for a cleaner format, and save it back to the repository, I decided to use <a href="https://github.com/tctree333/CVUSD-COVID19-Cases/blob/main/main.py">a Python script</a> with the <code>requests</code> and <code>pandas</code> libraries. While you could probably do this with bash scripting, I was more familiar with Python.</p> <p>I’m using <a href="https://github.com/features/actions">GitHub Actions</a> to run this script on a cron schedule and re-commit the data back to the repository.</p> <pre><code class="language-yaml"><span class="hljs-attr">name:</span> <span class="hljs-string">Scrape</span> <span class="hljs-string">data</span> <span class="hljs-attr">on:</span> <span class="hljs-attr">workflow_dispatch:</span> <span class="hljs-attr">schedule:</span> <span class="hljs-comment"># run every 4 hours from 5:06 AM PDT (4 AM PST) to 9:06 PM PDT (8 PM PST) Mon-Fri</span> <span class="hljs-bullet">-</span> <span class="hljs-attr">cron:</span> <span class="hljs-string">'6 12,16,20,0,4 * * 1-5'</span> <span class="hljs-attr">jobs:</span> <span class="hljs-attr">scrape:</span> <span class="hljs-attr">runs-on:</span> <span class="hljs-string">ubuntu-latest</span> <span class="hljs-attr">steps:</span> <span class="hljs-bullet">-</span> <span class="hljs-attr">uses:</span> <span class="hljs-string">actions/checkout@v2</span> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">Install</span> <span class="hljs-string">dependencies</span> <span class="hljs-attr">run:</span> <span class="hljs-string">pip</span> <span class="hljs-string">install</span> <span class="hljs-string">-r</span> <span class="hljs-string">requirements.txt</span> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">Fetch</span> <span class="hljs-string">latest</span> <span class="hljs-string">data</span> <span class="hljs-attr">run:</span> <span class="hljs-string">python</span> <span class="hljs-string">main.py</span> <span class="hljs-attr">env:</span> <span class="hljs-attr">LOGGER_DEPLOYMENT_ID:</span> <span class="hljs-string">$</span> <span class="hljs-attr">LOGGER_SECRET:</span> <span class="hljs-string">$</span> <span class="hljs-bullet">-</span> <span class="hljs-attr">name:</span> <span class="hljs-string">Commit</span> <span class="hljs-string">data</span> <span class="hljs-attr">run:</span> <span class="hljs-string">|- if [[ `git status --porcelain` ]]; then git config user.name github-actions git config user.email github-actions@github.com git add -A git commit -m "add latest data" git push fi </span></code></pre> <h2 id="visualizing-the-data" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210905-tracking-positive-covid-19-cases-in-my-school-district/#visualizing-the-data"><span>Visualizing the data</span></a></h2> <p>After I got the git scraping thing to work, I decided I wanted to graph the data in a <a href="https://www.google.com/sheets/about/">Google Sheet</a>. Since I’m scared of Google’s API authentication system, I used a <a href="https://developers.google.com/apps-script">Google Apps Script</a> deployed as a web app so I could avoid authenticating to Google in the Python script. If you haven’t tried it, Apps Script is a pretty powerful tool for interacting and connecting across Google Services. It’s just JavaScript and the documentation is decent, so I’d recommend trying it out!</p> <p>To create an <a href="https://developers.google.com/apps-script/guides/sheets">App Script attached to a Sheet</a>, go to Tools > Script Editor, which should open up an Apps Script editor. Besides deploying as a web app, you can also write custom spreadsheet functions!</p> <p>My App Script is below:</p> <pre><code class="language-javascript"><span class="hljs-keyword">const</span> <span class="hljs-variable constant_">SECRET_KEY</span> = <span class="hljs-string">'lol no'</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">doGet</span>(<span class="hljs-params">e</span>) { <span class="hljs-comment">// these are http query parameters</span> <span class="hljs-comment">// sig - hmac sha256 of "time|date|student|staff"</span> <span class="hljs-comment">// time - unix time in seconds</span> <span class="hljs-keyword">const</span> { sig, time } = e.<span class="hljs-property">parameter</span> <span class="hljs-keyword">const</span> { date, studentCases, staffCases } = e.<span class="hljs-property">parameter</span> <span class="hljs-keyword">const</span> currentTime = <span class="hljs-title class_">Math</span>.<span class="hljs-title function_">round</span>(<span class="hljs-title class_">Date</span>.<span class="hljs-title function_">now</span>() / <span class="hljs-number">1000</span>) <span class="hljs-comment">// time in seconds</span> <span class="hljs-keyword">if</span> ( <span class="hljs-title class_">Math</span>.<span class="hljs-title function_">abs</span>(currentTime - <span class="hljs-built_in">parseInt</span>(time)) > <span class="hljs-number">30</span> || <span class="hljs-comment">// invalid if timestamp isn't within 30 seconds of now</span> !<span class="hljs-title function_">verifySig</span>(<span class="hljs-string">`<span class="hljs-subst">${time}</span>|<span class="hljs-subst">${date}</span>|<span class="hljs-subst">${studentCases}</span>|<span class="hljs-subst">${staffCases}</span>`</span>, sig) ) { <span class="hljs-keyword">return</span> <span class="hljs-title class_">ContentService</span>.<span class="hljs-title function_">createTextOutput</span>(<span class="hljs-string">'invalid signature!'</span>) } <span class="hljs-keyword">const</span> sheet = <span class="hljs-title class_">SpreadsheetApp</span>.<span class="hljs-title function_">getActiveSpreadsheet</span>().<span class="hljs-title function_">getSheetByName</span>(<span class="hljs-string">'Raw'</span>) sheet.<span class="hljs-title function_">appendRow</span>([date, studentCases, staffCases]) <span class="hljs-keyword">return</span> <span class="hljs-title class_">ContentService</span>.<span class="hljs-title function_">createTextOutput</span>(<span class="hljs-string">'ok!'</span>) } <span class="hljs-keyword">function</span> <span class="hljs-title function_">verifySig</span>(<span class="hljs-params">content, sig</span>) { <span class="hljs-keyword">const</span> computedSig = <span class="hljs-title class_">Utilities</span>.<span class="hljs-title function_">computeHmacSha256Signature</span>( content, <span class="hljs-variable constant_">SECRET_KEY</span> ).<span class="hljs-title function_">reduce</span>(<span class="hljs-function">(<span class="hljs-params">acc, char</span>) =></span> { char = (char < <span class="hljs-number">0</span> ? char + <span class="hljs-number">256</span> : char).<span class="hljs-title function_">toString</span>(<span class="hljs-number">16</span>).<span class="hljs-title function_">padStart</span>(<span class="hljs-number">2</span>, <span class="hljs-string">'0'</span>) <span class="hljs-keyword">return</span> acc + char }, <span class="hljs-string">''</span>) <span class="hljs-keyword">return</span> sig === computedSig } </code></pre> <p>Because I’m exposing the script to the internet, I thought it might be better to add some kind of authentication using an HMAC of the data and a timestamp. This probably isn’t necessary since the deployment ID of the Apps Script is a secret in GitHub Actions, but it can’t hurt, right?</p> <h3 id="spreadsheet-organization" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210905-tracking-positive-covid-19-cases-in-my-school-district/#spreadsheet-organization"><span>Spreadsheet organization</span></a></h3> <p>An <a href="https://docs.google.com/spreadsheets/d/10LsiY12LBnNJ0OSryU8AnhroNbh19mwC-aJAL0H8oa8/edit?usp=sharing">example copy of the spreadsheet</a> is available if you want to follow along.</p> <p>My spreadsheet has two sub-sheets: <code>Raw</code> and <code>Filtered</code>. Raw data from the App Script gets appended to the bottom of the <code>Raw</code> sheet, and cleanup/analysis is performed in the <code>Filtered</code> sheet.</p> <p>Since the GitHub Action runs every 4 hours and sends data to the App Script even if the data doesn’t change, the <code>Raw</code> sheet contains duplicates (that I occasionally manually remove). This means that we only need to keep the latest data for each date, discarding the rest. We can do this with the <code>UNIQUE</code> function, as well as our friends <code>ARRAYFORMULA</code> and <code>VLOOKUP</code>.</p> <p>Cell <code>Filtered!G2</code> de-dupes date values with <code>=UNIQUE(Raw!$A$2:$A)</code>, and cells <code>H2:G2</code> find the respective data point with <code>=ARRAYFORMULA(IFNA(VLOOKUP($G$2:$G, Raw!$A$2:$D, 2)))</code>.</p> <p>Let’s break down this formula:</p> <ul> <li>ARRAYFORMULA allows us to use ranges in functions so we don’t need to copy the same formula into each cell</li> <li>VLOOKUP tries to find a given value in the first column of a range, then returns some value in that row</li> <li>IFNA prevents errors from being displayed if VLOOKUP fails</li> </ul> <p>In this case, the ARRAYFORMULA does mess with VLOOKUP’s range input, but it still gives the desired effect (when the Raw sheet has the latest data closer to the bottom), so I’m not too concerned.</p> <p>There is another, more important issue, though. The district’s data resets the numbers every month instead of displaying a running total, so if I directly graph the numbers, there will be a sudden drop at the beginning of each month. This is where the 10 other columns in the <code>Filtered</code> sheet come in.</p> <p>At a high level, I’m pulling out the month and previous month from each data point (columns E:F), finding the last values from each month (columns K:N), then adding the final number from the previous month (columns A:D). After all this work, we can finally graph these columns.</p> <p>Tip: If you don’t want these calculation columns to take up space, you can hide them! Select the columns, then right-click and select “Hide Columns E - N”.</p> <h2 id="conclusion" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210905-tracking-positive-covid-19-cases-in-my-school-district/#conclusion"><span>Conclusion</span></a></h2> <p>By creating this COVID-19 tracker, we’ve combined different techniques to scrape, store, and visualize data. Hopefully, our district can keep these numbers low with continued masking, vaccinations, testing, and quarantine. But even if we don’t, at least we have some cool graphs!</p>Improving Mobile and Keyboard Usability in TwemojiExplorer2021-06-09T00:00:00Z2022-04-18T21:47:31Zhttps://tomichen.com/blog/posts/20210609-improving-mobile-keyboard-usability/I recently built TwemojiExplorer, a website that gives convenient access to Twitter's Twemoji emoji set. Since the site is primarily a tool for my personal use, I did not put too much thought into keyboard or mobile usability during the initial development process. Now that I have a bit of time, I decided to tweak a couple of small things to make the experience better.<p>I recently built <a href="https://twemoji.tomichen.com/">TwemojiExplorer</a>, a website that gives convenient access to <a href="https://github.com/twitter/twemoji/">Twitter’s Twemoji</a> emoji set. Since the site is primarily a tool for my personal use, I did not put too much thought into keyboard or mobile usability during the initial development process. Now that I have a bit of time, I decided to tweak a couple of small things to make the experience better. If you’re interested in how I implemented and optimized the search functionality, you can <a href="https://tomichen.com/blog/posts/20210608-searching-displaying-large-datasets-svelte/">read about it here</a>.</p> <p>Most of these improvements are located in <a href="https://github.com/tctree333/TwemojiExplorer/commit/5951e2f72bb5a20ca53c1ddb78c81518bbea4f7f">this GitHub commit</a> if you want to follow along. You can also see a version of the site <a href="https://60c05994d4c4d30007fd8465--twemoji-explorer.netlify.app/">before I made these changes</a> and see the difference!</p> <h2 id="tabbing-through-elements" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210609-improving-mobile-keyboard-usability/#tabbing-through-elements"><span>Tabbing through elements</span></a></h2> <p>To access the actions for each emoji (copying SVG/codepoint), the user needs to use their mouse to hover over the emoji card. I’m using Javascript for this effect since in this case, I don’t want the actions to be displayed when Javascript is disabled. Since the only action a user can perform without Javascript is navigate to one of my favorite websites, Emojipedia, I decided to disable the action buttons when Javascript is disabled and put the Emojipedia link on the emoji image instead. Note: I would not recommend using Javascript for hover events in general. This is a special case and I have intentionally implemented it this way. Generally, you’ll want to use CSS’s <code>:hover</code> to keep hover events for users with Javascript disabled.</p> <h3 id="focus-styles" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210609-improving-mobile-keyboard-usability/#focus-styles"><span>Focus styles</span></a></h3> <p>Because the card itself is already a <code><button></code>, it’s tabbable by default. Keyboard users can also activate the button with the <code>on:click</code>, and can also tab through the actions when displayed. The first thing we can do is improve the <code>focus</code> state styles. I’m disabling the default focus styles with:</p> <pre><code class="language-css"><span class="hljs-comment">/* * The Emojipedia link should be styled as a button * so it has a `button` class that I'm targeting. */</span> <span class="hljs-selector-tag">button</span>, <span class="hljs-selector-class">.button</span> { <span class="hljs-attribute">outline</span>: <span class="hljs-number">2px</span> solid transparent; <span class="hljs-attribute">outline-offset</span>: <span class="hljs-number">2px</span>; } </code></pre> <p>Using a transparent outline instead of removing it altogether is important for <a href="https://blogs.windows.com/msedgedev/2020/09/17/styling-for-windows-high-contrast-with-new-standards-for-forced-colors/">Windows high contrast mode</a> users. It’s also how <a href="https://tailwindcss.com/docs/outline#remove-outlines">Tailwind’s <code>outline-none</code> utility</a> works.</p> <p>Now that we’ve removed the default focus styling, we need to add our own:</p> <pre><code class="language-css"><span class="hljs-selector-tag">button</span><span class="hljs-selector-pseudo">:focus</span>-visible, <span class="hljs-selector-class">.button</span><span class="hljs-selector-pseudo">:focus</span>-visible { <span class="hljs-attribute">box-shadow</span>: <span class="hljs-number">0</span> <span class="hljs-number">0</span> <span class="hljs-number">0</span> <span class="hljs-number">4px</span> <span class="hljs-built_in">hsl</span>(<span class="hljs-number">213</span>, <span class="hljs-number">89%</span>, <span class="hljs-number">47%</span>); } </code></pre> <p>Let’s break this down:</p> <ol> <li>We’re using the <code>focus-visible</code> <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/Pseudo-classes">pseudo-class</a> to only apply styles when the element is focused and the browser determines that there <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/:focus-visible">should be a visual focus</a> indicator. When the element is tabbed into, our styles will be applied, but if it’s clicked (which still triggers the <code>:focus</code> class), they won’t be.</li> <li>We’re using a <code>box-shadow</code> to apply focus styles instead of <code>outline</code> or <code>border</code> since they will follow the <code>border-radius</code> of our buttons.</li> <li>I also updated the color in a later commit to improve contrast.</li> </ol> <h3 id="tab-order" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210609-improving-mobile-keyboard-usability/#tab-order"><span>Tab order</span></a></h3> <p>If you take a look at the site <a href="https://60c05994d4c4d30007fd8465--twemoji-explorer.netlify.app/">before I made these changes</a> and try to tab through the elements, you’ll notice two things:</p> <ol> <li>The first tabbed element is a <code><div></code> that wraps the emoji sections, and</li> <li>the link around the emoji image that’s meant for users with disabled Javascript is also tabbable.</li> </ol> <p>I’m not entirely sure why the <code><div></code> is tabbable or if there’s a good reason for it to be, but I don’t think that should be happening, so I disabled it by adding the <code>tabindex="-1"</code> attribute. This takes the element out of the tab order, making it unreachable using the keyboard.</p> <p>We could do the same thing for the link around the image, but I think users with Javascript disabled might still want to tab through those links. We can meet this requirement by setting the attribute using Javascript. We’re also going to use Javascript to intercept clicks and prevent the browser from navigating to the page.</p> <pre><code class="language-html"><span class="hljs-tag"><<span class="hljs-name">script</span>></span><span class="language-javascript"> <span class="hljs-keyword">import</span> { onMount } <span class="hljs-keyword">from</span> <span class="hljs-string">'svelte'</span>; <span class="hljs-keyword">let</span> noJSAnchor; <span class="hljs-title function_">onMount</span>(<span class="hljs-function">() =></span> { noJSAnchor.<span class="hljs-property">tabIndex</span> = -<span class="hljs-number">1</span>; }); </span><span class="hljs-tag"></<span class="hljs-name">script</span>></span> <span class="hljs-tag"><<span class="hljs-name">a</span> <span class="hljs-attr">bind:this</span>=<span class="hljs-string">{noJSAnchor}</span> <span class="hljs-attr">on:click</span>=<span class="hljs-string">{(e)</span> =></span> {e.preventDefault()}} ><span class="hljs-tag"><<span class="hljs-name">img</span> /></span><span class="hljs-tag"></<span class="hljs-name">a</span>></span> </code></pre> <p>If Javascript is disabled, none of this will run, allowing users to navigate and tab normally.</p> <p>Note: I’m using Firefox on a Mac, which has some weird default tab behaviors that prevent me from tabbing through links. You can change this by going to System Preferences > Keyboard > Shortcuts and checking the “Use keyboard navigation to move focus between controls” checkbox at the bottom. You may also need to restart Firefox for the changes to take effect.</p> <h2 id="mobile-actions" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210609-improving-mobile-keyboard-usability/#mobile-actions"><span>Mobile Actions</span></a></h2> <p>If you try to access the old version on a mobile device, you’ll notice that tapping on the cards can also trigger the button that pops up, even if you’re only tapping once. This seems to be because phones will also trigger the <code>mouseover</code> and <code>mouseleave</code> events, even if there’s no mouse. When you tap on the card, your phone will trigger the <code>mouseover</code> event <strong>AND</strong> the <code>click</code> event in succession, which will open up the actions menu and also the buttons inside. This isn’t what we want at all!</p> <p>Ideally, one tap would open the options menu, and another tap would trigger the appropriate button. Luckily, we can detect, using media queries, if a device supports hovering as well as whether there is “fine” or “coarse” control over the pointer. For more information, see this <a href="https://css-tricks.com/interaction-media-features-and-their-potential-for-incorrect-assumptions/">CSS-Tricks article</a>.</p> <p>We can access media queries in Javascript with <code>[window.matchMedia](https://developer.mozilla.org/en-US/docs/Web/API/Window/matchMedia)</code>, which returns a <a href="https://developer.mozilla.org/en-US/docs/Web/API/MediaQueryList"><code>MediaQueryList</code></a> that has a <code>matches</code> property we can access.</p> <pre><code class="language-html"><span class="hljs-tag"><<span class="hljs-name">script</span>></span><span class="language-javascript"> <span class="hljs-keyword">function</span> <span class="hljs-title function_">showActions</span>(<span class="hljs-params"></span>) { <span class="hljs-keyword">if</span> (<span class="hljs-variable language_">window</span>.<span class="hljs-title function_">matchMedia</span>(<span class="hljs-string">'(hover: hover) and (pointer: fine)'</span>).<span class="hljs-property">matches</span>) { currentActive.<span class="hljs-title function_">set</span>(emoji.<span class="hljs-property">slug</span>) } } <span class="hljs-keyword">function</span> <span class="hljs-title function_">hideActions</span>(<span class="hljs-params"></span>) { currentActive.<span class="hljs-title function_">set</span>(<span class="hljs-literal">null</span>) } <span class="hljs-keyword">function</span> <span class="hljs-title function_">click</span>(<span class="hljs-params"></span>) { currentActive.<span class="hljs-title function_">set</span>(emoji.<span class="hljs-property">slug</span>) } </span><span class="hljs-tag"></<span class="hljs-name">script</span>></span> <span class="hljs-tag"><<span class="hljs-name">button</span> <span class="hljs-attr">on:mouseover</span>=<span class="hljs-string">"{showActions}"</span> <span class="hljs-attr">on:mouseleave</span>=<span class="hljs-string">"{hideActions}"</span> <span class="hljs-attr">on:click</span>=<span class="hljs-string">"{click}"</span>></span> <span class="hljs-comment"><!-- ... --></span> <span class="hljs-tag"></<span class="hljs-name">button</span>></span> </code></pre> <p>Here, we’re checking if the device’s primary input supports hovering and has fine control before showing the options menu. We’re not checking this on <code>mouseleave</code> since hiding the menu when the user taps away seems to be fine.</p> <p>Tip: If you use Firefox and Android, you can use <a href="https://developer.mozilla.org/en-US/docs/Tools/about:debugging">mobile debugging</a> to play around with these events and media queries. There are also similar features for <a href="https://www.browserstack.com/guide/how-to-debug-on-iphone">Safari on iOS</a> and <a href="https://developer.chrome.com/docs/devtools/remote-debugging/">Chrome on Android</a>.</p> <h2 id="conclusion" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210609-improving-mobile-keyboard-usability/#conclusion"><span>Conclusion</span></a></h2> <p>With these few tweaks, we can improve the user experience for people on mobile devices or people who use keyboards. While there are still improvements that can be made (using arrow keys to navigate the grid?), this seems like a decent place to be. There is a responsibility on people who build websites to ensure they are accessible to everyone, and I hope this article can help you think about the decisions you make while building websites.</p> <p>If you have any questions or comments, please reach out! I’d love to hear about things I’ve overlooked or any other feedback you may have.</p>Searching and Displaying Large Datasets with Svelte2021-06-08T00:00:00Z2022-04-18T21:47:31Zhttps://tomichen.com/blog/posts/20210608-searching-displaying-large-datasets-svelte/Searching large datasets can be a challenge, especially with live updates as the user types. Not only is re-rendering thousands of DOM nodes taxing on memory, but the heavy processing involved can also make the page unresponsive. Here, I'll share my process of working through these issues and improving the user experience on a new project.<p>Searching large datasets can be a challenge, especially if you’re rendering the results, and even more so if you’re updating results as the user is typing. Not only is re-rendering thousands of DOM nodes taxing on memory, doing so while the user types can make the text input unresponsive, creating a poor experience. Here, I’ll share my process of working through these issues and improving the user experience.</p> <h2 id="background" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210608-searching-displaying-large-datasets-svelte/#background"><span>Background</span></a></h2> <p>These last few days, I’ve been working on <a href="https://twemoji.tomichen.com/">TwemojiExplorer</a>, a website that allows easy access to Twitter’s Twemoji emoji set. All the emojis are there, sorted by category, with buttons to copy the SVG, the Unicode codepoint, and a link to one of my favorite websites, Emojipedia. When building websites, I occasionally want to sprinkle in some emojis to spice things up. The problem was that my process was a tad convoluted.</p> <ol> <li>First, I had to go to one of my favorite websites, Emojipedia, to look up the emoji and find the codepoint.</li> <li>Next, I had to go to Twemoji’s GitHub and search for the proper file.</li> <li>After I locate the correct emoji file, I can finally copy the SVG.</li> <li>Repeat for any other emojis I need.</li> </ol> <p>I decided to build a tool to help me, so the target audience of this site is myself. Since I’m lazy and this site primarily for my personal use, it’s not entirely mobile-friendly or keyboard accessible. Improving the mobile and keyboard experience is something I might improve in the future.</p> <p><strong>UPDATE:</strong> I improved the mobile and keyboard experience with a few small tweaks. You can <a href="https://tomichen.com/blog/posts/20210609-improving-mobile-keyboard-usability/">read all about it</a>!</p> <h2 id="the-problem" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210608-searching-displaying-large-datasets-svelte/#the-problem"><span>The Problem</span></a></h2> <p>I wanted this website to search all 1,805 emojis as the user is typing. No hitting the Enter key for me! There should be a card for each emoji with buttons to copy the SVG, codepoint, and a link to the corresponding page on one of my favorite websites, Emojipedia. I also wanted to try using <a href="https://github.com/elderjs/elderjs">Elder.js,</a> a static site generator using Svelte.</p> <p>I’m not going to be implementing a search library because that’s another project for another time, but I’ll be integrating one and (trying) to optimize the project.</p> <h2 id="attempt-%231%3A-fuse.js" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210608-searching-displaying-large-datasets-svelte/#attempt-%231%3A-fuse.js"><span>Attempt #1: Fuse.js</span></a></h2> <p>My first attempt was using <a href="https://fusejs.io/">Fuse.js</a> for the search. While I didn’t need fuzzy searching, we were planning on using Fuse for another project, so I wanted to try it out. Using Fuse turned out to be a bad idea, and after typing more than a few characters, it would lag out to oblivion. I do think you could make it work, especially with some of the tricks described in the rest of the article, but I decided to look for another solution.</p> <p>Another issue was how I was rendering the cards.</p> <pre><code class="language-html">{#each emojis as emoji} <span class="hljs-comment"><!-- $displayItems is a Svelte store containing an array of search results --></span> {#if !displayItems || $displayItems.find((obj) => obj.slug === emoji.slug)} <span class="hljs-tag"><<span class="hljs-name">Card</span> {<span class="hljs-attr">emoji</span>} /></span> {/if} {/each} </code></pre> <p>Not only was I iterating through each of the emojis, but I was also iterating through the entire search result each time. The performance of these two things combined was so unusable I didn’t even commit it to git. For more information on JavaScript’s <code>Array.find()</code>, see <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/find">MDN</a>.</p> <h2 id="attempt-%232%3A-flexsearch" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210608-searching-displaying-large-datasets-svelte/#attempt-%232%3A-flexsearch"><span>Attempt #2: FlexSearch</span></a></h2> <p>I searched around for lightweight, maintained, and performant libraries. I could’ve gotten away with not having those first two since I’m already shipping a ton of data to the client and I’m not going to be proactively maintaining this application anyway, but it’s just a habit.</p> <p>I stumbled upon <a href="https://github.com/nextapps-de/flexsearch">FlexSearch</a>, which checked all of my boxes, even if the documentation is a little confusing. To be honest, I’m not sure if I’m configuring it correctly, but it seems to work fine. I also updated how I was rendering the cards.</p> <pre><code class="language-html"><span class="hljs-comment"><!-- See https://github.com/tctree333/TwemojiExplorer/blob/dcfc640550/src/components/SearchBar.svelte --></span> <span class="hljs-tag"><<span class="hljs-name">script</span>></span><span class="language-javascript"> <span class="hljs-comment">// ...</span> displayItems.<span class="hljs-title function_">set</span>( res.<span class="hljs-title function_">reduce</span>(<span class="hljs-function">(<span class="hljs-params">acc, curr</span>) =></span> { acc[curr.<span class="hljs-property">emoji</span>] = <span class="hljs-literal">true</span> <span class="hljs-keyword">return</span> acc }, {}) ) </span><span class="hljs-tag"></<span class="hljs-name">script</span>></span> {#each emojis as emoji} <span class="hljs-comment"><!-- $displayItems is a Svelte store containing an object of {emoji: boolean} --></span> {#if !$displayItems || $displayItems[emoji.emoji] !== undefined} <span class="hljs-tag"><<span class="hljs-name">Card</span> {<span class="hljs-attr">emoji</span>} /></span> {/if} {/each} </code></pre> <p>Now, I was only iterating through all the emojis once, transforming the data, and accessing an object, which I believe is faster. Still not great, but it was (at least) usable. The biggest issue was when deleting the last character in the search bar, which was super slow. It would take up to 2 whole seconds after hitting the delete key for the character to be deleted, which doesn’t sound like much but feels super slow. This performance hit is because it’s going from rendering a subset of all the cards to rendering them all.</p> <p>To optimize this, I switched to iterating over the search results to reduce processing. Iterating the search results also allowed me to count the number of emojis in each category and hide categories with no emojis that match the search. I modified the search results to return a list of objects representing each category. The object contains a list of emojis. I can then iterate through both lists to render all the emoji cards.</p> <h2 id="improvement-attempt%3A-creating-a-flexsearch-index-for-each-emoji-category" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210608-searching-displaying-large-datasets-svelte/#improvement-attempt%3A-creating-a-flexsearch-index-for-each-emoji-category"><span>Improvement Attempt: Creating a FlexSearch index for each emoji category</span></a></h2> <p>Looking back, this optimization was probably unnecessary since I don’t think the bottleneck is in the search. I’m also searching all the categories anyway, which is not I/O bound, so using <code>Promise.all</code> to run them in “parallel” doesn’t boost performance. However, it did require some data restructuring, which at least reduced client-side transformations.</p> <p>See the commit on <a href="https://github.com/tctree333/TwemojiExplorer/commit/2663a4d">GitHub</a>. For more information on using asynchronous JavaScript functions in sequence or parallel, see this <a href="https://jrsinclair.com/articles/2019/how-to-run-async-js-in-parallel-or-sequential/">blog by James Sinclair</a>.</p> <p>This change didn’t noticeably improve anything, but it didn’t seem to hurt either, so I just left it in.</p> <h2 id="virtualizing-the-not-virtual-dom" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210608-searching-displaying-large-datasets-svelte/#virtualizing-the-not-virtual-dom"><span>Virtualizing the not virtual DOM</span></a></h2> <p>I realize that the bottleneck here was the rendering of the cards, so I thought I could use a virtualization library to avoid rendering cards that aren’t in the viewport.</p> <p>Virtualization works on the idea that if the user can’t see the element, there’s no reason it should be in the DOM. This strategy allows efficient rendering of lists that can have millions of items.</p> <p>However, two things make this tricky for this particular project:</p> <ol> <li>The grid of items is dynamic and changes depending on the search results.</li> <li>I want to scroll on the entire page, not in a box.</li> </ol> <p>I tried out a couple of different Svelte virtualization libraries, but none of them worked the way I wanted. I also tried writing my own using <a href="https://developer.mozilla.org/en-US/docs/Web/API/IntersectionObserver">IntersectionObserver</a>, but it was super glitchy and wack. While virtualization is a powerful technique, I haven’t found the proper implementation for this particular situation.</p> <h2 id="poor-man%E2%80%99s-virtualization%3A-the-%E2%80%9Cshow-more%E2%80%9D-button" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210608-searching-displaying-large-datasets-svelte/#poor-man%E2%80%99s-virtualization%3A-the-%E2%80%9Cshow-more%E2%80%9D-button"><span>Poor Man’s Virtualization: The “Show More” button</span></a></h2> <p>Since I was not able to implement virtualization, I decided to cheat and hide the elements anyway. The primary method for looking for emojis is through the search bar, so scrolling through all the emojis isn’t something you’re going to do, especially if you’re only partially through a search. I decided to limit the number of emojis initially displayed in each category to 23 (3 rows of 8 at max-width, without the Show More button).</p> <p>See the commit on <a href="https://github.com/tctree333/TwemojiExplorer/commit/614f3df39ae4d4daf379f906ead508cc64e54059">GitHub</a>.</p> <p>This solution is a little more aggressive but works great for my situation. With this optimization, the delay between hitting the delete key on the last character and the delete action is much shorter.</p> <pre><code class="language-html"><span class="hljs-tag"><<span class="hljs-name">div</span> <span class="hljs-attr">class</span>=<span class="hljs-string">"grid"</span>></span> <span class="hljs-comment"><!-- Initial emoji display --></span> {#each emojis.slice(0, showDefault) as emoji} <span class="hljs-tag"><<span class="hljs-name">Card</span> {<span class="hljs-attr">emoji</span>} /></span> {/each} <span class="hljs-comment"><!-- Check if there are emojis left --></span> {#if emojis.slice(showDefault).length > 0} {#if showAll} <span class="hljs-comment"><!-- Show them --></span> {#each emojis.slice(showDefault) as emoji} <span class="hljs-tag"><<span class="hljs-name">Card</span> {<span class="hljs-attr">emoji</span>} /></span> {/each} {:else} <span class="hljs-comment"><!-- Display the Show More button --></span> <span class="hljs-tag"><<span class="hljs-name">button</span> <span class="hljs-attr">on:click</span>=<span class="hljs-string">{()</span> =></span> { showAll = true; }}> <span class="hljs-tag"><<span class="hljs-name">p</span>></span>Show All<span class="hljs-tag"></<span class="hljs-name">p</span>></span> <span class="hljs-tag"></<span class="hljs-name">button</span>></span> {/if} {/if} <span class="hljs-tag"></<span class="hljs-name">div</span>></span> </code></pre> <p>This solution is a little more aggressive, but works great for my situation. With this optimization, the delay between hitting the delete key on the last character and it actually being deleted is much shorter.</p> <h2 id="debouncing-the-input" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210608-searching-displaying-large-datasets-svelte/#debouncing-the-input"><span>Debouncing the Input</span></a></h2> <p>Even with all these optimizations, there was still a little bit of a delay. The solution? Debouncing! Debouncing is used to prevent burst events from triggering a function too many times. For example, if you’re listening to mouse move events, moving the mouse can spam a burst of event triggers. Debouncing will reduce the number of function calls by triggering the callback after a delay period of no activity. Bouncing also happens in hardware, where a switch activation can bounce the metal contacts inside, creating multiple signals. If you’re interested in hardware switch debouncing, see <a href="https://www.youtube.com/watch?v=81BgFhm2vz8">Ben Eater’s video</a> on the 555 timer.</p> <p>For my project, I took inspiration from Josh W Comeau’s <a href="https://www.joshwcomeau.com/snippets/javascript/debounce/">debouncing snippet</a>, which uses a timeout to debounce events. Whenever the input box is modified, a timeout is started, which executes the search after 25 milliseconds. If another keypress occurs within those 25 milliseconds, the timeout restarts. I played around with a couple of values, and 25 milliseconds seemed to work fine for both responsiveness and performance. However, you may want to use a longer duration to execute the search when the user completely stops typing, since 25 milliseconds feels like it’s trying to be real-time but is lagging.</p> <p>See the commit on <a href="https://github.com/tctree333/TwemojiExplorer/commit/1eab77d06199d5f6cf6677d44df3b1f8518a30ea">GitHub</a>.</p> <pre><code class="language-html"><span class="hljs-tag"><<span class="hljs-name">script</span>></span><span class="language-javascript"> <span class="hljs-comment">// ...</span> <span class="hljs-keyword">let</span> debounceTimeout <span class="hljs-keyword">let</span> searchstring <span class="hljs-keyword">function</span> <span class="hljs-title function_">handleInput</span>(<span class="hljs-params"></span>) { <span class="hljs-built_in">clearTimeout</span>(debounceTimeout) debounceTimeout = <span class="hljs-built_in">setTimeout</span>(<span class="hljs-function">() =></span> { searchstring ? <span class="hljs-title function_">search</span>(searchstring, setResult) : <span class="hljs-title function_">setResult</span>(<span class="hljs-literal">null</span>) }, <span class="hljs-number">25</span>) } </span><span class="hljs-tag"></<span class="hljs-name">script</span>></span> <span class="hljs-comment"><!-- ... --></span> <span class="hljs-tag"><<span class="hljs-name">input</span> <span class="hljs-attr">type</span>=<span class="hljs-string">"text"</span> <span class="hljs-attr">bind:value</span>=<span class="hljs-string">"{searchstring}"</span> <span class="hljs-attr">on:input</span>=<span class="hljs-string">"{handleInput}"</span> <span class="hljs-attr">placeholder</span>=<span class="hljs-string">"Search Emojis!"</span> /></span> </code></pre> <h2 id="conclusion" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210608-searching-displaying-large-datasets-svelte/#conclusion"><span>Conclusion</span></a></h2> <p>Searching and rendering large datasets can be difficult, but hopefully, this article gave you some ideas on improving the user experience in your projects. Building this project was a lot of fun, and I hope it’ll be a valuable tool in my toolbox. I actually used the website itself during the building of the website :0 <img class="emoji" draggable="false" alt="🤯" src="https://tomichen.com/twemoji/72x72/1f92f.png" /></p>Rate-Limiting Your Serverless Endpoints Without A Database2021-06-01T00:00:00Z2022-04-18T21:47:31Zhttps://tomichen.com/blog/posts/20210602-rate-limiting-serverless-without-database/Serverless is all the rage in modern web development, especially serverless functions—threads that spin up for each request. This is great for keeping costs low, but it also means that each request starts fresh, so any rate-limiting information needs to be persisted to a database. Right? Perhaps not...<p>Serverless is all the rage in modern web development, since it’s highly scalable, low cost, and distributed. Frontend developers are also empowered to create fully-featured websites without needing to manage backend infrastructure. One component is serverless functions—threads that spin up when a request is made and turn off when they’re not needed. This is great for costs since you don’t need to pay for something that’s not running, but it also means that memory is cleared and each request starts fresh, so any rate-limiting information needs to be persisted to a database. Right?</p> <p><strong>NOTE:</strong> The methods presented below are not intended for any actual use for rate-limiting infrastructure and should be viewed as interesting and educational. For more information, check out the “Conclusion” section at the end of this post.</p> <h2 id="background" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210602-rate-limiting-serverless-without-database/#background"><span>Background</span></a></h2> <p>Over the weekend, I built <a href="https://iscancelled.com/"><code>iscancelled.com</code></a>, a fun website where you can cancel various things. The URL would end up being <code>*.iscancelled.com</code>. For grammatical accuracy, I also got <code>*.arecancelled.com</code>, just in case plurals are needed. To store the number of cancels each item has, I used <a href="https://upstash.com/">Upstash</a>, a hosted Redis service. Since the free tier only has 10,000 requests per day, I thought it might be interesting to try rate-limiting certain endpoints to prevent excessive spam. For more information about this project, see the <a href="https://github.com/tctree333/cancel-culture">GitHub repo</a>.</p> <p>Since the entire point was to prevent excessive operations, storing rate-limiting information in the database defeated the whole point. Since I thought each endpoint would spin up on every request and clean up after, I thought I was out of luck.</p> <p>That was when I stumbled upon this <a href="https://github.com/vercel/next.js/tree/canary/examples/api-routes-rate-limit">API route rate-limiting example</a> in the Next.js repo. This surprised me. There was no external database, but it still worked. What was going on?</p> <p>Note: I used Vercel for this project, so the following information was only tested on Vercel, though I suspect it may work on other platforms too.</p> <h2 id="exploring-the-code" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210602-rate-limiting-serverless-without-database/#exploring-the-code"><span>Exploring the Code</span></a></h2> <p>The code for rate-limiting is in <a href="https://github.com/vercel/next.js/blob/canary/examples/api-routes-rate-limit/utils/rate-limit.js"><code>utils/rate-limit.js</code></a>. Here, they use a least-recently-used (LRU) cache to store rate-limiting information. An LRU cache is basically a simple key-value store that evicts the least-used key once it hits a max size. When a key is read or set, the “recency” is updated. This implementation also uses a <code>maxAge</code>, which is checked when the key is read and deleted if the key is too old.</p> <pre><code class="language-javascript"><span class="hljs-keyword">const</span> <span class="hljs-variable constant_">LRU</span> = <span class="hljs-built_in">require</span>(<span class="hljs-string">'lru-cache'</span>) <span class="hljs-keyword">const</span> <span class="hljs-title function_">rateLimit</span> = (<span class="hljs-params">options</span>) => { <span class="hljs-keyword">const</span> tokenCache = <span class="hljs-keyword">new</span> <span class="hljs-title function_">LRU</span>({ <span class="hljs-attr">max</span>: <span class="hljs-built_in">parseInt</span>(options.<span class="hljs-property">uniqueTokenPerInterval</span> || <span class="hljs-number">500</span>, <span class="hljs-number">10</span>), <span class="hljs-attr">maxAge</span>: <span class="hljs-built_in">parseInt</span>(options.<span class="hljs-property">interval</span> || <span class="hljs-number">60000</span>, <span class="hljs-number">10</span>) }) <span class="hljs-keyword">return</span> { <span class="hljs-attr">check</span>: <span class="hljs-function">(<span class="hljs-params">res, limit, token</span>) =></span> <span class="hljs-keyword">new</span> <span class="hljs-title class_">Promise</span>(<span class="hljs-function">(<span class="hljs-params">resolve, reject</span>) =></span> { <span class="hljs-keyword">const</span> tokenCount = tokenCache.<span class="hljs-title function_">get</span>(token) || [<span class="hljs-number">0</span>] <span class="hljs-keyword">if</span> (tokenCount[<span class="hljs-number">0</span>] === <span class="hljs-number">0</span>) { tokenCache.<span class="hljs-title function_">set</span>(token, tokenCount) } tokenCount[<span class="hljs-number">0</span>] += <span class="hljs-number">1</span> <span class="hljs-keyword">const</span> currentUsage = tokenCount[<span class="hljs-number">0</span>] <span class="hljs-keyword">const</span> isRateLimited = currentUsage >= <span class="hljs-built_in">parseInt</span>(limit, <span class="hljs-number">10</span>) res.<span class="hljs-title function_">setHeader</span>(<span class="hljs-string">'X-RateLimit-Limit'</span>, limit) res.<span class="hljs-title function_">setHeader</span>( <span class="hljs-string">'X-RateLimit-Remaining'</span>, isRateLimited ? <span class="hljs-number">0</span> : limit - currentUsage ) <span class="hljs-keyword">return</span> isRateLimited ? <span class="hljs-title function_">reject</span>() : <span class="hljs-title function_">resolve</span>() }) } } <span class="hljs-keyword">export</span> <span class="hljs-keyword">default</span> rateLimit </code></pre> <p>Here, the <code>check</code> function first checks if the cache contains the token (which identifies a user). If it doesn’t, it defaults to an array containing a single zero. Next, it checks if the value in the array is a zero. This would mean that the token doesn’t exist, since if it did, there should be at least a value of 1 stored in the cache. If the token didn’t exist, it now sets the cache to the array referenced by <code>tokenCount</code>. Finally, it increments the first element in <code>tokenCount</code> by one.</p> <p>You may be wondering why the count is being wrapped in an array, especially when only one element is being used. I was wondering the same, so I ✨ thought about it ✨. Initially, I thought it was a way to avoid updating the “recency” of the cache, but then I realized that the “LRU” part of the LRU cache wasn’t even being used! Ideally, no keys will be evicted from maxing out capacity, since that would mean rate limits being cut short.</p> <p>After trying it out without the array wrapping, I discovered that my initial reaction wasn’t too far off. However, it was to prevent the <code>maxAge</code> timer from being reset, not the recency of the key. This works since arrays in Javascript are passed by reference, not by value, so when the inner number is updated with <code>tokenCount[0] += 1</code>, the value in the cache is also updated, without triggering a <code>maxAge</code> reset that you’d get with <code>cache.set()</code>. Primitives in Javascript, such as numbers, are passed by value, so this doesn’t work without the array.</p> <p>You can explore this yourself!</p> <pre><code class="language-javascript"><span class="hljs-comment">// x and y are primitives</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">swap</span>(<span class="hljs-params">x, y</span>) { <span class="hljs-keyword">let</span> tmp = x x = y y = tmp } <span class="hljs-keyword">let</span> x = <span class="hljs-number">1</span> <span class="hljs-keyword">let</span> y = <span class="hljs-number">2</span> <span class="hljs-title function_">swap</span>(x, y) <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(x, y) <span class="hljs-comment">// 1 2</span> <span class="hljs-comment">// x and y are arrays</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">swap</span>(<span class="hljs-params">x, y</span>) { <span class="hljs-keyword">let</span> tmp = x[<span class="hljs-number">0</span>] x[<span class="hljs-number">0</span>] = y[<span class="hljs-number">0</span>] y[<span class="hljs-number">0</span>] = tmp } <span class="hljs-keyword">let</span> x = [<span class="hljs-number">1</span>] <span class="hljs-keyword">let</span> y = [<span class="hljs-number">2</span>] <span class="hljs-title function_">swap</span>(x, y) <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(x, y) <span class="hljs-comment">// [ 2 ] [ 1 ]</span> </code></pre> <h2 id="exploring-the-platform" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210602-rate-limiting-serverless-without-database/#exploring-the-platform"><span>Exploring the Platform</span></a></h2> <p>Now that we understand what’s going on in the code, we can move on to the memory persistence after deployment. I wrote up a quick SvelteKit endpoint to see what was going on.</p> <pre><code class="language-typescript"><span class="hljs-keyword">import</span> <span class="hljs-keyword">type</span> { <span class="hljs-title class_">RequestHandler</span> } <span class="hljs-keyword">from</span> <span class="hljs-string">'@sveltejs/kit'</span> <span class="hljs-keyword">const</span> buffer = [] <span class="hljs-keyword">export</span> <span class="hljs-keyword">const</span> <span class="hljs-attr">get</span>: <span class="hljs-title class_">RequestHandler</span> = <span class="hljs-keyword">async</span> () => { buffer.<span class="hljs-title function_">push</span>({ <span class="hljs-attr">time</span>: <span class="hljs-keyword">new</span> <span class="hljs-title class_">Date</span>().<span class="hljs-title function_">toString</span>() }) <span class="hljs-keyword">return</span> { <span class="hljs-attr">body</span>: buffer } } </code></pre> <p>This initializes an array, appends the current time to it every request, and returns the array. After deploying, we can reload the page and watch the array grow bigger and bigger. However, after leaving it for about 15 minutes or so, the array would clear itself and start over. Interesting!</p> <p>This hinted to me that this memory persistence might have to do with cold starts and warm endpoints.</p> <p>When a serverless function endpoint is first requested, there is a short delay of a few hundred milliseconds to boot up the function. This is referred to as the “cold-start” time. Since having this delay impacts performance significantly, platforms do not destroy functions immediately. Instead, they keep them “warm”, so if another request comes within ~5-25 minutes, the cold-start time is not an issue. Memory and state are shared if a function is warm, so that’s why this persistence occurs. (<a href="https://workers.cloudflare.com/">Cloudflare Workers</a> does things a little differently, booting up the worker during the TLS handshake which lets them boast a 0ms cold-start time.)</p> <h2 id="conclusion" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210602-rate-limiting-serverless-without-database/#conclusion"><span>Conclusion</span></a></h2> <p>We’ve seen how it is possible to build a simple rate limit in a serverless function, without the need for an external persisted database. Since this method does rely on the function staying alive, it’s not 100% foolproof, especially for longer durations. Keeping the <code>maxAge</code> below 5 minutes or so would probably be a better idea, since if the function is killed for inactivity, there weren’t that many requests anyway. There might also be multiple functions active during periods of higher load, so it might not work there. For a simple, low-stakes, low-traffic site, this works fine, but you definitely should reconsider for critical or higher-traffic use cases.</p> <p><strong>UPDATE:</strong> It has been brought to my attention that if someone is flooding your endpoint with requests, new functions are spun up very quickly (which is exactly what these are designed to do). As a result, any volume of more than a couple of requests per second will defeat this rate-limiting method. Please consider this as something interesting and educational, NOT something you should be doing in production applications.</p> <p>I’ve also only tried this on Vercel (which I believe uses AWS Lambda under the hood), but I think it should work on other platforms too (at least the ones that use AWS). Try it and see!</p> <p>Finally, here is <a href="https://github.com/tctree333/cancel-culture/blob/main/src/lib/utils/limit.ts">the code</a> for rate-limiting used on <code>iscancelled.com</code>. I’m also identifying users by IP, exposed through <a href="https://vercel.com/docs/edge-network/headers#x-real-ip">Vercel’s <code>x-real-ip</code> header</a>.</p>The Creation of CapitalFish: An incremental game built with SvelteKit2021-04-11T00:00:00Z2022-04-18T21:47:31Zhttps://tomichen.com/blog/posts/20210412-the-creation-of-capitalfish/Over spring break, I decided to try out SvelteKit, which recently released its public beta. I thought it would be a great opportunity to build something fun. I already had the idea of creating an incremental game similar to Cookie Clicker, so what better way to learn a new thing than actually using that new thing? Join me as I rant about JS frameworks and explain how I created the game!<p>Over spring break, I decided to try out <a href="https://kit.svelte.dev/">SvelteKit</a>, a Svelte framework that provides client-side routing, server-side rendering, and all that fancy good stuff. SvelteKit recently released their public beta, so I thought it would be a great opportunity to build something fun. You can learn more about SvelteKit in their <a href="https://svelte.dev/blog/sveltekit-beta">announcement blog post</a>. I already had the idea of creating an incremental game similar to <a href="https://en.wikipedia.org/wiki/Cookie_Clicker">Cookie Clicker</a>, so what better way to learn a new thing than actually using that new thing?</p> <p>You can play the game at <a href="https://capitalfish.tomichen.com/"></a><a href="https://capitalfish.tomichen.com/">https://capitalfish.tomichen.com/</a>.</p> <h2 id="ranting-about-javascript-%F0%9F%A4%AC" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210412-the-creation-of-capitalfish/#ranting-about-javascript-%F0%9F%A4%AC"><span>Ranting about JavaScript 🤬</span></a></h2> <p>I’ve played around with <a href="https://nextjs.org/">Next.js</a> before, and I was impressed at how easy it made things. However, the biggest drawback of React and Next for me is how much JavaScript it ships. The Next tutorial project has a whole 47.5 kB of JS, which is kind of a lot considering it needs almost none of that. Compared to the <a href="https://capitalfish.tomichen.com/">game this post is about</a> at 18 kB, I’ll probably be more likely to use Svelte.</p> <p>I am aware that this isn’t a great comparison and there’s client-side routing and all that, but still. A 30 kB difference between a static blog and a literal game? Also, I don’t have that much experience in React/Next, so take my opinions with a grain of salt. Come to think of it, I have less than a week of experience with Svelte, so…</p> <p>While I could probably cut more if I rewrote the entire thing in plain vanilla JavaScript, I’m pretty sure that would be much less fun and take a lot more time. While it may be worth it for simple things (in which case, I prefer <a href="https://www.11ty.dev/">Eleventy</a>), I think Svelte has a good balance between developer ergonomics and performance.</p> <p>But I did not come here to rant about my obsession with file sizes while also simultaneously being too lazy to optimize my images. I am here to write about building a game.</p> <h2 id="building-the-game" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210412-the-creation-of-capitalfish/#building-the-game"><span>Building the game</span></a></h2> <p>The idea for this game came a couple of weeks before, where joking around with some friends led me to impulsively buy a domain with no clue what to do with it. After some deep introspection and consideration, I decided the best use was to build a cookie-clicker-esque incremental game. Since this would require lots of JavaScript, I decided to try out a JavaScript framework, choosing Svelte/SvelteKit for the reasons above.</p> <p>The bulk of the work was done in one day (~11 hours) with a couple of hours the night before to set things up/read the docs. The <a href="https://kit.svelte.dev/docs">SvelteKit docs</a> were very informational, and for Svelte itself, I mainly referenced <a href="https://svelte.dev/tutorial">their tutorial</a>, which had everything I needed. When researching different frameworks, I did the first few lessons, which was enough for me to get a general idea of what was going on and get started.</p> <p>After showing my friends, they seemed to enjoy it, so I slowly improved it over the next few days. Since the original contained some very mild personal information, I re-themed it to release it publicly, which is the version linked in this post. The code is also <a href="https://github.com/tctree333/capital-fish">available on GitHub</a> if you’re into that. Besides the images and text, the mechanics are the same. It’s also pretty simple to change the game theme and tweak values if you want, so feel free to do that! (though I will warn that my code is kinda bad even though I tried my best slacking off is inevitable and I could’ve split it into more components and I could’ve put styles in a different place and I could’ve done this and I could’ve done that and whatever at least it works lmao 😋 )</p> <p>Overall, building this game was a lot of fun. I learned a lot about Svelte, SvelteKit, and the amount of flexibility in the browser. While absolute positioning and changing top/left properties is not very performant (I think?) it works fine(ish)! There are still lots of bugs and issues and tweaking to gameplay that could happen, but I’m not going to worry about that. 😌</p>Converting BirdID's web API to FastAPI2021-01-19T00:00:00Z2022-04-18T21:47:31Zhttps://tomichen.com/blog/posts/20210119-converting-birdid-web-api/Over the weekend, I worked on converting BirdID's web API to FastAPI instead of Flask. While I did not notice many performance issues or other issues with Flask itself, I was noticing some errors relating to using asyncio.run to connect up Flask (which is synchronous) and a lot of the associated BirdID functions (since discord.py is asynchronous).<p>Over the weekend, I worked on converting BirdID’s web API to <a href="https://fastapi.tiangolo.com/">FastAPI</a> instead of <a href="https://flask.palletsprojects.com/en/1.1.x/">Flask</a>. While I did not notice too many performance issues or other issues with Flask itself, I was noticing some errors relating to using <code>asyncio.run</code> to connect up Flask (which is synchronous) and a lot of the associated BirdID functions (since <code>discord.py</code> is asynchronous).</p> <p>Also, the web API has not been worked on as much as the rest of BirdID. Functions are shared between the two (as they should be <img class="emoji" draggable="false" alt="😌" src="https://tomichen.com/twemoji/72x72/1f60c.png" />), so updates to the bot core improve the API. However, there are improvements to the web API itself that I want to make.</p> <h2 id="why-fastapi%3F" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210119-converting-birdid-web-api/#why-fastapi%3F"><span>Why FastAPI?</span></a></h2> <p>For this migration, the key things I wanted the new framework to have were a structure similar to Flask and native <code>async/await</code> support. The two main frameworks that fit this description were <a href="https://sanicframework.org/">Sanic</a> and <a href="https://fastapi.tiangolo.com/">FastAPI</a>. While Sanic was particularly appealing because it was structured so similarly to Flask, I eventually went with FastAPI because it had more features, such as data validation/typing. While the automatic docs generation is cool (<a href="https://orni-api.sciolyid.org/docs">check it out</a>), it wasn’t necessary for this project.</p> <p>As for performance, FastAPI’s docs say that Sanic is more comparable to Starlette since FastAPI is a higher level and adds more things on top (such as the data validation). As a result, Sanic does technically have better performance than FastAPI.</p> <p>After using FastAPI, the most convenient part for me is the Starlette/FastAPI <a href="https://www.starlette.io/middleware/">middleware</a> for CORS, signed session cookies with <a href="https://itsdangerous.palletsprojects.com/en/1.1.x/">itsdangerous</a>, <a href="https://sentry.io/welcome/">Sentry</a>, and more. I’m not sure if they would work with Sanic, but the Sanic documentation did not mention anything about ASGI middleware compatibility. I would not have known these things existed anyway. While I would probably have just implemented these things myself (poorly), it was a lot easier to use the built-in stuff.</p> <h2 id="the-actual-rewrite" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210119-converting-birdid-web-api/#the-actual-rewrite"><span>The actual rewrite</span></a></h2> <p>The actual process (<a href="https://github.com/tctree333/Bird-ID/pull/135">GH#135</a>) was not as bad as I initially expected. I worked on it through the 3 day weekend, and the bulk of the work was finished on the first day. I had a Science Olympiad competition/meetings on Saturday/Sunday, so this didn’t take that long in total.</p> <p>I spent a little bit of time during the week before to skim through the docs when I was choosing which framework to use. On the first day, I went one file at a time, starting from the configuration and then converting the routes. Since I thought this was going to be a lot more difficult, I went from what I thought were the easiest files to the hardest ones. I probably overestimated the amount of code there was—it’s only like 7 files.</p> <p>Anyway, it was mainly just renaming stuff to what FastAPI uses, adding the middleware, and using native <code>await</code>. For example, Flask’s <a href="https://flask.palletsprojects.com/en/1.1.x/blueprints/"><code>blueprints</code></a> are called <a href="https://fastapi.tiangolo.com/tutorial/bigger-applications/"><code>routers</code></a> in FastAPI, Flask’s <a href="https://flask.palletsprojects.com/en/1.1.x/quickstart/#accessing-request-data">funky global objects</a> are now passed in as a <a href="https://fastapi.tiangolo.com/advanced/using-request-directly/"><code>request</code> parameter</a>, and CORS, gzip, and sessions are all done with <a href="https://www.starlette.io/middleware/">ASGI middleware</a>.</p> <h2 id="other-updates" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210119-converting-birdid-web-api/#other-updates"><span>Other updates</span></a></h2> <p>Since I was already working on the API, I also fixed some other existing issues, which were mainly with the OAuth2 Discord authentication. We use <a href="https://authlib.org/">Authlib</a> for the OAuth2 client flow to get a user’s Discord ID so scores can sync with the bot. Since May 2020, we could only use version <code>0.14.1</code> because, for some reason, upgrading would break the login flow. During the rewrite, I switched to using the Starlette/FastAPI integration for Authlib, but there were still issues.</p> <p>It turns out that Authlib was storing the CSRF token inside the signed session cookie. I had <a href="https://web.dev/samesite-cookies-explained/"><code>Same-Site</code></a> on the cookie set to <code>Strict</code>, so the cookie wasn’t being sent when Discord redirected the user back to our application. Setting <code>Same-Site</code> to <code>Lax</code> fixed the issue. I’m not entirely clear on the security implications of this, but the worst thing an attacker could do is log someone out, which happens automatically after a few days anyway. Logging in isn’t that difficult either especially if you’re already logged into Discord. I would also prefer having separate signed cookies for the login flow and application session id.</p> <h2 id="performance" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210119-converting-birdid-web-api/#performance"><span>Performance</span></a></h2> <p>I’m not sure if this is faster than Flask, especially since a lot of the functions have already been improved with better caching performance. I also don’t know if there will be fewer errors in the long term since the new version has only been up for a day or so.</p> <p>According to the site analytics (we’re using <a href="https://plausible.io/">Plausible</a> with custom events), around 18 people have used it so far with about 2.1k bird checks (~26 hours after deploy), and Sentry hasn’t reported anything major.</p> <p>Before deploying the new version, I tried using <a href="https://github.com/rakyll/hey">hey</a> to send some load and see if FastAPI performs better. I know this probably isn’t the correct tool for the job, but hey <img class="emoji" draggable="false" alt="😉" src="https://tomichen.com/twemoji/72x72/1f609.png" />, it “works”. However, there wasn’t too much of a difference, especially when deployed onto the server environment. I think this could either be because I’m using the wrong tool, or because there isn’t <em>that</em> much blocking I/O. The main blocking I/O comes from fetching media from the Macaulay Library and we heavily cache it, which is shared between the bot and web processes.</p> <h2 id="conclusion" tabindex="-1"><a class="header-anchor" href="https://tomichen.com/blog/posts/20210119-converting-birdid-web-api/#conclusion"><span>Conclusion</span></a></h2> <p>Overall, this was pretty straightforward. FastAPI is pretty easy to use, so I’ll definitely consider it for future projects. Next up: a frontend refresh—I want to stop using jQuery (sidenote: jQuery isn’t that much younger than me, like whaaa?) and use vanilla JS instead. I also want to switch to 11ty/Tailwind. After that, I’ll look into switching <a href="https://github.com/tctree333/SciOly-ID/tree/master/sciolyid/web"><code>sciolyid</code>’s API</a> to FastAPI and add the practice functionality to <code>sciolyid</code>, then finally merge the <a href="https://orni.sciolyid.org/">BirdID</a> and <a href="https://sciolyid.org/">SciOly-ID</a> sites. Fun.</p>First Blog Post2020-12-21T00:00:00Z2022-06-03T02:52:09Zhttps://tomichen.com/blog/posts/20201221-first-post/Hello! Welcome to my blog! I've just spent the last few days adding this blog to the site—making and styling templates, updating fonts, and ensuring images are responsive. While a personal site is never truly finished, it's now in a state where I feel comfortable publishing.<p>Hello! Welcome to my blog!</p> <p>I’ve just spent the last few days adding this blog to the site—making and styling templates, updating fonts, and making images responsive. While a personal site is never <em>truly</em> finished, it’s now in a state where I feel comfortable publishing.</p> <p>This first post is primarily for me to put down my goals for this blog and the kind of content I want to publish. My main goal is to document my projects and how I solved problems that arose. Hopefully, this will help someone (maybe even myself) tackle these issues in the future. I also hope that writing these things down can help me reflect on the project and improve my skills.</p> <p>I may also post other content like tutorials or documentation if, and only if, I feel it would be helpful for either future me or someone else. I don’t want to repeat things that other people have already written about unless I believe I can do a much better job.</p> <p>For example, when I was trying to make <a href="https://tomichen.com/projects/scriscord/">Scriscord</a>, I was having a difficult time figuring out what I needed to do to get the darn thing to work. I could barely find any official documentation and only had a <a href="https://github.com/tctree333/scriscord#thanksother-resources">couple of Scratch discussion posts</a> to go off. I eventually resorted to reading through the Scratch source code as well as another extension I found. After finally getting it to work, I wanted to document some of what I did. If I had this blog at the time, I would probably have been more motivated to do so, but unfortunately, this knowledge has now been lost to time. How <img class="emoji" draggable="false" alt="😔" src="https://tomichen.com/twemoji/72x72/1f614.png" />.</p>