
Lately, I've been playing a lot of [Among Us](https://en.wikipedia.org/wiki/Among_Us) with my friends. It's a refreshing change of pace from the competitive FPS titles that we usually play. However, we've managed to inject a 'healthy' dose of competitiveness into our games in the form of a league that I built some software for. 

We recently crossed the 1000 games mark and players have been making feature requests and also submitting PRs! It has been open sourced at [healeycodes/among-us-friends](https://github.com/healeycodes/among-us-friends).

The league website is a React/Node/Express application that calculates a player's [Elo rating](https://en.wikipedia.org/wiki/Elo_rating_system) — a relative measure of one player's skill against another — as well as other performance statistics like crew/imposter win rate. It ranks players and graphs their recent performance on the home page. 

![The home page - a list of players, their win/loss rate, and Elo charts](preview.png)

When a player goes to their auto-generated player page, they can see a history of their games and who played on each side, the amount of Elo they won or lost each game, and an Elo history graph for the whole season. They also have a trophy section. Some of the trophies include: win streaks (5, 10, 15), being over 1300 Elo at the end of the season, best crew, best imposter, and more. As well as dynamic trophies, there are also manual trophies like Most Improved (which we vote on) and being a code contributor to the project.

![A player's trophy section](preview-trophies.png)

I manually track each game we play in a Google Sheets spreadsheet. This data is brought into the application via the Google Sheets API v4. To set this up, I grabbed the spreadsheets ID (which can be copied straight from the URL bar) and created a restricted API key via Google Console.

The data arrives in a series of rows with each row representing one game. An Among Us game can be split up into three parts of data. 

- The crew (7 or 8 players)
- The imposters (2 players)
- The winner (crew or imposters)
- The map short name (skeld, polus, mira)

Since a spreadsheet cannot be efficiently queried we bring in all the rows on every request (Sheets API usage is unlimited and free). If performance is ever an issue, caching can be added.

A player object is built for each person in the league and the following information is calculated.

- Crew wins/losses
- Imposter wins/losses
- Current Elo
- Elo history
- Their game history

A season object is built for each season that has been played (including the current season). This has information about trophies and other general stats such as map win/loss data.

## Elo ratings

For Elo calculations, I use the npm package `elo-rating`. In the Elo rating methodology, the _K-factor_ is the maximum possible adjustment for a game. I have based the league's system off the International Chess Federation's rules. Before a player reaches 30 games, their rating is more volatile. After 30 games, their K-factor drops from 40 to 20.

Elo was designed for games with two players. I was initially unsure how to solve the problem of Among Us having different team sizes. Other people have created rating systems for Among Us that separate a measure of a player's skill into crew/imposter. However, I made a product decision to reduce a player's performance down to a single number. This number is the average of a player's hidden crew Elo and hidden imposter Elo.

We alter our game settings (e.g. crew vision, kill cooldown) over time keep the game fun for impostors (so, making it harder for crew). By tracking map statistics, we have found data that goes against our commonly held beliefs. We thought The Skeld was easier to win as imposter than on Polus but it's actually the inverse (by ~10%).

Elo systems in video games are not tuned to be mathematically perfect but to reward and encourage players — a mixture of fun and correctness. Read more about this topic in [Analyzing Starcraft 2's Ranking System](http://sirlingames.squarespace.com/blog/2010/7/24/analyzing-starcraft-2s-ranking-system.html) by Sirlin.

We made the Elo system _less fair_ for players by rewarding those who play more with a tiny amount of Elo. Players that don't play for a long time in a season will have their Elo slowly decay over time. The players voted for these rules.

```javascript
const EloRating = require("elo-rating");

function EloChange(games) {
  // Based on https://ratings.fide.com/calculator_rtd.phtml
  // K = 40 for a player with less than 30 games
  // K = 20 afterwards
  const K = games > 30 ? 20 : 40;
  // From playerA's perspective, how much can be won or lost
  return function (playerA, playerB) {
    return [
      EloRating.calculate(playerA, playerB, true, K).playerRating - playerA,
      EloRating.calculate(playerA, playerB, false, K).playerRating - playerA
    ];
  };
}

// The tests for this look like:
// For new players
let eloChange = EloChange(0);
expect(eloChange(1200, 1400)[0]).toStrictEqual(30);
expect(eloChange(1200, 1400)[1]).toStrictEqual(-9);
// For everyone else
eloChange = EloChange(31);
expect(eloChange(1400, 1200)[0]).toStrictEqual(4);
expect(eloChange(1400, 1200)[1]).toStrictEqual(-15);
```

The frontend styling is mostly [water.css](https://watercss.kognise.dev/) with some in-line tweaking. The graphs are from the package [data-ui](https://github.com/williaster/data-ui) which has been depreciated in favor of [visx](https://github.com/airbnb/visx) — however, I found the data-ui examples easier to pick up and plug in.

The backend of the application is tested using Jest (w/ SuperTest for mocking requests to the server). The statistics functions are unit tested. All the tests are ran on commit/PR using GitHub Actions with the default Node YAML file that runs the following command against Node 14.x.

```yaml
    - run: cd back-end && yarn install && yarn test
```

## React

Initially, this application was hosted on Glitch and the pages were vanilla JavaScript. Over time, the league features grew and grew until it was taking a long time to add new functionality. It was also getting harder to onboard new contributors. I chose React because a few of us were familiar with it. I added react-router to handle the different pages (Ranking/Stats/About/Player).

I used Create React App to initialize the front-end and tried not to tweak it too much. With this project, my focus has been on the customers (e.g. my friends) over code quality. This is a refreshing change compared to the rest of my personal projects, where my focus is normally on clean code/testing/great patterns.

All the data comes in via a single `fetch` call to a serverless endpoint. It can take a few seconds to load but then there are no more API calls required. Clicking between the other pages is instant as the entire league data is stored in memory 🏃💨.

## Deployment

I refactored the Express application from Glitch to use [serverless-http](https://github.com/dougmoscrop/serverless-http) and [netlify-lambda](https://github.com/netlify/netlify-lambda). Now, it's a Jamstack app that deploys to Netlify on commits to the `main` branch. We also take advantage of the PR preview deployments.

When I was refactoring Glitch → Netlify I used these resources: [Netlify Functions + Express](https://github.com/netlify-labs/netlify-functions-express), [How to run Express.js apps with Netlify Functions](https://www.netlify.com/blog/2018/09/13/how-to-run-express.js-apps-with-netlify-functions/). One thing that tripped me up was fixing the API calls from the front-end to prepend `.netlify/functions/app/` to the routes.

## The Future

I have onboarded a few people to be able to use the Google Sheets system to add scores but it's tricky and there is no safety or restrictions on entering data. I have been looking at what kind of admin interfaces I could plug in while still using Google Sheets as a data source.


Making a casual game more competitive and arguing with my friends about Elo systems.

Writing Software for an Among Us League


In JavaScript, you can create random numbers using `Math.random()`. But what if we wanted to create our own random values in the browser without this function?

The [ECMAScript Language Specification](https://tc39.es/ecma262/#sec-math.random) defines the requirements of `Math.random()`:

> Returns a Number value with positive sign, greater than or equal to 0 but less than 1, chosen randomly or pseudo randomly with approximately uniform distribution over that range, using an implementation-dependent algorithm or strategy. This function takes no arguments.

> Each Math.random function created for distinct realms must produce a distinct sequence of values from successive calls.

## Number Generation

Here's an example of a number generator. It uses a [closure](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Closures) to maintain internal state and creates a sequence of numbers based off an initial seed value. Here the seed is fixed and is always initialized to `0`.

```javascript
Math.random = (function () {
  let seed = 0
  return function () {
    seed += 1
    return seed
  }
})()

// We can iterate through the sequence
Math.random() // 1
Math.random() // 2
Math.random() // 3
```

A pseudorandom number generator (PRNG) works in a similar manner. A PRNG maintains an internal state and applies math to that state every time a new random number is requested. The seed can be manual or automatic. In the [Go programming language](https://golang.org/pkg/math/rand/#New), you must seed `math/rand` yourself. In the browser, `Math.random` requests random data under the hood from the operating system (OS) to use as a seed.

PRNGs are deterministic. The same seed will always produce the same sequence of numbers. Often, a deterministic outcome is preferred. For example, to generate the same random events on all clients without them having to talk over a network. Or for reproducible performance benchmarks.

A hash function can be used to create a PRNG. In [spinning-balls](https://github.com/v8/v8/blob/4b9b23521e6fd42373ebbcb20ebe03bf445494f9/benchmarks/spinning-balls/v.js), one of Chrome's benchmarks, we can see an example of this:

```javascript
// v8/benchmarks/spinning-balls/v.js

// To make the benchmark results predictable, we replace Math.random
// with a 100% deterministic alternative.
Math.random = (function () {
  var seed = 49734321
  return function () {
    // Robert Jenkins' 32 bit integer hash function.
    seed = seed & 0xffffffff
    seed = (seed + 0x7ed55d16 + (seed << 12)) & 0xffffffff
    seed = (seed ^ 0xc761c23c ^ (seed >>> 19)) & 0xffffffff
    seed = (seed + 0x165667b1 + (seed << 5)) & 0xffffffff
    seed = ((seed + 0xd3a2646c) ^ (seed << 9)) & 0xffffffff
    seed = (seed + 0xfd7046c5 + (seed << 3)) & 0xffffffff
    seed = (seed ^ 0xb55a4f09 ^ (seed >>> 16)) & 0xffffffff
    return (seed & 0xfffffff) / 0x10000000
  }
})()
```

Like our number generator, it alters its internal state while calculating the next random number. This state-change allows the next call to produce a different number.

## More on Pseudorandom Number Generators

One of the oldest and most well known types of PRNG is the [linear congruential generator](https://en.wikipedia.org/wiki/Linear_congruential_generator) (LCG). Which, despite its somewhat scary name, does not require many lines of code.

@bryc provides an example and [a warning](https://github.com/bryc/code/blob/master/jshash/PRNGs.md#lcg-lehmer-rng):

> Commonly called a Linear congruential generator (LCG), but in this case, more correctly called a Multiplicative congruential generator (MCG) or Lehmer RNG. It has a state and period of 2^31-1. It's blazingly fast in JavaScript (likely the fastest), but its quality is quite poor.

```javascript
function LCG(a) {
  return function () {
    a = Math.imul(48271, a) | 0 % 2147483647
    return (a & 2147483647) / 2147483648
  }
}
```

(This is the first time I've come across `Math.imul()` — which provides [C-like 32-bit multiplication](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/imul#Description) of the two parameters.)

What does @bryc's comment, "its quality is quite poor" mean in this context? Well, given certain even seeds, this algorithm has a pattern when the final step (the division) is removed.

```javascript
// https://gist.github.com/blixt/f17b47c62508be59987b#gistcomment-2792771

// @bryc:
// "Looking at the output without the division, and in hexadecimal, the
// first bits are always the same. This shows a clear pattern in the
// first 8 bits of the output: 1000 000, and it happens each time,
// infinitely. This is mostly caused by using an even seed."
const LCG = (s) => (_) => (s = Math.imul(48271, s) >>> 0)
const nxt = LCG(3816034944)
for (let i = 0; i < 9; i++) {
  console.log(nxt().toString(16))
}

/* Outputs:
4b6c5580 <-- notice the last two digits
b04dc280 <--
9645a580
16717280
d974f580
5c9f2280
9a3a4580
f196d280
b5d59580 */
```

There are [many](https://en.wikipedia.org/wiki/Diehard_tests) ways to test the quality of randomness. Some of the methodology and results of these tests can be understood by a layperson. One of the [Diehard battery of tests](https://en.wikipedia.org/wiki/Diehard_tests) plays 200000 games of craps and looks at the distribution of wins and the number of throws each game.

There's also a test for LCGs called the [spectral test](https://en.wikipedia.org/wiki/Spectral_test) which plots the sequence in two or more dimensions. In the example below, we can see the [hyperplanes](https://en.wikipedia.org/wiki/Hyperplane) that the spectral test measures for.

![Hyperplanes of an LCG in three dimensions](lcg-3d.gif)

A PRNG eventually repeats its sequence. In this context, the _period_ is the length of steps until the cycle repeats. Simpler PRNGs such as [Mulberry32](https://github.com/bryc/code/blob/master/jshash/PRNGs.md#mulberry32) have a period as low as ~4 billion whereas the [Mersenne Twister](https://stackoverflow.com/a/38838863) has a period of `2^19,937 - 1`. In 2015, the V8 team [said](https://v8.dev/blog/math-random) that their implementation of `Math.random()` uses an algorithm called [xorshift128+](http://vigna.di.unimi.it/ftp/papers/xorshiftplus.pdf) which has a period of `2^128 - 1`. Its introduction can been seen in [this diff](https://github.com/v8/v8/blob/085fed0fb5c3b0136827b5d7c190b4bd1c23a23e/src/base/utils/random-number-generator.h#L102).

If a PRNG eventually repeats itself, you might wonder why we call it repeatedly. Why not use the first number and then reset the internal state with a new seed? The problem with this is that the seed needs to originate from somewhere. If we continue to ask the OS for more random data there is a chance that the call may block (as the OS waits for more randomness to be generated) and our program will stall.

## Entropy Required

So you've settled on a PRNG and replaced `window.Math.random`. You've shipped it to your users and, at first, everyone seems to be happy.

But wait! You forgot about the seed. And now your users are complaining about the sequence of random numbers they get. It's the same every time their customers' page loads. All of their software is predictable. As a result, the web games they built are easy to beat.

Huzaifa Sidhpurwala [reminds us](https://www.redhat.com/en/blog/understanding-random-number-generators-and-their-limitations-linux):

> Entropy is the measurement of uncertainty or disorder in a system. Good entropy comes from the surrounding environment which is unpredictable and chaotic.

When required, the generation of securely random numbers in the browser is performed by `Crypto.getRandomValues()` from the [Web Cryptography API](https://www.w3.org/TR/WebCryptoAPI/#Crypto-method-getRandomValues). Which is seeded by "a platform-specific random number function, the Unix `/dev/urandom` device, or other source of random or pseudorandom data."

The Linux [source](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/char/random.c) suggests where this pseudorandom data can come from:

> Sources of randomness from the environment include inter-keyboard timings, inter-interrupt timings from some interrupts, and other events which are both (a) non-deterministic and (b) hard for an outside observer to measure.

There are also hardware devices that use [quantum mechanical physical randomness](https://en.wikipedia.org/wiki/Hardware_random_number_generator#Quantum_random_properties).

You can find many [prominent examples](https://en.wikipedia.org/wiki/Random_number_generator_attack#Prominent_examples) of random number generator attacks which occurred because the wrong type (or not enough) entropy was used. Cloudflare [famously](https://www.cloudflare.com/learning/ssl/lava-lamp-encryption/) uses lava lamps as an entropy source. Since we are not attempting to create a secure algorithm, predictable sources of entropy like time are fine.

We can use `Date.now()` our seed state. This means that we will get a different random sequence for every millisecond. We could also use `performance.now()` which returns the length of time since the [time origin](https://developer.mozilla.org/en-US/docs/Web/API/DOMHighResTimeStamp#The_time_origin).

Other possible ways of getting entropy in the browser:

- `crypto.getRandomValues`, `crypto` key generation, or similar (feels like cheating)
- Mouse/touch events, [ambient light events](https://developer.mozilla.org/en-US/docs/Web/API/Ambient_Light_Events), mic/webcam noise (hard to use on page load)
- Geolocation API, Bluetooth API, or similar (need permission, doesn't work on page load)
- WebGL/video performance shenanigans
- Most APIs [listed here](https://developer.mozilla.org/en-US/docs/Web/API)

Here's our slower (because it's not native code) and unstable (because I haven't tested it) replacement for `Math.random()`. Also note that PRNGs have requirements for the seed state (e.g. prime numbers, 128-bit). Our algorithm doesn't comply with the [seed recommendations](http://vigna.di.unimi.it/ftp/papers/ScrambledLinear.pdf) for the Xoshiro family.

```javascript
// https://github.com/bryc/code/blob/master/jshash/PRNGs.md
// xoshiro128+ (128-bit state generator in 32-bit)
Math.random = (function xoshiro128p() {
  // Using the same value for each seed is _screamingly_ wrong
  // but this is 'good enough' for a toy function.
  let a = Date.now(),
    b = Date.now(),
    c = Date.now(),
    d = Date.now()
  return function () {
    let t = b << 9,
      r = a + d
    c = c ^ a
    d = d ^ b
    b = b ^ c
    a = a ^ d
    c = c ^ t
    d = (d << 11) | (d >>> 21)
    return (r >>> 0) / 4294967296
  }
})()

Math.random() // 0.5351827056147158
Math.random() // 0.2675913528073579
```

## So, Mission Accomplished?

Sadly it's impossible to create a fully ECMAScript compliant replacement for `Math.random()` since the specification requires "distinct realms [to] produce a distinct sequence of values from successive calls." A _realm_ roughly means a different global environment (e.g. a different window, or a different [WebWorker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API)). Our version cannot reach outside its realm thus cannot make this guarantee.

However, there have been proposals for a [Realms API](https://github.com/tc39/proposal-realms). It's not inconceivable that such an API would provide access to something like an incrementing realm id. This would give our algorithm the loophole it needs — access to Realm-unique entropy!

<small>Thanks to [JN~commonswiki](https://commons.wikimedia.org/wiki/File:Lcg_3d.gif) for the 3D GIF of the spectral test.</small>


Building a replacement for JavaScript's random number generator.

Creating Randomness Without Math.random


First some definitions.

> A **polyfill** will try to emulate certain APIs, so can use them as if they were already implemented.

> A **transpiler** on the other hand will transform your code and replace the respective code section by other code, which can already be executed.

(Thanks [Sirko](https://stackoverflow.com/a/31206361) for those!)

## Examples of polyfilling and transpiling

By polyfilling the [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) we make it usable in browsers where it isn't available by default. Another example would be making it available in its original functionality in Node.js.

By transpiling [Spread syntax](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax) (an ES6 feature) into ES5 compatible JavaScript, we end up with source code that is easier for us to write — and deployable code that works in older browsers!

Here's our ES6 code example.

```javascript
function speak() {
  console.log(...args);
}

speak(1, 2, 3);
```

Here is that same code transpiled into ES5 compatible code.

```javascript
"use strict";

function speak() {
  var _console;

  (_console = console).log.apply(_console, args);
}

speak(1, 2, 3);
```

(Technically, this is the [loose](https://babeljs.io/docs/en/babel-plugin-transform-spread#loose) transpilation because otherwise the above snippet would be eleven times longer.)

## Using `fetch` in old browsers

We love the Fetch API because it means we can avoid using [XMLHttpRequest](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest). Instead of providing a callback, we can use lovely [Promises](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise).

Here's a `fetch` call that prints out the status code of a request.

```javascript
fetch('https://httpstat.us')
    .then(res => console.log(res.status))
```

If we tried running that in Internet Explorer 11 (or Edge 13, or Chrome 39, and so on) we would get an error.

`window.fetch` would likely evaluate to undefined. We might get an error that looks like `Uncaught TypeError: window.fetch is not a function`.

It's easy to check one-off functionality for a feature on _Can I use_ — here's `fetch` [https://caniuse.com/#search=fetch](https://caniuse.com/#search=fetch). It isn't viable to check every feature that your code might use, so that's why we use things like [@babel/preset-env](https://babeljs.io/docs/en/babel-preset-env).

> `@babel/preset-env` is a smart preset that allows you to use the latest JavaScript without needing to micromanage which syntax transforms (and optionally, browser polyfills) are needed by your target environment(s). This both makes your life easier and JavaScript bundles smaller!

An even cooler feature of Babel is that it allows us to specify which platforms we want to support.

Why not just support _all_ platforms? Because then the JavaScript bundle we send to our users would get larger and larger every year and website performance would grind to a halt.

Babel's [Browserslist Integration](https://babeljs.io/docs/en/babel-preset-env#browserslist-integration) lets us forget about version numbers and instead use handy shortcuts. Let's say we wanted to support 99.75% of browsers and no dead browsers. We can add `"browserslist": "> 0.25%, not dead"` to our `package.json` file.

However, you might be reading this because you found out that `@babel/preset-env` doesn't include a `fetch` polyfill. This is an open issue on GitHub ([#9160](https://github.com/babel/babel/issues/9160)). It looks like it's going to stay this way.

> Babel only adds polyfill for ECMAScript methods (defined at https://tc39.github.io/ecma262/). Since fetch is defined by a web specification (https://fetch.spec.whatwg.org/), you need to add the polyfill yourself.

That's okay because we can use [github/fetch](https://github.com/github/fetch#usage) to polyfill it for us.

Either by replacing all instances of `fetch`.

```javascript
import 'whatwg-fetch'

window.fetch(...)
```

Or on a case-by-case basis.

```javascript
import { fetch as fetchPolyfill } from 'whatwg-fetch'

window.fetch(...)   // use native browser version
fetchPolyfill(...)  // use polyfill implementation
```

## Using `fetch` in Node.js

The Fetch API is common and people are fluent with it. It's productive if they can use it in all the JavaScript they write. Many people think it's available in Node.js by default. It's not but _there's a package for that_ (™).

[node-fetch/node-fetch](https://github.com/node-fetch/node-fetch) let's us use the API we're fluent with to make HTTP calls on the back-end. Underneath, it uses native Node.js functionality.

```javascript
// CommonJS
const fetch = require('node-fetch');

// ES Module
import fetch from 'node-fetch';
```

If you're looking for an isomorphic solution (this means using the same code in the browser and in Node.js) then you'll want Jason Miller's [isomorphic-unfetch](https://www.npmjs.com/package/isomorphic-unfetch) (but not for React Native, see [#125](https://github.com/matthew-andrews/isomorphic-fetch/issues/125)) or Leonardo Quixada's [cross-fetch](https://github.com/lquixada/cross-fetch).

These solutions will figure out which environment you're in and choose the correct polyfill.


A quick look at polyfilling, along with with solutions for both environments.

Polyfilling the Fetch API for Old Browsers and Node.js


I wanted to accept user comments on a Gatsby website and store them on GitHub. As in, I wanted the comments to go straight into a file called `comments.json` in my repository. So I could use something as simple as

```javascript
import comments from "../comments.json"
```

in my site's code. Without any databases. No third-party plugins making tens of networks requests.

[Netlify serverless functions](https://docs.netlify.com/functions/overview/) allowed me to use GitHub's API to make this repository change with the data from a submitted comment. It also hid my secret API credentials.

I built a prototype — [healeycodes/gatsby-serverless-comments](https://github.com/healeycodes/gatsby-serverless-comments) — that uses this flow:

1. 👩 User enters a comment and clicks submit.
2. ⚙️ A serverless function receives the data and hits GitHub's API.
3. 🔧 It reads the existing `comments.json` , appends the new comment, and saves.
4. 🚧 A new commit triggers a Netlify build.
5. ✅ The new version of the website is deployed!

The new comment is visible to users ~30 seconds ⏰ after the first click.

## The serverless function

Let's pick through the serverless function that receives the user's comment. It will make use of some constants that can be set through Netlify's website on _settings_ → _deploys_.

![GITHUB_PAT_TOKEN, GITHUB_REPO, GITHUB_USER](env-vars.png)

The function is written with _Node.js_ and exports a `handler` function, which is explained in [Netlify's documentation](https://docs.netlify.com/functions/build-with-javascript/#format).

```javascript
// comment.js

const fetch = require("node-fetch")

const auth = process.env.GITHUB_PAT_TOKEN
const repo = process.env.GITHUB_REPO
const user = process.env.GITHUB_USER
const api =
  "https://api.github.com/repos/" +
  user +
  "/" +
  repo +
  "/contents/src/comments.json"

exports.handler = async function(event, context, callback) {
  // Use the Contents API from GitHub
  // https://developer.github.com/v3/repos/contents/#get-contents
  const existingFile = JSON.parse(
    await fetch(api, {
      headers: {
        // Pass some kind of authorization
        // I'm using a personal access token
        Authorization:
          "Basic " + Buffer.from(user + ":" + auth)
            .toString("base64"),
      },
    }).then(res => res.text())
  )

  // The file's content is stored in base64 encoding
  // Decode that into utf-8 and then parse into an object
  let comments = JSON.parse(
    Buffer.from(existingFile.content, "base64").toString("utf-8")
  )

  // This is the user submitted comment
  // Perhaps we would do some validation here
  const newComment = JSON.parse(event.body)

  // Update the comments
  comments.push({
    author: newComment.author,
    email: newComment.email,
    message: newComment.message,
    date: Date.now(),
  })

  // Use the Contents API to save the changes
  const res = await fetch(api, {
    method: "PUT",
    headers: {
      Authorization:
        "Basic " + Buffer.from(user + ":" + auth).toString("base64"),
    },
    body: JSON.stringify({
      message: "New comment on " + new Date().toDateString(),

      // Turn that object back into a string and encoded it
      content: Buffer(JSON.stringify(comments)).toString("base64"),

      // Required: the blob SHA of the existing file
      sha: existingFile.sha,
    }),
  }).then(res => res.text())

  callback(null, {
    statusCode: 204,
  })
}
```

## Potential downsides

What if someone spams comments on your website? Well, you'll hit your build time limits pretty fast.

There's also a small window (10-100s of milliseconds between API calls) where two people comment at the same time and the older comment will be overwritten.

The fix for both of these is to alter our serverless function to open a pull-request with the comment change. Comments are now delayed but we've protected ourselves from malicious behavior and we can also screen comments for appropriateness. We won't lose any data but might rarely need to handle merge conflicts.

## My Netlify review

Netlify are betting big on [Jamstack](https://jamstack.org/) applications. It's a bet I would make too.

Their developer experience (DX) is up there with the best right now. It's rare I read about a product _just working_ and then it ends up doing so! Recently, Netlify's snappy deployments let me rush out changes to fix live issues within minutes.

What does this mean for their future success? Well, Tiny.cloud [points out](https://www.tiny.cloud/blog/developer-experience/) that:

> DX is kind of a big deal. Developers can play a huge role in the uptake of your product, especially when you consider they are likely to provide guidance on what tools their organization should invest in, even though the final decision usually happens at the executive level. 

Netlify's developer tooling lets me create prototypes like the one you're reading about without having to mess with configuration. [My Gatsby website](https://healeycodes.com/) is hosted with their generous free tier and transferring and hosting it has been hiccup-free.

I recommend them.


Use Netlify serverless functions to let users add comments to a static website.

Adding Comments to Gatsby with Netlify Serverless Functions + GitHub


My second passion alongside coding is writing. Whenever I can combine them, I do. 

![The analysis tool in action](analysis-tool.gif)

This weekend I used the [DEV API](https://docs.dev.to/api/) to build [a tool](https://dev-article-analysis.glitch.me/) that analyzes your articles over time. It's an interactive website where you can enter DEV usernames and get text statistics shown in pretty graphs. The metrics that I chose were reading level, ease of reading, and sentiment analysis.

It's open-sourced at [healeycodes/dev-article-analysis](https://github.com/healeycodes/dev-article-analysis)!

### Readability

The Flesch–Kincaid readability tests were developed in the U.S. Navy in the 70s. These scales are used to measure and compare books, newspapers, and articles. In Florida, they are used to encourage readable language in insurance policies.

> Florida Statutes Section 627.4145 (1) (a) The text achieves a minimum score of 45 on the Flesch reading ease test as computed in subsection (5) or an equivalent score on any other test comparable in result and approved by the office;

Flesch-Kincaid Grade scores text at a U.S. grade level. A score of 10.4 means that a tenth grader would be able to understand the article.

Flesch reading ease is a similar test with different weightings. It results in a score generally ranging between 0-100. Higher means easier to read. Wikipedia has a [table](https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests#Flesch_reading_ease) which relates score to grade level with some notes.

Technical articles, especially those with code excerpts, get wide-ranging scores. However, a user's writing style (e.g. a similar amount of code per article) means that a reasonable line can be drawn over time, perhaps leading to some basic conclusions. [text-readability](https://www.npmjs.com/package/text-readability) was used to calculate these scores in the project.

### Sentiment

Sentiment analysis helps us understand the emotional polarity of a text. For example, the phrase `I love cats, but I am allergic to them.` gives a comparative score of `0.1111111111111111`. To calculate this, a list of words (and emojis) are given weightings. -5 to 5, positive to negative. The only tokens in our sentence with scores are `{ allergic: -2 }, { love: 3 }`. We take the number of tokens, `9`, and calculate `(3 + -2) / 9` to find the comparative score. This example is used in the documentation of [sentiment](https://www.npmjs.com/package/sentiment).

The DEV API provides the text of an article in HTML or markdown, neither of which is friendly to analysis. This is why [remove-markdown](https://www.npmjs.com/package/remove-markdown) is used to reduce the `body_markdown` value to plaintext.

### Interaction

Users enter a DEV username. As the articles are retrieved, a progress status (and a random positive emoji) are sent to the page via WebSocket. This means the user isn't left sitting looking at a spinner.

I normally use Chart.js to render basic line charts but wondered if there were a simpler solution. I tried ApexCharts.js and didn't run into any problems. The online documentation is good and I searched and found the solution to one of my errors in a GitHub Issue. However, I might as well have used Chart.js because the level of customization I required meant the syntax for both libraries was near-identical.

The charts can be hovered to see article titles and their scores for each metric.

When arriving at the website, the graphs are already filled with precalculated data from my own DEV username (healeycodes). This is to help visitors better understand what the end result of their interaction will be.

### Deployment

I developed this Node/Express application on [Glitch](https://glitch.com/). Their cloud IDE and console lets me move fast and get feedback from people seconds after I've made a change. You can export the project to a GitHub project (it creates a `glitch` branch which you can open a pull request from).

![Glitch cloud IDE](glitch-ide.png)

Sometimes I want to upload image files and in the past I found this tricky. Glitch wants you to use their asset mechanism for this, which makes exporting the whole project to GitHub harder than it needs to be. My workaround is to upload the image to the project using the asset mechanism, copy the long URL they provide, `wget` this URL in the console which downloads it into the project, and finally using `mv` to change the name of the image.

This often brings the cloud IDE out of sync but entering `refresh` in console fixes this.


Building a tool for users to analyze their work over time.

DEV Article Analysis


I migrated from Jekyll to Gatsby recently and so far I've had a really neat time. The whole process took about a week of casual coding (a few hours here and there). The Gatsby ecosystem enabled me to quickly add a few features to my blog that I thought were missing; dark mode, better syntax highlighting, and the ability to design with components.

Gatsby starters are boilerplate Gatsby sites [maintained by the community](https://www.gatsbyjs.org/docs/starters/). One of the reasons I love them is that they use **Semantic HTML**. This is great because:

- It helps with search engine optimization — web crawlers are able to understand which parts of your pages are important.
- It helps with accessibility — for people who use non-traditional browsers and screenreaders.
- It helps with maintenance — I was able to pick up a starter and understand what the different parts of the template referred to due to the semantic tags.

Take this example from `gatsby-starter-blog` — the most popular starter and the base for my [current blog](https://healeycodes.com) (in-line styling removed).

```jsx
<article>
  <header>
    <h1>{post.frontmatter.title}</h1>
    <p>{post.frontmatter.date}</p>
  </header>
  <section dangerouslySetInnerHTML={{ __html: post.html }} />
  <hr />
  <footer>
    <Bio />
  </footer>
</article>
```

I've seen quite a few beginner web development resources that skip on semantic HTML and encourage what I'll call 'div-spamming'. The HTML5 spec weighs in on this [issue](https://html.spec.whatwg.org/multipage/grouping-content.html#the-div-element).

> Authors are strongly encouraged to view the div element as an element of last resort, for when no other element is suitable. Use of more appropriate elements instead of the div element leads to better accessibility for readers and easier maintainability for authors.

# Coming From Jekyll

I started blogging with Jekyll a year ago because I hosted my blog on GitHub pages and it was the static site generator with the least friction. It was a great choice at the time as it enabled me to get up and running straight away.

I've seen many people warning others (engineers in particular) to avoid rolling their own blogging solutions. The advice is that you should start writing and publishing first. This is because building a blog can function as procrastination and who knows if you actually enjoy blogging (the activity) or the idea of having blogged (the achievement).

With Jekyll, I used basic markdown and transferring written content to Gatsby wasn't too hard. Images had to be moved from one disorganized folder into separate folders. URLs were a bit of a pain and took 1.5hrs of manual work. I wanted all of my old posts to keep their location on the web so I added a front matter tag called `path` to override the default naming scheme. My old URLs were too long and included categories (which I'm still to implement) so the path scheme from now on will be the title only.

I extended `onCreateNode` in `gatsby-node.js`. I'm not sure if this is the best practice way to implement this feature but it works great so far. 

```js
exports.onCreateNode = ({ node, actions, getNode }) => {
  const { createNodeField } = actions

  if (node.internal.type === `MarkdownRemark`) {
    // Check to use legacy path
    const slug = node.frontmatter.path || createFilePath({ node, getNode })
    createNodeField({
      node,
      name: `slug`,
      value: slug,
    })
  }
}
```

# Syntax Highlighting

Code excerpts show up in a lot of my posts and I like them to be easy to parse.

I installed [gatsby-remark-prismjs](https://www.gatsbyjs.org/packages/gatsby-remark-prismjs/) for syntax highlighting and was up and running in about an hour with another hour spent tinkering styles to match my light/dark mode toggle. I use [New Moon Theme](https://github.com/taniarascia/new-moon) by Tania Rascia for my code excerpts. I couldn't find a version of the theme for PrismJS so I extracted the styling from Tania's (MIT-licensed) [blog](https://www.taniarascia.com/). My site's general color theme is custom.

One of the reasons I'm mentioning plugins is that I found it hard to integrate them with Jekyll and I feel like it wasn't just my inexperience with Ruby that was holding me back. Perhaps it's due to the hype surrounding Gatsby which means there are up-to-date resources. I've contributed one (small) open-source fix to the Jekyll project and I would still recommend it for anyone looking for a sensible system for HTML/CSS that has wide community support e.g., GitHub pages, Netlify, etc. If you want to avoid JavaScript, Jekyll is the way to go.

For my light/dark mode I use `gatsby-plugin-dark-mode` which works well out of the box and has good (but not great) documentation. For theme-toggling, I researched and found that a common pattern was to declare CSS variables in body scope and then to override these variables in _class_ scope. This way, the `dark` class can be added to the `<body>` tag which means `dark` CSS variables take precedence due to CSS Specificity. Classes are then toggled on and off the `<body>`, storing the preference in local browser storage.

```css
body {
  --bg: #eaeaeb;
  --textNormal: #414158;
}

body.dark {
  --bg: #21212c;
  --textNormal: #eaeaeb;
}
```

### Designing With Components

The first React component I wrote for my blog was for wrapping the `<ThemeToggler />` from `gatsby-plugin-dark-mode` into a component. It switches between a sun and a moon to let the user know which theme can be switched to. The base for this is the example code from the [docs](https://www.gatsbyjs.org/packages/gatsby-plugin-dark-mode/).

```jsx
<ThemeToggler>
  {({ theme, toggleTheme }) => (
    <label style={{ cursor: `pointer`, opacity: 0.8 }}>
      <input
        style={{ display: `none` }}
        type="checkbox"
        onChange={e => toggleTheme(e.target.checked ? "dark" : "light")}
        checked={theme === "dark"}
      />
      {theme === "dark" ? `☀️` : `🌑`}
    </label>
  )}
</ThemeToggler>
```

I've never used React as part of a blogging solution. I like the hierarchical UI approach that's encouraged. Including CSS-in-JS makes sense for the scale of my website. It's easier for me to reason about and quickly tinker with.

# How I Deploy

My website source exists in a GitHub repository. I write in markdown in VS Code, commit, and push. Netlify is connected to the repository and builds and deploys every commit to master. The build process takes 2m50s (30s locally).

I previously used Netlify for Jekyll and setting up either static site generator involved about 10 clicks and entering one or two build commands. The walkthrough covers everything.

Overall, the Gatsby experience has been very enjoyable. Everywhere I look in my online bubble (Twitter, DEV, lobste.rs) people are talking about Gatsby. It's nice to be part of the crowd.

I'm also happy that my site (despite being React-based) works fine without JavaScript enabled (barring the theme toggle, which I might hide with `<noscript>` styling).


Gatsby Sparks Joy


I love code puzzles. Always have. My deskmate told me about a puzzle that no one in the office had been able to solve when they shared it around. I was intrigued because of the challenge but also because it was a JavaScript puzzle.

I'd understand if it were a Project Euler problem (they can be mathy) or perhaps if no one had found the optimal solution to a LeetCode 'Hard' (I've seen some problems that once upon a time were research papers).

__The puzzle took me two weeks to solve__. I became stuck almost instantly and then it hung around in the back of my mind until the solution came to me. First, let's take a look at the easier version of the problem which helped me unlock the harder version.

Don't scroll too fast unless you want spoilers.

### [Codewars: Multi Line Task++: Hello World](https://www.codewars.com/kata/5935558a32fb828aad001213)

- Write a function `f` that returns `Hello, world!`.
- Every line must have at most 2 characters, and the total number of lines must be less than 40.

Without the line restriction. The solution is:

```javascript
function f () { return 'Hello, world!'; }
// or
const f = _ => 'Hello, world!';
```

I started splitting up the code and shortened the variable declaration. We can throw away the `const` and allow the function to exist in the global scope. We can also use template strings to break up the string into multiple lines.

Errors incoming.

```javascript
f
=_
=> // Uncaught SyntaxError: Unexpected token '=>'
`H
el
l,
 w
or
ld
!` // This string has newline characters in it!
```

My next idea was to define the function inside an object and then retrieve the function out of the object.

```javascript
f=
{g
()
{
return 'Hello, world!'; // Too long!
}}
[`
g
`[
1]
]

// 'Beautified'
f = {
        g() {
            return 'Hello, world!';
        }
    }
    [`
g
` [
        1
    ]]
```

No errors, I was declaring a function but now I couldn't figure out how to return from the _inner_ function without using the `return` keyword. It felt like I was close but I wasn't. I was still stuck on defining the string without newline characters as well.

### Finding inspiration: [JSFuck](http://www.jsfuck.com/)

> JSFuck is an esoteric and educational programming style based on the atomic parts of JavaScript. It uses only six different characters to write and execute code.

Reading through this project's [source code](https://github.com/aemkei/jsfuck/blob/master/jsfuck.js) really opened my mind to some parts of JavaScript that never really come up unless you're doing something like writing a library or code golfing.

Once I figured out how to remove the newline characters from the `Hello, world!` message (escaping with a backslash `\`) everything else fell into place. I could now use square brackets `[]` on objects to run pretty much anything. However, one of the requirements was to keep the total line count under 40 (one of the reasons why using JSFuck encoding was out of the question).

My idea was to create a new function by calling [Function.prototype.bind](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_objects/Function/bind) on a [String](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String) method. I used [String.prototype.trim](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/trim) because it had the shortest name (and conveniently got rid of any trailing newlines too).

```javascript
f=
''
[
'\
t\
r\
i\
m'
][
'\ // Get a new function where `this` is "Hello, world!"
b\
i\
n\
d'
]` // Tagged template, see below
H\
e\
l\
l\
o\
,\
 \
w\
o\
r\
l\
d\
!`
```

I also used tagged templates to pass `Hello, world!` as an argument to bind.

> Tags allow you to parse template literals with a function. The first argument of a tag function contains an array of string values.

Let's take it up a level to the harder version that started this journey!

### Codewars: [Multi Line Task∞: Hello World](https://www.codewars.com/kata/59a421985eb5d4bb41000031)

- Write a function `f` that returns `Hello, world!`.
- Every line must have __at most 1 character__, and the total number of lines must be less than 145.

Without having first solved the two-characters-per-line version I don't think I would have come close to answering this version.

The solution I went for is the same, we use `bind` on `trim` and pass the message as an argument (without template tags this time). To access the String object we use `[]+[]` which evaluates to `""`.

Since we can no longer escape the newline character from within a template string we have to use a workaround. The property names (`trim`, `bind`) and the message (`Hello, world!`) have to be built with concatenated variables.

We use [destructuring assignment](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Destructuring_assignment) on a template string and use empty slots in the first array to 'skip' assigning the newline character to anything. Like this:

```javascript
[,a,,b,,c] = 'xHxix!' // avoid the "x"s
a + b + c // Evaluates to: "Hi!"
```

I didn't optimize the solution any further once it passed the requirements. It's been left verbose to better explain what's going on (for example, we only need one "l" in the template string).

```javascript
[ // With destructuring assignment, start declaring variables
,
t
,
,
r
,
,
i
,
,
m
,
,
b
,
,
i
,
,
n
,
,
d
,
,
H
,
,
e
,
,
l
,
,
l
,
,
o
,
,
c // Comma
,
,
s
,
,
w
,
,
o
,
,
r
,
,
l
,
,
d
,
,
x // Exclamation mark
]
=
`
t
r
i
m
b
i
n
d
H
e
l
l
o
,
 
w
o
r
l
d
!
`
f // Start declaring our function
=
( // This evaluates to "" or, the String object
[
]
+
[
]
)
[ // `trim`
t
+
r
+
i
+
m
]
[ // `bind`
b
+
i
+
n
+
d
]
( // Can use parentheses or template tag syntax to call `bind`
H
+
e
+
l
+
l
+
o
+
c
+
s
+
w
+
o
+
r
+
l
+
d
+
x
)
```

<br>

I'm definitely taking a break from language-specific coding puzzles — give me logic over syntax! However, I'm glad I scratched this itch.

Writing JavaScript with a one character width restriction.

Hardest JavaScript Puzzle I've Ever Solved


CAPTCHAs are [not accessible](https://www.w3.org/TR/turingtest/) and in some cases not even effective but there's a lot to be learned by generating our own!

Find the source code for this article at [healeycodes/captcha-api](https://github.com/healeycodes/captcha-api)

## A solution for spam

Let's imagine a client who demands a solution for bot spam. They ask for an image and a string of the image's text. You call to mind every inscrutable jumbled mess of letters and numbers you've frustratingly failed to solve. You agree to the task nonetheless.

![Am I a human? Yes. Can I type phi off the top of my head? No. Picture of hard CAPTCHA](hard-captcha.png)

This client has a whole fleet of websites. Different sized CAPTCHAs are required in different places. They will provide a width and a height. This describes the specification of our API.

JavaScript is great for generating images because we can lean on the [Canvas API](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API). I've always found it to be handy to use with a lot of Stackoverflow content for when I'm stuck.

We don't want to generate our CAPTCHAs in browser-land because the bots that we're trying to keep out can inspect the source code, find the values in memory, and try all kinds of other tricky tactics.

## A Node.js service

Let's move it to the back-end to a service that can be called as desired. Someone has already solved the issue of accessing a Web API where there is not one, with [node-canvas](https://www.npmjs.com/package/canvas) or `npm i canvas`.

> [canvas] is an implementation of the Web Canvas API and implements that API as closely as possible.

We'll need to generate some random text each time. So let's write two functions to help us. For our API, we'll break logic down into functions that do one thing (and one thing well) so that the end result is easy to reason about and maintain.

```javascript
/* captcha.js */

// We'll need this later
const { createCanvas } = require("canvas");

// https://gist.github.com/wesbos/1bb53baf84f6f58080548867290ac2b5
const alternateCapitals = str =>
  [...str].map((char, i) => char[`to${i % 2 ? "Upper" : "Lower"}Case`]()).join("");

// Get a random string of alphanumeric characters
const randomText = () =>
  alternateCapitals(
    Math.random()
      .toString(36)
      .substring(2, 8)
  );
```

There's no way to automatically scale text in a canvas (just like in the browser *weeps*) so we'll need some helper functions for that too. Depending on the length of your CAPTCHA and how you want the text to be positioned inside the image, you may need to test run it. Here are some variables I prepared earlier.

```javascript
const FONTBASE = 200;
const FONTSIZE = 35;

// Get a font size relative to base size and canvas width
const relativeFont = width => {
  const ratio = FONTSIZE / FONTBASE;
  const size = width * ratio;
  return `${size}px serif`;
};
```

This scales the text so as long as the proportions of the canvas remain the same, we can expect a similar-looking image.

For this article, we're just going to rotate the text but there are tons of ways to distort the text to hide it from bots and I'd love to see what you come up with (try searching for "perspective transform canvas javascript").

When rotating a canvas, the value we pass is in [radians](https://en.wikipedia.org/wiki/Radian) so we need to multiply our random degrees by `Math.PI / 180`.

```javascript
// Get a float between min and max
const arbitraryRandom = (min, max) => Math.random() * (max - min) + min;

// Get a rotation between -degrees and degrees converted to radians
const randomRotation = (degrees = 15) => (arbitraryRandom(-degrees, degrees) * Math.PI) / 180;
```

No more helper functions, I promise. We're going to get to the real meat of it now. The logic is broken up into two functions. `configureText` takes a canvas object and adds and centers our random text. `generate` takes a width and height value (remember the specification we were given?) and returns a [Data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs) of a PNG image — our CAPTCHA. 

> Data URLs, URLs prefixed with the `data:` scheme, allow content creators to embed small files inline in documents.

```javascript
// Configure captcha text
const configureText = (ctx, width, height) => {
  ctx.font = relativeFont(width);
  ctx.textBaseline = "middle";
  ctx.textAlign = "center";
  const text = randomText();
  ctx.fillText(text, width / 2, height / 2);
  return text;
};

// Get a PNG dataURL of a captcha image
const generate = (width, height) => {
  const canvas = createCanvas(width, height);
  const ctx = canvas.getContext("2d");
  ctx.rotate(randomRotation());
  const text = configureText(ctx, width, height);
  return {
    image: canvas.toDataURL(),
    text: text
  };
};
```

We can consider all of the functions apart from `generate` to be private functions that shouldn't be used elsewhere, so let's just export this function.

```javascript
module.exports = generate;
```

## An API served by Express

So far we have one file, `captcha.js` which contains our image generation logic. To make this functionality available to be called by someone else we will serve it via an HTTP API. Express has the most community support for this kind of task.

The routes we'll host are:

- `/test/:width?/:height?/`
    - Used to get an image tag for manual testing.
- `/captcha/:width?/:height?/`
    - Used to get a CAPTCHA object for proper usage.

The question marks in the route here are the Express syntax for optional URL parameters. This means the client can provide none, the first one, or both. We'll validate that integers are passed as values (required by canvas) and if not we'll use sensible defaults.

The Express app in full:

```javascript
/* app.js */

const captcha = require("./captcha");
const express = require("express");
const app = express();

// Human checkable test path, returns image for browser
app.get("/test/:width?/:height?/", (req, res) => {
  const width = parseInt(req.params.width) || 200;
  const height = parseInt(req.params.height) || 100;
  const { image } = captcha(width, height);
  res.send(`<img class="generated-captcha" src="${image}">`);
});

// Captcha generation, returns PNG data URL and validation text
app.get("/captcha/:width?/:height?/", (req, res) => {
  const width = parseInt(req.params.width) || 200;
  const height = parseInt(req.params.height) || 100;
  const { image, text } = captcha(width, height);
  res.send({ image, text });
});

module.exports = app;
```

This Express app is exported so that we can test it. Our API is functional at this point. All we have to do is serve it which the following file takes care of.

```javascript
/* server.js */

const app = require("./app");
const port = process.env.PORT || 3000;

app.listen(port, () => console.log(`captcha-api listening on ${port}!`));
```

Navigating to `http://localhost:3000/test` rewards us with our basic CAPTCHA. Browsers will add a `body` and `html` tag if otherwise omitted.

![Our CAPTCHA](our-captcha.jpg)

## A valid Data URL

It's time to write some tests but first, put away your [unwieldy regular expressions](https://stackoverflow.com/a/1732454/10044202). There's a [library](https://www.npmjs.com/package/valid-data-url) that has already solved this problem. `valid-data-url` does exactly what it says on the tin.

I like to use Jest as my test runner. For no reason other than it's always worked for me and when it hasn't I've been able to find the answer. My setup is setting the `scripts` key in `package.json` like so:

```javascript
  "scripts": {
    "test": "jest"
  }
```

This is so I can type `npm test` (which is what many CI systems default to as well). Jest then finds and runs all of our tests.

Our app's test file imports the Express application object and uses `supertest` to mock HTTP requests against it. We use async/await syntax to reduce on callbacks.

```javascript
/* app.test.js */

const request = require("supertest");
const assert = require("assert");
const validDataURL = require("valid-data-url");
const app = require("../app");

describe("captcha", () => {
  describe("testing captcha default", () => {
    it("should respond with a valid data URL", async () => {
      const image = await request(app)
        .get("/captcha")
        .expect(200)
        .then(res => res.body.image);
      assert(validDataURL(image));
    });
  });

  describe("testing captcha default with custom params", () => {
    it("should respond with a valid data URL", async () => {
      const image = await request(app)
        .get("/captcha/300/150")
        .expect(200)
        .then(res => res.body.image);
      assert(validDataURL(image));
    });
  });
});
```

Given the size of this application (small) I'm content with leaving it at two integration tests.

## Constant integration with a GitHub Action

Since we used the standard npm test command (`npm test`) to configure our repository, we can set up a GitHub Action with a few clicks. This way, our application will be built and tested every time code is pushed.

![Build and test your JavaScript repository. Node.js Build and test a Node.js project with npm. "Set up this workflow". npm ci\nnpm run build --if-present\nnpm test](github-workflow.png)

Now we've got a sweet badge to show off!


Using Canvas, Express, and a GitHub Action.

Let's Build a CAPTCHA Generator with Node.js


Recently, I wrote an article about [how to write very fast JavaScript](/javascript/node/discuss/tutorial/2019/05/19/high-performance-javascript.html). Some of the examples took it to the extreme and became very quick at the cost of being totally unmaintainable. There's a middle ground between speed and comprehension and that's where _good code_ lives.

We're going to refactor some pieces of code based off real examples that I've come across. Sometimes I'll need to perform this kind of refactoring on my own code before submitting a PR. Other times I'll do a small refactor of existing code at the start of a [story](https://en.wikipedia.org/wiki/User_story) or bug to make my changes easier to implement.

## Scenario 1

We're an URL-shortening website, like TinyURL. We accept a long URL and return a short URL that forwards visitors to the long URL. We have two functions.

```javascript
// Unrefactored code

const URLstore = [];

function makeShort(URL) {
  const rndName = Math.random().toString(36).substring(2);
  URLstore.push({[rndName]: URL});
  return rndName;
}

function getLong(shortURL) {
  for (let i = 0; i < URLstore.length; i++) {
    if (URLstore[i].hasOwnProperty(shortURL) !== false) {
      return URLstore[i][shortURL];
    }
  }
}
```

Problem: what happens if `getLong` is called with a short URL that isn't in the store? Nothing is explicitly returned so `undefined` will be returned. Since we're not sure how we're going to handle that, let's be explicit and throw an error so that problems can be spotted during development.

Performance-wise, be careful if you're iterating through a flat array very often, especially if it's a core piece of your program. The refactor here is to change the data-structure of `URLstore`.

Currently, each URL object is stored in an array. We'll visualize this as a row of buckets. When we want to convert short to long, on average we need to check half of those buckets before we find the correct short URL. What if we have thousands of buckets, and we perform this hundreds of times a second?

The answer is to use some form of [hash function](https://en.wikipedia.org/wiki/Hash_function), which Maps and Sets [use under the surface](https://v8.dev/blog/hash-code). _A hash function is used to map a given key to a location in the hash table_. Below, this happens when we place our short URL into the store in `makeShort` and when we get it back out in `getLong`. Depending on how you're [measuring run time](https://en.wikipedia.org/wiki/Big_O_notation), the result is that _on average_ we only need to check one bucket — no matter how many total buckets there are!

```javascript
// Refactored code

const URLstore = new Map(); // Change this to a Map

function makeShort(URL) {
  const rndName = Math.random().toString(36).substring(2);
  // Place the short URL into the Map as the key with the long URL as the value
  URLstore.set(rndName, URL);
  return rndName;
}

function getLong(shortURL) {
  // Leave the function early to avoid an unnecessary else statement
  if (URLstore.has(shortURL) === false) {
    throw 'Not in URLstore!';
  }
  return URLstore.get(shortURL); // Get the long URL out of the Map
}
```

For those examples, we assumed that the random function wouldn't clash. 'Cloning TinyURL' is a common system design question and a very interesting one at that. What if the random function _does_ clash? It's easy to tack on addendums about scaling and redundancy.

## Scenario 2

We're a social media website where user URLs are generated randomly. Instead of random gibberish, we're going to use the `friendly-words` package that the Glitch team works on. They use this to generate the random names for your recently created projects!

```javascript
// Unrefactored code

const friendlyWords = require('friendly-words');

function randomPredicate() {
  const choice = Math.floor(Math.random() * friendlyWords.predicates.length);
  return friendlyWords.predicates[choice];
}

function randomObject() {
  const choice = Math.floor(Math.random() * friendlyWords.objects.length);
  return friendlyWords.objects[choice];
}

async function createUser(email) {
  const user = { email: email };
  user.url = randomPredicate() + randomObject() + randomObject();
  await db.insert(user, 'Users')
  sendWelcomeEmail(user);
}
```

It's often said that a function should do one thing. Here, `createUser` does one thing .. kinda. It creates a user. However, if we're thinking ahead to the future, there's a good chance (if our business is successful) that this function is going to grow very large indeed. So let's start early by breaking it up.

You may have also noticed that there is some duplicated logic in our random functions. The `friendly-worlds` package also offers lists for 'teams' and 'collections'. We can't go around writing functions for every option. Let's write **one** function that accepts a list of friendly things.

```javascript
// Refactored code

const friendlyWords = require('friendly-words');

const generateURL = user => {
  const pick = arr => arr[Math.floor(Math.random() * arr.length)];
  user.url = `${pick(friendlyWords.predicates)}-${pick(friendlyWords.objects)}` +
    `-${pick(friendlyWords.objects)}`; // This line would've been too long for linters!
};

async function createUser(email) {
  const user = { email: email };
  // The URL-creation algorithm isn't important to this function so let's abstract it away
  generateURL(user);
  await db.insert(user, 'Users')
  sendWelcomeEmail(user);
}
```

We separated some logic and reduced the number of lines of code. We inlined a function called `pick` that accepts an array of length 1 and up and returns a random choice, then we used a [template literal](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals) to build an URL.

## Strategies

Here are some straightforward to implement methods that can lead to easier to read code. There are no absolutes when it comes to clean code — there's always an edge case!

Return early from functions:

```javascript
function showProfile(user) {
  if (user.authenticated === true) {
    // ..
  }
}

// Refactor into ->

function showProfile(user) {
  // People often inline such checks
  if (user.authenticated === false) { return; }
  // Stay at the function indentation level, plus less brackets
}
```

Cache variables so functions can be read like sentences:

```javascript
function searchGroups(name) {
  for (let i = 0; i < continents.length; i++) {
    for (let j = 0; j < continents[i].length; j++) {
      for (let k = 0; k < continents[i][j].tags.length; k++) {
        if (continents[i][j].tags[k] === name) {
          return continents[i][j].id;
        }
      }
    }
  }
}

// Refactor into ->

function searchGroups(name) {
  for (let i = 0; i < continents.length; i++) {
    const group = continents[i]; // This code becomes self-documenting
    for (let j = 0; j < group.length; j++) {
      const tags = group[j].tags;
      for (let k = 0; k < tags.length; k++) {
        if (tags[k] === name) {
          return group[j].id; // The core of this nasty loop is clearer to read
        }
      }
    }
  }
}
```

Check for Web APIs before implementing your own functionality:

```javascript
function cacheBust(url) {
  return url.includes('?') === true ?
    `${url}&time=${Date.now()}` :
    `${url}?time=${Date.now()}`
}

// Refactor into ->

function cacheBust(url) {
  // This throws an error on invalid URL which stops undefined behaviour
  const urlObj = new URL(url);
  urlObj.searchParams.append('time', Date.now); // Easier to skim read
  return url.toString();
}
```

It's important to get your code right the first time because in many businesses there isn't much value in refactoring. Or at least, it's hard to convince stakeholders that eventually uncared for codebases will grind productivity to a halt.


Refactoring JavaScript for Performance and Readability (with Examples!)


You've just written a great piece of JavaScript. But when the running process stops, or the user refreshes, all of that nice data **disappears into the ether**.

Is this you?

When prototyping, or otherwise working on tiny projects, it can be helpful to manage some state without resorting to a database solution that wasn't designed for that creative itch you're trying to scratch.

We're going to explore some options that I wish I knew about when I started tinkering on the web. We'll look at JavaScript in the browser and Node.js on the back end. We'll also look at some lightweight databases that use the local file system.

## Node.js

First up is JSON serializing your data and saving it to disk. The MDN Docs have a great [article](https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Objects/JSON) if you haven't worked with JSON before.

```javascript
const fs = require('fs');

const users = {
    'Bob': {
        age: 25,
        language: 'Python'
    },
    'Alice': {
        age: 36,
        language: 'Haskell'
    }
}

fs.writeFile('users.json', JSON.stringify(users), (err) => {  
    // Catch this!
    if (err) throw err;

    console.log('Users saved!');
});
```

We created our users object, converted it to JSON with [JSON#stringify](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify) and called [fs#writeFile](https://nodejs.org/api/fs.html#fs_fs_writefile_file_data_options_callback). We passed in a filename, our serialized data, and an arrow function as a callback to execute when the write operation finishes. Your program will continue executing code in the meanwhile.

You can also use this method to write normal serialized data by passing in anything that can be cast to a string. If you're storing text data, you may find [fs#appendFile](https://nodejs.org/api/fs.html#fs_fs_appendfile_path_data_options_callback) useful. It uses an almost identical API but sends the data to the end of the file, keeping the existing contents.

There is a synchronous option, [fs#writeFileSync](https://nodejs.org/api/fs.html#fs_fs_writefilesync_file_data_options) but it isn't recommended as your program will be unresponsive until the write operation finishes. In JavaScript, you should aim to [Never Block](https://developer.mozilla.org/en-US/docs/Web/JavaScript/EventLoop#Never_blocking).

If you're dealing with CSV files, reach for the battle-hardened [node-csv](https://github.com/adaltas/node-csv) project.

Let's load those users back into our program with [fs#readFile](https://nodejs.org/api/fs.html#fs_fs_readfile_path_options_callback).

```javascript
fs.readFile('users.json', (err, data) => {
    // Catch this!
    if (err) throw err;
  
    const loadedUsers = JSON.parse(data);
    console.log(loadedUsers);
});
```

### Lightweight databases

SQLite uses a local file as a database — and is one of my favorite pieces of software in the world. It enables many of my smaller projects to exist with low maintenance and little deploying hassle.

Here are some facts about SQLite:
- The project has [711 times](https://www.sqlite.org/testing.html) as much test code and test scripts compared to other code.
- The developers pledge to keep it backward compatible through at least the year 2050.
- It's used on planes, in Android, and you probably interacted with it in [some way](https://www.sqlite.org/famous.html) on your way to this article today.

Seriously, [How SQLite Is Tested](https://www.sqlite.org/testing.html) is a wild ride.

In Node.js, we commonly use the `sqlite3` [npm package](https://www.npmjs.com/package/sqlite3). I'll be using some code from Glitch's `hello-sqlite` template, which you can [play around with and remix](https://glitch.com/edit/#!/remix/hello-sqlite) without an account.

```javascript
// hello-sqlite
let fs = require('fs');
let dbFile = './.data/sqlite.db'; // Our database file
let exists = fs.existsSync(dbFile); // Sync is okay since we're booting up
let sqlite3 = require('sqlite3').verbose(); // For long stack traces
let db = new sqlite3.Database(dbFile);
```

Through this `db` object, we can interact with our local database like we would through a connection to an outside database.

We can create tables.

```javascript
db.run('CREATE TABLE Dreams (dream TEXT)');
```

Insert data (with error handling).

```javascript
db.run('INSERT INTO Dreams (dream) VALUES (?)', ['Well tested code'], function(err) {
  if (err) {
    console.error(err);
  } else {
    console.log('Dream saved!');
    }
});
```

Select that data back.

```javascript
db.all('SELECT * from Dreams', function(err, rows) {
  console.log(JSON.stringify(rows));
});
```

You may want to consider [serializing some of your database queries](https://stackoverflow.com/a/42946402). _Each command inside the serialize() function is guaranteed to finish executing before the next one starts._ The sqlite3 [documentation](https://github.com/mapbox/node-sqlite3/wiki) is expansive. Keep an eye on the SQLite [Data Types](https://www.sqlite.org/datatype3.html) as they can be a little different to other databases.

If even SQLite seems like too much overhead for your project, consider [lowdb](https://github.com/typicode/lowdb) (also [remixable on Glitch](https://glitch.com/edit/#!/low-db)). lowdb is exciting because it's _a small local JSON database powered by Lodash (supports Node, Electron and **the browser**)_. Not only does it work as a wrapper for JSON files on the back end it also provides an API which wraps localStorage in the browser.

From their [examples](https://github.com/typicode/lowdb/tree/master/examples#browser):

```javascript
import low from 'lowdb'
import LocalStorage from 'lowdb/adapters/LocalStorage'

const adapter = new LocalStorage('db')
const db = low(adapter)

db.defaults({ posts: [] })
  .write()

// Data is automatically saved to localStorage
db.get('posts')
  .push({ title: 'lowdb' })
  .write()
```

## Browser

This brings us to the front end. [window#localStorage](https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage) is the modern solution to storing data in HTTP cookies — which MDN [doesn't recommend](https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Client-side_web_APIs/Client-side_storage#Old_fashioned_cookies) for storing things anymore.

Let's interact with them right now. If you're on desktop, open your dev console (F12 on Chrome) and see what DEV is storing for you:

```javascript
for (const thing in localStorage) {
  console.log(thing, localStorage.getItem(thing))
}

// Example of one thing:
// pusherTransportTLS {"timestamp":1559581571665,"transport":"ws","latency":543}
```

We saw how lowdb interacted with localStorage but for our small projects it's probably easier to talk to the API directly. Like this:

```javascript
// As a script, or in console
localStorage.setItem('Author', 'Andrew') // returns undefined
localStorage.getItem('Author') // returns "Andrew"
localStorage.getItem('Unset key') // returns null
```

It gets easier still: you can treat it like an object. Although, MDN recommends the API over this [shortcut](https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API/Using_the_Web_Storage_API#Basic_concepts).

```javascript
console.log(localStorage['Author']); // prints "Andrew"
```

If you don't want to store data on the user's computer forever (which can be cleared with `localStorage.clear()` **but don't run this on DEV**) you may be interested in [sessionStorage](https://developer.mozilla.org/en-US/docs/Web/API/Window/sessionStorage) which has a near identical API and only stores data while the user is on the page.

## End notes

I read somewhere that SQLite is used onboard the Internation Space Station in some capacity but I haven't been able to find a source. My fiancée wants you to know that SQLite _is_ a database and the title of this post is incorrect.


The title is a lie I also show how to use SQLite.

Saving Data in JavaScript Without a Database


I saw this question doing the rounds on social media. [Apparently](https://www.reddit.com/r/javascript/comments/3rb88w/ten_questions_ive_been_asked_most_more_than_once/cwmn60t/), [top companies](https://www.glassdoor.com/Interview/-1-Implement-a-loading-bar-that-animates-from-0-to-100-in-3-seconds-2-Start-loading-bar-animation-upon-a-button-cli-QTN_1393903.htm) are using it to screen front end engineers. My fiancée will be applying to jobs soon and I asked her to give it a go. She almost completed the final stage (with a little research) but a tricky recursion bug tripped her up. I wrote this article to help her out. Hopefully, you'll find this useful should you face similar themed questions!

There are a few variations but this question is normally asked in stages which get progressively harder.

#### 1. Implement a loading bar that animates from 0 to 100% in 3 seconds

This can be done purely with CSS. If something can be done purely with CSS I tend to go for that option. My rationale is that it's easier to refactor something that is pure CSS than trying to extend a quick JavaScript hack. CSS is very declarative, it's more straightforward to read and understand what's going on under the hood.

For a CSS only progress bar, I'm going to use two divs — a container and the progress bar — and `keyframes`. The important line here is `animation: 1s linear fill;`. There's a lot to talk about. What `transition-timing-function` are we going to use — `ease`, `linear`, a `cubic-bezier`?

At the very least, this quick answer shows that you know what `keyframes` and you can use it on a basic level.

```html
<div class="container">
  <div class="progress-bar"></div>
</div>
```

```css
.container {
  width: 300px;
  height: 50px;
  background-color: #D3D3D3;
}

.progress-bar {
  width: 100%;
  height: 50px;
  background-color: #90EE90;
  animation: 3s linear fill;
}

@keyframes fill {
    0% {
        width: 0%;
    }
    100% {
        width: 100%;
    }
}
```

__[CodePen](https://codepen.io/healeycodes/pen/pmKQzM)__

#### 2. Start loading bar animation upon a button click

Now we're moving into JavaScript land. I've always thought that `transition` was neat as heck so I'm going to use that with JavaScript to add a class to the `progress-bar` element. It enables you to "define the transition between two states of an element" [(MDN)](https://developer.mozilla.org/en-US/docs/Web/CSS/transition).

I'm making sure to cache the reference to the loading bar. I work on Smart TVs a lot and caching is one of the many tricks we use to keep everything snappy. HTML elements have [Element#classList](https://developer.mozilla.org/en-US/docs/Web/API/Element/classList) which is a fantastic API for interacting with classes. It's very safe to use. You can add multiple classes and it will only add one instance, you can also remove classes that don't exist without any errors.

[classList#toggle](https://developer.mozilla.org/en-US/docs/Web/API/Element/classList#Methods) is particularly useful. "When only one argument is present: Toggle the class value; i.e., if the class exists then remove it and return false, if not, then add it and return true" [(MDN)](https://developer.mozilla.org/en-US/docs/Web/API/Element/classList#Methods).

```html
<div class="container">
  <div class="progress-bar"></div>
</div>
<button onclick="loadBar()">Load</button>
```

```css
.container {
  width: 300px;
  height: 50px;
  background-color: #D3D3D3;
}

.progress-bar {
  width: 0%;
  height: 50px;
  background-color: #90EE90;
  transition: width 3s linear;
}

.load {
  width: 100%;
}
```

```javascript
const bar = document.querySelector('.progress-bar');

function loadBar () {
  bar.classList.add('load');
}
```

__[CodePen](https://codepen.io/healeycodes/pen/MdXzqb)__

#### 3. Queue multiple loading bars if the button is clicked more than once. Loading bar N starts animating with loading bar N-1 is done animating.

Here it gets more interesting. There is a way to iterate on our previous solution by removing and adding classes but that feels hacky. I think the intention is for you to use more JavaScript here. Technical interview questions don't really have a finish point. There's always restrictions, extensions, and what-ifs that can be thrown around. I'd love to see what you come up with before reading any further 👀.

There are a couple of traps here. You've got to make sure the time taken is exactly three seconds rather than a tick more or a tick less. And what is a good length for a tick? I suppose it depends on how wide your bar is. Going up by a percent each time seems like a sweet spot. Note: it's also better to manage your own state rather than relying on the DOM. 

Caching is really important here. You don't want to traverse the DOM for an element 33 times a second. Two functions are probably required for clean code. I went for a recursive timeout with a global flag to track whether the progress bar is running. We want to add one to the queue either way but we don't want two going at once or our bar will load twice as quick!

```html
<div class="container">
  <div class="progress-bar"></div>
</div>
<div>Queued bars: <span class="queued">0</span></div>
<button onclick="loadBar()">Load</button> 
```

```css
.container {
  width: 300px;
  height: 50px;
  background-color: #D3D3D3;
}

.progress-bar {
  width: 0%;
  height: 50px;
  background-color: #90EE90;
}
```

```javascript
const bar = document.querySelector('.progress-bar');
const queued = document.querySelector('.queued');

let loader = false;
let width = 0;
let count = 0;

function loadBar() {
  queued.innerText = ++count;
  if (loader === false) {
    bar.style.width = 0;
    tick();
  }
}

function tick() {
  loader = true;
  if (++width > 100) {
    queued.innerText = --count;
    width = 0;
    if (count < 1) {
      loader = false;
      return;
    }
  }
  bar.style.width = `${width}%`;
  setTimeout(tick, 30);
}
```

__[CodePen](https://codepen.io/healeycodes/pen/ZNRVBa)__

#### 4. Do the same thing but without timers!

Okay, they don't really ask this in the interview but someone mentioned `requestAnimationFrame` in the comments on DEV and I thought it would be fun to build an example using it while emulating the previous answer.

The logic is a lot shorter if you don't have to queue bar loads. I ended up deciding to use two coupled functions. Recently, I saw someone say that any instance of `else` is your code is a chance for a refactor. I've been thinking it over, and while no rule holds true, it has been influencing how I shape functions lately. Check it out.

```html
<div class="container">
  <div class="progress-bar"></div>
</div>
<div>Queued bars: <span class="queued">0</span></div>
<button onclick="loadBar(1)">Load</button> 
```

```css
.container {
  width: 300px;
  height: 50px;
  background-color: #D3D3D3;
}

.progress-bar {
  width: 0%;
  height: 50px;
  background-color: #90EE90;
}
```

```javascript
const bar = document.querySelector('.progress-bar');
const queued = document.querySelector('.queued');

let loading = false;
let count = 0;

function tick (timestamp, dist, duration) {
  const runtime = timestamp - starttime;
  let progress = runtime / duration;
  progress = Math.min(progress, 1);
  bar.style.width = `${dist * progress}%`;
  if (runtime > duration) {
    loading = false;
    count--;
    loadBar(0);
    return;
  }
  requestAnimationFrame (function(timestamp) {
      tick(timestamp, dist, duration)
  });
}

function loadBar (increment) {
  count += increment;
  queued.innerText = count;
  if (loading === true || count < 1) { return; }
  bar.style.width = 0;
  loading = true;
  requestAnimationFrame (function(timestamp) {
    starttime = timestamp;
    tick(timestamp, 100, 3000);
  });
}
```

__[CodePen](https://codepen.io/healeycodes/pen/EzpJNo)__

### End notes

Top marks if you have been shouting `<progress>` at me for the whole article. Yup, this [element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/progress) has been there since HTML5. You manage it by setting two attributes `max` and `value`. CSS Tricks have an [article](https://css-tricks.com/html5-progress-element/) on how to style and animate it. They also cover the different states, determinate and indeterminate — the latter meaning "progress unknown". These states are great because they give us a native way to communicate with the user.

This interview question is less about the perfect answer and more about how you communicate your thoughts as you go and the clarifications you ask about. Should there also be a numerical representation of percentage? Is this running on a low-powered device? If so, don't go up in one percent increments. Perhaps five or ten is better.

I think a good extension might be to ask an interviewee to build an interface that receives a WebSocket message describing the progress state and communicates that to the user.

What do you think of this interview question? Does it meet at the crossroads of problem-solving and browser knowledge for you?


Interview Question: Implement a Progress Bar


Premature optimization is the root of all evil. It's also the root of this article.

I like programming puzzles. I also like to go _fast_. We're going to take some LeetCode problems and solve them a few times, first improving runtime complexity in broad strokes and then looking for minor optimizations. We're after these wonderful words:

> faster than 100.00% of JavaScript online submissions

The environment we're targetting is `nodejs 10.15.0` with `--harmony` ([source](https://support.leetcode.com/hc/en-us/articles/360011833974-What-are-the-environments-for-the-programming-languages-)). The online judge system uses relatively small inputs for test cases as far as I can tell.

### First problem

[771. Jewels and Stones](https://leetcode.com/problems/jewels-and-stones/) ~ _You're given strings `J` representing the types of stones that are jewels, and `S` representing the stones you have.  Each character in `S` is a type of stone you have.  You want to know how many of the stones you have are also jewels._

A naive solution here is to loop through our stones, looping through the jewels for every stone. We'll be using standard for loops in this article as they are generally the fastest way of iterating data in JavaScript.

```javascript
let numJewelsInStones = function(J, S) {
    let myJewels = 0;
    // Jewels
    for (let i = 0; i < J.length; i++) {
        // Stones
        for (let j = 0; j < S.length; j++) { // Nested!
            if (J[i] === S[j]) {
                myJewels++;
            }
        }
    }
    return myJewels;
};
```

The runtime is quadratic, `O(N^2)`. Their online judge won't actually accept this solution! We get a big fat **Time Limit Exceeded**. Lesson? Nested for-loops should be avoided where possible.

Let's grab a [Set](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Set) to get rid of one of the loops. Reducing our runtime down to linear, `O(N)`. Looking up a Set in JavaScript is constant time, `O(1)`.

```javascript
let numJewelsInStones = function(J, S) {
    const jewels = new Set(J); // Set accepts an iterable object
    let myJewels = 0;
    for (let i = 0; i < S.length; i++) {
        if (jewels.has(S[i])) {
            myJewels++;
        }
    }
    return myJewels;
};
```

For this effort, we're rewarded with `faster than 97.84%`. I'm happy with this code. It's efficient and readable. If I needed drastically better performance, I might reach for a different technology than JavaScript. We have to walk the length of both strings at least once and there's no getting around that. We can't beat `O(N)` but we can make optimizations.

The stones and jewels are defined as letters. So `a-z` and `A-Z`. This means there are just 52 different buckets our values can fall into! We can use a boolean array instead of a Set. To convert an alphabetical letter into a number, we'll use its ASCII code point via [charCodeAt](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/charCodeAt). We'll set an index to `true` to represent a jewel.

However, there aren't boolean arrays in JavaScript. We could use a standard array and initialize it to length `52`. Or we could use [Int8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Int8Array) and allow the compiler to make additional optimizations. The typed array was ~6% faster when benchmarked with a range `0-52` of random characters entered as `J` and `S`.

Did you spot that our length is wrong? This is something I forgot as I was testing. There are seven characters between `z` and `A` on the ASCII code chart so the length required is actually 59.

![ASCII table](asciitable.png)

```javascript
let numJewelsInStones = function(J, S) {
    const jewels = new Int8Array(59);
    for (let i = 0; i < J.length; i++) {
        jewels[J.charCodeAt(i)-65] = 1;
    }
    let myJewels = 0;
    for (let i = 0; i < S.length; i++) {
        if (jewels[S.charCodeAt(i)-65] === 1) {
            myJewels++;
        }
    }
    return myJewels;
};
```

Et voila, our `100% fastest` [submission](https://leetcode.com/submissions/detail/229804303/). In my tests, this was actually twice as faster as the Set version. Other optimizations I skipped testing were caching lengths, using a while loop instead of a for loop, and placing the incrementor before the number (`++myJewels` vs `myJewels++`).

### Second problem

[345. Reverse Vowels of a String](https://leetcode.com/problems/reverse-vowels-of-a-string/) ~ _Write a function that takes a string as input and reverse only the vowels of a string._

A naive solution for this might be to loop through the array twice, replacing on the second loop. Let's try that out first.

```javascript
let reverseVowels = function(s) {
    const vowels = new Set(['a','e','i','o','u', 'A', 'E', 'I', 'O', 'U']);
    const reversed = [];
    let vowelsFound = [];
    // Find any vowels
    for (let i = 0; i < s.length; i++) {
        if (vowels.has(s[i])) {
            vowelsFound.push(s[i]);
        }   
    }
    // Build the final string
    for (let i = 0; i < s.length; i++) {
        if (vowels.has(s[i])) {
            reversed.push(vowelsFound.pop());
        } else {
            reversed.push(s[i]);
        }
    }
    return reversed.join('');
};
```

This nets us `faster than 97.00%`. The runtime is linear, `O(2N) -> O(N)`, and it reads well but I can't help but think we're looping the string one more time than we have to. Let's try a two-pointer approach. Walking in, step-by-step, from the front and back at the same time, swapping any vowels we see. If there's a middle vowel we just leave it.

```javascript
let reverseVowels = function(s) {
    const vowels = new Set(['a','e','i','o','u', 'A', 'E', 'I', 'O', 'U']);
    s = s.split('');
    let front = 0;
    let back = s.length - 1;
    while (front < back) {
        if (!vowels.has(s[front])) {
            front++;
            continue;
        }
        if (!vowels.has(s[back])) {
            back--;
            continue;
        }
        let temp = s[front];
        s[front] = s[back];
        s[back] = temp;
        front++;
        back--;
    }
    return s.join('');
};
```

We've reduced a full iteration! This gets us `faster than 98.89%` and it's at this point that we need to remember that LeetCode's benchmarks aren't conclusive nor are they consistent. It's not feasible for them to run a large number of iterations with a mixture of test cases. If you're practicing your puzzle solving, stop at `97%` and up. But that's not the point of this article, and, reader, I'm going to get that `100%` for you.

First I threw out the Set. The number of vowels is constant and we don't need all that hashing going on. I tried a switch statement but then found a chained if statement was faster. I discovered that in-lining this logic was faster than a function. I then reduced this down to an expression. What I'm trying to say is: the code coming up is gross. It's close-down-your-IDE-and-talk-a-walk gross. But .. [it's](https://leetcode.com/submissions/detail/229918811/) `faster than 100.00%`.

```javascript
let reverseVowels = function(s) {
    s = s.split('');
    let front = 0;
    let back = s.length - 1;
    while (front < back) {
        if (s[front] !== 'a' &&
            s[front] !== 'e' &&
            s[front] !== 'i' &&
            s[front] !== 'o' &&
            s[front] !== 'u' &&
            s[front] !== 'A' &&
            s[front] !== 'E' &&
            s[front] !== 'I' &&
            s[front] !== 'O' &&
            s[front] !== 'U') {
            front++;
            continue;
        }
        if (s[back] !== 'a' &&
            s[back] !== 'e' &&
            s[back] !== 'i' &&
            s[back] !== 'o' &&
            s[back] !== 'u' &&
            s[back] !== 'A' &&
            s[back] !== 'E' &&
            s[back] !== 'I' &&
            s[back] !== 'O' &&
            s[back] !== 'U') {
            back--;
            continue;
        }
        let temp = s[front];
        s[front++] = s[back];
        s[back--] = temp;
    }
    return s.join('');
};
```

(I'm sorry).

### Third problem

[509. Fibonacci Number](https://leetcode.com/problems/fibonacci-number/) ~ _Calculate the nth Fibonacci number_.

This is a common puzzle and it was the hardest to improve the runtime for because there are so few moving parts in the final solution. I'm sure some RNG was involved with LeetCode's grading too. Let's get the naive solution out of the way. The Fibonacci sequence is often used to teach recursion. However, the algorithm that is used has a runtime of `O(2^n)` (_very_ slow).

I actually crashed a browser tab by trying to calculate the 50th term with this function.

```javascript
let fib = function(N) {
    if (N < 2) {
        return N;
    }
    return fib(N - 1) + fib(N - 2);
}
```

We get `faster than 36.63%` for this answer. Ouch. In production, this is the kind of puzzle that can be solved by memoization (caching some of the work for later). This is the best solution because we only calculate up to the values that we need in linear time `O(N)` and then running the algorithm again for a term under that limit is constant time `O(1)`.

```javascript
const memo = [0, 1];
let fib = function(N) {
    if (memo[N] !== undefined) {
        return memo[N];
    }
    const result = fib(N - 1) + fib(N - 2);
    memo[N] = result;
    return result
};
```

`faster than 94.25%`. LeetCode doesn't store data between each run-through of our code so we'll have to try something different. We've interested in calculating *one* number of the sequence just *once*. I think we can throw away that array. Let's look at the iterative solution.

```javascript
let fib = function(N) {
    if (N < 2) {
        return N;
    }
    let a = 1;
    let b = 1;
    for (let i = 3; i <= N; ++i) {
        a = a + b;
        b = a - b;
    }
    return a;
};
```

If this looks a little different to other iterative versions you might have seen, it's because I avoided the third temporary variable that we have to use in JavaScript to swap values (there are other methods as well but they're too slow). I did some benchmarks and I found using arithmetic instead [was](https://leetcode.com/submissions/detail/229923152/)..  `faster than 100.00%`.


Solving Puzzles With High-Performance JavaScript


When I was starting out, I never understood how to tackle 'projects'. It was hard to understand the scope of a project from a glance, as well as the steps and considerations. I wanted to see behind the scenes. We all work differently but in this article I'm showing how I work when I get an idea-spark for something tiny. Come watch me scratch an itch.

I was on the MDN Web Docs earlier looking for something to spark a research journey for this week's article when I found that there was no random page link. A random page function would be useful for general revision of web technologies, allowing you to click through to things that sounded interesting.

![The project's main/only page](rnd-mdn-preview.png)

I began my journey by hacking away at the [index page](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Index) for JavaScript. This is often how I tackle front end web problems at my job too, using DevTools as the draft version of a site.

![Looking at the table rows](index-hacking.png)

In the HTML there's a table body located at `document.querySelector("#wikiArticle > table > tbody")`. On Chrome, you can right-click an element and go `Copy -> Copy JS Path` to find this. Each entry uses three table rows. I'm after the page link and also the summary. Let's separate the rows into groups that make up one entry each.

You can run these lines in your console.

```javascript
// Index table
const indexTable = document.querySelector("#wikiArticle > table > tbody");

// There are three <tr>s to a row
const allRows = indexTable.querySelectorAll('tr');
const chunkRows = [];
for (let i = 0; i < allRows.length; i += 3) {
    chunkRows.push([allRows[i], allRows[i + 1], allRows[i + 2]]);
}
```

I need to store a copy of the index page's data somewhere. Scraping the page per each request would be wasteful and slow. Storing them as JSON makes the most sense as I'll be delivering this over the web. A page snippet should have a link property and a text property. I'll take care of some of the templating here too.

```javascript
// Map these chunked rows into a row object
const pageEntries = [];
chunkRows.forEach(row => {
    const a = row[0].querySelector('a');
    const page = {
        link: `<a class="page-link "href="${a.href}">${a.innerText}</a>`,
        text: `<div class="page-text">${row[1].querySelector('td').innerText}</div>`
    }
    pageEntries.push(page);
})
```

I needed to decide how I was going to deliver this to users. I considered a Chrome extension but I wanted to prototype faster so I turned to Glitch and used their Express template.

I packaged the scraping logic into an async function and used [puppeteer](https://github.com/GoogleChrome/puppeteer) to run these commands in the context of the index page. (On Glitch, you have to use `--no-sandbox` which can be a security risk.). In the finished project, this script can be called manually with `node getNewPages.js` to update the entries on disk. (This would fit neatly into a cron job!).

From Chrome DevTool hackery to something a little more cohesive.

```javascript
// getNewPages.js

const fs = require('fs');
const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch({
      args: ['--no-sandbox'] // Required for Glitch
    });
    const page = await browser.newPage();
    await page.goto('https://developer.mozilla.org/en-US/docs/Web/JavaScript/Index');

    // Scan index table, gather links and summaries
    const pageEntries = await page.evaluate(() => {

        // Index table
        const indexTable = document.querySelector("#wikiArticle > table > tbody");

        // There are three <tr>s to a row
        const allRows = indexTable.querySelectorAll('tr');
        const chunkRows = [];
        for (let i = 0; i < allRows.length; i += 3) {
            chunkRows.push([allRows[i], allRows[i + 1], allRows[i + 2]]);
        }

        // Map these chunked rows into a row object
        const pageEntries = [];
        chunkRows.forEach(row => {
            const a = row[0].querySelector('a');
            const page = {
                link: `<a class="page-link "href="${a.href}">${a.innerText}</a>`,
                text: `<div class="page-text">${row[1].querySelector('td').innerText}</div>`
            }
            pageEntries.push(page);
        })
        return pageEntries;
    });

    // Save page objects to JSON on disk
    const pageJSON = JSON.stringify(pageEntries);
    fs.writeFile('./data/pages.json', pageJSON, function (err) {
        if (err) {
            return console.log(err);
        }
        console.log(`New pages saved! (JSON length: ${pageJSON.length})`);
    });

    await browser.close();
})();
```

You can load JSON files into Node with `require('./data/pages.json')` which keeps all of our page entries in memory for faster access (this is possible due to the fixed, small size of ~300kb). The rest of our web app is a wrapper around a random function.

```javascript
// server.js

// init project
const express = require('express');
const app = express();
app.use(express.static('public'));

const pages = require('./data/pages.json');
const rndPage = () => pages[Math.floor(Math.random() * pages.length)];

app.get('/', (req, res) => res.sendFile(__dirname + '/views/index.html'));

app.get('/rnd', (req, res) => res.send(rndPage()));

// listen for requests :)
const listener = app.listen(process.env.PORT, function() {
  console.log('Your app is listening on port ' + listener.address().port);
});
```

The only client-side JavaScript we need is a `fetch` call to update the page link and summary. As well as a button with a cute wobble animation to flip through the entries. There's a 1 in 897 chance of getting the same snippet twice, how would you solve this Let me know in the comments!

**Where to go from here:** there are other pages on the MDN Docs (other than JavaScript) that follow the same row pattern, meaning they can be scraped simply by changing the URL in our script.

You can [remix ](https://random-mdn-page.glitch.me/) the project to play with a copy of the code live or clone the repo: [healeycodes/random-mdn-page](https://github.com/healeycodes/random-mdn-page).

A Tiny Project, From Inception to Deployment


We're going to use Glitch as a free container platform to host our application. Better yet, it will have continuous deployment! Whenever we commit to our `master` branch on GitHub, a secure webhook will be sent to our project, which will update, build, and restart itself.

The route that receives this webhook is built into our Node.js/Express application. We'll be using Glitch's `hello-express` template to keep things simple. Inside the route, we need to run our git commands to get our updated files as well as any build and install commands.

*Note: Glitch auto-restarts Node.js projects by running `npm start`.*

![Choosing the template](create-express.png)

Create a new `hello-express` project on Glitch and add your secret to the `.env` file by adding `SECRET='randomized password here'`. There's one additional package we need, which can be installed via the Glitch UI inside package.json, or via console with `npm install body-parser`.

This is the `POST` route, along with including extra imports:

```javascript
const bodyParser = require('body-parser');
app.use(bodyParser.json());
const crypto = require('crypto');
const { execSync } = require('child_process');

app.post('/git', (req, res) => {
  const hmac = crypto.createHmac('sha1', process.env.SECRET);
  const sig  = 'sha1=' + hmac.update(JSON.stringify(req.body)).digest('hex');
  if (req.headers['x-github-event'] === 'push' &&
    crypto.timingSafeEqual(Buffer.from(sig), Buffer.from(req.headers['x-hub-signature']))) {
    res.sendStatus(200);
    const commands = ['git fetch origin master',
                      'git reset --hard origin/master',
                      'git pull origin master --force',
                      'npm install',
                      // your build commands here
                      'refresh']; // fixes glitch ui
    for (const cmd of commands) {
      console.log(execSync(cmd).toString());
    }
    console.log('updated with origin/master!');
    return;
  } else {
    console.log('webhook signature incorrect!');
    return res.sendStatus(403);
  }
});
```

We're interested in push events so we check the header. After, we perform a security check by creating an HMACSHA1 keyed hash with our secret and the webhook's body which will be repository information from GitHub. GitHub sends over a signature of the same body using our secret. We compare the signature with `timingSafeEqual` to avoid timing attacks. Hopefully, all is well and the two match up. Otherwise, we'll send a `403` code and skip the update.

If all is well, we send a `200` code back to GitHub and run our commands. We do this synchronously, using `execSync` since they depend upon each other, and we log the results in case there's any errors or information we need later. `refresh` is a Glitch environment command that resets the UI and fixes the file-tree in the sidebar (which plagued me for about half an hour this weekend!). The update process generally takes 5-15 seconds for small changes. Your application won't be available during this time.

![Exporting to GitHub](export-to-github.png)

Creating the GitHub webhook is actually quite straightforward but first some preparation. You'll need a repository with at least one file so that Glitch can export to it. Once you've got your repository set up, add it as the remote origin via Glitch console `git remote add origin {url}` and send over your project. It will become the `glitch` branch. Create a pull request and merge this to `master`. Otherwise, when the webhook fires, your project might load the wrong `master` changes and reset itself!

*I've seen some people set up this process pulling straight from the `glitch` branch but I prefer having `master` as the production version of a project.*

![Creating the webhook](create-webhook.png)

Head into the repository settings and create a new webhook with your Glitch project's secret. Make sure to choose `application/json` as the content-type. And that's it. Any `push` events with the correct signature will trigger an update/install/restart cycle of your Glitch project 🎉.

You can see what this looks like in a live project on the [PairCode repository](https://github.com/healeycodes/PairCode). It's a lightweight CodePen clone that I wrote at university and have been tinkering with lately. Glitch helped bring it back to life!

Reach out if you're having any tricky issues 👍.


Joining two of my favorite ecosystems together.

Continuous Deployment to Glitch via GitHub Webhooks


Something a lot of beginners struggle with is the concept of passing data between different programming languages. It's far more simple to understand that a value exists in a variable which can be passed around from function to function. However, to go beyond the program's edges we must serialize our data in some way. We'll look at two ways that these two languages can communicate. AJAX requests via the new Fetch API, and piping between local processes.

Serializing data means taking a value, object, or data structure, and translating it into a format that can be stored or transmitted. Most importantly, it needs to be put back together at the other end. Let's take look at JavaScript Object Notation (JSON). JSON is a human-readable format and is straightforward for machines to read and write. The [specification](https://www.json.org/) is small enough to read during one cup of coffee. Both JavaScript and Python have standard library methods to parse and write JSON.

## JSON Crash Course

JSON is built on two data structures. **Objects** — key/value pairs like a JavaScript `Object`, and a Python `Object` or `Dictionary`. **Arrays** — a series of data like a JavaScript `Array`, and a Python `List`.

```javascript
/* JavaScript
   Try this out in your developer console! */

const person = {"name":"Andrew", "loves":"Open Source"};
const asJSON = JSON.stringify(person);

// `person` is of type 'object'
console.log(`person is of type ${typeof person}`);

// `asJSON` is of type 'string'
console.log(`asJSON is of type ${typeof asJSON}`);

// We can convert it back to an object by parsing it
// `asObject` is of type 'object'
const asObject = JSON.parse(asJSON);
console.log(`asObject is of type ${typeof asObject}`);
```

Let's do the same in Python by using the standard library module `json`.

```python
# python

animal = {'type':'cat', 'age':12}
as_json = json.dumps(animal)

print(type(animal))  # prints '<class 'dict'>'
print(type(as_json))  # prints '<class 'str'>'

# now back again
as_object = json.loads(as_json)
print(type(as_object))  # prints '<class 'dict'>'
```

**Recap:** in JavaScript, you serialize to JSON with `JSON.stringify()` and parse with `JSON.parse()`. This works in the browser as well as Node.js. In Python, first import the `json` module then serialize with `json.dumps()` and parse with `json.loads()`.

## Talking via AJAX

In the past, this would be done with [XMLHttpRequest](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest) but the relatively new [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) is far more pleasant to use. First, we'll code up a small Python web server, and then we'll look at passing JSON back and forth with the browser.

[Flask](http://flask.pocoo.org/) is a 'microframework'. It's not only extremely fun to use, it's also great to prototype with. We'll use its [jsonify](http://flask.pocoo.org/docs/1.0/api/#flask.json.jsonify) module which writes/parses JSON as well as setting the correct response headers (an application/json mime type). It requires two [OS-specific commands](http://flask.pocoo.org/docs/1.0/quickstart/) to install and run a debug server. For myself on OS X, they were `pip install flask` and `FLASK_APP=app.py flask run`.

Code along or fork [this repository](https://github.com/healeycodes/talking-between-python-and-js) to grab the code for all the examples.

```python
# app.py
from flask import Flask, jsonify, request, render_template
app = Flask(__name__)

@app.route('/hello', methods=['GET', 'POST'])
def hello():

    # POST request
    if request.method == 'POST':
        print('Incoming..')
        print(request.get_json())  # parse as JSON
        return 'OK', 200

    # GET request
    else:
        message = {'greeting':'Hello from Flask!'}
        return jsonify(message)  # serialize and use JSON headers

@app.route('/test')
def test_page():
    # look inside `templates` and serve `index.html`
    return render_template('index.html')
```

With our server running and serving us a page we can run JavaScript on, let's talk in JSON! We'll send a GET request with the Fetch API and receive a greeting from Flask. Before writing a single line of client-side code, I always use [Postman](https://www.getpostman.com/) to test my web servers — it's free and one of the industry standard tools for API testing and development.

I'm running the following snippets in `<script>` tags inside `templates/index.html`. There's nothing else inside `index.html` so the rendered page is blank. Everything happens in the console.

```javascript
// GET is the default method, so we don't need to set it
fetch('/hello')
    .then(function (response) {
        return response.text();
    }).then(function (text) {
        console.log('GET response text:');
        console.log(text); // Print the greeting as text
    });

// Send the same request
fetch('/hello')
    .then(function (response) {
        return response.json(); // But parse it as JSON this time
    })
    .then(function (json) {
        console.log('GET response as JSON:');
        console.log(json); // Here’s our JSON object
    })
```

Awesome! We've got Python talking to client-side JavaScript using JSON for data serialization. Let's flip it and send JSON to Python from the browser. We'll use the Fetch API again but it will be a POST request instead of GET.

**Beginner tip:** Remember the difference between POST and GET. When you POST mail, you head to the post office with your letter filled with information. When you GET mail, you head to the post office again but this time you're picking up something that’s been left for you.

```javascript
// POST
fetch('/hello', {

    // Declare what type of data we're sending
    headers: {
      'Content-Type': 'application/json'
    },

    // Specify the method
    method: 'POST',

    // A JSON payload
    body: JSON.stringify({
        "greeting": "Hello from the browser!"
    })
}).then(function (response) { // At this point, Flask has printed our JSON
    return response.text();
}).then(function (text) {

    console.log('POST response: ');

    // Should be 'OK' if everything was successful
    console.log(text);
});
```

With these two core building blocks, we've conquered JSON communication via HTTP. However, do note that you should be adding `catch` to the end of these Promises.  I've only trimmed them for clarity. It's better to handle errors gracefully so we can tell the user that they are disconnected or that there's an error on our end. Docs for catch [here](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/catch).

If you're talking to a Node.js web server with Python, you will probably reach for the [requests](http://docs.python-requests.org/en/master/) module, which has syntax almost identical to the Fetch API.

## Talking via processes

We're going to spawn processes (both ways) so we can see what communication between Node.js and Python looks like. We'll listen to the [stdout](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout)) stream of the child process in both instances. Let's imagine that we have a program that reports data at irregular intervals. A temperature sensor. We want to listen to that program and store the values it reports.

Here's our fake sensor program in Python. It prints data to stdout. We'll catch it in Node.js.

```python
# sensor.py

import random, time
while True:
    time.sleep(random.random() * 5)  # wait 0 to 5 seconds
    temperature = (random.random() * 20) - 5  # -5 to 15
    print(temperature, flush=True, end='')
```

When piping information in this way, it's important to flush the stream so it reaches stdout when you expect it to ([why do we fflush() in C?](https://www.quora.com/Why-do-we-use-the-functions-fflush-stdin-and-fflush-stdout-in-c)) . More information on Python flushing [here](https://stackoverflow.com/a/35467658). We also make sure that the end of the printed statement contains no extra information (even though `parseFloat()` would clean it!) by default it would be the newline character `\n`.

So, we're Node.js and we want the current temperature as it's reported. Let's spawn `sensor.py` as a process and listen for the stdout event. _Piping_ the data between the two running languages.

```javascript
// temperature-listener.js

const { spawn } = require('child_process');
const temperatures = []; // Store readings

const sensor = spawn('python', ['sensor.py']);
sensor.stdout.on('data', function(data) {
    
    // Coerce Buffer object to Float
    temperatures.push(parseFloat(data));

    // Log to debug
    console.log(temperatures);
});
```

## Flip it and reverse it

Now, let's flip those roles. A Node.js sensor and a Python listener! This time we'll try a different method, using a newline character (`\n`) to delimit the different readings instead of waiting for an event. We'll add the data to a buffer until we hit a newline char. Once we do, we've collected a full reading and we can store it.

First, the equivalent sensor in Node.js.

```javascript
// sensor.js

function reportReading() {
    const temperature = (Math.random() * 20) - 5; // Range of -5 to 15
    process.stdout.write(temperature + '\n'); // Write with newline char
    setTimeout(reportReading, Math.random() * 5000); // Wait 0 to 5 seconds
}
reportReading();
```

Now in Python, a temperature listener program that will spawn the above code as a process.

```python
# temperature-listener.py

import sys
from subprocess import Popen, PIPE

temperatures = []  # store temperatures
sensor = Popen(['node', 'sensor.js'], stdout=PIPE)
buffer = b''
while True:

    # read sensor data one char at a time
    out = sensor.stdout.read(1)

    # after a full reading..
    if out == b'\n':
        temperatures.append(float(buffer))
        print(temperatures)
        buffer = b''
    else:
        buffer += out  # append to buffer
```

You can run `node temperature-listener.js` or `python temperature-listener.py` and the result will be the same. The array of temperatures will grow as new data arrives, and our debug log line will result in the following output.

```js
[ 3.8075910850643098 ]
[ 3.8075910850643098, -1.5015912681923482 ]
[ 3.8075910850643098, -1.5015912681923482, 11.97817663641078 ]
```

We've seen two different ways of communicating between Python and JavaScript but if either of these aren't for you — don't fret! There are many ways to pass data between these two languages. Not limited to: [named pipes](https://en.wikipedia.org/wiki/Named_pipe), TCP sockets, WebSockets, and file polling.


Let's learn how to pass data back and forth between languages.

Talking to Python from JavaScript (and Back Again!)


Let's learn about `window.localStorage` by building a tiny note-taking app with some simple HTML, CSS, and JavaScript. We'll code up a small UI and autosave the user's notes to their browser. My co-worker keeps all of his snippets on something just like this. I can't believe the mileage he gets out of 50-60 lines of code!

Most client-side storage interactions used to involve manipulating cookies but luckily we now have modern APIs. With cookies, the user sends everything stored with every request meaning worse performance for both parties.

Enter: `localStorage`. A key/value API that's available on the `window` object. Unlike `sessionStorage` which lasts for the during of the page session, `localStorage` has no expiration time.

All keys and values are converted to Strings. Let's add, update, and delete a key, and then try to read from an empty key. This is console input so the return value follows the command.

```javascript
// Add key - or overwrite existing key
window.localStorage.setItem('foo', 'bar')
> undefined

// Read a value from a key
window.localStorage.getItem('foo')
> "bar"

// Delete key - deleting empty yields same return value
window.localStorage.removeItem('foo')
> undefined

// Read from a deleted or unknown key
window.localStorage.getItem('foo')
> null
```

Other ways you might interact with localStorage involve clearing all keys and iterating through existing keys.

```javascript
// Get the read-only length
window.localStorage.length
> 6

// Get the nth key
window.localStorage.key(0)
> "algoliasearch-client-js"

// Delete all keys
window.localStorage.clear()
> undefined

// Iterate through all keys
for (let i = 0; i < window.localStorage.length; i++) {
    console.log(window.localStorage.key(i));
}
```

You can try some of these out in your developer console right here. DEV stores the following localStorage keys for me as I'm writing. Make sure **not to delete** any of these though! `algoliasearch-client-js`, `config_body_class`, `current_user`, `https://dev.to/new`, `pusherTransportTLS`.

#### The App

Here's what we'll be making. A small but practical note-taking app. Aside from spacing, it uses little styling ([send a pull request](https://github.com/healeycodes/tiny-note-taker)). What's important is how the app communicates with the client's browser storage.

[![Preview of tiny note taker](tiny-note-taker-preview.png)](https://healeycodes.github.io/tiny-note-taker/)

Hopefully, you're already thinking of the `localStorage` methods that we'll need. Let's use a ready-built HTML template that names the functions that we'll be using.

```html
<h5>Tiny Note Taker</h5>
<div>
    <button onclick="newNote()">New</button>
        <select id="notes" onchange="changeNote()"></select>
        <button onclick="deleteNote()">Delete</button>
</div>
<textarea id="editor" rows="10" onkeyup="saveNote()"
    onkeydown="checkEmpty()" autofocus></textarea>
<script src="./index.js"></script>
```

Take a second to think about how these functions might relate to the storage methods we saw just now, and how they will manipulate the DOM.

<hr>

`newNote()` Prompt the user for a note name and create it.

`changeNote()` Swap to the new note when the selected element is changed.

`deleteNote()` Delete the currently selected note.

`saveNote()` Save the currently selected note as the user types.

`checkEmpty()` — This one is less obvious.

If a user arrives at the page and starts typing before creating a new note, the name would be `""` an empty String. Instead, as they type _but before the note is saved_ name the current note `untitled`. We simplify our logic here by separating it into two events. `onkeydown` checks for an empty note and fixes if necessary. This means that `onkeyup` can simply save the note without any additional checks. Thus, `checkEmpty()`.

**Reminder**: Key events are [almost always](https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent#Usage_notes) called in this order `onkeydown -> onkeypress -> onkeyup` and they all have their place!

Let's start with our base case. Saving one note as the user types. Since we've placed the script include at the bottom of the HTML, we can grab a reference to the two elements we need without waiting for any document load events.

```javascript
const notes = document.querySelector('#notes');
const editor = document.querySelector('#editor');
```

Great. Those references will last for the lifetime of this script as we'll never delete those elements. Next, we need to title the empty-named note as untitled if the user types without first hitting the New button.

```javascript
/**
 * Check for empty note title.
 */
function checkEmpty() {
    if (notes.length === 0) {
        const untitled = document.createElement('option');
        untitled.innerText = 'untitled';
        notes.appendChild(untitled);
    }
}
```

Here we create an option element with JavaScript and make it available in the select dropdown. We don't need to do any saving yet, that will happen a few milliseconds later when the user lifts their finger off the current key.

**A few milliseconds later.**

```javascript
/**
 * Save editor text to storage under the current note.
 */
function saveNote() {
    window.localStorage.setItem(notes.value, editor.value);
}
```

There's `localStorage.setItem`! We know what that does. It will save the note text as the value and the note title (`"untitled"`) as the key. Now we have a problem to solve. If the user leaves and comes back, the select dropdown will be empty. Worse: they will overwrite their `"untitled"` note.

As our script runs, just after we grab references to those elements, let's populate the select dropdown with all the user's saved notes.

```javascript
const notes = document.querySelector('#notes');
const editor = document.querySelector('#editor');

// Load user's saved notes
for (let i = 0; i < window.localStorage.length; i++) {
    const newNote = document.createElement('option');
    newNote.innerText = window.localStorage.key(i);
    notes.appendChild(newNote);
}
changeNote();
```

Here we use the read-only `length` property of `localStorage` as well as its method `key()`. All saved notes are loaded into the select dropdown. We let the HTML operate as our state. This way we don't need to keep track of any of the notes in our JavaScript. We rely on simple functions to link up the DOM to the Storage API.

At the bottom of that last snippet, we saw `changeNote()` which loads the currently selected note into the textarea element a.k.a. the editor. This function is also called after a new note is created, which clears the editor. New notes are initialized with an empty String.

```javascript
/**
 * Change editor text to the currently selected note.
 */
function changeNote() {
    editor.value = window.localStorage.getItem(notes.value);
}
```

The Storage API abstracts away the heavy lifting for us.

User's need a way to name and create notes. `newNote()` does just that. It uses [Element#insertBefore](https://developer.mozilla.org/en-US/docs/Web/API/Node/insertBefore) to place the new note at the top of the selectbox. This method is called on the parent element. `let insertedNode = parentNode.insertBefore(newNode, referenceNode);`. What's neat about this method is that it accepts a null value as the second argument. Meaning we can call `insertBefore` on empty parent elements!

```javascript
/**
 * Ask the user to name their new note then create it.
 * Add this note to the select div then focus to it.
 */
function newNote() {
    const note = prompt('Name of note?');
    window.localStorage.setItem(note, '');

    const noteElem = document.createElement('option');
    noteElem.innerText = note;
    notes.insertBefore(noteElem, notes.firstChild);

    // Focus this note
    notes.value = note;
    changeNote();
}
```

Our note taking app is nearly finished. The final function required is for deleting notes. When a user is finished with a note, they can select it and then hit the Delete button. The trickiest part about this function is finding the right note to delete in the select dropdown. One of the drawbacks to using the DOM as our state. We have to loop through every option, checking the `value`.

```javascript
/**
 * Delete currently selected note
 */
function deleteNote() {
    const note = notes.value;
    window.localStorage.removeItem(note);
    editor.value = '';
    for (let i = 0; i < notes.length; i++) {
        const option = notes[i];
        if (option.value === note) {
            notes.removeChild(option);
        }
    }
}
```

We've now got a fully functioning note taking app, infinitely scalable through a CDN. We used the powerful [Storage API](https://developer.mozilla.org/en-US/docs/Web/API/Storage) and learnt how to use the DOM as application state. Go us! 🙌

Check out the final application code [here](https://github.com/healeycodes/tiny-note-taker), and the live version [here](https://healeycodes.github.io/tiny-note-taker/).


Learn the Storage API by Building a Note Taking App (Tutorial)


Let's look at how the DEV.to API can be used to embed a list of your recent posts. We'll also sort them by their positive reactions! All it takes is a splash of client-side JavaScript. The API is [not released](https://github.com/thepracticaldev/dev.to/issues/2187) yet, is not documented, and will likely change but I'll keep this post up to date.

Here's an [test page](https://healeycodes.github.io/embed-DEV-posts/example/) which uses the tiny library we'll be writing. We'll be creating a plain list to keep things extendable. You can also skip straight to the [repository](https://github.com/healeycodes/embed-DEV-posts) to check out the final code.

[![](test-embedded.png)](https://healeycodes.github.io/embed-DEV-posts/example/)

#### How easy is the DEV API to use?

[Easy](https://dev.to/api/articles?username=healeycodes). The articles route is `https://dev.to/api/articles?username=$user`. Originally, I experimented with getting more than the latest 30 posts, which is the default. This can be done by adding `&page=$pageNum` — but requesting multiple pages introduces a delay. Since it's impossible to know the number of pages of posts, you need to keep going until you hit an empty page. A late-loading list doesn't make for great UX.

Let's start by using the [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) to make a request from the user's browser.

```javascript
fetch(`https://dev.to/api/articles?username=healeycodes`)
    .then(function (response) {

        // Parse it as JSON
        return response.json();
    }).then(function (posts) {

        // An array of posts
        console.log(posts);
    })
```

Looking good. Let's neaten this up with some [JSDoc](http://usejsdoc.org/about-getting-started.html) comments so the code is more accessible. We'll also wrap it in a function.

```javascript
/**
 * Get a DEV user's post objects.
 * Only fetches previously 30 posts. Using `&page=X` is too slow.
 * @param {string} username - DEV username, e.g. 'ben'.
 * @returns {array} User's post objects.
 */
function getPosts(username) {

    // Assume that `api` is defined as a constant
    return fetch(`${api}/articles?username=${username}`)
        .then(function (response) {
            return response.json();
        })
        .then(function (posts) {

            // Sort the array in place with a compare function
            return posts.sort((a, b) => b.positive_reactions_count - a.positive_reactions_count);
        });
}
```

A list of 30 posts is maybe a bit much. We'll cut it down later. For now, notice how we sorted the array in place with a [compareFunction](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort#Parameters). We used a shorthand version. Compare functions are typically written by returning `-1`, `1`, or `0` when comparing the two parameters. However, all that matters is that the compare function is consistent.

We want to create a simple list that can be styled with CSS. Sensible class names might be `dev-feed-list` and `dev-feed-item`. We'll need an element to attach this list to as well so that should be an argument that our library takes. With JavaScript, we can add classes, and create lists and attach them on the fly like this:

```javascript
// An unordered list
const list = document.createElement('ul');

// A list item
const item = document.createElement('li');
item.classList.add('dev-feed-item');
item.innerText = 'I am a list item.';

// Let's create a relation between them
list.appendChild(item);

// Now between the list and the document, rendering the list
// body -> ul -> li
document.body.appendChild(list);
```

Let's create a function that builds the elements that will make up our embedded list. We'll use the `getPosts` function from earlier to get our sorted array of posts.

```javascript
/** 
 * Creates a DEV feed of posts.
 * @constructor
 * @param {Element} elem - Place list of posts inside this element.
 * @param {string} username - DEV username, e.g. 'ben'.
 * @param {number} numberOfPosts - Number of posts to list. 
 */
function createFeed(elem, username, numberOfPosts = 5) {
    const feed = document.createElement('ul');
    feed.classList.add('dev-feed-list');
    getPosts(username)

        // With our posts array
        .then(function (posts) {
            posts.length = numberOfPosts;
            posts.forEach(function (post) {

                // We create an element for each item
                const item = document.createElement('li');
                item.classList.add('dev-feed-item');

                // As well as a link for users to click through
                const link = document.createElement('a');
                link.href = post.url;
                link.innerText = post.title;

                // ul -> li -> a
                item.appendChild(link);
                feed.appendChild(item);
            });

            // Attach the list of posts, rendering it
            elem.appendChild(feed);
        });
}
```

If we bundle this all together as a script, we can generate lists by calling `createFeed(element, username, numberOfPosts)` where `element` is the container for the list and `username` is a valid DEV user, and `numberOfPosts` is the number of posts we want to render.

By allowing a custom number of posts to be fetched, as well as using sensible CSS classes, our library is extensible and can be used as a tiny module!


Embed Your Latest DEV Posts Anywhere (Tutorial)


_Dust off your ternary expressions, we're going in._

One-liners are tricky to maintain (and sometimes even hard to understand) but that doesn't stop them from being cool as hell. There's a certain satisfaction that comes after writing a terse solution.

This is a collection of some of my recent favorites. They will all run in your dev console so pop it open and try them out. I hope you'll share some of your own favorites in the comments!


#### Calendar Hacking

Ali Spittel [tweeted](https://twitter.com/ASpittel/status/1110548407800815617) this out recently. It solves a problem I've faced multiple times. If you swap the minus for a plus, it gives you the next seven days.

```javascript
// Create an array of the past seven days, inclusive
[...Array(7).keys()].map(days => new Date(Date.now() - 86400000 * days));
```


#### Random ID generation

This is my go-to function for creating UUIDs when prototyping. I've even seen people using it in production in the past. It's not secure but ... there are worse random generation functions out there.

```javascript
// Generate a random alphanumerical string of length 11
Math.random().toString(36).substring(2);
```


#### Quines

A [quine](https://en.wikipedia.org/wiki/Quine_(computing)) is a program that outputs its own source code. Quines have always fascinated me. I've got pretty close to completing my own quines a couple of times in different languages but details are the name of the game.

I've picked some winners for you. Credit to [Mama Fun Roll](https://codegolf.stackexchange.com/a/60148/78322), [PleaseStand](https://codegolf.stackexchange.com/a/270/78322), and [Peter Olson](https://codegolf.stackexchange.com/a/3173/78322) respectively for these three.

```javascript
// $=_=>`$=${$};$()`;$()
$=_=>`$=${$};$()`;$()

// eval(I="'eval(I='+JSON.stringify(I)+')'")
eval(I="'eval(I='+JSON.stringify(I)+')'")

// For those who like their quines via alert
// (function a(){alert("("+a+")()")})()
(function a(){alert("("+a+")()")})()
```


#### Scrape query parameters

Talk about non-maintainable code. This converts the page's query parameters to an object in 78 bytes! Thanks to Alex Lohr for code golfing it.

`?foo=bar&baz=bing` becomes `{foo: bar, baz: bing}`

```javascript
// Set the current page's query parameters to `q`
q={};location.search.replace(/([^?&=]+)=([^&]+)/g,(_,k,v)=>q[k]=v);q;
```

I'd like to see a minifier work that hard. 


#### Working clock

With a sprinkle of HTML, you can create a working clock with source code you could read out in one breath. I wrote this after a challenge from a co-worker. It ticks every second, updating the page with the current time.

```html
<body onload="setInterval(()=>document.body.innerHTML=new Date().toGMTString().slice(17,25))"></body>
```


#### Shuffle an array

Until Pythonistas show up with their `import random`, `random.shuffle(array)` solution, we're going to enjoy what we have. This has the bonus of having an infinitesimal chance of being an infinite loop (implementation depending). [Don't use in production](https://www.robweir.com/blog/2010/02/microsoft-random-browser-ballot.html).

```javascript
// Return a shuffled copy of an Array-like
(arr) => arr.slice().sort(() => Math.random() - 0.5)
```


#### Generate random hex code

ES7's [padEnd](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/padEnd) is a blessing. Along with `padStart`, they've made number to string conversions that much easier. Writing hex values right into JavaScript code is always pretty neat too.

```javascript
// Generate a random hex code such as `#c618b2`
'#' + Math.floor(Math.random() * 0xffffff).toString(16).padEnd(6, '0');
```


#### Pass the interview in style

The [infamous](https://blog.codinghorror.com/why-cant-programmers-program/) interview question answer but codegolfed. I researched and I don't think it can get any shorter than this.

```javascript
for(i=0;++i<101;console.log(i%5?f||i:f+'Buzz'))f=i%3?'':'Fizz'
```


#### Remove duplicates

This only works with primitives but it's still nifty. [Set](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Set) takes any iterable object, like an array `[1,2,3,3]`, and removes duplicates. The spread operator makes that set `[1,2,3]`.

```javascript
// Remove duplicates from the iterable `arr`
[...new Set(arr)]
```


#### A keyboard so real you can taste it

Okay, I don't really count this as a one-liner but it's too good not to include. A masterful codegolf solution by [edc65](https://codegolf.stackexchange.com/a/103226/78322). It's terse to a fault and codegolfed within an inch of its life but we should bask in its glory.

```javascript
// Return a ***3D*** ASCII keyboard as string
(_=>[..."`1234567890-=~~QWERTYUIOP[]\\~ASDFGHJKL;'~~ZXCVBNM,./~"].map(x=>(o+=`/${b='_'.repeat(w=x<y?2:' 667699'[x=["BS","TAB","CAPS","ENTER"][p++]||'SHIFT',p])}\\|`,m+=y+(x+'    ').slice(0,w)+y+y,n+=y+b+y+y,l+=' __'+b)[73]&&(k.push(l,m,n,o),l='',m=n=o=y),m=n=o=y='|',p=l=k=[])&&k.join`
`)()
```

It prints:
![Keyboard made out of ASCII text](ascii-keyboard.png)

Amazing.


Dust off your ternary expressions, we're going in.

JavaScript One-Liners That Make Me Excited


Travis CI offers free test builds for open source projects on GitHub. You'd be a fool not to take them up on their offer. Their email alerts have saved my projects many times before.

In this tutorial, we will be setting up an Express app for continuous integration (CI). Whenever we commit to our `master` branch, Travis CI will clone our repository, spin up a cloud build of linux, install any required dependencies, and run our tests! Hopefully, they pass!  If not, we'll be alerted.

#### Install

Set up a quick `package.json` file with: `npm init -y`. Then grab Express: `npm i express --save` as well as supertest and Jest, our development dependencies: `npm i supertest jest --save-dev`.

Or clone the [repository](https://github.com/healeycodes/earn-a-build-passing-badge), which serves as a live example of the project!

#### Taking the App Out of Express

A default hello world application with Express looks like this:

```javascript
// app.js

const express = require('express');
const app = express();
const port = 3000;

app.get('/', async (req, res) => res.status(200).send('Hello World!'));

app.listen(port, () => console.log(`Our app listening on port ${port}!`));
```

This works for manual testing. We can run this application and check that the right pages are being returned — but what if we have 50 pages with complicated logic? We want to automate this process. The first step is exporting our `app` object. When we run our tests, we don't need a live HTTP server.

Let's alter our hello world application.

```javascript
// app.js

const express = require('express');
const app = express();

app.get('/', async (req, res) => res.status(200).send('Hello World!'));

// Don't listen, just export
module.exports = app; // <--
```

Great. But how do we launch our application now? We'll use [separation of concerns](https://en.wikipedia.org/wiki/Separation_of_concerns) and place the call to `listen()` in another file called `server.js` (which also uses the `app` object!)

```javascript
// server.js

const app = require('./app');
const port = 3000;

app.listen(port, () => console.log(`Our app listening on port ${port}!`))

```

To launch our application, we now use `node server.js`. Let's add that to our `package.json` so people can simply use `npm start`. By default, Node.js will look for a `server.js` file but let's be explicit.

```json
"scripts": {
  "start": "node server.js"
},
```

#### The Tests

A common pattern is to place your tests inside a folder called `__tests__` in the root directory. Another pattern is to repeat the names of the files being tested with `.test` inserted before the `.js`. Thus, `__tests__/app.test.js`.

We'll be using Jest as a test runner. Jest will look inside `__tests__` as part of its default search and will run any test files it finds. You can use a custom test search with `--testMatch`.

> By default [Jest] looks for .js, .jsx, .ts and .tsx files inside of `__tests__` folders, as well as any files with a suffix of .test or .spec (e.g. Component.test.js or Component.spec.js). It will also find files called test.js or spec.js.

Inside our tests,  supertest will mock requests to our `app` object. Mocking requests is faster and more predictable than launching a server and using live requests. It also makes it easier to write [setup and teardown](https://jestjs.io/docs/en/setup-teardown) methods when they're required.

```javascript
// __tests__/app.test.js

const app = require('../app');
const request = require('supertest');

// `describe` is used for test components
describe('GET /', () => {
    
    // `it` is for individual tests
    it('responds with 200', async () => {
        await request(app)
            .get('/')
            .expect(200); // If the status code is not 200, this test will fail
    });
})
```

Let's add another line to our `package.json` so that our tests can be run with `npm test`. The reason we use `start` and `test` aliases is so that our software is predictable for developers picking it up for the first time, and so that it plays nice with other packages.

```json
"scripts": {
  "start": "node server.js",
  "test": "jest"
},
```

`npm test` yields the following output.

```bash
 PASS  __tests__/app.test.js
  GET /
    √ responds with 200 (39ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        2.681s
Ran all test suites.
```

#### Travis CI

Let's [get this code into a repository](https://help.github.com/en/articles/create-a-repo) on GitHub, and install the [Travis CI GitHub App](https://github.com/apps/travis-ci). Make sure that the repo you're testing has Travis CI enabled.

![Repository access](travis-repo-permissions.png)

As the Travis CI [tutorial](https://docs.travis-ci.com/user/tutorial/) tells us:

> Add a .travis.yml file to your repository to tell Travis CI what to do.

In our instance, it's that simple. We don't need to specify any additional settings beyond which version of Node.js we want the test build to use. Travis CI will use the default testing alias `npm test`.

```yml
# .travis.yml

language: node_js
node_js:
 - lts/* # Long Term Support
```

Committing and pushing this file to GitHub will queue up a test build immediately. You can watch the builds execute live at `travis-ci.com/{your-username}/{your-repo}`, and review them later to see where things went wrong. Use this page to grab the markdown for your Travis CI build status badge too!

![A live build](travis-build-passing.png)


How to setup Travis CI for a Node/Express project.

Earn a Build Passing Badge on GitHub! Testing Your Express App with Travis CI (Tutorial)


A little over six months ago, I completed my thesis on WebSocket server performance. I've had a chance to use them in (global) production, and lemme tell you, this technology kicks ass!

#### The Past

The model of the internet is request -> response. You send a request for a page by typing an address in your URL bar -> a web server then responds with data. The information you sent, and the information you received, has what's called a *header*, the rest (what you see on the page) is called the body.

![Request/response description](reqres.png)

These headers contain critical information about the data transfer. However, as we'll see later on, these headers and this request/response cycle can be problematic when we want to do things in real-time.

#### The Present

We've been discussing the HTTP protocol. *WebSockets* refer to the [WebSocket protocol](https://tools.ietf.org/html/rfc6455). Like `http://` and `https://`, we have `ws://` and `wss://` for WebSockets. (The *S* stands for secure). WebSockets are bidirectional. This means that messages can be sent both ways. There doesn't need to be a request and a message doesn't need to be a response. Once a connection has been made, you can sling data whichever way you wish. (Under the hood, here we're talking about [data frames](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers#Exchanging_Data_Frames)).

It all starts with the handshake. When you type in your developer console,

```javascript
    new WebSocket('ws://example.com:3000');
```

a special HTTP/S request is sent. It's special because, [among other quirks](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers#The_WebSocket_Handshake), the header contains `Upgrade: WebSocket` and `Connection: Upgrade`. The server, should it accept your hand in bidirectional matrimony, sends back a response with a similar header with the same two header key/values. At this point, the security you'll be using is negotiated. Both sides are saying Okay, let's do this.

On the client, opening a WebSocket connection looks like this.

```javascript
const socket = new WebSocket('ws://example.com:3000');

// Print incoming messages to the page
socket.onmessage = (msg) => document.body.innerText += msg;

socket.onclose = (event) => alert('uh oh');
```

On the server, we need to listen for these connections. We'll use the unofficial standard WebSockets library for Node.js called [websockets/ws](https://github.com/websockets/ws) or *ws* for short.

```javascript
// Include the library
const WebSocket = require('ws');

// Listen 👂
const wss = new WebSocket.Server({ port: 8080 });

// Every time we get a new connection
wss.on('connection', (ws) => {
  
  // Add a listener to it, log any messages
  ws.on('message', (msg) => {
    console.log('received: %s', msg);
  });

  // After the handshake, say Hi!
  ws.send('howdy');
});
```

Why is this technology useful? Seems like a pretty complicated way to send a message. Let's imagine that a financial website wants to show a stock ticker that updates every two seconds. Before WebSockets, you had to use *polling*. This meant requesting new information every X seconds, even if the information hadn't changed since the last update. Without performing frequent checks, it was impossible to stay in sync.

Clientside, it looked like this.

```javascript
const poller = (time) => {

// Make a request for some JSON (Instead of Fetch, we used to use `XML HttpRequest`!)
fetch('http://example.com/stocks.json')

  // A response will come
  .then((response) => {

    // Parse the data, continue down the chain
    return response.json();
  })
  .then((stockData) => {

    // Pass the fresh data to another function
    updateStocks(stockData);

    // Now, start it all over again - begin polling
    poller(time);
  });
}
```

This is called short-polling. There is an alternative called long-polling where the request is held paused at the server until new data is available or the request is about to timeout, in which case a blank response is sent and the cycle restarts. With long-polling, the data is sent back with low-latency and less bandwidth is used — it's a little more complex to setup though.

With polling, every request and response costs processing, memory, and bandwidth — think of the thick header sections for every cycle! With WebSockets, you perform a handshake and wait for new information — this uses exponentially less processing, memory, and bandwidth for both sides! The underlying [TCP](https://en.wikipedia.org/wiki/Transmission_Control_Protocol) pipe that carries the data back and forth doesn't sit perfectly idle though, for a good reason. The WebSocket protocol allows pings and pongs. These are heartbeats, or, keep-alive messages. Tiny data frames that shoot through the dark pipes under the seabed between computers to say *hello! I'm still here*.

The server pings and the client responds (by the spec, as soon as possible) with a pong. Websockets/ws has this built in. You can call `ws.ping()` at any point after the connection has been made. The endpoint pongs back which triggers the `pong` event. Your browser's `WebSocket class` does this automatically.

#### The Future

Next time you see Twitter has new posts for you, consider the hardworking sockets — and their little heartbeats — which brought you this information, and say thank you to [RFC 6455](https://tools.ietf.org/html/rfc6455) ❤️

[@healeycodes](https://twitter.com/healeycodes) for more personification of technology 🖲️


A little on the history of WebSockets and a brief overview on how to use them in JavaScript.

Kicking Butt and Taking Names With WebSockets (Tutorial)


This Sunday, I worked on an API that creates a GitHub badge based off a **project's mood**. Mood meaning the average time of day that the repository is worked on. I find that I work on different projects at different times of the day. In the morning, I skew towards back-end focused repositories. Maybe it's the coffee? ☕

[![Project Mood](p-m-header.png)](https://github.com/healeycodes/project-mood)

GitHub badges can either be generated dynamically by a library or via third-party services like [shields.io](https://shields.io). For [Project Mood](https://github.com/healeycodes/project-mood), we use [gh-badges](https://www.npmjs.com/package/gh-badges) — an npm package. Here's how a badge is built.

```javascript
const bf = new BadgeFactory();

// ...        

// Customize badge
const format = {
    text: ['project mood', timeOfDay],
    colorA: '#555',
    colorB: color,
    template: 'flat',
}

// Create SVG/add title
const title = `<title>average commit: ${parseInt(average)}:${parseInt((average % 1) * 60)}</title>`;
const svg = bf.create(format).replace(/>/, `>${title}`);
```

It's a prototype, and the library doesn't allow custom attributes, so we inject titles in with a RegEx replace. The sole route we make avaliable is `/:owner/:repo.svg` for example: `/healeycodes/project-mood.svg`. With Express, SVGs can be sent back much like a string would be.

```javascript
return res
    .status(200)
    .send(svg)
    .end();
```

The color of these badges is decided by scanning recent commits and finding the average time of day. The GitHub API requires a user agent and a [personal access token](https://github.com/settings/tokens). We process the commits with a map/reduce. JavaScript's `Date` responds well to time zone correction.

```javascript
// Options for the request call
const options = {
    url: `${api}repos/${req.params.owner}/${req.params.repo}/commits?${token}`,
    headers: {
        'User-Agent': process.env.USERAGENT
    }
};

// ...

// As part of the request callback, commits are scanned
const times = json.map(item => item.commit.author.date);
const average = times.reduce((sum, time) => {
    const d = new Date(time);
    const hours = d.getHours() + (d.getMinutes() / 60) + (d.getSeconds() / 60 / 60);
    return hours + sum;
}, 0) / times.length;
```

It takes ~0.75ms to generate a badge on a modern PC — this includes our title insertion method. Since there is no state being managed, this project would respond well to horizontal scaling. However, the roadmap describes some ways that scale can be managed without throwing money at the problem.

```text
🚗🚗🚗

- Caching:
    - Repositories should be scanned infrequently rather than per request.
    - We can store the most recently requested SVGs in memory.
    - Basically, don't generate the SVG for every request (which is used for the prototype).
- Blended colors depending on average time rather than fixed colors.
```

No project is complete without tests! A simple test plan, executed by a cloud-build, is one of my favorite markers to pass during development. For [Project Mood](https://github.com/healeycodes/project-mood), Mocha and SuperTest are paired with Travis CI. The Express app is exported when the `NODE_ENV` is set to 'test'.

```javascript
if (process.env.NODE_ENV === 'test') {
    module.exports = app;
} else {
    const PORT = process.env.PORT || 8080;
    app.listen(PORT, () => {
        console.log(`App listening on port ${PORT}`);
        console.log('Press Ctrl+C to quit.');
    });
}
```

This allows us to import it into `app.test.js` which will be called by `npm test`. Other enviromental values in use are `USERAGENT` which is required by the GitHub API, as well as `GHTOKEN`. The latter is set to be hidden by Travis CI so that public builds don't leak secrets.

![Travis CI](p-m-travis.png)

One of the lovely tests.

```javascript
// Entry - "mocha test/app.test.js --exit"

const request = require('supertest');
const app = require('../app');
const assert = require('assert');

/**
 * Test SVG request
 */
describe('GET /healeycodes/project-mood', () => {
    it('responds with an SVG', (done) => {
        request(app)
            .get('/healeycodes/project-mood.svg')
            .expect((res) => {
                // SVG XML Namespace
                assert(res.text.match(/http:\/\/www.w3.org\/2000\/svg/gmi) !== null);
                // Error message not present
                assert(res.text.match(/unknown/gmi) === null);
            })
            .expect(200, done);
    });
});
```

💬 [twitter/healeycodes](https://twitter.com/healeycodes) for complaints.


Creating GitHub badges based off the time when most people contribute to a GitHub project.

Creating Cute GitHub Badges Based Off a Project's Mood (Node.js)


Let's say we want to find every emoji as it's posted to Reddit. First, we'll define what an emoji is. The Unicode Consortium keeps their [data lists](http://unicode.org/Public/emoji/12.0/) avaliable. We'll bring them in via a JSON file. The file has descriptions and other things we don't need so we'll clean the data a bit and store it in an efficient `Set`. If we're checking whether a character is or isn't an emoji over 5k times a second we'll need the `O(1)` look-up time!

```python
import json

with open('emojis/emoji.json', encoding='utf8') as emoji_db:
    cleaned = filter(lambda e: 'emoji' in e, json.load(emoji_db))
    emojis = set(map(lambda e: e['emoji'], cleaned))
```

[![Emoji Streamer](emoji-streamer-preview.gif)](https://github.com/healeycodes/EmojiStreamer)

To check every Reddit comment as it comes into existance we'll use [PRAW](https://praw.readthedocs.io/en/latest/) (The Python Reddit API Wrapper). It's the defacto standard for interacting with the Reddit API. It's well documented and straightforward to use.

```python
import praw

client_id = os.environ['CLIENTID']
client_secret = os.environ['CLIENTSECRET']
reddit = praw.Reddit(client_id=client_id,
                     client_secret=client_secret,
                     user_agent='emoji-tracker v0.1')
while True:
    try:
        for comment in reddit.subreddit('all').stream.comments():
            for char in comment.body:
                if char in emojis:
                    print(char, flush=True)
    except Exception as e:
        print(e, file=sys.stderr)
```

Here we set up PRAW with our Reddit app's ID and secret. Once we have a Reddit instance, we can start scanning. PRAW will get all the comments while managing the API limits for us. At around 300k characters per minute, it's quite a lot of data for a human to comprehend but it's peanuts for a PC as long as it's setup properly.

Let's pipe any emojis we find out to `stdout` making sure to flush every `print` so we don't leave any hanging. Emojis sometimes come in slowly, and one at a time -- we want to make sure that our UI is updating as soon as possible to hold the user's attention for a little longer (for those users who enjoy watching a string of emojis grow and grow).

By the way, which of you is posting 40 crabs at a time and why!?

🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀

We'll catch these emojis in Node.js. express/express-ws was chosen so that we can simply host our static files while also making an emoji stream avaliable over a WebSocket. The `emoji-piper.py` file will be called as a child process. Let's attach some event listeners to catch the emojis as they are sent to `stdout`.

```javascript 
/* Whenever new emojis are ready, clear up CLOSING/CLOSED connections
    while sending emojis to the remaining clients */
const emojiStreamer = process;
emojiStreamer.stdout.on('data', (emoji) => {
    emoji = `${emoji}`;
    Array.from(clients, client => {
        if (client.readyState > 1) {
            clients.delete(client);
            return
        }
        client.send(emoji);
    });
});
```

We convert the data into a `String`, clean up closed or closing connections, and disperse the hot-off-the-press emojis. If the emojis were posted more often, say ten times more often, then it would make sense to place them into groups before passing them around (in Node.js or Python, wherever the closest bottleneck is).

Over in Front End Land, we let the user know when they're connected and let loose the fresh emojis. The reason we `split('\n')` here is to fix some unsyncing issues with the `stdout` pipeline -- a flaw in our design but necessary for rapid prototyping.

```javascript 
ws.onopen = () => {
    status.innerHTML = 'Connected ✔️';
}

ws.onmessage = (emojis) => {
    // Multiple emoji may arrive delimited by a new-line char
    emojiList = emojis.data.split('\n');
    emojiList.splice(emojiList.length - 1);
    emojiList.forEach(emoji => {
        render(emoji);
    });
}
```

Did you know that there is no consensus on what the plural of emoji is? Let me know at [@healeycodes](https://twitter.com/healeycodes) if you've been wincing at my usage of *emojis!*

This project is open sourced at [Emoji Streamer](https://github.com/healeycodes/EmojiStreamer) with Travis CI counting emojis and making sure our tests pass 👍


Building a website which live streams every emoji posted to Reddit.com.

Real-Time Streaming of Every Emoji Posted as a Comment to Reddit.com


Sudoku powered by the dark arts of `window.location.hash`, which means zero page refreshes are required. Unfortunately this means that the browser history gets spammed.

![Secret Sudoku](secret-sudoku.gif)

Playable [right here](https://healeycodes.github.io/secret-sudoku/), Secret Sudoku unlocked a series of unique experiences for me. It's not every day that you find yourself brute forcing a 19th-century newspaper puzzle. Or, better yet, getting Travis CI to do it for you.

```javascript
test('app plays without error', () => {
    const game = main(this, sudoku);
    game.startGame();
    for (let i = 0; i < 20000; i++) {
        // 0-81, 0-9
        if (game.play(Math.floor(Math.random() * 82), Math.floor(Math.random() * 10)) === true) {
            /* This test will usually complete the game six times over,
               adjust the difficulty randomly on completion */
            game.startGame(Math.floor(Math.random() * 6));
        };
    }
});
```

It turns out that all you need are some emotes, a few eventListeners, and a dream. Many thanks to Rob McGuire-Dale and his wonderful Sudoku [generator/solver](https://github.com/robatron/sudoku.js). And to Mathew Rayfield for showing that you can [animate URLs](http://matthewrayfield.com/articles/animating-urls-with-javascript-and-emojis/).

```javascript
// Main render functions
this.squaresFilled = () => this.board.join('').match(/[^.]/g).length
this.completeness = () => `${this.squaresFilled()}/81`;
this.start = () => `Secret_Sudoku_~_${this.completeness()}_~_row:`;
this.render = (extra = '') => {
    window.location.hash = `${this.start()}${this.rowMap[this.row]}__${this.prettifyRow(this.currRow(), this.cursor)}${extra}`;
}
```

Secret Sudoku is the perfect project to work on to help me relax in the evenings. It makes programming feel far more fun than logic abstraction should. Taking pull-requests today and every day hereafter.

#### [Repository](https://github.com/healeycodes/secret-sudoku)


Goofy project in which you can play Sudoku entirely within your URL bar.

I Ported Sudoku to Your URL Bar and It Plays like a Dream


Order trackers are a great example of AJAX-ness: a web page updating itself fluidly in the background (even though customers will be spam refreshing!)

![Order Tracker](order-tracker-header.png)

This past week, I created a minimal order tracker, and I'm going to highlight some of my design choices and share some excerpts from the most important sections of the tiny codebase.

Working back to front, we start with a Firebase real-time database.

![Firebase](order-firebase.png)

Here we're defining five stages that an order can be in, and we're also tracking the time the order-tracking instance was created for future logging.

The only packages we need are Express and the Firebase API. We use environment variables where sensible so that our app can be tested without using the production database.

```javascript
/* Firebase */
const admin = require('firebase-admin');
const serviceAccount = require(process.env.KEY);
admin.initializeApp({
    credential: admin.credential.cert(serviceAccount),
    databaseURL: process.env.DB
});
const db = admin.database();

/* Express */
const express = require('express');
const app = express();
const server = require('http').Server(app);
app.use(express.static('public'));
app.use(express.json());
```

There are just three short routes that create an API that we (the business) and our users' browsers can speak to. Note that authentication for public/private routes has been left as an exercise for the reader. As we'll see later, all web page content is statically hosted.

```javascript 
// Create a tracking instance
app.get('/private/orders', (req, res) => {
    const orderId = uuidv4();
    db.ref(`orders/${orderId}`).set({
        started: Date.now(),
        // Integer: 1-5 inclusive
        stage: 1
    })
        .then(() => {
            return res.send(orderId);
        })
        .catch((err) => {
            console.error(`Error creating tracking instance: ${err}`);
            return res.status(500).send('Server error');
        })
});

// Set a tracking instance's state
app.post('/private/orders', (req, res) => {
    db.ref('orders').child(req.body.orderId).set({
        // Clamp stage
        stage: Math.max(1, Math.min(5, req.body.stage))
    })
        .then(() => {
            return res.send('OK');
        })
        .catch((err) => {
            console.error(`Error setting tracking instance state: ${err}`);
            return res.status(500).send('Server error');
        })
});

// Client access to a tracking insance
app.get('/public/orders/:orderId', (req, res) => {
    const orderId = req.params.orderId;
    db.ref(`orders/${orderId}`)
        .once('value')
        .then(data => {
            order = data.val();
            if (order !== null) {
                return res.send(order);
            } else {
                console.error(`Unknown tracking instance requested: ${orderId}.`);
                return res.status(500).send('Server error');
            }
        })
        .catch((err) => console.error(`Order: ${orderId} errored: ${err}`));
});
```

On the front end, we grab some Font Awesome icons, throw them in some red/blue boxes with classes we can toggle, and we're almost done. In order to keep the scope as minimal as possible, we read the query parameter with JavaScript. This way, our microservice can host one static page, one .js file and one .css file.

```javascript 
// The query parameter `orderId` lets us provide order tracking
window.orderId = new URLSearchParams(window.location.search).get('orderId');

// If delivery in progress, check for new information every X seconds
window.localStage = null; // Integer: 1-5 inclusive
const checkStage = () => {
    fetch(`/public/orders/${window.orderId}`)
        .then(res => res.json())
        .then(data => {
            applyStage(data.stage);
        })
        .catch(err => console.error(`Error connecting to server: ${err}`))
    if (window.localStage < 5) {
        setTimeout(checkStage, 10000);
    }
}
```

A dash of CSS media queries for our mobile friends. This was as simple as flipping the flex grid and adding a little padding.

![Responsive](order-mobile.png)

I like to make sure that all of my projects are well tested. It helps me jump back into them after a break and stops me stomping through the code base breaking everything. For this app I chose SuperTest (which comes with Mocha).

For example, this test checks that all the data-plumbing is working correctly.

```javascript
/**
 * Test client accessing a tracking instance
 */
describe('GET /public/orders/:orderId', () => {
    it('respond with an order stage', (done) => {
        // Create an order instance
        request(app)
            .get('/private/orders')
            .end((err, res) => {
                request(app)
                    // Request this instance's stage from client route
                    .get(`/public/orders/${res.text}`)
                    .expect((res) => {
                        const stage = res.body.stage
                        // An integer within 1-5
                        assert(stage > 0 && stage < 6 );
                    })
                    .expect(200, done);
            });
    });
});
```

The [repo](https://github.com/healeycodes/order-tracking-microservice) uses Travis CI to run tests on every commit. Travis's offering to open source projects has been such a blessing to my journey as a developer, and helps me to build software that works (and tests) cross-platform!


Writing an order tracker with Node, Express, and Firebase.

Javascript posts