
[Golfcart](https://github.com/healeycodes/golfcart) is a minimal programming language inspired by Ink, JavaScript, and Python – implemented in Go. It's a toy programming language that I built to use for Advent of Code 2021.

```javascript
// Here's the classic interview question FizzBuzz
for i = 1; i < 101; i = i + 1 {
    log(if i % 3 == 0 and i % 5 == 0 {
        "FizzBuzz"
    } else if i % 3 == 0 {
        "Fizz"
    } else if i % 5 == 0 {
        "Buzz"
    } else {
        str(i)
    })
}
```

Another motivation was to learn how to write an interpreter from scratch.

During this project, I read [Crafting Interpreters](https://craftinginterpreters.com/) and [implemented the Lox programming language](https://github.com/healeycodes/hoot-language) using Python, and [partially ported Ink](https://github.com/healeycodes/quill) using Rust. Another introduction to interpreters I enjoyed was [A Frontend Programmer's Guide to Languages](https://thatjdanisso.cool/programming-languages). The [Ink blog](https://dotink.co/posts/) is also great.

(These resources electrified me. For the past few months all I've been thinking about is programming languages.)

For Golfcart, I began with a desire to design a small programming language that didn't use semi-colons or automatic semicolon insertion. So, no statements, and everything should be an expression that evaluates to a value. For example:
- `if/else if/else` evaluates to the successful branch
- A variable declaration evaluates to the value
- Setting a dict value evaluates to the value
- A for loop evaluates to the number of times the condition expression succeeded

```javascript
assert(
    // This runs five times
    for i = 0; i < 5; i = i + 1 {}, 5
)
```

However, I didn't realise how restrictive this design goal was. A problem I ran into early was accessing an item from a literal.

```javascript
[1][0] // This evaluates to [0] because Golfcart thinks it's two lists

// Instead, you do:
a = [1]
a[0]
```

While it's too late to add semi-colon separated statements to Golfcart, I have a new found appreciation for `;`.

## In the Beginning

I started small, with numbers and math operators (think `1 + 1 * 2`). Whenever I implemented a new type or a piece of syntax, I added a specification program with an assertion. When I came across a bug, I sometimes wrote an error program to purposefully throw an error. This project's tests `go test ./...` ensure that the specification programs and example programs run without any errors (an `assert()` call throws an error and quits) and that the error programs all throw errors.

Perhaps I was too focused on correctness on a micro-level.

The main problem with Golfcart is that there are slight differences between how Golfcart programs run in my head vs. in the interpreter. This is because I jumped to implementing the language and didn't spend enough time designing. Linus Lee (creator of Ink) has some interesting notes on designing small interpreters in [Build your own programming language](https://thesephist.com/posts/pl/#impl).

> In this phase, I usually keep a text document where I experiment with syntax by writing small programs in the new as-yet-nonexistent language. I write notes and questions for myself in comments in the code, and try to implement small algorithms and data structures like recursive mathematical functions, sorting algorithms, and small web servers.

If I had more predefined programs to start with (to run as tests), I would have noticed the divergence of how programs are actually evaluated early enough to re-think the design of Golfcart. Most of this project's example programs were written after the fact within the confines of the language's limitations.

One thing I got right was getting a REPL running from commit number one. Being able to quickly evaluate snippets became crucial to keeping me in a flow-state on this project.

Ultimately, I've learned a lot and this won't be my last language!

## More Syntax and Semantics

Golfcart is a dynamic strongly typed language with support for bools, strings, numbers (float64), lists, dicts, and nil (null). There is full support for closures and functions can alter any variable in a higher scope.

```javascript
counter = () => {
    n = 0
    () => {
        n = n + 1
        n
    }
}

my_counter = counter()
my_counter() // 1

assert(my_counter(), 2)
```

A Golfcart program is a series of expressions. Line breaks are optional and there are no semi-colons. The final expression is sent to stdout.

```javascript
a = 1 b = 2 assert(a + b, 3) // A successful assert() evaluates to nil
```

There are seven types. A type-check can be performed with `type()`.

```javascript
// Bools
true or false
true and true

// Numbers
1
1.1 + 1.1 // 2.2

// Strings
"multi-line
string"
"1" + "2" // "12"

// Lists
[1, 2]
nums = [3, 4]
nums.append(5) // [3, 4, 5]
[0] + [1] // [0, 1]

// Dicts
{a: 1} // Accessed by `.a` or `["a"]` like JavaScript
{b: n => n + 1} // Values can be any type
keys({a: 1}) // ["a"]

// Functions
_ => nil // All user-defined functions are anonymous, assignable by variable
n => n + 1
sum = (x, y) => x + y

// Nil
nil
nil == nil // true
```

The Fibonacci sequence.

```javascript
// Naive
t = time()
fib = n => if n == 0 {
    0
} else if n == 1 {
    1
} else {
    fib(n - 1) + fib(n - 2)
}
fib(20)
log("fib: " + str(time() - t))

// With memoization 
t = time()
cache = {"0": 0, "1": 1}
fib_memo = n => if cache[n] != nil {
    cache[n]
} else {
    cache[n] = fib_memo(n - 1) + fib_memo(n - 2)
}
fib_memo(20)
log("fib_memo: " + str(time() - t))
```

For more detailed examples, see:

- [Example programs](https://github.com/healeycodes/golfcart/tree/main/example%20programs)
- [Specification programs](https://github.com/healeycodes/golfcart/tree/main/example%20programs/spec%20programs)
- [Programs that purposefully throw errors](https://github.com/healeycodes/golfcart/tree/main/example%20programs/error%20programs)

(All the above are used as part of Golfcart's test suite)

## Implementation

Golfcart is a tree-walk interpreter. Its single dependency is the [Participle](https://github.com/alecthomas/participle) parsing library, which consumes a parser grammer written using Go structs and a RegEx-like syntax to create an abstract syntax tree (AST) (see [parser.go](https://github.com/healeycodes/golfcart/blob/main/pkg/golfcart/parse.go)). This library let me move fast and refactor parser bugs without headaches. Participle has great documentation and [examples](https://github.com/alecthomas/participle/tree/master/_examples) (e.g. how to parse BASIC, MicroC, GraphQL) and it's in active development.

Participle provides line numbers and column positions for each token. They are added to Golfcart's language values during evaluation (so some Golfcart errors have line numbers). All errors have error text, and hopefully enough information to find the problem and fix it but Golfcart lacks a mature stack trace.

A program runs like this: a piece of source code is turned into tokens by Participle's lexer. The lexer uses token definitions — for example, Golfcart's identifier is defined as: ```{"Ident", `\w]+`, nil}```). These tokens are parsed into an AST using struct definitions and the AST is evaluated (see [eval.go](https://github.com/healeycodes/golfcart/blob/main/pkg/golfcart/eval.go)).

Here's Golfcart's list literal:

```go
type ListLiteral struct {
    Pos lexer.Position

    // This matches any number of expressions (or none)
    // If there's more than one expression, they must be separated by a comma
    // `@@` means: recursively capture using the fields own type
    Expressions *[]Expression `"[" ( @@ ( "," @@ )* )? "]"`
    // For simple types, tokens can be parsed directly into Go values
    // e.g. bool, float, string, etc.
}
```

How does a literal node become a value? Let's take the `ListLiteral` as an example. An `Eval` method is defined for most nodes, including this one. Given a stack frame, this list literal becomes a list value.

```go
func (listLiteral ListLiteral) Eval(frame *StackFrame) (Value, error) {
    values := make(map[int]*Value, 0)
    if listLiteral.Expressions != nil {
        for _, expression := range *listLiteral.Expressions {
            result, err := expression.Eval(frame)
            if err != nil {
                return nil, err
            }
            values[len(values)] = &result
        }
    }
    return ListValue{val: values}, nil
}
```

Let's talk more about stack frames in Golfcart. A stack frame is a map of variables in scope. It's a recursive structure, every stack frame has a parent apart from the global frame. All functions are anonymous and create closures. Any variable referenced in a higher scope can be altered.

Examples explain this better than words.

```javascript
a = 1
a_function = () => a = 2 // Closure created
a_function() // When called, `a` is changed
a // 2

if true {
    // `b` is not defined in a higher scope
    // So, `b` is declared only within this scope
    b = 3
}
b // Error: cannot find value for key 'b'

c = nil
if true {
    // This assignment recursively looks in higher scopes for `c`
    // it's found and that value is altered
    c = 4
}
c // 4
```

<br>

When I needed to make a design decision for Golfcart, I drew from internal feelings about how computer programs _should_ run. It feels good to own a slice of my computing experience.

Check out the source and grab the binaries at [https://github.com/healeycodes/golfcart](https://github.com/healeycodes/golfcart) ⛳

## Learning Resources

I used these:

- [Build your own programming language](https://thesephist.com/posts/pl)
- [Crafting Interpreters](https://craftinginterpreters.com/)
- [A Frontend Programmer's Guide to Languages](https://thatjdanisso.cool/programming-languages)
- [The Ink blog](https://dotink.co/posts/)
- [Build your own programming language](https://thesephist.com/posts/pl/)

I've not tried but have heard great things about:

- [Writing An Interpreter In Go](https://interpreterbook.com/) (and the [sequel](https://compilerbook.com/))
- [Let's Build A Simple Interpreter](https://ruslanspivak.com/lsbasi-part1/)
- [The Super Tiny Compiler](https://github.com/jamiebuilds/the-super-tiny-compiler)


Creating the Golfcart Programming Language


- [Architecture](#architecture)
- [GPS module](#gps-module)
- [Printer module](#printer-module)
- [AM2302 module](#am2302-module)
- [Home module](#home-module)
- [Developer experience and testing](#developer-experience-and-testing)

<br>

I became interested in high precision GPS time after listening to a podcast about network clock synchronization on [Signals & Threads](https://signalsandthreads.com/clock-synchronization/). My original goal was to setup a local NTP time server that I could sync all my devices to. However, I got sidetracked along the way – and this is the story of my Raspberry Pi home dashboard.

For Christmas, I received a GPS receiver for my Raspberry Pi. It ended up in the box with the rest of my sensors, add-ons, and screens I've bought for my Pi. It was a sad box. I decided enough was enough and that I wanted to hook them all up at once. So I built a dashboard with a module system so I can add and remove sensors and functionality as time goes on. The code is [open source](https://github.com/healeycodes/pi) and has a Windows 98 theme powered by [98.css](https://github.com/jdan/98.css/). It currently has:

- A list of the visible GPS satellites (GT-U7 receiver)
- The live temperature and humidity of my living room (AM2302 sensor)
- A printer that prints out messages sent from mutuals (POS58 receipt printer)

![The home page of the dashboard displaying all three live modules.](dashboard-preview.png)

## Architecture

The dashboard is part of a mono repository that also contains the client code which runs on the Pi. The Pi spawns a thread for each module which polls the relevant sensor and sends the result to the server. The printer module is a special case in that it also fetches messages from the server and sends a confirmation as they are printed. The other modules just fling data up.

There are two config files (client and server) that allow modules to be toggled on and off so that you don't need my exact hardware setup. This also means that more than one Pi can be part of the same system. 

The client entry point `poller.py` is called when the Pi starts up (via a line in `/etc/rc.local`) and runs in the background. It takes two arguments – the server address and the password that each request sends.

![Architecture diagram of the three Raspberry Pi modules communicating with a Flask application on Heroku connected to a PostgreSQL database also on Heroku.](pi-architecture.png)

The server is hosted on a Heroku dyno and automatically deploys on commit. It's a Flask application backed by a PostgreSQL database (provisioned by Heroku). During local development, and when running the server tests, SQLite is used for developer convenience. While working on this project, I learned about [PEP 249](https://www.python.org/dev/peps/pep-0249/) (the Python Database API Specification) which is the reason my database code is portable between the two libraries I use – `sqlite` and `psycopg2`.

I'll break down for you what each module does.

## GPS module

In 2020, the U.S. government spent $1.71 billion maintaining the core Global Positioning System (GPS) program. The space segment consists of a nominal constellation of 24 satellites. These satellites transmit a one-way signal which a GPS device receives. A signal from four satellites is usually required to solve the geometric equations that provide your 3D position, speed, and the current time.

The government commits to broadcasting the GPS signal in space with:

- A global average user range error (URE) of ≤7.8 m (25.6 ft.), with 95% probability.
- A global average user range rate error (URRE) of ≤0.006 m/sec over any 3-second interval, with 95% probability.
- Distributed UTC time as maintained by the U.S. Naval Observatory (USNO) via the GPS signal in space with a time transfer accuracy relative to UTC(USNO) of ≤40 nanoseconds (billionths of a second), 95% of the time.

Here is my receiver which cost single-digit dollars.

![Holding my GT-U7.](gt-u7.jpeg)

There was a particular joy when I saw the database had been filled with a record of each of the 31 live GPS satellites (I had to wait for one earth rotation to catch 'em all). About 8-10 are visible at any one point. The satellites identify themselves by transmitting a pseudo-random noise (PRN) sequence along with the rest of their information. I use this to look up a mapping which transforms this two digit code into a name – like _USA-293_ aka _Magellan_, or _USA-132_ (which has been in service for 23 years). Until I looked into the data my GPS receiver was giving me I lived in a world where GPS was an abstraction – a blue circle on my phone showing my position on a map. Now I know the launch stories and how these satellites have been maintained.

To get a mapping of PRN -> satellite name I scraped the Wikipedia [List of GPS satellites](https://en.wikipedia.org/wiki/List_of_GPS_satellites). I later learned that a more accurate way to get a mapping is to check the USCG NAVCEN's [Constellation Status](https://www.navcen.uscg.gov/?Do=constellationStatus) page which is updated daily. From there, you can map a PRN to a unique space vehicle number (SVN). This mapping is necessary because PRNs change as satellites are retired and new ones are launched. It's also possible for a satellite to use different PRNs over its lifetime, like [the case of SVN 049](https://en.wikipedia.org/wiki/List_of_GPS_satellites#PRN_to_SVN_history) using PRNs 01, 24, 27, and 30.

My GT-U7 receiver uses the [NMEA 0183](https://gpsd.gitlab.io/gpsd/NMEA.html#_nmea_encoding_conventions) protocol (sometimes called NMEA sentences). By the way, that previous link goes to a 14k word article(!) so I was glad that I could abstract all of this away by using [gpsd](https://gpsd.gitlab.io/gpsd/), a GPS service daemon. The closest I came to interacting with NMEA sentences was to `cat` my device to check that it was alive.

`gpsd` has client libraries in C, C++, and Python. The [documentation](https://gpsd.gitlab.io/gpsd/client-howto.html) is top tier as well. [gpsmon](https://gpsd.gitlab.io/gpsd/gpsmon.html), a real-time GPS packet monitor and control utility, also impressed me with its slick graphical CLI.

While researching all this, I went down a bit of a rabbit hole reading about the way GPS service interruptions are announced. NAVCEN post a list of areas and times where GPS TT&E (testing, training activities, and exercises) are planned to occur. Both ahead of time, and during the GPS interference, [it's possible to alert NAVCEN](https://www.navcen.uscg.gov/?pageName=gpsServiceInterruptions) (or the FAA) to petition for the interference to be cancelled if there is a safety-of-life/safety-of-flight issue.

> Due to the fact that these training and testing activities can involve a number of aircraft, ships and/or other military equipment and up to hundreds of personnel, cancellation or postponement of a coordinated test should only occur under compelling circumstances. In general, only safety-of-life/safety-of-flight issues will warrant cancellation or postponement of an approved, coordinated GPS test.

Since the areas are mostly military zones, unless you're an unlucky firefighter this information can probably be discarded.

## Printer module

I used my POS58 receipt printer in a previous project where I printed out my twitter feed as new tweets arrived. I'm still porting around and editing the same slim library originally written by Vince Patron (who works at Brain Corp and who doesn't have a public email so I can thank him!) so setting up the printer module was pretty plug n' play.

![My tweets being printed live.](printer-in-action.gif)

My friend has a POS58 receipt printer as well and we ping each other messages. These printers are quite hardy – not too many moving parts and thermal paper is cheap. When enabled, the printer module makes a form page available at `/printer`. It's also 98-styled and polls an endpoint until the Pi pings the server and confirms that the message has been printed (on the ever growing strip of paper in the corner of my living room).

![The confirmation page after a message is submitted.](printer-confirm.png)

A more human-readable timestamp is a TODO and also a good first issue!

## AM2302 module

The AM2302 temperature and humidity sensor is the Hello World of Pi sensors. I use the deprecated library [adafruit/Adafruit\_Python\_DHT](https://github.com/adafruit/Adafruit_Python_DHT) because I've worked with it before. Since Linux can't guarantee the timing of calls to the sensor, you can sometimes get a null result when using it. I correct for this by not reporting empty results to the server.

I've found the accuracy of the two AM2302 sensors I've used to be quite good. On par with the other household sensors I have around. And if you're buying someone a Raspberry Pi as a gift, I recommend bundling this sensor along with it so the Pi is more than just a small computer you can SSH into.

## Home module

The home module conditionally renders the other module sections when a user visits the dashboard. The styling of the dashboard is provided by [@jdan](https://twitter.com/jdan)'s [98.css](https://jdan.github.io/98.css/). The other modules make a HTTP API available and the home page data is pulled in as the page loads. I didn't want to include a build step for the JavaScript (and there's only about 60 lines of non-library code) so I just wrote plain ES6. The one larger-than-necessary dependency I pulled in was moment.js which provides the relative-time-in-human-language feature of `a few seconds ago`, `an hour ago`, etc. I'm looking to remove this library ASAP because for my use-case I'm trading a 18.2kb cost for what is effectively sentence generation. Apart from the printer confirmation, which needs to be AJAX (unless I let the page hang perhaps?), this could all be server-side rendered. However, I wanted to leave open the option of a live updating dashboard so I setup HTTP APIs from the start.

## Developer experience and testing

I like to write, and use, software with very little setup. So it was important that the local database was SQLite instead of PostgreSQL. Another step I took was to not include migrations as part of this project. There's a single schema and it's applied when you instantiate a database connection.

Deploying the server to Heroku should take ~one minute. The only configuration steps are toggling which modules that should be enabled, setting a password in the configuration variables, and adding a PostgreSQL database.

For development, you run the following commands to boot up the Flask app on a UNIX-like OS.

```bash
git pull https://github.com/healeycodes/pi
cd server
pip install -r requirements.txt
export FLASK_APP=wsgi:app ;python3.9 -m flask run
```

GitHub Actions is used for CI and there are some end to end tests that cover all of the server modules.

The client code is the similar to the server; pull, install, and run. It uses the Python 3 version that comes with the latest Raspbian (v10). However, unlike with Flask's debug mode, there isn't auto-reloading. And in a lot of cases I found myself `sudo reboot`ing after every code change because the GPS device would get 'stuck' for lack of a better word.

I think that's the main thing I'll investigate for future projects that run on my own hardware. How to better manage the deploys and how to get logs in a browser.

Star/fork/complain about my architecture choices in the issue tracker at [healeycodes/pi](https://github.com/healeycodes/pi)!


Building a live dashboard with the help of a few hardware modules.

Listening to Satellites with my Raspberry Pi


Over the weekend, I wrote a program that tries to identify which DMC threads are used in embroidery hoop images. I didn't take a computer vision course at university — nor had I used OpenCV before — so it was fun to learn a few things and I hope you learn from my learnings.

It all started when my wife wanted to find out what colors were used in a few of the hoops posted to r/embroidery. Thus, I embarked on an over-engineered solution: a Python CLI that guesses and then overlays the DMC thread color palette on a given image.

![An embroidery hoop with a color palette overlain — the colors have DMC ids](example_out.jpg)

To get started, I reviewed 100 hoops recently posted to r/embroidery and saw that they were photographed straight on with the hoop near the center. Having a fairly uniform structure, I figured that a little computer vision script could help me out. The hoops being near-perfect circles made them a great target for OpenCV's [Hough Circle Transform](https://docs.opencv.org/master/da/d53/tutorial_py_houghcircles.html). However, the photographs were very noisy. Lots of things appeared like circles and the function took a long time due to the high resolution.

So to better target the hoop area, I used a series of destructive filters — the image is scaled down, converted to gray and then the following are applied: `GaussianBlur`, `medianBlur`, `adaptiveThreshold`, `erode`, `dilate`. As we see below, this makes the hoop more identifiable to `HoughCircles`.

![An image with each additional destructive filter applied](example_destructive_filters.jpg)

I tuned the values for these filters after running the program on the 100 r/embroidery posts I had. The full source code for the program I built can be found at [healeycodes/embroidery-vision](https://github.com/healeycodes/embroidery-vision).

```python
def apply_destructive_filters(image):
    """
    Apply a series of destructive filters with the aim of making circles more
    visible to the HoughCircles function.
    """
    kernel = np.ones((5, 5), np.uint8)

    filtered_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    filtered_image = cv2.GaussianBlur(filtered_image, (5, 5), 0)
    filtered_image = cv2.medianBlur(filtered_image, 5)
    filtered_image = cv2.adaptiveThreshold(
        filtered_image, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 3.5
    )
    filtered_image = cv2.erode(filtered_image, kernel, iterations=1)
    filtered_image = cv2.dilate(filtered_image, kernel, iterations=1)
    return filtered_image
```

It's common for multiple circles to be found. When this happens, I pick the largest and most central one. This can be seen in the previous image. The valid circle is green. I sometimes found partial circles that matched the hoop criteria (large and central). However, these were background shapes. Users rarely posted zoomed in images of hoops so I discarded circles that went beyond the edge.

The valid circle often included part of the hoop's border. I didn't want to consider the color of the border as a thread so I shrunk the hoop area by 1% to hide it.

## Color Palette 

There are 454 DMC thread colors. They're called things like `Beige Gray Ult Dk`. The goal is to take the RBG color of the photographed threads and find the closest matching DMC color. I don't account for lighting conditions (how would you do this?) which lowers the accuracy of the results. My program finds some threads and guesses wrong for the others. For nearest-color lookups, I used the k-d tree from [scipy.spatial.KDTree](https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.KDTree.html).

```python
def rgb_to_dmc(r, g, b):
    tree = sp.KDTree(rgb_colors)
    _, result = tree.query((r, g, b))
    # return the index of `rgb_colors`
    return result
```

When I tried calling this on every pixel in the hoop area, the palette it built had too many colors. When there was one kind of dark red thread in the image, the palette would have six or seven different DMC reds. I needed to better group colors together so I quantized the image.

```python
# https://stackoverflow.com/a/20715062
def quantize_image(image, div=64):
    """
    Reduces the number of distinct colors used in an image.
    """
    quantized = image // div * div + div // 2
    return quantized
```

Compressing the color space like this means that instead of finding multiple dark reds we just find one (hopefully the right one). Here's an image before and after the above function is applied.

![An example image without any color quantization, the same image with color quantization](example_quantized.jpg)

After improving the color accuracy somewhat, I arrived at my next problem. Speed.

Calling a k-d tree lookup for every pixel in a high resolution image was slow. For the example image I've been using in this post, there are 600k pixels (which mean the same number of lookups) but only 30 different colors. Adding a cache was a no-brainer. Python's built in Least Recently Used (LRU) cache worked great and now our 600k calls are performed in constant time.

```python
@lru_cache
def cached_rgb_to_dmc(r, g, b):
    return rgb_to_dmc(r, g, b)
```

With the colors found, I drew the palette over the image with the DMC identification numbers. The palette is also sent to standard output (a list of e.g. `#647 Beaver Gray Med 15.81%`)

![An example image's overlain palette](example_palette.jpg)

I found using OpenCV's APIs to be quite straight forward due to the amount of documentation and blog posts online. Whenever I had a question, a search brought me a vaguely related answer. Perhaps I underestimated the popularity of the library. The hardest part (as always) was figuring out the right question to ask.

```python
# overlay the color palette on top of the image
_, w, _ = original_image.shape
size = int(w / len(filtered))
y = size
for idx, color in enumerate(filtered):
    b, g, r = (
        dmc_colors[color[0]]["blue"],
        dmc_colors[color[0]]["green"],
        dmc_colors[color[0]]["red"],
    )
    cv2.rectangle(
        original_image, (size * idx, 0), ((size * idx) + size, size), (b, g, r), -1
    )
    cv2.putText(
        original_image,
        dmc_colors[color[0]]["floss"],
        (size * idx, size - 7),
        cv2.FONT_HERSHEY_SIMPLEX,
        0.5,
        (255 - b, 255 - g, 255 - r),
        1,
    )
```

_Update – two readers emailed in with advice._

- _One suggested converting RGB to a color space that better matches human visual perception, like CIELAB._
- _Another suggested compressing the color space by using K-means clustering (by guessing K as the number of threads)_

## A sparkle of presentation

I try to do most of my work in public on GitHub. This helps people (many more people than I ever expected to help) but I'm most interested in the effect it has on expanding my network of like-minded makers. It's a joy to receive a message, issue, or PR on any of my work.

When I make a repository public I aim to have a comprehensible README and some tests. For this project, the DMC color lookup is unit tested and example images are used for end to end testing. I added a GitHub Action so that it's easy for people to check their PRs are passing (it's also good to get an email when I've broken something by editing a source file on my phone!). This project is less likely to see collaboration as it's a learning project for me and not really a general tool. It's me-ware. Plus, it doesn't really work.

Lately, I've been using Mypy for my open source Python code. It's been catching bugs and enforcing better code patterns. I use a `requirements-dev.txt` file to pin the Mypy version for the CI pipeline. ~~But sadly, I couldn't get the OpenCV type stubs to work so I removed `mypy .` from my usual GitHub Action.~~ I had a Mypy issue where the type stubs for a few libraries weren't available but a reader emailed me how to fix this — by using a `mypy.ini` configuration file.

```ini
[mypy]
[mypy-cv2]
ignore_missing_imports = True
[mypy-scipy]
ignore_missing_imports = True
[mypy-numpy]
ignore_missing_imports = True
```

I've also found that adding repository tags aids discoverability.

![Thread emoji, Identify which DMC threads are used in embroidery hoop images. Topics: opencv hough-transform embroidery colorpalette dmc-color](tags.png)

Finally, I'll finish up with an open source license like MIT. I'd hate for someone to learn something from one of my toy projects but not be able to use it. I always try to present my work as if there are end users for it — a process that makes coming back to my code far easier than if I hadn't.


Picking the correct thread colors from embroidery hoop images.

Computer Vision and Embroidery


- [A dash of deep learning](#a-dash-of-deep-learning)
- [Automating GeoGuessr with Selenium](#automating-geoguessr-with-selenium)
- [Geolocation estimation](#geolocation-estimation)
- [Web demo](#web-demo)
- [The learnability of GeoGuessr](#the-learnability-of-geoguessr)

<br>

During this last lockdown in the UK, my wife and I have been playing GeoGuessr. It's slower paced than the computer games we normally play but it goes well with having an 11-week old baby who becomes more alive each and every day.

GeoGuessr is a _geographic discovery game_. You are dropped into a random Google Street View and tasked with pointing out your location on a map. You can look around, zoom, and follow the car's path through the local streets. 

![GeoGuessr](geoguessr.png)

We've started to take the Daily Challenge on GeoGuessr quite seriously. We show up every day and push for a new high score. In the Daily Challenge there's a three minute limit per round which, for us, is either filled with frantic clicking as we zip through the Australian outback (probably while mistaking it for South Africa), or debating back and forth about whether `ø` exists in the Swedish language.

I now have a lot of _I'll know it when I see it_ knowledge. I know Greenland on sight. My lost knowledge of country flags has returned, along with new knowledge of USA state flags, which countries drive on the right vs. left, which use KM vs M. I know pretty much every country-specific domain name (they're often on roadside adverts) – I won't forget [.yu](https://en.wikipedia.org/wiki/.yu) anytime soon.

Did you know that black and white guard rails are commonly found in Russia and Ukraine? Or that you can make out the blue EU bar on license plates through Google Street View's blurification? Read more in this _80,000 word_ guide – [Geoguessr - the Top Tips, Tricks and Techniques](https://somerandomstuff1.wordpress.com/2019/02/08/geoguessr-the-top-tips-tricks-and-techniques/).

> The red and white striped arrow pointing downwards indicates that you are in Japan, most likely on the island of Hokkaido or possibly on the island of Honshu near mountains.

![Utility pole with red and white striped arrow pointing downwards.](arrows.png)

## A dash of deep learning

I once read that machine learning is currently capable of doing anything a human can do in under one second. Recognize a face, pick out some text from an image, swerve to avoid another car. This got me thinking, and thinking led me to a paper called _Geolocation Estimation of Photos using a Hierarchical Model and Scene Classification_ by Eric Müller-Budack, Kader Pustu-Iren, and Ralph Ewerth. This paper treats "geolocalization as a classification problem where the earth is subdivided into geographical cells."

It predicts the GPS coordinates of photos. 

![Left: Workflow of the proposed geolocation estimation approach. Right: Sample
images of different locations for specific scene concepts.](paper-illustration.png)

Even indoor photos! (GeoGuessr's Daily Challenge will often trap you inside a museum).

Recently, the paper's authors released a PyTorch implementation and provided weights of a pre-trained `base(M, f*)` model with underlying ResNet50 architecture.

I presumed that the pretrained model would not map well to the sections of photospheres that I could scrape from GeoGuessr. For training data, the authors used "a subset of the Yahoo Flickr Creative Commons 100 Million dataset (YFCC100M)". Which contains "around five million geo-tagged images from Flickr [and] ambiguous photos of, e.g., indoor environments, food, and humans for which the location is difficult to predict."

What was interesting was that on the Im2GPS dataset, humans found the location of an image at country granularity (within 750km) 13.9% of the time but the Individual Scene Networks were able to do it 66.7% of the time!

![Table 6. Results on the Im2GPS (top) and Im2GPS3k (bottom) test sets. Percentage is the fraction of images localized within the given radius using the GCD distance.](table6.png)

So the question became: who is better at GeoGuessr, my wife (a formidable player) or the machine?

## Automating GeoGuessr with Selenium

To scrape screenshots of the current in-game location, I created a Selenium program that performs the following four times:

- Take a screenshot of the canvas
- Step forwards
- Rotate the view ~90 degrees

![An automated GeoGuessr run.](selenium.gif)

The number of times this happens is tuneable via `NUMBER_OF_SCREENSHOTS` in the snippet below.

```python
'''
Given a GeoGuessr map URL (e.g. https://www.geoguessr.com/game/5sXkq4e32OvHU4rf)
take a number of screenshots each one step further down the road and rotated ~90 degrees.
Usage: "python file_name.py https://www.geoguessr.com/game/5sXkq4e32OvHU4rf"
'''
from selenium import webdriver
import time
import sys
NUMBER_OF_SCREENSHOTS = 4


geo_guessr_map = sys.argv[1]

driver = webdriver.Chrome()
driver.get(geo_guessr_map)

# let JS etc. load
time.sleep(2)


def screenshot_canvas():
    '''
    Take a screenshot of the streetview canvas.
    '''
    with open(f'canvas_{int(time.time())}.png', 'xb') as f:
        canvas = driver.find_element_by_tag_name('canvas')
        f.write(canvas.screenshot_as_png)


def rotate_canvas():
    '''
    Drag and click the <main> elem a few times to rotate us ~90 degrees.
    '''
    main = driver.find_element_by_tag_name('main')
    for _ in range(0, 5):
        action = webdriver.common.action_chains.ActionChains(driver)
        action.move_to_element(main) \
            .click_and_hold(main) \
            .move_by_offset(118, 0) \
            .release(main) \
            .perform()


def move_to_next_point():
    '''
    Click one of the next point arrows, doesn't matter which one
    as long as it's the same one for a session of Selenium.
    '''
    next_point = driver.find_element_by_css_selector('[fill="black"]')
    action = webdriver.common.action_chains.ActionChains(driver)
    action.click(next_point).perform()


for _ in range(0, NUMBER_OF_SCREENSHOTS):
    screenshot_canvas()
    move_to_next_point()
    rotate_canvas()

driver.close()
```

The screenshots will also contain the GeoGuessr UI. I didn't look into stripping it.

## Geolocation estimation

I checked out the [PyTorch branch](https://github.com/TIBHannover/GeoEstimation/tree/pytorch), downloaded the pretrained model, and installed the dependencies via `conda`. I commend the repository's README. The [requirements](https://github.com/TIBHannover/GeoEstimation/tree/pytorch#requirements) section was quite clear and I didn't run into any problems on a fresh Ubuntu 20.04 box.

I picked the [World](https://www.geoguessr.com/maps/world) map on GeoGuessr for the showdown between human and machine. I gave the URL to my Selenium program and then ran inference against the four screenshots captured from GeoGuessr. 

The machine's trimmed results are below.

```bash
python -m classification.inference --image_dir ../images/

                                lat        lng
canvas_1616446493 hierarchy     44.002556  -72.988518
canvas_1616446507 hierarchy     46.259434  -119.307884
canvas_1616446485 hierarchy     40.592514  -111.940224
canvas_1616446500 hierarchy     40.981506  -72.332581
```

I presented the same four screenshots to my wife. She guessed a point in Texas. The actual starting location was over in Pennsylvania. The machine, indecisively, had four different guesses for each of the four screenshots. All of the machine's guesses were in the USA. Two very close ones and two further away.

![USA guesses.](usa-guesses.png)

If we take the average location, the machine wins this round!

We played two more follow up rounds and the final results were 2-1 to the machine. The machine got quite close with a street in Singapore but failed to get anywhere near a snowy street in Canada (Madeline got the town in seconds).

I learned after writing this post that there's some fantastic prior art on machine vs. human on the battleground of GeoGuessr. In _PlaNet - Photo Geolocation with Convolutional Neural Networks_, Tobias Weyand, Ilya Kostrikov, and James Philbin also tried to determine the location of a photo from just its pixels.

> To find out how PlaNet compares with human intuition, we let it compete against 10 well-traveled human subjects in a game of Geoguessr (www.geoguessr.com). 

> [H]umans and PlaNet played a total of 50 different rounds. PlaNet won 28 of the 50 rounds with a median localization error of 1131.7 km, while the median human localization error was 2320.75 km.

## Web demo

The authors of _Geolocation Estimation of Photos using a Hierarchical Model and Scene Classification_ built a pretty neat web-tool. Here I've run it against one of the Selenium screenshots.

> A graphical demonstration where you can compete against the deep learning approach presented in the publication can be found on: https://tibhannover.github.io/GeoEstimation/. We also created an extended web-tool that additionally supports uploading and analyzing your own images: https://labs.tib.eu/geoestimation

![A machine annotated image of the street in Pennsylvania.](web-tool.png)

## The learnability of GeoGuessr

There are many reasons why trying to beat GeoGuessr (by which we mean out performing a human very often) with machine learning might be easier than locating any human-taken photo.

Compared to general geolocation estimation, in GeoGuessr we are (almost always) trying to find out which road we are on. This means more effort can be made into recognizing things which are always there – like road markings or car makes and models (both often betray a country). Effort could be made to travel down roads looking for road signs which provide the country's language or the signage text could be used to search a look-up table.

There are other markers, that some in the GeoGuessr community consider cheating, that a learning framework may pick up.

Looking downwards in street view will reveal part of the vehicle that captured the current photosphere. For example, in Kenya the front of the street view car has a black snorkel. Much of Vietnam was recorded on a motorbike and you can often see the rider's helmet. Countries are often covered by the same car with a unique color or antenna.

![The snorkel car and the Vietnam rider's helmet.](snorkel.png)

In other places, there are rifts in the sky where the photosphere stitching looks torn (largely in Senegal, Montenegro and Albania). Across Africa, there are sometimes escort cars that follow the street view car. There are different camera generations – with different resolutions, types of halo, colouring, and blurring at the bottom of the sphere. In the bottom corner of the photosphere there is the credit message – it usually has "Google" and a year but will occasionally have a photographer's name.

By exploiting these, I wouldn't be surprised if a machine could take down even the best GeoGuessrers in a timed competition. In fact, I'll go as far to say we are one research grant away from being categorically worse at GeoGuessr than machines.

Applying photo geolocation estimation to the game of GeoGuessr.

GeoGuessing with Deep Learning


The challenge of searching static websites is effectively a solved problem with brilliant, yet widely unadopted, solutions.

These projects scan and index your content at build time and ship it to the browser with the rest of your static files. No out-of-your-control external service required. A small library in JavaScript (sometimes accompanied by WebAssembly) loads the index and allows instant searching and works just fine offline. Searching the complete Federalist Papers for multiple keywords takes less than 100µs in my Wasm [tech demo](https://github.com/healeycodes/crane-search) (I'll get to that later on).

The most popular libraries are written in JavaScript like [Lunr](https://lunrjs.com/), or [Fuse](https://github.com/krisk/Fuse), or [FlexSearch](https://github.com/nextapps-de/flexsearch). They provide pre-compiled indexes and [fuzzy](https://en.wikipedia.org/wiki/Approximate_string_matching), wildcard, and boolean searches. They have good documentation and are ready for production usage.

But I'm more interested in how the up-and-coming Wasm-powered projects work. Browser search is a space where Wasm can really find an edge. Wasm runs faster, loads quicker, and ships smaller binaries (in theory, in practice it takes some work to get there).

The two projects I looked at both use Rust as a backend language:

- [tinysearch](https://github.com/tinysearch/tinysearch) generates a small, self-contained Wasm module (with an embedded index). It aims for a small memory footprint by using Bloom filters — a space-efficient probabilistic data structure that results in a negligible amount of false positives. Read more in [A Tiny, Static, Full-Text Search Engine using Rust and WebAssembly](https://endler.dev/2019/tinysearch) by the creator, Matthias Endler.
- [Stork](https://stork-search.net/) focuses more on the developer experience with rich documentation and examples. Stork also pays keen attention to the user experience by providing a progress bar for loading the index, excerpts for the results (keywords highlighted), theme support, and more (see [Embedding the Interface](https://stork-search.net/docs/interface)). Learn about how it came to life in [Stork Turns One: Building a search tool for static sites with Rust and WebAssembly](https://jameslittle.me/blog/2020/one-year-of-stork/) by the creator, James Little.

Stork is more interesting to me because it focuses on the human experience — of the person who builds with it, and the person who searches. See the thin, quick, configurable UI below.

![The Stork instant search in-action](stork.gif)

In order to understand how these search tools work, and the design problems involved, I built a technical demo in Go, compiled to Wasm.

## Technical Demo

My technical demo is inspired by Stork and uses a near-identical configuration file setup. So it had to be named after a bird 🐦 too — [healeycodes/crane-search](https://github.com/healeycodes/crane-search).

Crane is two programs. The first program scans a group of documents and builds an efficient index. 1MB of text and metadata is turned into a 25KB index (14KB gzipped). The second program is a Wasm module that is sent to the browser along with a little bit of JavaScript glue code and the index. The result is an instant search engine that helps users find web pages as they type.

The full-text search engine is powered in part with code from Artem Krylysov's blog post [Let's build a Full-Text Search engine](https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full-text-search-engine/).

The core data-structure is an inverted index. It maps words to document IDs. This means we can check very quickly (i.e. 100µs within Wasm) which documents a word can be found in. At build time, we loop over the document files and add to the inverted index. We also create a list of results so we can relate a document ID to a document's metadata (defined in the configuration file). The metadata includes the title and URL so that we can display a list of search results in the browser.

We can return results for multiple keywords by looking at the common document IDs for each keyword. This takes linear time as the IDs are inserted into the index in order.

During the build stage, we perform normalization to make the search engine more helpful and accurate. For example, if you search for `fish` you probably want to match `Fishing`. Words are turned into lowercase, common words are dropped, and words are reduced into their root form (aka [stemming](https://en.wikipedia.org/wiki/Stemming) — Krylysov's example code uses [Snowball](https://github.com/kljensen/snowball) as a stemmer).

The document `"Alice likes to go fishing"` becomes `["alice", "go", "fish"]`. The search term `"Fishing"` is also turned into a token, at search-time it becomes `"fish"`.

To handle the document metadata, Crane's build program is passed the location of the configuration file. This tells it where to find the documents to index on disk, where they can be found on the static website, and what the search result should be titled.

```toml
[input]
files = [
    {
        path = "docs/essays/essay01.txt",
        url = "essays/essay01.txt",
        title = "Introduction"
    },
    # etc.
]

[output]
filename = "dist/federalist.crane"
```

The index and result data is combined into a single struct called a store. It's encoded using [encoding/gob](https://golang.org/pkg/encoding/gob/), a binary encoding that is faster and smaller than JSON ([one](https://gist.github.com/evalphobia/a2ba2636acbc112f68dcd89e8b81d349), [two](http://ugorji.net/blog/benchmarking-serialization-in-go)).

```go
file, err := os.Create(config.Output.Filename)
check(err)
defer file.Close()

encoder := gob.NewEncoder(file)
store := search.Store{
    Index:   index,
    Results: results,
}
err = encoder.Encode(store)
check(err)
```

The second part of Crane is the program that's compiled into Wasm and sent to the browser. It uses the Go standard library package [syscall/js](https://golang.org/pkg/syscall/js/).

> Package js gives access to the WebAssembly host environment when using the js/wasm architecture. Its API is based on JavaScript semantics.

Our browser program makes two functions available in the global JavaScript scope. A new goroutine is spawned when they're called.

```go
// Store is a shared type between both programs (build/browser)
var store search.Store

func main() {
    js.Global().Set("_craneLoad", js.FuncOf(load))
    js.Global().Set("_craneQuery", js.FuncOf(query))
    // Wait indefinitely,
    // required so that our Wasm module doesn't exit straight away
    select {}
}

func query(this js.Value, args []js.Value) interface{} {
    searchTerm := args[0].String()
    start := time.Now()
    matchedIDs := store.Index.Search(searchTerm)
    log.Printf("Search found %d documents in %v", len(matchedIDs), time.Since(start))

    var results []interface{}
    for _, id := range matchedIDs {
        results = append(results,
            map[string]interface{}{
                "title": store.Results[id].Title,
                "url":   store.Results[id].URL,
                "id":    store.Results[id].ID,
           })
    }

    // This will arrive as JSON
    return js.ValueOf(results)
}

// ..
```

Here's a high-level view of how everything fits together.

![Flow chart of the in-browser logic](crane-search-small.png)

A JavaScript support file, [wasm_exec.js](https://github.com/golang/go/blob/7be8358f70ff858f28b9aefe11986da25f1762bc/misc/wasm/wasm_exec.js), provides a Go class that connects JavaScript ⟷ Go. The Wasm module is loaded with [WebAssembly.instantiateStreaming](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WebAssembly/instantiateStreaming) ([polyfill available](https://github.com/golang/go/blob/b2fcfc1a50fbd46556f7075f7f1fbf600b5c9e5d/misc/wasm/wasm_exec.html#L17-L22)) and is passed to the Go instance. The store is downloaded via the Fetch API and is passed to the running Wasm module where the gob-encoded binary is decoded.

I wrote a ~50 line [class](https://github.com/healeycodes/crane-search/blob/main/dist/crane.js) that abstracts away most of this to make the project easier for a developer to interact with.

```javascript
const crane = new Crane("crane.wasm", "federalist.crane");
await crane.load()

const results = crane.query('some keywords');
console.log(results);
```

The result is, well, it's no Stork. But it works and it's fast. If there's one way to get a developer to respect the complexity of your project, have them build a technical demo of it with 1/50th of the features.

![Searching for keywords with Crane from the same dataset as Stork](crane.gif)

The binary is too large at around 3MB (but I didn't optimize for this). The solution for this is to read [Reducing the size of Wasm files](https://github.com/golang/go/wiki/WebAssembly#reducing-the-size-of-wasm-files) from the Go GitHub or read any of the blogs that come up when you search `shrink go wasm binary`. The best advice I found is to use [TinyGo](https://github.com/tinygo-org/tinygo) to generate the Wasm (the "Hello world" example is 575 bytes) and to limit the number of libraries you include. TinyGo actually artificially limits the libraries that you can use because it implements a subset of Go — which means the encoding/gob package is out (encoding/json too).

## Design Problems

If there's one thing I learned, it's that search is a reasonably solved problem. It's easy to get started; stand on the shoulders of algorithms and away you go. The key problem in this space is the developer experience. Configuring the build process of a static site can be tricky when things beyond the standard content (e.g. posts, images) need to be generated. In order to return metadata about each result, these types of search tools require a config (usually TOML or JSON) and generating these files adds complexity.

Setting up a search bar with any features beyond the basics (e.g. auto-complete, full keyboard support, highlighted excerpts) is non-trivial too, especially when you want it to feel good and always react fast.

It's also hard to build a solution that solves all use cases. Every time a feature is added (e.g. zero tolerance for false positives, fuzzy search, excerpts, partial matching) the index size grows by an order of magnitude. One rather esoteric idea that I toyed around with was building a search solution that split the index depending on the letters the user entered. Having an index for all the words that start with `a` and one for `b` and so on. However, English letter frequency is weighted towards things like `th` (see [English Letter Frequency Counts:
Mayzner Revisited](https://norvig.com/mayzner.html)).

There's also the issue of getting people to contribute to your project. Wasm is a reasonably young technology and building, testing, and debugging your code feels harder than it should be. Building a full-stack Wasm project always feels like it requires one-too-many steps. (For Go, it's worth calling out [wasmbrowsertest](https://github.com/agnivade/wasmbrowsertest) that automates in-browser testing.)

In [Introduction to WebAssembly](https://rsms.me/wasm-intro), the best low-level introduction to the technology I've seen, Rasmus Andersson writes:

> The web is an amazing concept that is currently falling short carrying a heavy legacy from the past. In order to bring truly great experiences to the web and blur the lines of what is and what isn’t a “native” experience, WebAssembly—or something else like it—needs to happen.

A search tool for static sites is a good fit for Wasm, and I hope to see projects in this space continue to evolve. It's a noble goal to build a project which actually has **four** design areas when you think about it:

- Back-end (build process)
- Back-of-the-front-end (search algorithm/Wasm)
- Front-of-the-front-end (results UI)
- Developer integration (APIs, docs)

So, thank your local open source developer today.

<br>

<small>Stork GIF sourced from https://github.com/jameslittle230/stork</small>


Building a technical demo to understand this blossoming area of technology.

WebAssembly Search Tools for Static Sites


I wanted to write a program that I could feed a bunch of novels and then produce similar text to the author's writing.

One method of generating fake but familiar looking text is to use a Markov chain generator. There is a fantastic Python library for doing this called [jsvine/markovify](https://github.com/jsvine/markovify) but I wanted to learn more about how it works under the hood so I implemented the algorithms from scratch!

Before we get to text generation, let's start by generating some fake weather. Skip the following section if you're familiar with Markov chains.

## Fake Weather Generation

I have some historical weather data from my town. The weather here is either sunny or rainy. When it's sunny, there's a good chance that it remains sunny the next day. It rarely rains but when it does it often rains for a few days.


Rather than using a naive probability (e.g. there's an ~83% chance it is sunny vs. rainy on any given day) we'll use a Markov chain to generate more realistic looking data. Our generated data will have streaks of weather which will more closely resemble real life patterns.

To be specific, if it's sunny there's a 10% chance it will be rainy the next day and a 90% chance it will stay sunny. If it's rainy then there's 50% chance it will be sunny the next day and a 50% chance it will stay rainy.

Here's a diagram of this two-state Markov process.

![A Markov chain with the sunny/rainy values as described above.](weather-markov-chain.png)

Instead of using weights to describe probability, let's distribute the states in a list that we randomly pick from. I find that defining Markov chains like this (while far more computationally expensive) is easier to debug.

```python
weather_chain = {
    'sun': ['sun', 'sun', 'sun', 'sun', 'sun', 'sun', 'sun', 'sun', 'sun', 'rain'],
    'rain': ['sun', 'rain']
}
```

We consume this model by picking a random starting state and using the current state to choose randomly from the possible future states. We do this over and over to generate a sequence.

```python
import random

# the initial state is chosen randomly
weather = [random.choice(list(weather_chain.keys()))]

for i in range(10):
    weather.append(random.choice(weather_chain[weather[i]]))
```

In this example output, we can see that rainy days are 'sticky' as we would expect from the model. 

```python
['rain', 'rain', 'rain', 'sun', 'sun', 'sun', 'sun', 'sun', 'rain', 'rain', 'sun']
```

## Fake Text Generation

Instead of having a predefined Markov chain like we saw in the previous section, let's build one from real data. The full source code for this article and the text corpuses can be found at [healeycodes/markov-chain-generator](https://github.com/healeycodes/markov-chain-generator).

The code excerpts assume that we're generating fiction. So the source text must have capital letters at the start of sentences and full stops at the end of sentences. We can use these two markers to generate text chunks that have a beginning and an end.

In our weather example, the state size was one — to decide the next step in the sequence, we only considered one previous day of weather. When it comes to generating language, a state size of one sometimes isn't big enough and the arrangement of words can be too random to be interesting. A state size of two is a good starting point. Going higher than two can produce text that is too similar to the original text corpus.

The following function builds a Markov chain in the same format as our weather example. It takes a source text and a state size and returns a dictionary where the keys are the current state and their values are a list of possible future states. The lists contain duplicates and this is how we handle the probabilities of future states.

```python
def build_model(source, state_size):
    '''
    Given a corpus and a state size, build a Markov chain.
    '''
    source = source.split()
    model = {}
    for i in range(state_size, len(source)):
        current_word = source[i]
        previous_words = ' '.join(source[i-state_size:i])
        if previous_words in model:
            model[previous_words].append(current_word)
        else:
            model[previous_words] = [current_word]

    return model
```

Given a tiny source of `'An apple is very good. An orange is very bad.'` and a state size of `2` it will produce the following Markov chain. Since the source was so small there are only four possible complete sentences.

```json
{
  "An apple":[
    "is"
  ],
  "apple is":[
    "very"
  ],
  "is very":[
    "good.",
    "bad."
  ],
  "very good.":[
    "An"
  ],
  "good. An":[
    "orange"
  ],
  "An orange":[
    "is"
  ],
  "orange is":[
    "very"
  ]
}
```

The following function consumes the Markov chain model and generates some fake text for us. To achieve a minimum length, we keep generating until we hit the minimum length and then keep going until we reach a token that ends with a full stop. To find a correct starting point, we pick a random key (two words) where the first character is a capital letter.

```python
def generate_text(model, state_size, min_length):
    '''
    Consume a Markov chain model (make sure to specify the <state_size> used)
    to generate text that is at least <min_length> size long.
    '''
    def get_new_starter():
        return random.choice([s.split(' ') for s in model.keys() if s[0].isupper()])
    text = get_new_starter()

    i = state_size
    while True:
        key = ' '.join(text[i-state_size:i])
        if key not in model:
            text += get_new_starter()
            i += 1
            continue

        next_word = random.choice(model[key])
        text.append(next_word)
        i += 1
        if i > min_length and text[-1][-1] == '.':
            break
    return ' '.join(text)
```

Here are those four possible complete sentences from our previous Markov chain. These can be combined infinitely by our function.


```python
'An apple is very bad.'
'An orange is very bad.'
'An orange is very good.'
'An apple is very good.'
```

## Some Examples

We can now feed in large amounts of text from an author and generate fake writing! In fact, any corpus that uses sentences will work with our program. For example, here is the result of feeding in a few Wikipedia articles.

> Cricket is more similar to dust devils and landspouts. They form when a homicide rate of 34.2 per 100,000 was reported. This included 15 officer-involved shootings. One shooting led to the latest hour of it; and lately, I know of but love, desperate love, the worst of all the more remote islands. At around the field. One of Wollstonecraft's most popular metaphors draw on military concepts: Disease is an early type of fiction that were quick to resort to violence. One of Wollstonecraft's favorite arguments.

Here's some Edgar Allen Poe.

> Count could recollect, it was never worth the trouble of the stranger. But, as usual, enveloped in frequent rolls, or bandages, of linen; but, in place of conference with the whole matter as a natural result of the river, and, plunging through a single slender gold chain, and throws a tranquil but magical radiance over all. I cannot enter into details just now: but it was found, on Sunday morning, that he was forced to allow, had ever suspected of existing in the heathen is unwonted; and fickle-mindedness has ever thought of this life and of cutting him off with a layer of plaster, thickly gilt and painted.

## Further Resources

The full source code and a few text corpuses on [healeycodes/markov-chain-generator](https://github.com/healeycodes/markov-chain-generator).

The sun/rain example was taken from [Wikipedia](https://en.wikipedia.org/wiki/Examples_of_Markov_chains#A_simple_weather_model). Victor Powell's article, [Markov Chains](https://setosa.io/ev/markov-chains/), was also helpful for my initial understanding and is worth checking out for the interactive graphics alone.

Some of the articles about Markov chains are a little inaccessible to those without a maths background. However, there's a Simple English version of Wikipedia. Many articles have an alternative page which you can find by replacing the `en` in the URL bar with `simple`. For example, the [simple version](https://simple.wikipedia.org/wiki/Markov_chain) of the Markov chain page.

Daniel Shiffman also covered Markov chains in a Coding Challenge on [The Coding Train](https://www.youtube.com/watch?v=eGFJ8vugIWA).


Generating random but familiar text by building Markov chains from scratch.

Generating Text With Markov Chains

I chose five personal websites out of the thirty or so I visit regularly and tried to understand what it is about them that utterly captures me.

Whether or not it is obvious to you, I have stolen many things from the following websites. Be it design tweaks, post ideas, or even turns of phrase. These websites are incredible and you should consume them.

Good artists borrow, great artists ~~steal~~ use the inspect tool.

# [arcana.computer](https://arcana.computer)

I have been following Justin Duke's writing for close to two years. His older page is now [depreciated](https://jmduke.com) and he posts to _arcana dot computer_, an [open source](https://github.com/jmduke/arcana.computer) website filled with catalogs and footnotes and light-touch design. The main framework is Jekyll, with dynamic content powered by Airtable. The [About](https://arcana.computer/miscellany/this-site.html) page of Justin's website explains the development/design/ideas behind the website in rich detail. So in this section I'll instead focus on how I _feel_ about the website.

![The index page of arcana.computer.](arcana.computer.png)

My philosophy towards personal websites matches Justin's. He writes:

> Lastly, if there’s anything I can convince you of: you should build a personal site, you should obsess over it, you should meticulously document it, and you should have quite a bit of fun doing so. (It’s worth it.)

The content throughout these pages is personal and reads true. Although it's not a journal as such, as I read through the notes and reviews I get a secret feeling that I'm looking somewhere I shouldn't be — like peeking in a hidden diary.

Justin writes about capturing his [media diet](https://arcana.computer/catalogs/media-diet):

> This started out as a lazy compulsion, but I’ve grown rather found of this habit over time. “You are what you eat”, and all that — I’ve realized that paying more attention to how I’m spending my consumptive time has made me more focused on consuming what I’m interested in, and not simply what’s easiest.

He talks about reviewing older sections of his media diet and how it helps him recollect that time in his life — "suddenly I am taken back to my old apartment on Capitol Hill, and my three weeks of funemployment before Stripe". When coming across old words that I have written, I've also experienced this almost-olfactory flashback of thoughts.

![The catalog listing of arcana.computer.](catalog.arcana.computer.png)

Overall, I am impressed with the breadth and depth of content on Justin's website, as well as how he's made Airtable work for him. I also like that sections are marked as in-progress. I like the personal structure to it. His methods for working on this website are similar to the goals of the [Long Now](https://en.wikipedia.org/wiki/Long_Now_Foundation) and that gels with me.

# [paulstamatiou.com](https://paulstamatiou.com)

Paul Stamatiou writes long form articles about his life and technology. If someone has a curiosity about a subject that he has covered (e.g. [creating with the iPad Pro](https://paulstamatiou.com/made-on-an-ipad-pro/), or [building a lightroom PC](https://paulstamatiou.com/building-a-windows-10-lightroom-photo-editing-pc/)) I wouldn't hesitate linking one of his articles to them — perhaps without even reading it — because of the consistent high quality I have come to expect.

![Paul Stamatiou's website.](paulstamatiou.com.png)

However, this quality comes with a cost, as he describes in [Writing more](https://paulstamatiou.com/writing-more/):

> I'm going to try something different and write more short-form posts here.

> Over the years my focus has been increasing the quality of my articles. They've ended up becoming increasingly time-consuming to create.

Although these articles (which for now he seems to have dubbed "briefs") are shorter and less researched than his other writing they read as complete entries to me. On another website, by another person, they would be complete blog posts.

The more things I write, the more hesitant I am to actually publish. So the way he talks about blogging in _Writing more_ resonates with me:

> I want to get back to what blogging felt like when I started in 2005. Back when posting a few sentences and publishing it within the same computing session was so easy and fun. Where expectations were low and it didn't have to be perfect.

He has written 1210 posts since 2005. My first thought goes to the build times of such a website! In 2011, he migrated from [Wordpress to Jekyll](https://paulstamatiou.com/how-to-wordpress-to-jekyll/). This year he [tweeted](https://twitter.com/Stammy/status/1307347164599922689) that he's looking at moving again:

> really, really want to migrate my jekyll blog to Hugo + Netlify but I have so many weird jekyll hacks and collections and templates/includes that I'm sure the migration would take months of spare time.

> probably faster to build a new site from the ground up, new CSS and all

Also this year, he was interviewed about his work (he's a designer at Twitter) and about his blog on [Thunder Nerds](https://www.thundernerds.io/2019/10/writing-a-blog-and-working-at-twitter-with-paul-stamatiou/). (More people should be interviewed about their technology blogs please.)

Paul is a photographer who generates fantastic photo sets and write ups with little animated maps of the location. His [photos](https://paulstamatiou.com/photos/), and the way they are arranged, is truly fantastic. He has of course written [thousands of words](https://paulstamatiou.com/photos/gear/) about his camera gear.

# [arp242.net](https://www.arp242.net)

Martin Tournoij is the creator of [Goat Counter](https://www.goatcounter.com/) and has posts dating back to 2013. His personal website is Jekyll-based and [open source](https://github.com/arp242/arp242.net). I usually run into his writing on [lobste.rs](https://lobste.rs) (a computing-focused community).

![Posts by Martin Tournoij](arp242-posts.png)

I like the one-page layout of the landing page. There's a list of posts and projects, a picture, and a link to his CV. My favorite post of his is [Analytics on personal websites](https://www.arp242.net/personal-analytics.html) — where he argues in part for vanity statistics:

> As for "vanity stats" or "stats to stroke your ego": I think that’s actually a valid use case as well. After you spent quite a bit of your spare time writing an article it’s just nice to know people are actually reading it. There’s nothing wrong with being validated – it’s a basic psychological need and I’m not a fan of casually dismissing it.

Later on, he wrestles with the fact that since he's the creator of an analytics tool, he doesn't want this website to turn into an advertising channel for it.

Martin doesn't shy away from controversial subjects on his blog. He writes about freedom and democracy, he pushes for empathy towards those he disagrees with. He writes without restraint which is admirable in itself.

He uses his own CSS template ([arp242/hello-css](https://github.com/arp242/hello-css)) which is worth a look. If you'll allow me to use a vague statement, his website has a unique visual readability to it.

# [joelcalifa.com](http://joelcalifa.com)

Joel Califa's website has whimsy. It doesn't take itself seriously.

![The third section of Joel Califa's website](joel-buttons.png)

Clicking this button makes text spawn and fly away and fade (the snippets are things from the old web like: `<i>`, `<frameset>`, `<marquee>`, etc). There's an illustration of his head that hides away when you hover near. The writing in the section headers frizzles with energy.

Joel's website successfully serves as both a work portfolio and a design blog. His _Work_ page describes itself as a "A sample of text-heavy case studies for patient visitors." The design blog has "Low frequency, high quality design articles."

My favorite post is [Tiny Wins](http://joelcalifa.com/blog/tiny-wins/):

> I recently shipped two things at GitHub that had an impact beyond my wildest dreams.

Where he discusses the work involved in designing dynamic favicons for the Pull Request page:

> Now browser tabs will always show a PR’s current build status.

As well as adding an arrow that signals which branch your changes are "flowing" into:

> Before releasing this, people would regularly confuse which branch would be merged into which.

He writes with authentic authority. He covers subjects that seem so obvious after you read them. Like in [Revisiting :Visited](http://joelcalifa.com/blog/revisiting-visited/), where research, the web specification, and its practical uses, are combined:

> A Nielsen study summed this up nicely over ten years ago, “People get lost and move in circles when websites use the same link color for visited and new destinations. To reduce navigational confusion, select different colors for the two types of links.”

> Can’t we, as an industry, get behind that reasoning? A “visited” link isn’t that far off from a “read” email. They both provide the user with the tacit understanding of where they’ve been.

Joel exists in the wonderful space between technology and design where he is addressing problems that directly relate to me. For example, his website is the first place I read [a serious defense](http://joelcalifa.com/blog/unsolicited-dating-advice/) of the `month/day/year` date ordering system.

It fills me with joy that I have only read half of his content.

# [rsms.me](https://rsms.me)

Rasmus Andersson has the prettiest website in this list. An elegant three-column layout that perfectly scales to the browser's width — dropping to two columns then one column. When a page is selected from the top right menu, the background changes to a rich color and shifts the menu up or down. With a wide enough browser, gray bars frame the website on either side.

![rsms.me main menu.](rasmus-menu.png)

Rasmus is the creator of the [Inter](https://rsms.me/inter/) font and uses it to great effect with bold hover colors.

![rsms.me hover effects.](rasmus-hover.png)

Even his [404 page](https://rsms.me/bad-url) is sharp.

After looking around more, I found a `<script>` tag that includes [main.js](https://rsms.me/res/main.js), a debug tool that must have been used to develop the grid layout (which I think is based on [rsms/raster](https://rsms.me/raster/)). Pressing alt+D or alt+G overlays a system of boxes and dots.

![Debug boxes and dots over the post list.](rasmus-boxes.png)

Rasmus's last post was in 2017 — [Introduction to WebAssembly](https://rsms.me/wasm-intro) — but it's one that I revisit often as it's a clear and detailed explanation of the technology.

His use of the right arrow symbol (U+2192) as well as the rounded hover effect on titles (e.g. "Projects" and "Thoughts and ideas") is something that I closely copied for my own website as it is just too perfect.

Like Paul Stamatiou, Rasmus has hundreds of articles and has been blogging for a long time – almost two decades. The earlier posts are more likely to be reblogs, quotes, links, and small thoughts. The kind of things that Twitter is now used for.

Unlike Twitter, the permanence of these small thoughts is poetic. Here, in 2002, a new font is announced next to something like a diary entry without a paragraph break in between:

> I’ve completed a new typeface. It’s a sweet little thing called Hovden Stitch. Yes your guess was correct. It looks like stitches, cross-stitches to be precise. Go get it for your mac or pc right here. Yesterday I hung out on a free festival here in Trollhättan. Laurel Music was great. Paola sucked. Laurel Music is playing at the next Club Feber in Trollhättan.

Hovden Stitch is alive today at [rsms/hovden-stitch](https://github.com/rsms/hovden-stitch).

A decade and one month ago, a cosy post called [The snow is gently falling](https://rsms.me/archive/2010/building-some-software-while-the-snow-is-gently) has a polaroid of a 15" MacBook, a notebook, a warm drink, and the snow outside. A single sentence reads:

> Building some software while the snow is gently falling down over a quiet Stockholm.

# On Reading Writing

You cannot meet someone through their writing. There is always something held back, something distorted. But still, I try. As I spend more time on the internet in these COVID-times, I find myself running into a problem that plagued me during my undergrad years: is it possible to truly connect with someone through language?

One cynical view of language is that it's referential. David Foster Wallace [writes](https://www.dalkeyarchive.com/a-conversation-with-david-foster-wallace-by-larry-mccaffery/) about this view:

> In order for language both to be meaningful and to have some connection to reality, words like “tree” and “house” have to be like little pictures, representations of little trees and houses. Mimesis. But nothing more. Which means we can know and speak of nothing more than little mimetic pictures. Which divides us, metaphysically and forever, from the external world.

Thinking about this stuff puts me in a real funk and I begin to think about reading and speaking like it's a battle. Like I am trapped inside with my secret language and you are trapped inside with yours. The glyphs arrive in my browser and I read and scroll and read and I'm trying to meet you somewhere between my head and yours — to not be alone.

People that inspire me the most (and that I steal from most regularly).

Personal Websites and Internet Writing


I have been learning chess (again) and how to program a chess engine (for the first time) over the last month. After skimming some introductory texts, I was convinced that building a simple chess engine — one that would put up a fair fight against a casual player — would take no more than a few days.

I was wrong.

But I made it there in the end and created a toy chess engine ([healeycodes/andoma](https://github.com/healeycodes/andoma)) that I am proud of. It can play a game of chess and solve simple chess puzzles like mate-in-two or mate-in-three. It has a slim UCI interface which means it can be hooked up to lichess.org via [lichess-bot](https://github.com/ShailChoksi/lichess-bot) — a bridge between the lichess API and chess bots.

The first speed bump in its development was grasping the computational the complexity of chess — how fast, and wide, the search tree grows. When a chess game starts, white can open in twenty different ways and black can respond in twenty different ways also. After the first full turn, there are 400 variations possible. After the third full turn, there are [over 119 million](https://en.wikipedia.org/wiki/Shannon_number#Shannon's_calculation).

Claude Shannon calculated that there are around `10^120` possible games of chess in his seminal paper _Programming a Computer for Playing Chess_ in 1950. In _Rage Against the Machines_, Nate Silver [quotes](https://fivethirtyeight.com/features/rage-against-the-machines/) Diego Rasskin-Gutman, who said:

> There are more possible chess games than the number of atoms in the universe.

Given unlimited resources, it actually doesn't take many lines of code to calculate every legal variation of chess. Here, the Python package [python-chess](https://python-chess.readthedocs.io/en/latest/) is used for board representation and legal move generation.

```python
from chess import Board, Move, STARTING_FEN

# an adjacency list
positions = {}

# depth-first search from a FEN string
def generate_tree(fen):
    board = Board(fen)
    legal_moves = list(board.legal_moves)
    if fen in positions:
        positions[fen] += legal_moves
    else:
        positions[fen] = legal_moves

    for move in legal_moves:
        board.push(move)
        next_fen = board.fen()
        board.pop()
        
        generate_tree(next_fen)

try:
    generate_tree(STARTING_FEN)
except RecursionError: 
    print(len(positions) + sum(len(p) for p in positions))
```

This program throws a `RecursionError` and prints `59691` — the number of different positions the search tree contained when it crashed. All we need to fix this, is another universe to run the program in.

Given that there are computational limits to abide by, as well as the time control rules of chess, improvements over a naive brute force search must be made.

## Evaluation

In order to search for good positions, it is necessary to understand what makes a good position _good_. The most simplistic way of describing a position's strength is to compare the total value material of each side.

Tomasz Michniewski, author of the [Simplified Evaluation Function](https://www.chessprogramming.org/Simplified_Evaluation_Function), defined some values that are "designed specifically to compensate for the lack of any other chess knowledge". This is perfect for me — a beginner chess player.

This snippet sums the material on the initial board using Michniewski's piece values.

```python
import chess

piece_values = {
    chess.PAWN: 100,
    chess.ROOK: 500,
    chess.KNIGHT: 320,
    chess.BISHOP: 330,
    chess.QUEEN: 900,
    chess.KING: 20000
}

board = chess.Board(chess.STARTING_FEN)
white_material = 0
black_material = 0

for square in chess.SQUARES:
    piece = board.piece_at(square)
    if not piece:
        continue
    if piece.color == chess.WHITE:
        white_material += piece_values[piece.piece_type]
    else:
        black_material += piece_values[piece.piece_type]
```

Michniewski also provides [piece-square tables](https://www.chessprogramming.org/Piece-Square_Tables), which alter the value of a piece depending on which square it sits on. For example, it's better for pawns to progress up the board and it's better for knights to be near the center of the board.

The bonus of a square may be positive, neutral, or negative. The piece-square tables are presented from White's POV and must be mirrored for Black.

```text
# a knight's bonuses depending on square
-50,-40,-30,-30,-30,-30,-40,-50,
-40,-20,  0,  0,  0,  0,-20,-40,
-30,  0, 10, 15, 15, 10,  0,-30,
-30,  5, 15, 20, 20, 15,  5,-30,
-30,  0, 15, 20, 20, 15,  0,-30,
-30,  5, 10, 15, 15, 10,  5,-30,
-40,-20,  0,  5,  5,  0,-20,-40,
-50,-40,-30,-30,-30,-30,-40,-50,
```

For the king, Michniewski provides two tables — one for the middle game and one for the end game. He defines the end game as being either if:

> Both sides have no queens or

> Every side which has a queen has additionally no other pieces or one minorpiece maximum.

<br>

The evaluation of a chess board is one of the things that's kept me interested in chess engines. Evaluation rules are easy to add and take away. I refactored the code from Go to Python to be able to prototype different rules faster.

After piece-square tables, one might look at pawn structure, mobility, center control, connectivity, trapped pieces, king safety, space, tempo, and other patterns (this list is taken from the Chess Programming Wiki's [Evaluation](https://www.chessprogramming.org/Evaluation) page).

## Searching With Minimax

[Minimax](https://en.wikipedia.org/wiki/Minimax) is a search algorithm that finds the next optimal move by minimizing the potential loss in a worst case scenario. Chess is a game of perfect information — by looking at the board it's possible to know exactly what an opponent is capable of. However, this search for moves is limited by the evaluation function and the depth that computing resources are able to reach.

The search space is a tree of legal moves which grows exponentially at every level (the average branching factor is around 35). By the time the tree is explored, the path to many future boards is known as well as which path restricts the opponent's possible gains the most.

The leaf nodes of the tree return the evaluation of their current state. Non-leaf nodes inherit their value from a descendant node. Eventually, the recursive function reduces down to a value for the given board.

This function can be used to pick the next best move by calling it on every legal move available in the current turn. A great visual resource for this algorithm is Sebastian Lague's [Algorithms Explained – minimax and alpha-beta pruning](https://www.youtube.com/watch?v=l-hh51ncgDI).

```python
def minimax(board, depth, maximizing_player):
    if depth == 0 or board.is_game_over():
        return evaluate(board)
    if maximizing_player:
        value = -float('inf')
        for move in board.legal_moves:
            board.push(move)
            value = max(value, minimax(board, depth - 1, False))
            board.pop()
	    return value
    else:
        value = float('inf')
        for move in board.legal_moves:
            board.push(move)
            value = min(value, minimax(board, depth - 1, True))
            board.pop()
        return value
```

## Alpha-beta pruning

My chess engine uses [alpha-beta pruning](https://en.wikipedia.org/wiki/Alpha%E2%80%93beta_pruning) as an improvement over the naive minimax algorithm — which does not fare well against the exponential nature of chess. Branches of the search tree can be eliminated when it is clear that another branch shows more promise. This significantly reduces the number of moves required to be generated.

![Branches of a minimax search tree being stopped early](alpha-beta-pruning.png)

By reducing the depth of branches that will not bear fruit we can search deeper down the better parts of the tree.

The speed of alpha-beta pruning can be increased by applying [move ordering](https://www.chessprogramming.org/Move_Ordering). This is where the more promising branches of the search tree are searched first — which means less time is spent in the worst branches as they will be cut off early.

Move ordering cannot be 100% accurate but it's a powerful optimization.

In [my engine](https://github.com/healeycodes/andoma), a cheap (but not perfect) `move_value` function is used to sort the initial legal move nodes from best to worst. The logic of this function is capture in its docstring:

```python
'''
How good is a move?
A promotion is great.
A weaker piece taking a stronger piece is good.
A stronger piece taking a weaker piece is bad.
Also consider the position change via piece-square table.
'''
```

## Communication and the UCI Protocol

The [Universal Chess Interface](https://github.com/healeycodes/andoma/blob/main/uci-interface.md) (UCI) is a open protocol to hook up chess engines to user interfaces. The communication is done through standard input and standard output and messages end with `\n`. The move format is long algebraic notation — like `e2e4`, or `e1g1` for white short castling, and an example of a promotion to queen is `a7a8q`.

There are many configuration commands in the specification and it initially seemed overwhelming. After debugging, I found that not many were required to get a chess engine to the _Hello World_ stage.

In order to hook my chess engine up to lichess via [lichess-bot](https://github.com/ShailChoksi/lichess-bot), I implemented the following. These commands are send to the engine from lichess-bot:

- `uci` — the engine reports its name, authors, and `uciok`
- `isready` — the engine reports that it's ready: `readyok`
- `position startpos moves e2e4` — the engine sets it's internal state to match the list of moves
- `go` — the engine should now calculate and respond with the next best move, like `bestmove g8f6`

## A Promise

I have found a great joy interacting with the chess community over the last month. Lichess is a fantastic resource and endlessly fun to play on. The UI is slick and light — and the post-match analysis is revealing and simple to use. It's [open source](https://github.com/ornicar/lila) and relies on donations and sponsorships.

I took breaks from writing this article to play chess against Dad on a real board. We're missing a white rook and use a pencil sharpener as a replacement piece. He has been telling me stories about playing chess decades ago — before IBM's [Deep Blue](https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)) emerged and beat Garry Kasparov, the reigning world champion, on its second attempt in 1997.

If you are within my social circle, you will have experienced me evangelizing chess and chess engines over the last month — now that I have published this, I promise to chill out a little bit.

## Other Resources

To build [healeycodes/andoma](https://github.com/healeycodes/andoma), I used the following resources (and recommend all of them).

- [Chess Programming Wiki](https://www.chessprogramming.org/Main_Page) — an incredible wealth of knowledge exists here. I spent hours reading through intermediate and advanced concepts just for fun
- [Algorithms Explained – minimax and alpha-beta pruning](https://www.youtube.com/watch?v=l-hh51ncgDI) — Sebastian Lague's breakdowns of algorithms are very approachable and this one is no different
- [A step-by-step guide to building a simple chess AI](https://www.freecodecamp.org/news/simple-chess-ai-step-by-step-1d55a9266977/) — this JavaScript-based tutorial introduces concepts with explicit code snippets. The source code of the final solution is quite readable too
- [thomasahle/sunfish](https://github.com/thomasahle/sunfish) — a simple, feature complete engine to learn from 
- [zserge/carnatus](https://github.com/zserge/carnatus) — a clone of sunfish in Go

<br>

<small>Thanks to [Jez9999](https://commons.wikimedia.org/wiki/File:AB_pruning.svg) for the _Alpha-Beta pruning example_ image.</small>


Exploring the computational complexity of chess. Code snippets in Python so you can do the same.

Building My Own Chess Engine


Lately, I've been playing a lot of [Among Us](https://en.wikipedia.org/wiki/Among_Us) with my friends. It's a refreshing change of pace from the competitive FPS titles that we usually play. However, we've managed to inject a 'healthy' dose of competitiveness into our games in the form of a league that I built some software for. 

We recently crossed the 1000 games mark and players have been making feature requests and also submitting PRs! It has been open sourced at [healeycodes/among-us-friends](https://github.com/healeycodes/among-us-friends).

The league website is a React/Node/Express application that calculates a player's [Elo rating](https://en.wikipedia.org/wiki/Elo_rating_system) — a relative measure of one player's skill against another — as well as other performance statistics like crew/imposter win rate. It ranks players and graphs their recent performance on the home page. 

![The home page - a list of players, their win/loss rate, and Elo charts](preview.png)

When a player goes to their auto-generated player page, they can see a history of their games and who played on each side, the amount of Elo they won or lost each game, and an Elo history graph for the whole season. They also have a trophy section. Some of the trophies include: win streaks (5, 10, 15), being over 1300 Elo at the end of the season, best crew, best imposter, and more. As well as dynamic trophies, there are also manual trophies like Most Improved (which we vote on) and being a code contributor to the project.

![A player's trophy section](preview-trophies.png)

I manually track each game we play in a Google Sheets spreadsheet. This data is brought into the application via the Google Sheets API v4. To set this up, I grabbed the spreadsheets ID (which can be copied straight from the URL bar) and created a restricted API key via Google Console.

The data arrives in a series of rows with each row representing one game. An Among Us game can be split up into three parts of data. 

- The crew (7 or 8 players)
- The imposters (2 players)
- The winner (crew or imposters)
- The map short name (skeld, polus, mira)

Since a spreadsheet cannot be efficiently queried we bring in all the rows on every request (Sheets API usage is unlimited and free). If performance is ever an issue, caching can be added.

A player object is built for each person in the league and the following information is calculated.

- Crew wins/losses
- Imposter wins/losses
- Current Elo
- Elo history
- Their game history

A season object is built for each season that has been played (including the current season). This has information about trophies and other general stats such as map win/loss data.

## Elo ratings

For Elo calculations, I use the npm package `elo-rating`. In the Elo rating methodology, the _K-factor_ is the maximum possible adjustment for a game. I have based the league's system off the International Chess Federation's rules. Before a player reaches 30 games, their rating is more volatile. After 30 games, their K-factor drops from 40 to 20.

Elo was designed for games with two players. I was initially unsure how to solve the problem of Among Us having different team sizes. Other people have created rating systems for Among Us that separate a measure of a player's skill into crew/imposter. However, I made a product decision to reduce a player's performance down to a single number. This number is the average of a player's hidden crew Elo and hidden imposter Elo.

We alter our game settings (e.g. crew vision, kill cooldown) over time keep the game fun for impostors (so, making it harder for crew). By tracking map statistics, we have found data that goes against our commonly held beliefs. We thought The Skeld was easier to win as imposter than on Polus but it's actually the inverse (by ~10%).

Elo systems in video games are not tuned to be mathematically perfect but to reward and encourage players — a mixture of fun and correctness. Read more about this topic in [Analyzing Starcraft 2's Ranking System](http://sirlingames.squarespace.com/blog/2010/7/24/analyzing-starcraft-2s-ranking-system.html) by Sirlin.

We made the Elo system _less fair_ for players by rewarding those who play more with a tiny amount of Elo. Players that don't play for a long time in a season will have their Elo slowly decay over time. The players voted for these rules.

```javascript
const EloRating = require("elo-rating");

function EloChange(games) {
  // Based on https://ratings.fide.com/calculator_rtd.phtml
  // K = 40 for a player with less than 30 games
  // K = 20 afterwards
  const K = games > 30 ? 20 : 40;
  // From playerA's perspective, how much can be won or lost
  return function (playerA, playerB) {
    return [
      EloRating.calculate(playerA, playerB, true, K).playerRating - playerA,
      EloRating.calculate(playerA, playerB, false, K).playerRating - playerA
    ];
  };
}

// The tests for this look like:
// For new players
let eloChange = EloChange(0);
expect(eloChange(1200, 1400)[0]).toStrictEqual(30);
expect(eloChange(1200, 1400)[1]).toStrictEqual(-9);
// For everyone else
eloChange = EloChange(31);
expect(eloChange(1400, 1200)[0]).toStrictEqual(4);
expect(eloChange(1400, 1200)[1]).toStrictEqual(-15);
```

The frontend styling is mostly [water.css](https://watercss.kognise.dev/) with some in-line tweaking. The graphs are from the package [data-ui](https://github.com/williaster/data-ui) which has been depreciated in favor of [visx](https://github.com/airbnb/visx) — however, I found the data-ui examples easier to pick up and plug in.

The backend of the application is tested using Jest (w/ SuperTest for mocking requests to the server). The statistics functions are unit tested. All the tests are ran on commit/PR using GitHub Actions with the default Node YAML file that runs the following command against Node 14.x.

```yaml
    - run: cd back-end && yarn install && yarn test
```

## React

Initially, this application was hosted on Glitch and the pages were vanilla JavaScript. Over time, the league features grew and grew until it was taking a long time to add new functionality. It was also getting harder to onboard new contributors. I chose React because a few of us were familiar with it. I added react-router to handle the different pages (Ranking/Stats/About/Player).

I used Create React App to initialize the front-end and tried not to tweak it too much. With this project, my focus has been on the customers (e.g. my friends) over code quality. This is a refreshing change compared to the rest of my personal projects, where my focus is normally on clean code/testing/great patterns.

All the data comes in via a single `fetch` call to a serverless endpoint. It can take a few seconds to load but then there are no more API calls required. Clicking between the other pages is instant as the entire league data is stored in memory 🏃💨.

## Deployment

I refactored the Express application from Glitch to use [serverless-http](https://github.com/dougmoscrop/serverless-http) and [netlify-lambda](https://github.com/netlify/netlify-lambda). Now, it's a Jamstack app that deploys to Netlify on commits to the `main` branch. We also take advantage of the PR preview deployments.

When I was refactoring Glitch → Netlify I used these resources: [Netlify Functions + Express](https://github.com/netlify-labs/netlify-functions-express), [How to run Express.js apps with Netlify Functions](https://www.netlify.com/blog/2018/09/13/how-to-run-express.js-apps-with-netlify-functions/). One thing that tripped me up was fixing the API calls from the front-end to prepend `.netlify/functions/app/` to the routes.

## The Future

I have onboarded a few people to be able to use the Google Sheets system to add scores but it's tricky and there is no safety or restrictions on entering data. I have been looking at what kind of admin interfaces I could plug in while still using Google Sheets as a data source.


Making a casual game more competitive and arguing with my friends about Elo systems.

Writing Software for an Among Us League


I first heard about the [Ink](https://dotink.co/) programming language when I came across [Polyx](https://github.com/thesephist/polyx). Polyx is a productivity suite written in Ink that includes homegrown replacements for Dropbox and Trello as well as a personal relationship manager and a read-it-later service. [Linus Lee](https://thesephist.com/) is the sole author of Ink and Polyx. I read through the source code of Polyx because I was interested in owning my own personal infrastructure — and this was the start of my journey with Ink!

> A functional language that takes after modern JavaScript and Go

Ink exists in the area between a hobby project and a fully grown programming language. It has [documentation](https://dotink.co/docs/), open source [projects](https://dotink.co/docs/projects/), and it's actively developed with regular releases. It is easy to extend, and the source code is clear and understandable. It's got warts, sure, but you could write an application with it that gets you a customer and earns you a dollar.

I sent an email to Linus to chat about the language and he pointed me to some of his newer Ink projects that contained the most idiomatic code to learn from (which were [September](https://github.com/thesephist/september) and [inkfmt](https://github.com/thesephist/inkfmt)). He also fast tracked a planned VS Code [syntax highlighting extension](https://github.com/thesephist/ink-vscode) when he found out what editor I was using!

I spent a few weeks learning the language and created [Ink by Example](https://inkbyexample.com/) — a hands-on introduction to Ink using annotated example programs. Why was my first major project a learning resource? Well, writing about a topic helps me understand it but trying to teach a topic leads me to the hard questions that build mastery.

With technical topics, you meet the same problems that arise when trying to absorb a book. In [Why books don’t work](https://andymatuschak.org/books/), Andy Matuschak writes:

> Have you ever had a book like this—one you’d read—come up in conversation, only to discover that you’d absorbed what amounts to a few sentences? I’ll be honest: it happens to me regularly. Often things go well at first. I’ll feel I can sketch the basic claims, paint the surface; but when someone asks a basic probing question, the edifice instantly collapses

Until I explain a topic in a permanent medium (one that exists outside my own head) I don't know what I don't know. I fix this by building a structure from the basic principles all the way to the tricky nodes at the end of the graph. This can be via notes, an article, or a project.

## Building, Learning

The best way to learn a language is to build something. Ideally, something that solves a personal problem (this motivation will drive you through the quagmires to victory). The structure of Ink by Example is ~~modelled after~~ stolen from Go by Example.

I enjoy language resources that zoom in on a tiny slice of the syntax and give you clean examples. Usually, this starts with printing to console.

```ink
std := load('../vendor/std')
log := std.log

log('hello world')
```

If someone is fluent in another programming language they will want to know how to do _X_ in _Y_. They seek to translate the building blocks that they're familiar with; data structures, functions, and system interfaces.

The home page of Ink by Example presumes that you know what question you're asking. It's designed for an intermediate programmer.

![Hello World, Values, IO, Loops, Control Flow, Lists, Maps, Functions, Files, HTTP, Random, Sorting, Execing Processes](list.png)

When building and learning at the same time I like a resource that _shows how something works_. The 'how' — not the 'why'. A section of code annotated with enough information to get you started. A section of code that you can copy, change two lines, and ship!

![The Random example page that explains how rand() and urand() work](example.png)

The build tool chain for the project is powered by Ink and the [repository](https://github.com/healeycodes/inkbyexample) builds and deploys to Netlify on commits to the main branch. 

I set it up to be fairly hackable. There are two HTML templates (bases for index and example) that are imported as strings and formatted with Ink's `std.format`. The order that the examples are shown is controlled by `examples.ink`. The program files are structured like a table with documentation and code in parallel cells.

The program files are turned into executable code and evaluated when the test or build commands are ran. This was useful during development because it gave me full certainty that these code examples actually worked. (Unit tests would have been better!)

The templates are compiled and written to `/public` as HTML files, along with a few static files like CSS and an `og:image`.

For syntax highlighting, I read through another Ink project called [September](https://github.com/thesephist/september) and saw that it provided a print command that sent Ink source code to the terminal with syntax highlighting via ANSI escape codes. I imported the files required for highlighting and altered the escape code functions to instead wrap the lines in `<span>` elements with different class names.

```ink
` before: if comment, apply ansi.Gray function `
(Tok.Comment) -> Gray

` after: if comment, wrap in span to target via class in HTML `
(Tok.Comment) -> s => '<span class="c">' + s + '</span>'
```

The annotated examples programs are designed to print out a lot of data. This is rendered under the source code as if the file has been 'ran' in a terminal to create a natural feel for an intermediate programmer and to show the shape of the data we're dealing with.

![The section of output under the annotated program as if it has been ran via terminal](output.png)

Since everything builds to a folder called `/public`, the Netlify configuration is just two lines long. The build time is 17 seconds long.

```toml
[build]
  publish = "public/"
  command = "make build-linux"
```

## Why Learn Ink At All?

Sometimes I am too career-driven in the languages and technologies that I pick up. So I wanted to make sure that I was still learning to explore and be creative — unencumbered by StackOverflow surveys that detail what technologies make you most employable. And what is more esoteric than a language that only two people actively code with (to my knowledge: myself and Linus).

I find Ink enjoyable to write code with. It's terse, functional, and for small solutions it's extremely clear to read. Programs are easy to share and deploy; a binary and a script. After reading some of Linus's passionate [technical articles](https://dotink.co/posts/) about Ink I felt an unexplainable yearning to try it out. So I did.

The future of Ink sounds exciting. I caught up with Linus a few days ago and he hinted at an experimental implementation written in Rust. He suggested some language problems that might be fixed too. He also pointed me towards resources on interpreters and compilers which I've been devouring. Who knows — maybe I'll be writing about my own programming language one day soon.

Picking up Ink and giving back to the community.

Learning the Ink Programming Language

I wanted to learn more about web analytics so I budgeted myself a few hours and built a toy analytics system.

The goal was to collect information about page views in a _privacy focused_ manner so that no GDPR notice is required. With that in mind, I made these decisions:

- Events on a website aren't attributed to a particular user (no reading or setting cookies)
- No personally-identifying information is stored on disk

The system should be able to be used on a third-party host (e.g. GitHub Pages) and the alterations to an existing codebase should be minimal i.e. including a HTML snippet. The web server that captures the tracking requests can be hosted on anything that runs Flask.

The following analytics should be available:

- What pages users visit
- Where users are referred from
- What browsers and screen sizes are used

## Tracking Pixels

A page view can be captured by sending a request to the analytics server. This happens when a browser loads an image. By including an 1x1 transparent image (a tracking pixel) we can initiate this request.

I implemented two methods, one that works with JavaScript enabled and one that works without.

- Dynamically add an `<img>` element with the data encoded in the search parameters of the URL
- (In the case of no JavaScript) Render an `<img>` element with the path and title in the search parameters of the URL

They are toggled by the `<script>`/`<noscript>` pattern.

We'll get some data from the search parameters and the rest from the request's headers. Here's the HTML snippet that includes both methods.

```html
<script>
  // The tracking pixel path can be hosted anywhere
  // but here it's located on the same host for the test page
  const website = document.location.protocol + "//" + document.location.host
  const url = new URL(website + "/pixel.gif")

  // '/'
  url.searchParams.append("path", location.pathname)

  // 'Analytics Test Page'
  url.searchParams.append("title", document.title)

  // 'https://www.google.com'
  url.searchParams.append("referrer", document.referrer)

  // '320,568'
  url.searchParams.append(
    "resolution",
    window.screen.width + "," + window.screen.height
  )

  const img = document.createElement("img")
  img.src = url
  // When the element exists in the DOM, the request is made
  document.body.appendChild(img)
</script>
<noscript>
  
  <img src="/pixel.gif?path=%2F&title=Analytics%20Test%20Page" />
</noscript>
```

When the user's browser has JavaScript disabled there is far less information available. For either solution, the user's country and their browser information can be captured on the analytics server.

## Serving The GIF

Let's start with a Flask application that serves a tracking pixel.

```python
# analytics.py

import base64
from flask import Flask, Response
app = Flask(__name__)


@app.route("/")
def index():
    # serve the HTML snippet for testing
    return render_template("index.html")


@app.route('/pixel.gif')
def pixel():
    # transparent 1x1 GIF, 43 bytes
    gif = base64.b64decode(
        'R0lGODlhAQABAIAAAP8AAP8AACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==')
    return Response(gif, mimetype="image/gif")
```

This can be run by exporting the `FLASK_APP` variable:

```bash
#!/usr/bin/env bash
export FLASK_APP=analytics.py
flask run
```

The first time I loaded the test page, I ran into a `net::ERR_BLOCKED_BY_CLIENT` error in the console. My tracking pixel was blocked by my own ad blocker. Perhaps it's allergic to 1x1 images.

After toggling my ad blocker, the Flask logs showed that the information we want is being sent, and also that we responded with a 200 status code.

```bash
"GET /pixel.gif?path=%2F&title=Analytics%20Test%20Page&referrer=&resolution=1920%2C1080 HTTP/1.1" 200 -
```

By using the search parameters object, URL encoding (also known as percent-encoding) was used automatically. For example, `%2F` decodes to `/`. You can read more about the reasoning behind this type of encoding in the [URI Generic Syntax](https://tools.ietf.org/html/std66#page-12) RFC.

> A percent-encoded octet is encoded as a character triplet, consisting of the percent character "%" followed by the two hexadecimal digits representing that octet's numeric value. For example, "%20" is the percent-encoding for the binary octet "00100000" (ABNF: %x20), which in US-ASCII corresponds to the space character (SP).

## Anonymizing User Data

Depending on how your application is deployed you may have a reverse-proxy server before the request reaches Flask. So `request.remote_addr` might describe a local address. The reverse-proxy server will usually add the user's IP address to the `X-Forwarded-For` header. We can check for that with a fallback to `request.remote_addr`.

```python
ip = request.headers.get('X-Forwarded-For', request.remote_addr)
user_agent = request.user_agent.string
```

The `User-Agent` header describes the device. For Chrome it looks something like this: `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36`. Some mobiles also add the firmware version. We don't want to keep this, or an IP address, on file because both pieces can be identifiable information which would require a GDPR notice.

Rather than keeping the exact `User-Agent` we can strip away the information that's most important to us — browser and major version.

```python
# 'chrome 84'
user_agent = request.user_agent.browser + ' ' + \
    request.user_agent.version.split('.')[0]
```

Converting an IP address to a country is a little harder. A simple solution is to use a free geolocation API but shipping IP addresses to a third-party service isn't exactly privacy focused. I found [GeoIP2-python](https://github.com/maxmind/GeoIP2-python), a package that can be combined with a free [GeoLite2 country database](https://dev.maxmind.com/geoip/geoip2/geolite2/). With this method, the geolocation happens without any user data leaving our server.

After downloading the database, the package can be used to get a country from an IP:

```python
import geoip2.database
# this is expensive, so use this for multiple requests
reader = geoip2.database.Reader('./GeoLite2-Country.mmdb')
country = ''
try:
    response = reader.country('127.0.0.1')
    country = response.country.name
except geoip2.errors.AddressNotFoundError:
    pass
```

We could also check for [Accept-Language](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Language) via `request.accept_languages`.

The rest of the page view information comes from the search parameters and we use default values to handle the case where JavaScript isn't enabled and less information is sent.

```python
timestamp = int(time.time())
path = request.args.get('path', '')
title = request.args.get('title', '')
referrer = request.args.get('referrer', '')
width = request.args.get('resolution', ['']).pop(0)
height = request.args.get('resolution', ['']).pop()
```

## Storing Page Views

I used `sqlite3` from the standard library to save and load this data. Normally, I'd use SQLAlchemy but for this experiment I only need two functions (save/load) and I don't think I'll be extending it.

I need to be able to:

- Store the tracking information for a page view
- Retrieve tracking information for a slice of time

For tiny prototypes like this, a pattern I sometimes use is checking if the table exists whenever the module is imported. This aggressive check makes it easier to share this code without thinking about the database setup. The database file either exists or will be created.

```python
# db.py

import sys
import sqlite3
DB = 'analytics.db'

conn = sqlite3.connect(DB)
c = conn.cursor()
c.execute('''CREATE TABLE IF NOT EXISTS data (
    id INTEGER PRIMARY KEY,
    timestamp INTEGER,
    path TEXT,
    title TEXT,
    referrer TEXT,
    country TEXT,
    device TEXT,
    width INTEGER,
    height INTEGER
    )''')
conn.commit()
conn.close()


def save(timestamp=0, path='', title='', referrer='', country='', device='', width=0, height=0):
    conn = sqlite3.connect(DB)
    c = conn.cursor()
    statement = '''INSERT INTO data (
        timestamp,
        path,
        title,
        referrer,
        country,
        device,
        width,
        height
        ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)'''
    c.execute(statement, [timestamp, path, title,
                          referrer, country, device, width, height])
    conn.commit()
    conn.close()


def load(start=0, end=sys.maxsize):
    conn = sqlite3.connect(DB)
    conn.row_factory = sqlite3.Row
    c = conn.cursor()
    c.execute(
        '''SELECT * FROM data WHERE timestamp >= ? and timestamp < ?''', [start, end])
    result = [dict(row) for row in c.fetchall()]
    conn.close()
    return result
```

Here's the final tracking pixel view that saves all the data we have access to using the `save` function from above.

```python
@app.route('/pixel.gif')
def pixel():
    ip = request.headers.get('X-Forwarded-For', request.remote_addr)
    country = ''
    try:
        response = reader.country(ip)
        country = response.country.name
    except geoip2.errors.AddressNotFoundError:
        pass
    device = request.user_agent.browser + ' ' + \
        request.user_agent.version.split('.')[0]

    timestamp = int(time.time())
    path = request.args.get('path', '')
    title = request.args.get('title', '')
    referrer = request.args.get('referrer', '')
    width = int(request.args.get('resolution', '').split(',').pop(0))
    height = int(request.args.get('resolution', '').split(',').pop())

    db.save(timestamp=timestamp,
            path=path,
            title=title,
            referrer=referrer,
            country=country,
            device=device,
            width=width,
            height=height)

    gif = base64.b64decode(  # transparent 1x1 GIF, 43 bytes
        'R0lGODlhAQABAIAAAP8AAP8AACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==')
    return Response(gif, mimetype="image/gif")
```

## Displaying User Analytics

It's common to want to view analytics for a given time period like the previous week. So the view that displays user analytics should accept two parameters that define the start time and end time of the slice.

A minimal page should look something like this:

```text
Sun Aug 2 10:32:01 2020 - Sun Aug 9 10:32:01 2020

Paths
('/', 2)

Referrers
('', 2)

Countries
('', 2)

Devices
('safari 13', 2)

Resolutions
('414x736', 2)
```

When no parameters are passed, I chose to load all data from the beginning of time.

The analytics view that accomplishes all of that looks like this. `Counter` from `collections` saves a few lines of code when aggregating page views.

```python
@app.route("/analytics")
def analytics():
    start = int(request.args.get('start', 0))
    end = int(request.args.get('end', time.time()))
    data = db.load(start=start, end=end)
    paths = Counter()
    referrers = Counter()
    countries = Counter()
    devices = Counter()
    resolutions = Counter()
    for page_view in data:
        countries[page_view['country']] += 1
        paths[page_view['path']] += 1
        referrers[page_view['referrer']] += 1
        devices[page_view['device']] += 1
        resolutions['{}x{}'.format(
            page_view['width'], page_view['height'])] += 1
    start_formatted = datetime.fromtimestamp(start).strftime("%c")
    end_formatted = datetime.fromtimestamp(end).strftime("%c")
    return render_template('analytics.html',
                           start=start_formatted,
                           end=end_formatted,
                           paths=paths.most_common(),
                           referrers=referrers.most_common(),
                           countries=countries.most_common(),
                           devices=devices.most_common(),
                           resolutions=resolutions.most_common())
```

The template `analytics.html` renders a page with the `start` and `end` as the title and the properties in table rows.

For the full source code and installation instructions see the [repository on GitHub](https://github.com/healeycodes/privacy-focused-analytics).

## Improvements

Without user sessions, it's impossible to calculate bounce rate and other information about user behavior. To enable user sessions, it needs to be possible to connect together a series of page views. To keep this process GDPR-friendly, one could use a similar method to [GoatCounter](https://github.com/zgoat/goatcounter) (a big influence on this project).

GoatCounter creates a server-side hash by combining the website's ID, a `User-Agent`, an IP address, and a rotating salt. Since the hashing is a one-way operation, this "[identifies] the client without storing any personal information directly."

In GoatCounter's [GDPR notes](https://www.goatcounter.com/gdpr), there are a few aspects of this method worth highlighting.

> It’s prohibitively expensive to retrieve the IP address from the hash.

> It’s true that certain “additional information” from other parties could reveal more – such as correlating the User-Agent and location – but would be hard, and the retrieved data would be limited [...] To determine whether a personal is identifiable “account should be taken of all the means reasonably likely to be used”, and this doesn’t strike me as reasonably likely.

The strongest argument in favor of this approach is:

> If a user (i.e. a customer on your site) would contact me for their rights to have insight in their data and/or have it removed, then I would have no way to reliably do so, even if they would provide me with most of their computer’s information. It might be possible if they provide their browsing history, but if you have full access to all their browsing data then what do you need GoatCounter’s data for?

However, there are privacy implications when using hashed IP addresses, user agents, and similar pieces of data. Read more in [Introduction to the hash function as a personal data pseudonymisation technique](https://edps.europa.eu/data-protection/our-work/publications/papers/introduction-hash-function-personal-data_en) on the European Data Protection Supervisor.

### Granular Interactions

With or without session support, it might also be useful to track click events. These events could be configured manually, or automatically e.g. by attaching an event listener to every button which spawns new tracking pixels.

If further tracking is required, analytics systems tend to have an API that allows for custom events to be defined and captured.

## Further Reading

There are three open source web analytics systems that I drew inspiration from for this article. The best way to learn more about privacy focused analytics is to read their documentation, arguments for their design choices, and their codebases!

- [GoatCounter](https://github.com/zgoat/goatcounter) - [live demo](https://stats.arp242.net/)
- [simple-web-analytics.com](https://github.com/ihucos/simple-web-analytics.com) - [live demo](https://simple-web-analytics.com/app#demo)
- [Fathom Lite](https://github.com/usefathom/fathom) - [live demo](https://app.usefathom.com/share/lsqyv/pjrvs#/?from=2020-08-02%2000%3A00%3A00&site=184&to=2020-08-09%2023%3A59%3A59)

Learning more about user data collection by writing my own analytics system.

Privacy Focused Analytics From Scratch


Lately, I've been revisiting concepts and algorithms that I got stuck on when I was learning to code. One of them is the boids flocking simulation.

We'll look at how this program can be written in Go and delivered via WebAssembly. You can find all the source code for this article, as well as the [WASM demo](https://healeycodes.github.io/boids/), on [healeycodes/boids](https://github.com/healeycodes/boids).

![A flock of dark colored boids swimming around a white area](preview.gif)

A boid is a simulated bird-like object, a bird-oid. Each boid looks at their local flockmates and uses three rules to respond to their behavior — _separation_, _alignment_, and _cohesion_.

This artificial life program was first developed by Craig Reynolds. In [Flocks, Herds, and Schools: A Distributed Behavioral Model](http://www.red3d.com/cwr/papers/1987/boids.html) he describes it:

> The simulated flock is an elaboration of a particle system, with the simulated birds being the particles. The aggregate motion of the simulated flock is created by a distributed behavioral model much like that at work in a natural flock; the birds choose their own course.

## Ebiten "A dead simple 2D game library for Go"

[Ebiten](https://ebiten.org/) is well documented and has a lively community (`#ebiten` channel in [Gophers Slack](https://blog.gopheracademy.com/gophers-slack-community/)). I started learning it by extending the [example programs](https://ebiten.org/examples/) to get used to the API.

Let's look at the overall structure of our program.

When you build a game with Ebiten, you must implement the [ebiten.Game](https://pkg.go.dev/github.com/hajimehoshi/ebiten#Game) interface. We define a `Game` struct to store our flock of boids as well as the methods required by Ebiten to run our game. `Update` is called every tick to handle the game logic, `Draw` is called every frame to draw the screen, and `Layout` is used to define the game's screen size (which can be 'stretched' to fit the window).

```go
type Game struct {
    flock  Flock
    inited bool
}

// Create all of the boids, give them some random velocity
func (g *Game) init() {}

// Apply logic to our flock
func (g *Game) Update(screen *ebiten.Image) error {}

// Draw all of the boids
func (g *Game) Draw(screen *ebiten.Image) {}

// Define the game's logical screen size
func (g *Game) Layout(outsideWidth, outsideHeight int) (int, int) {}

// Set the window size, the title, and pass an instance of Game to
// ebiten.RunGame
func main() {}
```

### Game Loops

Almost every game that you've played uses a programming pattern called a [Game Loop](https://gameprogrammingpatterns.com/game-loop.html).

> A game loop runs continuously during gameplay. Each turn of the loop, it processes user input without blocking, updates the game state, and renders the game. It tracks the passage of time to control the rate of gameplay.

The frequency of a game loop is sometimes referred to as the tick rate of a game. A single iteration of the loop is a _tick_.

## Simulation Logic

In our program, a boid has the following properties:

- Position — the coordinates of the boid in the world
- Velocity — how fast and in what direction the boid is moving
- Acceleration — we change the velocity by this value during the next tick

```go
type Boid struct {
    imageWidth   int
    imageHeight  int
    position     Vector2D // Exposes `X`, `Y`, and some calculation methods
    velocity     Vector2D
    acceleration Vector2D
}
```

Every tick, we need to handle three things for every boid in our flock:

- Check edges — check if the boid has gone off the edge of the screen
- Apply rules — using the three rules, set every boid's acceleration
- Apply movement — apply acceleration to the boid's velocity

```go
type Flock struct {
    boids []*Boid
}

func (flock *Flock) Logic() {
    for _, boid := range flock.boids {
        boid.ApplyEdges()
        boid.ApplyRules(flock.boids)
        boid.ApplyMovement()
    }
}
```

To handle boids hitting the edge of the screen, we wrap them around by swapping their position to the opposite side. Their velocity means they keep flying in the same relative direction.

After we've figured out the boid's acceleration by applying the three rules, we need to update its position and velocity, and reset acceleration.

```go
func (boid *Boid) ApplyMovement() {
    boid.position.Add(boid.velocity)
    boid.velocity.Add(boid.acceleration)
    boid.velocity.Limit(maxSpeed)
    boid.acceleration.Multiply(0.0)
}
```

By adding acceleration to the boid's existing velocity, and then limiting it by a maximum speed, we get smooth movement that tends towards the desired direction and speed. Velocity can be thought of as momentum and acceleration as steering.

## The Three Rules

In the following code excerpts, you may notice that the logic is not very optimized. This is to make it easier to understand. On every tick, every boid considers the position of every other boid. We would say that the asymptotic complexity is `O(n^2)`.

Different rules use different 'inclusion' zones. Whether another boid is _local_ to another boid depends on which rule we are applying.

```go
maxForce             = 1.0
maxSpeed             = 4.0
alignPerception      = 75.0
cohesionPerception   = 100.0
separationPerception = 50.0
```

The last three values are really fun to play with and greatly affect the shape and contortions of a flock of boids. `maxForce` limits the steering of a boid and `maxSpeed` is the ultimate speed limit.

I've expanded some of Craig Reynolds' low-resolution diagrams to help explain the three rules that define a boid's behavior.

### Separation

> Steer to avoid crowding local flockmates.

The closer another boid is to us, the more motivated we are to move away from it. The white circle represents `separationPerception`. Boids outside of this area are not considered. This rule encourages boids to move into free space.

![A boid looking at its local area and deciding to move away from the average position of other boids](separation.png)

(In the canonical version of the Boid program, a boid doesn't consider other boids behind itself by applying a field of view restriction.)

```go
separationSteering := Vector2D{}
separationTotal := 0

for _, other := range restOfFlock {
    d := boid.position.Distance(other.position)

    // A boid shouldn't consider itself
    if boid != other {

        // It should only consider local boids
        if d < separationPerception {
            separationTotal++

            // We want to steer away from other boids
            // If a boid is very close, consider it more strongly
            diff := boid.position
            diff.Subtract(other.position)
            diff.Divide(d)
            separationSteering.Add(diff)
        }
    }
}

// If there are no local boids, we keep the same direction and speed
if separationTotal > 0 {
    separationSteering.Divide(float64(separationTotal))
    separationSteering.SetMagnitude(maxSpeed)
    separationSteering.Subtract(boid.velocity)
    separationSteering.SetMagnitude(maxForce * 1.2)
}

// `acceleration` will be the sum of all three rules
// we'll then take the average and apply it to `velocity`
boid.acceleration.Add(separationSteering)
```

### Alignment

> Steer towards the average heading of local flockmates.

We know the direction of a boid by looking at the velocity. We sum and take the average of other nearby boids within the `alignPerception` area. This rule encourages boids to travel together.

![A group of boids with lines indicating their heading, the boid in question is about to adjust to this](alignment.png)

```go
alignSteering := Vector2D{}
alignTotal := 0

for _, other := range restOfFlock {
    d := boid.position.Distance(other.position)
    if boid != other {
        if d < alignPerception {
            alignTotal++
            alignSteering.Add(other.velocity)
        }
    }
}

if alignTotal > 0 {
    alignSteering.Divide(float64(alignTotal))
    alignSteering.SetMagnitude(maxSpeed)
    alignSteering.Subtract(boid.velocity)
    alignSteering.Limit(maxForce)
}

boid.acceleration.Add(alignSteering)
```

### Cohesion

> Steer to move toward the average position of local flockmates

To be a _cohesive_ boid, one must head towards the center of the clump. This may seem at odds with _separation_ but these two rules work together to let boids arrange themselves equally within a space. Although, the reality of this _arranging_ can look quite chaotic!

![Our boid has a red arrow pointing to a green point in space. There are lines from the other boids that lead to this average center.](cohesion.png)

```go
cohesionSteering := Vector2D{}
cohesionTotal := 0

for _, other := range restOfFlock {
    d := boid.position.Distance(other.position)
    if boid != other {
        if d < cohesionPerception {
            cohesionTotal++
            cohesionSteering.Add(other.position)
        }
    }
}

if cohesionTotal > 0 {
    cohesionSteering.Divide(float64(cohesionTotal))
    cohesionSteering.Subtract(boid.position)
    cohesionSteering.SetMagnitude(maxSpeed)
    cohesionSteering.Subtract(boid.velocity)
    cohesionSteering.SetMagnitude(maxForce * 0.9)
}

boid.acceleration.Add(cohesionSteering)
```

Each rule should be considered equally, so after summing the rules together, we divide by three to find the average.

```go
boid.acceleration.Divide(3)
```

## Building And Deploying

It's time to get this simulation in front of users! Lucky for us, Ebiten supports many platforms. (The Nintendo Switch is even on the [roadmap](https://ebiten.org/blog/roadmap2020.html#Ebiten_2.1_-_March,_2021).)

The current release (v1.11.4), supports:

- Windows, macOS, Linux, FreeBSD
- Android, iOS
- Web browsers (Chrome, Firefox, Safari and Edge)
  - via WebAssembly or GopherJS

We will be compiling to [WebAssembly](https://developer.mozilla.org/en-US/docs/WebAssembly). With WebAssembly as a compilation target, we can deliver our Go program to anyone running a modern browser. I'm currently hosting it on [GitHub Pages](https://healeycodes.github.io/boids/).

What are the benefits of WebAssembly? In brief:

- Fast, efficient, and portable
- Executes at 'near-native' speed
- Runs in a secure sandbox

To get our Boid program ready for users, we build the `.wasm` file.

```bash
GOOS=js GOARCH=wasm go build -o dist/boids.wasm github.com/healeycodes/boids
```

We set the build's target operating system to `js` as well as setting the architecture to `wasm`. If you're setting this up from scratch, you'll need a copy of `wasm_exec.js` from the Go source — you can copy it locally from your machine or get it from [golang/go](https://github.com/golang/go/blob/master/misc/wasm/wasm_exec.js).

To display the simulation in a browser, we use [WebAssembly.instantiate](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WebAssembly/instantiate). Here's what our index page looks like:

```html
<main>
    <p>Loading..</p>
</main>

<script src="dist/wasm_exec.js"></script>
<script>
  (function () {
    const go = new Go();
    // WebAssembly.instantiateStreaming is more efficient
    // but doesn't work on Safari
    fetch('dist/boids.wasm')
      .then(response => response.arrayBuffer())
      .then(bytes => WebAssembly.instantiate(bytes, go.importObject))
      .then(result => {
        document
          .querySelector('main')
          .remove();
        go.run(result.instance);
      });
  })();
</script>
```

The full source is available on [the repository](https://github.com/healeycodes/boids).

## Where to go from here?

We looked at how to create an Ebiten game, the emergent behavior of the Boid program, and delivering a Go program via WebAssembly. But there's still more to explore!

Expand this solution with:

- Field of vision support (boids shouldn't look behind 👀)
- QuadTree optimization
- Different `maxSpeed`/`maxForce` for each boid
- Graphical interface for live-editing of values

Or use a different language:

- JavaScript/p5.js with [Daniel Shiffman](https://www.youtube.com/watch?v=mhjuuHl6qHM&vl=en)
- C#/Unity with [Sebastian Lague](https://www.youtube.com/watch?v=bqtqltqcQhw) in 3D

I'd love to see what you come up with!

<hr>

<small>Reynolds, C. W. (1987) Flocks, Herds, and Schools: A Distributed Behavioral Model, in Computer Graphics, 21(4) (SIGGRAPH '87 Conference Proceedings) pages 25-34.</small>

<small>Nystrom, R. (2020) Game Programming Patterns/Sequencing Patterns</small>


The famous flocking simulation built with the Ebiten game library.

Boids in WebAssembly Using Go


In JavaScript, you can create random numbers using `Math.random()`. But what if we wanted to create our own random values in the browser without this function?

The [ECMAScript Language Specification](https://tc39.es/ecma262/#sec-math.random) defines the requirements of `Math.random()`:

> Returns a Number value with positive sign, greater than or equal to 0 but less than 1, chosen randomly or pseudo randomly with approximately uniform distribution over that range, using an implementation-dependent algorithm or strategy. This function takes no arguments.

> Each Math.random function created for distinct realms must produce a distinct sequence of values from successive calls.

## Number Generation

Here's an example of a number generator. It uses a [closure](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Closures) to maintain internal state and creates a sequence of numbers based off an initial seed value. Here the seed is fixed and is always initialized to `0`.

```javascript
Math.random = (function () {
  let seed = 0
  return function () {
    seed += 1
    return seed
  }
})()

// We can iterate through the sequence
Math.random() // 1
Math.random() // 2
Math.random() // 3
```

A pseudorandom number generator (PRNG) works in a similar manner. A PRNG maintains an internal state and applies math to that state every time a new random number is requested. The seed can be manual or automatic. In the [Go programming language](https://golang.org/pkg/math/rand/#New), you must seed `math/rand` yourself. In the browser, `Math.random` requests random data under the hood from the operating system (OS) to use as a seed.

PRNGs are deterministic. The same seed will always produce the same sequence of numbers. Often, a deterministic outcome is preferred. For example, to generate the same random events on all clients without them having to talk over a network. Or for reproducible performance benchmarks.

A hash function can be used to create a PRNG. In [spinning-balls](https://github.com/v8/v8/blob/4b9b23521e6fd42373ebbcb20ebe03bf445494f9/benchmarks/spinning-balls/v.js), one of Chrome's benchmarks, we can see an example of this:

```javascript
// v8/benchmarks/spinning-balls/v.js

// To make the benchmark results predictable, we replace Math.random
// with a 100% deterministic alternative.
Math.random = (function () {
  var seed = 49734321
  return function () {
    // Robert Jenkins' 32 bit integer hash function.
    seed = seed & 0xffffffff
    seed = (seed + 0x7ed55d16 + (seed << 12)) & 0xffffffff
    seed = (seed ^ 0xc761c23c ^ (seed >>> 19)) & 0xffffffff
    seed = (seed + 0x165667b1 + (seed << 5)) & 0xffffffff
    seed = ((seed + 0xd3a2646c) ^ (seed << 9)) & 0xffffffff
    seed = (seed + 0xfd7046c5 + (seed << 3)) & 0xffffffff
    seed = (seed ^ 0xb55a4f09 ^ (seed >>> 16)) & 0xffffffff
    return (seed & 0xfffffff) / 0x10000000
  }
})()
```

Like our number generator, it alters its internal state while calculating the next random number. This state-change allows the next call to produce a different number.

## More on Pseudorandom Number Generators

One of the oldest and most well known types of PRNG is the [linear congruential generator](https://en.wikipedia.org/wiki/Linear_congruential_generator) (LCG). Which, despite its somewhat scary name, does not require many lines of code.

@bryc provides an example and [a warning](https://github.com/bryc/code/blob/master/jshash/PRNGs.md#lcg-lehmer-rng):

> Commonly called a Linear congruential generator (LCG), but in this case, more correctly called a Multiplicative congruential generator (MCG) or Lehmer RNG. It has a state and period of 2^31-1. It's blazingly fast in JavaScript (likely the fastest), but its quality is quite poor.

```javascript
function LCG(a) {
  return function () {
    a = Math.imul(48271, a) | 0 % 2147483647
    return (a & 2147483647) / 2147483648
  }
}
```

(This is the first time I've come across `Math.imul()` — which provides [C-like 32-bit multiplication](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/imul#Description) of the two parameters.)

What does @bryc's comment, "its quality is quite poor" mean in this context? Well, given certain even seeds, this algorithm has a pattern when the final step (the division) is removed.

```javascript
// https://gist.github.com/blixt/f17b47c62508be59987b#gistcomment-2792771

// @bryc:
// "Looking at the output without the division, and in hexadecimal, the
// first bits are always the same. This shows a clear pattern in the
// first 8 bits of the output: 1000 000, and it happens each time,
// infinitely. This is mostly caused by using an even seed."
const LCG = (s) => (_) => (s = Math.imul(48271, s) >>> 0)
const nxt = LCG(3816034944)
for (let i = 0; i < 9; i++) {
  console.log(nxt().toString(16))
}

/* Outputs:
4b6c5580 <-- notice the last two digits
b04dc280 <--
9645a580
16717280
d974f580
5c9f2280
9a3a4580
f196d280
b5d59580 */
```

There are [many](https://en.wikipedia.org/wiki/Diehard_tests) ways to test the quality of randomness. Some of the methodology and results of these tests can be understood by a layperson. One of the [Diehard battery of tests](https://en.wikipedia.org/wiki/Diehard_tests) plays 200000 games of craps and looks at the distribution of wins and the number of throws each game.

There's also a test for LCGs called the [spectral test](https://en.wikipedia.org/wiki/Spectral_test) which plots the sequence in two or more dimensions. In the example below, we can see the [hyperplanes](https://en.wikipedia.org/wiki/Hyperplane) that the spectral test measures for.

![Hyperplanes of an LCG in three dimensions](lcg-3d.gif)

A PRNG eventually repeats its sequence. In this context, the _period_ is the length of steps until the cycle repeats. Simpler PRNGs such as [Mulberry32](https://github.com/bryc/code/blob/master/jshash/PRNGs.md#mulberry32) have a period as low as ~4 billion whereas the [Mersenne Twister](https://stackoverflow.com/a/38838863) has a period of `2^19,937 - 1`. In 2015, the V8 team [said](https://v8.dev/blog/math-random) that their implementation of `Math.random()` uses an algorithm called [xorshift128+](http://vigna.di.unimi.it/ftp/papers/xorshiftplus.pdf) which has a period of `2^128 - 1`. Its introduction can been seen in [this diff](https://github.com/v8/v8/blob/085fed0fb5c3b0136827b5d7c190b4bd1c23a23e/src/base/utils/random-number-generator.h#L102).

If a PRNG eventually repeats itself, you might wonder why we call it repeatedly. Why not use the first number and then reset the internal state with a new seed? The problem with this is that the seed needs to originate from somewhere. If we continue to ask the OS for more random data there is a chance that the call may block (as the OS waits for more randomness to be generated) and our program will stall.

## Entropy Required

So you've settled on a PRNG and replaced `window.Math.random`. You've shipped it to your users and, at first, everyone seems to be happy.

But wait! You forgot about the seed. And now your users are complaining about the sequence of random numbers they get. It's the same every time their customers' page loads. All of their software is predictable. As a result, the web games they built are easy to beat.

Huzaifa Sidhpurwala [reminds us](https://www.redhat.com/en/blog/understanding-random-number-generators-and-their-limitations-linux):

> Entropy is the measurement of uncertainty or disorder in a system. Good entropy comes from the surrounding environment which is unpredictable and chaotic.

When required, the generation of securely random numbers in the browser is performed by `Crypto.getRandomValues()` from the [Web Cryptography API](https://www.w3.org/TR/WebCryptoAPI/#Crypto-method-getRandomValues). Which is seeded by "a platform-specific random number function, the Unix `/dev/urandom` device, or other source of random or pseudorandom data."

The Linux [source](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/char/random.c) suggests where this pseudorandom data can come from:

> Sources of randomness from the environment include inter-keyboard timings, inter-interrupt timings from some interrupts, and other events which are both (a) non-deterministic and (b) hard for an outside observer to measure.

There are also hardware devices that use [quantum mechanical physical randomness](https://en.wikipedia.org/wiki/Hardware_random_number_generator#Quantum_random_properties).

You can find many [prominent examples](https://en.wikipedia.org/wiki/Random_number_generator_attack#Prominent_examples) of random number generator attacks which occurred because the wrong type (or not enough) entropy was used. Cloudflare [famously](https://www.cloudflare.com/learning/ssl/lava-lamp-encryption/) uses lava lamps as an entropy source. Since we are not attempting to create a secure algorithm, predictable sources of entropy like time are fine.

We can use `Date.now()` our seed state. This means that we will get a different random sequence for every millisecond. We could also use `performance.now()` which returns the length of time since the [time origin](https://developer.mozilla.org/en-US/docs/Web/API/DOMHighResTimeStamp#The_time_origin).

Other possible ways of getting entropy in the browser:

- `crypto.getRandomValues`, `crypto` key generation, or similar (feels like cheating)
- Mouse/touch events, [ambient light events](https://developer.mozilla.org/en-US/docs/Web/API/Ambient_Light_Events), mic/webcam noise (hard to use on page load)
- Geolocation API, Bluetooth API, or similar (need permission, doesn't work on page load)
- WebGL/video performance shenanigans
- Most APIs [listed here](https://developer.mozilla.org/en-US/docs/Web/API)

Here's our slower (because it's not native code) and unstable (because I haven't tested it) replacement for `Math.random()`. Also note that PRNGs have requirements for the seed state (e.g. prime numbers, 128-bit). Our algorithm doesn't comply with the [seed recommendations](http://vigna.di.unimi.it/ftp/papers/ScrambledLinear.pdf) for the Xoshiro family.

```javascript
// https://github.com/bryc/code/blob/master/jshash/PRNGs.md
// xoshiro128+ (128-bit state generator in 32-bit)
Math.random = (function xoshiro128p() {
  // Using the same value for each seed is _screamingly_ wrong
  // but this is 'good enough' for a toy function.
  let a = Date.now(),
    b = Date.now(),
    c = Date.now(),
    d = Date.now()
  return function () {
    let t = b << 9,
      r = a + d
    c = c ^ a
    d = d ^ b
    b = b ^ c
    a = a ^ d
    c = c ^ t
    d = (d << 11) | (d >>> 21)
    return (r >>> 0) / 4294967296
  }
})()

Math.random() // 0.5351827056147158
Math.random() // 0.2675913528073579
```

## So, Mission Accomplished?

Sadly it's impossible to create a fully ECMAScript compliant replacement for `Math.random()` since the specification requires "distinct realms [to] produce a distinct sequence of values from successive calls." A _realm_ roughly means a different global environment (e.g. a different window, or a different [WebWorker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API)). Our version cannot reach outside its realm thus cannot make this guarantee.

However, there have been proposals for a [Realms API](https://github.com/tc39/proposal-realms). It's not inconceivable that such an API would provide access to something like an incrementing realm id. This would give our algorithm the loophole it needs — access to Realm-unique entropy!

<small>Thanks to [JN~commonswiki](https://commons.wikimedia.org/wiki/File:Lcg_3d.gif) for the 3D GIF of the spectral test.</small>


Building a replacement for JavaScript's random number generator.

Creating Randomness Without Math.random


A graph is a structure that represents the connections between objects. They can be used for social network analysis, working out someone's [Erdős number](https://en.wikipedia.org/wiki/Erd%C5%91s_number), finding the shortest path through a maze, and many other [applications](https://en.wikipedia.org/wiki/Graph_theory#Applications).

We'll be calculating the path between two countries with the least border crossings. The method we'll use is [Breadth-First Search](https://en.wikipedia.org/wiki/Breadth-first_search) (BFS).

## What's in a Graph?

Graphs are made up of vertices and edges. Below is the representation of France and its land border crossings.

![A graph of France and its land borders. Country codes are uses for the vertices.](france-connections.png)

An example of a vertex is `FR` or `DE` — they are also known as points or nodes. Here they represent countries.

An example of an edge is `{FR, IT}` or `{FR, BE}` — they are also known as lines, links, or connections. Here they represent the land border crossing between two countries.

Let's build a slightly more complicated graph. If our goal is to find the shortest path between two countries then we need a structure containing all of the countries of the world. We'll actually need multiple graphs because some sets of countries aren't connected to other sets of countries (and islands are just lone vertices).

All the code for this post can be found on [healeycodes/country-borders](https://github.com/healeycodes/country-borders).

We can get all the data we need from `borders.csv`. This file is structured as `country_code, country_name, country_border_code, country_border_name`.

As a side-quest, we'll also store a relationship between country codes and their full names to make our output more user-friendly later on.

```python
import csv

country_border_graph = {}
country_code_lookup = {}

with open('borders.csv') as csvfile:
    reader = csv.reader(csvfile, delimiter=',')
    next(reader, None)  # skip the headers
    for row in reader:
        country_code_lookup[row[0]] = row[1]
        country_a = row[0]
        country_b = row[2]

        if country_a in country_border_graph:
            country_border_graph[country_a].append(country_b)
        else:
            country_border_graph[country_a] = [country_b]
```

A graph is a mathematical structure and there is no singular way to represent them in code. Here, we have chosen a dictionary that maps a country to a list of its neighbors.

If we isolate France inside `country_border_graph`, it looks like this:

```python
{
    'FR': ['AD', 'BE', 'DE', 'IT', 'LU', 'MC', 'ES', 'CH']
}
```

## Displaying Graphs

So how do we print this to the screen? First we alter the structure of our data then we plug it into some libraries.

In order to build a [DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) for the `pandas` library, we need to create two lists that describe all of our edges. We do that by iterating through the structure and adding a vertex to two lists (`from_verts` and `to_verts`) when we want to describe a connection.

```python
from_verts = []
to_verts = []
for country in country_border_graph:
    for i in range(0, len(country_border_graph[country])):
        if country_border_graph[country][0] == '':
            # skip islands
            continue
        from_verts.append(country)
        to_verts.append(country_border_graph[country][i])
```

For France, the lists would look like this:

```python
# from_verts
['FR', 'FR', 'FR', 'FR', 'FR', 'FR', 'FR', 'FR']

# to_verts
['AD', 'BE', 'DE', 'IT', 'LU', 'MC', 'ES', 'CH']
```

Using `pandas`, `networkx`, and `matplotlib` we can display our non-islands in a popout window.

```python
import pandas as pd
import networkx as nx
import matplotlib.pyplot as plt


def show(from_verts, to_verts):
    df = pd.DataFrame(
        {'from': from_verts, 'to': to_verts})
    G = nx.from_pandas_edgelist(df, 'from', 'to')

    nx.draw(G,
            with_labels=True,
            node_size=1500,
            node_color="skyblue",
            node_shape="o",
            linewidths=4,
            font_size=25,
            font_color="grey",
            font_weight="bold",
            width=4,
            edge_color="grey",
            dpi=1000)
    plt.show()
```

The result is below. (I briefly attempted to visually _unsquish_ the vertices from each other but didn't think it was worth the effort. If you want to display this data more aesthetically, start by checking out this [StackOverflow answer](https://stackoverflow.com/a/54876985).)

![A graph of all the non-islands of the world](squished-non-islands.png)

While this image, at first glance, looks similar to a world map the positions of the countries are not tied to their position on Earth.

## Shortest paths

You can calculate the shortest path between some countries by hand. Take the path from Canada to Mexico as an example. At the lowest point of the largest set in the above image, we can see that `CA` is connected to `US` which is connected to `MX`. It's clear there is only one route between Canada and Mexico and therefore we have found the shortest path.

What if there are multiple paths? Let's take the path from France to China as an example.

Breadth-First Search (BFS) works by first looking at all of the nearby vertices and then working further and further away from the start point. By using `FR` as a starting point, our algorithm will first check `['AD', 'BE', 'DE', 'IT', 'LU', 'MC', 'ES', 'CH']`. When it doesn't find China among those, it will then look at the neighbors of `AD` and then the neighbors of `BE` and so on.

In the diagram below, we can see the order in which BFS would visit each vertex of a tree.

![Nodes, numbered 1-to-12 in a tree](bfs-path.png)

The algorithm we use has been adapted from an [outdated essay](https://www.python.org/doc/essays/graphs/) on Python.org. Eryk Kopczyński corrected the efficiency of the essay's pathing function ten years after its publication date. In linear time, it computes and stores the direct paths from the start point to every other point and then returns the path from the start point to the endpoint.

His original code returns the path in a strange format. `FR` to `CN` is `[[[[['FR'], 'DE'], 'PL'], 'RU'], 'CN']`. We can unfurl this nested list with `flatten`.

```python
from collections import Iterable


def flatten(l):
    # flatten a nested list
    # https://stackoverflow.com/a/2158532

    for el in l:
        if isinstance(el, Iterable) and not isinstance(el, (str, bytes)):
            yield from flatten(el)
        else:
            yield el


def find_shortest_path(graph, start, end):
    # breadth first search (BFS)
    # adapted from https://www.python.org/doc/essays/graphs/

    dist = {start: [start]}
    q = [start]
    while len(q):
        at = q.pop(0)
        for country in graph[at]:
            if country not in dist:
                dist[country] = [dist[at], country]
                q.append(country)

    raw_path = dist.get(end)
    return list(flatten(dist.get(end)))
```

We can use this function on the original `country_border_graph` dictionary we built. With the `country_code_lookup` we made, we can print out user-friendly information.

```python
france = country_border_graph["FR"]
france_names = [country_code_lookup[code] for code in france]

print(f'France is connected to {len(france_names)} countries.')
print(f'They are {", ".join(france_names)}.')

path = find_shortest_path(country_border_graph, 'FR', 'CN')
path_names = [country_code_lookup[code] for code in path]

print(f'The shortest path from France to China is: {" -> ".join(path_names)}.')
```

This prints:

```text
France is connected to 8 countries.
They are Andorra, Belgium, Germany, Italy, Luxembourg, Monaco, Spain, Switzerland.
The shortest path from France to China is: France -> Germany -> Poland -> Russian Federation -> China.
```

To display this path, we can use the `show` function we defined earlier.

```python
path = find_shortest_path(country_border_graph, 'FR', 'CN')

from_verts = []
to_verts = []
for i in range(1, len(path)):
    # we're drawing a line
    # so we describe each edge 
    from_verts.append(path[i-1])
    to_verts.append(path[i])

graph.show(from_verts, to_verts)
```

![A line of nodes, FR-DE-PL-RU-CN](fr-to-cn.png)

We can now visually show the shortest path between any two connected countries.

(If we attempted to use this code to calculate the path between two non-connected countries, e.g. `GB` and `CN`, the code would throw an error. Let's blame geography for this instead of the code's author.)

<small>Thanks to [Alexander Drichel](https://commons.wikimedia.org/wiki/File:Breadth-first-tree.svg) for the Breadth-First Tree diagram.</small>


Calculating the path between two countries with the least border crossings.

A Practical Introduction to Graphs (Network Diagrams)


First some definitions.

> A **polyfill** will try to emulate certain APIs, so can use them as if they were already implemented.

> A **transpiler** on the other hand will transform your code and replace the respective code section by other code, which can already be executed.

(Thanks [Sirko](https://stackoverflow.com/a/31206361) for those!)

## Examples of polyfilling and transpiling

By polyfilling the [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) we make it usable in browsers where it isn't available by default. Another example would be making it available in its original functionality in Node.js.

By transpiling [Spread syntax](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax) (an ES6 feature) into ES5 compatible JavaScript, we end up with source code that is easier for us to write — and deployable code that works in older browsers!

Here's our ES6 code example.

```javascript
function speak() {
  console.log(...args);
}

speak(1, 2, 3);
```

Here is that same code transpiled into ES5 compatible code.

```javascript
"use strict";

function speak() {
  var _console;

  (_console = console).log.apply(_console, args);
}

speak(1, 2, 3);
```

(Technically, this is the [loose](https://babeljs.io/docs/en/babel-plugin-transform-spread#loose) transpilation because otherwise the above snippet would be eleven times longer.)

## Using `fetch` in old browsers

We love the Fetch API because it means we can avoid using [XMLHttpRequest](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest). Instead of providing a callback, we can use lovely [Promises](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise).

Here's a `fetch` call that prints out the status code of a request.

```javascript
fetch('https://httpstat.us')
    .then(res => console.log(res.status))
```

If we tried running that in Internet Explorer 11 (or Edge 13, or Chrome 39, and so on) we would get an error.

`window.fetch` would likely evaluate to undefined. We might get an error that looks like `Uncaught TypeError: window.fetch is not a function`.

It's easy to check one-off functionality for a feature on _Can I use_ — here's `fetch` [https://caniuse.com/#search=fetch](https://caniuse.com/#search=fetch). It isn't viable to check every feature that your code might use, so that's why we use things like [@babel/preset-env](https://babeljs.io/docs/en/babel-preset-env).

> `@babel/preset-env` is a smart preset that allows you to use the latest JavaScript without needing to micromanage which syntax transforms (and optionally, browser polyfills) are needed by your target environment(s). This both makes your life easier and JavaScript bundles smaller!

An even cooler feature of Babel is that it allows us to specify which platforms we want to support.

Why not just support _all_ platforms? Because then the JavaScript bundle we send to our users would get larger and larger every year and website performance would grind to a halt.

Babel's [Browserslist Integration](https://babeljs.io/docs/en/babel-preset-env#browserslist-integration) lets us forget about version numbers and instead use handy shortcuts. Let's say we wanted to support 99.75% of browsers and no dead browsers. We can add `"browserslist": "> 0.25%, not dead"` to our `package.json` file.

However, you might be reading this because you found out that `@babel/preset-env` doesn't include a `fetch` polyfill. This is an open issue on GitHub ([#9160](https://github.com/babel/babel/issues/9160)). It looks like it's going to stay this way.

> Babel only adds polyfill for ECMAScript methods (defined at https://tc39.github.io/ecma262/). Since fetch is defined by a web specification (https://fetch.spec.whatwg.org/), you need to add the polyfill yourself.

That's okay because we can use [github/fetch](https://github.com/github/fetch#usage) to polyfill it for us.

Either by replacing all instances of `fetch`.

```javascript
import 'whatwg-fetch'

window.fetch(...)
```

Or on a case-by-case basis.

```javascript
import { fetch as fetchPolyfill } from 'whatwg-fetch'

window.fetch(...)   // use native browser version
fetchPolyfill(...)  // use polyfill implementation
```

## Using `fetch` in Node.js

The Fetch API is common and people are fluent with it. It's productive if they can use it in all the JavaScript they write. Many people think it's available in Node.js by default. It's not but _there's a package for that_ (™).

[node-fetch/node-fetch](https://github.com/node-fetch/node-fetch) let's us use the API we're fluent with to make HTTP calls on the back-end. Underneath, it uses native Node.js functionality.

```javascript
// CommonJS
const fetch = require('node-fetch');

// ES Module
import fetch from 'node-fetch';
```

If you're looking for an isomorphic solution (this means using the same code in the browser and in Node.js) then you'll want Jason Miller's [isomorphic-unfetch](https://www.npmjs.com/package/isomorphic-unfetch) (but not for React Native, see [#125](https://github.com/matthew-andrews/isomorphic-fetch/issues/125)) or Leonardo Quixada's [cross-fetch](https://github.com/lquixada/cross-fetch).

These solutions will figure out which environment you're in and choose the correct polyfill.


A quick look at polyfilling, along with with solutions for both environments.

Polyfilling the Fetch API for Old Browsers and Node.js


Engineering tickets often end up meaning a lot. For the product but also emotionally for the individual. It's the engineer's job to be involved in the whole process from before the ticket even exists. Engineers refine the ticket, estimate the ticket, and shuffle it through columns on a sprint board until customers receive a feature.

The way products are built varies at every company. A team I sit metres away from at work do things so differently that there would be a learning curve if I joined them. But the concepts remain the same.

The fun part about writing this will be hearing about all the different ways y'all make software.

## Feature or fix?

A ticket can be a feature that needs to be completed, a bug that requires solving, or a chore that needs doing (e.g. code clean-up, documentation creation). A _spike_ is a type of ticket that means the situation is unclear. Perhaps it's because there are unknowns about the technical implementation. Perhaps what's required is checking where a tricky bug is originating from. A spike can sometimes mean: go and figure out what tickets need to be created to solve this problem and then create those tickets (the hydra of tickets).

In the [Agile](https://en.wikipedia.org/wiki/Agile_software_development) methodology, tickets shouldn't be time boxed but spikes are one of the exceptions. Without a time-limit, when does the engineering stop if unknowns are involved? It's far better to write down your findings and get your team's input. Sometimes spikes are easier than they seem and the development work can be completed within the time-limit and everyone is happy.

## Where do tickets come from?

Here are some ways that tickets can come into existence from nothing.

- A customer complains about a bug and a support engineer documents it
- A project manager or team lead plans a piece of the project
- Some technical debt is created while working on another ticket
- A developer realizes that their current ticket requires additional parts
- Another department (e.g. data scientists) require some work to be done

## Where do tickets live?

Most tickets exist in a sad place called the backlog. It's sad because there are always tickets in there that you want to complete. Things that genuinely improve the product (or your ease of working on the product).

Backlogs usually increase in size until they become unmanageable. At which point they can be filtered and trimmed by certain criteria. _Won't do_ is a common reason to close a ticket — as in: we realized that there is an issue here but we don't have any plans to actually fix the problem. I've seen date cut-off limits where after a year of a ticket existing if it hasn't been completed then it's put out of its misery and archived. A well trimmed backlog is the sign of a well managed engineering team.

Tickets are always archived, never deleted. If there once was a bug that then becomes a bigger bug in the future, you'll wish there was a trail to follow when it falls on your lap.

Tickets are brought out of the backlog into a refinement section. These tickets are then discussed in a backlog refining meeting. Questions are raised and answered before the ticket can continue on its journey. Sometimes the questions can't be answered and a comment is added. A comment might tag someone with a TODO like _@alice to gather more requirements from (other team who requested the feature)_. This ticket will be looked at again in the next meeting.

In a backlog refinement meeting, you might ask:

- Is this possible?
- Is this within our area of responsibility (is this another team's job)?
- Roughly, what steps are involved in solving this?
- How long will this take? 

A good rule for completing projects is: don't do something if you don't actually need to do it. This sounds silly until you realise how often it is broken.

## Estimation

[Planning poker](https://en.wikipedia.org/wiki/Planning_poker) is a ritual that helps get the average estimation for a ticket from a group of engineers. It works best when everyone reveals their estimations at the same time as this removes biases. Planning poker cards are often used because everyone can flip them over at the same time.

The unit of time is 'points' which can seem abstract at first. It's common to hear things like _oh that's just a one-pointer_ or _hm, 13 points is a little high, can we break that up into smaller tickets?_.

Planning poker cards have a modified Fibonacci sequence on them. 1, 2, 3, 5, 8, 13, 20, 40, and 100. Why? The International Scrum Institute has a [good summary](https://www.scrum-institute.org/Effort_Estimations_Planning_Poker.php):

> The reason for using the Fibonacci sequence is to reflect the uncertainty in estimating larger items. A high estimate usually means that the story is not well understood in detail or should be broken down into multiple smaller stories. Smaller stories can be estimated in greater detail.

When tickets are pointed they can be placed into _sprints_ in a sprint planning meeting ([What is a Sprint in Scrum?](https://www.scrum.org/resources/what-is-a-sprint-in-scrum)). A sprint is a time frame where a goal is worked towards by completing tickets. Traditionally, a working product is demoed to stakeholders at the end of a sprint so that it can be worked on iteratively taking feedback into account.

One sprint follows another. At the end of a sprint there are reviews and retrospectives to ensure that the ways of working are productive and pleasant for everyone involved.

## Interacting with tickets

When I pick up a ticket, I move it to a column called _In Progress_. This automatically assigns me to the ticket and puts my avatar in the corner of the card. This stops people doing duplicate work and helps product managers and quality assurance engineers see who is working on what.

Sprint boards usually have a column for tickets that are in review. GitHub notifications can get lost in a sea of other emails so being able to see what work I can review for my teammates really helps me.

The act of dragging these small cards in the sprint board's UI feels significant. Moving the ticket through the columns, left-to-right, indicates and broadcasts the ticket's progress. The final column, which means the work is done (meaning deployed to production or handed off to another team) is a happy place.

Good tickets are written so they can be scanned efficiently. They use [Markdown](https://en.wikipedia.org/wiki/Markdown) to achieve this by marking sections and highlights. Bullet pointed lists are used in abundance. Good tickets also probably have:

- A user story — the requirements from the POV of a user
- Technical requirements if relevant
- Links to other relevant tickets or previous work
- For front-end tickets, links to designs
- Tags! Tags give the team visibility into blockers and future problems

I like to keep tickets updated so that stakeholders don't need to chase me for an update. I do this by leaving comments about my progress.

I've normally been lucky at companies I've worked at in that code reviews are integrated to the issue tracking system. So 'attaching' a pull-request to a ticket is effortless and is achieved with a single click or comment.

## Document the world

I like documentation. I like the flair that people add. Emojis, GIFs (practical or memes), and embedded videos. I spend a lot of time thinking about the best way to communicate ideas and problems — and I'm very happy that software engineering has allowed me to work on writing for humans. Lean writing that conveys a point. The least words required.

Some days I will be adding comments to technical designs, or explaining a pull-request that grew in size and requires a lot of context to understand. Other days I'll be breaking down a technology for the wider team by assuming the reader has minimal coding knowledge.

If you go the extra mile to throw up an internal documentation page after solving a problem, I appreciate you. I see you editing the onboarding documentation, keeping it tight, the instructions clear and friendly, and I thank you.


Engineering tickets often end up meaning a lot. For the product but also emotionally for the individual.

The Flow of Knowledge in a Team (or Why I Dig Tickets)


I wanted to accept user comments on a Gatsby website and store them on GitHub. As in, I wanted the comments to go straight into a file called `comments.json` in my repository. So I could use something as simple as

```javascript
import comments from "../comments.json"
```

in my site's code. Without any databases. No third-party plugins making tens of networks requests.

[Netlify serverless functions](https://docs.netlify.com/functions/overview/) allowed me to use GitHub's API to make this repository change with the data from a submitted comment. It also hid my secret API credentials.

I built a prototype — [healeycodes/gatsby-serverless-comments](https://github.com/healeycodes/gatsby-serverless-comments) — that uses this flow:

1. 👩 User enters a comment and clicks submit.
2. ⚙️ A serverless function receives the data and hits GitHub's API.
3. 🔧 It reads the existing `comments.json` , appends the new comment, and saves.
4. 🚧 A new commit triggers a Netlify build.
5. ✅ The new version of the website is deployed!

The new comment is visible to users ~30 seconds ⏰ after the first click.

## The serverless function

Let's pick through the serverless function that receives the user's comment. It will make use of some constants that can be set through Netlify's website on _settings_ → _deploys_.

![GITHUB_PAT_TOKEN, GITHUB_REPO, GITHUB_USER](env-vars.png)

The function is written with _Node.js_ and exports a `handler` function, which is explained in [Netlify's documentation](https://docs.netlify.com/functions/build-with-javascript/#format).

```javascript
// comment.js

const fetch = require("node-fetch")

const auth = process.env.GITHUB_PAT_TOKEN
const repo = process.env.GITHUB_REPO
const user = process.env.GITHUB_USER
const api =
  "https://api.github.com/repos/" +
  user +
  "/" +
  repo +
  "/contents/src/comments.json"

exports.handler = async function(event, context, callback) {
  // Use the Contents API from GitHub
  // https://developer.github.com/v3/repos/contents/#get-contents
  const existingFile = JSON.parse(
    await fetch(api, {
      headers: {
        // Pass some kind of authorization
        // I'm using a personal access token
        Authorization:
          "Basic " + Buffer.from(user + ":" + auth)
            .toString("base64"),
      },
    }).then(res => res.text())
  )

  // The file's content is stored in base64 encoding
  // Decode that into utf-8 and then parse into an object
  let comments = JSON.parse(
    Buffer.from(existingFile.content, "base64").toString("utf-8")
  )

  // This is the user submitted comment
  // Perhaps we would do some validation here
  const newComment = JSON.parse(event.body)

  // Update the comments
  comments.push({
    author: newComment.author,
    email: newComment.email,
    message: newComment.message,
    date: Date.now(),
  })

  // Use the Contents API to save the changes
  const res = await fetch(api, {
    method: "PUT",
    headers: {
      Authorization:
        "Basic " + Buffer.from(user + ":" + auth).toString("base64"),
    },
    body: JSON.stringify({
      message: "New comment on " + new Date().toDateString(),

      // Turn that object back into a string and encoded it
      content: Buffer(JSON.stringify(comments)).toString("base64"),

      // Required: the blob SHA of the existing file
      sha: existingFile.sha,
    }),
  }).then(res => res.text())

  callback(null, {
    statusCode: 204,
  })
}
```

## Potential downsides

What if someone spams comments on your website? Well, you'll hit your build time limits pretty fast.

There's also a small window (10-100s of milliseconds between API calls) where two people comment at the same time and the older comment will be overwritten.

The fix for both of these is to alter our serverless function to open a pull-request with the comment change. Comments are now delayed but we've protected ourselves from malicious behavior and we can also screen comments for appropriateness. We won't lose any data but might rarely need to handle merge conflicts.

## My Netlify review

Netlify are betting big on [Jamstack](https://jamstack.org/) applications. It's a bet I would make too.

Their developer experience (DX) is up there with the best right now. It's rare I read about a product _just working_ and then it ends up doing so! Recently, Netlify's snappy deployments let me rush out changes to fix live issues within minutes.

What does this mean for their future success? Well, Tiny.cloud [points out](https://www.tiny.cloud/blog/developer-experience/) that:

> DX is kind of a big deal. Developers can play a huge role in the uptake of your product, especially when you consider they are likely to provide guidance on what tools their organization should invest in, even though the final decision usually happens at the executive level. 

Netlify's developer tooling lets me create prototypes like the one you're reading about without having to mess with configuration. [My Gatsby website](https://healeycodes.com/) is hosted with their generous free tier and transferring and hosting it has been hiccup-free.

I recommend them.


Use Netlify serverless functions to let users add comments to a static website.

Adding Comments to Gatsby with Netlify Serverless Functions + GitHub


I realized I was asking Google Assistant the same questions over and over again. Like _What's the current weather?_ or _What's on my calendar for today?_ So I set out to build a small dashboard for my Raspberry Pi that I could check instead.

I bought a _Waveshare 2.7inch e-Paper HAT (B)_ for ~$20. This version comes with a printed circuit board that slots right onto the general-purpose input/output (GPIO) pins of a Raspberry Pi. You can also use the included wires to avoid blocking the unused GPIO pins.

![e-Paper PCB, manual, GPIO wires, screws](product.jpg)

I followed the installation instructions from the [Waveshare wiki](https://www.waveshare.com/wiki/2.7inch_e-Paper_HAT_(B)) and fired up one of the hello world programs — there are [versions in C and Python](https://github.com/waveshare/e-Paper). The steps to get this going, as well as the example programs, are quite accessible compared to other Raspberry Pi hardware accessories.

![Hello world program, sample text and images on the screen](hello-world.jpg)

I started picking through the source code of the Python examples to understand how [epd-library](https://pypi.org/project/epd-library/) works. The version _B_ of this screen is tri-colored, it can display white, black, and red, and the resolution is 264x176. The examples start by clearing the screen (filling it with white) and then create two layers which are [Image](https://pillow.readthedocs.io/en/stable/reference/Image.html) objects from the [Pillow library](https://pillow.readthedocs.io/en/stable/). One of the layers is the black content and the other layer is the red content. These are then passed to the render function `EPD::display`.

I've extracted and commented the key parts of `examples/epd_2in7b_test.py `.

```python
# initialize the EPD class
epd = epd2in7b.EPD()
epd.init()

# start with a clear (white) background
epd.Clear()

# define a simple font
font24 = ImageFont.truetype(os.path.join(picdir, 'Font.ttc'), 24)

# create empty image objects/layers
HBlackimage = Image.new('1', (epd.height, epd.width), 255)
HRedimage = Image.new('1', (epd.height, epd.width), 255)

# render the black section
black_draw = ImageDraw.Draw(HBlackImage)
black_draw.text((1, 1), top_text, font=FONT_LARGER, fill=0)

# calculate the depth of the black section
top_depth = black_draw.textsize(top_text, font=FONT_LARGER)[1] + 5

# render the red section under the black section
red_draw = ImageDraw.Draw(HRedImage)
red_draw.text((1, top_depth), bottom_text, font=FONT_SMALLER, fill=0)

epd.display(epd.getbuffer(HBlackImage), epd.getbuffer(HRedImage))
```

For the prototype, I'll show the current weather and some news headlines from the BBC. [Open Weather](https://openweathermap.org/) and [News API](https://newsapi.org/) both have generous free-tiers. For each of these features, I'll write functions that return text which I can create layers from.

I'll be using [formatted strings](https://docs.python.org/3.5/library/stdtypes.html#str.format) from Python 3.5 (as opposed to the superior [f-strings](https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-pep498)) because that's what [Raspian](https://www.raspberrypi.org/downloads/raspbian/) (the default Raspberry Pi operating system) comes with.

This function talks to Open Weather using [PyOWM](https://pypi.org/project/pyowm/), a wrapper library.

```python
def get_weather():
    '''
    Get the formatted weather description.
    E.g. `Richmond, GB: Clear sky`
    '''
    owm = pyowm.OWM(OPEN_WEATHER_KEY)
    observation = owm.weather_at_place(LOCATION)
    w = observation.get_weather()
    detailed = w.get_detailed_status().capitalize()

    temp_data = w.get_temperature('celsius')
    current_temp = str(temp_data["temp"])
    min_temp = str(temp_data["temp_min"])
    max_temp = str(temp_data["temp_max"])
    text = '{}: {}\n{}°C ({}°C - {}°C)'

    return text.format(LOCATION, detailed, current_temp, min_temp, max_temp) 
```

News API sadly doesn't have an up-to-date library available so I use [requests](https://pypi.org/project/requests/) to make an HTTP call and parse the data. I can display about three headlines (including an extra wrapped line for each). The text needs to be manually [wrapped](https://en.wikipedia.org/wiki/Line_wrap_and_word_wrap) to stop it going off-screen.

```python
def get_news(limit=3):
    '''
    Get some news headlines from the BBC formatted in a list.
    E.g. `- Acclaimed Swedish author Per Olov Enquist dies
    - PM's return to work 'a boost for the country'
    - 'Myth that Sweden has not taken serious steps'`
    '''
    url = 'https://newsapi.org/v2/top-headlines?sources=bbc-news&apiKey={}'.format(NEWS_API_KEY)
    resp = requests.get(url=url)
    data = resp.json()
    articles = [article['title'] for article in data['articles'][:limit]]
    
    wrapper = textwrap.TextWrapper(width=33)
    text = ''
    for headline in articles:
        text += '- {}\n'.format(wrapper.fill(text=headline))
    return '{}\n'.format(text)
```

With these two sections of text, I create the image layers. I want the weather to be on the black-colored layer with a slightly larger font. The news will be underneath on the red-colored layer. Here's a function that takes an instance of the EPD class as well as two sections of text.

```python
def show_text(epd, top_text='', bottom_text=''):
    '''
    Given an EPD instance, and sections of text, render the text.
    '''
    HBlackImage = Image.new('1', (epd2in7b.EPD_HEIGHT, epd2in7b.EPD_WIDTH), 255)
    HRedImage = Image.new('1', (epd2in7b.EPD_HEIGHT, epd2in7b.EPD_WIDTH), 255)

    # render the black section
    black_draw = ImageDraw.Draw(HBlackImage)
    black_draw.text((1, 1), top_text, font=FONT_LARGER, fill=0)

    # calculate the depth of the black section
    top_depth = black_draw.textsize(top_text, font=FONT_LARGER)[1] + 5

    # render the red section under the black section
    red_draw = ImageDraw.Draw(HRedImage)
    red_draw.text((1, top_depth), bottom_text, font=FONT_SMALLER, fill=0)

    epd.display(epd.getbuffer(HBlackImage), epd.getbuffer(HRedImage))
```

The two Image objects contain simple data and don't 'know' what color they are. They look like this when rendered to bitmap format.

![A weather status](top-text.png)

Notice the space that the second layer leaves for the first layer so that the text doesn't overlap.

![News headlines](bottom-text.png)

The main function of this program makes sure that we only display the dashboard during hours when someone will be awake. I want to avoid [ghosting](https://en.wikipedia.org/wiki/Screen_burn-in) which the documentation warns about. The EPD class that gets passed to the `show_text` function is initialized here.

```python
def main():
    try:
        epd = epd2in7b.EPD()
        logging.info("Init and Clear")
        epd.init()
        epd.Clear()


        show_text(epd, top_text=get_weather(), bottom_text=get_news())

        # leave the dashboard up for half an hour
        time.sleep(60*30)

        logging.info("Clear...")
        epd.init()
        epd.Clear()
            
    except IOError as e:
        logging.info(e)
        sys.exit()
        
    except KeyboardInterrupt:    
        logging.info("ctrl + c:")
        epd2in7b.epdconfig.module_exit()
        sys.exit()

while True:
    t = datetime.datetime.now()

    # rest the screen outside this range
    if 7 <= t.hour <= 22:
        # update the dashboard
        main()
    else:
        time.sleep(30)
```

Let's see it in action!

![E-ink screen rendering both sections of text](prototype.jpg)

I'm not sure if I'll flesh this prototype out into an open source project so for now the full source code is in a [gist](https://gist.github.com/healeycodes/38c256c747eac46ac4ffc881e93bf095).

## Final thoughts

I didn't invest any time in working with the four buttons on the left of the screen (they're labelled KEY1, KEY2, etc. in the above picture) but I think it would be worthwhile to integrate them in a fleshed out dashboard — they connect as standard GPIO buttons.

I've learned that e-ink screens are worth the investment for me and I might look into getting a much larger version. The viewing angle is like that of a [Kindle](https://en.wikipedia.org/wiki/Amazon_Kindle) and although it's small it's beautiful to look at and has a more 'real' feel than a laptop or smartwatch screen.

I look forward to seeing this technology progress and the commercial applications that arise. I saw that Whole Foods Market were using them in-store back [in 2016](https://web.archive.org/web/20170107033856/http://www.pamplinmedia.com/lor/48-news/314796-193219-thoughtfully-simple). It would be nice to have less LEDs staring at me whenever I leave the house!

A prototype Raspberry Pi dashboard to save me talking to my smart assistant.

Hacking Together an E-ink Dashboard


Madeline and I moved our living room around so our desks could be up against the blinds in the lightest area. We have our full work setups here. We carried the gear home in a duffle bag and went back a week later for the [chairs](https://en.wikipedia.org/wiki/Aeron_chair) which, while being amongst the most ergonomic objects I've ever touched, rattled like heck as we wheeled them home.

![Our living room, a sofa, rug, long desk, two setups](living-room.jpg)

When we have meetings at the same time, one of us normally goes into another room. It depends on how much we're speaking. When we're both talking we tend to get louder and louder, without realizing it, until we're almost shouting.

This is where I listen to lofi hip hop streams and build software and record GIFs to make my pull requests all pretty. My MacBook Pro 2019 is paired with a small £12 bluetooth keyboard because my work keyboard won't fit comfortably on the kitchen table.

![Laptop, a small keyboard](laptop.png)

Last week, I was in the other room and Madeline pranked me by turning the brightness and volume up and down on my MacBook via this keyboard (which I had left behind). I was in a casual call with someone at the time and I was describing, while bewildered, all of the weird symptoms until I heard Madeline stifling laughter on the other side of the door.

I spend a lot of time on Slack, Google Hangouts, and Discord. On Slack, my team has a private team channel for work memes and things that relate to the [sprint](https://en.wikipedia.org/wiki/Scrum_Sprint). I find that when working remote, it's best to have conversations in channels rather than private messages because this shares knowledge and context among the team. Messages in channels are also searchable by the team (even though Slack's search feature leaves much to be desired). I found that it was worth learning a few Slack shortcuts to increase my productivity e.g. CMD+K to quickly navigate, UP ARROW to edit your last message.

We use Hangouts for our regularly scheduled meetings, all synced by Google Calendar. Hangouts has slightly worse quality than Zoom and seems to use up more battery. Discord has been great for pair programming. The stream feature is being improved almost every day. Discord is better for when I'm working closely with someone on a feature but we only need each other every few minutes. In between these moments we can go silent or just idly chat.

I've been quite into videos from a train driver's point of view. They're long and uneventful and beautiful. I leave these 12hr streams on my third screen. These videos are often from Norway, where there is the [slow television](https://en.wikipedia.org/wiki/Slow_television) movement.

The usual meeting schedule, and the cadence of our sprints, is the same as before we all went remote. Before, we each worked 1-2 days remotely each week so the transition hasn't been too bad. I have been trying to be positive and I've been looking forward to reviewing our remote practices for when we get back to the office. It's always been a priority for me to make sure that anyone remote is included fully in discussions and activities at the office.

One trick for connecting the remote person to office life is to let them lead the daily standup while they share their screen (usually viewing the sprint board). This ensures that the remote person understands everything that is going on and feels comfortable asking for clarifications. It ensures that everyone is conscious of the microphone setup at the office too (for when there is one shared microphone e.g. a standup meeting). Alternatively, everyone at the office can dial into the meeting separately to make it a level playing ground.

![Richmond Park](richmond-park.jpg)

At lunch Madeline and I usually go for runs in Richmond Park. We work hard to keep two meters+ away from people. It's been beautiful to watch the season change into Summer.

I have been drinking more coffee — cold brew and instant — especially in the mornings. We discovered the joy of barista-style oat milk (shaking the carton plenty before pouring) and it has ruined all other types of oat milk for us! It's apparently made from liquid oats (I looked it up but still don't know what it means) which keeps the milk from separating when the coffee is left still.

Deploying code has been scarier while remote. At the office there is always someone around to help with problems even though the way of grabbing them is the same (we message on Slack if someone's headphones are in). However, the only problem I've run into is GitHub being down, which can affect our CI/CD systems. I'm lucky that Madeline works at the same company as me. She's in a different team but we have been sneaking in some pairing sessions and helping each other out. I've learned more about the area that her team works in. Our teams sit a few meters away at the office but our processes are surprisingly different. It's been fun to learn what works for them and then to take it back to my team. Madeline and I have been speeding up each other's features by calling out questions across the kitchen table when we're stuck.

A junior co-worker told me that they are googling more and getting less stuck because they aren't embarrassed by the idea of other people looking at their screen. I told them that I sometimes barely understand what is happening on my screen let alone anyone else's screen in the vicinity. They also told me they are learning quicker because they are seeking out help less.

My work tickets lately have been all over the stack, touching a little bit of everything. It's been fun and keeps things fresh. I hope soon to be able to deep dive into a technology like I did a few months ago when we shipped a Vue.js prototype.

We have a moodboard at the office — a big whiteboard off in one corner with a grid of days and people's names. We draw pictures, or write words, to describe our days so we can look back in [retrospective meetings](https://en.wikipedia.org/wiki/Scrum_(software_development)#Sprint_retrospective) (once per two-week sprint). Before, when it was rarer to be remote, we sometimes had people describe their moodboard drawings for someone else to quickly sketch up which I enjoyed doing even when my doodles came out goofy. Now, we have moved to a fully remote moodboard with Miro. Which means we can use GIFs! On Miro, you can also see ghost-like cursors, belonging to faraway engineers, move around the shared board.

Our plant growing has extended to things that are consumable. With garlic, onion and spring onion, as well as pak choy, we hope to grow enough to make one meal within lockdown! It is meditative to care for them. I have lost count of the things that are growing in this apartment. And I'm thankful that Madeline is the main waterer.

![Potted cloves of garlic](plants.jpg)

The bi-monthly hackday was held remotely and Madeline and I spun up a _QUAKE III: Team Arena_ server from scratch and deployed it so that we can use it for the #gaming Slack channel at work. We used to have the server hosted on someone's laptop. Our humorous presentation was about client/server latency and why the person hosting the game had an unfair advantage. We also started building a Vue.js app to help us keep track of goods around the house that we buy on different schedules e.g. kitchen roll, and black bin bags but ran out of time to fully flesh it out.

I've been reading more in the evenings. I bought The Electric State by Simon Stålenhag after admiring his art on Twitter. I've been reading articles from [Lobsters](https://lobste.rs/), [Hacker News](https://news.ycombinator.com/), as well as The New Yorker — I really liked [The Friendship That Made Google Huge](https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge).

Hope y'all are staying safe 👋


Writing code, working (almost) solo, and exploring remote-life during the UK lockdown.

How I'm Working Remotely


Screens tend to keep me awake in the evening so I wondered if I could print out my Twitter home timeline, live, as I read a book in the evening.

Here's the final project in action:

![Ticket tape in action, a receipt printer printing out textual tweets](in-action.gif)

I was gifted a receipt printer for Christmas to use with my Raspberry Pi. A _POS58 USB Thermal Receipt Printer_ also known as a _ZJ-5890K_. It's sold by a few different brands under different names.

I couldn't get the drivers to work on Linux, MacOS, or Windows. I was stuck until I found [vpatron/usb\_receipt\_printer](https://github.com/vpatron/usb_receipt_printer) — a guide that walks through setting the device up on a Raspberry Pi. It shows how to write to the USB port directly (a tactic I previously tried and failed). The included demo program solved all of my printing problems. Thanks Vince Patron!

## Tweepy

Twitter has a powerful API but it's a little cryptic at first glance. Tweepy is a Python library for accessing the Twitter API — it has good documentation and community support.

In order to use Tweepy, we need to create an application on Twitter's Developer platform to get the keys and tokens we need to authenticate.

![Keys, secret keys and access token management page](keys-and-tokens.png)

We will have three files in total.

- `config.py` — for managing our keys.
- `poller.py` — for polling Twitter's API via Tweepy, and converting tweets to text.
- `posprint.py` — for calling the receipt printer.

We save the keys in a separate file. A better way of managing keys is for your application to access them at runtime through [environment values](https://en.wikipedia.org/wiki/Environment_variable). This can be safer (it avoids keys accidentally being pushed to GitHub) and lets our application use different keys in different environments (e.g. local versus live production deployment).

```python
# config.py

consumer_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
consumer_secret = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
access_token = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
access_token_secret = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
```

Twitter has a streaming data API but it was harder to set up than the chosen solution.

We ask for the latest 20 tweets every minute and check that we haven't seen them before. Then we format, clean, and print the new ones.

```python
# poller.py

import tweepy
from emoji import demojize
from time import sleep
from collections import OrderedDict

import config
import posprint

auth = tweepy.OAuthHandler(
    config.consumer_key,
    config.consumer_secret
)
auth.set_access_token(
    config.access_token,
    config.access_token_secret
)

api = tweepy.API(auth)

# we use an ordered dictionary so we can remove old tweets
# from the data structure. Our program may be long-running
# and we don't want to run out of memory
seen_statuses = OrderedDict()

while True:
    public_tweets = api.home_timeline(tweet_mode='extended')
    for status in public_tweets:

        # we're interested in statuses we haven't seen before
        if status.id not in seen_statuses:

            # store the statuses we use
            seen_statuses[status.id] = status

            # dump old statuses that can't appear again
            if len(seen_statuses) > 20:
                seen_statuses.popitem()

            user = status.user.screen_name + ':\n'

            # get the full tweet text regardless of type
            # https://github.com/tweepy/tweepy/issues/935
            if hasattr(status, 'retweeted_status'):
                tweet = user + status.retweeted_status.full_text
            else:
                tweet = user + status.full_text

            # trim unprintable characters that POS58 can't output
            # by turning emojis into textual representation.
            # 😅 becomes :grinning_face_with_sweat:
            tweet = demojize(tweet).encode(
                encoding='ascii',
                # throwaway non ascii characters
                errors='ignore'
            ).decode()

            posprint.output(tweet)

    # twitter restricts statuses/home_timeline to once per minute
    sleep(65)
```

Let's modify Vince's demo program [vpatron/usb\_receipt\_printer](https://github.com/vpatron/usb_receipt_printer) to export the function we call above — `posprint.output(tweet)` — with our textual tweet.

```python
# posprint.py

import usb.core
import usb.util
import textwrap

"""Demo program to print to the POS58 USB thermal receipt printer. This is
labeled under different companies, but is made by Zijiang. See
http://zijiang.com

MIT License — Copyright (c) 2019 Vince Patron
"""

# In Linux, you must:
#
# 1) Add your user to the Linux group "lp" (line printer), otherwise you will
#    get a user permissions error when trying to print.
#
# 2) Add a udev rule to allow all users to use this USB device, otherwise you
#    will get a permissions error also. Example:
#
#    In /etc/udev/rules.d create a file ending in .rules, such as
#    33-receipt-printer.rules with the contents:
#
#   # Set permissions to let anyone use the thermal receipt printer
#   SUBSYSTEM=="usb", ATTR{idVendor}=="0416", ATTR{idProduct}=="5011", MODE="666"

def output(data):
    # find our device
    # 0416:5011 is POS58 USB thermal receipt printer
    dev = usb.core.find(idVendor=0x0416, idProduct=0x5011)

    # was it found?
    if dev is None:
        raise ValueError('Device not found')

    # disconnect it from kernel
    needs_reattach = False
    if dev.is_kernel_driver_active(0):
        needs_reattach = True
        dev.detach_kernel_driver(0)

    # set the active configuration. With no arguments, the first
    # configuration will be the active one
    dev.set_configuration()

    # get an endpoint instance
    cfg = dev.get_active_configuration()
    intf = cfg[(0,0)]

    ep = usb.util.find_descriptor(
        intf,
        # match the first OUT endpoint
        custom_match = \
        lambda e: \
            usb.util.endpoint_direction(e.bEndpointAddress) == \
            usb.util.ENDPOINT_OUT)

    assert ep is not None

    # print!
    lines = textwrap.wrap(data, width=30)
    for line in lines:
        ep.write(line + '\n')
    ep.write('\n\n\n\n')

    # reattach if it was attached originally
    dev.reset()
    if needs_reattach:
        dev.attach_kernel_driver(0)
        print('Reattached USB device to kernel driver')
```

Our program is started by running `poller.py`.

If everything went correctly, our POS58 printer should output tweets as they arrive in the following width-restricted format which avoids breaking words where possible.

```text
jessfraz: Ive started using
Pocket to keep articles I want
to read later and also
combined it with a bot to read
RSS feeds. Its alright so far.
What are some tools like this
you cant live without?
```

Find the project files on [GitHub](https://github.com/healeycodes/twitter-ticker-tape). Thanks again to [Vince Patron](https://github.com/vpatron) for his comprehensive guide!


Building a fun toy with my Raspberry Pi, Python, and Tweepy.

Twitter Ticker Tape with a POS58 Receipt Printer


I'm in year two of technical blogging and I've learned a lot. I have far _far_ more views and comments than I thought I would at this point. At the beginning I was throwing my thoughts into the void and it felt good. Reading [**The Developer's Guide to Content Creation**](https://www.developersguidetocontent.com/) reminded me of those roots. It inspired me.


I love seeing people blog about their passions and thoughts especially when it's on a personal website. Personal websites are cool and weird and make the internet more enjoyable for me. If there's a link in your bio there's a high chance I'll click it.

So I'm very happy, on this quiet and unusually cold Sunday, that there are resources for those who are starting out on this journey!

## The Best Beginner Introduction to Blogging

Without a doubt [The Developer's Guide to Content Creation](https://www.developersguidetocontent.com/) is the best resource for those looking to start writing content for developers. Before reading it, I had a shortlist of topics that I thought such a resource should cover and it nailed every one. Morillo's writing is clear and engaging. She tells us how to write content that is accessability in presentation but also in content. She reminds us how not to write as well:

- _"Just follow these simple steps"_
- _"It’s easy to put together"_
- _"The answer is obvious"_

> I’ve realized that many writers of technical content use these words to make the subject matter seem less intimidating to the reader. They genuinely want to remove friction for their users, but they’re using language that makes the reader feel like the problem is with them.

## Rich in Resources

The Developer's Guide to Content Creation also contains copious amounts of worksheets to inspire you, help you strategize, and put together a content calendar.

There are resources linked throughout the book that compliment each section. There's also an example of _how_ to insert resources into an article in a more descriptive manner. To mirror one of Morillo's examples, take GitHub's trending page.

- "GitHub has a list of trending repositories. Check it out [here](https://github.com/trending)."
- "[Search through GitHub's list of trending repositories ](https://github.com/trending)."

For a reader, the second version flows better.

Morillo recommends comprehensive resources like [DigitalOcean's Technical Writing Guidelines](https://www.digitalocean.com/community/tutorials/digitalocean-s-technical-writing-guidelines) as well as tools that take seconds to set up and run on your writing.

> [AlexJS](https://alexjs.com/) is an open-source tool that will help you "spot insensitive, inconsiderate writing."

> [Hemingway App](http://www.hemingwayapp.com/) is an online editor that helps you simplify your writing. It highlights confusing sections, areas that are too long, and more.

To add my own, reading other technical blogs can be a good resource and I'd recommend Hacker Noon's [shortlist for Personal Developer Blog of the Year](https://hackernoon.com/personal-developer-blog-of-the-year-hacker-noon-noonies-awards-2019-hz2tu32ql) to get started. Drew DeVault posts [great content](https://drewdevault.com/) and [will occasionally pay out $20](https://drewdevault.com/make-a-blog) to people starting a technical blog on an open platform (i.e. not Medium). Google have also released a free [course on technical writing](https://developers.google.com/tech-writing) although I believe it's aimed more towards those writing technical documentation.

## The Shape of a Post

Morillo talks a lot about how technical blogs are consumed and how they appear to users.

> Before you start writing, you need to understand how people read online. People don’t read every word and every sentence; they scan. They’re scanning for the most relevant information and things that pop out at them will command their attention.

Thinking about the shape of a post can help during the editing stage. It's a way to cut through complexity and grab readers who would have otherwise failed to skim read blocks of text and given up. It also helps those who come to your post only looking for that one line of code that will unblock them.

We're reminded that people scan for "keywords, section headings, bullets, anchor text (when hyperlinked), words that are bolded or italicized for emphasis".

<br>

The Developer's Guide to Content Creation covers topics that were getting dusty in the back of my mind as well as delivering a nuggets of wisdom. Even as I write this post, I'm considering Morillo's advice. It's for this reason that I would recommend this book to anyone writing content for developers no matter their experience. It has changed the way I will approach writing for my personal blog as well as for my company's blog.

Following this book, Morillo is currently writing [The Developer's Guide to Book Publishing](https://www.developersguidetocontent.com/book-publishing-guide) — I'm looking forward to it!


Discussing an amazing resource for those writing for developers.

Book Review: The Developer's Guide to Content Creation


My second passion alongside coding is writing. Whenever I can combine them, I do. 

![The analysis tool in action](analysis-tool.gif)

This weekend I used the [DEV API](https://docs.dev.to/api/) to build [a tool](https://dev-article-analysis.glitch.me/) that analyzes your articles over time. It's an interactive website where you can enter DEV usernames and get text statistics shown in pretty graphs. The metrics that I chose were reading level, ease of reading, and sentiment analysis.

It's open-sourced at [healeycodes/dev-article-analysis](https://github.com/healeycodes/dev-article-analysis)!

### Readability

The Flesch–Kincaid readability tests were developed in the U.S. Navy in the 70s. These scales are used to measure and compare books, newspapers, and articles. In Florida, they are used to encourage readable language in insurance policies.

> Florida Statutes Section 627.4145 (1) (a) The text achieves a minimum score of 45 on the Flesch reading ease test as computed in subsection (5) or an equivalent score on any other test comparable in result and approved by the office;

Flesch-Kincaid Grade scores text at a U.S. grade level. A score of 10.4 means that a tenth grader would be able to understand the article.

Flesch reading ease is a similar test with different weightings. It results in a score generally ranging between 0-100. Higher means easier to read. Wikipedia has a [table](https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests#Flesch_reading_ease) which relates score to grade level with some notes.

Technical articles, especially those with code excerpts, get wide-ranging scores. However, a user's writing style (e.g. a similar amount of code per article) means that a reasonable line can be drawn over time, perhaps leading to some basic conclusions. [text-readability](https://www.npmjs.com/package/text-readability) was used to calculate these scores in the project.

### Sentiment

Sentiment analysis helps us understand the emotional polarity of a text. For example, the phrase `I love cats, but I am allergic to them.` gives a comparative score of `0.1111111111111111`. To calculate this, a list of words (and emojis) are given weightings. -5 to 5, positive to negative. The only tokens in our sentence with scores are `{ allergic: -2 }, { love: 3 }`. We take the number of tokens, `9`, and calculate `(3 + -2) / 9` to find the comparative score. This example is used in the documentation of [sentiment](https://www.npmjs.com/package/sentiment).

The DEV API provides the text of an article in HTML or markdown, neither of which is friendly to analysis. This is why [remove-markdown](https://www.npmjs.com/package/remove-markdown) is used to reduce the `body_markdown` value to plaintext.

### Interaction

Users enter a DEV username. As the articles are retrieved, a progress status (and a random positive emoji) are sent to the page via WebSocket. This means the user isn't left sitting looking at a spinner.

I normally use Chart.js to render basic line charts but wondered if there were a simpler solution. I tried ApexCharts.js and didn't run into any problems. The online documentation is good and I searched and found the solution to one of my errors in a GitHub Issue. However, I might as well have used Chart.js because the level of customization I required meant the syntax for both libraries was near-identical.

The charts can be hovered to see article titles and their scores for each metric.

When arriving at the website, the graphs are already filled with precalculated data from my own DEV username (healeycodes). This is to help visitors better understand what the end result of their interaction will be.

### Deployment

I developed this Node/Express application on [Glitch](https://glitch.com/). Their cloud IDE and console lets me move fast and get feedback from people seconds after I've made a change. You can export the project to a GitHub project (it creates a `glitch` branch which you can open a pull request from).

![Glitch cloud IDE](glitch-ide.png)

Sometimes I want to upload image files and in the past I found this tricky. Glitch wants you to use their asset mechanism for this, which makes exporting the whole project to GitHub harder than it needs to be. My workaround is to upload the image to the project using the asset mechanism, copy the long URL they provide, `wget` this URL in the console which downloads it into the project, and finally using `mv` to change the name of the image.

This often brings the cloud IDE out of sync but entering `refresh` in console fixes this.


Building a tool for users to analyze their work over time.

DEV Article Analysis


I migrated from Jekyll to Gatsby recently and so far I've had a really neat time. The whole process took about a week of casual coding (a few hours here and there). The Gatsby ecosystem enabled me to quickly add a few features to my blog that I thought were missing; dark mode, better syntax highlighting, and the ability to design with components.

Gatsby starters are boilerplate Gatsby sites [maintained by the community](https://www.gatsbyjs.org/docs/starters/). One of the reasons I love them is that they use **Semantic HTML**. This is great because:

- It helps with search engine optimization — web crawlers are able to understand which parts of your pages are important.
- It helps with accessibility — for people who use non-traditional browsers and screenreaders.
- It helps with maintenance — I was able to pick up a starter and understand what the different parts of the template referred to due to the semantic tags.

Take this example from `gatsby-starter-blog` — the most popular starter and the base for my [current blog](https://healeycodes.com) (in-line styling removed).

```jsx
<article>
  <header>
    <h1>{post.frontmatter.title}</h1>
    <p>{post.frontmatter.date}</p>
  </header>
  <section dangerouslySetInnerHTML={{ __html: post.html }} />
  <hr />
  <footer>
    <Bio />
  </footer>
</article>
```

I've seen quite a few beginner web development resources that skip on semantic HTML and encourage what I'll call 'div-spamming'. The HTML5 spec weighs in on this [issue](https://html.spec.whatwg.org/multipage/grouping-content.html#the-div-element).

> Authors are strongly encouraged to view the div element as an element of last resort, for when no other element is suitable. Use of more appropriate elements instead of the div element leads to better accessibility for readers and easier maintainability for authors.

# Coming From Jekyll

I started blogging with Jekyll a year ago because I hosted my blog on GitHub pages and it was the static site generator with the least friction. It was a great choice at the time as it enabled me to get up and running straight away.

I've seen many people warning others (engineers in particular) to avoid rolling their own blogging solutions. The advice is that you should start writing and publishing first. This is because building a blog can function as procrastination and who knows if you actually enjoy blogging (the activity) or the idea of having blogged (the achievement).

With Jekyll, I used basic markdown and transferring written content to Gatsby wasn't too hard. Images had to be moved from one disorganized folder into separate folders. URLs were a bit of a pain and took 1.5hrs of manual work. I wanted all of my old posts to keep their location on the web so I added a front matter tag called `path` to override the default naming scheme. My old URLs were too long and included categories (which I'm still to implement) so the path scheme from now on will be the title only.

I extended `onCreateNode` in `gatsby-node.js`. I'm not sure if this is the best practice way to implement this feature but it works great so far. 

```js
exports.onCreateNode = ({ node, actions, getNode }) => {
  const { createNodeField } = actions

  if (node.internal.type === `MarkdownRemark`) {
    // Check to use legacy path
    const slug = node.frontmatter.path || createFilePath({ node, getNode })
    createNodeField({
      node,
      name: `slug`,
      value: slug,
    })
  }
}
```

# Syntax Highlighting

Code excerpts show up in a lot of my posts and I like them to be easy to parse.

I installed [gatsby-remark-prismjs](https://www.gatsbyjs.org/packages/gatsby-remark-prismjs/) for syntax highlighting and was up and running in about an hour with another hour spent tinkering styles to match my light/dark mode toggle. I use [New Moon Theme](https://github.com/taniarascia/new-moon) by Tania Rascia for my code excerpts. I couldn't find a version of the theme for PrismJS so I extracted the styling from Tania's (MIT-licensed) [blog](https://www.taniarascia.com/). My site's general color theme is custom.

One of the reasons I'm mentioning plugins is that I found it hard to integrate them with Jekyll and I feel like it wasn't just my inexperience with Ruby that was holding me back. Perhaps it's due to the hype surrounding Gatsby which means there are up-to-date resources. I've contributed one (small) open-source fix to the Jekyll project and I would still recommend it for anyone looking for a sensible system for HTML/CSS that has wide community support e.g., GitHub pages, Netlify, etc. If you want to avoid JavaScript, Jekyll is the way to go.

For my light/dark mode I use `gatsby-plugin-dark-mode` which works well out of the box and has good (but not great) documentation. For theme-toggling, I researched and found that a common pattern was to declare CSS variables in body scope and then to override these variables in _class_ scope. This way, the `dark` class can be added to the `<body>` tag which means `dark` CSS variables take precedence due to CSS Specificity. Classes are then toggled on and off the `<body>`, storing the preference in local browser storage.

```css
body {
  --bg: #eaeaeb;
  --textNormal: #414158;
}

body.dark {
  --bg: #21212c;
  --textNormal: #eaeaeb;
}
```

### Designing With Components

The first React component I wrote for my blog was for wrapping the `<ThemeToggler />` from `gatsby-plugin-dark-mode` into a component. It switches between a sun and a moon to let the user know which theme can be switched to. The base for this is the example code from the [docs](https://www.gatsbyjs.org/packages/gatsby-plugin-dark-mode/).

```jsx
<ThemeToggler>
  {({ theme, toggleTheme }) => (
    <label style={{ cursor: `pointer`, opacity: 0.8 }}>
      <input
        style={{ display: `none` }}
        type="checkbox"
        onChange={e => toggleTheme(e.target.checked ? "dark" : "light")}
        checked={theme === "dark"}
      />
      {theme === "dark" ? `☀️` : `🌑`}
    </label>
  )}
</ThemeToggler>
```

I've never used React as part of a blogging solution. I like the hierarchical UI approach that's encouraged. Including CSS-in-JS makes sense for the scale of my website. It's easier for me to reason about and quickly tinker with.

# How I Deploy

My website source exists in a GitHub repository. I write in markdown in VS Code, commit, and push. Netlify is connected to the repository and builds and deploys every commit to master. The build process takes 2m50s (30s locally).

I previously used Netlify for Jekyll and setting up either static site generator involved about 10 clicks and entering one or two build commands. The walkthrough covers everything.

Overall, the Gatsby experience has been very enjoyable. Everywhere I look in my online bubble (Twitter, DEV, lobste.rs) people are talking about Gatsby. It's nice to be part of the crowd.

I'm also happy that my site (despite being React-based) works fine without JavaScript enabled (barring the theme toggle, which I might hide with `<noscript>` styling).


Gatsby Sparks Joy


I love code puzzles. Always have. My deskmate told me about a puzzle that no one in the office had been able to solve when they shared it around. I was intrigued because of the challenge but also because it was a JavaScript puzzle.

I'd understand if it were a Project Euler problem (they can be mathy) or perhaps if no one had found the optimal solution to a LeetCode 'Hard' (I've seen some problems that once upon a time were research papers).

__The puzzle took me two weeks to solve__. I became stuck almost instantly and then it hung around in the back of my mind until the solution came to me. First, let's take a look at the easier version of the problem which helped me unlock the harder version.

Don't scroll too fast unless you want spoilers.

### [Codewars: Multi Line Task++: Hello World](https://www.codewars.com/kata/5935558a32fb828aad001213)

- Write a function `f` that returns `Hello, world!`.
- Every line must have at most 2 characters, and the total number of lines must be less than 40.

Without the line restriction. The solution is:

```javascript
function f () { return 'Hello, world!'; }
// or
const f = _ => 'Hello, world!';
```

I started splitting up the code and shortened the variable declaration. We can throw away the `const` and allow the function to exist in the global scope. We can also use template strings to break up the string into multiple lines.

Errors incoming.

```javascript
f
=_
=> // Uncaught SyntaxError: Unexpected token '=>'
`H
el
l,
 w
or
ld
!` // This string has newline characters in it!
```

My next idea was to define the function inside an object and then retrieve the function out of the object.

```javascript
f=
{g
()
{
return 'Hello, world!'; // Too long!
}}
[`
g
`[
1]
]

// 'Beautified'
f = {
        g() {
            return 'Hello, world!';
        }
    }
    [`
g
` [
        1
    ]]
```

No errors, I was declaring a function but now I couldn't figure out how to return from the _inner_ function without using the `return` keyword. It felt like I was close but I wasn't. I was still stuck on defining the string without newline characters as well.

### Finding inspiration: [JSFuck](http://www.jsfuck.com/)

> JSFuck is an esoteric and educational programming style based on the atomic parts of JavaScript. It uses only six different characters to write and execute code.

Reading through this project's [source code](https://github.com/aemkei/jsfuck/blob/master/jsfuck.js) really opened my mind to some parts of JavaScript that never really come up unless you're doing something like writing a library or code golfing.

Once I figured out how to remove the newline characters from the `Hello, world!` message (escaping with a backslash `\`) everything else fell into place. I could now use square brackets `[]` on objects to run pretty much anything. However, one of the requirements was to keep the total line count under 40 (one of the reasons why using JSFuck encoding was out of the question).

My idea was to create a new function by calling [Function.prototype.bind](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_objects/Function/bind) on a [String](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String) method. I used [String.prototype.trim](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/trim) because it had the shortest name (and conveniently got rid of any trailing newlines too).

```javascript
f=
''
[
'\
t\
r\
i\
m'
][
'\ // Get a new function where `this` is "Hello, world!"
b\
i\
n\
d'
]` // Tagged template, see below
H\
e\
l\
l\
o\
,\
 \
w\
o\
r\
l\
d\
!`
```

I also used tagged templates to pass `Hello, world!` as an argument to bind.

> Tags allow you to parse template literals with a function. The first argument of a tag function contains an array of string values.

Let's take it up a level to the harder version that started this journey!

### Codewars: [Multi Line Task∞: Hello World](https://www.codewars.com/kata/59a421985eb5d4bb41000031)

- Write a function `f` that returns `Hello, world!`.
- Every line must have __at most 1 character__, and the total number of lines must be less than 145.

Without having first solved the two-characters-per-line version I don't think I would have come close to answering this version.

The solution I went for is the same, we use `bind` on `trim` and pass the message as an argument (without template tags this time). To access the String object we use `[]+[]` which evaluates to `""`.

Since we can no longer escape the newline character from within a template string we have to use a workaround. The property names (`trim`, `bind`) and the message (`Hello, world!`) have to be built with concatenated variables.

We use [destructuring assignment](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Destructuring_assignment) on a template string and use empty slots in the first array to 'skip' assigning the newline character to anything. Like this:

```javascript
[,a,,b,,c] = 'xHxix!' // avoid the "x"s
a + b + c // Evaluates to: "Hi!"
```

I didn't optimize the solution any further once it passed the requirements. It's been left verbose to better explain what's going on (for example, we only need one "l" in the template string).

```javascript
[ // With destructuring assignment, start declaring variables
,
t
,
,
r
,
,
i
,
,
m
,
,
b
,
,
i
,
,
n
,
,
d
,
,
H
,
,
e
,
,
l
,
,
l
,
,
o
,
,
c // Comma
,
,
s
,
,
w
,
,
o
,
,
r
,
,
l
,
,
d
,
,
x // Exclamation mark
]
=
`
t
r
i
m
b
i
n
d
H
e
l
l
o
,
 
w
o
r
l
d
!
`
f // Start declaring our function
=
( // This evaluates to "" or, the String object
[
]
+
[
]
)
[ // `trim`
t
+
r
+
i
+
m
]
[ // `bind`
b
+
i
+
n
+
d
]
( // Can use parentheses or template tag syntax to call `bind`
H
+
e
+
l
+
l
+
o
+
c
+
s
+
w
+
o
+
r
+
l
+
d
+
x
)
```

<br>

I'm definitely taking a break from language-specific coding puzzles — give me logic over syntax! However, I'm glad I scratched this itch.

Writing JavaScript with a one character width restriction.

Hardest JavaScript Puzzle I've Ever Solved


CAPTCHAs are [not accessible](https://www.w3.org/TR/turingtest/) and in some cases not even effective but there's a lot to be learned by generating our own!

Find the source code for this article at [healeycodes/captcha-api](https://github.com/healeycodes/captcha-api)

## A solution for spam

Let's imagine a client who demands a solution for bot spam. They ask for an image and a string of the image's text. You call to mind every inscrutable jumbled mess of letters and numbers you've frustratingly failed to solve. You agree to the task nonetheless.

![Am I a human? Yes. Can I type phi off the top of my head? No. Picture of hard CAPTCHA](hard-captcha.png)

This client has a whole fleet of websites. Different sized CAPTCHAs are required in different places. They will provide a width and a height. This describes the specification of our API.

JavaScript is great for generating images because we can lean on the [Canvas API](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API). I've always found it to be handy to use with a lot of Stackoverflow content for when I'm stuck.

We don't want to generate our CAPTCHAs in browser-land because the bots that we're trying to keep out can inspect the source code, find the values in memory, and try all kinds of other tricky tactics.

## A Node.js service

Let's move it to the back-end to a service that can be called as desired. Someone has already solved the issue of accessing a Web API where there is not one, with [node-canvas](https://www.npmjs.com/package/canvas) or `npm i canvas`.

> [canvas] is an implementation of the Web Canvas API and implements that API as closely as possible.

We'll need to generate some random text each time. So let's write two functions to help us. For our API, we'll break logic down into functions that do one thing (and one thing well) so that the end result is easy to reason about and maintain.

```javascript
/* captcha.js */

// We'll need this later
const { createCanvas } = require("canvas");

// https://gist.github.com/wesbos/1bb53baf84f6f58080548867290ac2b5
const alternateCapitals = str =>
  [...str].map((char, i) => char[`to${i % 2 ? "Upper" : "Lower"}Case`]()).join("");

// Get a random string of alphanumeric characters
const randomText = () =>
  alternateCapitals(
    Math.random()
      .toString(36)
      .substring(2, 8)
  );
```

There's no way to automatically scale text in a canvas (just like in the browser *weeps*) so we'll need some helper functions for that too. Depending on the length of your CAPTCHA and how you want the text to be positioned inside the image, you may need to test run it. Here are some variables I prepared earlier.

```javascript
const FONTBASE = 200;
const FONTSIZE = 35;

// Get a font size relative to base size and canvas width
const relativeFont = width => {
  const ratio = FONTSIZE / FONTBASE;
  const size = width * ratio;
  return `${size}px serif`;
};
```

This scales the text so as long as the proportions of the canvas remain the same, we can expect a similar-looking image.

For this article, we're just going to rotate the text but there are tons of ways to distort the text to hide it from bots and I'd love to see what you come up with (try searching for "perspective transform canvas javascript").

When rotating a canvas, the value we pass is in [radians](https://en.wikipedia.org/wiki/Radian) so we need to multiply our random degrees by `Math.PI / 180`.

```javascript
// Get a float between min and max
const arbitraryRandom = (min, max) => Math.random() * (max - min) + min;

// Get a rotation between -degrees and degrees converted to radians
const randomRotation = (degrees = 15) => (arbitraryRandom(-degrees, degrees) * Math.PI) / 180;
```

No more helper functions, I promise. We're going to get to the real meat of it now. The logic is broken up into two functions. `configureText` takes a canvas object and adds and centers our random text. `generate` takes a width and height value (remember the specification we were given?) and returns a [Data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs) of a PNG image — our CAPTCHA. 

> Data URLs, URLs prefixed with the `data:` scheme, allow content creators to embed small files inline in documents.

```javascript
// Configure captcha text
const configureText = (ctx, width, height) => {
  ctx.font = relativeFont(width);
  ctx.textBaseline = "middle";
  ctx.textAlign = "center";
  const text = randomText();
  ctx.fillText(text, width / 2, height / 2);
  return text;
};

// Get a PNG dataURL of a captcha image
const generate = (width, height) => {
  const canvas = createCanvas(width, height);
  const ctx = canvas.getContext("2d");
  ctx.rotate(randomRotation());
  const text = configureText(ctx, width, height);
  return {
    image: canvas.toDataURL(),
    text: text
  };
};
```

We can consider all of the functions apart from `generate` to be private functions that shouldn't be used elsewhere, so let's just export this function.

```javascript
module.exports = generate;
```

## An API served by Express

So far we have one file, `captcha.js` which contains our image generation logic. To make this functionality available to be called by someone else we will serve it via an HTTP API. Express has the most community support for this kind of task.

The routes we'll host are:

- `/test/:width?/:height?/`
    - Used to get an image tag for manual testing.
- `/captcha/:width?/:height?/`
    - Used to get a CAPTCHA object for proper usage.

The question marks in the route here are the Express syntax for optional URL parameters. This means the client can provide none, the first one, or both. We'll validate that integers are passed as values (required by canvas) and if not we'll use sensible defaults.

The Express app in full:

```javascript
/* app.js */

const captcha = require("./captcha");
const express = require("express");
const app = express();

// Human checkable test path, returns image for browser
app.get("/test/:width?/:height?/", (req, res) => {
  const width = parseInt(req.params.width) || 200;
  const height = parseInt(req.params.height) || 100;
  const { image } = captcha(width, height);
  res.send(`<img class="generated-captcha" src="${image}">`);
});

// Captcha generation, returns PNG data URL and validation text
app.get("/captcha/:width?/:height?/", (req, res) => {
  const width = parseInt(req.params.width) || 200;
  const height = parseInt(req.params.height) || 100;
  const { image, text } = captcha(width, height);
  res.send({ image, text });
});

module.exports = app;
```

This Express app is exported so that we can test it. Our API is functional at this point. All we have to do is serve it which the following file takes care of.

```javascript
/* server.js */

const app = require("./app");
const port = process.env.PORT || 3000;

app.listen(port, () => console.log(`captcha-api listening on ${port}!`));
```

Navigating to `http://localhost:3000/test` rewards us with our basic CAPTCHA. Browsers will add a `body` and `html` tag if otherwise omitted.

![Our CAPTCHA](our-captcha.jpg)

## A valid Data URL

It's time to write some tests but first, put away your [unwieldy regular expressions](https://stackoverflow.com/a/1732454/10044202). There's a [library](https://www.npmjs.com/package/valid-data-url) that has already solved this problem. `valid-data-url` does exactly what it says on the tin.

I like to use Jest as my test runner. For no reason other than it's always worked for me and when it hasn't I've been able to find the answer. My setup is setting the `scripts` key in `package.json` like so:

```javascript
  "scripts": {
    "test": "jest"
  }
```

This is so I can type `npm test` (which is what many CI systems default to as well). Jest then finds and runs all of our tests.

Our app's test file imports the Express application object and uses `supertest` to mock HTTP requests against it. We use async/await syntax to reduce on callbacks.

```javascript
/* app.test.js */

const request = require("supertest");
const assert = require("assert");
const validDataURL = require("valid-data-url");
const app = require("../app");

describe("captcha", () => {
  describe("testing captcha default", () => {
    it("should respond with a valid data URL", async () => {
      const image = await request(app)
        .get("/captcha")
        .expect(200)
        .then(res => res.body.image);
      assert(validDataURL(image));
    });
  });

  describe("testing captcha default with custom params", () => {
    it("should respond with a valid data URL", async () => {
      const image = await request(app)
        .get("/captcha/300/150")
        .expect(200)
        .then(res => res.body.image);
      assert(validDataURL(image));
    });
  });
});
```

Given the size of this application (small) I'm content with leaving it at two integration tests.

## Constant integration with a GitHub Action

Since we used the standard npm test command (`npm test`) to configure our repository, we can set up a GitHub Action with a few clicks. This way, our application will be built and tested every time code is pushed.

![Build and test your JavaScript repository. Node.js Build and test a Node.js project with npm. "Set up this workflow". npm ci\nnpm run build --if-present\nnpm test](github-workflow.png)

Now we've got a sweet badge to show off!


Using Canvas, Express, and a GitHub Action.

Let's Build a CAPTCHA Generator with Node.js


My [first program](https://github.com/healeycodes/conways-game-of-life) in Go was Conway's Game of Life. This time I made an [in-memory HTTP caching server](https://github.com/healeycodes/in-memory-cache-over-http) with similar methods to Memcached like increment/decrement and append/prepend.

I use caching pretty often but I had never coded up a Least Recently Used (LRU) cache by hand before. Neither had I used Go's `net/http` or `container/list` packages. Both packages are elegant and have great documentation and readable source code — the latter being one of my favorite things about Go.

With my first program, I threw everything in a file called `main.go` and called it a day. This time I created two packages.

- api — an HTTP server which responds to GET requests like `/set?key=name&value=Andrew&expire=1571577784` and `/get?key=name`.
- cache — an LRU cache that allows an expire time and a max number of keys.

## Caching

As an LRU cache fills up and needs to forget an item it will choose the one that was _last accessed_ the _longest time ago_. It also allows lookups in constant time. To build mine, I mapped strings to doubly linked list elements.

```go
// Store contains an LRU Cache
type Store struct {
	mutex *sync.Mutex
	store map[string]*list.Element
	ll    *list.List
	max   int // Zero for unlimited
}
```

Each list element is a Node. We also store the key inside the Node so that when the cache fills up, we can do a reverse lookup from the back of the list to remove that item from the map.

```go
// Node maps a value to a key
type Node struct {
	key     string
	value   string
	expire  int  // Unix time
}
```

The `mutex` field of the Store allows the cache to avoid having concurrent readers and writers to the data structures. The default behavior of `net/http` is to spawn a goroutine for every request. In [some cases](https://groups.google.com/forum/#!msg/golang-nuts/HpLWnGTp-n8/hyUYmnWJqiQJ) it appears to be okay to have multiple concurrent map readers but I played it safe and every cache operation is guarded by a mutex.

In a previous version of this article, I exported the mutex from the cache and locked/unlocked in the API's middleware. However, this meant that the application may be bottlenecked by HTTP read/write speeds (a friendly commentator pointed this out). Instead of changing where the middleware locked/unlocked to avoid the read/write limitation, I chose to hide the mutex inside the cache to make the application safer for future maintainers while also gaining the performance boost.

There is not much in the middleware right now apart from some basic logging that helps during development. Having an overall middleware usually cuts down on duplicate code.

```go
// Middleware
func handle(f func(http.ResponseWriter, *http.Request)) func(w http.ResponseWriter, r *http.Request) {
	return func(w http.ResponseWriter, r *http.Request) {
		if getEnv("APP_ENV", "") != "production" {
			fmt.Println(time.Now(), r.URL)
		}
		f(w, r)
	}
}
```

## Give key, get value

In Go, the HTTP protocol is a first-class citizen. Clients and servers are simple (and extensible). Writing the `/get` method for my server uses six lines.

```go
// Get a key from the store
// Status code: 200 if present, else 404
// e.g. ?key=foo
func Get(w http.ResponseWriter, r *http.Request) {
	value, exist := s.Get(r.URL.Query().Get("key"))
	if !exist {
		http.Error(w, "", 404)
		return
	}
	w.Header().Set("content-type", "text/plain")
	w.Write([]byte(value))
}
```

The cache method that maps to this route is more complex. It looks for a key in the map and checks that it is valid (not expired or due for cleanup). It returns `(string, bool)` — (the value or an empty string, true if the value was found). If a string is found, its Node is moved to the front of the list because it is now the most recently accessed.

If the key is expired then it's passed to the delete method which will remove the key from the map and the Node will be passed to the garbage collector.

```go
// Get a key
func (s *Store) Get(key string) (string, bool) {
	s.mutex.Lock()
	defer s.mutex.Unlock()
	current, exist := s.store[key]
	if exist {
		expire := int64(current.Value.(*Node).expire)
		if expire == 0 || expire > time.Now().Unix() {
			s.ll.MoveToFront(current)
			return current.Value.(*Node).value, true
		}
	}
	return "", false
}
```

I've been reaching for `defer` in my other programming languages recently. It helps one better manage the lifetime of objects. It's explained in a Go [blog post](https://blog.golang.org/defer-panic-and-recover).

> Defer statements allow us to think about closing each file right after opening it, guaranteeing that, regardless of the number of return statements in the function, the files will be closed.

The syntax `current.Value.(*Node).value` performs a type assertion on the list element providing access to the underlying Node pointer. If it's the wrong type, this will trigger a panic. Type assertions can also return two values if requested, the second being a boolean whether the assertion succeeded: `value, ok := current.Value.(*Node).value`. 

## Insert into cache

Putting something into the cache means either creating a new Node at the front of the list or updating the details of a pre-existing Node and moving that to the front of the list. If we go over the maximum number of keys then we delete the Node with the oldest last-accessed time.

The expire parameter is optional.

```go
// Set a key
func (s *Store) Set(key string, value string, expire int) {
	s.mutex.Lock()
	defer s.mutex.Unlock()
	s.set(key, value, expire)
}

// Internal set
func (s *Store) set(key string, value string, expire int) {
	current, exist := s.store[key]
	if exist != true {
		s.store[key] = s.ll.PushFront(&Node{
			key:    key,
			value:  value,
			expire: expire,
		})
		if s.max != 0 && s.ll.Len() > s.max {
			s.delete(s.ll.Remove(s.ll.Back()).(*Node).key)
		}
		return
	}
	current.Value.(*Node).value = value
	current.Value.(*Node).expire = expire
	s.ll.MoveToFront(current)
}
```

Since many other cache methods require 'set' and 'delete' functionality, there are internal `set` and `delete` methods to avoid duplication of code. Terminology note: a method is a function on an instance of an object.

When removing all keys from the cache we can lean on the garbage collector to do the hard stuff for us by removing all existing references to the objects.

```go
// Flush all keys
func (s *Store) Flush() {
	s.mutex.Lock()
	defer s.mutex.Unlock()
	s.store = make(map[string]*list.Element)
	s.ll = list.New()
}
```

The full list of methods is Set, Get, Delete, CheckAndSet, Increment, Decrement, Append, Prepend, Flush, and Stats.

<br>

This project took me two Sunday mornings and I continue to warm towards Go. It's not as terse as other languages but remains understandable. It tends to lead to vertical code as opposed to horizontal code which also aids readability. It also brings all of the benefits of a static language without requiring a lot of boilerplate.

So far, I've had a great 'Google experience' alongside my Go programming. Looking up solutions normally leads to a sensible and well-explained answer. When I'm heading in the wrong direction, I normally find out after searching rather than running into problems further down the line. But perhaps this is because the language is quite new and there are fewer results for the incorrect version!

Check out the code on [GitHub](https://github.com/healeycodes/in-memory-cache-over-http).


Cloning Memcached with Go


I’m new to Go. Over the last week, I completed [A Tour of Go](https://tour.golang.org/welcome/1) and tonight I wrote my first program, an interactive [Conway's Game of Life](https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life). I used [Ebiten](https://github.com/hajimehoshi/ebiten) which describes itself as *a dead simple 2D game library in Go*. Given its current stage of development, the [documentation](https://ebiten.org/) and [examples](https://github.com/hajimehoshi/ebiten/tree/master/examples) are amazing. It's also cross-platform 🎮.

![Conway's Game of Life GIF](conwaysgameoflife.gif)

Most of my professional experience is with dynamic languages like JavaScript and Python. When I start learning a language I like to run through the official tutorial and then get stuck-in building a game. It brings me back to my roots. I got into programming to make solo games but found out that I love writing high-performance things for the web (front or back!).

It's very rewarding to make things on your screen move, dance, and die. This is one of the reasons that learning web development is so much fun. Except nothing dies on the web (0.03% of the web still use Internet Explorer 5/6/7!).

### My Reasons for Learning Go

I love the single binary deployment of Go. It's totally opposite to Node/Python and it's a lot easier for me to share small things with my friends. People on my team at work are really excited by Go. It's fun to talk to them about the language and I know if I continue to learn they will be interested in hearing about my experience. With other people picking it up, there's a chance that we'll develop some Go microservices in the future too.

I could easily say that I'm learning Go because I like building high-performance applications for the web (which certainly excites me) but one of the main reasons is that it's new and exciting. There's an online buzz for the language similar to the Rust community. I enjoy learning and seeking that spark when something clicks. It's also well-documented and supported (e.g. StackOverflow threads), and there seem to be regular conferences and meetups near me.

### Conway's Game of Life

Conway's Game of Life is a _cellular automaton_ but that doesn't mean a lot to most people. Imagine a grid of cells that exist in two states: alive or dead. The 'animation' that you see is actually successive generations being rendered to the screen. There are four rules that help decide the next generation's state. Wikipedia describes them thusly.

1. Any live cell with fewer than two live neighbours dies, as if by underpopulation.
2. Any live cell with two or three live neighbours lives on to the next generation.
3. Any live cell with more than three live neighbours dies, as if by overpopulation.
4. Any dead cell with exactly three live neighbours becomes a live cell, as if by reproduction.

There needs to be an initial state. As we saw from the rules, an empty board will not produce cells

The `rand` package in Go _produces a deterministic sequence of values each time a program is run_. I don't want my board's default state to be the same on every load so I seed it with an ever-changing variable: time.

The following function takes a pointer to my `Game` struct which has a property called board. We edit the board that is stored in memory so we don't need to return anything.

```go
// Given an empty board, give it a random state
func giveState(g *Game) {
	rand.Seed(time.Now().UnixNano()) // <-- Different every time
	for x := 0; x < RES; x++ {
		for y := 0; y < RES; y++ {
			if rand.Intn(15) == 1 {
				g.board[x][y] = 1
			}
		}
	}
}
```

Every cell has a 1-in-15 chance of being alive during the first generation. Ebiten's update function runs at 60fps per second and I create a new generation for every tick. I print out the current generation number in the top left.

![Generation number](gennumber.png)

Conway's Game of Life seems to be a rite of passage for Computer Science students. At least, that's what I've learned from the web. It didn't come up in any of my classes so it's been fun to play around with it and read about the amazing things people have created with it. For example, a [replica of a digital clock](https://codegolf.stackexchange.com/a/111932) that has even time-steps.

### Interaction

![Interaction GIF](interaction.gif)

If you click the board, the nearby cells have their states flipped. You can drag around the screen to make the cells cascade. Animated patterns have a simple pleasure about them. Ebiten has a function for clicking and I call it on every update tick. Here we use multiple inline assignments to set the value of `x` and `y`. The type of these variables is inferred at compile time.

```go
if ebiten.IsMouseButtonPressed(ebiten.MouseButtonLeft) {
	x, y := ebiten.CursorPosition()
	interaction(x, y, g)
}
```

As low-level as Golang is, it seems to keep out of your way by default.

### Go Modules

I understand that Go's module system is a fairly recent addition. I created a mod file by running `go mod init github.com/my/repo` and then whenever I built or ran my code the dependencies were automatically updated. Changing development setup was as easy as cloning the repository and running `go install`.

I've also found VS Code's Go plugin to be incredibly useful, with it performing more helpful actions than I'm probably aware of! This has helped me focus on learning the language. For anyone looking for a way into Go, I recommend [A Tour of Go](https://tour.golang.org/welcome/1). For resources beyond that, who knows — I'm testing a few myself and welcome recommendations!

Check out the [code](https://github.com/healeycodes/conways-game-of-life) on my GitHub.


My First Golang Program


In my contract with my ISP, they gave me a range of speeds to expect but also a speed that was the guaranteed minimum. If they aren't able to keep above this minimum, I'm eligible for a discount a few times per year.

I didn't want to sit spamming speedtest.net to check when my speeds were getting low. I wanted to know if there had been dips. Were the dips at the same times? Was there a pattern? I needed data and some data visualization to know if/when I might be eligible for any discounts and whether it was worth pursuing.

## Something like this

![Bandwidth graph](bandwidth-results.png)

Running on my trusty Raspberry Pi. Powered via a USB cable from the router and connected via ethernet so the results (barring other network use) should be reliable.

![Raspberry Pi](bandwidth-pi.jpg)

## Over-designed

I started this project off way too complicated. I used a web browser automation framework called Selenium to visit Netflix's fast.com bandwidth test to check my speed. It used a headless Chromium instance.

```python
# wait for test to finish i.e. when 'your speed message' is shown
WebDriverWait(driver, 120).until(
    expected_conditions.presence_of_element_located(
        (By.CSS_SELECTOR, FAST_COMPLETE_CSS))
)
```

Not only was this quite brittle and vulnerable to CSS changes (which had to be hardcoded, see above) it also meant that any users of this project needed to get headless Chromium setup for Raspbian (the default Linux distro for Raspberry Pi). After troubleshooting this and getting it running, I realized it was over-designed.

Using this method remains an option in the project (`browsertest.py`) but it gives less accurate results on Raspberry Pis due to the browser overhead. Note: this project works on any platform but is mainly targetted towards Pis.

I discovered late in the game that speedtest.net have a great CLI library and I should have used it from the start. Here's how I get the download speed in `clitest.py`.

```python
import speedtest

# ..

def get_speed():
    """
    Use Speedtest CLI to test bandwidth speed.
        :return: Download speed in Mbps
    """
    s = speedtest.Speedtest()
    s.download()
    results_dict = s.results.dict()
    return results_dict['download'] / 1048576  # convert bits to megabits
```

This script runs via crontab and is pointed at the server. A command similar to `python clitest.py 'https://server-location/save' 'password'` runs every half an hour. The Windows alternative is Task Scheduler but I believe it's clunkier. 

## Back-end

I thought it would be neat to be able to check my speeds from anywhere so I created a Glitch project to store, receive, and host the results. It's an Express/Node project that has two API routes.

Results are sent to `/save` along with a password, which is set via an environmental variable on Glitch.

A slice of results can be read from `/read` in JSON. Here's what the Express route for that looks like.

```javascript
// get bandwidth test results for graphing here
app.get("/read", function(request, response) {
  const data = db.get("results").value();
  const prepared = data.map(s => {
    return { x: s.date, y: Number(s.speed).toFixed(3) };
  });
  const trimmed = prepared.slice(Math.max(prepared.length - 48, 1));
  response.send(trimmed); // send a slice of results
});
```

For storage, I wanted something that didn't require any setup at all. `lowdb` is a small local JSON database and it's perfect because there's only ever going to be one process reading or writing, and the write event occurs every half an hour or so. `lowdb` creates the 'database' file if it doesn't already exist.

## Quick data visualization

Chart.js is the go-to library for graphs in JavaScript and uses the Canvas API. It's batteries-included and looks nice by default (but perhaps I'm just used to the style!). It's about fifty lines, including the API call.

```javascript
fetch('/read')
    .then(response => response.json())
    .then(json => renderGraph(json));
const safeDate = time => new Date(parseInt(time)).toUTCString();
const renderGraph = (speedData) => {
    let ctx = document.getElementById('myChart').getContext('2d');
    let myChart = new Chart(ctx, {
    type: 'scatter',
    data: {
        datasets: [{
            data: speedData,
            backgroundColor: () => 'rgba(255, 99, 132, 0.2)',
            borderColor: () => 'rgba(255, 99, 132, 1)',
            borderWidth: 1,
            pointRadius: 5,
        }]
    },
    options: {
        scales: {
            xAxes: [{
                type: 'linear',
                position: 'bottom',
                ticks: {
                    userCallback: (label, index, labels) => safeDate(label)
                },
                scaleLabel: {
                    display: true,
                    labelString: 'Date'
                }
            }],
            yAxes: [{
                scaleLabel: {
                    display: true,
                    labelString: 'Mbps'
                },
                ticks: {
                    beginAtZero: true
                }
            }],
        },
        title: {
            display: true,
            text: 'Bandwidth Test Results'
        },
        legend: {
            display: false,
        },
        tooltips: {
            callbacks: {
                label: function(tooltipItem, data) {
                return `${tooltipItem.value} Mbps @ ${safeDate(tooltipItem.label)}`;
                }
            }
        }
    }
    });
}
```

I find it easy to play around with Chart.js and like JavaScript it's very productive if you want to throw something together. The docs are great and it's a large enough library that common searches will find helpful StackOverflow answers.

## Where to go from here

So far my speeds have been hovering around the guaranteed minimum, usually dipping just under around 8pm local time. At current, I have no reason to complain! A pleasant surprise.

It will be straightforward to diagnose any future bandwidth problems as I now have an up-to-date record of my speeds. Hopefully, some of this project's code will help you diagnose any simple problems with your bandwidth throughout the day.


I Built a Bot to Try and Get Money Back From My Internet Provider


I learnt about pair programming at university when we were studying Agile methodologies. At the time it seemed very full-on — almost scary. I pictured myself stumbling over my keyboard as my co-worker facepalmed behind me. In reality, pair programming sessions have helped me to evolve as a developer and I actually find it more relaxing to tackle something as a team of two.

![Some fast-paced pair programming](fast-pair-programming.gif)

> Do you mind if I drive?

_Driving_ means being in control of the mouse/keyboard. _Navigating_ is watching, and depending on the pair's style, might be dictating what is to be written or there to raise questions, problems, and muse on design. Find a more detailed break down of the roles [here](https://gist.github.com/healeycodes/5acc53131957f6a96a281c89890c7706) (first written by Jordan Poulton).

I saw a study which claimed that during a conversation we give the other person around three times longer to respond to us than we give ourselves when it is our turn to respond. I think that the same holds true while pair programming because I've felt under pressure at times. I can remember plenty of instances of myself making endless typos (that I missed until they were pointed out) and forgetting basic structures of code, worrying that the navigator was getting bored while waiting for me to finish my code-thought. However, I can't remember any of my co-workers ever doing anything of the sort. Everyone has uniformly been brilliant.

It may seem odd to the uninitiated that two developers working on one ticket can be more efficient than if they were working on two tickets separately. One of the reasons is the level of focus and that can be maintained by the driver. Someone watching you code will catch silly errors that your linter might not. This saves building and testing time. Instead of googling easy questions (and context switching) you can throw them behind you and stay in your IDE. Being able to discuss ideas helps with creative problem-solving. I find that my mind wanders less because my partner is always there to point me back to our design. Rabbit holes can be nipped in the bud — e.g. one of us saying "Let's not optimize this part yet".

For me, it's a comfy atmosphere, it's fun. I've seen it referred to as an "automatic bi-directional rubber ducky".

When debugging, the best course of action is to break the problem down to its smallest parts — removing assumptions and finding out which parts you know for sure 100% work. However, it's hard to get rid of your own assumptions and later on you'll find out that something you held to be true was, in fact, the path to the solution. While pair programming, it's easier to approach the problem with more logic as there is a second pair of eyes that must be convinced of every assumption being held.

Pairing is good for when:

- You're stuck!
- You're new and want mentoring
- The team needs shared knowledge
- There is one time-sensitive feature

### Some tickets don't require it

I've read about peoples' experiences at companies where _every line of code_ is pair programmed. I'm curious about the benefit of such a requirement. I imagine that the [bus factor](https://en.wikipedia.org/wiki/Bus_factor) of a team would be incredibly low. But I'm thinking about the stories and bugs where I'm working in parts of the codebase that I know well. Where most of the work is setting up boilerplate code and some tests.

My favourite setup that I've experienced so far is a team where pair programming was left up to the developers. If we felt that a ticket required pair programming to be efficient then all we had to do was let our manager know that was the plan.

### As a mentor

Over the past year, I've been teaching my wife how to code. The largest jumps in her knowledge are when we've tackled a feature or a project as a pair-team. It gives me a chance to observe how she goes about tackling a problem and lets me guide her down the right path (or at least close to it). While I'm driving, the most helpful things she's reported watching are the steps that aren't coding but are more knowledge-based e.g. setting up a dev environment, proper version control, different test suites and CI/CD integrations.

### N.b.

This reads a little like propaganda from the Pair Programming Institute (tm). Some people don't mix well with pair programming and may find it distressing and inefficient. And there's nothing wrong with that at all!

My Experience with Pair Programming


## What's a cache?

A cache stores data so that future requests can receive it faster. Generating a component or view in a dynamic web application can be costly — especially when database calls are involved. Saving the computed result to a cache means that serving the next request for the same data will be much more efficient.

Web applications can use a cache running on the same web server, a remote cache, or a distributed system. Cloud service companies offer APIs for highly scalable distributed caching. In simple, single-threaded applications you could even use an object as a cache! I've seen this used in tiny Node.js services.

## Okay, give me an example

You may have heard of Memcached or Redis. They both allow key/value storage as well as storing data in other data-structures. Here's a high-level example of cache usage. Let's say our web application receives a request at `/books/312` where 312 is the id of a book entity. We want to return a dynamic page with details about this book and perhaps some reviews that people have left for the book.

In pseudocode, applying a cache to this situation looks like this:

```text
do we have key '/books/312' in our cache?
if so:
    return that value as a response
else:
    generate a page
    store page in cache at '/books/321/'
    return page as a response
```

In this example, getting the details of book 321 involves a call to our database. Grabbing recent reviews may involve a separate database call too. We then might generate the page through a server-side template. All of this computation can be avoided when a user visits the same page soon after this original request.

Database calls should be avoided when we can find the correct information in a different way. Local caches operate with sub-millisecond response time. Caches also remove the last used item when they fill up. Overall, they're pretty hardy but setting a sensible timeout value is important.

Memcached can be deployed using distributed servers that don't communicate with each other. Only the client (our web app) knows about the cache servers. Much like a hash table operates, a key is hashed and a server is chosen to get/set the key.

Here's a high-level example of that:


```text
call a hash function on 'books/321'
turn the hashed value into an index
check the corresponding cache server
get or set 'books/321/' from this server
```

The amount of cache servers that the web app knows about is unlimited. It scales horizontally — which means more servers can be added to deal with higher loads.

Here are some issues you might face when using a caching system:

- Users receive old data
- Caches fill up too fast
- Requests are too unique to be cached
- Race conditions

## A race condition?

Here's an example of a race condition involving a cache. We want to display an old school hit counter on our website. When a user requests the page, we'll check what the current hit count is, increase it by one, and overwrite the key.

Ideally, it works like this:

```text
get 'hit_count' from cache
increase by one
save new 'hit_count' to cache
```

If our web app is multithreaded or distributed in some way, the cache can be hit twice at the same time. Both server instances increase the hit count they receive back from the cache. Both server instances then save that value back. From the cache's perspective, a key is requested twice and then saved twice. Both instances of the web app request the current hit value, `10`, add one so that they both have `11` and then they save that to the cache. The true hit count is `12` and some data has been lost!

This problem is actually solved by Memcached which offers increment and decrement functions. For more complex situations where an atomic operation is required, Memcached has a Compare-And-Set feature. Where you use a retry-loop attempting to perform an operation without losing any data along the way.

Here's a [great blog post](http://neopythonic.blogspot.com/2011/08/compare-and-set-in-memcache.html) covering CAS in more detail.

## Give me examples in code

Let's take a look at some examples of this in real code. We'll be using Flask and Django, the most popular Python web frameworks. Both come with robust out-of-the-box solutions.

Flask is built on Werkzeug which means we can use APIs from `werkzeug.contrib.cache` without including any extra modules. Flask has a page on caching patterns [right here](http://flask.pocoo.org/docs/1.0/patterns/caching/).

Here's some code from a prototype app I wrote recently with added comments

```python
# SimpleCache is a dev tool and is not threadsafe
# but it's very easy to swap in MemcachedCache or GAEMemcachedCache
from werkzeug.contrib.cache import SimpleCache

def create_app():
    app = Flask(__name__)
    # this line could be `cache = MemcachedCache(['127.0.0.1:11211'])`
    cache = SimpleCache()

    @app.route('/nearby/<postcode>/<radius>')
    def nearby_stores(postcode, radius):
        # have we performed this calculation for these arguments?
        if cache.get('{}{}'.format(postcode, radius)) is not None:
            return cache.get('{}{}'.format(postcode, radius))

        # otherwise do some calculations
        # ..

        # store for next time we run with these arguments
        cache.set('{}{}'.format(postcode, radius), jsonified)
        return jsonified
```

A cache check in this app can save processing, a database call, and one third party request! However, using a cache module from Werkzeug does add some unnecessary lines that an extension like `Flask-Cache` gets rid of. Instead of manually checking the cache and setting the cache, we can use a decorator which uses `request.path` (which is configurable).

So the start of that function would look like this:

```python
    @app.route('/nearby/<postcode>/<radius>')
    @cache.cached(timeout=120)
    def nearby_stores(postcode, radius):
```

# What about Django?

Like Flask, Django has a [fantastic page](https://docs.djangoproject.com/en/2.2/topics/cache/) on setting up caching for your application. Its cache framework is very similar to the cases we've just seen. The largest difference will be the initial configuration. Django can also save your cached data in the database you're using. The configuration is more mature than Flask as well.

Here's an example from their docs:

```python
from django.views.decorators.cache import cache_page

@cache_page(60 * 15)
def my_view(request):
```

# The hardest part

Structuring your application and figuring out which computations can be saved and used at a later time is the hardest part! Caching web requests for simple web applications is a fairly solved problem. How you wield your caching sword is up to you.

An Introduction to Caching on the Web, With Examples in Python


When all other resources fail me, technical blogs come to the rescue. They provide insight into my problem, clues about the design of a potential solution, and sometimes humor or even an emotional connection. I collect new tech blogs. I devour them.

I write to provide that same benefit, (I try) to help other engineers and to promote free and open software (FLOSS — [Free/Libre and Open Source Software](https://www.gnu.org/philosophy/floss-and-foss.en.html)). I go back and correct errors, add asides, links, and errata. I work on new content but improve the old stuff too.

Aside from the innumerable benefits, open-source software is vital for a long-lasting website. Closed software will always come to an end. The EOL date for Adobe Flash is 2020 — software that was used on [28.5% of all websites in 2011](https://opensource.com/alternatives/flash-media-player). Its demise has been met by cheers but people are grappling with the prospect of [lost content](https://github.com/open-source-flash/open-source-flash).

Let's say I plan to revamp my [personal website](https://healeycodes.com/) into a brief footprint on the sands of time. What can I do to extend the shelf-life of the second iteration of its production system? What web standards are sure to be kicking as the century turns over and [The Clock of the Long Now](https://en.wikipedia.org/wiki/Clock_of_the_Long_Now) ticks again.

## Static vs dynamic

To define, I'll say that a [Jekyll](https://jekyllrb.com/) website is static and a Node.js/Express program with database calls is an example of dynamic. A static site generator takes some files, usually text-based, and builds a website into a folder which can be dispersed by any web server. These generators are smart, they build interactive nav bars, minify all of your files, some even make API calls to get up-to-date information.

[Gatsby](https://www.gatsbyjs.org/) is a static site generator that takes advantage of modern browser features. It uses progressive/responsive image loading, offline access, and [provides](https://www.gatsbyjs.org/features/) a React-based design system with a bustling eco-system surrounding it. But if I have some content that I want to provide to the world for a century, Gatsby seems like overcomplicating the problem. It adds many dependencies and is working on [accessibility](https://www.gatsbyjs.org/blog/2019-04-18-gatsby-commitment-to-accessibility/) where other generators are accessible by default because they're flat sites.

At the very worst, a static site generator will generate HTML files which can simply be read by the human eye. They can be archived and left. Content management systems like Blogspot or SquareSpace generate a pile of JavaScript which pulls your files from a propriety location, likely without a published spec.

## Markdown

Markdown sounds like the tool for the job (but only if a spec is included). Since its [invention](https://daringfireball.net/projects/markdown/) Markdown has empowered READMEs, blogs, forums, and more. I use it to drive a number of things including this article. However, it exists under a kind of living tree doctrine. Services add their own features and interpretations (which is a good thing) but for our century-long plan, we'll need a document that explains how the markings should be interpreted to display the content in its originally planned manner.

> I think of the oak beams in the ceiling of College Hall at New College, Oxford. Last century, when the beams needed replacing, carpenters used oak trees that had been planted in 1386 when the dining hall was first built. The 14th-century builder had planted the trees in anticipation of the time, hundreds of years in the future, when the beams would need replacing. Did the carpenters plant new trees to replace the beams again a few hundred years from now?
<small> — WIRED SCENARIOS: THE MILLENNIUM CLOCK DANNY HILLIS</small>

## Maintenance

An ideal software stack might have one dependency, a run-time from a too-big-to-fail programming language. Perhaps a static site generator written in [pure Python](https://github.com/lepture/mistune). Maybe even that is overcomplicating it. Some blogs that I enjoy are created via [Makefile](https://github.com/mitsuhiko/lucumr)! The [Lindy effect](https://en.wikipedia.org/wiki/Lindy_effect) is a theory that means _the life expectancy of technology is proportional to its current age_. I would like to avoid revamping every five years ⁠— or sixteen times in this coming century. Maybe the ideal technology is in fact from a decade ago and is not present in my lexicon.

Dead links are a problem but a link crawler can be run on a cron job. I use a similar method and go back to correct links to resources that have moved — or I use an alternative resource. If a source disappears entirely, I'll write a footnote. I know some writers back up all of their sources to create a self-contained web — which is a noble, librarian-like goal.

Continuous improvement of the lightweight design _and_ the content will be necessary. Or we could just write as professors do, take Peter Norvig's article on [solving sudoku with Python](https://norvig.com/sudoku.html). It's written like an RFC spec and apart from the two graphs is replicable by a century-old typewriter.

## Content that mellows

Through repetitive practice, my writing has and will improve over time. I don't want to leave my old articles to rot. Posts about specific technologies can be locked (with minor trimming) but there are others which aim to be timeless. I wonder about a schedule where content is re-edited and parts re-written every X months. Performing a quick readthrough of many of my old articles leads to me dropping a resource here and an extra example there. Evergreen.

It's at this point that proper version control becomes important as changing an article can change the meaning, which can invalidate an offsite citation. Providing access to different versions of articles sounds non-trivial for a static site generator. It sounds like Wikipedia's [system](https://en.wikipedia.org/wiki/Help:Page_history)!

If you are interested in solving any of these problems with me you should [contact me](mailto:healeycodes@gmail.com)!

## Tech blog recommendations

- [Julia Evans](https://jvns.ca/)
- [Peter Norvig](https://norvig.com/)
- [Drew DeVault](https://drewdevault.com/)
- [Alice Goldfuss](https://blog.alicegoldfuss.com/)
- [Tristan Hume](http://thume.ca/archive.html)


What If I Want My Website to Last for 100 Years?


Genius [suspected](https://bgr.com/2019/06/17/genius-vs-google-lyrics-results-on-search-copied-from-lyrics-site/) their lyrics were being scraped by Google but they needed a way to prove it. So they watermarked some of their lyrics with a message in Morse code. It read __Red Handed__. It was encoded using different types of apostrophes. I suspect they used _U+0027 APOSTROPHE_ and _U+2019 RIGHT SINGLE QUOTATION MARK_ since those are the most commonly used but I can't be sure.

Let's write up a solution to this problem: encoding a message in apostrophes via Morse code. We'll assume that just those two apostrophes are available. I'm choosing Python because it's nice for string manipulation.

The important parts: our two markers, and the alphabet in Morse.

```python
APOSTROPHE = '\''
RIGHT_MARK = '’'

morse = [".-", "-...", "-.-.", "-..", ".", "..-.", "--.", "....",
         "..", ".---", "-.-", ".-..", "--", "-.", "---", ".--.",
         "--.-", ".-.", "...", "-", "..-", "...-", ".--", "-..-",
         "-.--", "--.."]
```

First, we need a way of converting text to Morse code. `'cat'` becomes `'-.-..--'` (which in a way looks like a sleeping cat!).

```python
# flatten a 2D list
flatten = lambda l: [item for sublist in l for item in sublist]

# :text: a message made up of [a-zA-Z]
def to_morse(message):
    m = [morse[ord(c)-97] for c in message.lower()]
    return flatten(m)
```

I've chosen _APOSTROPHE_ to represent the dot in Morse as it has a lower Unicode value. _RIGHT SINGLE QUOTATION MARK_ represents a dash. Once we have our message in Morse, we convert it to an apostrophe message using substitution. We then iterate over the source text until we hit an apostrophe and exchange it for the next character in our message. Take a look.

```python
# :message: a message to encode
# :source: a source text with apostrophes
def insert_morse(message, source):
    # a list of dots and dashes
    morse = to_morse(message)
    # a list of the two types of apostrophes
    secret = [APOSTROPHE if m == '.' else RIGHT_MARK for m in morse]
    inserted = []
    for c in source:
        if len(secret) > 0 and (c == APOSTROPHE or c == RIGHT_MARK):
            inserted.append(secret.pop(0))
        else:
            inserted.append(c)
    return ''.join(inserted)
```

If you're wondering how to decode a source text that has such a message encoded within it's actually far simpler.

```python
# :source: a source encoded via insert_morse
def retrieve_morse(source):
    morse = []
    for c in source:
        if c == APOSTROPHE:
            morse.append('.')
        elif c == RIGHT_MARK:
            morse.append('-')
    return ''.join(morse)
```

Let's look at an example of this in action.

```python
source_text = 'Hello \'\' World, \'\' no \'\' secrets \' here!'
secret = 'cat'

encoded_message = insert_morse(secret, source_text)
retrieved_message = retrieve_morse(encoded_message)

print(encoded_message) # Hello ’' World, ’' no '’ secrets ’ here!
print(retrieved_message) # -.-..--
```

Pretty neat. If you thought that was cool, wait until you find out how people have been using [zero-width spaces](https://en.wikipedia.org/wiki/Zero-width_space) to get up to more advanced fingerprinting trickery.

The exercise of decoding __and__ understanding a Morse message without spaces is left up to the reader! I'd love to hear your thoughts and resources on hiding data in plain sight on the web.

Grab the code from this article [on GitHub](https://github.com/healeycodes/hidden-in-apostrophes).


Using character codes to hide in plain sight.

A Tutorial on Tricking Google: Encoding Messages in Apostrophes


_"You should contribute to open-source." ⁠— The Internet._

Maybe you've seen this advice. Either from those who believe in free software — or as a suggestion to aid job seeking. Many people (some of my mentees included) report that GitHub is scary. The projects on there are mammoth-sized compared to any team project they've worked on. The people working on these projects are, like, _serious people_.

This is true and this is not true. There really isn't anything to worry about. Let's talk about what's involved. I'll be using my _tiny_ [contribution](https://github.com/jekyll/minima/pull/379) I made to Jekyll yesterday as an example. There are [lots](https://www.firsttimersonly.com/) of resources for finding first-timer issues but there are a few steps that these guides miss out on.

## Find a problem

It's going to be easier to find (or remember) a bug in the software that you use every day compared to getting up to speed with a project you haven't heard of before. Let's take Jekyll's minima theme for example. Way back, I wanted my website to look different but I didn't have the time to write a theme from scratch — so I've been [hacking away](https://healeycodes.com/) at this theme for a while now.

This is how I found the bug. I went into developer tools (F12) and set the view-size to be _iPhone 5/SE_, which is relatively small by modern standards, to test a page of my website. Some developers won't be considering devices of this size (but they should). I saw that the responsive footer wasn't appearing correctly. Instead of two stacked columns, there were two columns and one was empty.

![Minima broken columns](minbroke.png)

At first, I presumed that some of my hacking must have caused this. So I went over to the [jekyll/minima](https://jekyll.github.io/minima/) repository to double check the live demo page. I found the same problem (note: this problem is now fixed on master). The problem was an erroneous class — `.one-half` which was not behaving responsively. (I discovered this through some Dev Tools tinkering).

It needed a [media query](https://developer.mozilla.org/en-US/docs/Web/CSS/Media_Queries/Using_media_queries). Media queries are how our websites change styling depending on the device's settings, e.g. resolution. __You don't need to understand the whole tech stack behind the project you're fixing__. For example, I haven't used Sass that much but I knew about its CSS variable system. I looked around for a breakpoint variable that was already being used and which complimented the footer columns at all sizes. I settled on `$on-large` — resulting in a fix to the problem.

![Minima fixed columns](minfixed.png)

## Fork the branch

Check the contributing guidelines. This is more important than any advice I can give you. For jekyll/minima, the [guidelines](https://github.com/jekyll/minima#contributing) were pretty simple: "Bug reports and pull requests are welcome," as well as a link to the code of conduct. It should also tell you which branch to send your changes to.

jekyll/minima is developed on `master` but usually there will be a staging branch for you to fork. Head to the GitHub repository in your browser and click the _Fork_ button, then clone the branch to your computer. The community recommends learning the command line tools for git interaction but the [GitHub Desktop](https://desktop.github.com) client works fine too. If there's a test suite, run the tests before changing anything to ensure that your environment is stable.

## Create a pull request

Make your changes and [write a good commit message](https://github.com/erlang/otp/wiki/writing-good-commit-messages). Mine was "Fix footer stacking issue for smaller view sizes". Push the changes to your fork. There are ways to create a pull request via command line but I visit my fork's repository and hit the _Pull request_ button at the top right of the file list.

I've always thought that writing a pull request is just as important as the code. If your team members can't understand the proposed change at a high level then they won't be able to critique it successfully. Meaning that your pull request could be denied (or accepted when it shouldn't be). For bugs, I describe the bug, include an image if appropriate, then explain what changes are required to fix it. If there are important, tech-y implementation details then I place them at the very end.

Perhaps a final look over the contributing guidelines (check `CONTRIBUTING.md` if you can't find it) and then it's time to submit. Check how often pull requests are being read, commented on, and merged. Make sure you understand how long it may take for a response so your expectations are met. (For my [small fix](https://github.com/jekyll/minima/pull/379) it was less than a day).

## Handling criticism

You may want to tick _Allow edits from maintainers_ to make small stylistic changes easier to happen. A GitHub user thanked me for my change and saw that there were further fixes in the area I was working in. I had a look but couldn't come up with a clean solution to the extra fixes. For the extra fixes, a small architectural rejig was required and I didn't have time to further study the code base. Basically, it's okay to say no — we're all volunteering our time here.

Separate yourself from your pull request. Your code does not represent _you_. For example, you may not be familiar with design patterns used elsewhere in the project. The worst thing that will happen is someone advising you to try again. You'll be shocked at how helpful and welcoming people are in the open-source community.

We want you to succeed! If you're experiencing pull request anxiety, throw up a one-line change on one of my [toy projects](https://github.com/healeycodes). I'll greet ya with a smile 😊.


Making a small change to the Jekyll project.

Your First Open-Source Pull Request: a Walkthrough


Recently, I wrote an article about [how to write very fast JavaScript](/javascript/node/discuss/tutorial/2019/05/19/high-performance-javascript.html). Some of the examples took it to the extreme and became very quick at the cost of being totally unmaintainable. There's a middle ground between speed and comprehension and that's where _good code_ lives.

We're going to refactor some pieces of code based off real examples that I've come across. Sometimes I'll need to perform this kind of refactoring on my own code before submitting a PR. Other times I'll do a small refactor of existing code at the start of a [story](https://en.wikipedia.org/wiki/User_story) or bug to make my changes easier to implement.

## Scenario 1

We're an URL-shortening website, like TinyURL. We accept a long URL and return a short URL that forwards visitors to the long URL. We have two functions.

```javascript
// Unrefactored code

const URLstore = [];

function makeShort(URL) {
  const rndName = Math.random().toString(36).substring(2);
  URLstore.push({[rndName]: URL});
  return rndName;
}

function getLong(shortURL) {
  for (let i = 0; i < URLstore.length; i++) {
    if (URLstore[i].hasOwnProperty(shortURL) !== false) {
      return URLstore[i][shortURL];
    }
  }
}
```

Problem: what happens if `getLong` is called with a short URL that isn't in the store? Nothing is explicitly returned so `undefined` will be returned. Since we're not sure how we're going to handle that, let's be explicit and throw an error so that problems can be spotted during development.

Performance-wise, be careful if you're iterating through a flat array very often, especially if it's a core piece of your program. The refactor here is to change the data-structure of `URLstore`.

Currently, each URL object is stored in an array. We'll visualize this as a row of buckets. When we want to convert short to long, on average we need to check half of those buckets before we find the correct short URL. What if we have thousands of buckets, and we perform this hundreds of times a second?

The answer is to use some form of [hash function](https://en.wikipedia.org/wiki/Hash_function), which Maps and Sets [use under the surface](https://v8.dev/blog/hash-code). _A hash function is used to map a given key to a location in the hash table_. Below, this happens when we place our short URL into the store in `makeShort` and when we get it back out in `getLong`. Depending on how you're [measuring run time](https://en.wikipedia.org/wiki/Big_O_notation), the result is that _on average_ we only need to check one bucket — no matter how many total buckets there are!

```javascript
// Refactored code

const URLstore = new Map(); // Change this to a Map

function makeShort(URL) {
  const rndName = Math.random().toString(36).substring(2);
  // Place the short URL into the Map as the key with the long URL as the value
  URLstore.set(rndName, URL);
  return rndName;
}

function getLong(shortURL) {
  // Leave the function early to avoid an unnecessary else statement
  if (URLstore.has(shortURL) === false) {
    throw 'Not in URLstore!';
  }
  return URLstore.get(shortURL); // Get the long URL out of the Map
}
```

For those examples, we assumed that the random function wouldn't clash. 'Cloning TinyURL' is a common system design question and a very interesting one at that. What if the random function _does_ clash? It's easy to tack on addendums about scaling and redundancy.

## Scenario 2

We're a social media website where user URLs are generated randomly. Instead of random gibberish, we're going to use the `friendly-words` package that the Glitch team works on. They use this to generate the random names for your recently created projects!

```javascript
// Unrefactored code

const friendlyWords = require('friendly-words');

function randomPredicate() {
  const choice = Math.floor(Math.random() * friendlyWords.predicates.length);
  return friendlyWords.predicates[choice];
}

function randomObject() {
  const choice = Math.floor(Math.random() * friendlyWords.objects.length);
  return friendlyWords.objects[choice];
}

async function createUser(email) {
  const user = { email: email };
  user.url = randomPredicate() + randomObject() + randomObject();
  await db.insert(user, 'Users')
  sendWelcomeEmail(user);
}
```

It's often said that a function should do one thing. Here, `createUser` does one thing .. kinda. It creates a user. However, if we're thinking ahead to the future, there's a good chance (if our business is successful) that this function is going to grow very large indeed. So let's start early by breaking it up.

You may have also noticed that there is some duplicated logic in our random functions. The `friendly-worlds` package also offers lists for 'teams' and 'collections'. We can't go around writing functions for every option. Let's write **one** function that accepts a list of friendly things.

```javascript
// Refactored code

const friendlyWords = require('friendly-words');

const generateURL = user => {
  const pick = arr => arr[Math.floor(Math.random() * arr.length)];
  user.url = `${pick(friendlyWords.predicates)}-${pick(friendlyWords.objects)}` +
    `-${pick(friendlyWords.objects)}`; // This line would've been too long for linters!
};

async function createUser(email) {
  const user = { email: email };
  // The URL-creation algorithm isn't important to this function so let's abstract it away
  generateURL(user);
  await db.insert(user, 'Users')
  sendWelcomeEmail(user);
}
```

We separated some logic and reduced the number of lines of code. We inlined a function called `pick` that accepts an array of length 1 and up and returns a random choice, then we used a [template literal](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals) to build an URL.

## Strategies

Here are some straightforward to implement methods that can lead to easier to read code. There are no absolutes when it comes to clean code — there's always an edge case!

Return early from functions:

```javascript
function showProfile(user) {
  if (user.authenticated === true) {
    // ..
  }
}

// Refactor into ->

function showProfile(user) {
  // People often inline such checks
  if (user.authenticated === false) { return; }
  // Stay at the function indentation level, plus less brackets
}
```

Cache variables so functions can be read like sentences:

```javascript
function searchGroups(name) {
  for (let i = 0; i < continents.length; i++) {
    for (let j = 0; j < continents[i].length; j++) {
      for (let k = 0; k < continents[i][j].tags.length; k++) {
        if (continents[i][j].tags[k] === name) {
          return continents[i][j].id;
        }
      }
    }
  }
}

// Refactor into ->

function searchGroups(name) {
  for (let i = 0; i < continents.length; i++) {
    const group = continents[i]; // This code becomes self-documenting
    for (let j = 0; j < group.length; j++) {
      const tags = group[j].tags;
      for (let k = 0; k < tags.length; k++) {
        if (tags[k] === name) {
          return group[j].id; // The core of this nasty loop is clearer to read
        }
      }
    }
  }
}
```

Check for Web APIs before implementing your own functionality:

```javascript
function cacheBust(url) {
  return url.includes('?') === true ?
    `${url}&time=${Date.now()}` :
    `${url}?time=${Date.now()}`
}

// Refactor into ->

function cacheBust(url) {
  // This throws an error on invalid URL which stops undefined behaviour
  const urlObj = new URL(url);
  urlObj.searchParams.append('time', Date.now); // Easier to skim read
  return url.toString();
}
```

It's important to get your code right the first time because in many businesses there isn't much value in refactoring. Or at least, it's hard to convince stakeholders that eventually uncared for codebases will grind productivity to a halt.


Refactoring JavaScript for Performance and Readability (with Examples!)


You've just written a great piece of JavaScript. But when the running process stops, or the user refreshes, all of that nice data **disappears into the ether**.

Is this you?

When prototyping, or otherwise working on tiny projects, it can be helpful to manage some state without resorting to a database solution that wasn't designed for that creative itch you're trying to scratch.

We're going to explore some options that I wish I knew about when I started tinkering on the web. We'll look at JavaScript in the browser and Node.js on the back end. We'll also look at some lightweight databases that use the local file system.

## Node.js

First up is JSON serializing your data and saving it to disk. The MDN Docs have a great [article](https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Objects/JSON) if you haven't worked with JSON before.

```javascript
const fs = require('fs');

const users = {
    'Bob': {
        age: 25,
        language: 'Python'
    },
    'Alice': {
        age: 36,
        language: 'Haskell'
    }
}

fs.writeFile('users.json', JSON.stringify(users), (err) => {  
    // Catch this!
    if (err) throw err;

    console.log('Users saved!');
});
```

We created our users object, converted it to JSON with [JSON#stringify](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify) and called [fs#writeFile](https://nodejs.org/api/fs.html#fs_fs_writefile_file_data_options_callback). We passed in a filename, our serialized data, and an arrow function as a callback to execute when the write operation finishes. Your program will continue executing code in the meanwhile.

You can also use this method to write normal serialized data by passing in anything that can be cast to a string. If you're storing text data, you may find [fs#appendFile](https://nodejs.org/api/fs.html#fs_fs_appendfile_path_data_options_callback) useful. It uses an almost identical API but sends the data to the end of the file, keeping the existing contents.

There is a synchronous option, [fs#writeFileSync](https://nodejs.org/api/fs.html#fs_fs_writefilesync_file_data_options) but it isn't recommended as your program will be unresponsive until the write operation finishes. In JavaScript, you should aim to [Never Block](https://developer.mozilla.org/en-US/docs/Web/JavaScript/EventLoop#Never_blocking).

If you're dealing with CSV files, reach for the battle-hardened [node-csv](https://github.com/adaltas/node-csv) project.

Let's load those users back into our program with [fs#readFile](https://nodejs.org/api/fs.html#fs_fs_readfile_path_options_callback).

```javascript
fs.readFile('users.json', (err, data) => {
    // Catch this!
    if (err) throw err;
  
    const loadedUsers = JSON.parse(data);
    console.log(loadedUsers);
});
```

### Lightweight databases

SQLite uses a local file as a database — and is one of my favorite pieces of software in the world. It enables many of my smaller projects to exist with low maintenance and little deploying hassle.

Here are some facts about SQLite:
- The project has [711 times](https://www.sqlite.org/testing.html) as much test code and test scripts compared to other code.
- The developers pledge to keep it backward compatible through at least the year 2050.
- It's used on planes, in Android, and you probably interacted with it in [some way](https://www.sqlite.org/famous.html) on your way to this article today.

Seriously, [How SQLite Is Tested](https://www.sqlite.org/testing.html) is a wild ride.

In Node.js, we commonly use the `sqlite3` [npm package](https://www.npmjs.com/package/sqlite3). I'll be using some code from Glitch's `hello-sqlite` template, which you can [play around with and remix](https://glitch.com/edit/#!/remix/hello-sqlite) without an account.

```javascript
// hello-sqlite
let fs = require('fs');
let dbFile = './.data/sqlite.db'; // Our database file
let exists = fs.existsSync(dbFile); // Sync is okay since we're booting up
let sqlite3 = require('sqlite3').verbose(); // For long stack traces
let db = new sqlite3.Database(dbFile);
```

Through this `db` object, we can interact with our local database like we would through a connection to an outside database.

We can create tables.

```javascript
db.run('CREATE TABLE Dreams (dream TEXT)');
```

Insert data (with error handling).

```javascript
db.run('INSERT INTO Dreams (dream) VALUES (?)', ['Well tested code'], function(err) {
  if (err) {
    console.error(err);
  } else {
    console.log('Dream saved!');
    }
});
```

Select that data back.

```javascript
db.all('SELECT * from Dreams', function(err, rows) {
  console.log(JSON.stringify(rows));
});
```

You may want to consider [serializing some of your database queries](https://stackoverflow.com/a/42946402). _Each command inside the serialize() function is guaranteed to finish executing before the next one starts._ The sqlite3 [documentation](https://github.com/mapbox/node-sqlite3/wiki) is expansive. Keep an eye on the SQLite [Data Types](https://www.sqlite.org/datatype3.html) as they can be a little different to other databases.

If even SQLite seems like too much overhead for your project, consider [lowdb](https://github.com/typicode/lowdb) (also [remixable on Glitch](https://glitch.com/edit/#!/low-db)). lowdb is exciting because it's _a small local JSON database powered by Lodash (supports Node, Electron and **the browser**)_. Not only does it work as a wrapper for JSON files on the back end it also provides an API which wraps localStorage in the browser.

From their [examples](https://github.com/typicode/lowdb/tree/master/examples#browser):

```javascript
import low from 'lowdb'
import LocalStorage from 'lowdb/adapters/LocalStorage'

const adapter = new LocalStorage('db')
const db = low(adapter)

db.defaults({ posts: [] })
  .write()

// Data is automatically saved to localStorage
db.get('posts')
  .push({ title: 'lowdb' })
  .write()
```

## Browser

This brings us to the front end. [window#localStorage](https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage) is the modern solution to storing data in HTTP cookies — which MDN [doesn't recommend](https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Client-side_web_APIs/Client-side_storage#Old_fashioned_cookies) for storing things anymore.

Let's interact with them right now. If you're on desktop, open your dev console (F12 on Chrome) and see what DEV is storing for you:

```javascript
for (const thing in localStorage) {
  console.log(thing, localStorage.getItem(thing))
}

// Example of one thing:
// pusherTransportTLS {"timestamp":1559581571665,"transport":"ws","latency":543}
```

We saw how lowdb interacted with localStorage but for our small projects it's probably easier to talk to the API directly. Like this:

```javascript
// As a script, or in console
localStorage.setItem('Author', 'Andrew') // returns undefined
localStorage.getItem('Author') // returns "Andrew"
localStorage.getItem('Unset key') // returns null
```

It gets easier still: you can treat it like an object. Although, MDN recommends the API over this [shortcut](https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API/Using_the_Web_Storage_API#Basic_concepts).

```javascript
console.log(localStorage['Author']); // prints "Andrew"
```

If you don't want to store data on the user's computer forever (which can be cleared with `localStorage.clear()` **but don't run this on DEV**) you may be interested in [sessionStorage](https://developer.mozilla.org/en-US/docs/Web/API/Window/sessionStorage) which has a near identical API and only stores data while the user is on the page.

## End notes

I read somewhere that SQLite is used onboard the Internation Space Station in some capacity but I haven't been able to find a source. My fiancée wants you to know that SQLite _is_ a database and the title of this post is incorrect.


The title is a lie I also show how to use SQLite.

Saving Data in JavaScript Without a Database


I saw this question doing the rounds on social media. [Apparently](https://www.reddit.com/r/javascript/comments/3rb88w/ten_questions_ive_been_asked_most_more_than_once/cwmn60t/), [top companies](https://www.glassdoor.com/Interview/-1-Implement-a-loading-bar-that-animates-from-0-to-100-in-3-seconds-2-Start-loading-bar-animation-upon-a-button-cli-QTN_1393903.htm) are using it to screen front end engineers. My fiancée will be applying to jobs soon and I asked her to give it a go. She almost completed the final stage (with a little research) but a tricky recursion bug tripped her up. I wrote this article to help her out. Hopefully, you'll find this useful should you face similar themed questions!

There are a few variations but this question is normally asked in stages which get progressively harder.

#### 1. Implement a loading bar that animates from 0 to 100% in 3 seconds

This can be done purely with CSS. If something can be done purely with CSS I tend to go for that option. My rationale is that it's easier to refactor something that is pure CSS than trying to extend a quick JavaScript hack. CSS is very declarative, it's more straightforward to read and understand what's going on under the hood.

For a CSS only progress bar, I'm going to use two divs — a container and the progress bar — and `keyframes`. The important line here is `animation: 1s linear fill;`. There's a lot to talk about. What `transition-timing-function` are we going to use — `ease`, `linear`, a `cubic-bezier`?

At the very least, this quick answer shows that you know what `keyframes` and you can use it on a basic level.

```html
<div class="container">
  <div class="progress-bar"></div>
</div>
```

```css
.container {
  width: 300px;
  height: 50px;
  background-color: #D3D3D3;
}

.progress-bar {
  width: 100%;
  height: 50px;
  background-color: #90EE90;
  animation: 3s linear fill;
}

@keyframes fill {
    0% {
        width: 0%;
    }
    100% {
        width: 100%;
    }
}
```

__[CodePen](https://codepen.io/healeycodes/pen/pmKQzM)__

#### 2. Start loading bar animation upon a button click

Now we're moving into JavaScript land. I've always thought that `transition` was neat as heck so I'm going to use that with JavaScript to add a class to the `progress-bar` element. It enables you to "define the transition between two states of an element" [(MDN)](https://developer.mozilla.org/en-US/docs/Web/CSS/transition).

I'm making sure to cache the reference to the loading bar. I work on Smart TVs a lot and caching is one of the many tricks we use to keep everything snappy. HTML elements have [Element#classList](https://developer.mozilla.org/en-US/docs/Web/API/Element/classList) which is a fantastic API for interacting with classes. It's very safe to use. You can add multiple classes and it will only add one instance, you can also remove classes that don't exist without any errors.

[classList#toggle](https://developer.mozilla.org/en-US/docs/Web/API/Element/classList#Methods) is particularly useful. "When only one argument is present: Toggle the class value; i.e., if the class exists then remove it and return false, if not, then add it and return true" [(MDN)](https://developer.mozilla.org/en-US/docs/Web/API/Element/classList#Methods).

```html
<div class="container">
  <div class="progress-bar"></div>
</div>
<button onclick="loadBar()">Load</button>
```

```css
.container {
  width: 300px;
  height: 50px;
  background-color: #D3D3D3;
}

.progress-bar {
  width: 0%;
  height: 50px;
  background-color: #90EE90;
  transition: width 3s linear;
}

.load {
  width: 100%;
}
```

```javascript
const bar = document.querySelector('.progress-bar');

function loadBar () {
  bar.classList.add('load');
}
```

__[CodePen](https://codepen.io/healeycodes/pen/MdXzqb)__

#### 3. Queue multiple loading bars if the button is clicked more than once. Loading bar N starts animating with loading bar N-1 is done animating.

Here it gets more interesting. There is a way to iterate on our previous solution by removing and adding classes but that feels hacky. I think the intention is for you to use more JavaScript here. Technical interview questions don't really have a finish point. There's always restrictions, extensions, and what-ifs that can be thrown around. I'd love to see what you come up with before reading any further 👀.

There are a couple of traps here. You've got to make sure the time taken is exactly three seconds rather than a tick more or a tick less. And what is a good length for a tick? I suppose it depends on how wide your bar is. Going up by a percent each time seems like a sweet spot. Note: it's also better to manage your own state rather than relying on the DOM. 

Caching is really important here. You don't want to traverse the DOM for an element 33 times a second. Two functions are probably required for clean code. I went for a recursive timeout with a global flag to track whether the progress bar is running. We want to add one to the queue either way but we don't want two going at once or our bar will load twice as quick!

```html
<div class="container">
  <div class="progress-bar"></div>
</div>
<div>Queued bars: <span class="queued">0</span></div>
<button onclick="loadBar()">Load</button> 
```

```css
.container {
  width: 300px;
  height: 50px;
  background-color: #D3D3D3;
}

.progress-bar {
  width: 0%;
  height: 50px;
  background-color: #90EE90;
}
```

```javascript
const bar = document.querySelector('.progress-bar');
const queued = document.querySelector('.queued');

let loader = false;
let width = 0;
let count = 0;

function loadBar() {
  queued.innerText = ++count;
  if (loader === false) {
    bar.style.width = 0;
    tick();
  }
}

function tick() {
  loader = true;
  if (++width > 100) {
    queued.innerText = --count;
    width = 0;
    if (count < 1) {
      loader = false;
      return;
    }
  }
  bar.style.width = `${width}%`;
  setTimeout(tick, 30);
}
```

__[CodePen](https://codepen.io/healeycodes/pen/ZNRVBa)__

#### 4. Do the same thing but without timers!

Okay, they don't really ask this in the interview but someone mentioned `requestAnimationFrame` in the comments on DEV and I thought it would be fun to build an example using it while emulating the previous answer.

The logic is a lot shorter if you don't have to queue bar loads. I ended up deciding to use two coupled functions. Recently, I saw someone say that any instance of `else` is your code is a chance for a refactor. I've been thinking it over, and while no rule holds true, it has been influencing how I shape functions lately. Check it out.

```html
<div class="container">
  <div class="progress-bar"></div>
</div>
<div>Queued bars: <span class="queued">0</span></div>
<button onclick="loadBar(1)">Load</button> 
```

```css
.container {
  width: 300px;
  height: 50px;
  background-color: #D3D3D3;
}

.progress-bar {
  width: 0%;
  height: 50px;
  background-color: #90EE90;
}
```

```javascript
const bar = document.querySelector('.progress-bar');
const queued = document.querySelector('.queued');

let loading = false;
let count = 0;

function tick (timestamp, dist, duration) {
  const runtime = timestamp - starttime;
  let progress = runtime / duration;
  progress = Math.min(progress, 1);
  bar.style.width = `${dist * progress}%`;
  if (runtime > duration) {
    loading = false;
    count--;
    loadBar(0);
    return;
  }
  requestAnimationFrame (function(timestamp) {
      tick(timestamp, dist, duration)
  });
}

function loadBar (increment) {
  count += increment;
  queued.innerText = count;
  if (loading === true || count < 1) { return; }
  bar.style.width = 0;
  loading = true;
  requestAnimationFrame (function(timestamp) {
    starttime = timestamp;
    tick(timestamp, 100, 3000);
  });
}
```

__[CodePen](https://codepen.io/healeycodes/pen/EzpJNo)__

### End notes

Top marks if you have been shouting `<progress>` at me for the whole article. Yup, this [element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/progress) has been there since HTML5. You manage it by setting two attributes `max` and `value`. CSS Tricks have an [article](https://css-tricks.com/html5-progress-element/) on how to style and animate it. They also cover the different states, determinate and indeterminate — the latter meaning "progress unknown". These states are great because they give us a native way to communicate with the user.

This interview question is less about the perfect answer and more about how you communicate your thoughts as you go and the clarifications you ask about. Should there also be a numerical representation of percentage? Is this running on a low-powered device? If so, don't go up in one percent increments. Perhaps five or ten is better.

I think a good extension might be to ask an interviewee to build an interface that receives a WebSocket message describing the progress state and communicates that to the user.

What do you think of this interview question? Does it meet at the crossroads of problem-solving and browser knowledge for you?


Interview Question: Implement a Progress Bar


Premature optimization is the root of all evil. It's also the root of this article.

I like programming puzzles. I also like to go _fast_. We're going to take some LeetCode problems and solve them a few times, first improving runtime complexity in broad strokes and then looking for minor optimizations. We're after these wonderful words:

> faster than 100.00% of JavaScript online submissions

The environment we're targetting is `nodejs 10.15.0` with `--harmony` ([source](https://support.leetcode.com/hc/en-us/articles/360011833974-What-are-the-environments-for-the-programming-languages-)). The online judge system uses relatively small inputs for test cases as far as I can tell.

### First problem

[771. Jewels and Stones](https://leetcode.com/problems/jewels-and-stones/) ~ _You're given strings `J` representing the types of stones that are jewels, and `S` representing the stones you have.  Each character in `S` is a type of stone you have.  You want to know how many of the stones you have are also jewels._

A naive solution here is to loop through our stones, looping through the jewels for every stone. We'll be using standard for loops in this article as they are generally the fastest way of iterating data in JavaScript.

```javascript
let numJewelsInStones = function(J, S) {
    let myJewels = 0;
    // Jewels
    for (let i = 0; i < J.length; i++) {
        // Stones
        for (let j = 0; j < S.length; j++) { // Nested!
            if (J[i] === S[j]) {
                myJewels++;
            }
        }
    }
    return myJewels;
};
```

The runtime is quadratic, `O(N^2)`. Their online judge won't actually accept this solution! We get a big fat **Time Limit Exceeded**. Lesson? Nested for-loops should be avoided where possible.

Let's grab a [Set](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Set) to get rid of one of the loops. Reducing our runtime down to linear, `O(N)`. Looking up a Set in JavaScript is constant time, `O(1)`.

```javascript
let numJewelsInStones = function(J, S) {
    const jewels = new Set(J); // Set accepts an iterable object
    let myJewels = 0;
    for (let i = 0; i < S.length; i++) {
        if (jewels.has(S[i])) {
            myJewels++;
        }
    }
    return myJewels;
};
```

For this effort, we're rewarded with `faster than 97.84%`. I'm happy with this code. It's efficient and readable. If I needed drastically better performance, I might reach for a different technology than JavaScript. We have to walk the length of both strings at least once and there's no getting around that. We can't beat `O(N)` but we can make optimizations.

The stones and jewels are defined as letters. So `a-z` and `A-Z`. This means there are just 52 different buckets our values can fall into! We can use a boolean array instead of a Set. To convert an alphabetical letter into a number, we'll use its ASCII code point via [charCodeAt](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/charCodeAt). We'll set an index to `true` to represent a jewel.

However, there aren't boolean arrays in JavaScript. We could use a standard array and initialize it to length `52`. Or we could use [Int8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Int8Array) and allow the compiler to make additional optimizations. The typed array was ~6% faster when benchmarked with a range `0-52` of random characters entered as `J` and `S`.

Did you spot that our length is wrong? This is something I forgot as I was testing. There are seven characters between `z` and `A` on the ASCII code chart so the length required is actually 59.

![ASCII table](asciitable.png)

```javascript
let numJewelsInStones = function(J, S) {
    const jewels = new Int8Array(59);
    for (let i = 0; i < J.length; i++) {
        jewels[J.charCodeAt(i)-65] = 1;
    }
    let myJewels = 0;
    for (let i = 0; i < S.length; i++) {
        if (jewels[S.charCodeAt(i)-65] === 1) {
            myJewels++;
        }
    }
    return myJewels;
};
```

Et voila, our `100% fastest` [submission](https://leetcode.com/submissions/detail/229804303/). In my tests, this was actually twice as faster as the Set version. Other optimizations I skipped testing were caching lengths, using a while loop instead of a for loop, and placing the incrementor before the number (`++myJewels` vs `myJewels++`).

### Second problem

[345. Reverse Vowels of a String](https://leetcode.com/problems/reverse-vowels-of-a-string/) ~ _Write a function that takes a string as input and reverse only the vowels of a string._

A naive solution for this might be to loop through the array twice, replacing on the second loop. Let's try that out first.

```javascript
let reverseVowels = function(s) {
    const vowels = new Set(['a','e','i','o','u', 'A', 'E', 'I', 'O', 'U']);
    const reversed = [];
    let vowelsFound = [];
    // Find any vowels
    for (let i = 0; i < s.length; i++) {
        if (vowels.has(s[i])) {
            vowelsFound.push(s[i]);
        }   
    }
    // Build the final string
    for (let i = 0; i < s.length; i++) {
        if (vowels.has(s[i])) {
            reversed.push(vowelsFound.pop());
        } else {
            reversed.push(s[i]);
        }
    }
    return reversed.join('');
};
```

This nets us `faster than 97.00%`. The runtime is linear, `O(2N) -> O(N)`, and it reads well but I can't help but think we're looping the string one more time than we have to. Let's try a two-pointer approach. Walking in, step-by-step, from the front and back at the same time, swapping any vowels we see. If there's a middle vowel we just leave it.

```javascript
let reverseVowels = function(s) {
    const vowels = new Set(['a','e','i','o','u', 'A', 'E', 'I', 'O', 'U']);
    s = s.split('');
    let front = 0;
    let back = s.length - 1;
    while (front < back) {
        if (!vowels.has(s[front])) {
            front++;
            continue;
        }
        if (!vowels.has(s[back])) {
            back--;
            continue;
        }
        let temp = s[front];
        s[front] = s[back];
        s[back] = temp;
        front++;
        back--;
    }
    return s.join('');
};
```

We've reduced a full iteration! This gets us `faster than 98.89%` and it's at this point that we need to remember that LeetCode's benchmarks aren't conclusive nor are they consistent. It's not feasible for them to run a large number of iterations with a mixture of test cases. If you're practicing your puzzle solving, stop at `97%` and up. But that's not the point of this article, and, reader, I'm going to get that `100%` for you.

First I threw out the Set. The number of vowels is constant and we don't need all that hashing going on. I tried a switch statement but then found a chained if statement was faster. I discovered that in-lining this logic was faster than a function. I then reduced this down to an expression. What I'm trying to say is: the code coming up is gross. It's close-down-your-IDE-and-talk-a-walk gross. But .. [it's](https://leetcode.com/submissions/detail/229918811/) `faster than 100.00%`.

```javascript
let reverseVowels = function(s) {
    s = s.split('');
    let front = 0;
    let back = s.length - 1;
    while (front < back) {
        if (s[front] !== 'a' &&
            s[front] !== 'e' &&
            s[front] !== 'i' &&
            s[front] !== 'o' &&
            s[front] !== 'u' &&
            s[front] !== 'A' &&
            s[front] !== 'E' &&
            s[front] !== 'I' &&
            s[front] !== 'O' &&
            s[front] !== 'U') {
            front++;
            continue;
        }
        if (s[back] !== 'a' &&
            s[back] !== 'e' &&
            s[back] !== 'i' &&
            s[back] !== 'o' &&
            s[back] !== 'u' &&
            s[back] !== 'A' &&
            s[back] !== 'E' &&
            s[back] !== 'I' &&
            s[back] !== 'O' &&
            s[back] !== 'U') {
            back--;
            continue;
        }
        let temp = s[front];
        s[front++] = s[back];
        s[back--] = temp;
    }
    return s.join('');
};
```

(I'm sorry).

### Third problem

[509. Fibonacci Number](https://leetcode.com/problems/fibonacci-number/) ~ _Calculate the nth Fibonacci number_.

This is a common puzzle and it was the hardest to improve the runtime for because there are so few moving parts in the final solution. I'm sure some RNG was involved with LeetCode's grading too. Let's get the naive solution out of the way. The Fibonacci sequence is often used to teach recursion. However, the algorithm that is used has a runtime of `O(2^n)` (_very_ slow).

I actually crashed a browser tab by trying to calculate the 50th term with this function.

```javascript
let fib = function(N) {
    if (N < 2) {
        return N;
    }
    return fib(N - 1) + fib(N - 2);
}
```

We get `faster than 36.63%` for this answer. Ouch. In production, this is the kind of puzzle that can be solved by memoization (caching some of the work for later). This is the best solution because we only calculate up to the values that we need in linear time `O(N)` and then running the algorithm again for a term under that limit is constant time `O(1)`.

```javascript
const memo = [0, 1];
let fib = function(N) {
    if (memo[N] !== undefined) {
        return memo[N];
    }
    const result = fib(N - 1) + fib(N - 2);
    memo[N] = result;
    return result
};
```

`faster than 94.25%`. LeetCode doesn't store data between each run-through of our code so we'll have to try something different. We've interested in calculating *one* number of the sequence just *once*. I think we can throw away that array. Let's look at the iterative solution.

```javascript
let fib = function(N) {
    if (N < 2) {
        return N;
    }
    let a = 1;
    let b = 1;
    for (let i = 3; i <= N; ++i) {
        a = a + b;
        b = a - b;
    }
    return a;
};
```

If this looks a little different to other iterative versions you might have seen, it's because I avoided the third temporary variable that we have to use in JavaScript to swap values (there are other methods as well but they're too slow). I did some benchmarks and I found using arithmetic instead [was](https://leetcode.com/submissions/detail/229923152/)..  `faster than 100.00%`.


Solving Puzzles With High-Performance JavaScript


When I was starting out, I never understood how to tackle 'projects'. It was hard to understand the scope of a project from a glance, as well as the steps and considerations. I wanted to see behind the scenes. We all work differently but in this article I'm showing how I work when I get an idea-spark for something tiny. Come watch me scratch an itch.

I was on the MDN Web Docs earlier looking for something to spark a research journey for this week's article when I found that there was no random page link. A random page function would be useful for general revision of web technologies, allowing you to click through to things that sounded interesting.

![The project's main/only page](rnd-mdn-preview.png)

I began my journey by hacking away at the [index page](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Index) for JavaScript. This is often how I tackle front end web problems at my job too, using DevTools as the draft version of a site.

![Looking at the table rows](index-hacking.png)

In the HTML there's a table body located at `document.querySelector("#wikiArticle > table > tbody")`. On Chrome, you can right-click an element and go `Copy -> Copy JS Path` to find this. Each entry uses three table rows. I'm after the page link and also the summary. Let's separate the rows into groups that make up one entry each.

You can run these lines in your console.

```javascript
// Index table
const indexTable = document.querySelector("#wikiArticle > table > tbody");

// There are three <tr>s to a row
const allRows = indexTable.querySelectorAll('tr');
const chunkRows = [];
for (let i = 0; i < allRows.length; i += 3) {
    chunkRows.push([allRows[i], allRows[i + 1], allRows[i + 2]]);
}
```

I need to store a copy of the index page's data somewhere. Scraping the page per each request would be wasteful and slow. Storing them as JSON makes the most sense as I'll be delivering this over the web. A page snippet should have a link property and a text property. I'll take care of some of the templating here too.

```javascript
// Map these chunked rows into a row object
const pageEntries = [];
chunkRows.forEach(row => {
    const a = row[0].querySelector('a');
    const page = {
        link: `<a class="page-link "href="${a.href}">${a.innerText}</a>`,
        text: `<div class="page-text">${row[1].querySelector('td').innerText}</div>`
    }
    pageEntries.push(page);
})
```

I needed to decide how I was going to deliver this to users. I considered a Chrome extension but I wanted to prototype faster so I turned to Glitch and used their Express template.

I packaged the scraping logic into an async function and used [puppeteer](https://github.com/GoogleChrome/puppeteer) to run these commands in the context of the index page. (On Glitch, you have to use `--no-sandbox` which can be a security risk.). In the finished project, this script can be called manually with `node getNewPages.js` to update the entries on disk. (This would fit neatly into a cron job!).

From Chrome DevTool hackery to something a little more cohesive.

```javascript
// getNewPages.js

const fs = require('fs');
const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch({
      args: ['--no-sandbox'] // Required for Glitch
    });
    const page = await browser.newPage();
    await page.goto('https://developer.mozilla.org/en-US/docs/Web/JavaScript/Index');

    // Scan index table, gather links and summaries
    const pageEntries = await page.evaluate(() => {

        // Index table
        const indexTable = document.querySelector("#wikiArticle > table > tbody");

        // There are three <tr>s to a row
        const allRows = indexTable.querySelectorAll('tr');
        const chunkRows = [];
        for (let i = 0; i < allRows.length; i += 3) {
            chunkRows.push([allRows[i], allRows[i + 1], allRows[i + 2]]);
        }

        // Map these chunked rows into a row object
        const pageEntries = [];
        chunkRows.forEach(row => {
            const a = row[0].querySelector('a');
            const page = {
                link: `<a class="page-link "href="${a.href}">${a.innerText}</a>`,
                text: `<div class="page-text">${row[1].querySelector('td').innerText}</div>`
            }
            pageEntries.push(page);
        })
        return pageEntries;
    });

    // Save page objects to JSON on disk
    const pageJSON = JSON.stringify(pageEntries);
    fs.writeFile('./data/pages.json', pageJSON, function (err) {
        if (err) {
            return console.log(err);
        }
        console.log(`New pages saved! (JSON length: ${pageJSON.length})`);
    });

    await browser.close();
})();
```

You can load JSON files into Node with `require('./data/pages.json')` which keeps all of our page entries in memory for faster access (this is possible due to the fixed, small size of ~300kb). The rest of our web app is a wrapper around a random function.

```javascript
// server.js

// init project
const express = require('express');
const app = express();
app.use(express.static('public'));

const pages = require('./data/pages.json');
const rndPage = () => pages[Math.floor(Math.random() * pages.length)];

app.get('/', (req, res) => res.sendFile(__dirname + '/views/index.html'));

app.get('/rnd', (req, res) => res.send(rndPage()));

// listen for requests :)
const listener = app.listen(process.env.PORT, function() {
  console.log('Your app is listening on port ' + listener.address().port);
});
```

The only client-side JavaScript we need is a `fetch` call to update the page link and summary. As well as a button with a cute wobble animation to flip through the entries. There's a 1 in 897 chance of getting the same snippet twice, how would you solve this Let me know in the comments!

**Where to go from here:** there are other pages on the MDN Docs (other than JavaScript) that follow the same row pattern, meaning they can be scraped simply by changing the URL in our script.

You can [remix ](https://random-mdn-page.glitch.me/) the project to play with a copy of the code live or clone the repo: [healeycodes/random-mdn-page](https://github.com/healeycodes/random-mdn-page).

A Tiny Project, From Inception to Deployment


We're going to use Glitch as a free container platform to host our application. Better yet, it will have continuous deployment! Whenever we commit to our `master` branch on GitHub, a secure webhook will be sent to our project, which will update, build, and restart itself.

The route that receives this webhook is built into our Node.js/Express application. We'll be using Glitch's `hello-express` template to keep things simple. Inside the route, we need to run our git commands to get our updated files as well as any build and install commands.

*Note: Glitch auto-restarts Node.js projects by running `npm start`.*

![Choosing the template](create-express.png)

Create a new `hello-express` project on Glitch and add your secret to the `.env` file by adding `SECRET='randomized password here'`. There's one additional package we need, which can be installed via the Glitch UI inside package.json, or via console with `npm install body-parser`.

This is the `POST` route, along with including extra imports:

```javascript
const bodyParser = require('body-parser');
app.use(bodyParser.json());
const crypto = require('crypto');
const { execSync } = require('child_process');

app.post('/git', (req, res) => {
  const hmac = crypto.createHmac('sha1', process.env.SECRET);
  const sig  = 'sha1=' + hmac.update(JSON.stringify(req.body)).digest('hex');
  if (req.headers['x-github-event'] === 'push' &&
    crypto.timingSafeEqual(Buffer.from(sig), Buffer.from(req.headers['x-hub-signature']))) {
    res.sendStatus(200);
    const commands = ['git fetch origin master',
                      'git reset --hard origin/master',
                      'git pull origin master --force',
                      'npm install',
                      // your build commands here
                      'refresh']; // fixes glitch ui
    for (const cmd of commands) {
      console.log(execSync(cmd).toString());
    }
    console.log('updated with origin/master!');
    return;
  } else {
    console.log('webhook signature incorrect!');
    return res.sendStatus(403);
  }
});
```

We're interested in push events so we check the header. After, we perform a security check by creating an HMACSHA1 keyed hash with our secret and the webhook's body which will be repository information from GitHub. GitHub sends over a signature of the same body using our secret. We compare the signature with `timingSafeEqual` to avoid timing attacks. Hopefully, all is well and the two match up. Otherwise, we'll send a `403` code and skip the update.

If all is well, we send a `200` code back to GitHub and run our commands. We do this synchronously, using `execSync` since they depend upon each other, and we log the results in case there's any errors or information we need later. `refresh` is a Glitch environment command that resets the UI and fixes the file-tree in the sidebar (which plagued me for about half an hour this weekend!). The update process generally takes 5-15 seconds for small changes. Your application won't be available during this time.

![Exporting to GitHub](export-to-github.png)

Creating the GitHub webhook is actually quite straightforward but first some preparation. You'll need a repository with at least one file so that Glitch can export to it. Once you've got your repository set up, add it as the remote origin via Glitch console `git remote add origin {url}` and send over your project. It will become the `glitch` branch. Create a pull request and merge this to `master`. Otherwise, when the webhook fires, your project might load the wrong `master` changes and reset itself!

*I've seen some people set up this process pulling straight from the `glitch` branch but I prefer having `master` as the production version of a project.*

![Creating the webhook](create-webhook.png)

Head into the repository settings and create a new webhook with your Glitch project's secret. Make sure to choose `application/json` as the content-type. And that's it. Any `push` events with the correct signature will trigger an update/install/restart cycle of your Glitch project 🎉.

You can see what this looks like in a live project on the [PairCode repository](https://github.com/healeycodes/PairCode). It's a lightweight CodePen clone that I wrote at university and have been tinkering with lately. Glitch helped bring it back to life!

Reach out if you're having any tricky issues 👍.


Joining two of my favorite ecosystems together.

Continuous Deployment to Glitch via GitHub Webhooks


_Boyer–Moore majority vote algorithm_ — I love this algorithm because it's amazing _and_ approachable. I first saw it on a LeetCode discuss thread, and it blew everyone away. Some people were, like, irate. [169. Majority Element](https://leetcode.com/problems/majority-element/).

This problem, common on competitive coding sites, has a solution that was discovered in 1980 but went unpublished until 1991 because of its emphasis on Fortran and mechanical verification.

**Problem:** Given a list of votes with a majority (_n_/2 + 1), declare the leader. As in, the most frequently occurring vote. It is possible to get the result in linear time _O(n)_ AND constant space _O(1)_.

Examples:

`[0, 1, 1, 0, 1, 0, 1, 1] => 1 is the majority element`

`['a', 'b', 'a', 'c', 'b', 'c', 'a', 'a'] => 'a' is the majority here`

A naive solution might look like this. We'll use a Dictionary to keep track of all the votes as well as storing the highest number of votes we've seen.

```python
def majority_vote(votes):
    leader = None
    max_votes = 0
    candidates = dict()

    for i in votes:
        # if seen before
        if i in candidates:
            # count their vote
            candidates[i] += 1
            # and check if they're leading
            if candidates[i] > max_votes:
                leader = i
                max_votes = candidates[i]
        else:
            candidates[i] = 1

    return leader
```

The above accomplishes a correct solution in linear time _O(n)_ using linear space _O(n)_. We can do better. One pass, **without** counting every element.

_MJRTY_ or _A Fast Majority Vote Algorithm_ was discovered in the Computer Science Laboratory of SRI International in 1980 by Robert S. Boyer and J Strother Moore. They were assisting a colleague who was working on fault tolerance.

In their humorous paper, they imagined a convention center filled with voters, carrying placards boasting the name of their chosen candidate. Each voter representing an index of the list.

> Suppose a floor fight ensues

They opined that the voters might knock each other out simultaneously, going only for the opposing team. After the mess, the voter/s left standing would represent the majority.

> Here is a bloodless way the chairman can simulate the pairing phase.

Their algorithm improves on our naive solution by removing the data structure (the Dictionary). Converted here from Fortran to Python.

```python
def majority_vote_improved(votes):
    # after one vote, we have a leader
    leader = votes[0]
    count = 1

    for i in range(1, len(votes)):
        # the lead may grow
        if votes[i] == leader:
            count += 1
        else:
            # or shrink
            count -= 1

        # and they may be replaced
        if count == 0:
            leader = votes[i]
            count = 1

    return leader
```

![A graph of this algorithm](graph.png)

Thus, the majority is found in linear time _O(n)_ with constant space _O(1)_.

<br>

Check out a step-by-step walkthrough on Moore's [website](https://www.cs.utexas.edu/~moore/best-ideas/mjrty/index.html). More on [Wikipedia](https://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_majority_vote_algorithm) too.

> The entire effort of specifying MJRTY and getting the 61 verification conditions proved required about 20 man hours. [...] about 55 minutes of computer time to prove the final list of 66 theorems.

<br>

<small>MJRTY - A Fast Majority Vote Algorithm, with R.S. Boyer. In R.S. Boyer (ed.), Automated Reasoning: Essays in Honor of Woody Bledsoe, Automated Reasoning Series, Kluwer Academic Publishers, Dordrecht, The Netherlands, 1991, pp. 105-117.</small>


Breaking down the Boyer–Moore majority vote algorithm with examples in Python.

Fun With Linear Time: My Favorite Algorithm


My B.A. is in Creative Writing. I applied to study a Masters in Computer Science to get better at theory, fill the gaps in my knowledge, and to be a better software engineer. Here are a few things that weren't on the curriculum.


### Markdown

Using [markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) is straightforward and can be learned in fives minutes. However, effective use of it is a little harder. Writing good technical documentation is a separate skill to Computer Science. Have you ever visited a repository on GitHub and the README just clicks with you? Your questions: answered. Your eyes: rewarded.

Directing where a visitor looks and shaping their experience of your text is something that is understated. I've been asked to review portfolios and have found fantastic libraries with textbook clean code and patterns out the wazoo. But all of this hidden behind a cold markdown page with minimal formatting. A few short sentences squashed together doing zero justice to the beautiful code a few clicks further. Most hiring managers would have stopped right there.

Studying Computer Science in academia is quite separate from the reality of programming, which is all about humans connecting. We write code for humans, not machines. No one wants to start reading code without an intro. A nicely marked up README softens the blow — and will [help you get that job](https://dev.to/healeycodes/how-to-write-an-awesome-github-readme-2ldc).

### Clean code

With deadlines looming and marks not awarded for neat code, it's hard to learn to write code cleanly. Going through a few review processes at my first job shaped me up fast though. There's a big difference between writing code for a professor — with the only requirement being that it works and is 'readable' — and writing code for a team that you care about. Projects in academia are built and thrown away for the most part. We all grumble about the code behind some fantastic papers. (Although, a few of my classmates did open source [parts of our projects](https://github.com/healeycodes/websocket-benchmarker)).

People have their own definition of clean code. For me, I'm interested in writing something that can be easily extended or altered in about three to four months. Code with comments only where necessary, cached variables to help cohesion, everything formatted per the style guide and then further for readability. I wish someone told me at school: don't forget the human.

### Code review

While there were one or two opportunities to work in teams, it wasn't a high priority to set up a code review process. All coding — from [launching spaceships](https://coder.today/tech/2017-11-09_nasa-coding-standards-defensive-programming-and-reliability-a-postmortem-static-analysis./) to writing iOS apps — will involve reviewing other people's code and preparing your own work for review. There are two parts here, reviewing and being reviewed.

Reviewing code is a practice in building mental models and applying empathy. If you're reviewing properly, you're not only concerned about how the code reads line by line but how it affects the code base today and also this time next year. You have to ask questions and provide feedback with the utmost empathy. You are not a scorned theatre critic — you are trying to help your _teammate_ improve their code. Be polite, err on the side of being courteous. Though, when you have a rapport, slather on those GIFs.

Preparing code for review is a skill in itself. Your review should be a logical chunk. As small as makes sense. Describing the problem or story your code is solving is important too (I personally use markdown for this). It will arm your buddy as they go to battle in the depths of an attempt at your first bug (sorry, Jamie). It will save them rolling their chair over to you and going _so about that review_.

### The edges of languages

At school we applied languages like Java, JS, and Python, to solve algorithmic problems. We built databases, management systems, navigated mazes, and touched a bit of full stack too. We optimized. We did all this with the core tools. Data structures, algorithms, Object Oriented Programming, and a dash of math. However, I had to learn on my own how `setTimeout`, `delete`, and `this` interact inside a production code base. No one told me about decorators, or how ES6/7/8 was bringing us into the future. I like bugs that take me somewhere I haven't been before — when I crack open the spine of StackOverflow, the esoteric and unknown interactions. For JavaScript, I can recommend [You Don't Know JS](https://github.com/getify/You-Dont-Know-JS) and [Eloquent JavaScript](https://eloquentjavascript.net/) (both free). For Python: [So you want to be a Python expert?](https://www.youtube.com/watch?v=cKPlPJyQrt4).

### ~

Although I worked hard, I learned more by trying to be [creative on the web](https://github.com/healeycodes), entering [game jams](https://healeycodes.com/projects/), and hacking away at weird projects. I pretty much consider myself to be self-taught except I'm lucky enough to put the letters _MSc_ on my resume. The skills that I used to consider paramount have fallen away. I once thought being a programmer was all about eureka moments and collecting a library of algorithms in your brain.

When I hear that someone new is joining our team, I don't worry how good they are at writing code. I want to know that they'll listen to my comments in their review (and tell me how I'm wrong!). I want to be able to communicate with them about technical subjects. I want to learn from them but not just about code-related things. Most of all, I hope they are fun to work with.


What They Don't Teach You at a Computer Science Masters


The following isn't good advice. It's just _advice_. And even that's a stretch.

I used to be really into those productivity hacks. No zero days, Pomodoro, writing up your goals for the day, starting a task for five minutes, etc. Eventually, they merged into a document of rules. I wanted to work as a coder. I had digested all of the advice I could take. Blogs and READMEs, complex graph charts that spider out describing your career in buzzwords you don't even know yet. Enough of the analysis paralysis, I said, and typed up some rules in a small London coffee shop. To these, I stuck.

The following sections describe some of these rules and the results (interviews, job offers) but do note that my success is probably unrelated to every word printed here.

### Write code in two languages every day

For me, this was JavaScript and Python. I was aiming for full stack and back end positions. If I was stuck on my projects, I would code up data structures from scratch in both languages. The repetition helped them stick. I wanted to be ready to write relatively clean code in either language without any preparation, should I be called to interview. Solving problems that are challenging to you is never a waste of time if your goal is to learn.

Sites like [LeetCode](https://leetcode.com/) and [HackerRank](https://www.hackerrank.com/) have questions that you can sort to get progressively harder. What's important is that you realize that the goal is not a correct answer — it's _understanding_. Pushing at the edge of your knowledge should spawn mini study sessions where you chase down different implementations of the solution so you understand the whys and hows. I found that personal blogs always had the best break-downs.

### Read yourself to sleep with a textbook

If you've seen my other writing, you'll know that I won't stop recommending _Grokking Algorithms_. I chose the Computer Science textbooks that I enjoyed, not necessarily ones that would help in my job search. These books didn't have to have code or math in them, e.g. _Code: The Hidden Language .._ or _The Mythical Man-Month_. A lot of the stuff I did during my job search was to get in the right headspace. This is embarrassing to say but I did the things that I pictured a good software engineer doing. Which I suppose is like wanting to be a good basketball player and spending your post-practice time standing in center court just staring at the hoop. It's good to just rest some times, ya know.

### Study interview question trivia every day

I wasn't trying to memorize, which would have felt cheap, I was trying to understand what these questions implied in order to direct my learning. They are asked for a reason. `setTimeout` trivia questions are really about asynchronous programming and the Event Loop. I had to know the answer and be able to speak about the subject it represented. These sessions were scary first as they further revealed everything I didn't know (and still don't!). 

Not often will you have to call `delete` but it's good to know what it does and _doesn't_ [do](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/delete) (I was asked about this in an interview). Searches like `most popular JavaScript interview questions` will find you the lists I'm talking about. These questions can get pretty esoteric so your mileage may vary.

### Three testable projects with instant interaction

Three felt like a good number. I worked backward — coming up with the blurb and the bullet points for the projects on the resume first. Building the projects came second. They each needed a test suite, e.g. `npm test`, `pytest`. In my mind, good engineers test everything. I suppose I still believe that. Getting tests written early on speeds up the later stages of development.

What do I mean by instant interaction? I guessed that someone reviewing my resume would spend about four seconds per each hyperlinked project — and that most wouldn't even click. I hosted all three projects on a Digital Ocean [Droplet](https://www.digitalocean.com/products/droplets/), local to my region. Digital Ocean have some great guides for getting started with Linux too. I didn't want to use free options, like Heroku's free tier that sleep when not in use and have to wake up when a visitor arrives — there go my four seconds.

Once they arrived on the page I wanted them to be able to interact with the core idea of the project instantly, e.g. no sign-up or authing required. I'll skim over my projects to give you a better idea.

**CodePen Teams Clone**

I was (and still am) very into pair programming and created a low latency solution for remote collaboration coding. I cloned the design of CodePen but piped the display data over WebSocket to the whole room. Teammates could see live edits in about 25ms. Rooms could be forked too. When arriving on the home page, the _Create Room_ button was front and center.

**Game Jam Highscores API**

This was a free solution I ran for a few years for game devs to instantly set up a highscores backend by just calling our RESTful API. Upon loading the home page, you were greeted by your API key, and there were links prefilled with this key to test out the features.

**Lightweight Google Analytics Clone**

I cloned Google Analytics in the most basic way possible, allowing people to roll their own analytics. The angle was privacy-friendly and people were only tracked for 24 hours. I linked to the demo page on my resume, with animated graphs that popped up with page views, unique visitors, avg. time on page, bounce rate.

Back to the rules.

### Read ten resumes a day

People post tech resumes all over the place but I found the most high quality were on [r/cscareerquestions](https://www.reddit.com/r/cscareerquestions/). After all the studying, whether you got an interview or not was almost totally weighted by your resume. Reading all these resumes helped me understand what my own was competing with. It gave me ideas for sections, page design, and even projects. I also helped out by pointing out grammatical errors wherever I saw them. I took my time with them — even printed them out sometimes and marked them with a pen. There's historical precedence for this kind of behavior, Hunter S. Thompson typed out the entirety of The Great Gatsby and A Farewell to Arms (on a typewriter no less!).

### Cover letter with genuine connection

Cover letters seem to be more beneficial over here in the UK. I count the little text boxes that companies offer with your application as a cover letter too. I fought hard to write something genuine as I really believe that sincerity wins over everything else. I liked to read the company's tech blog if they had one and if not I often checked out the source code of their website and tried to find interesting things to comment on. I even found a leaked API key once. In most of my interviews, topics from my cover letter were brought up.

### Pace and plan for phone interviews

Print out everything beforehand and lay it out on the bed. The job post, my personalized resume for that job, the companies bio page, the interviewer's LinkedIn (although, don't allude to this) — it helped me understand their background and seeing their face was relaxing. It helped remind me that people are almost unanimously friendly and want you to succeed. I've always paced around the room for phone interviews. I alternated between pacing and lying flat on my back.

### Travel to interview location the day before

I traveled down to the interview's location the day before so that part of my journey would be stress-free. I visualized myself interviewing, getting the job, and working there — like I was going on my new commute. Being familiar with the area made me more comfortable when I ended up in the room and that's what it's all about really. An hour or so snapshot of yourself. Nothing could prepare me for getting lost in the building though, which actually happened twice.

Following my application, I ended up with a phone interview rate of 10%, all of which invited me to interview in person. I approached test projects with the same obsessive gusto I've unfortunately shown in this article.


The Weird Rules I Set Myself That Got Me a Job


I read through the earliest README I could [find](http://pdp-10.trailing-edge.com/decuslib10-04/01/43,50322/read.me.html). Written in 1975 by William J. Earl at the CS department of UIC. The text is a little dry but also surprisingly relatable, some 44 years later. _"Due to a bug in the compiler, this function does not compile correctly"_.

A README is a reflection of how a repository is maintained. A good one doesn't necessarily mean an active, bug-free project with perfect tests. But it suggests that the owner cares about **you**, the user (or future maintainer). A good README tells you everything you need to know to use the project and get involved. It _sells_ the project — but concurrently respects a visitor's time by letting them know if they need a different solution.

When using your GitHub profile as part of an application, READMEs are important in a different way. They showcase technical writing skills — the ability to communicate well and put together a software-related document. Expecting someone to dive into your code without providing an abstraction explaining the project is a pretty big ask.

## Clear description

Someone should be able to use your software without reading a line of code.

First, change the default title that GitHub provides — turn `python-ml-project-for-cat-lovers-2` into `Cat Crawler - Classify Cat GIFs`. The next step is explaining your project in its simplest form. Many people use a one-liner at the very top.

> A bot that downloads and indexes Cat GIFs

As you add more content to your lead description, you raise the chance of cognitive overload. After editing the language down, use titles, line breaks, and spacing to further break it up. (Two new lines for a new paragraph and `<br>` for a break. [Cheatsheet](https://help.github.com/en/articles/basic-writing-and-formatting-syntax).) Don't shy away from product logos and screenshots. Unlike other technical documentation, multimedia performs well here. For the more esoteric repositories, a background reading section can help.

If your repository is fun or lighthearted then your description should show this! [Strunk & White](https://en.wikipedia.org/wiki/The_Elements_of_Style) have their place but the internet is also a hangout for cool coders being creative. Shoutout to [not-paid](https://github.com/kleampa/not-paid), which slowly fades a website until it becomes invisible at a certain date, should the client not pay their dues.

## Usage

How does someone use your project? If it's an API, have a section of code with the most basic interaction. Your full documentation can be further down the page or at another location. [facebook/react](https://github.com/facebook/react) shows their usage with an example — the smallest way someone could use React. Use single backticks to quote code, and three backticks on new lines for codes blocks. Put the language right after the first three backticks for specific highlighting.

```jsx
function HelloMessage({ name }) {
  return <div>Hello {name}</div>;
}

ReactDOM.render(
  <HelloMessage name="Taylor" />,
  document.getElementById('container')
);
```

Show the output of this interaction. If you can use a GIF in any way then do so! GIFs are easy for humans to parse and they transmit a lot of data into your reader's mind. [alexfoxy/laxxx](https://github.com/alexfoxy/laxxx), a library for smooth web animations, does this perfectly.

![alexfoxy/laxxx](laxjs.gif)

I use the open source tool [ShareX](https://getsharex.com/) to create my GIFs. It's as simple as selecting an area of your screen. Another solution I recommend, also open source, is [LICEcap](https://www.cockos.com/licecap/).

## Installing

Hopefully, after seeing your project in action, your visitor wants to install it. This section is sometimes titled as _Getting Started_. Every project should have this, even if it's just `npm install catcrawler`. If it's a static site then say so! `host the parent dir on a webserver`. Assume knowledge of the basic tools. You don't need to explain what `pip` or `npm` is but you should list every command that you would run on a new setup.

[DEV](https://github.com/thepracticaldev/dev.to/#getting-started) has a thorough section for getting a build up and running, which is a great model to follow if you're aiming to be accessible. It's good practice to spin up a virtual machine and try to replicate your installation guide — assuming you don't have automatic build tests to do this for you.

## Badges

![example badge](example-badge.png)

Mostly standardized by [badges/shields](https://github.com/badges/shields), GitHub badges are one of the first things a visitor sees as they scroll down. Build status badges describe the stability of a project. Other badges might show the activity of a repository with commits per month, or the number of maintainers. They aren't compulsory but much like GIFs, they are a huge bonus.

[shields.io](https://github.com/badges/shields) have an API for creating your own badges, and they also have an npm [package](https://www.npmjs.com/package/gh-badges) which is pleasant to use. I picked it up a few weeks ago and had some badges up and running in under an hour! Another npm alternative is [badger](http://badges.github.io/badgerbadgerbadger/). Python has [pybadges](https://github.com/google/pybadges) by Google.

If you're looking to earn your first `build: passed` badge, I wrote an [article](https://dev.to/healeycodes/earn-a-build-passing-badge-on-github--testing-your-express-app-with-travis-ci-1m2f) here on DEV that walks you through getting started with continuous integration on GitHub.

## Contributing

If you're seeking them, a section for contributors is helpful and welcoming. There is a GitHub [standard](https://help.github.com/en/articles/setting-guidelines-for-repository-contributors) for adding a `CONTRIBUTING.md` file to the root dir. This may include a code of conduct and general guidelines for finding issues and building a pull request. A lot of beginners are anxious about the process of helping out open source projects and walking them through step-by-step can alleviate some of this. If you're not sure where to start, I recently saw a code of conduct [generator](https://github.com/uyouthe/code-of-conduct) which I thought was pretty neat. I know that some of my friends will only support repositories with strict guidelines and rules for maintainer interaction.

## License

A license is the first thing I look for when I'm seeking a solution at work. When creating a repository through GitHub, there is an option for choosing a license which will generate a `LICENSE.md` file for you. GitHub also has a [page](https://help.github.com/en/articles/licensing-a-repository) on this file — and they are the creators of [choosealicense.com](https://choosealicense.com/), a fantastic guide walking you through all the options. I personally use MIT for my open source code. Some people have strong opinions about licensing, especially when it comes to [GPL](https://tldrlegal.com/license/gnu-general-public-license-v3-(gpl-3)). _"Any modifications to or software including (via compiler) GPL-licensed code must also be made available under the GPL along with build & install instructions."_

## Templates

[There](https://github.com/RichardLitt/standard-readme) [are](https://github.com/zalando/zalando-howto-open-source/blob/master/READMEtemplate.md) [many](https://github.com/noffle/common-readme) README templates out there. They're great in a pinch but I've found that smaller projects don't fit neatly into this kind of box and the resulting text comes across a little cold. Mature projects benefit more from templates but given the amount of developer time that has probably into the software, it's worth it to have a custom solution.

[This](https://gist.github.com/PurpleBooth/109311bb0361f32d87a2) is my favorite as it's straight to the point and also has two subsections on tests. If you have any tests, this should be mentioned in your README. When cloning a project, running tests is the first thing I do to make sure my setup is ready for development.

## Other sections

Once you're past all the important sections, fill your README to your heart's delight. I may be in the minority but I love browsing GitHub looking for new things and discovering how they're built. I appreciate a detailed repository with lots of example code. For me to get involved with a project, it's important to see that the maintainers care at least as much as I do.

Explore [trending](https://github.com/trending) for your programming language to get an idea for the standard layout. For inspiration, check out two of my recent favorites [Gatsyby](https://github.com/gatsbyjs/gatsby) and [lax.js](https://github.com/alexfoxy/laxxx). Most of all, make your documentation sing.


How to Write an Awesome GitHub README


Something a lot of beginners struggle with is the concept of passing data between different programming languages. It's far more simple to understand that a value exists in a variable which can be passed around from function to function. However, to go beyond the program's edges we must serialize our data in some way. We'll look at two ways that these two languages can communicate. AJAX requests via the new Fetch API, and piping between local processes.

Serializing data means taking a value, object, or data structure, and translating it into a format that can be stored or transmitted. Most importantly, it needs to be put back together at the other end. Let's take look at JavaScript Object Notation (JSON). JSON is a human-readable format and is straightforward for machines to read and write. The [specification](https://www.json.org/) is small enough to read during one cup of coffee. Both JavaScript and Python have standard library methods to parse and write JSON.

## JSON Crash Course

JSON is built on two data structures. **Objects** — key/value pairs like a JavaScript `Object`, and a Python `Object` or `Dictionary`. **Arrays** — a series of data like a JavaScript `Array`, and a Python `List`.

```javascript
/* JavaScript
   Try this out in your developer console! */

const person = {"name":"Andrew", "loves":"Open Source"};
const asJSON = JSON.stringify(person);

// `person` is of type 'object'
console.log(`person is of type ${typeof person}`);

// `asJSON` is of type 'string'
console.log(`asJSON is of type ${typeof asJSON}`);

// We can convert it back to an object by parsing it
// `asObject` is of type 'object'
const asObject = JSON.parse(asJSON);
console.log(`asObject is of type ${typeof asObject}`);
```

Let's do the same in Python by using the standard library module `json`.

```python
# python

animal = {'type':'cat', 'age':12}
as_json = json.dumps(animal)

print(type(animal))  # prints '<class 'dict'>'
print(type(as_json))  # prints '<class 'str'>'

# now back again
as_object = json.loads(as_json)
print(type(as_object))  # prints '<class 'dict'>'
```

**Recap:** in JavaScript, you serialize to JSON with `JSON.stringify()` and parse with `JSON.parse()`. This works in the browser as well as Node.js. In Python, first import the `json` module then serialize with `json.dumps()` and parse with `json.loads()`.

## Talking via AJAX

In the past, this would be done with [XMLHttpRequest](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest) but the relatively new [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) is far more pleasant to use. First, we'll code up a small Python web server, and then we'll look at passing JSON back and forth with the browser.

[Flask](http://flask.pocoo.org/) is a 'microframework'. It's not only extremely fun to use, it's also great to prototype with. We'll use its [jsonify](http://flask.pocoo.org/docs/1.0/api/#flask.json.jsonify) module which writes/parses JSON as well as setting the correct response headers (an application/json mime type). It requires two [OS-specific commands](http://flask.pocoo.org/docs/1.0/quickstart/) to install and run a debug server. For myself on OS X, they were `pip install flask` and `FLASK_APP=app.py flask run`.

Code along or fork [this repository](https://github.com/healeycodes/talking-between-python-and-js) to grab the code for all the examples.

```python
# app.py
from flask import Flask, jsonify, request, render_template
app = Flask(__name__)

@app.route('/hello', methods=['GET', 'POST'])
def hello():

    # POST request
    if request.method == 'POST':
        print('Incoming..')
        print(request.get_json())  # parse as JSON
        return 'OK', 200

    # GET request
    else:
        message = {'greeting':'Hello from Flask!'}
        return jsonify(message)  # serialize and use JSON headers

@app.route('/test')
def test_page():
    # look inside `templates` and serve `index.html`
    return render_template('index.html')
```

With our server running and serving us a page we can run JavaScript on, let's talk in JSON! We'll send a GET request with the Fetch API and receive a greeting from Flask. Before writing a single line of client-side code, I always use [Postman](https://www.getpostman.com/) to test my web servers — it's free and one of the industry standard tools for API testing and development.

I'm running the following snippets in `<script>` tags inside `templates/index.html`. There's nothing else inside `index.html` so the rendered page is blank. Everything happens in the console.

```javascript
// GET is the default method, so we don't need to set it
fetch('/hello')
    .then(function (response) {
        return response.text();
    }).then(function (text) {
        console.log('GET response text:');
        console.log(text); // Print the greeting as text
    });

// Send the same request
fetch('/hello')
    .then(function (response) {
        return response.json(); // But parse it as JSON this time
    })
    .then(function (json) {
        console.log('GET response as JSON:');
        console.log(json); // Here’s our JSON object
    })
```

Awesome! We've got Python talking to client-side JavaScript using JSON for data serialization. Let's flip it and send JSON to Python from the browser. We'll use the Fetch API again but it will be a POST request instead of GET.

**Beginner tip:** Remember the difference between POST and GET. When you POST mail, you head to the post office with your letter filled with information. When you GET mail, you head to the post office again but this time you're picking up something that’s been left for you.

```javascript
// POST
fetch('/hello', {

    // Declare what type of data we're sending
    headers: {
      'Content-Type': 'application/json'
    },

    // Specify the method
    method: 'POST',

    // A JSON payload
    body: JSON.stringify({
        "greeting": "Hello from the browser!"
    })
}).then(function (response) { // At this point, Flask has printed our JSON
    return response.text();
}).then(function (text) {

    console.log('POST response: ');

    // Should be 'OK' if everything was successful
    console.log(text);
});
```

With these two core building blocks, we've conquered JSON communication via HTTP. However, do note that you should be adding `catch` to the end of these Promises.  I've only trimmed them for clarity. It's better to handle errors gracefully so we can tell the user that they are disconnected or that there's an error on our end. Docs for catch [here](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/catch).

If you're talking to a Node.js web server with Python, you will probably reach for the [requests](http://docs.python-requests.org/en/master/) module, which has syntax almost identical to the Fetch API.

## Talking via processes

We're going to spawn processes (both ways) so we can see what communication between Node.js and Python looks like. We'll listen to the [stdout](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout)) stream of the child process in both instances. Let's imagine that we have a program that reports data at irregular intervals. A temperature sensor. We want to listen to that program and store the values it reports.

Here's our fake sensor program in Python. It prints data to stdout. We'll catch it in Node.js.

```python
# sensor.py

import random, time
while True:
    time.sleep(random.random() * 5)  # wait 0 to 5 seconds
    temperature = (random.random() * 20) - 5  # -5 to 15
    print(temperature, flush=True, end='')
```

When piping information in this way, it's important to flush the stream so it reaches stdout when you expect it to ([why do we fflush() in C?](https://www.quora.com/Why-do-we-use-the-functions-fflush-stdin-and-fflush-stdout-in-c)) . More information on Python flushing [here](https://stackoverflow.com/a/35467658). We also make sure that the end of the printed statement contains no extra information (even though `parseFloat()` would clean it!) by default it would be the newline character `\n`.

So, we're Node.js and we want the current temperature as it's reported. Let's spawn `sensor.py` as a process and listen for the stdout event. _Piping_ the data between the two running languages.

```javascript
// temperature-listener.js

const { spawn } = require('child_process');
const temperatures = []; // Store readings

const sensor = spawn('python', ['sensor.py']);
sensor.stdout.on('data', function(data) {
    
    // Coerce Buffer object to Float
    temperatures.push(parseFloat(data));

    // Log to debug
    console.log(temperatures);
});
```

## Flip it and reverse it

Now, let's flip those roles. A Node.js sensor and a Python listener! This time we'll try a different method, using a newline character (`\n`) to delimit the different readings instead of waiting for an event. We'll add the data to a buffer until we hit a newline char. Once we do, we've collected a full reading and we can store it.

First, the equivalent sensor in Node.js.

```javascript
// sensor.js

function reportReading() {
    const temperature = (Math.random() * 20) - 5; // Range of -5 to 15
    process.stdout.write(temperature + '\n'); // Write with newline char
    setTimeout(reportReading, Math.random() * 5000); // Wait 0 to 5 seconds
}
reportReading();
```

Now in Python, a temperature listener program that will spawn the above code as a process.

```python
# temperature-listener.py

import sys
from subprocess import Popen, PIPE

temperatures = []  # store temperatures
sensor = Popen(['node', 'sensor.js'], stdout=PIPE)
buffer = b''
while True:

    # read sensor data one char at a time
    out = sensor.stdout.read(1)

    # after a full reading..
    if out == b'\n':
        temperatures.append(float(buffer))
        print(temperatures)
        buffer = b''
    else:
        buffer += out  # append to buffer
```

You can run `node temperature-listener.js` or `python temperature-listener.py` and the result will be the same. The array of temperatures will grow as new data arrives, and our debug log line will result in the following output.

```js
[ 3.8075910850643098 ]
[ 3.8075910850643098, -1.5015912681923482 ]
[ 3.8075910850643098, -1.5015912681923482, 11.97817663641078 ]
```

We've seen two different ways of communicating between Python and JavaScript but if either of these aren't for you — don't fret! There are many ways to pass data between these two languages. Not limited to: [named pipes](https://en.wikipedia.org/wiki/Named_pipe), TCP sockets, WebSockets, and file polling.


Let's learn how to pass data back and forth between languages.

Talking to Python from JavaScript (and Back Again!)


Let's learn about `window.localStorage` by building a tiny note-taking app with some simple HTML, CSS, and JavaScript. We'll code up a small UI and autosave the user's notes to their browser. My co-worker keeps all of his snippets on something just like this. I can't believe the mileage he gets out of 50-60 lines of code!

Most client-side storage interactions used to involve manipulating cookies but luckily we now have modern APIs. With cookies, the user sends everything stored with every request meaning worse performance for both parties.

Enter: `localStorage`. A key/value API that's available on the `window` object. Unlike `sessionStorage` which lasts for the during of the page session, `localStorage` has no expiration time.

All keys and values are converted to Strings. Let's add, update, and delete a key, and then try to read from an empty key. This is console input so the return value follows the command.

```javascript
// Add key - or overwrite existing key
window.localStorage.setItem('foo', 'bar')
> undefined

// Read a value from a key
window.localStorage.getItem('foo')
> "bar"

// Delete key - deleting empty yields same return value
window.localStorage.removeItem('foo')
> undefined

// Read from a deleted or unknown key
window.localStorage.getItem('foo')
> null
```

Other ways you might interact with localStorage involve clearing all keys and iterating through existing keys.

```javascript
// Get the read-only length
window.localStorage.length
> 6

// Get the nth key
window.localStorage.key(0)
> "algoliasearch-client-js"

// Delete all keys
window.localStorage.clear()
> undefined

// Iterate through all keys
for (let i = 0; i < window.localStorage.length; i++) {
    console.log(window.localStorage.key(i));
}
```

You can try some of these out in your developer console right here. DEV stores the following localStorage keys for me as I'm writing. Make sure **not to delete** any of these though! `algoliasearch-client-js`, `config_body_class`, `current_user`, `https://dev.to/new`, `pusherTransportTLS`.

#### The App

Here's what we'll be making. A small but practical note-taking app. Aside from spacing, it uses little styling ([send a pull request](https://github.com/healeycodes/tiny-note-taker)). What's important is how the app communicates with the client's browser storage.

[![Preview of tiny note taker](tiny-note-taker-preview.png)](https://healeycodes.github.io/tiny-note-taker/)

Hopefully, you're already thinking of the `localStorage` methods that we'll need. Let's use a ready-built HTML template that names the functions that we'll be using.

```html
<h5>Tiny Note Taker</h5>
<div>
    <button onclick="newNote()">New</button>
        <select id="notes" onchange="changeNote()"></select>
        <button onclick="deleteNote()">Delete</button>
</div>
<textarea id="editor" rows="10" onkeyup="saveNote()"
    onkeydown="checkEmpty()" autofocus></textarea>
<script src="./index.js"></script>
```

Take a second to think about how these functions might relate to the storage methods we saw just now, and how they will manipulate the DOM.

<hr>

`newNote()` Prompt the user for a note name and create it.

`changeNote()` Swap to the new note when the selected element is changed.

`deleteNote()` Delete the currently selected note.

`saveNote()` Save the currently selected note as the user types.

`checkEmpty()` — This one is less obvious.

If a user arrives at the page and starts typing before creating a new note, the name would be `""` an empty String. Instead, as they type _but before the note is saved_ name the current note `untitled`. We simplify our logic here by separating it into two events. `onkeydown` checks for an empty note and fixes if necessary. This means that `onkeyup` can simply save the note without any additional checks. Thus, `checkEmpty()`.

**Reminder**: Key events are [almost always](https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent#Usage_notes) called in this order `onkeydown -> onkeypress -> onkeyup` and they all have their place!

Let's start with our base case. Saving one note as the user types. Since we've placed the script include at the bottom of the HTML, we can grab a reference to the two elements we need without waiting for any document load events.

```javascript
const notes = document.querySelector('#notes');
const editor = document.querySelector('#editor');
```

Great. Those references will last for the lifetime of this script as we'll never delete those elements. Next, we need to title the empty-named note as untitled if the user types without first hitting the New button.

```javascript
/**
 * Check for empty note title.
 */
function checkEmpty() {
    if (notes.length === 0) {
        const untitled = document.createElement('option');
        untitled.innerText = 'untitled';
        notes.appendChild(untitled);
    }
}
```

Here we create an option element with JavaScript and make it available in the select dropdown. We don't need to do any saving yet, that will happen a few milliseconds later when the user lifts their finger off the current key.

**A few milliseconds later.**

```javascript
/**
 * Save editor text to storage under the current note.
 */
function saveNote() {
    window.localStorage.setItem(notes.value, editor.value);
}
```

There's `localStorage.setItem`! We know what that does. It will save the note text as the value and the note title (`"untitled"`) as the key. Now we have a problem to solve. If the user leaves and comes back, the select dropdown will be empty. Worse: they will overwrite their `"untitled"` note.

As our script runs, just after we grab references to those elements, let's populate the select dropdown with all the user's saved notes.

```javascript
const notes = document.querySelector('#notes');
const editor = document.querySelector('#editor');

// Load user's saved notes
for (let i = 0; i < window.localStorage.length; i++) {
    const newNote = document.createElement('option');
    newNote.innerText = window.localStorage.key(i);
    notes.appendChild(newNote);
}
changeNote();
```

Here we use the read-only `length` property of `localStorage` as well as its method `key()`. All saved notes are loaded into the select dropdown. We let the HTML operate as our state. This way we don't need to keep track of any of the notes in our JavaScript. We rely on simple functions to link up the DOM to the Storage API.

At the bottom of that last snippet, we saw `changeNote()` which loads the currently selected note into the textarea element a.k.a. the editor. This function is also called after a new note is created, which clears the editor. New notes are initialized with an empty String.

```javascript
/**
 * Change editor text to the currently selected note.
 */
function changeNote() {
    editor.value = window.localStorage.getItem(notes.value);
}
```

The Storage API abstracts away the heavy lifting for us.

User's need a way to name and create notes. `newNote()` does just that. It uses [Element#insertBefore](https://developer.mozilla.org/en-US/docs/Web/API/Node/insertBefore) to place the new note at the top of the selectbox. This method is called on the parent element. `let insertedNode = parentNode.insertBefore(newNode, referenceNode);`. What's neat about this method is that it accepts a null value as the second argument. Meaning we can call `insertBefore` on empty parent elements!

```javascript
/**
 * Ask the user to name their new note then create it.
 * Add this note to the select div then focus to it.
 */
function newNote() {
    const note = prompt('Name of note?');
    window.localStorage.setItem(note, '');

    const noteElem = document.createElement('option');
    noteElem.innerText = note;
    notes.insertBefore(noteElem, notes.firstChild);

    // Focus this note
    notes.value = note;
    changeNote();
}
```

Our note taking app is nearly finished. The final function required is for deleting notes. When a user is finished with a note, they can select it and then hit the Delete button. The trickiest part about this function is finding the right note to delete in the select dropdown. One of the drawbacks to using the DOM as our state. We have to loop through every option, checking the `value`.

```javascript
/**
 * Delete currently selected note
 */
function deleteNote() {
    const note = notes.value;
    window.localStorage.removeItem(note);
    editor.value = '';
    for (let i = 0; i < notes.length; i++) {
        const option = notes[i];
        if (option.value === note) {
            notes.removeChild(option);
        }
    }
}
```

We've now got a fully functioning note taking app, infinitely scalable through a CDN. We used the powerful [Storage API](https://developer.mozilla.org/en-US/docs/Web/API/Storage) and learnt how to use the DOM as application state. Go us! 🙌

Check out the final application code [here](https://github.com/healeycodes/tiny-note-taker), and the live version [here](https://healeycodes.github.io/tiny-note-taker/).


Learn the Storage API by Building a Note Taking App (Tutorial)


Let's look at how the DEV.to API can be used to embed a list of your recent posts. We'll also sort them by their positive reactions! All it takes is a splash of client-side JavaScript. The API is [not released](https://github.com/thepracticaldev/dev.to/issues/2187) yet, is not documented, and will likely change but I'll keep this post up to date.

Here's an [test page](https://healeycodes.github.io/embed-DEV-posts/example/) which uses the tiny library we'll be writing. We'll be creating a plain list to keep things extendable. You can also skip straight to the [repository](https://github.com/healeycodes/embed-DEV-posts) to check out the final code.

[![](test-embedded.png)](https://healeycodes.github.io/embed-DEV-posts/example/)

#### How easy is the DEV API to use?

[Easy](https://dev.to/api/articles?username=healeycodes). The articles route is `https://dev.to/api/articles?username=$user`. Originally, I experimented with getting more than the latest 30 posts, which is the default. This can be done by adding `&page=$pageNum` — but requesting multiple pages introduces a delay. Since it's impossible to know the number of pages of posts, you need to keep going until you hit an empty page. A late-loading list doesn't make for great UX.

Let's start by using the [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) to make a request from the user's browser.

```javascript
fetch(`https://dev.to/api/articles?username=healeycodes`)
    .then(function (response) {

        // Parse it as JSON
        return response.json();
    }).then(function (posts) {

        // An array of posts
        console.log(posts);
    })
```

Looking good. Let's neaten this up with some [JSDoc](http://usejsdoc.org/about-getting-started.html) comments so the code is more accessible. We'll also wrap it in a function.

```javascript
/**
 * Get a DEV user's post objects.
 * Only fetches previously 30 posts. Using `&page=X` is too slow.
 * @param {string} username - DEV username, e.g. 'ben'.
 * @returns {array} User's post objects.
 */
function getPosts(username) {

    // Assume that `api` is defined as a constant
    return fetch(`${api}/articles?username=${username}`)
        .then(function (response) {
            return response.json();
        })
        .then(function (posts) {

            // Sort the array in place with a compare function
            return posts.sort((a, b) => b.positive_reactions_count - a.positive_reactions_count);
        });
}
```

A list of 30 posts is maybe a bit much. We'll cut it down later. For now, notice how we sorted the array in place with a [compareFunction](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort#Parameters). We used a shorthand version. Compare functions are typically written by returning `-1`, `1`, or `0` when comparing the two parameters. However, all that matters is that the compare function is consistent.

We want to create a simple list that can be styled with CSS. Sensible class names might be `dev-feed-list` and `dev-feed-item`. We'll need an element to attach this list to as well so that should be an argument that our library takes. With JavaScript, we can add classes, and create lists and attach them on the fly like this:

```javascript
// An unordered list
const list = document.createElement('ul');

// A list item
const item = document.createElement('li');
item.classList.add('dev-feed-item');
item.innerText = 'I am a list item.';

// Let's create a relation between them
list.appendChild(item);

// Now between the list and the document, rendering the list
// body -> ul -> li
document.body.appendChild(list);
```

Let's create a function that builds the elements that will make up our embedded list. We'll use the `getPosts` function from earlier to get our sorted array of posts.

```javascript
/** 
 * Creates a DEV feed of posts.
 * @constructor
 * @param {Element} elem - Place list of posts inside this element.
 * @param {string} username - DEV username, e.g. 'ben'.
 * @param {number} numberOfPosts - Number of posts to list. 
 */
function createFeed(elem, username, numberOfPosts = 5) {
    const feed = document.createElement('ul');
    feed.classList.add('dev-feed-list');
    getPosts(username)

        // With our posts array
        .then(function (posts) {
            posts.length = numberOfPosts;
            posts.forEach(function (post) {

                // We create an element for each item
                const item = document.createElement('li');
                item.classList.add('dev-feed-item');

                // As well as a link for users to click through
                const link = document.createElement('a');
                link.href = post.url;
                link.innerText = post.title;

                // ul -> li -> a
                item.appendChild(link);
                feed.appendChild(item);
            });

            // Attach the list of posts, rendering it
            elem.appendChild(feed);
        });
}
```

If we bundle this all together as a script, we can generate lists by calling `createFeed(element, username, numberOfPosts)` where `element` is the container for the list and `username` is a valid DEV user, and `numberOfPosts` is the number of posts we want to render.

By allowing a custom number of posts to be fetched, as well as using sensible CSS classes, our library is extensible and can be used as a tiny module!


Embed Your Latest DEV Posts Anywhere (Tutorial)


Dead links and images and be frustrating for visitors. Manually checking for them can be even more frustrating! We're going to build a bot that crawls a website for missing resources using just the Python standard library.

![Bot in action](bot-in-action.gif)

Let's talk design goals. We want to run a command and have the whole website checked for dead resources. This means some crawling will be involved.

```bash
$ python deadseeker.py 'https://healeycodes.com/'
> 404 - https://docs.python.org/3/library/missing.html
> 404 - https://github.com/microsoft/solitare2
```

More technically, the bot should parse all HTML tags on the given page looking for `href` and `src` attributes. If it finds any, it should send a GET request and log any HTTP error codes. If it finds local pages (e.g. `/about/`, `/projects/`) it should queue them to scan later. As we check links, let's add them to a set so we only check them once.

Python has [html.parser](https://docs.python.org/3/library/html.parser.html#module-html.parser) for us — a simple HTML and XHTML parser. Let's take a look at how it works.

```python
from html.parser import HTMLParser

# extend HTMLParser
class MyHTMLParser(HTMLParser):
    # override `handle_starttag`
    def handle_starttag(self, tag, attrs):
        print(f'Encountered a start tag: {tag}')
        print(f'And some attributes: {attrs}')

parser = MyHTMLParser()
parser.feed('<html><body><a href="https://google.com">Google</a></body></html>')
```

This prints:

```bash
> Encountered a start tag: a
> And some attributes: [('href', 'https://google.com')]
```

That's the heavy lifting handled for us. How about the requests? Python has [urllib.request](https://docs.python.org/3/library/urllib.request.html#module-urllib.request). It has `urllib.request.urlopen` which we'll use to send our GET requests. Most of the time you'll be using the third-party library [requests](http://docs.python-requests.org) but our requirements are slim enough that we can do it vanilla!

Let's check if Google is up, we're expecting a HTTP status code of 200 (OK).

```python
>>> import urllib.request
>>> r = urllib.request.urlopen('https://google.com')
>>> r.status
200
```

Some websites will return 403 (Forbidden) because our user agent will betray us as the bot we are. By default, it will look like `User-Agent: Python-urllib/3.7`. We can get around this by disguising ourselves with a different user agent. Responsible bots will check [robots.txt](http://www.robotstxt.org/) to check the website's rules first!

We start with imports, grabbing everything we'll need. We also store a reference to a user agent string. This one means _user is browsing on a recent Chrome build_.

```python
import sys
import urllib
from urllib import request, parse
from urllib.parse import urlparse, urljoin
from urllib.request import Request
from html.parser import HTMLParser
from collections import deque

search_attrs = set(['href', 'src'])
agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'
```

We've also imported one data structure and declared another one — deque and set. [deque](https://docs.python.org/3/library/collections.html#collections.deque) is a 'list-like container with fast appends and pops on either end'. We'll use this as a simple queue — adding local pages as we find them and scanning them in a first-in-first-out manner. Our [Set](https://docs.python.org/3/library/stdtypes.html#set) usage is simpler, we'll be checking if we've already sent a request to a link before doing so and adding it. We could have used a [List](https://docs.python.org/3/tutorial/datastructures.html) in both instances but it would have less computationally efficient.

We extend the `HTMLParser` into `LinkParser` — the core of our program. We use [super()](https://docs.python.org/3/library/functions.html#super) to refer to the parent constructor that we're overriding.

```python
class LinkParser(HTMLParser):
    def __init__(self, home):
        super().__init__()
        self.home = home
        self.checked_links = set()
        self.pages_to_check = deque()
        self.pages_to_check.appendleft(home)
        self.scanner()
```

When we create an instance of this class we pass it the home page of our website. We store this as `self.home` so we can use it to check if a link we come across is a local page. As you can see at the end of the constructor, we start scanning right away — but what does 'scanning' mean?

```python
def scanner(self):
    # as long as we still have pages to parse
    while self.pages_to_check:

        # take the first page added
        page = self.pages_to_check.pop()

        # send a request to it using our custom header
        req = Request(page, headers={'User-Agent': agent})
        res = request.urlopen(req)

        # check that we're about to parse HTML (e.g. not CSS)
        if 'html' in res.headers['content-type']:
            with res as f:

                # read the HTML and assume that it's UTF-8
                body = f.read().decode('utf-8', errors='ignore')
                self.feed(body)
```

As `feed` parses the HTML, it will encounter tags and call `handle_starttag`, `handle_endtag`, and other methods. We've overridden `handle_starttag` with our own method that checks the attributes for the keys we're looking for. When we come across `<a href="http://google.com">Google</a>` we want to extract the `href` value. Similarly for `<img src="/cute_dog.png">` we want the `src` value.

```python
def handle_starttag(self, tag, attrs):
    for attr in attrs:
        # ('href', 'https://google.com')
        if attr[0] in search_attrs and attr[1] not in self.checked_links:
            self.checked_links.add(attr[1])
            self.handle_link(attr[1])
```

Reminders: To loop through an iterable, you can use `for thing in things:` and in the following block, you refer to each item by the first variable, `thing`. To check whether something is in a set, you use `item in a_set` which returns a Boolean. You can add with `a_set.add(item)`.

```python
def handle_link(self, link):
    # check for a relative link (e.g. /about/, /blog/)
    if not bool(urlparse(link).netloc):

        # fix if we need to, we can't send a request to `/about/`
        link = urljoin(self.home, link)

    # attempt to send a request, seeking the HTTP status code
    try:
        req = Request(link, headers={'User-Agent': agent})
        status = request.urlopen(req).getcode()

    # we're expecting errors (dead resources) so let's handle them
    except urllib.error.HTTPError as e:
        print(f'HTTPError: {e.code} - {link}')  # (e.g. 404, 501, etc)
    except urllib.error.URLError as e:
        print(f'URLError: {e.reason} - {link}')  # (e.g. conn. refused)

    # otherwise, we got a 200 (OK) or similar code!
    else:

        # remove this in production or we won't spot our errors
        print(f'{status} - {link}')

    # build a queue of local pages so we crawl the entire website
    if self.home in link:
        self.pages_to_check.appendleft(link)
```

A complete dead resource crawler in under 50 lines of vanilla code. Praise Python. Many people consider the expansive standard library to be one of the reasons for its popularity. The last thing we need to do is call our class, passing it the first argument after the script.

```python
LinkParser(sys.argv[1])  # e.g. 'https://healeycodes.com/'
```

Here's the [repository](https://github.com/healeycodes/Broken-Link-Crawler) with the final code. The tutorial comments have been cleaned up. Happy botting 😀


Crawling a website for missing resources using the Python standard library.

Build a Python Bot to Find Your Website's Dead Links (Tutorial)


Usually, code golf means one of two things:

- The act of shortening a section of code.
- A wonderfully obscure community on the internet who compete to write _very_ short code. To some, they are seen as a group that 'takes the joke too far' but to me they are **artisans**.

Code golf challenges are not always about writing the shortest code though. Some competitions score on [creativeness](https://codegolf.stackexchange.com/questions/21835/most-creative-way-to-display-42). Some questions are even just asking _is this possible?_ — and the results are beautiful. Later on, we'll see a RegEx that only matches itself.

The most common measure for a code golf competition is the amount of bytes the answer requires, generally this can be understood as the number of UTF-8 characters. `print('Hello, World!')` would be a 22 byte answer. Like golf, the lowest score wins.

For 'Print every ASCII character your program doesn't have', [Umbrella](https://codegolf.stackexchange.com/a/15497/78322) is the current leader for JavaScript with 84 bytes — which is relatively long.

```javascript
// Alerts 'bcdfghijkmnpquvxyz6840'
"!#$%&*+,-13:<=>?@[\]^_`{|}~AERTOWS";alert('BCDFGHIJKMNPQUVXYZ'.toLowerCase()+95*72)
```

Let's see an extreme example, the [GolfScript](https://codegolf.stackexchange.com/a/12376/78322) entry. Language reference [here](http://www.golfscript.com/golfscript/quickref.html).

```golfscript
# Outputs the remaining ASCII characters
{`),32>^.}.~
```

GolfScript is _quite_ hard to follow. Most of the time, we write code for humans — not machines. In code golf one writes to score points. The standard programming languages, while often used, can be restrictive. For example, I have never seen a C# or Java answer come top in a shortest-answer-wins challenge.

#### Esoteric languages

An esoteric language:

- experiments with [weird ideas](https://esolangs.org/wiki/Hexagony).
- might be hard to [program in](https://esolangs.org/wiki/Unreadable).
- might even be designed as a [joke](https://esolangs.org/wiki/Brainfuck).
- is likely [not practical](https://esolangs.org/wiki/JSFuck).

Much like mainstream programming languages have evolved over time, so have code golfing languages. These languages are designed with different goals. Some want to be able to write the [smallest programs](https://esolangs.org/wiki/Pyth). Some are written [just because](http://esolangs.org/wiki/Shakespeare).

[Hexagony](https://github.com/m-ender/hexagony) is a two-dimensional programming language. Let's take a look at a program that prints 'Hello, World!'. You can run this online at [tio.run](https://tio.run/##y0itSEzPz6v8/19BQcFDwVohVcGaS0EhB8hKAWItLiBRBMT5QBzOBRKGsLUUTLgUjIHQWsFBQU/BDqjHWsEIyLdRiAGyFUzAavT//wcA) a.k.a. Try It Online — an open source playground of **636** languages.

```hexgony
   H ; e ;
  l ; d ; *
 ; r ; o ; W
l ; ; o ; * 4
 3 3 ; @ . >
  ; 2 3 < \
   4 ; * /
```

Intrigued? One of my favorite pieces of code golf writing is the language creator's [description](https://codegolf.stackexchange.com/a/57706/78322) of his primality testing program. Reading about these solutions, and experimenting with my own, exercises my problem-solving muscle. It helps me to learn the edges of my languages.

#### Community

Similar to the open source community, code golf is often a collaborative effort. Languages are developed in the open, and questions will find helpful answers. Competition 'entries' will also receive polite assistance. I remember my first attempt at code golf, I received welcome messages and suggestions for shaving off bytes.

The [Programming Puzzles & Code Golf](https://codegolf.stackexchange.com/) section of StackExchange is the most active code golf community on the internet and this is where you can find many of these languages in use. [Esolangs](https://esolangs.org/wiki/Main_Page) is a community wiki where the most esoteric languages are covered in-depth. The #codegolf tag on Twitter is slow but welcoming.

#### RegEx that only matches itself

As promised, [this](https://codegolf.stackexchange.com/questions/28821/regex-that-only-matches-itself) post is a fun one. The poster begins by stating that _this may well be impossible_ before elaborating on the challenge. Fortunately, [jimmy23013](https://codegolf.stackexchange.com/users/25180/jimmy23013) shows us that not only is this challenge possible but is solvable quite tersely indeed.

```bash
# RegEx that only matches itself
# in (PCRE) Perl Compatible Regular Expressions
<^<()(?R){2}>\z|\1\Q^<()(?R){2}>\z|\1\Q>

# A more widely compatible version:
/^\/()(?R){2}\/\z|\1\Q^\/()(?R){2}\/\z|\1\Q/
```

#### Is this number prime?

Let's start with a Python [answer](https://codegolf.stackexchange.com/a/57650/78322) (59 bytes). It's as naive as prime-checking gets but is creatively short. It demonstrates a typical code golf answer: _solve the problem the long way round then apply syntax tricks_. (Comments my own).

```python
# take a number via stdin
n=int(input())
# build an array of this number's factors from 1 to n
# if there is only one factor (the number one) then it is prime!
print([i for i in range(1,n)if n%i==0]==[1])
```

The leading JavaScript answer is a little harder to pick apart.

```javascript
// Alert true or false given an input number
alert(!/^(?!(..+)\1+$)../.test(prompt()))
```

The [poster](https://codegolf.stackexchange.com/a/57692/78322) tells us that it uses a 'cool unary regex to determine primality'. Further research found [this](https://iluxonchik.github.io/regular-expression-check-if-number-is-prime/) ~5000 word article which explains said RegEx expression. As you've probably started to realize, code golf is like programming in that it's a rabbit hole that doesn't end. But it does get more rewarding the further you dive in.

#### Elsewhere on the web

[JS1k](https://js1k.com/) is a JavaScript demo competitive for 'fancy pancy JavaScript demos' that come in under 1024 bytes. They are very visually impressive.

There's also the [The International Obfuscated C Code Contest](https://www.ioccc.org/) which began in 1984. It's closer to treasure hunting than golf though! Here's a maze generator entry from 1995, by Carlos Duarte ([source](https://www.ioccc.org/years.html#1995_cdua)):

![1995/cdua](c-maze.png)

Honorary shoutouts to two of my favorite esolangs [05AB1E](https://github.com/Adriandmen/05AB1E) and [Jelly](https://github.com/DennisMitchell/jellylanguage). They both place well in challenges and are usually accompanied by well-written explanations which are a great entry point for learning them.


Answered: What the Heck is Code Golf?


_Dust off your ternary expressions, we're going in._

One-liners are tricky to maintain (and sometimes even hard to understand) but that doesn't stop them from being cool as hell. There's a certain satisfaction that comes after writing a terse solution.

This is a collection of some of my recent favorites. They will all run in your dev console so pop it open and try them out. I hope you'll share some of your own favorites in the comments!


#### Calendar Hacking

Ali Spittel [tweeted](https://twitter.com/ASpittel/status/1110548407800815617) this out recently. It solves a problem I've faced multiple times. If you swap the minus for a plus, it gives you the next seven days.

```javascript
// Create an array of the past seven days, inclusive
[...Array(7).keys()].map(days => new Date(Date.now() - 86400000 * days));
```


#### Random ID generation

This is my go-to function for creating UUIDs when prototyping. I've even seen people using it in production in the past. It's not secure but ... there are worse random generation functions out there.

```javascript
// Generate a random alphanumerical string of length 11
Math.random().toString(36).substring(2);
```


#### Quines

A [quine](https://en.wikipedia.org/wiki/Quine_(computing)) is a program that outputs its own source code. Quines have always fascinated me. I've got pretty close to completing my own quines a couple of times in different languages but details are the name of the game.

I've picked some winners for you. Credit to [Mama Fun Roll](https://codegolf.stackexchange.com/a/60148/78322), [PleaseStand](https://codegolf.stackexchange.com/a/270/78322), and [Peter Olson](https://codegolf.stackexchange.com/a/3173/78322) respectively for these three.

```javascript
// $=_=>`$=${$};$()`;$()
$=_=>`$=${$};$()`;$()

// eval(I="'eval(I='+JSON.stringify(I)+')'")
eval(I="'eval(I='+JSON.stringify(I)+')'")

// For those who like their quines via alert
// (function a(){alert("("+a+")()")})()
(function a(){alert("("+a+")()")})()
```


#### Scrape query parameters

Talk about non-maintainable code. This converts the page's query parameters to an object in 78 bytes! Thanks to Alex Lohr for code golfing it.

`?foo=bar&baz=bing` becomes `{foo: bar, baz: bing}`

```javascript
// Set the current page's query parameters to `q`
q={};location.search.replace(/([^?&=]+)=([^&]+)/g,(_,k,v)=>q[k]=v);q;
```

I'd like to see a minifier work that hard. 


#### Working clock

With a sprinkle of HTML, you can create a working clock with source code you could read out in one breath. I wrote this after a challenge from a co-worker. It ticks every second, updating the page with the current time.

```html
<body onload="setInterval(()=>document.body.innerHTML=new Date().toGMTString().slice(17,25))"></body>
```


#### Shuffle an array

Until Pythonistas show up with their `import random`, `random.shuffle(array)` solution, we're going to enjoy what we have. This has the bonus of having an infinitesimal chance of being an infinite loop (implementation depending). [Don't use in production](https://www.robweir.com/blog/2010/02/microsoft-random-browser-ballot.html).

```javascript
// Return a shuffled copy of an Array-like
(arr) => arr.slice().sort(() => Math.random() - 0.5)
```


#### Generate random hex code

ES7's [padEnd](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/padEnd) is a blessing. Along with `padStart`, they've made number to string conversions that much easier. Writing hex values right into JavaScript code is always pretty neat too.

```javascript
// Generate a random hex code such as `#c618b2`
'#' + Math.floor(Math.random() * 0xffffff).toString(16).padEnd(6, '0');
```


#### Pass the interview in style

The [infamous](https://blog.codinghorror.com/why-cant-programmers-program/) interview question answer but codegolfed. I researched and I don't think it can get any shorter than this.

```javascript
for(i=0;++i<101;console.log(i%5?f||i:f+'Buzz'))f=i%3?'':'Fizz'
```


#### Remove duplicates

This only works with primitives but it's still nifty. [Set](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Set) takes any iterable object, like an array `[1,2,3,3]`, and removes duplicates. The spread operator makes that set `[1,2,3]`.

```javascript
// Remove duplicates from the iterable `arr`
[...new Set(arr)]
```


#### A keyboard so real you can taste it

Okay, I don't really count this as a one-liner but it's too good not to include. A masterful codegolf solution by [edc65](https://codegolf.stackexchange.com/a/103226/78322). It's terse to a fault and codegolfed within an inch of its life but we should bask in its glory.

```javascript
// Return a ***3D*** ASCII keyboard as string
(_=>[..."`1234567890-=~~QWERTYUIOP[]\\~ASDFGHJKL;'~~ZXCVBNM,./~"].map(x=>(o+=`/${b='_'.repeat(w=x<y?2:' 667699'[x=["BS","TAB","CAPS","ENTER"][p++]||'SHIFT',p])}\\|`,m+=y+(x+'    ').slice(0,w)+y+y,n+=y+b+y+y,l+=' __'+b)[73]&&(k.push(l,m,n,o),l='',m=n=o=y),m=n=o=y='|',p=l=k=[])&&k.join`
`)()
```

It prints:
![Keyboard made out of ASCII text](ascii-keyboard.png)

Amazing.


Dust off your ternary expressions, we're going in.

JavaScript One-Liners That Make Me Excited


We're going to write a Reddit bot that listens for specific phrases and responds to comments that contain them. We'll learn how the Reddit API can be interacted with in Python, while making a bot buddy of our own.

#### Bottiquette

You've probably heard of the [three laws of robotics](https://en.wikipedia.org/wiki/Three_Laws_of_Robotics) but have you heard of Reddit's [bottiquette](https://www.reddit.com/wiki/bottiquette)? It's a list of rules that will help you create a polite bot that won't get in anyone's way. The most important rule is not to impersonate a human with your bot. While it's okay to cast up- and down-votes, the command to do so must come directly from a human.

#### Setup

You'll need [PRAW](https://praw.readthedocs.io/en/latest/index.html), the _The Python Reddit API Wrapper_. Install this package with: `pip install praw`. Its got extensive documentation, and there's also a [r/redditdev](https://www.reddit.com/r/redditdev/) subreddit should you need more inspiration or assistance.

Create a __script__ [Reddit app](https://www.reddit.com/prefs/apps/), and note down the client id and client secret.

![Reddit's Create Application page](rb-app.png)

Make sure you don't commit these to a repository! With Python, you can access environmental variables with `os.environ`, a mapping object. Below is a hello world PRAW app. Since our bot will be performing account actions, we'll also need the account's username and password. With passive bots, this isn't necessary. There are also [other ways to authenticate](https://praw.readthedocs.io/en/latest/getting_started/authentication.html).

```bash
# Setting environment variables on different platforms:
Linux/OS X  - export ENV_VAR='data'
Windows Cmd - ENV_VAR='data'
Powershell  - $env:ENV_VAR='data' 
```

```python
import os
import praw

# a reddit instance
reddit = praw.Reddit(client_id=os.environ['CLIENTID'],
                     client_secret=os.environ['CLIENTSECRET'],
                     user_agent='Reddit bot tutorial v0.1-1',
                     username=os.environ['CLIENTUSER'],
                     password=os.environ['CLIENTPASS'])

# with incorrect authentication information, this will error with:
#   prawcore.exceptions.OAuthException: invalid_grant error processing request
print(reddit.user.me())
```

Since our sensitive information is defined elsewhere, we can commit this without leaking any details. It's also easier to test and let's other developers work more easily with this code.

#### All the comments

This is a fun one to do. My challenge to you: try and read every comment as it's posted to Reddit. This really gives you an idea of the scale of data that's available.

```python
# with our reddit instance, get an endless stream of comments
for comment in reddit.subreddit('all').stream.comments():
    print(comment.body)
```

Here, `comments` is a generator function that yields new comments as they become available. As with most APIs, there are rate limits with the Reddit API. Lucky for us, PRAW manages these in the background for us and keeps us in check.

We want to filter comments for specific phrases. Along the same lines as our environment variables, we're also going to genericize this feature. [argparse](https://docs.python.org/3/library/argparse.html), part of Python's standard library, allows command line arguments to be easily configured.

```python
import argparse

# setup argparse
parser = argparse.ArgumentParser(description='Reply to certain Reddit comments')
parser.add_argument('--search', dest='search',
                    help='Search for these comma-delimited phrases. E.g., "one phrase, another phrase"')
args = parser.parse_args()

# split by commas, strip extra white space
search = [phrase.strip() for phrase in args.search.split(',')]

# python args.py --search "test phrase, two, three"
# prints: ['test phrase', 'one', 'two']
print(search)
```

#### Replying to comments

Reddit's [r/test](https://www.reddit.com/r/test/) is a good battleground to try this out. We'll use [Comment#reply](https://praw.readthedocs.io/en/latest/code_overview/models/comment.html#praw.models.Comment.reply). Since we'll be searching comment bodies for our phrases, we'll already have a reference to a comment object, and we can simply use its `reply` method. Otherwise, you can find create a comment object with an id.

```python
# with a comment reference
comment.reply('hello! signed, a bot.')

# find a comment by id
comment = reddit.comment(id='dxolpyc')
comment.reply('hi again!')
```

You may need two accounts to properly test this, as newly created Reddit accounts can't post two comments instantly, i.e., posting a target phrase and then a posting reply to that comment.

#### Where we are so far

Let's check how our bot is shaping up. You'll see I've used `argparse` to genericize the search phrases. We're making a configurable bot, rather than a hardcoded one that's harder to extend and work with in the future.

```python
import os
import praw
import argparse

# setup argparse
parser = argparse.ArgumentParser(description='Reply to certain Reddit comments')
parser.add_argument('--search', dest='search',
                    help='Search for these comma-delimited phrases. E.g., "one phrase, another phrase"')
parser.add_argument('--reply', dest='reply',
                    help='Reply to target comments with this phrase')
args = parser.parse_args()

# our search phrases, also works with a single word
search = [phrase.strip() for phrase in args.search.split(',')]
# our reply
reply = args.reply

# reddit instance
reddit = praw.Reddit(client_id=os.environ['CLIENTID'],
                     client_secret=os.environ['CLIENTSECRET'],
                     user_agent='Reddit bot tutorial v0.1-1',
                     username=os.environ['CLIENTUSER'],
                     password=os.environ['CLIENTPASS'])

# an endless stream of comments
for comment in reddit.subreddit('all').stream.comments():
    for phrase in search:
        if phrase in comment.body:
            comment.reply(reply)
            # for debugging
            print(f'Replied to {comment.id}!')
            break

```

#### Where to go from here

- Clone the [tutorial repository](https://github.com/healeycodes/Reddit-Bot-Tutorial) and get hacking!
- Get the poster's name and use it in your reply: `comment.author.name`
- Host your bot on a Raspberry Pi or in the cloud with [Heroku](https://devcenter.heroku.com/categories/python-support).
- Connect your bot to a backend service, like I did with my [EmojiStreamer](https://github.com/healeycodes/EmojiStreamer) project, which streams every single emoji posted to Reddit.

#### More

I recently joined [coding coach](https://mentors.codingcoach.io/). If you're developing a Reddit bot, or need dev assistance in general, I can help out!


The basics of building a Reddit bot that listens for phrases and replies to them.

Need a Friend? Write a Reddit Bot with Python to Follow You Around (Tutorial)


Travis CI offers free test builds for open source projects on GitHub. You'd be a fool not to take them up on their offer. Their email alerts have saved my projects many times before.

In this tutorial, we will be setting up an Express app for continuous integration (CI). Whenever we commit to our `master` branch, Travis CI will clone our repository, spin up a cloud build of linux, install any required dependencies, and run our tests! Hopefully, they pass!  If not, we'll be alerted.

#### Install

Set up a quick `package.json` file with: `npm init -y`. Then grab Express: `npm i express --save` as well as supertest and Jest, our development dependencies: `npm i supertest jest --save-dev`.

Or clone the [repository](https://github.com/healeycodes/earn-a-build-passing-badge), which serves as a live example of the project!

#### Taking the App Out of Express

A default hello world application with Express looks like this:

```javascript
// app.js

const express = require('express');
const app = express();
const port = 3000;

app.get('/', async (req, res) => res.status(200).send('Hello World!'));

app.listen(port, () => console.log(`Our app listening on port ${port}!`));
```

This works for manual testing. We can run this application and check that the right pages are being returned — but what if we have 50 pages with complicated logic? We want to automate this process. The first step is exporting our `app` object. When we run our tests, we don't need a live HTTP server.

Let's alter our hello world application.

```javascript
// app.js

const express = require('express');
const app = express();

app.get('/', async (req, res) => res.status(200).send('Hello World!'));

// Don't listen, just export
module.exports = app; // <--
```

Great. But how do we launch our application now? We'll use [separation of concerns](https://en.wikipedia.org/wiki/Separation_of_concerns) and place the call to `listen()` in another file called `server.js` (which also uses the `app` object!)

```javascript
// server.js

const app = require('./app');
const port = 3000;

app.listen(port, () => console.log(`Our app listening on port ${port}!`))

```

To launch our application, we now use `node server.js`. Let's add that to our `package.json` so people can simply use `npm start`. By default, Node.js will look for a `server.js` file but let's be explicit.

```json
"scripts": {
  "start": "node server.js"
},
```

#### The Tests

A common pattern is to place your tests inside a folder called `__tests__` in the root directory. Another pattern is to repeat the names of the files being tested with `.test` inserted before the `.js`. Thus, `__tests__/app.test.js`.

We'll be using Jest as a test runner. Jest will look inside `__tests__` as part of its default search and will run any test files it finds. You can use a custom test search with `--testMatch`.

> By default [Jest] looks for .js, .jsx, .ts and .tsx files inside of `__tests__` folders, as well as any files with a suffix of .test or .spec (e.g. Component.test.js or Component.spec.js). It will also find files called test.js or spec.js.

Inside our tests,  supertest will mock requests to our `app` object. Mocking requests is faster and more predictable than launching a server and using live requests. It also makes it easier to write [setup and teardown](https://jestjs.io/docs/en/setup-teardown) methods when they're required.

```javascript
// __tests__/app.test.js

const app = require('../app');
const request = require('supertest');

// `describe` is used for test components
describe('GET /', () => {
    
    // `it` is for individual tests
    it('responds with 200', async () => {
        await request(app)
            .get('/')
            .expect(200); // If the status code is not 200, this test will fail
    });
})
```

Let's add another line to our `package.json` so that our tests can be run with `npm test`. The reason we use `start` and `test` aliases is so that our software is predictable for developers picking it up for the first time, and so that it plays nice with other packages.

```json
"scripts": {
  "start": "node server.js",
  "test": "jest"
},
```

`npm test` yields the following output.

```bash
 PASS  __tests__/app.test.js
  GET /
    √ responds with 200 (39ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        2.681s
Ran all test suites.
```

#### Travis CI

Let's [get this code into a repository](https://help.github.com/en/articles/create-a-repo) on GitHub, and install the [Travis CI GitHub App](https://github.com/apps/travis-ci). Make sure that the repo you're testing has Travis CI enabled.

![Repository access](travis-repo-permissions.png)

As the Travis CI [tutorial](https://docs.travis-ci.com/user/tutorial/) tells us:

> Add a .travis.yml file to your repository to tell Travis CI what to do.

In our instance, it's that simple. We don't need to specify any additional settings beyond which version of Node.js we want the test build to use. Travis CI will use the default testing alias `npm test`.

```yml
# .travis.yml

language: node_js
node_js:
 - lts/* # Long Term Support
```

Committing and pushing this file to GitHub will queue up a test build immediately. You can watch the builds execute live at `travis-ci.com/{your-username}/{your-repo}`, and review them later to see where things went wrong. Use this page to grab the markdown for your Travis CI build status badge too!

![A live build](travis-build-passing.png)


How to setup Travis CI for a Node/Express project.

Earn a Build Passing Badge on GitHub! Testing Your Express App with Travis CI (Tutorial)



Notice how I said _live cam_ and not _live stream_? It's because we'll be dealing with normal images. Go back about 15 years and this technology was exciting. Live weather cams and nature cams were deadly cool. Image-based live streams are also a little more robust when the connection between our Raspberry Pi and web server can be unreliable (e.g., WiFi).

![My Pi!](my-pi.jpg)


This method can hit around 5-10fps depending on the image resolution. There's a `benchcamera.py` script in the [repository](https://github.com/healeycodes/Raspberry-Pi-Live-Cam) that you can run on your Pi to test. Full installation instructions for this project can also be found in the repo.

Let's work back to front, starting with the web server. The Flask app we'll be coding can be hosted [anywhere](https://www.google.com/search?q=host+flask+app+free). Its job is to receive images from the Pi, store the latest image and serve this image to clients. Temporarily storing this image is the trickiest part of this project. We want our app to be scalable to, say, a few hundred clients. Flask apps are normally hosted via servers like uWSGI or Gunicorn and they handle the load in part by using threads.

A different thread of the app is created for each user. These instances can't share state at the application level. We solve this by storing the image data in a local SQLite database called `global.db`. Dealing with multiple readers and writers is easy for databases. They handle it by using _locks_. This ensures that our Flask app won't ask for an image that is halfway through being written to the database and then send corrupted data to the client.

#### Flask App

```python
# app.py

import os
import sqlite3
from flask import Flask, request, g
app = Flask(__name__)

DATABASE = 'global.db'


# helper method, allows database access within a controller
def get_db():
    db = getattr(g, '_database', None)
    if db is None:
        db = g._database = sqlite3.connect(DATABASE)
    return db


# create our database structure, which is akin to a dict with one key
def init_db():
    db = sqlite3.connect(DATABASE)
    cur = db.cursor()
    cur.execute('''CREATE TABLE IF NOT EXISTS store
                (id INTEGER PRIMARY KEY, image BLOB)''')
    cur.execute("INSERT OR IGNORE INTO store (id, image) VALUES (1, '')")
    db.commit()
    db.close()


# initialize db
init_db()


# close the database connection after every request ends
@app.teardown_appcontext
def close_connection(exception):
    db = getattr(g, '_database', None)
    if db is not None:
        db.close()
```

When we start our app, we need to make sure that there is a table, and a slot in that table to store images. At the same time, we don't want to overwrite existing images. So we create a table only if it doesn't exist already `CREATE TABLE IF NOT EXISTS` and then we create the one row that we will be using with similar logic `INSERT OR IGNORE`.

The upload controller (also known as a _handler_) exists on the root path and only accepts POST requests. It also checks for an environmental value called `PASSWORD`. (Query parameters are insecure over HTTP but [more secure](https://stackoverflow.com/questions/2629222/are-querystring-parameters-secure-in-https-http-ssl) over HTTPS).

```python
# recieve images and write to db as BLOB if the password is correct
@app.route('/', methods=['POST'])
def update_image():
    db = get_db()
    cur = db.cursor()
    if request.args.get('password') != os.environ['PASSWORD']:
        return '', 400
    else:
        image = [request.data]
        cur.execute(
            "UPDATE store SET image=? WHERE id=1", image)
        db.commit()
        return '', 200
```

We can now receive images! We will only store one image: the latest one. Serving that image to any client that makes a request is straightforward.

```python
# share images naively and let browsers interpret the BLOB as jpeg
@app.route('/live.jpeg')
def get_image():
    cur = get_db().cursor()
    image = cur.execute("SELECT image FROM store WHERE id=1").fetchone()[0]
    return image, 200
```

If we host a link to this image as `<img src="/live.jpeg" />` our clients will see a still picture. The longer they stay on the page, the more out-of-date this image starts to get. They could always refresh to force an update but that's wasteful. A simple solution is to use JavaScript to force a refresh of just the image using `setInterval`. We can use this logic to build a live cam widget. Our test route uses this so we can make sure our images are being received.

```python
# test route that mimics having a live cam
@app.route('/test')
def test_image():
    return '''<img src="/live.jpeg" /><script>setInterval(() =>
        document.querySelector(\'img\').src = \'/live.jpeg?\' + Date.now(), 150)</script>'''
```

We cache-bust to make sure that the browser doesn't think that it already has the image source in its cache. One way to cache-bust is to add a unique query string to the end of a request (that ultimately does nothing) like the current date in milliseconds: `?1552907804662`.

```bash
$ PASSWORD='123'
$ FLASK_APP=app.py flask run
 * Running on http://localhost:5000/
```

For installation and running questions related to Flask, see their [docs](http://flask.pocoo.org/).

#### Raspberry Pi Script

The script that runs on the Pi will capture images and send them via a POST request. We install our modules with `pip install requests picamera`. The script is run via `python camera.py` or `python3 camera.py`.

```python
# camera.py

import io
import time
import requests
import picamera

# change this line to your Flask app's address!
url = 'http://192.168.1.100:5000/?password=123'

framerate = 90
quality = 100
res = (1280, 720)
with picamera.PiCamera(framerate=framerate, resolution=res) as camera:
    time.sleep(2)  # camera warm-up time
    while True:
        try:
            image = io.BytesIO()
            camera.capture(image, 'jpeg', quality=quality, use_video_port=True)
            r = requests.post(url, data=image.getvalue())
            # place a `time.sleep` here if you want a slower live cam
        except:
            time.sleep(5)  # wait for WiFi/server to come back

```

You will probably want to run this as a [startup script](https://www.google.com/search?q=rc.local) for maximum uptime. If there's a network problem, the script will keep attempting to post an image until it gets through. If there's a problem with your Pi, our Flask app will keep serving the latest image it received. Robust!


#### Possible Improvements

Some thoughts to improve the project but make it a worse tutorial.

- Add a listener to SQLite and cache the latest image in memory.
- Upload images via WebSocket (save bandwidth and latency).
- Serve images to clients over WebSocket as they arrive (same as above).
- Use Flask for uploading images, serve images as a static file via HTTP server (scales infinitely).
- When receiving images, scale them to different resolutions (e.g., `/live800x600.jpeg`).

<br>

Reach out, or raise an issue on [GitHub](https://github.com/healeycodes/Raspberry-Pi-Live-Cam), if you're having problems 🛠️.


A lo-fi solution for streaming images from your Raspberry Pi to the web.

Coding a Live Cam for the Raspberry Pi in Python (Tutorial)


After watching my flappy bird bounce and respond to my keystrokes, a fire was lit inside me. I just had completed a Unity tutorial at age 22. It had been a childhood dream to create software — to be an _engineer_. But it never really happened. So I decided to make it happen.

![Flappy bird](flappy-bird.gif)

Video game tutorials seemed pretty accessible so I kept that up for a few weeks. Brute force learning about functions, types, and classes. I delved away from the instructions and tinkered where I could but it didn't feel like I was actually creating anything, only copying. I was waiting for the results from my Creative Writing degree to come in. I had been working with some web technologies throughout my time at university but it was small potatoes compared to where I wanted to go, to where I am now.

I read an article about a conversion Masters in Computer Science. These are full Masters programmes that aim to bring passionate students up to the level of a bachelors graduate and beyond, including a heavy thesis. These courses really push you and many institutions have 'suggested' learnings for before term starts. (They're required if you want to excel). If these courses sound intense — maybe even scary — it's because they are. Often times I found myself researching concepts in order to learn other concepts in order to go back to the lecture notes!

I applied to many but ended up choosing a distance-learning MSc. I respected the institution and the alumni buzz grabbed my attention. The first modules were in Java so over the summer I prepared by completing both parts of the popular Java MOOC [Object-Oriented programming with Java](http://moocfi.github.io/courses/2013/programming-part-1/). I knew that I would be choosing any optional web-related modules so I completed most of [freeCodeCamp](https://www.freecodecamp.org/) as well. Alongside this, I was also entering game jams and creating [small games](https://healeycodes.com/projects/) in Unity with C#. This whole journey was possible with one rule: make things. Small programs, CLI programs, goofy games, silly bots.

Any fears that I had about the online delivery of my course were soothed by the active Discord chatrooms that the students maintained and the rapid pace at which professors would reply to questions — as well as the interactive live-streamed lectures. A year-long intensive course was exactly what I needed to support everything I was learning from building my [own applications](https://healeycodes.com/projects/). The module choices were flexible but still forced us to learn the fundamentals like algorithms and data structures, databases, and practical software development.

Before long, I had to start thinking about applying for jobs. This is where things got scary. Everything I had done up to this point was without risk. If you study hard, you usually get good grades, if you keep trying to build toy projects and don't give up then you will usually succeed. But applying for jobs? Interviewing? Failure is part of the process.

I stepped away from the safety of my fun apps and curriculums and I built three portfolio projects. Each one aimed to solve a problem that I or someone else in my community was facing. They were all full stack and unit tested, they had databases attached and were hosted on real servers at real hostnames — I wanted all the trimmings. I ended up building a website analytics service, a low-latency pair-programming solution, and a website for enabling game jam developers' to track user highscores. All open sourced.

Cover letters seem to catch a bunch of flak online but I think most of my interviews were related to the personal paragraph or so that I wrote for each company. If I didn't feel like what I was putting down was sincere then I started over. Often, the things that I spoke about were brought up on the phone or in-person. I applied country-wide and had a ~10% response rate for phone interviews. I spoke to hiring managers as I paced around my room and surprisingly passed all of them. I stayed up late working on coding tests. Chipping away at the rockface of documentation. The timed algorithm tests caught me sweating too.

In-person interviews were more exciting. After all this time, I finally got to speak to real people about my programming passions! I had been couped up in my room for so long that I probably came across as over-excited. I failed most of them. Which was the wrong way to think about it at the time. There are some interviews that you just can't pass. I walked out of each one thinking: that was the one. Each one felt perfect until the email finally came.

The interview that I ended up passing (for the job that I came home from earlier today) felt like every other one. I walked out grinning and called my fianceé and told her that this one, _this one_ was different. She laughed, hopeful but cautiously optimistic. I was surprised that my last-minute reviewing of HTTP status codes paid off — that part felt a bit surreal.

I've been studying and coding ever since. In my last post, I spoke about what my [day-to-day life](/webdev/career/javascript/2019/03/14/day-in-the-life.html) is like at this office. In my next, I might write about what I think makes a good portfolio project 🕹️.


An overview of how I learned to code. Also a bit on my experience during my CS Masters.

How Flappy Bird Got Me My Start in Software


*This blog is for my past self who worried about what they would be doing day-to-day after graduating.*

Last year, I completed my Masters in Computer Science and I've been working as a full stack software engineer ever since! I'm going to talk about how I work on projects outside of work, how I get my tasks done at work, and the miscellanea in between.



#### 7:00am

An iPhone alarm, maybe two.

#### 7:10am

I eat cereal and read [hackernews](https://news.ycombinator.com/) and some programming subreddits, as well as tech/regional news, and some Twitter. Sometimes I'll tweak one of my projects, something small — a docstring or a section of CSS — and push those changes to GitHub. It's good to continuously work on projects and carry them around in my mind so that ideas can come to me throughout the day. I use [GitHub Desktop](https://desktop.github.com/) at home because my branches don't get too complex, and it's great for reviewing small diffs.

#### 7:30am

I wake my fianceé with a coffee. She's a heavy-sleeping student who depends on a large cup to start her day.

I cycle to work and it takes around 15 minutes. There's a cycle path most of the way. If it's raining, I'll have a change of clothes in my bag but I don't change very often.

#### 7:55am

I'm usually the first person at the office and I turn on the lights for the engineering section. I catch up on any emails that I missed yesterday evening. I leave earlier than most and don't have any work communications on my phone.

I like to start the day by reviewing code. So ideally there are some fresh diffs for me to pick through. Our stack has a lot of breadth and I find myself reviewing anything from JavaScript (Node.js and browser) to C++ to PHP. We've got Angular and React (both TypeScript), as well as some crazy legacy stuff.

I'll perform some default coding work here also — tackling a ticket, merging branches, and a dash of technical writing.

#### 9:30am

People slowly arrive until our scrum standup meeting. I'm normally there first, I obsessively carry around a pen and post-it notes, a habit from when I was new and didn't want to forget anything. At scrum meetings, you say what you did yesterday, if anything is blocking you, and what you will be doing that day. Our 'scrum master' will direct or correct people as needed. It's a fun, relaxed atmosphere. As well as coders, there will sometimes be product management, technical writers, and support staff.

On average, I have about 0.5 meetings a day. These can be design meetings, catch-ups, [sprint](https://en.wikipedia.org/wiki/Scrum_Sprint)-related, or general company events. For the rest of the time, I'm coding by myself and helping or being helped by those around me. Sometimes we'll do some pair-programming when we think it will be beneficial. We all have a lot of autonomy. We work hard to get things done in time for the end of the sprint but if they don't get done then that's fine too, resources will be reallocated, no one gets in trouble for missing deadlines as long as they're open about it. In fact, I can't remember anyone getting in trouble for anything.

#### 12pm

Although we have semi-flexible working hours, the whole company takes lunch at 12 for an hour. Some people take a shorter lunch to get home earlier. I'll take the first 15 minutes to have a snack, call my fianceé, and stretch my legs. For the remaining 45 minutes, I will study or write. I use this time to read documentation or create 'hello world' applications in new tech stacks. I do tutorials, read technical blog posts, and take coding challenges. If I'm writing, I do it longhand with a biro in a A4 lined pad. All of this takes me out of my own head. My lunchtimes feel long and relaxing.

#### 1pm

For the rest of the day, I continue with default work. At around 3pm, it's clear that everyone's most productive hours are over. People don't work slower on purpose, it just seems to happen. We manage this by doing non-intensive but still time-consuming work. Testing applications, writing documentation, taking non-essential meetings.

#### 4:30pm

My day is over. I commit to any branches I have open, even if it's just to say "End of day commit. I'm mid-way through building X". I play games with some of my deskmates so it won't be the last time that I speak to them that day. I wipe the rain off my bike's seat and cycle home. I have to go through a wind-valley, so it often feels like three-times the distance. I've been slowed down to walking speed before.

#### 5:30pm

After eating dinner with my fianceé and catching a show, I'll load up one of my projects and start developing. I've been tinkering around with my website recently. That's my guilty pleasure because it's never "productive" or at least it doesn't feel like it. The things that give me an endorphin rush are coding, writing, and designing. I like testing too, surprisingly, but I build my web applications to be testable from the get-go. If there's a longhand blog to write up I'll try and get it out of the way early so there can be some time between finishing it and editing it. My undergrad was in Creative Writing so writing technical blog posts is probably the easiest part of my day. It's an amalgamation of the two things I love most.

#### 8pm

I'll take a break at eight for some video games unless I'm pushing to complete a personal task. My main game at the moment is Overwatch on PC. I will still feel a swirling of ideas and code-flow-structures as I play but I suppose it's at this point that I start to "unwind" but not that it feels like there's anything to unwind from.

After playing games, I'll edit blog posts, and add small things to my projects — most of the good ones I open source on [github.com/healeycodes](https://github.com/healeycodes). By adding small things, I mean refactoring small functions, adding some UI flair. I make sure not to start anything major in the evening. I will sometimes do a tutorial if I feel like the day has been unproductive — I'll choose an area of compsci that I don't normally touch, for fun.

In the evenings, I like to read. Lately, I've been reading [Grokking Algorithms](https://www.manning.com/books/grokking-algorithms) which is a very accessible but still in-depth book on algorithms and data structures. As far as fiction goes, I've been reading Ben Lerner as well. I work out, shower, and hit the hay.

#### Weekends

I try and complete a small project end-to-end every weekend. This last weekend, it was [Project Mood](https://github.com/healeycodes/project-mood).

<br>

#### Backmatter

Recently, I've joined Twitter to become more of a part of the developer community (I want to give back!) — I'm at [@healeycodes](https://twitter.com/healeycodes). I'm also looking for fun open source projects (big, small, or tiny). Feel free to reach out.

🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴🚴

Living, working, building, and drinking too much coffee.

A Day in the Life of a Mediumly Productive Software Engineer


A little over six months ago, I completed my thesis on WebSocket server performance. I've had a chance to use them in (global) production, and lemme tell you, this technology kicks ass!

#### The Past

The model of the internet is request -> response. You send a request for a page by typing an address in your URL bar -> a web server then responds with data. The information you sent, and the information you received, has what's called a *header*, the rest (what you see on the page) is called the body.

![Request/response description](reqres.png)

These headers contain critical information about the data transfer. However, as we'll see later on, these headers and this request/response cycle can be problematic when we want to do things in real-time.

#### The Present

We've been discussing the HTTP protocol. *WebSockets* refer to the [WebSocket protocol](https://tools.ietf.org/html/rfc6455). Like `http://` and `https://`, we have `ws://` and `wss://` for WebSockets. (The *S* stands for secure). WebSockets are bidirectional. This means that messages can be sent both ways. There doesn't need to be a request and a message doesn't need to be a response. Once a connection has been made, you can sling data whichever way you wish. (Under the hood, here we're talking about [data frames](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers#Exchanging_Data_Frames)).

It all starts with the handshake. When you type in your developer console,

```javascript
    new WebSocket('ws://example.com:3000');
```

a special HTTP/S request is sent. It's special because, [among other quirks](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers#The_WebSocket_Handshake), the header contains `Upgrade: WebSocket` and `Connection: Upgrade`. The server, should it accept your hand in bidirectional matrimony, sends back a response with a similar header with the same two header key/values. At this point, the security you'll be using is negotiated. Both sides are saying Okay, let's do this.

On the client, opening a WebSocket connection looks like this.

```javascript
const socket = new WebSocket('ws://example.com:3000');

// Print incoming messages to the page
socket.onmessage = (msg) => document.body.innerText += msg;

socket.onclose = (event) => alert('uh oh');
```

On the server, we need to listen for these connections. We'll use the unofficial standard WebSockets library for Node.js called [websockets/ws](https://github.com/websockets/ws) or *ws* for short.

```javascript
// Include the library
const WebSocket = require('ws');

// Listen 👂
const wss = new WebSocket.Server({ port: 8080 });

// Every time we get a new connection
wss.on('connection', (ws) => {
  
  // Add a listener to it, log any messages
  ws.on('message', (msg) => {
    console.log('received: %s', msg);
  });

  // After the handshake, say Hi!
  ws.send('howdy');
});
```

Why is this technology useful? Seems like a pretty complicated way to send a message. Let's imagine that a financial website wants to show a stock ticker that updates every two seconds. Before WebSockets, you had to use *polling*. This meant requesting new information every X seconds, even if the information hadn't changed since the last update. Without performing frequent checks, it was impossible to stay in sync.

Clientside, it looked like this.

```javascript
const poller = (time) => {

// Make a request for some JSON (Instead of Fetch, we used to use `XML HttpRequest`!)
fetch('http://example.com/stocks.json')

  // A response will come
  .then((response) => {

    // Parse the data, continue down the chain
    return response.json();
  })
  .then((stockData) => {

    // Pass the fresh data to another function
    updateStocks(stockData);

    // Now, start it all over again - begin polling
    poller(time);
  });
}
```

This is called short-polling. There is an alternative called long-polling where the request is held paused at the server until new data is available or the request is about to timeout, in which case a blank response is sent and the cycle restarts. With long-polling, the data is sent back with low-latency and less bandwidth is used — it's a little more complex to setup though.

With polling, every request and response costs processing, memory, and bandwidth — think of the thick header sections for every cycle! With WebSockets, you perform a handshake and wait for new information — this uses exponentially less processing, memory, and bandwidth for both sides! The underlying [TCP](https://en.wikipedia.org/wiki/Transmission_Control_Protocol) pipe that carries the data back and forth doesn't sit perfectly idle though, for a good reason. The WebSocket protocol allows pings and pongs. These are heartbeats, or, keep-alive messages. Tiny data frames that shoot through the dark pipes under the seabed between computers to say *hello! I'm still here*.

The server pings and the client responds (by the spec, as soon as possible) with a pong. Websockets/ws has this built in. You can call `ws.ping()` at any point after the connection has been made. The endpoint pongs back which triggers the `pong` event. Your browser's `WebSocket class` does this automatically.

#### The Future

Next time you see Twitter has new posts for you, consider the hardworking sockets — and their little heartbeats — which brought you this information, and say thank you to [RFC 6455](https://tools.ietf.org/html/rfc6455) ❤️

[@healeycodes](https://twitter.com/healeycodes) for more personification of technology 🖲️


A little on the history of WebSockets and a brief overview on how to use them in JavaScript.

Kicking Butt and Taking Names With WebSockets (Tutorial)


#### Code review is the best part of the job

I went from being afraid of being judged to embracing team-spirit. Today, I find myself completing a ticket and looking forward to placing it in review. I really enjoy the feedback I receive from my peers and the conversations that start as a result. It's a great chance to learn more about the project's architecture and patterns. In a past life, I was a Creative Writing student so I understand that editing something *always* makes it better.

I like giving as much as getting. I was encouraged from day one to join in the review process. I understand how to optimize JavaScript quite well so the first comments I made were offering advice for the performance critical parts of the code base that came through review — all of which were well received.

Software engineering as a field has so much breadth that even people starting out in the industry have something to offer a team. Whether they read a tech blog, built a project in the stack, or went deep in the documentation.

#### Coding is one of the least important parts of the job

Don't get me wrong, if I start slinging bad code into review, I'm going to run into issues. However, I've found that the most important skill at my job is communication. I need to understand what my project manager is telling me so that I can translate it into code. I need to then listen to my manager's feedback about my approach. If I can't explain my design to my teammates then it will take them longer to understand my (hopefully self-documenting) code.

I was part of a group of three designing a new module when I realised how my communication has grown. Much like the ideal piece of code, the best way to communicate is with simple, almost boring language — all backed by the correct abstraction. Figuring out the exact requirements, and the right tools and modules avaliable to you is 85% of the battle in completing a ticket.

```javascript
/* Boring vs. Fancy.
 *
 * Task: Given a one or two digit Number, zero-pad values below ten
 */

// Fancy - uses String-tricks
('0' + foo).slice(-2);

// Plain old logic - significantly faster!
(foo < 10) ? ('0' + foo) : foo;

// ES7 gives us a native pad operator but it's not optimized for this situation either
foo + ''.padStart(1, 0);
```

I tackled a high-priority bug — something that crippled our product — I dug for a whole work day in order to find the one character, a single byte, that needed to be removed in order to fix the whole thing. That was a good day. Other weeks, I blast out boilerplate like they're paying me one penny a word.

<br>

The worst part about my job? The winter weather commute!


Two Things About My First In-house Software Job That I Didn't See Coming


This Sunday, I worked on an API that creates a GitHub badge based off a **project's mood**. Mood meaning the average time of day that the repository is worked on. I find that I work on different projects at different times of the day. In the morning, I skew towards back-end focused repositories. Maybe it's the coffee? ☕

[![Project Mood](p-m-header.png)](https://github.com/healeycodes/project-mood)

GitHub badges can either be generated dynamically by a library or via third-party services like [shields.io](https://shields.io). For [Project Mood](https://github.com/healeycodes/project-mood), we use [gh-badges](https://www.npmjs.com/package/gh-badges) — an npm package. Here's how a badge is built.

```javascript
const bf = new BadgeFactory();

// ...        

// Customize badge
const format = {
    text: ['project mood', timeOfDay],
    colorA: '#555',
    colorB: color,
    template: 'flat',
}

// Create SVG/add title
const title = `<title>average commit: ${parseInt(average)}:${parseInt((average % 1) * 60)}</title>`;
const svg = bf.create(format).replace(/>/, `>${title}`);
```

It's a prototype, and the library doesn't allow custom attributes, so we inject titles in with a RegEx replace. The sole route we make avaliable is `/:owner/:repo.svg` for example: `/healeycodes/project-mood.svg`. With Express, SVGs can be sent back much like a string would be.

```javascript
return res
    .status(200)
    .send(svg)
    .end();
```

The color of these badges is decided by scanning recent commits and finding the average time of day. The GitHub API requires a user agent and a [personal access token](https://github.com/settings/tokens). We process the commits with a map/reduce. JavaScript's `Date` responds well to time zone correction.

```javascript
// Options for the request call
const options = {
    url: `${api}repos/${req.params.owner}/${req.params.repo}/commits?${token}`,
    headers: {
        'User-Agent': process.env.USERAGENT
    }
};

// ...

// As part of the request callback, commits are scanned
const times = json.map(item => item.commit.author.date);
const average = times.reduce((sum, time) => {
    const d = new Date(time);
    const hours = d.getHours() + (d.getMinutes() / 60) + (d.getSeconds() / 60 / 60);
    return hours + sum;
}, 0) / times.length;
```

It takes ~0.75ms to generate a badge on a modern PC — this includes our title insertion method. Since there is no state being managed, this project would respond well to horizontal scaling. However, the roadmap describes some ways that scale can be managed without throwing money at the problem.

```text
🚗🚗🚗

- Caching:
    - Repositories should be scanned infrequently rather than per request.
    - We can store the most recently requested SVGs in memory.
    - Basically, don't generate the SVG for every request (which is used for the prototype).
- Blended colors depending on average time rather than fixed colors.
```

No project is complete without tests! A simple test plan, executed by a cloud-build, is one of my favorite markers to pass during development. For [Project Mood](https://github.com/healeycodes/project-mood), Mocha and SuperTest are paired with Travis CI. The Express app is exported when the `NODE_ENV` is set to 'test'.

```javascript
if (process.env.NODE_ENV === 'test') {
    module.exports = app;
} else {
    const PORT = process.env.PORT || 8080;
    app.listen(PORT, () => {
        console.log(`App listening on port ${PORT}`);
        console.log('Press Ctrl+C to quit.');
    });
}
```

This allows us to import it into `app.test.js` which will be called by `npm test`. Other enviromental values in use are `USERAGENT` which is required by the GitHub API, as well as `GHTOKEN`. The latter is set to be hidden by Travis CI so that public builds don't leak secrets.

![Travis CI](p-m-travis.png)

One of the lovely tests.

```javascript
// Entry - "mocha test/app.test.js --exit"

const request = require('supertest');
const app = require('../app');
const assert = require('assert');

/**
 * Test SVG request
 */
describe('GET /healeycodes/project-mood', () => {
    it('responds with an SVG', (done) => {
        request(app)
            .get('/healeycodes/project-mood.svg')
            .expect((res) => {
                // SVG XML Namespace
                assert(res.text.match(/http:\/\/www.w3.org\/2000\/svg/gmi) !== null);
                // Error message not present
                assert(res.text.match(/unknown/gmi) === null);
            })
            .expect(200, done);
    });
});
```

💬 [twitter/healeycodes](https://twitter.com/healeycodes) for complaints.


Creating GitHub badges based off the time when most people contribute to a GitHub project.

Creating Cute GitHub Badges Based Off a Project's Mood (Node.js)


Let's say we want to find every emoji as it's posted to Reddit. First, we'll define what an emoji is. The Unicode Consortium keeps their [data lists](http://unicode.org/Public/emoji/12.0/) avaliable. We'll bring them in via a JSON file. The file has descriptions and other things we don't need so we'll clean the data a bit and store it in an efficient `Set`. If we're checking whether a character is or isn't an emoji over 5k times a second we'll need the `O(1)` look-up time!

```python
import json

with open('emojis/emoji.json', encoding='utf8') as emoji_db:
    cleaned = filter(lambda e: 'emoji' in e, json.load(emoji_db))
    emojis = set(map(lambda e: e['emoji'], cleaned))
```

[![Emoji Streamer](emoji-streamer-preview.gif)](https://github.com/healeycodes/EmojiStreamer)

To check every Reddit comment as it comes into existance we'll use [PRAW](https://praw.readthedocs.io/en/latest/) (The Python Reddit API Wrapper). It's the defacto standard for interacting with the Reddit API. It's well documented and straightforward to use.

```python
import praw

client_id = os.environ['CLIENTID']
client_secret = os.environ['CLIENTSECRET']
reddit = praw.Reddit(client_id=client_id,
                     client_secret=client_secret,
                     user_agent='emoji-tracker v0.1')
while True:
    try:
        for comment in reddit.subreddit('all').stream.comments():
            for char in comment.body:
                if char in emojis:
                    print(char, flush=True)
    except Exception as e:
        print(e, file=sys.stderr)
```

Here we set up PRAW with our Reddit app's ID and secret. Once we have a Reddit instance, we can start scanning. PRAW will get all the comments while managing the API limits for us. At around 300k characters per minute, it's quite a lot of data for a human to comprehend but it's peanuts for a PC as long as it's setup properly.

Let's pipe any emojis we find out to `stdout` making sure to flush every `print` so we don't leave any hanging. Emojis sometimes come in slowly, and one at a time -- we want to make sure that our UI is updating as soon as possible to hold the user's attention for a little longer (for those users who enjoy watching a string of emojis grow and grow).

By the way, which of you is posting 40 crabs at a time and why!?

🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀🦀

We'll catch these emojis in Node.js. express/express-ws was chosen so that we can simply host our static files while also making an emoji stream avaliable over a WebSocket. The `emoji-piper.py` file will be called as a child process. Let's attach some event listeners to catch the emojis as they are sent to `stdout`.

```javascript 
/* Whenever new emojis are ready, clear up CLOSING/CLOSED connections
    while sending emojis to the remaining clients */
const emojiStreamer = process;
emojiStreamer.stdout.on('data', (emoji) => {
    emoji = `${emoji}`;
    Array.from(clients, client => {
        if (client.readyState > 1) {
            clients.delete(client);
            return
        }
        client.send(emoji);
    });
});
```

We convert the data into a `String`, clean up closed or closing connections, and disperse the hot-off-the-press emojis. If the emojis were posted more often, say ten times more often, then it would make sense to place them into groups before passing them around (in Node.js or Python, wherever the closest bottleneck is).

Over in Front End Land, we let the user know when they're connected and let loose the fresh emojis. The reason we `split('\n')` here is to fix some unsyncing issues with the `stdout` pipeline -- a flaw in our design but necessary for rapid prototyping.

```javascript 
ws.onopen = () => {
    status.innerHTML = 'Connected ✔️';
}

ws.onmessage = (emojis) => {
    // Multiple emoji may arrive delimited by a new-line char
    emojiList = emojis.data.split('\n');
    emojiList.splice(emojiList.length - 1);
    emojiList.forEach(emoji => {
        render(emoji);
    });
}
```

Did you know that there is no consensus on what the plural of emoji is? Let me know at [@healeycodes](https://twitter.com/healeycodes) if you've been wincing at my usage of *emojis!*

This project is open sourced at [Emoji Streamer](https://github.com/healeycodes/EmojiStreamer) with Travis CI counting emojis and making sure our tests pass 👍


Building a website which live streams every emoji posted to Reddit.com.

Real-Time Streaming of Every Emoji Posted as a Comment to Reddit.com


Sudoku powered by the dark arts of `window.location.hash`, which means zero page refreshes are required. Unfortunately this means that the browser history gets spammed.

![Secret Sudoku](secret-sudoku.gif)

Playable [right here](https://healeycodes.github.io/secret-sudoku/), Secret Sudoku unlocked a series of unique experiences for me. It's not every day that you find yourself brute forcing a 19th-century newspaper puzzle. Or, better yet, getting Travis CI to do it for you.

```javascript
test('app plays without error', () => {
    const game = main(this, sudoku);
    game.startGame();
    for (let i = 0; i < 20000; i++) {
        // 0-81, 0-9
        if (game.play(Math.floor(Math.random() * 82), Math.floor(Math.random() * 10)) === true) {
            /* This test will usually complete the game six times over,
               adjust the difficulty randomly on completion */
            game.startGame(Math.floor(Math.random() * 6));
        };
    }
});
```

It turns out that all you need are some emotes, a few eventListeners, and a dream. Many thanks to Rob McGuire-Dale and his wonderful Sudoku [generator/solver](https://github.com/robatron/sudoku.js). And to Mathew Rayfield for showing that you can [animate URLs](http://matthewrayfield.com/articles/animating-urls-with-javascript-and-emojis/).

```javascript
// Main render functions
this.squaresFilled = () => this.board.join('').match(/[^.]/g).length
this.completeness = () => `${this.squaresFilled()}/81`;
this.start = () => `Secret_Sudoku_~_${this.completeness()}_~_row:`;
this.render = (extra = '') => {
    window.location.hash = `${this.start()}${this.rowMap[this.row]}__${this.prettifyRow(this.currRow(), this.cursor)}${extra}`;
}
```

Secret Sudoku is the perfect project to work on to help me relax in the evenings. It makes programming feel far more fun than logic abstraction should. Taking pull-requests today and every day hereafter.

#### [Repository](https://github.com/healeycodes/secret-sudoku)


Goofy project in which you can play Sudoku entirely within your URL bar.

I Ported Sudoku to Your URL Bar and It Plays like a Dream


Order trackers are a great example of AJAX-ness: a web page updating itself fluidly in the background (even though customers will be spam refreshing!)

![Order Tracker](order-tracker-header.png)

This past week, I created a minimal order tracker, and I'm going to highlight some of my design choices and share some excerpts from the most important sections of the tiny codebase.

Working back to front, we start with a Firebase real-time database.

![Firebase](order-firebase.png)

Here we're defining five stages that an order can be in, and we're also tracking the time the order-tracking instance was created for future logging.

The only packages we need are Express and the Firebase API. We use environment variables where sensible so that our app can be tested without using the production database.

```javascript
/* Firebase */
const admin = require('firebase-admin');
const serviceAccount = require(process.env.KEY);
admin.initializeApp({
    credential: admin.credential.cert(serviceAccount),
    databaseURL: process.env.DB
});
const db = admin.database();

/* Express */
const express = require('express');
const app = express();
const server = require('http').Server(app);
app.use(express.static('public'));
app.use(express.json());
```

There are just three short routes that create an API that we (the business) and our users' browsers can speak to. Note that authentication for public/private routes has been left as an exercise for the reader. As we'll see later, all web page content is statically hosted.

```javascript 
// Create a tracking instance
app.get('/private/orders', (req, res) => {
    const orderId = uuidv4();
    db.ref(`orders/${orderId}`).set({
        started: Date.now(),
        // Integer: 1-5 inclusive
        stage: 1
    })
        .then(() => {
            return res.send(orderId);
        })
        .catch((err) => {
            console.error(`Error creating tracking instance: ${err}`);
            return res.status(500).send('Server error');
        })
});

// Set a tracking instance's state
app.post('/private/orders', (req, res) => {
    db.ref('orders').child(req.body.orderId).set({
        // Clamp stage
        stage: Math.max(1, Math.min(5, req.body.stage))
    })
        .then(() => {
            return res.send('OK');
        })
        .catch((err) => {
            console.error(`Error setting tracking instance state: ${err}`);
            return res.status(500).send('Server error');
        })
});

// Client access to a tracking insance
app.get('/public/orders/:orderId', (req, res) => {
    const orderId = req.params.orderId;
    db.ref(`orders/${orderId}`)
        .once('value')
        .then(data => {
            order = data.val();
            if (order !== null) {
                return res.send(order);
            } else {
                console.error(`Unknown tracking instance requested: ${orderId}.`);
                return res.status(500).send('Server error');
            }
        })
        .catch((err) => console.error(`Order: ${orderId} errored: ${err}`));
});
```

On the front end, we grab some Font Awesome icons, throw them in some red/blue boxes with classes we can toggle, and we're almost done. In order to keep the scope as minimal as possible, we read the query parameter with JavaScript. This way, our microservice can host one static page, one .js file and one .css file.

```javascript 
// The query parameter `orderId` lets us provide order tracking
window.orderId = new URLSearchParams(window.location.search).get('orderId');

// If delivery in progress, check for new information every X seconds
window.localStage = null; // Integer: 1-5 inclusive
const checkStage = () => {
    fetch(`/public/orders/${window.orderId}`)
        .then(res => res.json())
        .then(data => {
            applyStage(data.stage);
        })
        .catch(err => console.error(`Error connecting to server: ${err}`))
    if (window.localStage < 5) {
        setTimeout(checkStage, 10000);
    }
}
```

A dash of CSS media queries for our mobile friends. This was as simple as flipping the flex grid and adding a little padding.

![Responsive](order-mobile.png)

I like to make sure that all of my projects are well tested. It helps me jump back into them after a break and stops me stomping through the code base breaking everything. For this app I chose SuperTest (which comes with Mocha).

For example, this test checks that all the data-plumbing is working correctly.

```javascript
/**
 * Test client accessing a tracking instance
 */
describe('GET /public/orders/:orderId', () => {
    it('respond with an order stage', (done) => {
        // Create an order instance
        request(app)
            .get('/private/orders')
            .end((err, res) => {
                request(app)
                    // Request this instance's stage from client route
                    .get(`/public/orders/${res.text}`)
                    .expect((res) => {
                        const stage = res.body.stage
                        // An integer within 1-5
                        assert(stage > 0 && stage < 6 );
                    })
                    .expect(200, done);
            });
    });
});
```

The [repo](https://github.com/healeycodes/order-tracking-microservice) uses Travis CI to run tests on every commit. Travis's offering to open source projects has been such a blessing to my journey as a developer, and helps me to build software that works (and tests) cross-platform!


Writing an order tracker with Node, Express, and Firebase.

Cloning Dominos Order Tracker as a Microservice with Node.js and Firebase!


WebSockets run a large part of the web today. But which servers and frameworks are the best? Well, that depends on how you define best. If you're after raw performance then the following post may be of interest to you. I will be going over some of my design notes for a small benchmark program I wrote in Python with [asyncio](https://docs.python.org/3/library/asyncio.html) and [websockets](https://github.com/aaugustin/websockets).

![WebSocket Benchmarker](wsbenchheader.png)

#### [WebSocket Benchmarker](https://github.com/healeycodes/websocket-benchmarker)

I wrote my Masters' thesis on benchmarking WebSockets servers and frameworks, and coded a benchmark program using Node.js and [ws](https://github.com/websockets/ws). The program, while effective in reaching my research goal, was not fit for publication. The goals with [WebSocket Benchmarker](https://github.com/healeycodes/websocket-benchmarker) were to learn more about asynchronous Python and to create something that other people could use. Usability here meaning some end-to-end tests, maintainability, and nicely commented code with Docstrings:

<br>

```python
async def client(state):
    '''A WebSocket client, which sends a message and expects an echo
    `roundtrip` number of times. This client will spawn a copy of itself afterwards,
    so that the requested concurrency-level is continuous.

    Parameters
    ----------
    state : Dictionary
        A Dictionary-like object with the key `clients` --
        the number of clients spawned thus far.

    Returns
    -------
    string
        A statement when the max number of clients have been spawned.'''
```

<br>

> Await expression - Suspend the execution of coroutine on an awaitable object.

This program allows someone to fake a number of concurrent clients that connect to an echo server (the WebSocket implementation being benchmarked) and send a series of messages. The roundtrip time for each message is measured, logged, and lightly analyzed at the benchmark's close.

Asynchronous design means that we don't need to manually poll things to see when they're finished. And rather than passing callbacks as often done in the past, asyncio lets us `await` things. It's concurrency but easy.

>`await`, similarly to `yield from`, suspends execution of `read_data` coroutine until `db.fetch` awaitable completes and returns the result data.

The 'clients' are coroutine functions that exist in the asyncio event loop. They await opening connections, await sending messages, await recieving messages, and await he closing handshake. After this they spawn a copy of themselves and await that too! This is how concurrency is achieved. There is always the same number of clients. Never more or less.

```python
# create an amount of client coroutine functions to satisfy args.concurrency
con_clients = [client] * concurrency

# pass them all a 'link' to the same state Dictionary
state = dict({'clients': 0})

# run them concurrently
main = asyncio.gather(*[i(state) for i in con_clients])
loop = asyncio.get_event_loop()
loop.run_until_complete(main)
```

Once you're inside an asynchonrous function, it really is as simple as it sounds:

`response = await websocket.recv()`.

Now control flow will go elsewhere and return when appropriate.

I hope to use this software to update my personal benchmark rankings of WebSocket servers and frameworks and write another blog post about the results (and the trade-offs that seeking raw performance often leads to)!

Contributions to the [repository](https://github.com/healeycodes/websocket-benchmarker) are most welcome ❤️.


Benchmarking websocket servers and frameworks with async Python.

Benchmarking WebSocket Servers with Python!


I'm a big fan of all three problems and solutions today. I must say that these Easys are getting harder than their counterparts though. But I do often sort by acceptance rate high->low.

There's also a 'perfect' solution included for the *Move Zeroes* problem!

### [283. Move Zeroes](https://leetcode.com/problems/move-zeroes/)

Problem: Move all the zeroes to the end of the array, in-place.

The problem statement also wants a minimum of total operations.

In production, this would be a perfect use-case of Python's timsort `.sort()` -- although, this sometimes creates additional space which excludes the algorithm for this problem. It also uses more operations than we need to!

The trick is realizing that the zero characters are essentially empty space. I'll show you what I mean.

```python
class Solution(object):
    def moveZeroes(self, nums):
        """
        :type nums: List[int]
        :rtype: void Do not return anything, modify nums in-place instead.
        """
        place_nums = 0
        for i in range(len(nums)):
            if nums[i] != 0:
                if i > place_nums:
                    nums[place_nums] = nums[i]
                place_nums += 1
        
        for i in range(place_nums, len(nums)):
            nums[i] = 0

        # return is excluded as per problem statement
```

We place all the numbers at the start of the array and then overwrite the part of the array that should now be zeros.

`Runtime: 32 ms, faster than 100.00% of Python online submissions for Move Zeroes.`

Runtime complexity: `O(n)`.

Space complexity: `O(n)`.

### [21. Merge Two Sorted Lists](https://leetcode.com/problems/merge-two-sorted-lists/)

Problem: Merge two sorted linked list, return as a linked list

I chose to solve this recursively as it really cuts down on the if/else blocks. The code is a little easier to read as well.

```python
# Definition for singly-linked list.
# class ListNode(object):
#     def __init__(self, x):
#         self.val = x
#         self.next = None

class Solution(object):
    def mergeTwoLists(self, l1, l2):
        """
        :type l1: ListNode
        :type l2: ListNode
        :rtype: ListNode
        """
        if not l1 and not l2:
            return None
        elif not l1:
            return l2
        elif not l2:
            return l1
        
        node = ListNode(0)
        if l1.val < l2.val:
            node.val = l1.val
            node.next = self.mergeTwoLists(l1.next, l2)
        else:
            node.val = l2.val
            node.next = self.mergeTwoLists(l1, l2.next)
            
        return node
```

However, cute as this solution is, there will be stack issues with long lists.

Runtime complexity: `O(n)`.

Spacetime complexity: `O(n)`.

### [617. Merge Two Binary Trees](https://leetcode.com/problems/merge-two-binary-trees/)

Problem: Merge two binary trees, that is to say, fold them onto one and other, summing overlapping nodes.

This uses a similar recursive strategy to the linked list problem earlier.

```python
# Definition for a binary tree node.
# class TreeNode(object):
#     def __init__(self, x):
#         self.val = x
#         self.left = None
#         self.right = None

class Solution(object):
    def mergeTrees(self, t1, t2):
        """
        :type t1: TreeNode
        :type t2: TreeNode
        :rtype: TreeNode
        """
        if not t1:
            return t2
        if not t2:
            return t1
        
        result = TreeNode(t1.val + t2.val)
        result.left = self.mergeTrees(t1.left, t2.left)
        result.right = self.mergeTrees(t1.right, t2.right)
        return result
```

Runtime complexity: `O(n)`.

Spacetime complexity: `O(n)`.


Solutions for: Move Zeroes, Merge Two Sorted Lists, and Merge Two Binary Trees.

Leetcode - Episode 17 - Moving, Shaking, and Folding (3 x E)


More algorithms today. Pretty efficient ones at that. All above 90% ranked on the first submission. I'm beginning to find a lot of patterns that can be applied to the different problems. Most of the Easy binary tree questions are guilty of this.

I'm really happy that I finally solved *Flipping an Image* -- I had been in the thicket of a messy solution for a couple days but I was able to Pythonize it (read: refactor it)! I got hung up on solving it optimally before peaking at any resources.

### [832. Flipping an Image](https://leetcode.com/problems/flipping-an-image/)

Problem: Flip a binary matrix and then invert it.

The trick is doing both steps at once. In other words, you don't want two for loops.

```python
class Solution(object):
    def flipAndInvertImage(self, A):
        """
        :type A: List[List[int]]
        :rtype: List[List[int]]
        """
        for x in range(len(A)):
            A[x] = A[x][::-1]
            for y in range(len(A[x])):
                A[x][y] ^= 1
        
        return A
```

Outrageously straightforwards when it's reduced down to four lines.

Runtime complexity: `O(n)`.

Spacetime complexity: `O(n)`.

### [872. Leaf-Similar Trees](https://leetcode.com/problems/leaf-similar-trees/)

Problem: Do these two trees have similar leaves -- read left-to-right.

We're going to the edge of these trees and comparing the concatenated values found therein, or, there-out-there.

```python
# Definition for a binary tree node.
# class TreeNode(object):
#     def __init__(self, x):
#         self.val = x
#         self.left = None
#         self.right = None

class Solution(object):
    def leafSimilar(self, root1, root2):
        """
        :type root1: TreeNode
        :type root2: TreeNode
        :rtype: bool
        """
        self.root1_store = []
        self.root2_store = []
        def bfs_recurse(node, store):
            if not node.left and not node.right:
                store.append(node.val)
            else:
                if node.left:
                    bfs_recurse(node.left, store)
                if node.right:
                    bfs_recurse(node.right, store)
        if root1:
            bfs_recurse(root1, self.root1_store)
        if root2:
            bfs_recurse(root2, self.root2_store)
        return self.root1_store == self.root2_store
```

First try, pre-optimization: `Runtime: 32 ms, faster than 99.02% of Python online submissions.`.

Runtime complexity: `O(n)`.

Spacetime complexity: `O(n)`.

### [559. Maximum Depth of N-ary Tree](https://leetcode.com/problems/maximum-depth-of-n-ary-tree/)

Problem: What is the deepest level of this n-ary tree?

For this, we'll need a full tree traversal and some way of keeping track of the maximum depth so far. Lately, I've been reaching for instance variables in situations like this.

```python
"""
# Definition for a Node.
class Node(object):
    def __init__(self, val, children):
        self.val = val
        self.children = children
"""
class Solution(object):
    def maxDepth(self, root):
        """
        :type root: Node
        :rtype: int
        """
        self.depth = 0
        def dfs_recurse(node, depth):
            depth += 1
            for c in node.children:
                dfs_recurse(c, depth)
            self.depth = max(depth, self.depth)
        if root:
            dfs_recurse(root, 0)
        return self.depth
```

I'm going to check out some other solutions that don't use pseudo-globals.

Runtime complexity: `O(n)`.

Space complexity: `O(n)`.


Solutions for: Flipping an Image, Leaf-Similar Trees, and Maximum Depth of N-ary Tree.

Leetcode - Episode 16 - Pretty Efficient (3 x E)


I felt like it was time to knock some of these tree questions out. I find that solving tree questions in Python leads to simple, elegant code.

I went over Heap's Algorithm for solving permutation problems earlier. It hasn't fully *clicked* with me yet though. More studying is required -- as well as checking out alternative solutions. Heap's feels a little heavy-handed to me.

### [226. Invert Binary Tree](https://leetcode.com/problems/invert-binary-tree/)

Problem: Invert a binary tree.

I like the check I use here before I invert -- `if node.left or node.right:`. It's instantly obvious that it will work for all three siutations, two leafs, one leaf, None. 

```python
# Definition for a binary tree node.
# class TreeNode(object):
#     def __init__(self, x):
#         self.val = x
#         self.left = None
#         self.right = None

class Solution(object):
    def invertTree(self, root):
        """
        :type root: TreeNode
        :rtype: TreeNode
        """
        def recurse(node):
            if node.left or node.right:
                node.left, node.right = node.right, node.left
            if node.left:
                recurse(node.left)
            if node.right:
                recurse(node.right)
            return
        
        if root:
            recurse(root)
        return root
```

This problem is a little bit of a meme. I expected it to be harder than it was but I have been revising tree algorithms recently..

Runtime complexity: `O(n)`.

Spacetime complexity: `O(n)`.

### [590. N-ary Tree Postorder Traversal](https://leetcode.com/problems/n-ary-tree-postorder-traversal/)

Problem: Return the postorder traversal of a tree's values.

To think about this in another way, return the values as you would read the tree from the bottom up line by line.

```python
"""
# Definition for a Node.
class Node(object):
    def __init__(self, val, children):
        self.val = val
        self.children = children
"""
class Solution(object):
    def postorder(self, root):
        """
        :type root: Node
        :rtype: List[int]
        """
        vals = []
        queue = []
        if root:
            queue.append(root)
        while queue:
            cur = queue.pop()
            vals.append(cur.val)
            queue.extend(cur.children)
        
        return vals[::-1]
```

I recently learned (or, confirmed) that Python doesn't create new memory when slicing or creating an iterator, e.g. `[1,2][::-1]` / `reversed([1,2])`. It seems obvious now that I think about it.

Runtime complexity: `O(n)`.

Space complexity: `O(n)`.

### [589. N-ary Tree Preorder Traversal](https://leetcode.com/problems/n-ary-tree-preorder-traversal/)

Problem: Return the preorder traversal of a tree's values.

Another way to think about preorder traversal is Depth-First-Search order.

```python
"""
# Definition for a Node.
class Node(object):
    def __init__(self, val, children):
        self.val = val
        self.children = children
"""
class Solution(object):
    def preorder(self, root):
        """
        :type root: Node
        :rtype: List[int]
        """
        dfs_order = []
        
        def recurse(node):
            dfs_order.append(node.val)
            if node.children is not None:
                for i in range(0, len(node.children)):
                    recurse(node.children[i])
        if root:
            recurse(root)
        
        return dfs_order
```

I think this is a fairly terse but understandable solution but you can only make DFS so complicated.

Runtime complexity: `O(n)`.

Spacetime complexity: `O(n)`.


Solutions for: Invert Binary Tree, N-ary Tree Postorder Traversal, and N-ary Tree Preorder Traversal.

Leetcode - Episode 15 - Three Tree Qs (3 x E)


I'm not too excited about any of my solutions today. They're all fairly optimal but I suppose I would class them as 'entry-level' array problems. They didn't require any three-dimensional thinking.

I hope to put aside some time soon to study a new topic (e.g., permutation) that will unlock more difficult challenges and research areas within algorithms. That's the good thing about the streak, if I wait too long then I'll be forced to study more just to keep it up!

### [896. Monotonic Array](https://leetcode.com/problems/monotonic-array/submissions/)

Problem: Is this array monotonic? I.e., only decreasing or only increasing.

If it's static then it's also monotonic.

I over-wrote the solution to this problem. It's optimal but there's too much code.

```python
class Solution(object):
    def isMonotonic(self, A):
        """
        :type A: List[int]
        :rtype: bool
        """
        def all_increase(arr):
            for i in range(1, len(arr)):
                if arr[i] < arr[i-1]:
                    return False
            return True
                
        def all_decrease(arr):
            for i in range(1, len(arr)):
                if arr[i] > arr[i-1]:
                    return False
            return True
        
        if A[0] > A[len(A)-1]:
            return all_decrease(A)
        else:
            return all_increase(A)
```

The reason there's so much code is because I didn't want to use a flag. Flags don't (usually) read like Clean Code to me. I suppose there's a line with algorithm problems like these, and certainly coding competitions, where standards must come second.

Runtime complexity: `O(n)`.

Spacetime complexity: `O(1)`.

### [485. Max Consecutive Ones](https://leetcode.com/problems/max-consecutive-ones/submissions/)

Problem: Given a binary array, find the max number of `1`s in a row.

This is a fairly standard algorithm you first see when learning how to programming. You keep track of the streak and when it ends check it against the current max streak. Rinse and repeat.

```python
class Solution(object):
    def findMaxConsecutiveOnes(self, nums):
        """
        :type nums: List[int]
        :rtype: int
        """
        max_ones = 0
        streak = 0
        for idx, val in enumerate(nums):
            if val != 1:
                max_ones = max(streak, max_ones)
                streak = 0
            else:
                streak += 1
        
        return max(streak, max_ones)
```

In some languages I believe it will be quicker to use if/else statements instead of a `max` call -- [Java discussion](https://stackoverflow.com/questions/2103606/is-math-maxa-b-or-abab-faster-in-java).

Runtime complexity: `O(n)`.

Space complexity: `O(1)`.

### [217. Contains Duplicate](https://leetcode.com/problems/contains-duplicate/)

Problem: Given an array of integers, are there any duplicates?

There are three ways to go about solving this problem.

- Runtime: `O(n^2)`, Space: `O(1)` -- for each *num* check the whole array for duplicates.
- Runtime: `O(nlgn)`, Space: `depends` -- sort the array, compare neighbors.
- Runtime: `O(n)`, Space: `O(n)` -- apply a Set or Dictionary in some way.

I use the last option, in a short and dramatic Pythonic manner.

```python
class Solution(object):
    def containsDuplicate(self, nums):
        """
        :type nums: List[int]
        :rtype: bool
        """
        return len(set(nums)) < len(nums)
```

This performs well millisecond-wise in their rankings. It's also closer to production code than the other three options.

Runtime complexity: `O(n)`.

Spacetime complexity: `O(n)`.


Solutions for: Monotonic Array, Max Consecutive Ones, and Contains Duplicate.

Leetcode - Episode 14 - Short, Easy Solutions (3 x E)


Today has been a good day.

I drank one coffee more than I should have and completed my daily three problems in a comfortably short time with each solution also having a comfortably short runtime.

As I hoped, each successful problem has served as a building block for tackling harder problems. The mini-research I do after getting an optimal solution is vital. The Leetcode discussion board, and the wider internet, information that really clicks because the problem that the information helps to solve is still fresh in my head.

### [938. Range Sum of BST](https://leetcode.com/problems/range-sum-of-bst/)

Problem: Given a binary search tree, return the sum of all values between `L` and `R`.

I started out by writing a recursive algorithm that summed the entirety of a BST. I then added logic to check that I actually wanted to sum node's value.

The trick here is skipping the parts of the tree that cannot hold any relevant values. If the `node.left` has a value smaller than `L` then we are not interested in any deeper parts of that sub-tree.

```python
# Definition for a binary tree node.
# class TreeNode(object):
#     def __init__(self, x):
#         self.val = x
#         self.left = None
#         self.right = None

class Solution(object):
    def rangeSumBST(self, root, L, R):
        """
        :type root: TreeNode
        :type L: int
        :type R: int
        :rtype: int
        """
        def recursive_search(node, sum_vals=0):
            if node.val >= L and node.val <= R:
                sum_vals += node.val
            if node.left and node.val >= L:
                sum_vals += recursive_search(node.left, 0)
            if node.right and node.val <= R:
                sum_vals += recursive_search(node.right, 0)
            return sum_vals
        
        return recursive_search(root)
```

Runtime complexity: We may have to search the whole tree, so: `O(n)`.

Spacetime complexity: Nothing is created, so `O(1)`? Either that, or in the worst case we hold the entire tree in memory (on the stack).

### [807. Max Increase to Keep City Skyline](https://leetcode.com/problems/max-increase-to-keep-city-skyline/)

Problem: Given a 2D grid, find the total possible increase to values so that the max(row) and max(column) isn't increased.

Okay, that is a simplification of the problem statement but it's a little hard to understand with a quick scan.

I knew that two iterations would be required and that we needed to keep track of a value for each row and column.

```python
class Solution(object):
    def maxIncreaseKeepingSkyline(self, grid):
        """
        :type grid: List[List[int]]
        :rtype: int
        """
        max_increase = 0
        
        # to store the highest building for x/y view
        max_x = [0] * len(grid)
        max_y = [0] * len(grid[0])
        
        # find the highest buildings
        for x in range(0, len(grid)):
            for y in range(0, len(grid[x])):
                max_x[x] = max(grid[x][y], max_x[x])
                max_y[y] = max(grid[x][y], max_y[y])
        
        # sum the possible building increases
        for x in range(0, len(grid)):
            for y in range(0, len(grid[x])):
                max_increase += min(max_x[x], max_y[y]) - grid[x][y]
        
        return max_increase
```

One of the optimizations here is to use `min` and `max` instead of manually checking for higher/lower values.

Runtime complexity: `O(2n)` -> `O(n)`.

Space complexity: `O(2*sqrt(n)` -> `O(sqrt(n)` -- an interesting one.

### [791. Custom Sort String](https://leetcode.com/problems/custom-sort-string/)

Problem: Return a permutation of string `T` so that the letters are sorted in accordance with the order of string `S`.

Note: `S` has unique characters and will not necessarily contain every letter of the alphabet.

I'm a big fan of my solution to this problem. I'll take any opportunity to use a Set!

```python
from string import ascii_letters  
class Solution(object):
    def customSortString(self, S, T):
        """
        :type S: str
        :type T: str
        :rtype: str
        """
        letters = set(ascii_letters)
        vals = dict()
        idx = 0
        
        # determine the value of given letters
        for i in S:
            vals[i] = idx
            letters.remove(i)
            idx += 1
        
        # the value doesn't matter for the rest
        for i in letters:
            vals[i] = idx
            
        T = list(T)
        T.sort(key=lambda c: vals[c])
        return ''.join(T)
```

There is an optimization here where the unpointed letters are removed from the sorting sequence as their order doesn't matter.

Runtime complexity: We throw away the `nlgn` sorting time, so: `O(n)`.

Spacetime complexity: This would be `O(1)` due to `S` having unique characters but we can't sort an immutable string in-place!


Solutions for: Range Sum of BST, Max Increase to Keep City Skyline, and Custom Sort String.

Leetcode - Episode 13 - Finding My Stride (3 x M)


Today we're doing 'Saturday Morning Coffee' solutions -- instead of '10pm Work Night' solutions!

Continuing with the streak of string-based problems, all of the code here is 'optimal' -- i.e., *~97th* percentile or higher in the rankings.

### [953. Verifying an Alien Dictionary](https://leetcode.com/problems/verifying-an-alien-dictionary/)

Problem: Given the `order` of an alien alphabet, are these given alien `words` ordered lexiographically?

I started with a naive solution to this, which was to iterate `order` into a Dictionary for fast character look-up. I then compared the given list against a `sorted()` version of itself. I.e., `return words == sorted(words, key=some lambda)`. I wasn't happy with the runtime and I was wasting space -- `sorted` creates an addtional list in memory. Also, the function being passed to the lambda converted every word to a numerical list -- more created space, more complexity.

I then tested using a list of `[None] * 26` to use as a look-up 'Dictionary' by converting the ASCII letters to their numerical values but it was actually slower than using a normal Dictionary which was surprising. So, the Dictionary had to stay.

My improvement was to use a compare function, letter by letter, so that no extra space was created. In some cases only the first letter of each word needed to be checked -- instead of converting each whole word.

```python
class Solution(object):
    def isAlienSorted(self, words, order):
        """
        :type words: List[str]
        :type order: str
        :rtype: bool
        """
        
        # fast char-value checking
        vals = dict()
        for idx, val in enumerate(order):
            vals[val] = idx
        
        
        for i in range(0, len(words)-1):
            w1, w2 = words[i], words[i+1]
            flag = 0
            
            for j in range(min(len(w1), len(w2))):
                if vals[w1[j]] < vals[w2[j]]:
                    # w1 is winner
                    flag = 1
                    break
                elif vals[w1[j]] > vals[w2[j]]:
                    # w2 is winner
                    return False

            # w1 and w2 have equal comparable chars
            if flag != 1:
                if len(w1) > len(w2):
                    return False
        
        return True
```

This is one of my fastest and cleanest solutions yet.

`Runtime: 32 ms, faster than 100.00% of Python online submissions for Verifying an Alien Dictionary.`

Runtime complexity: `O(n)`.

Spacetime complexity: Assuming an `order` of 26: `O(1)`.

### [917. Reverse Only Letters](https://leetcode.com/problems/reverse-only-letters/)

Problem: Reverse the letters in string `S` leaving all other characters in place.

I used a simple 'two pointer' solution to this, similar to a version that I used for *345. Reverse Vowels of a String*.

```python
from string import ascii_letters
class Solution(object):
    def reverseOnlyLetters(self, S):
        """
        :type S: str
        :rtype: str
        """
        S = list(S)
        letters = set(ascii_letters)
        start = 0
        end = len(S)-1
        while not start > end:
            if S[start] not in letters:
                start += 1
                continue
            if S[end] not in letters:
                end -= 1
                continue
            S[start], S[end] = S[end], S[start]
            start += 1
            end -= 1
        return ''.join(S)
```

After researching, and checking the runtime percentile (~97th), I believe this to be the most optimal Python solution to this problem.

Runtime complexity: `O(n)`.

Space complexity: `O(1)`.

### [844. Backspace String Compare](https://leetcode.com/problems/backspace-string-compare/)

Problem: Given string `S` and `T`, are they equal when written in a text editor when `#` is backspace?

One of my early solutions was creating two lists, e.g., `[None] * len(S)`, but I realised that this wasn't optimal and may create more space than required.

I reached for `deque` -- Python's implementation of a Double Ended Queue. With backspace equalling a `pop()` the rest was pretty straight forward. 

```python
from collections import deque
class Solution(object):
    def backspaceCompare(self, S, T):
        """
        :type S: str
        :type T: str
        :rtype: bool
        """
        s = deque()
        for i in S:
            if i == '#':
                if len(s) > 0:
                    s.pop()
                else:
                    continue
            else:
                s.append(i)
            
        t = deque()
        for i in T:
            if i == '#':
                if len(t) > 0:
                    t.pop()
                else:
                    continue
            else:
                t.append(i)

        return s == t
```

I saw a faster solution to this problem that doesn't use any data structures. Instead it uses while loops and a lot of checking. It's about twice as many lines of code as mine and, while impressive, is incredibly hard to parse!

<br>

Runtime complexity: `O(n)`.

Spacetime complexity: `O(n)`.


Solutions for: Verifying an Alien Dictionary, Reverse Only Letters, and Backspace String Compare.

Leetcode - Episode 12 - Starting Early (3 x E)


Watch out string problems: I'm coming for you.

I really feel like I'm graduating from String School and feel fairly comfortable with a few different techniques for solving string-based puzzles. I get my fastest run times in this area as well.

Not too much else to report. I've also been reading up on permutation algorithms and reviewing the trade-offs between BFS and DFS.

### [345. Reverse Vowels of a String](https://leetcode.com/problems/reverse-vowels-of-a-string/)

Problem: Reverse the vowels of a given string.

I instantly knew that I wanted to go for a two pointer solution for this. I'm not sure if there's a more efficient way, in fact.

```python
class Solution(object):
    def reverseVowels(self, s):
        """
        :type s: str
        :rtype: str
        """
        s = list(s)
        vowels = set('aeiouAEIOU')
        
        start = 0
        end = len(s)-1
        while not start >= end:
            if s[start] not in vowels:
                start += 1
                continue
            if s[end] not in vowels:
                end -= 1
                continue
            s[start], s[end] = s[end], s[start]
            start += 1
            end -= 1
        return ''.join(s)
```

I saw on the discussion board that some people were getting faster runtimes by using a case statement instead of a set for vowel look-up. I'm not sure whether that's gaming the Leetcode online judge or whether it's actually faster than using the hash function via Set look-up. Will probably depend on different languages and their implementations.

Runtime complexity: `O(n)`.

Spacetime complexity: `O(1)`.

### [205. Isomorphic Strings](https://leetcode.com/problems/isomorphic-strings/)

Problem: Given two strings, check if they are isomorphic.

I.e., do they have the same pattern of letters. E.g., `zoo` matches `bee`. (`011` -> `011`).

I wanted to compare some integer lists representing string patterns.

```python
class Solution(object):
    def isIsomorphic(self, s, t):
        """
        :type s: str
        :type t: str
        :rtype: bool
        """
        count = 0
        letters = dict()
        
        s_list = []
        for i in s:
            if i not in letters:
                count += 1
                letters[i] = count
            s_list.append(letters[i])
            
        count = 0
        letters = dict()
        
        t_list = []
        for i in t:
            if i not in letters:
                count += 1
                letters[i] = count
            t_list.append(letters[i])
            
        return s_list == t_list
```

While linear in runtime, this solution felt a little clumsy to me. However, the only improvement that I found on the forums was to use arrays instead of look-up tables since the test cases were all ASCII letters.

Runtime complexity: `O(n)`.

Space complexity: `O(n)`.

### [151. Reverse Words in a String](https://leetcode.com/problems/reverse-words-in-a-string/)

Problem: Reverse the words in a string, remove leading and trailing spaces, have no more than one space between words.

I was surprised to see that this is classed as Medium difficulty but I suppose with the space-character edge cases, if I were not using Python (which, also, is fantastic at string manipulation), it would be a little harder.

```python
class Solution(object):
    def reverseWords(self, s):
        """
        :type s: str
        :rtype: str
        """
        return ' '.join(s.split()[::-1])
```

A pretty neat one-liner. `s.split()` takes care of all space-character edge cases for us!

My (undeservedly) fastest runtime percentile yet: *99.17th*. The fastest solution available returns early for an empty string which is a small optimization.

Runtime complexity: `O(n)`.

Spacetime complexity: `O(n)`.


Solutions for: Reverse Vowels of a String, Isomorphic Strings, and Reverse Words in a String.

Leetcode - Episode 11 - Faster than 99.17% (1 x M, 2 x E)


I'm really learning a lot by reading the discussion boards after submitting each of my near-optimal solutions -- especially, optimization. I'm also noticing holes in my toolset. E.g., bitwise -- as we'll see in just a second.

This Leetcode series started after I felt like I wasn't getting that much practical knowledge going over algorithm lectures and books. I knew enough of the basics to start putting solving and basic analysis into practise .. so I did. Solving three problems everyday increase my producivity as well which is a huge bonus.

The most beneficial path to me at this point is to use both types of resources.

Since I have a fixed number that I aim for each day, I am finding myself shying away to the more simple problems but this isn't nesscarily a bad thing. The problems, the research, and the post-solve reading serve as building blocks for taking on the harder ones.

### [137. Single Number II](https://leetcode.com/problems/single-number-ii/)

Problem: Given an array of integers, find the only element that doesn't appear three times.

I knew that the most optimal solution to this problem was going to be by bitwise methods but it wasn't coming to me. My bitwise capabilities feel below par.

I attempted to solve it as best as I could and ended up with a neat linear-time program.

```python
from collections import Counter
class Solution:
    def singleNumber(self, nums):
        """
        :type nums: List[int]
        :rtype: int
        """
        ints = Counter()
        maybes = set()
        for i in nums:
            maybes.add(i)
            ints[i] += 1
            if ints[i] == 3:
                maybes.remove(i)
        for i in maybes:
            return i
```

For every element in the array, there are a few more Dictionary operations than one might like but they are all constant time, and the code is clean and precise.

Runtime complexity: `O(n)`.

Spacetime complexity: `O(n)`.

### [509. Fibonacci Number](https://leetcode.com/problems/fibonacci-number/)

Problem: Given `n` find the `n`th Fibonacci number.

It's a shame that this problem doesn't support memoization because that would make it a little more interesting.

As it is, the only mistep you can make is going for a heavy-handed recursive solution. Mine just iterates and adds.

```python
class Solution:
    def fib(self, N):
        """
        :type N: int
        :rtype: int
        """
        if N == 0:
            return 0
        
        last = 1
        curr = 1
        i = 2
        while i < N:
            temp = curr
            curr += last
            last = temp
            i += 1
        return curr
```

A fairly neat solution. Only beaten if you use math which brings a constant runtime. E.g., using a formula for the n^th Fibonacci sequence in terms of the golden ratio.

Runtime complexity: `O(n)`.

Space complexity: `O(1)`.

### [383. Ransom Note](https://leetcode.com/problems/ransom-note/)

Problem: Given a ransom note string and a magazine string work out whether the ransom note can be constructed from the magazine.

I first attempted to solve this with a pair of Dictionaries, then a single Dictionary, then I remembered that when dealing with single letters it's usual viable to use an array instead.

```python
import string
class Solution(object):
    def canConstruct(self, ransomNote, magazine):
        """
        :type ransomNote: str
        :type magazine: str
        :rtype: bool
        """
        letters = [0] * 26
        for i in magazine:
            letters[string.lowercase.index(i)] += 1
        
        for i in ransomNote:
            letters[string.lowercase.index(i)] -= 1
            if letters[string.lowercase.index(i)] < 0:
                return False
        
        return True
```

Mostly pretty simple solutions today. I worked through some tree problems but ran into some edge cases. Presumably at some point I will post up a whole swathe of tree and graph problems!

Runtime complexity: `O(n)`.

Spacetime complexity: `O(1)`.


Solutions for: Single Number II, Fibonacci Number, and Ransom Note.

Leetcode - Episode 10 - Short Solutions (1 x M, 2 x E)


The three-a-day streak nearly ended today.

I came close to completing a couple of Medium problems but I didn't want to brute force the answers. I have a set of unwritten rules about what I'm allowed to look up in a book or google. All solutions need to be within the range of 'optimal' as well.

The goal is to get better at solving problems and algorithm analysis -- not to get the green text that reads *Accepted*. I ended up searching for some Easys to complete so I could get to bed on time.

### [169. Majority Element](https://leetcode.com/problems/majority-element/)

Problem: Given an array of size `n` find the majority element that appears more than `n/2` times.

I wrote about the algoritm that solves this problem for the first post on this blog. The *Boyer–Moore majority vote algorithm*.

Let's take a look at its linear runtime and constant space goodness.

```python
class Solution:
    def majorityElement(self, nums):
        """
        :type nums: List[int]
        :rtype: int
        """
        counter = 1
        current = nums[0]
        for i in nums[1:]:
            if i != current:
                counter -= 1
                if counter < 0:
                    current = i
                    counter = 1
            else:
                counter += 1
        return current
```

Its all about tracking who is in front of the pack and swapping when a challenger comes along.

Runtime complexity: `O(n)`.

Spacetime complexity: `O(1)`.

### [520. Detect Capital](https://leetcode.com/problems/detect-capital/)

Problem: Judge whether a word uses correct capitals.

The three types of correctness: `'IRE'`, `'topcoder'`, and `'Bing'`.

I wanted to test the runtime between using Python's built-ins and a custom solution for Leetcode's test cases.

So, the complex way:

```python
class Solution:
    def detectCapitalUse(self, word):
        """
        :type word: str
        :rtype: bool
        """
        def is_cap(c):
            return ord(c) < 97
        
        if is_cap(word[0]) and len(word) > 1 and is_cap(word[1]):
            for c in word:
                if not is_cap(c):
                    return False
        elif is_cap(word[0]):
            for c in word[1:]:
                if is_cap(c):
                    return False
        else:
            for c in word[1:]:
                if is_cap(c):
                    return False

        return True
```

This had a runtime of 76ms.

Instead, we could have just written:

```python
class Solution:
    def detectCapitalUse(self, word):
        """
        :type word: str
        :rtype: bool
        """
        return word.isupper() or word.islower() or word.istitle()
```

This *much* shorter version had a runtime of .. 76ms. The exact same result!

I'm fairly sure that as `n` tends to infinity, the more complex solution will surely win out.

Runtime complexity: `O(n)`.

Space complexity: `O(1)`.

### [620. Not Boring Movies](https://leetcode.com/problems/not-boring-movies/)

Problem: Given a SQL table `cinema` find movies with an odd numbered ID and a description that is not (literally) 'boring'.

Not much to comment on this one. I read through the discussion board and some people found that using `(not .. )` for their test expressions ran slower but I suppose that will be down to implementation details.

```SQL 
# Write your MySQL query statement below
SELECT * FROM cinema WHERE id % 2 != 0 AND description != 'boring' ORDER BY rating DESC;
```

<br>

Runtime complexity: `O(n)`.

Spacetime complexity: `O(n)`.


Solutions for: Majority Element, Detect Capital, and Not Boring Movies.

Leetcode - Episode 9 - Trudging Through (3x E)


Three Mediums today. I looked at the Parentheses problem last night so it's been bouncing around my head all day.

My fiancée was actually at a Python meet-up while I was solving these. We're a real pair of Pythonistas.

### [535. Encode and Decode TinyURL](https://leetcode.com/problems/encode-and-decode-tinyurl/)

Problem: Create an URL shortening service.

This is more of a design problem than a programming problem but there are still some tricks involved. There are many popular ways to solve the URL-shortening problem.

I decided to give URLs an incrementing ID kinda like a primary key which then gets encoded in base64. However, a lower base with alphanumeric captials is probably better suited for browsers. An advantage to my solution is that the early URLs will be even shorter than the six characters that Leetcode suggests.

As soon as you move away from a one-server model, this solution will fail because it begins to get harder to keep track of the incrementing ID.

```python
import base64

class Codec:
    def __init__(self):
        self.urls = dict()
        self.url_count = 0
    
    def encode(self, longUrl):
        """Encodes a URL to a shortened URL.
        
        :type longUrl: str
        :rtype: str
        """
        self.url_count += 1
        url_base64 = base64.b64encode(bytes(self.url_count))
        self.urls[url_base64] = longUrl
        return url_base64

    def decode(self, shortUrl):
        """Decodes a shortened URL to its original URL.
        
        :type shortUrl: str
        :rtype: str
        """
        return self.urls[shortUrl]

# Your Codec object will be instantiated and called as such:
# codec = Codec()
# codec.decode(codec.encode(url))
```

I liked a solution in the discussions tab that proposed a random choice that keeps picking a random six character long alphanumeric-captial string until it finds one that isn't already stored. This scales better with multiple servers.

This will probably be the only problem in this series with a `O(1)` runtime! It's basically a big clever Dictionary.

Runtime complexity: `O(1)`.

Spacetime complexity: `O(n)`.

### [701. Insert into a Binary Search Tree](https://leetcode.com/problems/insert-into-a-binary-search-tree/)

Problem: Given the root node of a BST, insert a given value (that is guaranteed to not already appear in the BST).

I went with an iterative search for this one. The solution is pretty straightforward.

```python
# Definition for a binary tree node.
# class TreeNode:
#     def __init__(self, x):
#         self.val = x
#         self.left = None
#         self.right = None

class Solution:
    def insertIntoBST(self, root, val):
        """
        :type root: TreeNode
        :type val: int
        :rtype: TreeNode
        """
        node = root
        while True:
            if node.val > val:
                if node.left is None:
                    node.left = TreeNode(val)
                    return root
                else:
                    node = node.left
            else:
                if node.right is None:
                    node.right = TreeNode(val)
                    return root
                else:
                    node = node.right
```

Runtime complexity: `O(lg n)`.

Space complexity: `O(1)` -- I think. I believe we are destorying references as we go so the stack size remains constant.

### [921. Minimum Add to Make Parentheses Valid](https://leetcode.com/problems/minimum-add-to-make-parentheses-valid/)

Problem: Given a string of parentheses, calculate the minimum number of additions to make it 'valid'.

A common solution to this problem is to used `O(n)` space and use a stack. However, that stack can be implemented with an integer instead.

```python
class Solution(object):
    def minAddToMakeValid(self, S):
        """
        :type S: str
        :rtype: int
        """
        parens = 0
        corrections = 0
        for i in S:
            if i == '(':
                parens += 1
            elif parens > 0:
                parens -= 1
            else:
                corrections += 1
        
        return corrections + parens
```

I like this solution because it's simple to read and follow.

<br>

Runtime complexity: `O(n)`.

Spacetime complexity: `O(1)`.


### [824. Goat Latin](https://leetcode.com/problems/goat-latin/)

Problem: Translate a sentence `S` into Goat Latin.

Rules:

- If there is a starting constant, move it to the end.
- Add `ma` to the end. Add an `a` for every word in the sentence so far.

I was surprised to find that it's quicker to use Python string operations as opposed to creating an array to fit the new word into. I think if you were able to create fixed size arrays in Python then it might be quicker to keep `extend` an array. However, whatever implementation of Python that Leetcode are using is probably doing this under the surface.

```python
class Solution(object):
    def toGoatLatin(self, S):
        """
        :type S: str
        :rtype: str
        """
        vowels = set('aeiouAEIOU')
        S = S.split(' ')
        
        for idx, val in enumerate(S):
            if S[idx][0] not in vowels:
                S[idx] = S[idx][1:] + S[idx][0]
            S[idx] = S[idx] + 'ma' + (idx + 1) * 'a'
        return ' '.join(S)
```

Python really shines here, keeping the logic compact.

<br>

Runtime complexity: `O(n)`.

Spacetime complexity: `O(n)`.

<br>

2.69% of Leetcode's problems solved!


Solutions for: Encode and Decode TinyURL, Insert into a Binary Search Tree, Minimum Add to Make Parentheses Valid, and Goat Latin.

Leetcode - Episode 8 - Breaking Out Some Mediums (3x M, 1x E)


This blog is now live! A lot of these posts were just sitting on my computer but it's now hooked up to GitHub Pages via Jekyll. GitHub Pages was very pleasant to use and extend.

In solving news: I'm focusing on using more Python languages features at the moment.

### [1. Two Sum](https://leetcode.com/problems/two-sum/)

Problem: Given an array of integers and a target, return the indices of the two numbers that add up to the target.

I really like the solution to this problem because it's so simple when you think about it. When faced with the problem initially it can be hard to see any route to a `O(n)` runtime.

```python
class Solution(object):
    def twoSum(self, nums, target):
        """
        :type nums: List[int]
        :type target: int
        :rtype: List[int]
        """
        num_dict = dict()
        for idx, val in enumerate(nums):
            if target - val in num_dict:
                return [num_dict[target - val], idx]
            else:
                num_dict[val] = idx

```

We iterate through the array adding everything to a Dictionary (after checking that the other number we're looking for hasn't been added yet). Awkwardly, we store the value as the key and the indice as the value.

Runtime complexity: `O(n)`.

Spacetime complexity: `O(n)`.

### [242. Valid Anagram](https://leetcode.com/problems/valid-anagram/)

Problem: Given two strings `t` and `s`, dermine if `t` is an anagram of `s`.

While I was working through my solution here, I felt like I was doing too much work to solve the problem, which turned out to be true.

```python
from collections import Counter
class Solution(object):
    def isAnagram(self, s, t):
        """
        :type s: str
        :type t: str
        :rtype: bool
        """
        if len(s) != len(t):
            return False
        
        s_count = Counter()
        t_count = Counter()
        for i in s:
            s_count[i] += 1
        for i in t:
            t_count[i] += 1
        
        for key, val in s_count.items():
            if key not in t_count:
                return False
            if s_count[key] != t_count[key]:
                return False
            
        return True
```

Runtime complexity: Once through the string with `O(1)` Dictonary additions, once through the `s` Dictionary with `O(1)` Dictionary checks: `O(2n)` -> `O(n)`.

Space complexity: `O(n)`.

After researching, it seems that the fastest solution would be to use two arrays of size 26, incrementing and decrementing the values per each character. Then we check the arrays against each other.

Another ['solution'](https://stackoverflow.com/a/17004897):

> Fastest algorithm would be to map each of the 26 English characters to a unique prime number. Then calculate the product of the string. By the fundamental theorem of arithmetic, 2 strings are anagrams if and only if their products are the same.

### [58. Length of Last Word](https://leetcode.com/problems/length-of-last-word/)

Problem: Given a string, return the length of the last word.

I start from the end of the string and work backwards and take care to perform a low number of checks (about `word_len` or `s[i]`) per character. (It turns out that it wasn't a low enough number of checks!)

```python
class Solution(object):
    def lengthOfLastWord(self, s):
        """
        :type s: str
        :rtype: int
        """
        word_len = 0
        for i in range(len(s)-1, -1, -1):
            if s[i] != ' ':
                word_len += 1
            elif word_len != 0:
                break
        
        if word_len == 0:
            return 0
        
        return word_len
```

Tricky test cases with this one, like `' '` and `a`.

Some people used two loops, one starting from the end to find the last space character, and one to walk back through towards the end counting the non-space characters. This is more optimal because less checks are performed. For example: `'a b '`. When there is extra whitespace on the end, my algoritm performs roughly 1.5x as many operations.

<br>

Runtime complexity: `O(n)`

Spacetime complexity: `O(1)`.


Solutions for: Two Sum, Valid Anagram, and Length of Last Word.

Leetcode - Episode 7 - Getting More Pythonic (3x E)


I've been going through the problems roughly sorted by their acceptance rate: higher `->` lower. Today, the difficulty is slowly creeping up. I'm looking forward to tackling some Mediums soon.

### [500. Keyboard Row](https://leetcode.com/problems/keyboard-row/)

Problem: Given a list of `words`, return only those that can be typed using one keyboard row

This was a neat problem to solve. At one point, I was `lower()`ing the words to account for capitalization but simply adding capital letters to the Sets decreased the runtime by ~13%.

```python
class Solution(object):
    def findWords(self, words):
        """
        :type words: List[str]
        :rtype: List[str]
        """
        rows = [set('qwertyuiopQWERTYUIOP'),
                set('asdfghjklASDFGHJKL'),
                set('zxcvbnmZXCVBNM')]
        one_row_words = []
        
        for word in words:
            for row in rows:
                if word[0] in row:
                    if all(char in row for char in word[1:]):
                        one_row_words.append(word)
                    break
        return one_row_words
```

I made sure to break once the word's row had been found to save on loop cycles over 'dead' words.

Runtime complexity: `O(n)`.

Spacetime complexity: `O(1)`.

### [937. Reorder Log Files](https://leetcode.com/problems/reorder-data-in-log-files/)

Problem: Given an array of `logs` with alphanumeric identifiers at the start, sort them so that *letter-only* logs come before *digit-only* logs.

I've added in some comments where there are specific rules but I recommend checking their description as this one's problem statement is harder to understand than any solution.

This was a fun one to solve.

```python
class Solution(object):
    def reorderLogFiles(self, logs):
        """
        :type logs: List[str]
        :rtype: List[str]
        """
        letter_logs = []
        digit_logs = []
        for log in logs:
            log = log.split()
            # check ascii value to determine digit or letter log
            if ord(log[1][0]) < 66:
                # the digit-logs should be put in their original order
                digit_logs.append(log)
            else:
                letter_logs.append(log)
        
        # the letter-logs are ordered lexicographically ignoring identifier,
        # with the identifier used in case of ties
        letter_logs.sort(key=lambda x: x[1:] + x[0:1])
                         
        return [' '.join(log) for log in letter_logs + digit_logs]
```

I'm getting a little better with Python generators. I really like how they feel to use, and how they affect the structure of these programs.

<br>

Runtime complexity: Separate logs `O(n)`, sort letter logs `O(n log n)`, join logs `O(n)`. Ergo: `O(n)`.

Spacetime complexity: `O(n)`.

### [136. Single Number](https://leetcode.com/problems/uncommon-words-from-two-sentences/)

Problem: Given an array `N` of integers, numbers all appear twice except for one. Return that integer.

I knew this could be XORed but I couldn't remember so implemented it with a Set and made a note to check after.

```python
class Solution(object):
    def singleNumber(self, nums):
        """
        :type nums: List[int]
        :rtype: int
        """
        nums_set = set()
        for i in nums:
            if i in nums_set:
                nums_set.remove(i)
            else:
                nums_set.add(i)
        for i in nums_set:
            return i
```

Runtime complexity: `O(n)`.

Space complexity: `O(n)`.

And now without that extra space, using XOR.

```python
class Solution(object):
    def singleNumber(self, nums):
        """
        :type nums: List[int]
        :rtype: int
        """
        result = 0
        for i in nums:
            result ^= i
        return result
```

Runtime complexity: `O(n)`.

Space complexity: `-`.

<br>

See you next time.


Solutions for: Keyboard Row, Reorder Log Files, and Single Number.

Leetcode - Episode 6 - Progressively Harder (3x E)


Title refers to my runtime rankings against other users *not* how long it took me to complete these. Ran into a couple of stubborn bugs, one of which was failing to read the question properly because I jumped ahead - doh.

### [884. Uncommon Words from Two Sentences](https://leetcode.com/problems/uncommon-words-from-two-sentences/)

Problem: Return the uncommon words from two sentences `A` and `B`

Note: a word is uncommon if it appears no more than once in either string.

This solution works regardless of the amount of sentences given. Thinking about finding uncommon words in just *two* sentences seems like a `O(n^2)` trap.

```python
from collections import Counter
class Solution(object):
    def uncommonFromSentences(self, A, B):
        """
        :type A: str
        :type B: str
        :rtype: List[str]
        """
        word_counter = Counter()
        for word in A.split(' ') + B.split(' '):
            word_counter[word] += 1

        uncommon_words = []
        for key, val in word_counter.items():
            if val == 1:
                uncommon_words.append(key)
        
        return uncommon_words
```

Add all words to a Counter Dictionary, return the words with a value of `1`.

Runtime complexity: `O(n)`.

Space complexity: `O(n)`.

### [700. Search in a Binary Search Tree](https://leetcode.com/problems/search-in-a-binary-search-tree/)

Problem: Search a Binary Tree.

My solution is a recursive search. 

```python
# Definition for a binary tree node.
# class TreeNode(object):
#     def __init__(self, x):
#         self.val = x
#         self.left = None
#         self.right = None

class Solution(object):
    def searchBST(self, root, val):
        """
        :type root: TreeNode
        :type val: int
        :rtype: TreeNode
        """
        if not root:
            return None
        if root.val == val:
            return root
        elif val < root.val:
            return self.searchBST(root.left, val)
        
        return self.searchBST(root.right, val)
```

<br>

Runtime complexity: `O(log n)`.

Spacetime complexity: `-`.

Nothing complicated to see here.

### [412. Fizz Buzz](https://leetcode.com/problems/fizz-buzz/)

Problem: Implement the [infamous](https://blog.codinghorror.com/why-cant-programmers-program/) FizzBuzz algorithm.

This was my first time 'solving' this in Python. I almost forgot about having to shift the iteration range (1-100 instead of 0-99).

```python
class Solution:
    def fizzBuzz(self, n):
        """
        :type n: int
        :rtype: List[str]
        """
        str_reps = []
        
        for i in range(1, n+1):
            if i % 3 == 0 and i % 5 == 0:
                str_reps.append('FizzBuzz')
            elif i % 3 == 0:
                str_reps.append('Fizz')
            elif i % 5 == 0:
                str_reps.append('Buzz')
            else:
                str_reps.append(str(i))
        
        return str_reps
```

Runtime complexity: `O(n)`.

Spacetime complexity: `-`.

<br>

Day 5 complete, 16 problems solved in 2019.


Solutions for: Uncommon Words from Two Sentences, Search in a Binary Search Tree, and Fizz Buzz.

Leetcode - Episode 5 - Fastest Solutions Yet (3x E)

Andrew Healey's blog and personal website. Software engineer, writer, creator.

Recent (78 articles)

Most Popular