The app in action

I like Word Hunt. I don’t even have an iPhone (I use a Pixel running GrapheneOS), but I borrow them so often to play that my “obsession” with the game has become a bit.

I created this web app at the height of this in summer of 2024. I was initially motivated by questions about optimal play and score distributions but ended up building a solver that finds all words and on a board and the maximum achievable score.

You can try it out here. You can also view the source code on GitHub.

The Game

A screenshot of a Word Hunt board

For the uninitiated, Word Hunt is a lot like Boggle. The goal of the game is to find as many words as you can on a board like above in the time allotted. You construct words by connecting adjacent letters on the grid (e.g., “laser” would be a word on this board, though I leave finding it as an exercise for the reader). Diagonal moves are allowed and each tile can only be used once in a word. Longer words give you more points.

Reading Boards

A webcam image before and after preprocessing and board identification. All squares found are boxed in red. Those identified as tiles, in green.

Since games are timed at 80 seconds, having users type in each letter of their board was not an option. I opted instead to use OpenCV to read in the board from an image.

The logic for this was relatively simple:

• First, do standard pre-processing (convert the image to grayscale and binarize)
• Use OpenCV to find contours (edges of shapes)
• Approximate each contour as a quadrilateral
• Compare contour positions and dimensions to look for a large square with a 4x4 grid of smaller squares inside of it. If that’s found:
  • Split the image into its 16 tiles
  • Iterate through them left to right, top to bottom, and use template matching* to correspond each to a letter.

If all tiles are found and each matches a character, the result is a 2D array of characters representing the game board.

Hunting Words

An animation showing early stopping with a trie. The paths branching from the red-boxed R and M are skipped because no English words begin with 'SR' or 'SATM'.

Solving the board by brute force (trying all possible letter combinations) is greatly suboptimal and not likely to run quick enough on more limited hardware. Instead, I used a trie, which stores an entire dictionary as a tree where branches are possible next characters and nodes indicate whether a string is a valid prefix or word. Importantly, a node only has branches for next characters that could lead to a real English word, not always for all 26 letters. Resultant dead ends allow the program to stop exploring paths that can’t lead to valid words. For example, since no English word starts with ‘wx’, the program can immediately skip further checks once it reaches that combination.

In pseudo-code, the algorithm looks like this:

• Initialize an empty list of words
• For each tile on the board:
  1. Initialize an empty word string
  2. Append the current tile’s letter to the word string
  3. Check if the current string is a valid prefix according to the trie
     • If it is not, end this branch
  4. Check if the current string is a valid word according to the trie
     • If it is, add it to the word list
  5. For each possible next move:
     • Branch to that tile
     • Go back to step 2 and repeat

Though using a trie doesn’t technically improve the worst case time complexity, the early stopping significantly helps the average.

Scoring

To rank words and calculate a board’s maximum score, I needed a function that tells how many points a word is worth. Empirically, I found this to be

with \(n\) being the length of the word. (Words of length 1 and 2 are not valid in this game.)

Web App

A still of the web UI

My initial implementation of this was a Python CLI, but I thought a web app would look nicer and be easier to use. Luckily, OpenCV has a JS library that’s pretty much one-to-one with the Python version, so porting that over was pretty easy. Reimplementing the solve logic was also pretty straightforward, though I did end up implementing the trie from scratch rather than using a library like I did in Python.

The most time consuming part was without a doubt the UI, which I painstakingly styled to match GamePigeon’s as closely as possible.*

A demo of the assistant. Some sections have been sped up and are clearly marked.

This is a quick showcase of something I’ve been working on for the past couple of weeks. There’s still a lot of work to be done, but I’m writing this now to avoid repeating previous mistakes—I put off writing about my last, davinci-based chat bot until it was ‘presentable’, and by then it had been obsoleted by GPT-3.5-turbo and ChatGPT plugins.

Even in a more polished state, this bot isn’t going to do everything for you. I use it for brainstorming, troubleshooting, high-level design, and tedious things like making Tkinter interfaces. I’d already used ChatGPT a fair amount in my day-to-day, and it’s been noticeably more convenient to be able to just say “computer” and ask questions without having to tab over to OpenAI and type them out.

My end goal is a smart assistant that can control my lights, music, etc., and perform internet queries slightly more complicated than what I could do with “Ok, Google”—this would make my commutes more productive/bearable and help me cut back on screen time in general.

Capabilities

Hardcoded

I programmed the following capabilities into the bot myself:

• Speech recognition and synthesis with Porcupine and Cobra, whisper.cpp, and ElevenLabs
• A simple console-style interface made with Tkinter
• A scheduler for executing commands at fixed intervals
• Persistent memory with naive recall (retrieves relevant memories by finding the nearest neighbors to each query) built with Pinecone
• The ability to create and manage commands (arbitrary Python code)

Self-implemented

The bot gave itself the ability to do the following upon my request, completely through dialogue*:

• Read text files
• Read text from documents (pdf, doc, etc.)
• Create files and directories
• Get a filepath from the user with a file selection dialog
• View open windows
• Scrape text from a given window with Tesseract OCR
• Perform Google searches
• Browse webpages (scrape and read their text)
• Securely store secrets**
• Run arbitrary bash script with user permission

It also implemented for itself:

• A Gmail address; the bot checks the inbox periodically and responds to any new mail.
• A Telegram bot; the bot uses long polling (its idea) and responds to messages pretty much instantaneously.

The bot is decently capable of stringing these together to get things done, but it does sometimes need a little encouragement. It could “email my brother’s school calling in sick for him” without intervention, but not “create a to-do list app in Angular with Auth0” (yet).

Design

This will probably be subject to change—I have a million different ideas to try out.

Prompting

In a system message, I tell GPT-4 that it is “Alex, a speech-enabled assistant and conversation partner.” I give it a few personality guidelines, and instruct it to use the following XML tags in its responses:

<input medium="console">...</input> - Represents user's typed console input
<input medium="speech">...</input> - Represents user's spoken input
<command-output>...</command-output> - Output yielded from commands you execute

<output medium="speech">...</output> - Read the text aloud to the user
<output medium="console">...</output> - Print the text to the console
<command>...</command> - Execute the enclosed command
<var name="VARNAME">...</var> - Store the content in VARNAME, use as $VARNAME in commands
<thought>...</thought> - For planning and reasoning (mandatory before each tag)
<schedule name="NAME" description="DESCRIPTION" instructions="INSTRUCTIONS" period="PERIOD">COMMAND</schedule> - Schedules a command to be run ever PERIOD seconds if set and to be handled with the provided instructions. If period is unset, the task will retrigger after every completion. The description should give instructions on what to do with output.
<unschedule>NAME</unschedule> - Unschedule a command to be run
<listen /> - Listen for user audio
<end-of-response /> - End your message (mandatory)

Right now these are hardcoded in, but I plan soon to construct this boiler on startup based on the configured interfaces and their associated tags.

I then let it know what commands it has access to, give it a few more guidelines, examples, and start giving it user input.

Response Handling

I’ve tried my best not to build a monolith. The system is broken into specialized components. I’ve built six so far, which can be mixed and matched with no problem.

• Brain: Formats input and feeds it to GPT, pipes output to other components
• Listener: Listens for wake words, processes speech, and sends transcriptions to the brain as input
• Executor: Manages and executes commands. Also synthesizes speech (yes, this does need to be separated into two components)
• Scheduler: Handles creation, deletion, and execution of scheduled commands
• Recaller: Prepends all input with relevant “memories” retrieved from vectorized historical chatlog
• Terminal: The earlier-mentioned Tkinter interface that facilitates text-based I/O

Components are effectively arranged in a loop (interfaces -> afferent components -> brain -> efferent components -> interfaces -> etc.) by a Mediator, which facilitates inter-component communication. Signals are passed from component to component as JSON objects and are either transformed, acted upon, or ignored and passed on unchanged.

A simplified animated representation of a three-component configuration

This modular structure makes the actual main method pretty simple:

comms = Mediator()

scheduler = Scheduler()
brain = Brain()
executor = Executor(stream_audio=True)
terminal = Terminal()
recaller = Recaller()

comms.register_responder(brain)
comms.register_interface(executor)
comms.register_interface(terminal)
comms.register_interface(scheduler)
comms.add_afferent(recaller)

scheduler.start()
executor.start()
terminal.start()

Note: This entire architecture was loosely inspired by a lesson from my neuroscience-minor girlfriend (she likes to study by teaching) that touched on efferent and afferent pathways. I myself am not a neuroscience major, and I understand that my use of the terms here might not be completely correct. For our purposes, afferent components intercept/process signals heading towards the brain, and efferent, those away.

Commands

One of the few commands the bot has hardcoded into it is create_command:

create_command --command_name="" --description="" --script_content="" --dependencies=""

This creates a command for the bot to use later, just as it used this one. The script content is saved to a Python module in a commands package, and a virtual environment is created for it with the provided dependencies. In the description for create_command, the bot is instructed to include an execute method whose return value will be output when the command is called. The arguments are inferred from this method’s signature, and they, along with the command name and description, are saved to a JSON file which is used to construct the boilerplate on startup.

When a bot-created command is called, the appropriate module is loaded with importlib and the execute method is called from the command’s virtual environment.

Future Plans

• Let the bot spawn (containerized?) AutoGPT instances
• Save on tokens by letting the bot quote previous messages by reference (i.e. named variables) instead of verbatim regeneration
• Modularize components/interfaces and allow the bot to create them
• Dynamically generate boilerplate based on configured components
• Better-than-naive memory recall
• Use recall to identify useful commands per query instead of including them all in boiler
• Use GPT-3.5 for intermediate logic
• Give the bot eyes with MiniGPT-4
• Fork Cool-Retro-Term for a cooler terminal interface

I haven’t released the source yet since the architecture of the whole system is not yet stable, and because it’s still all tangled up with my own utility libraries, but if you’d like to collaborate or are interested in this project in any way, feel free to reach out to me at ken@kenmyers.io.

A demo of the app

This is a webapp I made in December 2020 and am just now getting to writing about. You can view it here.

The idea is to selectively breed melodies with a human in the loop as the fitness test.

Algorithm

More accurately, we breed music generators, or “organisms” as I will refer to them here. Each organism has a set of equations that determine how it generates melodies—this would be the metaphorical genome.

The algorithms I used to both generate organisms’ equations and the subsequent melodies are designed by myself and completely arbitrary.

Generating Melodies

Each organism has two member equations—probability distributions for rhythm/note duration and for pitch. These functions take a pitch/duration as an input, and yield the relative probability of it being output by the organism. Each equation is composed of constants, trigonometric and algebraic operators, variables for the durations and pitches of the melody’s last three notes, and, of course, the input variable \(x\).

A graph of a probability distribution for pitch with the equation \(5cos^2(\dfrac{x^2}{20}-1) + x^{\frac{2}{3}}\)

For example, in the graph above, the peaks around \(x = \pm 9\) tell us that a note nine scale degrees above or below the center note is most likely to be output by this organism, and the dip at \(x = 0\) that the center note itself is least.

The organisms generate melodies note by note, randomly picking pitch and duration with these distributions.

Populating The Bracket

First generation organisms are initialized with randomly generated equations of varying length.

After that, each organism generates two melodies that are put against each other in a randomly seeded bracket. The next generation is created from the winning N (four in my current implementation) through permutation and mutation.

Permutation populates the bracket with every possible combination of winning equations, each organism contributing half its “genome.”

Mutation does one of four things:

• Replaces a random operator/operand in an equation with another
• Scales a constant by a random factor
• Deletes a random operator, and operand(s) if the operator is not unary
• Inserts a random operator (and operand(s))

Each generation comprises all permutations of winning genomes, N mutants (five, currently), the previous generation’s unaltered winners, and N=3 random new organisms.

Implementation

A webapp lets multiple users vote on brackets at the same (or not same) time. This made sense for this project since multiple users means faster generation cycles and less bias.

API

The API is written in Python with Flask, hosted on Heroku alongside a PostgreSQL database for bracket data, and has two endpoints: one to request a pair to vote on, and one to cast a vote.

The first takes GET requests with no arguments and yields two base64-encoded MIDI melodies and their respective identifiers, if any are available.

The second takes POST requests with arguments for the pair and loser’s identifiers, and eliminates the loser from the bracket. If the pair was the last of a bracket tier, new pairs are generated for the next tier. If the pair was the last of the entire bracket, a new bracket is generated from the top organisms as described above.

UI

A still of the UI

The UI is written in plain HTML, CSS, and JavaScript. It is minimalistically designed with graphics I made myself. A random background color is selected and a pair fetched on page load. There are buttons to hear and select each song, cast your vote, and that’s about it.

You can view the source for the UI here and a demo here.

Results

The app was a success in that it functions without error and correctly implements the algorithms described above. The algorithm itself, however, is naive—so far, the melodies do not appear to be growing any more pleasant, nor my methods of genetic recombination to yield organisms similar to their two parents for that matter. I am interested to see what would come of this idea if it used something like Magenta’s MusicVAE to represent melodies in a latent space, where operations performed on melodies have more intuitive results.

Distance optimized map of the 10 closest stars to Earth, Sol included. This one has an average error of around 8.9%.

There was something of a family debate over whether a reduced-dimensionality starmap could still be accurate enough as to be useful, so I made a covid-quarantine experiment of it.

Process

The idea is pretty simple: create a 2D starmap in which the distances between each star are as accurate as possible.

Generation of a map of our 30 nearest stars.

I tried to implement something like a Kohonen map. I’m not sure if it actually falls under that name, but it does follow the same principles of iteratively changing the position of each node towards a more ideal state, which in this case would be one in which the distances between each pair on the board are most accurate. Here’s my algorithm:

• Randomly initialize each item’s position.
• Repeat the following n times or until convergence:
  • For each item in the dataset:
    1. Calculate and store the average pairwise error (discrepancy between true distance and distance on the map) between the given item and all other items.
    2. Shift the item’s position by a certain increment in both directions on each axis, calculating and storing the item’s average error for each possible move.
    3. Commit the move which yields the lowest error, and move onto the next item.
  • If the average error of the whole map has not changed since last iteration, either
    • Decrease the increment size if the results will still be significant, or otherwise
    • Stop. The map’s error has converged.

A few nuances have been glossed over in this overview. You can view the full code on my Github.

Results

One of the principal arguments against the usability of these maps was that “it’s like putting cities in a line.” I made sure to generalize my code for data of any dimension and fed it the largest 20 cities in Texas. If you’re from around here, I think you’d agree with me that this map looks like it’d be helpful to any one-dimensional creatures attempting to traverse the state.

The 20 largest cities in Texas plotted linearly, optimized for distance. (The Metroplex is a bit clustered, as expected.)

Here are a few more of the starmaps I’ve generated, without any post-processing. (You can click on any of these to view them in more detail.)

10 nearest stars. Average error of about 8.88%

20 nearest stars. Average error of about 13.2%

50 nearest stars. Average error of about 16.4%

100 nearest stars. Average error of about 16.0%

As for the error indication, the blue lines represent specific distances that are over a user-inputted threshold (in this case, 85%), the red text simply lists the average error of all distances involving the given star, and the red halos are proportional to said error. All of this is toggle-able.

Something to note is that these maps are quite sensitive to initial conditions, so you may have to re-roll a few times until you get a map you’re happy with. In my experience, the average map error for a given dataset could fall anywhere between 10-30%, but I’m sure this number changes when you’re working with something other than stars or cities.

Beyond Starmaps

Even though the code is capable, there’s not much use in stepping down higher-dimensional data to one, two, or three dimensions because, since it is primarily intended to be used as a cartography tool, my program does not normalize the data you feed it, and uniform higher dimensional coordinates are hard to come by. (If you were, say, a novelist trying to world-build a universe in which there were seven spatial dimensions, then yes, it could be useful.)

I intend to continue working on this project. Besides the obvious refactoring, bug-squashing, and polishing, I’d also like to add support for wrap-around/toroidal space and perhaps globular, a GUI that shows live generation (like the GIF you saw earlier), and the ability to import CSV files.