I’m writing this guide in hopes to demystify backpacking out of NYC for car-less beginner backpackers like me.

Where to go

Without a car, you’re bound by the limits of public transportation. Luckily, the subway and adjacent commuter rail have great access to nature. The spots mentioned here have bespoke shelters as well - in both parks, it is illegal to camp outside designated shelters or campsites.

Appalachian Trail via Metro-North

From Grand Central station, you can take the Metro-North up to the Appalachian Trail Metro-North station. This stop is only serviced on the weekends and holidays - on weekdays, go to Pawling and walk or Uber up to the trail.

As the name suggests, this drops you directly onto the Appalachian Trail. From here, the closest shelter is Telephone Pioneers Shelter. This shelter has both a lean-to and some space to camp. Go a few miles past the shelter and you’ll find Nuclear Lake, a beautiful spot to stop and pull some water using your water filter. This is especially useful since the creek near Telephone Pioneers is frequently dry.

On your way back, I’d highly recommend walking down to Pawling! It’s a cute town in which to grab coffee or food after a long weekend.

Harriman State Park via NJ Transit

Harriman is my personal favorite due to its wide range of trails and shelters. From the train-accessible entrance, there are many forks to take. You can have a new experience every time!

To get to Harriman, take a train from Penn Station to Secaucus Junction and another to the Tuxedo stop. The Tuxedo line runs every day but at irregular times - make sure to plan ahead on the NJ Transit app so you know when to head out and back.

Once in the park, start your trek to one of the many shelters. Download the Harriman Trail Map PDF on your phone or use AllTrails to find your way around. Note that on holidays, the main Harriman shelter areas can get quite crowded. If you have a car, this is a good opportunity to start your hike at places less accessible by train.

Gear

Backpacking gear chat is centered around one thing: weight. You’ll be carrying your pack the whole way, so even slight differences in weight matter. This is what the Ultralight movement aims to optimize for - we even have our own r/nycultralight!

Backpacking gear can get expensive, so it makes sense to simultaneously optimize for cost. The upshot is that the gear is quite durable and will last you for years, but the downside is the initial cost is quite high. Once you have a good setup though, you can take trip after trip for just the cost of food and transportation!

Backpack

Most backpacking packs sit in the 55-65 liter range - this balances holding enough stuff while staying light. When getting a pack, the most important thing to check is the fit. A good backpack will have waist and chest straps with a hard backplate - this allows it to sit on your back, evenly distributing weight along your back. A good fit is extremely important, so unless you’re experienced, start at someplace like REI where you can feel out different bags.

Tent

Your tent is frequently the largest item in your backpack, so it’s important to get one that’s right for you! There are many cheap ($50) tents on Amazon, but buy at your own risk - standard camping tents are bulkier and heavier than backpacking tents. They can get you started but will slow you down on longer hauls. I’m currently using this one, and it’s absolutely painful to backpack with; it takes up more than half of my backpack alone. Upgrade coming soon.

There are websites that can give better tent reviews than me, but you’re mostly optimizing for sleeping area size vs. packed size/weight. I like a two-person tent even for myself so I can lounge out. If you’re looking to go as light as possible, some popular tents forego any inbuilt supports and use your trekking poles instead.

Sleeping

You can use a standard sleeping bag for backpacking. I have the REI TrailBreak 30, and I’m super happy with it! Advanced users can venture into the world of quilts, but that’s above my paygrade.

To give yourself some cushion at night, you’ll need some sleeping pad. The main options are:

1. Air inflatable: These require some work to inflate and are prone to accident since popping the mattress is game over. However, they pack the smallest and are the most comfortable! These are my personal favorite - I use this cheap one.
2. Self-inflating pads: These provide some cushion and autoinflation via foam, but they also seal in the air to provide a more comfortable surface. They’re quite comfortable but come at a price. The nice ones are very expensive and the cheap ones lose their ability to inflate after being stored for too long.
3. Closed-cell foam: This is what a lot of legit backpackers go with. It’s impossible to pop and hence deflate, and it can double as a seat cushion. It’s also probably the least comfortable. Side sleepers beware!

If you’ll be camping in the cold, pay attention to the R value of your sleeping pad. This describes how much insulation your pad will give you from the ground. Remember that you can always increase the R value of your setup by adding a closed-cell foam pad underneath your existing pad.

Misc

Some important but smaller items:

• Water filter: allows you to cut down on your total weight by avoiding packing water. The straws attach to standard water bottles and are very easy to work with.
• Toilet paper & trowel: nuff said
• Bear bags: absolutely critical for hiking in bear country (most of the American Northeast)
• Regular plastic bags: you’ll never regret another bag or two for trash or general organization.
• Flashlight/battery pack: The night is scary without a flashlight, and your phone may die quickly. I’m happy with this one off Amazon
• First aid kit: Some band-aids or bandage kit are great in a pinch

Eating

For your first trip, it’s easiest to pack cold food to avoid having to buy a stove. You can cold soak or go with some trail classics like pasta salad, trail mix, tuna salad, or sandwiches. I always bring some sardines and crackers for quick and filling snacks.

That said, a camping stove and pot are a huge upgrade - they make the cold more bearable, and they allow you to cut down on your total weight. I have the Soto Windmaster and the MSR Titan 900. The windmaster and a fuel canister fit inside the Titan, so my whole stove setup only takes the space of my pot.

The easiest (and quite tasty) foods to cook are the pre-packaged camping meals. These can get pricey and are high in sodium, so to mix things up, you could try:

• Rice and lentils with tinned fish over the stove - my personal favorite
• Any instant microwave meal like mashed potatoes
• Dehydrated meals from the internet. Renee and Tim have some great stuff.
• Anything else you can dream up :)

Some final advice

Before closing out, in no particular order:

• Check weather and temperature before heading out. For me, the optimal times are fall and spring when the weather is 50-75°F.
• Tell friends you’re leaving and when you’ll be back so they know to be suspicious if you don’t make it back.
• If your iPhone does not have satellite SOS capabilities, consider buying an SOS beacon.
• Remember that you’re a guest in nature. Practice Leave No Trace so we can keep backpacking for years to come.

Now go have some fun in nature with your friends!

Credit to Emily Oster on Twitter.

I juiced this base artifact up a little using Cursor chat, and now it looks much better! View it here.

Big thanks to Simon Willison for giving me the inspiration to make this simple visualizer.

My best example of this is the Kumon method of teaching. Kumon works by having students read some material about their next unit. After reading this material, they do worksheets in the classroom and at home. These worksheets are graded when they come in for their next session, and if they’re doing well enough, they get the opportunity to take a test and graduate to the next level. Students will learn arithmetic at a very young age and become proficient quite quickly. This makes for cosmetically impressive results - I’ve never seen anyone do multiplication faster than a 7-year-old who’s been doing Kumon for years.

I think these results are a mirage. While students are able to progress through the units, they do it by learning tricks. They learn the keep-change-flip method of dividing fractions, or they learn to solve quadratic equations by memorizing the quadratic formula. These tricks will get you the right answer, but they don’t help you build any deeper intuition about what’s going on. Learning the quadratic formula won’t help you understand differentiation better, but completing the square will! The keep-change-flip method won’t help you understand negative exponents, but splitting up the fractions into their component parts will. In both cases, the “trick” will get you to the right answer reliably and is fast to learn. It won’t, however, help you generalize onto future situations.

I’m picking on Kumon, but I don’t think the problem is exclusive to that system; they’re just the worst offenders that I know of (I was a Kumon tutor in a past life). I’ve had plenty of math teachers try to teach through these short-sighted tricks. I don’t blame them - their explicit reward system is only tied to standardized test results and students’ grades. In this system, I would also try to shuttle along kids so I could get along with my job. It’s only the best teachers, the ones truly in it for the love of the game, that put in the effort to teach differently. I’ve had a few of these kinds of teachers, and they changed my life.

The way we teach kids how to learn in school has ripple effects on the rest of their lives. In fact, I think this problem only rears its head in higher-level education or in the real world. In a real job, these tricks don’t exist in the same way: there’s no textbook with which to learn shortcuts to the “right answer” if such a provably right answer even exists. Instead, you’re forced to take in your surroundings and build a mental model for how your domain works. Building a strong understanding of your domain helps when doing routine tasks, and it’s literally required when trying to build something novel.

I’m particularly interested in this because so much of my “education” has happened outside the classroom. I’ve learned primarily through side projects and learning on the job. In either case, I didn’t read a book and memorize some tricks; nor did I just try to accomplish the task in the most expedient way. Instead, I’ve tried to focus on the process of building my DAG. By focusing on building a wider understanding of the system, I’ve been able to generalize to new situations and build novel solutions. For me, this deeper understanding is also more durable: when I can slot information into my DAG, it sticks with me near-permanently.

While it’d be nice to neatly wrap up this post with a suggestion on how to incentivize this teaching style, I honestly don’t know how. Pedagogy reforms like Common Core have largely been a disaster - this style of instruction requires a deep understanding of the material and the teaching ability to execute on this understanding. Even if we had a fleet of teachers all of whom were capable of teaching like this, they aren’t incentivized to. The benefits of this way of thinking play out over years, so you can’t reward this behavior on a per-teacher basis. It would never show up on a standardized test.

It may be that we need new interactive styles of instruction to unlock this, but these beliefs would need to be embedded into those systems. All I know is that when I’m teaching, I’ll continue to encourage my students to think about the wider picture. I hope that you will too.

This was infuriating because I’ve never had trouble finding my voice orally. My whole life I’ve been speaking, teaching, and debating with passion. These experiences always felt different to me: when writing I felt friction just arranging my words onto the page, but when speaking, I felt a flow that allowed me to say exactly what I meant exactly how I wanted. At the time, I saw writing as an immutable weakness of mine. Some people are good at some things, some people are good at other things.

This feeling changed for me when I started to write about things I enjoyed. I have a distinct memory of writing a long essay about Edward Snowden at some summer camp - I turned it in expecting for it to be ripped apart, but I was met with praise instead. I built a part of my identity around this weakness, accepting that some people are good at some things while some are good at others. While there’s some truth here, this way of viewing myself was artificially limiting.

And so I wrote about why to use fish not bash; about why I should buy a Pebble; about random bits of network engineering. This moved me in the right direction, yet it wasn’t enough. Even though I started to write about things that lit me up, I still had expectations I was holding onto. Whether it was my boss, teacher, or some imaginary hyper-critical netizen, I was worried that someone more skilled than me would read my work and look down on me. For this reason, I wrote almost exclusively technical posts. Tech is something I’ve always been confident in; it allowed me to write without fear of judgement.

I have so much more to say than just explainers about tech! It’s only been through writing braindumps that I’ve felt comfortable getting those words out, though. By giving myself a space to write with no expectations, I’ve written some of my most proud words. I’ve had the chance to profess my model of teaching, thank the best mentor I’ve ever had, reflect on friendship, and much more. In letting go of my fear of being perceived, I’ve finally found my voice.

But now that I’ve found it, what will I say?

This style of work is foreign in a lot of the industry where design docs are the primary planning vehicle. Ideally, design docs let you gather feedback and de-risk the execution of an idea before you start writing code. This is not my experience. In particular, when executing refactors, the devil is in the details - small edges in how an API is used can change large parts of its implementation. With design docs, it’s very difficult to foresee these edges; a working prototype can teach you much more.

This isn’t to say that design docs aren’t valuable! After building my prototype, I’ll reflect on it and write a design doc so that others are aware of my plans and can give any guiding feedback. I do find, though, that having a working prototype tends to end a lot of debate. We can get lost in the “is this possible” weeds; the point of a prototype is to remove this ambiguity.

After making my low-quality prototype and getting light consensus, I PR in my prototype bit-by-bit. In this pass, I’ll write nice docs, write tests, and smooth out the hairy edges of my prototype. At this point, I’m effectively doing a rewrite of my previous implementation, so I get to make all the decisions I wish I had made in the first place! I think this results in higher-quality code than writing it cold off a design doc.

This approach is not reasonable at all scales. At some point, it’s infeasible to write a prototype, and you’re better off designing with trusted co-authors. This is especially true in larger orgs with cross-functional dependencies. That said, I think you can push prototypes far further than people expect with mocking and getting your hands dirty in others’ codebases.

This also relies on a particular skill: being able to write bad code very quickly. The core element of execution is the ability to write a prototype quickly enough where you’re not slowing down your total velocity. Luckily, I’m great at writing bad code.

As a 16-year-old being given the opportunity to do network engineering part-time, I was just grateful to have a job that I enjoyed. I quickly picked up Linux and networking basics, but it was soon on to the next thing. Once I picked up one skill, I was given a new assignment with higher complexity. This process of laddering never ceased.

Most of the time, I didn’t believe that I could do the next task, but Jon had an almost maniacal belief in me. This belief gave me no choice to but to push myself into learning and getting the job done. I’d be at client sites with broken networks and no choice other than to fix the issue. I’d be staring down borked libvirt configurations with no choice but to figure out the right hardware options. Over time, I grew to love this feeling - the feeling of never knowing the answer, always having to work to figure it out.

I now crave this experience. I want to constantly push myself into more difficult work, and I want to excel in that work. In part, this is out of a love for the process, but it’s also out of a feeling of obligation. I feel that I’ve seen through a one-way door; I can’t go back to not pushing myself when I know how much more I can accomplish.

The challenge is that nobody will ever push me as hard as Jon did. I’ll never have someone with that maniacal belief in my abilities, so in his place, I have to push myself. I hope that I can one day pass on the favor to someone else.

It’s not only that this is hard but also that it forces you to teach suboptimally to each student. When building a single piece of curriculum to teach everyone, you have to build up from your students’ lowest point of shared knowledge. This means you’ll go too slowly for the high-performers, and you’ll inevitably leave some students in the dust. In school we try to control for this with prerequisites and waitlist screening forms, but these tools are far too coarse-grained to be useful.

Written materials like textbooks solve some of this since students can move at their own pace, but they don’t solve the interactivity problem. The only way to learn at max difficulty & speed is to do so interactively. When you learn something, you must think hard about it and test your understanding by asking probing questions. Textbooks can teach you the information, but they can’t help you test the boundaries of your understanding.

We use broadcasting because it’s economically scalable. One teacher can broadcast to 30 students. One textbook can reach millions. In a perfect world, we’d be able to give each student their own PhD-level tutor who can tailor instruction to exactly their knowledge and skill level. Sadly, this is cost-prohibitive.

I have high hopes for AI tutors as a way to solve this problem. Seeding an LLM with high quality data will allow users to interactively learn about the topic of their choice in a high-bandwidth, personalized, and social way. Hopefully, my kids will be much smarter than I am.

When I think about that 7th grader booting up the Codecademy Javascript course, all I can do is remember the joy of those early days. I remember building my choose your own adventure batman game. I remember making a silly little paint application. I remember building Tennis Scorekeeper+ (the plus was so people would think it was the pro version). Most of all, I remember the complete joy of writing code with no strings attached. Just in it for the love of the game.

Being in it “for the love of the game” is still the core of my experience. There’s a lot of noise out there about the next wave to jump on. There’s a lot of ladder climbing that will undeniably increase your TC or land you a job at . While I don’t have a moral issue with any of that (get that bag), it’s just never been motivating for me. I’ve always just wanted to work on problems I find interesting with people I like. And maybe do some good for the world.

I want to hold onto this for as long as I possibly can. I see far too many people jaded about their jobs, playing the corporate game so they can get that new apartment. While playing the ladder will lead to short-term benefits, I genuinely believe that the long arc of time favors those who are just trying to do their best for the sake of doing their best.

The last 12 years have been filled with surprise after surprise. Software has brought me success that I could have never imagined. It’s taken me far from home to a place I thought I would despise but instead love. It’s brought me my closest friends in life.

The road here has been long, and it’s been paved with the goodwill of so many around me. The support of my family; the camaraderie of my friends; the wisdom of my mentors. I’m grateful for all of it, and I look forward to continuing to pave it for many years.

I just wish my dad was here to see it.

services.syncthing = {
  enable = true;
  user = user;
  configDir = "/home/${user}/.config/syncthing";
  dataDir = "/home/${user}/.config/syncthing/db";

  overrideDevices = true;
  overrideFolders = true;

  settings = {
    devices = {
      # map from device name to ID
    };

    folders = {
      # generic sync folder
      "cccjw-5fcyz" = {
        path = "/home/${user}/sync";
        devices = [
          # list of device names to share with
        ];
      };
    };
  };
};

To modularize this, I created a Nix function with the user parameter:

# ./custom/syncthing.nix
{ user }: {
  # same config from above
  ...
}

Then, in my individual NixOS configuration files, I can just import it in my imports line and pass in the right param:

# ./systems/asahi/default.nix
{ config, pkgs, ... }:

{
  imports = [
    ./asahi-hardwarecfg.nix
    ( import ../../custom/syncthing.nix { user = "peyton"; } )
  ];
...

And voilà! I’ve now DRYed up my configuration files. Here’s the config for the curious.

In my head, I model these relationships as a DAG. Each node is a concept, and concepts have arrows to the other concepts that depend on them. In order to fully understand one node, you must first explore all the nodes that point to it. You have to bottom out the recursion at some point by taking some facts as axiomatic, but IMO, the further you can go down the better.

As a teacher, your job is to make this DAG exist in the heads of your students. Doing this requires an understanding of two things: how you’ve built your DAG and the current state of your students’ DAGs. These are both deceptively difficult things to understand.

The obvious one comes first: it’s impossible to know what your students’ level of understanding is. The best you can do is try to guess based on past experience and filter admits to your class based on some criteria. But no matter what you do, different students will come in at different levels of understanding, and students will have gaps in their knowledge you never predicted.

The harder part for me, though, is grokking how my own understanding is built. When you’re deep in a subject, you’ve developed an intuitive understanding so deep that you don’t even realize it’s there. When teaching Kubernetes, you preach the wonders of its networking model only to realize its value is lost on kids who’ve never dealt with HA systems. It’s not that you expected them to understand high availability - it’s just that you forgot it was an integral part of your thought process.

It’s for all these reasons that I think most professors are very bad teachers. High-level research staff are so far away from undergrads that they can’t begin to empathize with their mindset. And they certainly can’t introspect enough to fully understand how their DAG is built.

Hopefully in the future we’ll see more classes taught by students, TAs that are undergrads, and tenure structures that reward the truly exceptional instructors.

The hard part here is that I think it’s super important to stay up to date on things! It’s probably a good idea to keep track of the LLM space, but it’s so hard to avoid the flood of grifters. Somehow I’ve managed to avoid getting sucked into these pits, but I’ve never been sure why.

It took way too long to come to this realization, but for me, it’s all about the people I’m around. My friends are the most talent-dense group of people I’ve met in my life, and I learn so much just by being around them. Whatever online gurus can offer, you can get 1000x that just by being around people who love the game just as much as you do.

For me, this usually just means seeing what lands in the group chat every day. Whether it’s the corporate drama of the day or someone going way too deep on their website’s build system, I wake up every day looking forward to learning something new.

And all they’ve taught me doesn’t hold a candle to how they’ve been there for me ❤️

If you’re starting to trying to be a musician or artist—something like that—because you wanna make money, because you wanna do a job, that’s- that’s the wrong way. You have to do this because you love it. And it doesn’t matter if you broke, you still gon’ do it. I mean, I go out to jam sessions, and I play regardless of whether I’m getting a check or not. It’s-, it’s about whether I, uh—you have to love this thing, man! You have to love it and breathe it and—It’s your morning coffee. It’s your food. That’s why you become an artist.

Interview with Roy Hargrove: https://www.youtube.com/watch?v=mIaONoYeDh4

The greatest intro to ever hit an album: https://open.spotify.com/track/1ZwejHvd2KmKCWHn9HpAEw

If you need generalizability, I totally get it. Things like chat applications require a high level of semantic awareness, and the model has to respond in a manner that’s convincing enough to its users. In cases where you want the LLM to produce something human-like, it makes sense that the brains would need to be a little juiced up.

That said, LLMs are a whole lot more than just bots we can chat with. There are some domains that have a tightly-scoped set of inputs and require the model to always respond in a similar way. Something like SQL autocomplete is a good example - completing a single SQL query requires a very small context window, and it requires no generalized knowledge of the English language. Structured extraction is similar: you don’t need 2.7B parameters to go from remind me at 7pm to walk the dog to { "time": "7pm", "reminder": "walk the dog" }.

I say all this because inference is expensive. Not only is it expensive in terms of raw compute - maintaining the infrastructure required to run models also gets pretty complicated. You either end up shelling out money for in-house talent or paying some provider to do the inference for you. In either case, you’re paying big money every time a user types remind me to eat a sandwich.

I think the future will be full of much smaller models trained to do specific tasks. Some tooling to build these already exists, and you can even run them in the browser. This mode of deployment is inspiring to me, and I’m optimistic about a future where 15M params is all you need.

But this experience is not for the faint of heart.

When the package (and version) you need is inside of nixpkgs, you’re living the good life. You can just add the package name to your package list, hit a switch, and start using the package.

When you need an older version of a package, you end up trolling through git history and locking nixpkgs to the revision containing the version you want. If you want a newer version, you might have to update the derivation yourself and PR it in!

You can avoid some of these headaches by writing an overlay, but depending on how cleanly the nixpkgs derivation is written, this can range from “wow that was easy” to “why have I spent 10 hours on this”.

And in the apocalyptic case where you can’t get a derivation written, you might need to build an FHS environment. But even if you do this, programs outside the FHS environment can’t use the FHS tooling. This leaves you either putting everything inside this FHS environment or gitting gud at writing derivations.

I’ll keep using NixOS, but boy do I miss the days of curl https://cooltool.com/install.sh | sh.

Asahi can technically work with any distro, but the “officially supported” (insane levels of air quotes here) is the Fedora Asahi remix. I ran this for a while and had a really excellent experience. It was the smoothest out-of-the-box Linux experience I’ve ever had. Low bar there, but we’ll take what we can get.

Asahi can work with NixOS, and I’ve had a pretty good experience running this setup. There are some prebuilt NixOS modules to make this easier.

The Asahi subreddit is a good place to go for more info and support.

One benefit in practice is that it’s a really good package manager. Nixpkgs has a ton of packages, and it remains relatively stable even if you follow its “unstable” branch. Smarter people than me probably know why this is, but I’m guessing that the “complete view of the world” allows Nix to make much stronger guarantees about what will work and what won’t.

It’s also great at managing user-level configuration. Home manager has friendly configuration syntax into a ton of commonly-used programs. Using home manager was my first “wtf” moment with Nix. I’ve been trying to solve the reproducible home problem for so long, and Nix just fixed all my problems.

If you’re looking to try out Nix, install it and migrate your home bit-by-bit to home manager! You’ll be shocked at how easy it is to set up. If you want to get a little deeper, this book on NixOS and flakes is an amazing resource.

$ ollama serve &
$ ollama run llama2:latest
>>> say whatever you want to say to llama

The real magic, though, is in how easy it is to define new model types with different system prompts and parameters:

$ cat Modelfile
FROM llama2:latest

TEMPERATURE 1
SYSTEM """You are an assistant who will write Peyton's blog posts for him"""
$ ollama create peytonhelper -f Modelfile
$ ollama run peytonhelper
>>> say something to my helper

In the past, I’ve used ctransformers for building stuff like this, but it’s a bit too low-level. I don’t care to learn the proper prompting setup for all these models.

I don’t use Spacemacs, but I feel the same way about it. The way it uses Vim motions to create easy-to-use chords feels so intuitive to me. I’ve taken the Spacemacs vibe and created my own set of keybindings in the same spirit that work for me.

I use <Space> as my leader with all my actions hanging off of it. Some examples:

• <Space>{h,j,k,l} - move directionally in a pane of split windows
• <Space>f - symbol search over the project
• <Space><Space> - open a pane with all my recently-edited files

I’ll add more in the future, but for now, you can see some examples in my VSCode config.

Where I’ve had much more reliable results is in using LLMs for extracting structured results from unstructured input and in using them for text manipulation. These both rely less on the LLM being able to “think” and more on it being able to follow direct instructions.

Recipes

In 2021, I made a recipes page, and I seeded it with some recipes. Sadly, I never ended up adding more. While the main reason for this is my laziness, it’s also just hard to make good recipes! Formatting ingredients, giving easy-to-follow instructions, and just making the new page is a real pain.

This is a bummer for me since for any of my recipes, I can easily articulate in words how to make it, but I find writing those instructions down very difficult. Recently, I’ve turned to voice recognition and LLMs for assistance on this.

At a high level, I want to enable the following workflow:

• Dictate a recipe using my voice
• Have an LLM generate a nicely-formatted recipe
• Put that recipe on my website
• Dictate any modifications I want to make and have them reflected in the recipe.

Voice Transcription

The first step of all of this is to go from voice -> text. I don’t have much familiarity with recording in Python, so I mostly asked ChatGPT to write the code and told it what was broken. The code for this isn’t all that interesting, but it provides a nice interface that I can use to record audio:

# This will record audio to the file listen.RECORDING_FILE
listen.record_audio()

To go from this recording file to text, I used a python port of the incredible whisper.cpp project. I am consistently stunned at how well whisper.cpp works and how quickly it runs. It outperforms the in-built MacOS dictation by such a wide margin that you’ll never want to use Siri again.

With whisper set up, I now have a super simple function I can use to get voice input:

def get_voice_input(whisp: Whisper) -> str:
    listen.record_audio()
    return whisp.transcribe_from_file(listen.RECORDING_FILE)

Here’s where we are so far:

• Dictate a recipe using my voice
• Have an LLM generate a nicely-formatted recipe
• Put that recipe on my website
• Dictate any modifications I want to make and have them reflected in the recipe.

Function calls

I alluded to function calls before, but it’s worth taking some time to talk about what they really are. In June, OpenAI announced an extension to their API to allow GPT chat completions to call functions instead of just responding with text. You simply provide a JSON schema describing the available functions, and the LLM will make a decision on whether to call a function and what arguments to call it with.

This has primarily been billed as a tool to build agents. For example, if you want to build an agent that can tell you the weather, you can give it a “get_current_weather” function that’ll make a call out to some weather API. Then, when you ask it “what’s the weather”, it’ll make the function call and respond to your question in plain english using the results of the call.

The powerful thing here is that the function calling is extremely reliable in only ever making calls that conform to the JSON schema. I’ve had very few issues, and I was able to fix any issues by prompting the LLM more clearly.

Autogenerated schemas

JSON schemas are pretty arduous to type out, in particular because you have to define your schema once in code and once in the JSON schema you provide to the LLM. To alleviate this pain, there’s a library that generates OpenAI-compatible JSON schemas straight from Pydantic objects. This allows you to define type-safe schemas that LLMs can understand “for free”.

From the docs, here’s an example of using the library to extra user information from input:

class UserDetails(OpenAISchema):
    """User Details"""
    name: str = Field(..., description="User's name")
    age: int = Field(..., description="User's age")

completion = openai.ChatCompletion.create(
    model="gpt-3.5-turbo-0613",
    functions=[UserDetails.openai_schema],
    messages=[
        {"role": "system", "content": "I'm going to ask for user details. Use UserDetails to parse this data."},
        {"role": "user", "content": "My name is John Doe and I'm 30 years old."},
    ],
)

user_details = UserDetails.from_response(completion)
print(user_details)  # UserDetails(name="John Doe", age=30)

You might be able to see where I’m going with this :)

Recipe extraction

In order to render a recipe onto a page, I need to know its components. I thought the following were pretty typical attributes of a recipe:

• Description
• Steps
• Ingredients
• Total Time
• Active Time

Using the function call library, I wrote up a schema for recipes:

class RecipeDetails(OpenAISchema):
    """RecipeDetails are the details describing a recipe. Details are precise and
reflect the input from the user."""
    description: str = Field(..., description="casual description of every recipe step in approximately 200 characters")
    steps: List[str] = Field(..., description="""specific list of all steps in the recipe, in order.
If multiple steps can be combined into one, they will.""")
    recipe_title: str = Field(..., description="title of the recipe in 20 characters or less")
    ingredients: List[Ingredient] = Field(..., description="list of all ingredients in the recipe")
    total_time: int = Field(..., description="total time in minutes required for this recipe")
    active_time: int = Field(..., description="total active (non-waiting) cooking time in minutes required for this recipe")

I then wrote a generic function to extract any schema from unstructured text and called it on the transcript:

transcription = get_voice_input(w)
recipe = llm.get_shema_obj("RecipeDetails", elems.RecipeDetails, transcription)

Now, recipe is a structured object that I can pull all of my recipe information out of.

• Dictate a recipe using my voice
• Have an LLM generate a nicely-formatted recipe
• Put that recipe on my website
• Dictate any modifications I want to make and have them reflected in the recipe.

Rendering the recipe

I already use Hugo to host my blog, so I won’t fix what ain’t broke. First, when I create a new recipe, I make a small markdown file inside of my content/recipes directory with some metadata and a reference to the JSON-formatted recipe info:

+++
title = "The Good Lemonade"
description = "Refreshing and sweet lemonade like they served at the country fair"
date = "2023-08-06"
total_time = "1440"
active_time = "20"
+++

{{< ai_recipe url="data/recipes/country_fair_lemonade.json" >}}

This ai_recipe shortcode is a pretty simple template that displays all the elements of the JSON recipe. The JSON is just the RecipeDetails class serialized.

And that’s it for the first pass! I can now speak into my mic and get a nicely-formatted recipe on my website!

• Dictate a recipe using my voice
• Have an LLM generate a nicely-formatted recipe
• Put that recipe on my website
• Dictate any modifications I want to make and have them reflected in the recipe.

Modifications

These recipes are never perfect in the first iteration. The ingredients are typically overspecified, and the steps definitely suffer from miscommunications.

To solve this, I built a new mode into my REPL for performing modifications. I just ask the user for a link to the recipe they’d like to modify, and I pull the unique identifier out of the URL. I then read in the JSON for that recipe so I can kick off modification:

url = nice_input("give the url for the recipe: ")
# Regular expression pattern
pattern = r'https?://[^/]+/recipes/ai_([^/]+)/?'

# Use re.search to find matches
match = re.search(pattern, url)

if match:
	recipe_name = match.group(1)

	json_path = os.path.join(presentation.hugo_base_dir, "data", "recipes", f"{recipe_name}.json")
	with open(json_path, 'r') as f:
		file_contents = f.read()
	recipe_dict = json.loads(file_contents)
	recipe = parse_obj_as(elems.RecipeDetails, recipe_dict)

Modification REPL

In order to make modifications relative to the current state, I need to show the LLM what the current state of the file is. I don’t want to pass in a full HTML page, so I made a text analog of my webpage using jinja:

Recipe Title: {{ recipe.recipe_title }}

Total time: {{ recipe.total_time }} minutes
Active time: {{ recipe.active_time }} minutes

Recipe Description: {{ recipe.description }}

Recipe Ingredients:
{% for item in recipe.ingredients -%}
- {{ item.quantity }} {{ item.unit }} {{ item.name }}
{% endfor %}
Recipe steps:
{% for step in recipe.steps -%}
- {{ step }}
{% endfor %}

I then ask the user for voice input and instruct the LLM to make modifications to the recipe according to the user’s instructions:

modifications = get_voice_input(w)
new_recipe = llm.make_modifications(modifications, recipe)
# use the same parsing as before to turn unstructured recipe into pydantic class
recipe = parse_recipe(new_recipe)

I can then write out the new recipe, and the user can look in their live Hugo view to see if it looks good:

There are some cases where the LLM will make modifications too aggressively or it’s just easier for me to make these changes manually. For these cases, I open up a vim buffer with the recipe and allow the user to make whatever changes they wish. I use the LLM to parse the required info out the same as before.

• Dictate a recipe using my voice
• Have an LLM generate a nicely-formatted recipe
• Put that recipe on my website
• Dictate any modifications I want to make and have them reflected in the recipe.

Conclusion

This tool was a ton of fun to build and taught me to think more precisely about how to use LLMs. I really believe that the current assistant/agent-focused discourse is pretty misguided. While assistant applications may come into the mainstream, I think that in the near term, LLMs will be useful as tools to get jobs done rather than agents to replace humans.

Local LLM epilogue

I’ve been playing around with local LLMs a lot as well. They are not even close to GPT quality, but they’re getting better every day. Hopefully Llama 2’s commercial license will open the floodgates.

With grammar based sampling being merged into llama.cpp, I think we’re a JSON-trained model away from being able implement function calling locally. Some have already taken a stab at it, and I have high hopes for restricted sampling being the key to eeking out better performance from local models.

At some point in the future, I’d like to rip out the OpenAI calls here and replace them with a local model. That would be sweet.

How Golinks Work

The goal of a golink is to allow the user to input a short URL like go/gh/ and be redirected to a longer url such as https://github.com/. These should also be path-preserving, so go/gh/pawalt should go to https://github.com/pawalt.

This is primarily achieved by reconfiguring DNS with a special mapping for go and serving a redirect whenever we see one of our redirect paths.

DNS lookup

When the browser (or any HTTP client) looks up a URL like go/gh, it first has to know what IP to send the request to. To do this, it looks up the host of the request (in this case go) using a DNS server. We need to get some access to our DNS server so that we can set the go entry to our IP.

Additionally, if we want to serve redirects on hosts other than go (gh -> github.com for example), we’ll need to update DNS with each host that we want to redirect from.

Serving the redirect

Now that a request for go/ has made it to our application, we need a way to serve a redirect to our destination URL. In order to serve the redirect, we’ll run a web server that accepts request on all routes. When it gets a request, it looks up the host and path in its list of redirects. If it finds a matching redirect rule, it’ll serve an HTTP redirect (usually 302) to the destination URL.

The combination of DNS + HTTP redirect means requests for any source URL will get automatically redirected to their destination URLs.

Editing the redirect list

With the mechanics in place, we need a way to edit the mappings from source to destination URL. For these, we can simply run a web server that retrieves a list of the current mappings and displays them, allowing the user to edit the list. When the user saves a new set of mappings, it must be stored ✨ somewhere ✨, and DNS must be reconfigured for any new hosts that are added.

With each component, we get a workflow that looks like:

1. User adds a new redirect in web editor
2. New redirect is saved, and DNS is reconfigured with any new hosts
3. User can make a request and be redirected
   1. DNS points the host to the right IP
   2. The webserver serves a redirect to the desired URL

Implementation

To implement this, I’ve got two devices I’m talking to - my router for the DNS and the computer running the redirect webserver. In this case, I’m using a fly.io VM to host the redirect webserver. Each of these devices might be on completely separate networks with no way to talk to each other.

Tailscale

To connect my devices, I’m using Tailscale. You can think of Tailscale as a zero-config peer-to-peer VPN solution, although it’s really more than that (words on this later). Tailscale gives each of my devices an IP in the 100.x.y.z range and connects them using a WireGuard tunnel. This means that even on different networks, my laptop can directly access both devices on consistent IPs.

With a little extra work, we can even get this connected to the entire home network, meaning non-Tailscale devices can still use golinks. To do this, I designate my router as a subnet router, meaning that Tailscale knows to route traffic for my home network through it. In this case, my home network is 172.27.0.0/16, so any requests for that range will go through my router, 100.2.3.4.

Host file

First, we need to get DNS overriding working - requests for URLs like go/ should be sent to our VM in Fly. We can achieve this by adding a hosts file to our router’s DNS settings. I’m using OpenWRT on my router which uses Dnsmasq for DHCP and DNS. Just as /etc/hosts lets us define custom host -> IP mappings on our computers, we can add a hosts file to Dnsmasq for our own custom mappings.

The hosts file will use the Fly VM’s tailscale IP as the IP to connect to, and it’ll store the mappings in comments. This information is our entire state which means we can just use this file instead of any databases. Here’s how a table of redirects would be formatted down in our hosts file:

root@OpenWrt:~# cat /etc/hosts.d/golinks.hosts
# this hosts file property of me!!! dont touch


100.79.24.134 gh # gh -> https://github.com
100.79.24.134 gh # gh/pl -> https://github.com/pennlabs
100.79.24.134 gh # gh/pw -> https://github.com/pawalt
100.79.24.134 go # go/gh -> https://github.com
100.79.24.134 go # go/mon -> https://www.youtube.com/watch?v=b2F-DItXtZs

Serving the redirect

Serving the redirect is fairly straightforward - when a user makes a web request to the Fly VM, it will look up the link in its redirect map and serve a redirect (302 Found) to the redirect destination. If it can’t find a matching redirect, it’ll serve a 404.

Editing the redirect list

To view and edit the redirects, the same webserver that does the redirecting serves a UI on go/_/hosts. I use a weird path (/_/hosts) so there’s a low chance of collision with any URL I’d actually want to go to via a golink.

When the user hits Submit, the list of redirects is POSTed to /_/hosts. The list is then persisted in the router’s golinks host file (mine is /etc/hosts.d/golinks.hosts).

Mapping persistence

As previously mentioned, the information on the mappings is stored entirely in the router’s hosts file so as to not have any local state to deal with. In an ideal world, I could just use SFTP to read and write the mapping data in the hosts file, but this normally requires me to mint an ssh key and distribute it to both Fly and my router.

Luckily, Tailscale SSH obviates the need for any of that! By running Tailscale SSH on my router¹, I can SSH into it via any device I’m logged into. This means that my Fly VM can SSH into the router without providing any credentials - simply by being on the tailnet, it has the authorization to SSH².

Reading

To read in a mapping, the application will SFTP into the router and read the hosts file (default /etc/hosts.d/golinks.hosts). It’ll then parse out the comments and update its stored mapping to reflect the mappings in the comments.

This list is refreshed every 5 minutes in case of a out-of-band edit to the file.

Writing

Before writing out the hosts file, I have to answer a question: what IP should be advertised to the clients i.e. what IP does the Fly VM have that is routable from my laptop? It’s the Tailscale IP! To get the Tailscale IP, I dial a UDP connection to 100.100.100.100, one of Tailscale’s managed IPs. Since it’s Tailscale-managed, the source IP on this connection is my Tailscale IP.

To render out the hosts file, I have a very simple template:

# this hosts file property of me!!! dont touch  
  
{{ range .lines }}  
{{ .ip }} {{ .host }} # {{ .comment }}  
{{- end }}

I render this template and SFTP it over to the router to save the new configuration, but there’s one last trick. Dnsmasq won’t immediately see the new host file, so it might be a few minutes before I can use my link. However, if Dnsmasq receives a SIGHUP, it will reload all its config - including the hosts files. To send the SIGHUP, I use a quick command: kill -HUP $(ps | grep dnsmasq | grep -v grep | cut -d " " -f1). There’s definitely a better way to do this with syscall.Kill, but that’s a task for later.

Now that we have persistence, our end to end process looks like:

1. User adds a new redirect at go/_/hosts
2. New redirect is saved via sftp to /etc/hosts.d/golinks.hosts, reconfiguring Dnsmasq
3. User can make a request and be redirected
   1. DNS points the host to the Fly VM
   2. The webserver serves a redirect to the desired URL

Conclusion

In a small bit of code, we’ve managed to get network-wide golinks and keep them on the go! I’ve got Tailscale installed on all my devices, so this solution works wherever I am on whatever device I’m using. This new SSH feature has changed the way I see the product - from a VPN solution to a network mesh with trust built in. I hope to see more features like this in the future!

If you want to check out any of the code for this project, head over to my homelab repo. Just be warned: the code is definitely homelab-quality.

The Work

As soon as I started this job, I immediately loved the work. I latched onto networking and sysadmin work in a way that I had never latched onto any schoolwork in my life. Work provided me constant challenges that I could bash my head against and eventually overcome. When I conquered one challenge, the next was sitting there, waiting for me to run it over.

Additionally, I received praise like I had never before in my life. In school I was a high performer, but I was rarely top of my class. Even when I was, it never felt gratifying. Grades have never been a motivating factor for me. They’ve always felt secondary to the main goal - learning the content. At work however, I got to work on hard problems, and when I solved them, the people around me were impressed and praised me for my work.

The Anxiety

This praise was not as simple as “great job.” Most frequently it was along the lines of “great job - aren’t you 16??” At the beginning, this praise was strictly a positive influence on me, but as I grew, it haunted me. What if I could no longer achieve that level of praise? What if as I grew older, I lost my “wonder kid magic”?

As I grew, the happiness I got from that praise turned to anxiety. Instead of seeking to do the best job for the company and our clients, I sought to gain the maximum amount of praise possible. Instead of admitting where I had doubts about my solutions, I glossed over my doubts and presented my work as flawless.

The Issues

Perhaps one of my best examples was when I was working on installing kernel bypass routing software on one of our routers. I worked on installing this software all morning and never got it working 100% right. Before I went to lunch, though, I told my boss we were all good and that I hadn’t caused any issues. I came back from lunch to my boss asking “Hey Peyton, any idea why half our sites went down an hour ago?” I had glossed over issues before this, but this is one of the only times it actually bit me in the ass.

I was anxious not only about my performance but also about getting older! So much of my praise at the time was contextualized as “I can’t believe you did X as a Y year old!” My greatest fear was that one day I would wake up and no longer be that wonder child. I feared growing older and just being some random dude, indistinguishable from the millions of other engineers out there.

Growth

As I’ve grown, I’ve realized that fear of age was unfounded. The older I get, the more I realize that being good at what I do is enough. I don’t need to be some special kid; I just need to be good at what I do and to love what I do.

I’ve also realized that sweeping my sketchy implementation details under the rug actually makes me worse at my job! These details compound, and before long, the collective burden of small issues becomes unbearable. Building solid systems that stand the test of time is ultimately more fulfilling, even if I have to admit that I don’t always know what to do.

Some of these realizations came after working on CockroachDB this summer. When working on an open-source codebase, there is nowhere to hide. All of my work was open to the public, and as such, it was scrutinized thoroughly. I frequently thought to myself “eh this isn’t the cleanest way to do things but it’ll work.” Literally every time I thought that, the “clean way” was brought up in code review.

Additionally, the RFC process forced me to think out every detail of how my solution would work before implementing it. This was an extremely rewarding experience as instead of sweeping details under the rug as I might’ve done before, I got the opportunity to receive feedback on all the minutiae of my design from people far more knowledgeable than me.

I look forward to many more years of building robust systems and being honest as I build them.

NOTE: This post was cross-posted from the Penn Labs blog! Check it out over there as well and stay around to see some of the cool work the team does: https://pennlabs.org/blog/spot-postmortem

Over the past month, our infrastructure has had some intermittent issues, so we wanted to drop a quick postmortem describing what the problem was and how we fixed it so we can solve these problems more quickly in the future!

Terminology

Before starting, we should define some terminology:

• Pod - one instance of a running application (ex. Penn Courses Backend)
• Node - a physical machine that our pods run on
• Cluster - a collection of nodes that our pods run on
• Kubernetes - a piece of software built to move pods among nodes such that nodes don’t get overwhelmed (use too much CPU/RAM)

Background

For our cluster, we use AWS Spot Nodes. These are machines that may, at any time, be shut down by AWS for use in other places. In exchange for this, we pay a lower price for these nodes. Nodes going down is not a problem for us since we’re running in Kubernetes which will automatically move our applications to our other nodes if one goes down.

When a spot node goes down, all applications (I’ll use the word pods interchangeably) are moved to the other nodes. A new node is then brought up and the pods (hopefully) rebalance between the available nodes.

Additionally, a few weeks ago, we switched from many smaller nodes to 3 higher-resource nodes. This gave us some cost savings as fewer high-resource nodes are cheaper than many low-resource nodes.

Symptoms

Symptoms first appeared around the time of the Path@Penn launch.

We started getting reports of applications failing with no text in the browser other than Service Unavailable. This allowed us to immediately know that this was an infrastructure-wide issue for a few reasons:

• If multiple products are failing in the same way, then there cannot be a single-application code change that caused this. It must be that some part of the underlying infrastructure is broken.
• Service Unavailable is an error that typically only our load balancer, Traefik will throw. It is almost never sent by actual application code.

Initial Debugging

Upon looking at the cluster with kubectl, we saw that many pods were stuck in the ContainerCreating state. This means that a pod has been assigned to a node, but for some reason, the node cannot start the pod up.

To debug this further, we looked at Datadog, our monitoring system. Datadog showed between 1 and 2 nodes at 100% CPU usage with the third sitting almost idle at ~10% usage.

At this point, the question becomes: If Kubernetes is balancing pods, why are some nodes at such higher resource usage than the others?

Deduction

After further investigation, it looks like the following was happening:

1. A spot node gets scheduled to terminate
2. All its pods are drained to the other two nodes
3. The remaining two nodes don’t have enough available resources to host all our pods and spike to 100% CPU usage
4. 100% CPU usage renders the two nodes unusable, and they stop responding to both Kubernetes and Datadog.
5. Even when the third node comes back online, pods cannot be rebalanced to it because the other two nodes are unreachable due to load.

The underlying cause of this issue is that while 3 nodes is enough for our cluster, 2 is not. When a spot node termination happens, we don’t have enough resources to handle the load, and it creates a cascading failure throughout our infrastructure.

Solution

To solve this issue, we’ve scaled our cluster up to 5 nodes. This should mean that even if we lose a node due to spot lifecycle, we’ve got plenty of compute to handle things.

Why did this just happen now?

This is the main question we’ve been asking ourselves. Intuitively, it seems like we should have run into these issues as soon as we switched to only using 3 nodes.

However, these issues are not just a symptom of having 3 nodes. They’re a symptom of having 3 nodes while under high load. The Path@Penn launch was the first time we had high load on our products since moving to 3 nodes. Therefore, while this problem existed all along, we didn’t notice it until our systems were under stress.

Next Steps

To solve these issues more quickly in the future, we should have better alerting and monitoring around application failures. Specifically:

• Alerting when we have a high number of 503 Service Unavailable responses. These almost always indicate an infrastructure-level error.
• Alerting when we have high node CPU usage. While pod CPU usage hasn’t been great signal for us (pods occasionally spike), high node usage almost always indicates a problem.

Reach Out

If any of this looks like it’s up your alley or you wanna learn more about our mission, be sure to email us at contact@pennlabs.org or apply to be a part of Labs! We’ve got some fantastic teams working on interesting problems with a direct impact on campus.

Intro

The key idea of these thighs is that we want to end up with bulk pre-cooked chicken that can enjoyed by itself or as an ingredient to other meals. For this, we have a few desirable qualities:

• Tenderness: If we want thighs that can be re-cooked, they must be relatively tender after their first cook.

  We’ll achieve this by using thighs (high fat content) and by using an instant pot (locks in moisture).

• Base flavor: We want the thighs to have a flavor that will make them appealing to eat alone but will not overpower dishes.

  Our garlic soy sauce will provide a subtle but helpful flavor boost to cut some of the “chicken taste” out of the thighs.

• Manageability: I only want to cook chicken 1-2 times a week. I should be able to make 8+ chicken thighs at once.

  The instant pot is a one-size-fits-all tool for doing exactly this.

• Healthy: If I’m going to eat these almost every day, they better not kill me.

  While chicken thighs aren’t as lowfat as breasts, they’re still pretty good for you, and we aren’t adding anything too egregious to them. If you’re watching calories or want to pump your protein consumption, you can switch to breasts at the expense of initial tenderness and recookability.

Ingredients & Recipe

Prepping

1. Mince garlic and add all marinade ingredients to mixing bowl. The measurements on this are very inexact, so taste and test different combinations yourself! Just taste and adjust until you like it.

   Steps below are optional but highly recommended

2. Dry thighs with a paper towel and season all sides with salt and pepper

3. Heat large pan on high heat with a neutral oil like canola until the oil is almost smoking.

   Cast-iron or carbon steel are best here for their heat retention, but nonstick is perfectly fine.

4. Cook thighs until they develop a hard sear.

   The texture from this sear is what will give the thighs texture, so don’t skimp on this if you want to eat these solo!

5. Remove thighs, taking care to not remove the sear.

   If you’re in a nonstick, the thighs will come off with no problem, but if you’re in a cast iron or nonstick, use your heavy-duty tools to make sure the sear comes off with the chicken. The worst feeling is building a great sear and it getting stuck to the pan.

Once your thighs are seared, they should look something like this (peep the marinade in the back):

Veggie interlude

If you’re looking to cook up some veggies for the week, the fond left in your searing pan will make for some amazing sautéed vegetables. Just cut up some veggies, throw them in with bit of olive oil, and wait for delicious veggies that go with any meal. I’ll usually do something like the following:

1. Add olive oil with diced onions, sliced mushrooms, sliced celery
2. Once onions are translucent, add sliced carrots and diced bell pepper
3. Once carrots have softened up, add sliced garlic, red pepper flakes, and salt and pepper to taste

Wait until everything’s combined and store for later use! I love this combination standalone or along with proteins like these chicken thighs.

Finalizing

Now that we’ve got thighs and marinade, add both to the instant pot and cook on high pressure for 3 minutes. If you didn’t sear the thighs, do 5 minutes instead.

I’ll typically put a thigh on rice, spoon some of the marinade from the pot over top and garnish with green onions. Here’s what it looks like! These are actually breasts, but the thighs will look similar.

If you cooked up vegetables earlier, this is a great time to reuse them. Add those veggies on top of the chicken + rice for a full meal.

Storing

Separate the thighs and marinade, and store them however you desire in your fridge. The thighs will keep for a little over a week, and the marinade will keep for a few weeks.

To reheat this meal, just throw a thigh + rice + a spoonful of marinade in a bowl. Pop it in the microwave for 1.5 minutes, and you’ve got a hot meal ready to go.

Other meals

For a twist on an omelette, try an Oyakomelette! It adds a little body to an omelette to turn it from a light breakfast into a hearty lunch or dinner.

If you’re looking for a grilled chicken salad, just slice the thighs, sear them off in a pan, and follow my recipe (or don’t; it’s just a salad).

Finally, if you’re looking for a greasy late-night meal, make yourself a quesadilla. Fry up diced chicken in olive oil, combine with cheese and carmelized onions, pack the filling into a tortilla, sear in a pan, and you’re good to go. I have made more quesadillas with this stuff than I’d like to count, and they never disappoint.

Don’t let these ideas limit you, though! You can use these thighs wherever you would use chicken. I’ve made fried rice, tacos, sandwiches, and more with this stuff.

Conclusion

These thighs have served me very well, and I’m sure they’ll serve you as well. Just make sure to stay creative with it! These being pre-cooked gives you lots of extra creative time to explore whatever dishes you desire.

NOTE: This post was cross-posted from the Penn Labs blog! Check it out over there as well and stay around to see some of the cool work the team does: https://pennlabs.org/blog/building-icarus

As of recent, we’ve moved our infrastructure away from Dokku and onto Kubernetes. Especially for an organization with high turnover (nature of being a club), you have to carefully consider the organizational impacts of moving to a technology as complex as Kubernetes. This post should serve as a bit of insight into how we dealt with that challenge and built technology to solve it.

Background

Typically when organizations decide to adopt Kubernetes, their organizational structure takes one of two patterns (or a combination of both):

Pattern 1: Embedded SREs

In this pattern, each team has one or more Site Reliability Engineers embedded in each team. These SREs are responsible for managing the deployment infrastructure and general architecture for the product. Since this model gives teams Kubernetes experts right in the teams, these teams are responsible for managing their own Kubernetes manifests and general deployment strategy.

This system works great for teams with complicated application architectures whose applications might require special features like persistent volumes, cross-datacenter availability, complex scheduling requirements, etc. Think teams running Cassandra clusters, Solr clusters, or database services. In this model, SREs on each team can fit the Kubernetes configs to each team’s needs. The clear drawback, however, is that your number of SREs required scales linearly with your number of teams.

Because of the (relative) simplicity of Penn Labs’ applications and only having 2-3 SREs to service 5 teams, we determined that this pattern was not an option.

Pattern 2: Platform Team

In this pattern, there is a single team (or organization) that is responsible for building a high-level platform that other teams can use to deploy their applications. Building a platform involves significant up-front engineering work and a strong team to build the automation and interfaces required in order to create a frictionless deployment experience.

This approach typically works best for applications with simple architectures. If you can restrict your applications to following similar architectures, you can build a platform that takes advantage of those shared architecture, abstracting away the common configuration required by your applications.

In Labs, all of our web applications follow similar architectures: a single frontend (usually React) and a monolithic backend (usually Django). Since we have a common, simple architecture and we don’t have the personell to man every team with an SRE, we decided to go with the platform approach. This approach fit nicely into our organizational structure since we already have a platform team that manages our authentication system.

Platform Abstractions

There are quite a few tools that fit together to make this system work (Hashicorp Vault, Grafana, etc.), but for the purposes of this post, I’m going to focus solely on the tool that developers actually use: Icarus.

Kubernetes configs are hard and complicated. Kubernetes aims to be able to support any containerized application architecture possible, so it exposes every knob you could possibly want to turn. In simple setups, as you can see below, this leads to nothing other than repetition and confusion:

---
# Source: icarus/templates/services.yaml
apiVersion: v1
kind: Service
metadata:
  name: "testsite-serve"
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 80
  selector:
    name: "testsite-serve"
---
# Source: icarus/templates/deployments.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: "testsite-serve"
  namespace: default
  labels:
    name: "testsite-serve"
spec:
  replicas: 1
  selector:
    matchLabels:
      name: "testsite-serve"
  template:
    metadata:
      labels:
        name: "testsite-serve"
    spec:
      containers:
        - name: "worker"
          image: "pennlabs/website:latest"
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
          envFrom:
            - secretRef:
                name: test-secret
---
# Source: icarus/templates/ingresses.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: testsite-serve
  namespace: default
spec:
  rules:
    - host: "pennlabs.org"
      http:
        paths:
          - path: "/"
            backend:
              serviceName: testsite-serve
              servicePort: 80
  tls:
    - hosts:
        - "pennlabs.org"
      secretName: pennlabs-org-tls
---
# Source: icarus/templates/certificates.yaml
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: pennlabs-org
  annotations:
    "helm.sh/resource-policy": keep
spec:
  secretName: pennlabs-org-tls
  dnsNames:
  - "pennlabs.org"
  - "*.pennlabs.org"
  issuerRef:
    name: wildcard-letsencrypt-prod
    kind: ClusterIssuer
    group: cert-manager.io

As mentioned earlier, in our applications we can make the following assumptions to make our lives easier:

• There will be exactly one container per application
• That application, if exposed to the world, will speak HTTP and want to be secured with HTTPS
• That application will use secrets synced into Kubernetes by our Vault secret sync job

With just these three assumptions, we can radically simplify our required configuration to the following for this example:

deploy_version: 0.1.15

applications:
  - name: serve
    image: pennlabs/website
    secret: test-secret
    ingress:
      hosts:
        - host: pennlabs.org
          paths: ['/']

Let’s dive into how this is possible.

Helm

At the root of the transformation from Icarus file to full Kubernetes manifest is a tool called Helm. Helm describes itself as a “package manager for Kubernetes”, but for the purposes of this article, think of it as a templating tool that happens to be good at deploying to Kubernetes.

Helm operates off the Go template package. This package, much like Python’s Jinja or Ruby’s ERB, lets you define a template file, specifying where to put information the user provides. Take the example of our service template. This template defines a basic service with parameters:

apiVersion: v1
kind: Service
metadata:
  name: {{ $app_id | quote }}
spec:
  type: {{ .svc_type }}
  ports:
    - port: {{ .port }}
      targetPort: {{ .port }}
  selector:
    name: {{ $app_id | quote }}

Helm lets you define default values, so in Icarus, we set .svc_type to be ClusterIP by default and .port to be 80 by default. We also set $app_id to be <repository_name>-<application_name>. Applying these rules, we get the resulting configuration that we see above:

# Source: icarus/templates/services.yaml
apiVersion: v1
kind: Service
metadata:
  name: "testsite-serve"
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 80
  selector:
    name: "testsite-serve"

Icarus works simply by applying substitutions like this to create the final Kubernetes manifests, and then we use the DigitalOcean API to automate the deployment of these manifests to our actual cluster. I’m glossing over a lot of detail here, but feel free to check out our deploy orb for the specifics.

End Result

There’s been a lot of talk here about organizational philosophy and YAML templating, but let’s circle back to what this system actually gives us.

When one of our developers wants to create a new application, they can, completely independently, create a Git repo, add in a small CI config file, create their Icarus file, and add their proper secrets into Vault. Once that’s done, they can Git push up, and their application will be published to the domain of their choice in just a few minutes. If you ask me, that’s pretty cool.

Reach Out

If any of this looks like it’s up your alley or you wanna learn more about our mission, be sure to contact us at contact@pennlabs.org or apply to be a part of Labs! We’ve got some fantastic teams working on interesting problems with a direct impact on campus.

NOTE: FaunaDB the company died, so this approach no longer is in use. I work for Modal now, so I re-implemented the backend using Modal. Sadly I lost all my data, so please give me kudos if you like the articles!

When diving into WebRTC recently, I ran into this great article on the limitations of WebRTC, particularly related to its unreliability in doing NAT traversal. At the end of the article, I saw that it had this neat “kudos” button that when I hovered over it, upped the kudos count for the post. It turns out that kudos are a feature of the Svtle platform.

While I’m not interested in moving my site over to Svtle, I wanted that button, so I decided to make it. The button is comprised of 3 main parts:

1. CSS animation for the expanding/contracting and the color change
2. Netlify serverless function for tracking kudos count
3. Client-side JavaScript glue to hook up the animation to the serverless function

CSS

When I started coding (2013ish), CSS quickly became my mortal enemy. Nothing made sense to me, and the rules felt very inconsistent. So much so, in fact, that I quickly gave up on writing my own CSS and just used Bootstrap (now Bulma) for every project moving forward. Naturally, this had me dreading working with CSS for this project.

Once I got past some initial bumps, though, I found that it turns out that 7 years of CSS improvement and Sass really have made CSS much friendlier. The way the final animation works is roughly:

1. Start the dot off as a small size, $dotSize
2. On hover, fill out to the size of the ring, $filledSize. Animate just using the transition: 1s attribute.
3. After 1s of hovering, run some JavaScript (timing method described below) to change the ring and circle classes to filled, which gives them a green color and the circle a slightly smaller size than the ring. Transition time 0.2s felt right for this.

These steps created a nice pop out and back in that I liked, so now I could move on to writing my server-side functions.

Netlify Functions

Before this, I had been hosting my site on Github Pages. Pages worked great for me since I had no dynamic content, but this button threw a wrench in that. Moving my site over was largely uninteresting (to Netlify’s credit), so I’ll just focus on the functions here.

For storing like counts, I use FaunaDB, a serverless database with a generous free tier. Fauna is super easy to set up with Netlify. I’ll note how to bootstrap at the end of this article. In my Fauna collection, I store documents that contain the title of the article with a list of which client IDs have given it kudos. Example:

{
  "title": "automating-k3s-deployment-on-proxmox",
  "users": [
    "bb344507-a78f-4c6d-a71d-362056a8e6d8",
    "e894ef53-7a4a-4b6b-b081-e28adba82a04",
    "755408c9-f671-4ff0-b577-30c8075f3936",
    "768839bf-39b6-432e-96f2-38d5e1aa575a",
    "8ce3057f-22bb-47d7-9c29-c2dad1205fa0"
  ]
}

Once I hooked up Fauna, I simply created two serverless functions:

• get-kudos.js - This returns a JSON response with the number of kudos and whether or not the current user has given the post kudos. Example:

  {
    "numClicked": 5,
    "userClicked": true
  }
  

• add-kudos.js - This adds the current user to the list of users who have given kudos to a post.

Client-side JavaScript

To keep track of which users had liked which post, I generated UUIDs according to the UUIDv4 spec and stored them in local storage. At page load, I simply make an AJAX request to my get-kudos function to check if the logged in user has given the page kudos. If they have, I immediately add the filled CSS class to the ring and circle.

For keeping track of whether or not the user has hovered in the circle for 1 second, I used the mouseenter and mouseleave events. I have a variable lastTime which starts of as 0. When you enter the circle, lastTime gets set to the current UNIX time and a 1 second delayed function gets dispatched. When you leave the circle, lastTime gets set to 0.

After one second of entering the circle, the aforementioned delayed function is run which checks if lastTime is 1 second or more before the current time. If it is, the add-kudos request is sent off, and the ring and circle becomes filled!

This completes the code, creating the kudos effect:

On Your Site

This button is pretty self-contained, so it shouldn’t be too hard to add to your own site if you’re hosted on Netlify. Here are some rough guidelines:

1. Copy over my functions directory
2. Hook up FaunaDB and rename your database to personal-site

   $ netlify addons:create fauna
   $ netlify addons:auth fauna
   # now go in and change your database to personal-site
   $ cd functions
   $ npm install
   $ netlify dev:exec node createdb.js
   

3. Copy in my kudos JavaScript and kudos SCSS into your project and import them into your page
4. Add a kudos container to your post template
5. Make sure your netlify.toml is set to npm install for your functions.

Conclusion

After not doing any web development for so long, it was good to stretch those muscles a bit. With Hugo and Netlify, this process required basically no infrastructure work on my part, which was exactly what I was looking for.

Hopefully this was helpful, and if you enjoyed this article, feel free to give me some kudos :)

All of this changed when I found out about k3s. k3s allows you to run a full Kubernetes-compliant cluster without “a PhD in k8s clusterology”. Its main innovation is packaging all the normal Kubernetes components into a single binary and refactoring some authentication to make it much easier for nodes to join the cluster.

Combined with some Ansible and Terraform knowledge, I decided to get cooking on some automation to make spinning up a new cluster a simple command.

Prerequisites

There are a few things I’m not going to cover in this guide that you need to get started:

• Install Terraform
  • Just download Terraform from the downloads page and drop it somewhere in your $PATH
• Install the Terraform Proxmox provider
  • Use the go install command found in the README
  • Drop the binaries into ~/.terraform.d/plugins/
• Have a Proxmox host
• Install Ansible

Building a cloud-init Ubuntu template

In order for Terraform to work smoothly with new VMs, we need to make a template VM that has cloud-init on it. Cloud-init adds some packages to the VM that makes automatic provisioning possible. Thankfully, Proxmox has pretty good support for it.

SSH into your Proxmox host, and enter these commmands to create an Ubuntu VM with cloud-init:

$ wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
# Export whatever storage pool you want to use to hold your VM images. Mine just happens to be named hermes_data
$ export STORAGE_POOL=hermes_data
$ qm create 8000 --memory 2048 --net0 virtio,bridge=vmbr0
$ qm importdisk 8000 bionic-server-cloudimg-amd64.img $STORAGE_POOL
Formatting '/data/images/8000/vm-8000-disk-0.raw', fmt=raw size=2361393152
    (100.00/100%)
$ qm set 8000 --scsihw virtio-scsi-pci --scsi0 $STORAGE_POOL:8000/vm-8000-disk-0.raw
update VM 8000: -scsi0 hermes_data:8000/vm-8000-disk-0.raw -scsihw virtio-scsi-pci
$ qm set 8000 --name ubuntu-ci
update VM 8000: -name ubuntu-ci

Now we have a VM with all the appropriate cloud-init packages installed. Now we just have to attach the hardware that cloud-init requires, and we can template the VM.

$ qm set 8000 --ide2 $STORAGE_POOL:cloudinit
update VM 8000: -ide2 hermes_data:cloudinit
Formatting '/data/images/8000/vm-8000-cloudinit.qcow2', fmt=qcow2 size=4194304 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16
$ qm set 8000 --boot c --bootdisk scsi0
update VM 8000: -boot c -bootdisk scsi0
$ qm set 8000 --serial0 socket --vga serial0
update VM 8000: -serial0 socket -vga serial0
$ qm template 8000

There we go! We now have a template VM that we can build our k3s nodes off of.

Deploying the VMs

Now, on whatever machine you have Ansible and Terraform installed on, clone down my proxmox-k3s repo:

$ git clone https://github.com/Pwpon500/proxmox-k3s
Cloning into 'proxmox-k3s'...
remote: Enumerating objects: 64, done.
remote: Counting objects: 100% (64/64), done.
remote: Compressing objects: 100% (38/38), done.
remote: Total 64 (delta 3), reused 64 (delta 3), pack-reused 0
Unpacking objects: 100% (64/64), done.
$ cd proxmox-k3s/proxmox-tf/prod

Now, edit the main.tf file to reflect the IPs of all your nodes as well as what SSH keys you want to use. Next, export the appropriate variables so that the Proxmox Terraform provider can connect to the Proxmox API:

$ export PM_API_URL="https://<node_ip>:8006/api2/json"<Paste>
$ export PM_USER=root@pam
$ export PM_PASS=<your_pass_here>

If you don’t set any of these, you’ll be prompted for them whenever you do a terraform plan or a terraform apply.

Without any further ado, let’s create some VMs!

$ terraform init
$ terraform plan
$ terraform apply

These commands will have a lot of output, but I don’t want to paste it all here. It should be clear if things have worked.

Wait a few minutes for the VMs to finish doing their cloud-init inital configuration, and continue to the next step.

Applying k3s Configs

Now, go into the ansible-roles directory. All you need to edit here is the inventory.toml file. Change the IPs to whatever yours are, and add as many hosts as you created. The only important thing to remember is to have exactly 1 master node and to make the rest workers.

Once you’ve entered in the appropriate IPs, apply your playbook:

$ ansible-playbook -i inventory.toml playbook.yml -u ubuntu

This command will take a few minutes to complete, but if it finishes successfully, you have a fully working k3s cluster!

Testing Out the Cluster

To test out the cluster, we’re going to go into the master node and use the in-built k3s kubectl:

$ ssh ubuntu@172.30.100.80
ubuntu@k3s-node-0:~$ sudo k3s kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
k3s-node-0   Ready    master   2d    v1.14.4-k3s.1
k3s-node-1   Ready    worker   2d    v1.14.4-k3s.1
k3s-node-2   Ready    worker   2d    v1.14.4-k3s.1
k3s-node-3   Ready    worker   2d    v1.14.4-k3s.1

Looks pretty good. We have all the nodes properly joined up. Let’s start a pod just to test things out:

$ sudo k3s kubectl run lab-nginx --image=nginx --port=80
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/lab-nginx created
$ sudo k3s kubectl expose deployment lab-nginx --type=NodePort
service/lab-nginx exposed
$ sudo k3s kubectl port-forward svc/lab-nginx 8080:80 &
$ curl localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Woohoo! Looks to me like this thing’s working. If you want to get out the kubeconfig to run kubectl from your local machine, it’s in /etc/rancher/k3s/k3s.yaml.

Now if you don’t want it anymore, you can do a simple terraform destroy in the proxmox-tf directory, and you’re back to where you started.

Conclusion

Obviously, what we’ve done here is exciting because we get a Kubernetes cluster with a pretty simple process. It’s also crazy useful that we can now spin up entire VM clusters with just a simple command. Before, I had to do this through the Proxmox Web UI, but not anymore.

I hope all this was helpful! I had a ton of fun with this, and there’s more of this kind of material coming in the future.

To get to MVP, I had to implement the following functionality:

• Logging
• Config file parsing

Combining all the previous updates, we get the following feature set for the MVP:

• Packet listening
• Packet forwarding over UDP
• Direct reply from backends, allowing the load balancer to only have to handle incoming traffic
• Dynamic backend registration and deregistration
• Backend commands and health checks
• Logging
• Config file parsing

I’ll briefly cover the improvements so far and then give a short demo of Caplance working.

Logging

For logging, I decided to use the Logrus package. It’s frequently used by other packages, and it was super easy to switch to. I had already implemented logging with the log package from the go stdlib, so by importing log github.com/sirupsen/logrus in its place, I got instant compatibility. From there, I changed the log levels as appropriate. I’m using the following log levels:

• Debug
• Info
• Error
• Fatal
• Panic

I’d like to eventually get away from panic and just call a graceful stop function with an error code, but that’s nitpicking pretty hard considering where Caplance is right now. Here’s a little demo of what the logging looks like:

Load balancer:

Backend:

Config File Parsing

For parsing configuration files, I used Viper. Viper allows me to use JSON, HCL, TOML, or YAML to write my config files so long as they follow my defined structure. The structure for configuration files follows this struct:

type config struct {
    Client struct {
        ConnectIP string
        DataIP    string
        Name      string
    }
    Server struct {
        MngIP           string
        BackendCapacity int
    }
    VIP  string
    Test bool

    HealthRate   int
    ReadTimeout  int
    WriteTimeout int

    Sockaddr string
}

The following defaults are also set:

viper.SetDefault("Test", false)
viper.SetDefault("HealthRate", 20)
viper.SetDefault("RegisterTimeout", 10)
viper.SetDefault("ReadTimeout", 30)
viper.SetDefault("WriteTimeout", 10)
viper.SetDefault("Sockaddr", "/var/run/caplance.sock")

In YAML, an example config file looks like this:

vip: 10.0.0.50

client:
  dataIP: 10.0.0.2
  name: backend-1

server:
  mngIP: 10.0.0.1
  backendCapacity: 53

This config file accepts all given defaults, only filling in the required fields. I’ll write more detailed documentation on the config file format soon, but for now, this will do. What’s really cool about Viper is that I can just tell it where to go looking for config files, give it a struct to put data into, and it does everything else for me:

func readConfig() {
    if configLocation != "" {
        viper.SetConfigFile(configLocation)
    }

    err := viper.ReadInConfig()
    if err != nil {
        log.Fatal("Failed to read in config: " + err.Error())
    }

    conf = &config{}
    err = viper.Unmarshal(conf)
    if err != nil {
        log.Fatal("Failed to unmarshal config into struct: " + err.Error())
    }
}

Here, I’m just telling Viper that if a specific config file location exists, use it. Then, I read in the config and unmarshal it into the conf struct for later use. Pretty cool stuff.

Demo

Now for the fun part! I’ve prepared a brief demo just to show that Caplance works as we expect it to.

In this demo, I’ll have the following hosts:

The virtual IP for the cluster that the client will be calling to is 10.0.0.50. Let’s start things up! I’m only going to show output from h1 and h2 right now, but h2 and h3 are showing the same as h2.

h1 $ ../caplance server -f confs/simple_1.yaml
INFO[0000] Reading in config file
INFO[0000] Starting load balancer

h2 $ ../caplance client -f confs/simple_1.yaml
INFO[0000] Reading in config file
INFO[0000] Starting client
2019/07/07 12:42:39 rpc.Register: method "Start" has 2 input parameters; needs exactly three

Now, I’m going to start nginx on the backends and see what happens when I curl from h1!

h2 $ nginx -p "" -c confs/nginx_1.conf

h5 $ ./repeat_curl.sh 10
Welcome to Onion Backend 2!
Welcome to Onion Backend 1!
Welcome to Onion Backend 2!
Welcome to Onion Backend 2!
Welcome to Onion Backend 1!
Welcome to Onion Backend 1!
Welcome to Onion Backend 2!
Welcome to Onion Backend 2!
Welcome to Onion Backend 3!
Welcome to Onion Backend 2!

Looks like it’s working! In this limited example, it seems that we’re probing 3 much less than the others, but that’s just random chance. If we run this over many more iterations and count occurences out, we see things level out:

h5 $ ./repeat_curl.sh 1000 | sort | uniq -c
    330 Welcome to Onion Backend 1!
    336 Welcome to Onion Backend 2!
    333 Welcome to Onion Backend 3!

Looks like things are working out! If you want to run this demo yourself, all the required files are right in the repo.

Conclusion

This was by far the most difficult project I’ve ever worked on, so having it done feels pretty surreal. Before signing off, I want to thank Davis, Armaan, and Ben for listening to me rant and rant about Caplance. Having an ear to talk to is unimaginably helpful when working on a project like this.

Despite some of the language I’ve been using, this is most certainly not the last work I’ll be doing on Caplance. I still have much to learn and features I want to implement. Until then, however, thanks for reading, and I’ll see you next time.

• Backend registration
• The following controls from the backends:
  • PAUSE
  • DEREGISTER
  • RESUME
  • HEALTH
• caplancectl to tell a running client process to issue one of those commands
• Graceful stop for backends
• Packet listening on NFQUEUE
• Refactor project structure

If you’ve been keeping track, you’ll notice that this puts us very close to the Caplance MVP! All that’s left is config file parsing and some housekeeping.

Here are some of the more technically interesting parts of this revision of Caplance:

NFQUEUE

If you’ve got some time and an appetite for some very interesting debugging, I highly recommend taking a look at my post about NFQUEUE. If you don’t, here’s the TL;DR:

I switched how listening works in Caplance again. Now, I’m using NFQUEUE, an iptables option to hand off packets from an iptables rule to a program in userspace before any packet processing is done.

Backend Communication

One of the things I wanted to learn from this project was how to write a system for keeping track of state between two hosts. I looked (extensively) into using RPC for this, but it just didn’t seem like the right move. RPC is designed to make “remote process calls”, but I don’t really need that. I need a way to keep track of state. I could’ve mangled RPC to do this, but I think just writing it myself turned out to be more elegant.

The first step in creating this was creating the interface. I created a simple interface called Communicator to write data, read data, and close the underlying connection:

// Communicator is the connection manager for a backend
type Communicator interface {
    ReadLine() (string, error)
    WriteLine(data string) error
    Close() error
}

To actually implement this, I created a TCPCommunicator struct and filled in the methods.

// TCPCommunicator is an implementation of BackendCommunicator over TCP
type TCPCommunicator struct {
    reader       *bufio.Reader
    writer       *bufio.Writer
    conn         net.Conn
    readTimeout  time.Duration
    writeTimeout time.Duration
}

This struct has a reader and writer to get data from the underling connection. It also has a readTimeout and writeTimeout. While the writeTimeout is fairly mundane, the readTimeout turns out to be surprisingly useful.

In order to deregister a backend after a certain period of inactivity, I can just use the SetReadDeadline(t time.Time) error function with the readTimeout. Then, if the read errors out, and the error is a timeout, I know that the inactivity window has closed. Baking this logic into the TCPCommunicator, I now have a powerful tool that I can use on both the backend and load balancer side!

I also implemented the following commands that the client can send to the server:

• PAUSE - client requesting to stay registered but not have packets forwarded to it
• DEREGISTER - client requesting to deregister
• RESUME - client requesting to have packets forwarded after a PAUSE
• HEALTH - message sent every few seconds to stop server from deregistering the client due to inactivity

The way these are implemented isn’t particularly interesting. I’m just using a bufio reader to read data off the connection and parsing it with simple string tools.

Caplancectl

This isn’t the most complex part of Caplance by any means, but I was blown away by how simple the net/rpc package made writing caplancectl, so I wanted to share.

Caplancectl connects to Caplance using a unix socket that Caplance creates at /var/run/caplance.sock. Caplance listens just as it would with a TCP listener:

c.unixSock, err = net.Listen("unix", SOCKADDR)

Now that I’ve got that socket, I can hand it off to the rpc package, and the rest of the networking is done for me!

rpc.Register(c) // registering the client to receive requests over rpc
rpc.HandleHTTP()
http.Serve(c.unixSock, nil)

Now that I’ve got the RPC listening, all I have to do is define some methods that the RPC can execute. The RPC registers any methods from the registered object that take two arguments for which the latter is a pointer and returns an error. More formally, the methods must have this signature:

func (t *T) MethodName(argType T1, replyType *T2) error

For example, my pause function looks like this:

// Pause command from caplancectl
func (c *Client) Pause(req *string, reply *string) error {
    err := c.pause()
    if err == nil {
        *reply = "Pause request sent"
    } else {
        *reply = "Pause request encountered an error: " + err.Error()
    }
    return nil
}

That’s all that needs to be done from the server side! If you think that’s easy, the client side is even easier.

On the client side, I can just use the RPC package to dial and then call whatever method I want. For example, if I wanted to issue a pause request, I would only need this code:

client, err := rpc.DialHTTP("unix", SOCKADDR)
if err != nil {
    log.Fatal(err)
}

var reply string
err = client.Call("Client.Pause", "", &reply)
if err != nil {
    log.Fatal(err)
}
fmt.Println(reply)

Pretty cool, right?

Future Work

I’m pretty close to MVP. To get there, I have to implement the following:

• Config file parsing
• Non-stdout logging

I’ve got some more features I’d like to implement, but we’re gonna focus on MVP for now. These shouldn’t be too hard to get implemented, so keep you eyes peeled for update 3!

Types of Message

When establishing a TCP connection, there are 4 possible packet types you could see:

• SYN (S in tcpdump) - Always the first message sent. It is the message from the client to the server effectively saying “I would like to connect.”
• SYN-ACK (S. in tcpdump) - The server’s reply to a SYN if it wishes to accept the connection.
• ACK (A or . in tcpdump) - The client’s reply to a SYN-ACK, fully establishing the TCP connection.
• RESET (R in tcpdump) - The server’s response to a SYN if it wishes to refuse the connection.

In a successful connection, the order will be SYN, SYN-ACK, ACK. In an unsuccessful connection, the order will be SYN, RESET.

Operating Systems and TCP

When a program wants to use TCP, it asks the OS to “bind” to a TCP port. Then, when packets come into that TCP port, the OS hands off the packets to the program. The program can then reply to those packets however it wants (accepting or rejecting the incoming connections).

If a program is not bound to a port and a SYN is sent to that port, the OS immediately replies to the SYN with a RESET.

In the case of Caplance, the load balancer isn’t listening on a specific TCP port. Instead, it listens for any and all TCP packets. As a result, no bindings are created by Caplance.

The Bug

Ok, now on to the fun part. I had finally gotten to the point in Caplance where I could send a request to the load balancer’s IP and have a backend respond to the request, or so I thought. For the purposes of this demo, h1 is the load balancer, h2 is the backend, and h3 is the client. 10.0.0.50 is the virtual IP that the load balancer is serving. Let’s start up a netcat TCP server on h2 and try to send a request from h3:

h2 $ nc -l 10.0.0.50 8080

h3 $ telnet 10.0.0.50 8080
Trying 10.0.0.50...
telnet: Unable to connect to remote host: Connection refused

Well, what can you expect really? Things never work on the first try. This time, let’s do the exact same thing but use tcpdump to see what’s happening on h2.

h2 $ tcpdump -i any host 10.0.0.50 and tcp -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
20:57:32.941669 IP 10.0.0.3.36040 > 10.0.0.50.8080: Flags [S], seq 252176171, win 29200, options [mss 1460,sackOK,TS val 2262321703 ecr 0,nop,wscale 9], length 0
20:57:32.941701 IP 10.0.0.50.8080 > 10.0.0.3.36040: Flags [S.], seq 492629165, ack 252176172, win 28960, options [mss 1460,sackOK,TS val 4192526147 ecr 2262321703,nop,wscale 9], length 0

Here, tcpdump is showing us that the backend is seeing 2 packets. One is a SYN from the client to the server, and one is a SYN-ACK from the server to the client. If you remember, these are exactly the messages we expect … minus an ACK from the client.

At this point, the logical conclusion is that packets are able to flow from client to server, but the ones going server to client aren’t making it. Just for sanity, let’s see what the tcpdump output looks like for the client.

h3 $ tcpdump -i any host 10.0.0.3 and tcp -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
21:11:45.055143 IP 10.0.0.3.37388 > 10.0.0.50.8080: Flags [S], seq 3009015791, win 29200, options [mss 1460,sackOK,TS val 2263173829 ecr 0,nop,wscale 9], length 0
21:11:45.059240 IP 10.0.0.50.8080 > 10.0.0.3.37388: Flags [R.], seq 0, ack 3009015792, win 0, length 0
21:11:45.059248 IP 10.0.0.50.8080 > 10.0.0.3.37388: Flags [S.], seq 2629426318, ack 3009015792, win 28960, options [mss 1460,sackOK,TS val 4193378273 ecr 2263173829,nop,wscale 9], length 0

… What?

At this point, I’m baffled. Let’s recap what’s going on here. The server is seeing a SYN and SYN-ACK. However, the client is seeing a SYN, SYN-ACK, and RESET. How is it possible that the client could be sent both a SYN-ACK and a RESET from the same source??

If you’re interested in this stuff, I invite you to sit for a minute and think about why this might be happening. This is one of the more interesting bugs I’ve run into in a while. The presence of a SYN-ACK and RESET concurrently really interested me.

The Bug Revealed

Remember when I said that Caplance isn’t binding to a specific port? That turns out to be the key to the puzzle here. Since Caplance listens for all TCP connections instead of a single port, it never sends a “bind” request to the OS. This means that the OS will still respond to all incoming TCP connections with a RESET.

To clarify, let’s think about the connection establishment journey:

1. SYN leaves the client
2. SYN hits the load balancer
   1. Because the load balancer isn’t actually bound to a TCP port, the OS replies to the client with a RESET
   2. The load balancer forwards the SYN to the backend
3. SYN hits the backend
   1. Backend replies to the client with SYN-ACK
4. Client receives RESET
5. Client receives SYN-ACK

As is typical, the actual bug is pretty simple in hindsight, but it can be hard to find in the moment.

Enter NFQUEUE

So now we’re in a conundrum: we want to stop the OS from seeing packets destined for the VIP, but we also want the OS to give packets destined to the VIP to our program so it can forward then to the appropriate backend. This is where NFQUEUE comes into play.

NFQUEUE is a iptables filter rule that queues up incoming packets onto a queue, waiting to be processed by some program in userspace. Let’s see an example rule.

The following rules will drop all tcp and udp packets destined for the IP 10.0.0.50:

$ iptables -A INPUT -d 10.0.0.50/32 -p tcp -j DROP
$ iptables -A INPUT -d 10.0.0.50/32 -p udp -j DROP

What if instead, we want to put all tcp and udp destined for 10.0.0.50 onto a NFQUEUE with the id 0? We can simply modify our previous rule as follows:

$ iptables -A INPUT -d 10.0.0.50/32 -p tcp -j NFQUEUE --queue-num 0
$ iptables -A INPUT -d 10.0.0.50/32 -p udp -j NFQUEUE --queue-num 0

Now, we have a single source (queue 0) that we can consume all our packets off of! There are bindings to do this in many languages, but I’m using Go for Caplance, so here’s a snippet of how this looks in Go.

b.nfq, err = netfilter.NewNFQueue(0, 100, netfilter.NF_DEFAULT_PACKET_SIZE)
if err != nil {
    log.Panicln(err)
}
packetChan := b.nfq.GetPackets()
stopped := false
for !stopped {
    select {
    case packet := <-packetChan:
        b.packets <- packet.Packet.Data()
        packet.SetVerdict(netfilter.NF_DROP)
    case sig := <-b.stopChan:
        b.stopChan <- sig
        stopped = true
    }
}

Don’t worry if you don’t understand the Go specifics of this code. The important thing is that we’re creating a NFQUEUE receiver called b.nfq. Then, we create a channel called packetChan off of which we can consume whatever packets come in to queue 0.

Running It

Let’s make sure this works! If it does, we should see whatever we type into h3 popping up in the TCP server on h2.

h3 $ telnet 10.0.0.50 8080
Trying 10.0.0.50...
Connected to 10.0.0.50.
Escape character is '^]'.
Is this working?
It is!!!!!
 ^]
telnet> close
Connection closed.

h2 $ nc -l 10.0.0.50 8080
Is this working?
It is!!!!

We did it! NFQUEUE was the answer to our problems.

Conclusion

When I found out about NFQUEUE, I was dumbstruck. It solved so many of my problems - creating a single source from which I could consume packets, stopping the load balancer from replying to the wrong packets, and overall just cleaning up my code. If you’ve got a project like this, I highly recommend looking at NFQUEUE as a potential option.

1. Ubiquity

   I use Neovim for writing pretty much all my other code (except Java), so it was weird to use VSCode for this one purpose.

2. Terminal Integration

   I hate the VS Code terminal. It doesn’t have the character support I want, and it overrides a bunch of keys (for example, Ctrl+f for fish autocomplete). I’m also in the terminal already when I’m doing Go development, so being able to just type vim and instantly be in my editor is a blessing.

Before I start, it’s important to note that I’m currently doing all this on Ubuntu 18.04 with Go 1.12, provided via the longsleep/golang-backports ppa. For more info on how to install Go, visit the wiki

Basics

The first bit of configuration is my Go tab settings. These are standard for Go, and even if you don’t set them, gofmt will enforce them anyway.

au FileType go set noexpandtab
au FileType go set shiftwidth=4
au FileType go set softtabstop=4
au FileType go set tabstop=4

I’m using Vundle for my package management. Frankly, I should be using vim-plug, but I haven’t taken the 5 minutes to move over. Vundle does the job just fine.

The first important package to have is vim-go. This plugin is absolutely fantastic, providing pretty much all the base go features you need. In my configuration for this plugin, I turn on syntax highlighting for basically everything. I also like goimports over gofmt because it sorts my imports, so I use that as the default formatter. Finally, I turn on highlighting of the variable my cursor is on, and I enable type hints in airline.

" use goimports not gofmt to format
let g:go_fmt_command = "goimports"
" syntax highlight all the things
let g:go_highlight_build_constraints = 1
let g:go_highlight_extra_types = 1
let g:go_highlight_fields = 1
let g:go_highlight_functions = 1
let g:go_highlight_methods = 1
let g:go_highlight_operators = 1
let g:go_highlight_structs = 1
let g:go_highlight_types = 1
" highlight variables across file
let g:go_auto_sameids = 1
" vim-go get type info in airline
let g:go_auto_type_info = 1

Once this is all set, and the plugin is installed, run :GoInstallBinaries to get the latest version of all required Go binaries.

File Explorer

One of the main reasons I switched from Neovim to VS Code for Go development in the first place was the presence of a persistent file browser. I had used NERDTree before, but the fact that it didn’t persist through tabs made pretty useless for me. However, recently, I found out about :NERDTreeMirror, and it took NERDTree from a nice idea to a killer plugin for me.

:NERDTreeMirror does exactly what it sounds like. It starts the NERDTree file browser in the current tab, mirroring the state of the other already-open trees. This allows NERDTree to effectively act like a persistent file browser. The only problem is that this command isn’t automatically invoked when a new tab is opened, breaking the illusion of a persistent file browser. To fix this, I wrote my own new tab function that checks if NERDTree is already open, and if it is, :NERDTreeMirror is called after the new tab is created. I map this to <leader>t, which in my case, is \t.

function! IsNerdTreeEnabled()
  return exists('t:NERDTreeBufName') && bufwinnr(t:NERDTreeBufName) != -1
endfunction
function! TreeTab()
  if IsNerdTreeEnabled()
    execute 'tabe'
    execute 'NERDTreeMirror'
  else
    execute 'tabe'
  endif
endfu
nnoremap <leader>t :call TreeTab()<CR>

My other config settings, in order

1. Automatically open NERDTree if no arguments are passed into vim
2. Open NERDTree to the current file when \v is typed
3. Close NERDTree if it is the only window left

" auto-open if no args are set
autocmd VimEnter * if !argc() | NERDTree | endif
" open on \v
nnoremap <silent> <Leader>v :NERDTreeFind<CR>
" close if only window left
autocmd bufenter * if (winnr("$") == 1 && exists("b:NERDTree") && b:NERDTree.isTabTree()) | q | endif

Linting

I use Ale for linting. It’s very fast, and most importantly, it’s dumb easy to set up. I’ve struggled for hours with Neomake only to get it half-working, but Ale worked out of the box. Just make sure you have golint installed, and you’re good to go.

I also manually configure the error and warning characters and disable the location list for errors. It’s more annoying for me than it is helpful.

" Error and warning signs.
let g:ale_sign_error = '⤫'
let g:ale_sign_warning = '⚠'
let g:ale_set_loclist = 0

Autocomplete

Autocomplete is notoriously hard to set up in Vim/Neovim, and unfortunately, I don’t have silver bullet to fix that. I’m using deoplete and deoplete-go for better Go support. To use the latter, make sure you have gocode installed. I’m still having some problems with errors on startup, but the solution is very good once gocode has a chance to start up.

My config options, in order

1. start deoplete at startup
2. allow me to cycle through autocomplete options with the tab key
3. set a order of preference for go autocomplete suggestions

" deoplete settings
let g:deoplete#enable_at_startup = 1
inoremap <expr><TAB>  pumvisible() ? "\<C-n>" : "\<TAB>"
let g:deoplete#sources#go#sort_class = ['package', 'func', 'type', 'var', 'const']

A few tips for setup:

• Make sure the python and python3 Neovim providers are installed. They can be installed with the following commands:

  $ pip install neovim
  $ pip3 install neovim
  

• Make sure all necessary Go binaries are installed. Ensure this with the :GoInstallBinaries command in Neovim.

• Add a make instruction in Vundle for deoplete-go so that it gets made properly.

  Plugin 'zchee/deoplete-go', { 'do': 'make'}
  

Misc

Some other plugins I use are:

• auto-pairs for automatically closing parentheses and brackets
• vim-airline for a fantastic statusline
• vim-devicons to add type icons to NERDTree
• ctrlp.vim for fast fuzzy search

Conclusion

With these plugins, I can now use Neovim without losing any of the functionality I got in VS Code! If you’re interested, here’s my vimrc. Happy coding!

• listen for and ingest whole IP packets
• select the backend for a packet based on a combination of IP and source port
• encapsulate packets in UDP and send them to the appropriate backend
• attach a specified virtual IP

I’ve also implemented some extremely rudimentary testing and setup, but I’m going to rework those once the project structure solidifies some more.

Here’s some detail on the more interesting technical parts of the project so far:

Listening

I initially listened for packets with a go interface for pcap, a packet capture protocol that allows the user to select an interface and “sniff” its packets. If you’ve used tcpdump, you’ve used pcap. With pcap, I could tap into the interface that the virtual IP was using and filter for only the packets destined for the VIP. Then, I could read the packets right off the interface. There were two problems with this approach:

1. I wasn’t actually doing listening in a conventional way. If you pull up ss -4l, you wouldn’t see any entry for Caplance (because I’m not technically listening. I’m creating a TAP device). That’s a pretty big problem just because I want Caplance to work well with conventional Unix tools.
2. Closing a pcap device from go is stupid slow. In my tests, closing the device took upwards of 30 seconds, and the device needs to be closed before Caplance can terminate. This is unacceptable, and it was ultimately what did in the pcap approach for me.

Enter IPConn. IPConn is a listener for go that listens at the IP level. That means, when you do a read off of it, it reads in from the IP layer down, exactly what I want. Since it’s a built-in go listener, it also closes extremely quickly. Whereas before I needed to set up a filter, find the appropriate device, etc., with IPConn, my listening can be reduced to this simple function:

func (b *Balancer) listenWithConn(conn *net.IPConn, pool *sync.Pool) {
	for {
		buf := pool.Get().([]byte)
		n, err := conn.Read(buf)
		if err != nil {
			log.Println("could not read from connection")
			continue
		}
		toSend := rawPacket{buf, n}
		b.packets <- toSend
	}
}

To provide a bit of context around this function, pool is a pool of buffers that the IPConn can use to store data from reads. b.packets is a channel (concurrency-safe queue) of packet for the worker goroutines to read packets off of.

Packet Forwarding

While I would to use normal routing rules to route incoming packets over GRE tunnels to backends, that won’t work here. If we were always forwarding the same IP range of packets to the same backends, we could use iptables, but for this project, we want to use consistent hashing. Because of this, we have to write listening (as seen above) and packet forwarding ourselves.

First, it’s important to note that there’s no good GRE library for go right now. The only real way to create a GRE tunnel in go is to make calls to the kernel to tell it to construct one. There’s no way to have a GRE object, for instance, that you can just write data to in the same way you do with a TCP connection. I may take this up as a future project.

Because of the lack of a library, I would have to write data directly to the wire. My initial idea was to (again) use pcap. My algorithm was as follows:

1. Create GRE tunnel to backend (with netlink)
2. Manually create GRE header for backend
3. For each packet destined for the backend:
   1. Create appropriate GRE header for the packet
   2. Marshal GRE header into a []byte
   3. Append the packet onto the marshalled header
   4. Perform some crazy pcap magic to figure out mac addresses of the next hop on the way to the backend
   5. Write the encapsulated packet to the right physical interface

Ya know, now that I type it all out, that was an awful plan.

After a lot of frustration and reflection, I asked myself the pivotal question, “Why am I so set on GRE in the first place?” The only reason I could come up with was that Google used it, and that’s a pretty bad reason. Thinking about it more, I realized I could just use a UDP connection in place of GRE and achieve the same results I wanted.

Now, my algorithm is as follows:

1. Dial a UDP connection to each backend
2. Write each packet to the appropriate backend with the built-in Write(data []byte) method

That’s a wee bit simpler. It’s not as easy to show in a single piece of code as listening, but it wasn’t too hard to write.

Graceful Stop

I implemented graceful stop, which turned out to be way harder than I expected. I’m anticipating writing about it in the future, though, so I’m not going to include details on how it works right now. It’s also very likely that I’ll change how it’s implemented. If you want to see the current iteration of it right now, you can find the code here.

Future Work

Obviously, I’m not done yet. I still need to implement the following:

• Configuration file
• Environment variable configuration
• Client registration
• Client health checks
• Client packet injestion
• Optimization
• More that I don’t even know exists yet

I’m crazy excited to get started on all those items, and I can’t wait to write about them once I figure them out.

During this time, I learned a significant amount about myself and the process of improvement, so I thought I’d share a bit about what I’ve learned and some of my personal opinions on the topic. While reading, keep in mind that what I’m saying is based purely off my own experience and is likely not applicable to all who read this.

Making improvements small

I’ve made the mistake of trying to make large improvements many times. The danger with this is that you set expectations that you will almost certainly not be able to meet. The subsequent dropping of some of the expectations just weakens the others, and eventually, the whole system falls apart. The hard part is that the desire to make huge improvements often comes from the best place in the heart! Wanting to improve is an admirable thing, and wanting to radically improve yourself is even more admirable. Unfortunately, for the vast majority of people, I just don’t think quick radical improvement is feasible.

Obviously, the solution here is to make your improvements small. For me, the easiest way to make my improvements small is to first make the rules for my improvement unambiguously specific. After the rules have been fully specified, I can easily see if what I’m doing is too much or not. I work on those rules for a week, and afterward, I reflect on whether or not they were too strenuous. Typically, I can tell that they were too strenuous simply if I was able to stick ot them or not.

Allowing rules to change

You’re never going to get your rules right the first time. In fact, they shouldn’t! That’s why you try out the rules in 1-week increments in the first place. The important thing is to track what rules are useful, which aren’t, and what rules you add/change throughout the week.

For example, in my week 1, I decided to track my time with an app called Clockify. The only problem is that Clockify didn’t help me at all. Timing myself on tasks made near no difference. However, halfway through the week, I found an app called Forest that was hugely helpful for me. It not only timed me but also locked me out of my phone, which ended up being the kick I needed. I noted that I was using Forest instead of Clockify in my reflection, and the change was done.

I should note, however, that allowing rules to change doesn’t permit just dropping rules. If you just start dropping rules, it cheapens the values of the rest, and eventually, you’ll give up on those. If you are getting rid of a rule or changing a rule, it’s imperative that you have a good reason for doing so and that you record the change.

The eventual end of it all

At the end of the day, making an improvement a week, no matter how small, piles up (that’s the idea). For me, after about 7 weeks of the project, I was done. The project had given me a lot to think about, and it really helped me get on track at a time where my life was pretty disorderly.

While I’m done adding improvements for the time being, I’m still using many of the improvements I made during the project! I’m still:

• Tracking tasks in Todoist
• Focusing with Forest
• Exercising 5+ times/week
• Paying attention to my screen time and reading my reports
• Blogging (obviously)
• Budgeting with Mint

I’d really encourage anyone reading to try out a project like this. It’s certainly helped me get myself on track in ways that I’ve been attempting for years now. There’s a link to my Github repo at the top, but I’ll drop it again here:

https://github.com/Pwpon500/improvement

This program requires that the students have access to a rack with a managed switch and at least one server capable of virtualization. First, these resources are often difficult for schools to procure. Then, even if the school can procure them, only a single student or group of students can work on the labs at a time. Furthermore, working on labs at home isn’t even a possibility since you can’t take a whole rack home.

Mininet aims to solve these problems.

What is Mininet?

Mininet is a network simulation tool for Linux. The goal of it is to create a “realistic virtual network” with minimal overhead. Since it aims for minimal overhead, it shares both filesystem space and PID space with the host it operates on. Both of these can be avoided, however, with the private directories host option in Mininet.

One cool thing about Mininet is how lightweight it is. Since it does basically no isolation other than creating some virtual kernels, it spins up and down in seconds, using minimal memory. This means it can run on a device as weak as a Raspberry Pi!

The Mininet API

Another amazing part of Mininet is its Python API. This API lets you programmatically create new networks, interact with them, and destroy them. Take the following simple example:

#!/usr/bin/env python2
""" This script provides a basic switch topo."""

from functools import partial
from mininet.topo import SingleSwitchTopo
from mininet.net import Mininet
from mininet.cli import CLI
from mininet.node import Host
from mininet.log import setLogLevel


def simple_test():
    "Create and test a simple network"
    topo = SingleSwitchTopo(k=5)
    private_dirs = ['/run', ('/var/run', '/tmp/%(name)s/var/run'), '/var/mn']
    host = partial(Host, privateDirs=private_dirs)
    net = Mininet(topo=topo, host=host)
    net.start()
    CLI(net)
    net.stop()


if __name__ == '__main__':
    # Tell mininet to print useful information
    setLogLevel('info')
    simple_test()

In these few lines of code, I have a file that will automatically create a network with 5 hosts on it where each host has PID separation. Now, instead of having to manually specify all these options, the user can just run “sudo ./setup.py”, and they’re consoled in.

The possibilities only begin here. Things get even crazier with custom topologies.

Teaching Example

I could go on and on about Mininet, but the power of it becomes most evident when you actually see how it can be used. Here, I’m working on building up some labs based around Mininet for my friends to learn network engineering with. Feel free to try the labs and contribute your own!

1. DNS level
2. Network level/Virtual IP level
3. Datacenter level

While I was familiar with the Datacenter level (aka reverse proxy load balancing) and DNS load balancing (achieved partially through round robin DNS), I had never looked at network-level load balancing. After reading this section, I was fascinated with how their network-level load balancing works.

After some research online, I realized that the system they were referring to was their load balancer called Maglev. In 2016, Google released a paper detailing Maglev and how they built it. If you’re interested in the nitty-gritty details, you can read the paper here. It’s a dense but fascinating read. I could spend a huge amount of time going into all the details of what makes Maglev interesting, but instead, I’ll focus on two aspects: Maglev hashing and packet encapsulation.

Introduction

Before anything else, I’ll describe the function of a network load balancer. The job of a network load balancer is to distribute packets destined for a “virtual IP address” to a set of backends evenly. Take the following picture as an example:

In this example, the job of Maglev is to deliver all of the packets coming in from the public internet evenly to backends 1-4.

Maglev Hashing

The first part of load balancing is actually deciding which backend to send an incoming packet to. In network load balancing, we are just looking at the IP layer, so we can’t keep track of things like which backend has the most TCP connections open to it. Since we can’t keep track of current state, we have to figure out how to distribute packets evenly just based on some attribute of the packet. Assume we pick out some attribute of each packet (this is usually a hashed combination of source IP and source port) and call id id. Then, we can treat id as an integer and evenly distribute packets to backends using the function backend = id (mod n) where n is our number of backends. Assuming an even distribution of IDs, this method will evenly distribute packets to backends, and it will keep sending packets from the same source to the same backend.

So, is it that simple? It seems like we’ve achieved what we want to - even distribution - with no downsides. Consider what happens when our n increases or decreases by even 1, however. In this case, since we don’t know the range of our id hash function, which backend our packets go to could be completely changed for all connections. This would mean that all open client connections would be broken, which is certainly not desirable behavior. To see this in action, let’s look at the example of where to send incoming packets with 5 backends versus with 4 backends.

As you can see, all the packet flows are disrupted except that for the packet with id 561. Since we will certainly be adding and removing backends live, this is unacceptable behavior. This is where consistent hashing comes to the rescue. Consistent hashing is very complicated, but the short of it is that it provides a way for minimal disruption when adding and removing backends. With consistent hashing, instead of all the packet flows being disrupted, at most 1/n (where n is the number of the backends) of the packets flows are disrupted.

There are many trade-offs with different kinds of consistent hashing that this article explains better than I can, but in essence, there are three things you can optimize for in consistent hashing: memory usage, lookup speed, and hashtable rebuild speed. You can typically have 2 of these at once, but you can never have all 3 without making some other very significant tradeoff. Google’s solution, called Maglev Hashing, optimizes for the first two. It assumes node failures are uncommon, and in making that assumption, it can get low memory usage with high lookup speed and minimal disruption when new backends are added but poor rebuild speed when a backend fails. This is a pretty reasonable assumption, and it turns out to work extremely well in the context of network load balancing.

Now that we know how to pick backends, let’s actually talk about how we send data to backends.

Packet Encapsulation

First, let’s consider the reverse proxy method of load balancing. In this method, connections are taken in by the load balancer. The balancer then initiates a new connection to the appropriate backend for the connection, and proxies the connections together, allowing the client to indirectly talk to a backend. This method looks like this:

In this design, the load balancer has to do the following:

• Keep track of all active connections
• Proxy data between connections for all connections
• Pass both ingress and egress traffic for all the backends

While this is acceptable (and even desired) for a datacenter load balancer, we can’t pass packets at the scale a network load balancer needs to using this design. This is where packet encapsulation comes into play.

The solution that Google found to this problem was to use the following algorithm:

1. Maglev hash the packet to determine its backend
2. Wrap the packet in a layer of GRE (generic routing encapsulation)
3. Send the encapsulated packet to the desired backend
4. Have the backend break the packet of the encapsulation and fully process it
5. Have the backend directly reply to this packet.

Did you catch that last step? Since we’re using encapsulation, the backend is seeing the original packet as received by the load balancer, so it can directly reply to the packet. We can see this in action on an example packet flow:

The fact that the backend can directly reply is a huge benefit of the encapsulation method. To see why, consider the use case of YouTube. When a user requests to see a video, the request payload is very small, only containing metadata about what video they want to watch. The reply, however, is gigantic since it’s a full video of up to 8K quality! With this method, the load balancer only only has to worry about the request, meaning it can handle drastically more traffic than a reverse proxy can.

Personally, I thought this was the most interesting part of Maglev. I had only ever thought to use GRE as a site-to-site VPN, but this gave me a whole new outlook on what it could do.

Conclusion

Maglev brings a ton of interesting concepts to the table and combines them in a way we’ve never really seen before. I could only cover two things here, but tech like ECMP and RPC also play a huge role in how Maglev works. If you’re interested, I highly encourage reading the paper.

Thanks for reading! Look below for a shameless plug.

Shameless Plug

As I said, I found Google’s ideas on network load balancers fascinating. In fact, I found it so interesting that I decided to implement it myself! If you’re interested in seeing how some of these ideas are implemented, I’m writing my own version of Maglev called Caplance here. At the time of writing this, I’m working on setting up automated testing with Mininet, and I’ll probably write more in the coming weeks as I make more progress on the load balancer.

Who needs college really?

The argument against a CS degree typically takes two main points:

• You can learn CS on your own
• Any current technologies won’t be taught in school anyway

I think the first point is much less valid than people give it credit for. It’s easy to learn programming on your own, but learning computer science on your own is far more challenging. While programming is perhaps the most critical part of computer science, being a good programmer is a drastically different thing from being a good software engineer. Concepts like proofs, time complexity, and complex algorithms are also crucial to fully developing as a software engineer, and these concepts are extremely challenging to learn on your own. I’ll go more into this idea in the next section.

This next point is actually where I think the aforementioned Devon Crawford gets it right. Colleges simply cannot stay current with state-of-the-art technologies. Take Devon’s example of Kubernetes. Kubernetes is the state-of-the-art in container orchestration, and it has pretty much taken over the container orchestration space, at least in open source. Despite all of this, most universities don’t have classes even touching it, much less actually going into how it works. This is a huge weakness for colleges simply because of how much hiring is skills-based. Having something like “Kubernetes proficiency” is a huge boost to any CS resume, but because of the nature of how long it takes to build and establish a new class, colleges will never be able to keep up.

The case for college

To touch on the self-teaching side of CS, let me provide a personal anecdote:

In my junior year of high school, my programming competition team started to become seriously competitive. We started to place highly at tournaments, and it became clear that if we wanted to reach the next level, we would have to learn real algorithms. It became my job to learn all things graph theory. In the span of a few weeks, I learned BFS, DFS, Dijkstra’s, and Prim’s - or so I thought.

Fast forward to my second semester of college. Now, I’m re-learning all of those algorithms, and I’m realizing just how wrong I was about all of them. For one thing, my old versions of the algorithms all ran in at least n squared time, if not n cubed. Getting a formal education around these has taught me how to do these in linear or logarithmic time. My understanding of them has also far deepened, teaching me how to use them in a variety of different contexts I would have never imagined before.

It’s primarily this experience that has made me understand the value of a formal education. While I “knew” these algorithms before, I now realize how little I actually understood about them. I think idea holds for a multitude of computer science concepts.

To address colleges’ difficulty in teaching new technologies, I actually don’t think that’s the job of the college. The job of the college is to provide the student with the necessary background as to be able to learn new tools quickly. The job of the college is not to teach current technologies, because if it was, those skills would be useless when that tool fell out of favor. The best thing a college can do is what the good ones are doing - provide the students with a solid foundation to build on. If the student has the right background, they’ll be able to learn any tool you can throw at them.

College also provides a multitude of other benefits not listed here including, but not limited to, establishing professional connections, facilitating learning outside one’s comfort zone, and providing a safe space for young people to grow into the world.

Conclusion

While I understand the ideas behind the nay-sayers of college, I ultimately disagree with them. I think that in the long-term, going to college is one of the best decisions you could make.

• At the start of the turn, each player must roll the 6 dice.
• After rolling, the player sets aside all the dice that scored.
• The player can then choose to roll again to potentially get more points or to take the points they have already scored.
• If the player rolls and scores no points, they get 0 points and their turn is over.
• If the player uses up all 6 dice, they can recycle all of them and roll all 6 again.

The ways to score are as follows (dice cannot be double counted when scoring):

The winner is the first person to score 10,000 points. Being the competitive person I am, I decided to give the game some analysis and see if I could determine the expected value for a roll in order to figure out if I should roll or not. I chose to use Go for this because I figured I’d be doing a good amount of brute forcing, so I wanted a pretty fast language that I was familiar with. I also just like writing Go.

Pure Brute Force

At first, my idea was to do 2 simple things to generate expected value:

1. Use backtracking recursion to generate all the possible rolls
2. Find all the ways to partition the dice in each roll and take the maximum scoring partition

Starting with my backtracking method, I quickly wrote the following

func backtrack(rolls []int, toRoll int) float64 {
	if toRoll == 0 {
		return float64(maxScore(rolls))
	}

	sum := 0.0
	for i := 1; i <= 6; i++ {
		sum += backtrack(append(rolls, i), toRoll-1)
	}
	return sum / 6
}

The idea here is fairly simple: I loop through all 6 possible die rolls, sum them up, and average them. When I’ve used up all of my rolls, I score the resulting dice.

Scoring was not quite as simple as I had hoped, though. Partitioning a set is a little complicated, especially in a language like Go with no real concept of set algebra. The general algorithm I found is as follows:

1. If there is only one element, return the set containing just that element. Otherwise, do the following:
2. Remove the first element from the set and generate the partitions for the remaining set.
3. For each generated partition, do the following
   1. Add the partition containing {removed element} + generated partition
   2. For each set in the generated partition:
      1. Add the partition containing {removed element + set} + generated partition \ set

func generatePartitions(rolls []int) [][][]int {
	if len(rolls) == 1 {
		return [][][]int{{{rolls[0]}}}
	}

	firstElem := rolls[0]
	rest := generatePartitions(rolls[1:])
	toReturn := [][][]int{}

	for _, elem := range rest {
		toReturn = append(toReturn, append(elem, []int{firstElem}))

		for i, set := range elem {
			removed := make([][]int, len(elem))
			copy(removed, elem[:])
			removed = append(removed[:i], removed[i+1:]...)
			toReturn = append(toReturn, append(removed, append(set, firstElem)))
		}
	}

	return toReturn
}

Then, I take all these partitions, score each one, and return the max of all the scores:

func maxScore(rolls []int) int {
	max := 0
	partitions := generatePartitions(rolls)
	for _, partition := range partitions {
		score := scoreSuperSet(partition)
		if score > max {
			max = score
		}
	}

	if max > 0 {
		return curScore + max
	}
	return 0
}

After writing this code, however, I realized that it had quite a few problems:

• Backtracking could have to check up to 6^6 (46,656) rolls
• Each roll would have to be partitioned in every way possible. This follows the Bell numbers, so in the worst case, there are 203 possible partitions to check. Combined with the previous number, in the worst case, I have to check 947,1168 partitions.
• I’m only accounting for one roll! The part of Farkle that makes it so hard to analyze is the ability to roll multiple times, but if you take a look at my code, I’m only checking the possibilities for the direct next roll.

Let’s make a checklist for things to fix:

• Be smarter than backtracking all possibilities
• Find a way to score a roll in constant time
• Account for possibly making multiple more rolls

Improving Scoring

Something didn’t sit right with me regarding how I did scoring. It seemed totally wrong that I would need to generate up to 203 combinations and take their max score. After all, I don’t consider all possible combinations when scoring my rolls playing with my family. I group the dice according to their numbers and see if the groups fit into any of the scoring categories. Turns out, that works pretty well for real scoring. Here’s what I came up with:

1. Go through all the dice and record how many of each number there are. Also record which dice appear with which frequency

func scoreRoll(rolls []int, score int, depth int) float64 {
  var occurences [6]int
  for _, elem := range rolls {
    occurences[elem-1]++
  }
  var frequencies [7][]int

  for i := 0; i < 6; i++ {
    occurs := occurences[i]
    frequencies[occurs] = append(frequencies[occurs], i)
  }
...

2. Compare the frequencies against the rules of the game

I won’t include this part since it’s just a series of if/else statements. You can find the code at this iteration here if interested.

3. Count the number of remaining 1s and 5s

if occurences[0] < 3 && countIndiv {
toReturn += float64(100 * occurences[0])
numUsed += occurences[0]
}
if occurences[4] < 3 && countIndiv {
toReturn += float64(50 * occurences[4])
numUsed += occurences[4]
}

4. If there is no score, the player gets no points. Otherwise, they get their scored points plus the score they already have

if toReturn == 0 {
	return 0
}
return toReturn + float64(score)

Now, I have a scoring algorithm that will work in constant time. Let’s update that checklist:

• Be smarter than backtracking all possibilities
• Find a way to score a roll in constant time
• Account for possibly making multiple more rolls

More Backtracking

Taking extra rolls into account is actually pretty simple in theory. When you score each roll, take the expected value of rolling again. If that expected value is higher than what you’ve already scored, use it. Otherwise, keep your current score. This can be simply implemented as follows:

expectedRoll := backtrack([]int{}, numLeft, int(toReturn))
if expectedRoll > float64(toReturn) {
	toReturn = expectedRoll
}

I ran this code and waited. I waited some more, and I waited even longer. Looking back, I committed the cardinal sin of recursion: recursion without a base case. Each roll would check the expected value of another roll, and the cycle would infinitely repeat. To fix this, I added a simple depth parameter to backtrack and implemented it as follows:

func backtrack(rolls []int, toRoll int, score int, depth int) float64 {
	if depth >= maxDepth {
		return 0
	}

maxDepth is just a global variable that the user sets to define how deep they want to recurse. I then pass depth into my scoring function and change my expected roll call to implement depth:

expectedRoll := backtrack([]int{}, numLeft, int(toReturn), depth+1)

Now, theoretically, I have an algorithm that’ll go to whatever arbitrary depth I want. However, upon running this, I can only go to a depth of 2 before the program explodes running time. Since I’m backtracking, each added level of depth makes the running time go up exponentially. So I’ve fixed one problem, but just amplified another. Let’s update the checklist:

• Be smarter than backtracking all possibilities
• Find a way to score a roll in constant time
• Account for possibly making multiple more rolls

Finding A Model

Faced with no idea for how to do this without backtracking, I had the idea of dynamic programming. If I could just find a decent model for the expected values, I could get rid of all the costly backtracking. I added support for command line flags and generated all the expected values from 1 to 400 with the following command (this is in fish but a similar thing can be done in bash):

for x in (seq 400)
    ./farkle-odds -dice 1 -score $x
end

I used plot.ly to plot all this out and saw something pretty amazing:

{:width="750px”}

It’s linear! Realizing this, I generated lines of best fit for each of the numbers of dice. I then added the approxScore method in to use the lines of best fit to approximate the expected value of a score:

func approxScore(toRoll int, score int) float64 {
	return mVals[toRoll-1]*float64(score) + bVals[toRoll-1]
}

Generating Lines

After some plotting of greater depths, I concluded that the greater depths are almost linear, and any deviations from a linear model were not significant enough to substantially impact the generated expected values. Using this linearity to my advantage, I used the following algorithm to generate coefficients for a linear model for any arbitrary depth:

1. Use backtrack to find 2 expected values
2. Use simple algebra to find the m and b values for the line between those points
3. Record those m and b values in JSON to a file
4. Replace the old m and b values with the new ones and repeat the process

Here’s what that looks like in actual code:

func generateCoeffs() {
	for i := 1; i <= 50; i++ {
		newM := []float64{}
		newB := []float64{}
		for j := 1; j <= 6; j++ {
			// find high and low vals for this iteration
			lowVal := backtrack([]int{}, j, 0)
			highVal := backtrack([]int{}, j, 400)
			// use high and low vals to generate line
			newM = append(newM, (highVal-lowVal)/400)
			newB = append(newB, lowVal)
		}
		coeffs.MCoeffs = newM
		coeffs.BCoeffs = newB
		writeVals(newM, newB, i)
	}
	fmt.Println("Coefficients generated and written")
}

Now, we can finally check off that last box!

• Be smarter than backtracking all possibilities
• Find a way to score a roll in constant time
• Account for possibly making multiple more rolls

Conclusion

After some refactoring, I’m really happy with the result of this little experiment. The linear model isn’t a perfect fit for the expected values, but it’s damn good and takes constant time to run.

If you’re interested in seeing what the code looks like, it can be found here. Make sure to let me know if there are any improvements I can make.

Also, here is some of the plotted data of the expected values for different dice rolls:

Depth 1 Depth 2

Why VPLS?

With VPLS, you can deliver a layer 2 circuit over a routed backbone. This lets you extend that circuit from one location to another over a route that you can easily control. If the backbone is not routed, you leave the path-finding up to spanning tree; while that works for smaller networks, the bigger your layer 2 domain gets, the less-reliable spanning tree gets. Eventually, it may start taking spaghetti-like paths, and there will be little you can do about it.

VPLS is also useful in allowing the extension of layer 2 domain to a remote site. For example, in the case of VoIP phones, VPLS would allow the phones to be directly connected to the remote VoIP server, requiring no configuration whatsoever at the client site.

VPLS on a High Level

VPLS works by creating MPLS pseudowires (point-to-point layer 2 circuits) to every node you want to be part of your bridged domain. You then add each pseudowire as well as the physical interface you want to be “bridged into” the VPLS domain. In basic terms, you have now created a “virtual switch,” plugged each pseudowire into it, and plugged your physical interface into it.

The Setup

This setup is completely virtualized, but it can be replicated easily with physical servers as well. There will be one provider and three provider edges. Each router will have a unique router-id by which it will be identified in OSPF and LDP. That address will be assigned to a secondary loopback interface (lo1) and advertised to the other routers using OSPF. The setup looks like this: {:width="500px”}

Each PE will be directly attached to the provider with a /30. I’m using the 172.30.2.X IP scheme, but you can use whatever you want. Just make sure to match up those addresses between the provider and PE. Similarly, my use of 10.0.0.X for the router-id’s can changed to whatever you want.

Global Configuration

On each of the nodes, you will have to enable some services. I’m doing this in /etc/rc.conf.local. Append these two lines at the bottom to enable OSPF and LDP. Also make sure to wipe any lines disabling OSPF or LDP out of /etc/rc.conf and /etc/rc.conf.local:

ospfd_flags=""
ldpd_flags=""

You’ll also need to give each router its appropriate router-id on its loopback interface. You can put this in /etc/rc.local or use the hostname.XXX format. I prefer the hostname.XXX format.

/etc/hostname.lo1:

inet 10.0.0.1 255.255.255.255
description "id_loopback"

Make sure to change that 10.0.0.X address for each router.

Provider Configuration

The provider is the simplest node to configure since it just acts as a glorified label switch.

/etc/ospfd.conf:

router-id 10.0.0.1

area 0.0.0.0 {
    interface lo1
    interface re0
    interface re1
    interface re2
}

/etc/ldpd.conf:

router-id 10.0.0.1

address-family ipv4 {
    interface re0
    interface re1
    interface re2
}

/etc/hostname.re0:

inet 172.30.2.1 255.255.255.252
mpls
description "p1_edge"

It’s critical that you have the mpls line in your configuration. This lets OpenBSD know to treat that interface as a provider-facing interface. Repeat this interface configuration for each PE. Make sure to use a different IP for each interface as well as a different subnet. I recommend using consecutive /30 blocks (172.30.2.0/30, 172.30.2.4/30, 172.30.2.8/30).

PE Configuration

The configuration of PEs is similar to that of the provider, but pseudowires have to be created to each other PE. Each PE has two physical interfaces. re0 is the provider-facing interface, and re1 is the client-facing interface.

/etc/ospfd.conf:

router-id 10.0.0.2

area 0.0.0.0 {
    interface lo1
    interface re0
}

First, we have to create our pseudowires. These follow the mpwX naming convention. All we have to do in /etc/hostname.XXX is create them and bring them up. LDPD takes care of the rest. We also have to bring up our physical client-facing interface.

/etc/hostname.mpw0:

create
up

/etc/hostname.re1:

up

We also need to bring up our bridge interface and add our pseudowires and our physical interface to it.

/etc/hostname.bridge0:

add re1
add mpw0
add mpw1
up
description "vpls_bridge"

Remember that in this LDP configuration, re0 is provider-facing, and re1 is client-facing. If you want some extra information on how ldpd.conf works, check out the OpenBSD ldpd.conf man page. /etc/ldpd.conf:

router-id 10.0.0.2

address-family ipv4 {
    interface re0
}

l2vpn pe1 type vpls {
    bridge bridge0
    interface re1

    pseudowire mpw0 {
        neighbor-id 10.0.0.3
        pw-id 100
    }
    pseudowire mpw1 {
        neighbor-id 10.0.0.4
        pw-id 100
    }
}

Repeat this process with the rest of the PEs, changing the appropriate IPs and router-id’s. After that’s all done, restart ldpd and ospfd, use /etc/netstart to bring up interfaces, and you’re ready to go!

You can find my full configuration here. Use this if you need any extra help figuring out the configurations for the nodes.

Testing

To test your VPLS setup, connect clients to your physical interfaces on all your PEs, and give them all static IPs in the same subnet. Try to ping each other and see if the pings return. If they do, you did it! You’ve now created a functional VPLS network.

Diagnostics

More likely, however, your network doesn’t work. Don’t worry! OpenBSD has some great tools for viewing LDP status. Here are some common commands and what they look like on a working PE:

pe1 / root / 23:18:29
> ~ # ldpctl show neighbor
AF   ID              State       Remote Address    Uptime
ipv4 10.0.0.1        OPERATIONAL 10.0.0.1        00:03:53
ipv4 10.0.0.3        OPERATIONAL 10.0.0.3        00:03:11
ipv4 10.0.0.5        OPERATIONAL 10.0.0.5        00:03:11
pe1 / root / 23:18:29
> ~ # ldpctl show l2vpn pseudowire
Interface   Neighbor        PWID           Status
mpw0        10.0.0.3        100            UP
mpw1        10.0.0.4        100            UP

You can also use ospfctl to if you think the problem may lie in the routing.

Conclusion

VPLS is some amazing technology, and hopefully, you can implement it yourself with the help of this post.