I decided to pick up a ZSA Voyager after justifying the price with potential ergonomic benefits. The decision was more an ergonomic one and excuse to learn a slightly better way of typing. That said I don’t get RSI from typing on anything except mac keyboards and using the mac trackpad which is pretty awful ergonomically.

My experience

The first thing I noticed was that typing on it was surprisingly difficult to initially get adjusted to. It felt like I might give up after the first 2 days of training because I was so used to row staggered keyboards and kept on making typos. I was averaging 30-60WPM on the first few days with very high peaks that were mostly luck. My average speed was low and my accuracy was worse.

The reason for this is because the ZSA voyager has a columnar layout which is fundamentally different to regular keyboards. A columnar layout places the keys in alignment with the columns as opposed to the rows. I had heard from a coworker with who had tried a columnar keyboard that typing on columnar was extremely frustrating and difficult so I was anticipating a long onboarding process in figuring out how to actually use the thing.

With training I found that that once you stick through it for a day or two it becomes significantly easier. I was able to pick up columnar pretty quickly strangely enough. Learning columnar is easier than learning an entirely different layout.

I type normally at 170-200WPM on row staggered keyboards on typing tests (avg 155 wpm with grammar) and was able to reach this in 10 days with 1 hour of daily practice on keybr.com.

Typing symbols or non alphabetical characters is noticeably difficult since you only have 52 keys and have to remap them, so I spent some time figuring out a layout that worked for me. I did not stick with the default layout. I still don’t consider myself super fast at typing the symbols as my layout isn’t fully optimal, but the symbols which I press frequently are easier to press with my layout and feel more comfortable than a regular keyboard.

Remapping your own keys opens you up to decision fatigue which is something I think a lot of people would find annoying. There’s even stuff like homerow mods to keep most of your important keys on the homerhow, which I haven’t learned. To get over this decision fatigue hump I accepted that my layout would have tradeoffs that make it suboptimal.

I frequently changed my layout to things that made more sense and with the help of Oryx (the official software ZSA makes to do this) it felt really natural and easy to do. By far the least frustrating experience I’ve had with keyboards since I didn’t have to use QMK or flash firmware manually without the help of a nice interface. If you are curious, here is the latest layout I am using.

Speed results

My speed results on keybr.com show that the first few days of typing I was messing up 1/10 words and not going very fast, but as time went on my accuracy and average speed improved. Right now without any grammar being typed I average around 175wpm with the ZSA

• Day 1: top speed 168wpm, avg 60wpm, acc 91.7%
• Day 2: top speed 168wpm, avg 102wpm, acc 90.18%
• Day 3: top speed 157wpm, avg 115wpm, acc 92.05%
• Day 4: top speed 156wpm, avg 127wpm, acc 94.6%
• Day 5: top speed 170wpm, avg 132wpm, acc 95.38%
• Day 6: top speed 180wpm, avg 145wpm, acc 95.81%
• Day 7: top speed 178wpm, avg 146wpm, acc 96.7%
• Day 8: top speed 180wpm, avg 155wpm, acc 96.89%
• Day 9: top speed 180wpm, avg 156wpm, acc 95.67%
• Day 10: top speed 169wpm, avg 150wpm, acc 96.22%
• Day 11: top speed 183wpm, avg 160wpm, acc 96.68%
• Day 12: top speed 189wpm, avg 165wpm, acc 96.87%

I highly recommend using keybr for training since it lets you practice typing case insensitive words, which practically is the vast majority of your typing experience anyway

By day 20 I hit 203wpm. It’s possible I’m the fastest ZSA Voyager typist in the world!

The thumb cluster

I was very surprised how pleasant it is to use your thumbs for space, backspace, enter and a layer modifier. This is currently how I use the keyboard and it is very intuitive to type on and doesn’t cause me any issues.

I don’t think I can go back to regular keyboards after having experienced the ergonomic benefits of having my fingers not have to move around as much.

The downsides

The main things I find that are downsides

• All of the switches that come with the board are loud, and the keyboard is noticeably louder than a regular keyboard. You have to buy silent ones online separately. I have bought separate silent switches to help with this.
• Not a whole lot of tenting options
• Price is a bit too steep for a keyboard ($365)
• Very easy to misstype on if you are used to switches that require more actuation force
• Cost, price point is almost double what it should be

Anecdotally I’ve also heard that the lack of palm support combined with your hand size may impact how comfortable you feel using the keyboard. For every 9 users that have a good experience with the board there is 1 that doesn’t really like it. Personally I found it very comfortable to type on even without palm support (using a desk mat).

TRRS port

There is a known defect with the TRRS ports where either both ports will break or only one port will break, but it results in the keyboard not receiving enough power to work. The board I received started having this issue shortly after use. I sent it in under warranty, and ZSA’s support was incredible. I received a replacement keyboard within a week under no cost.

My main recommendation here is to just be aware of the warranty, but I don’t think most people will encounter any problems.

Still, it’s not ideal that this defect exists at all so keep this in mind if you are considering buying the keyboard.

The upsides

In no particular order

• Super portable
• Ergonomic. It can’t be overstated how much better it feels to type on a split keyboard with a flat profile like this if you used regular keyboards
• Best in class customization software, entirely in browser (Oryx)
• 2 year warranty
• Comes with a portable case

Conclusion

I highly recommend this keyboard if you are already used to split keyboards and want something portable/don’t mind the steep price point. Would I recommend it for people beginning to learn split keyboards though? No. There’s probably better options out there to start with like a Kinesis or even a Glove80.

This post describes the encapsulation that is used and explains some of the motivations and a blurb on flow hashing.

What is the networking stack

When you hear the “networking stack” you need understand it in terms of layers of abstraction that build on top of each other. At the lowest level you have physical data transmission, the next level you have ethernet frames (data link), the next level contains IP frames (network layer) and then finally a TCP or UDP frame. Colloquially we just refer to these as layers 1-4.

One way to understand the usefulness of the networking stack is in terms of what constraints you have: when you go up the networking stack you have less constraints. TCP on the transport layer enforces a socket per connection and has strict ordering rules, but when you get to layer 3 (IP aka TUN layer) you can inspect TCP, as well as UDP packets, as well as everything else (eg ICMP packets).

To oversimplify, understanding these layers is important for understanding how you can process packets quickly for something like sending packets from one partition/network to another, because intuitively you need additional layers to store meaningful routing metadata.

Geneve encapsulation

RFC 8926 describes Geneve encapsulation.

Geneve encapsulation is most relevant from sections 3.1 to 3.3. From a technical perspective the main thing you need to know to understand GENEVE is the following: it exists as a layer 4 (transport layer) UDP way of encapsulating layer 2 - layer 4 information for network virtualization. So you can send these GENEVE encapsulated packets using a user space socket but it communicates a lot more information because it has both an inner and outer packet frame. The inner frame can contain an entire ethernet frame which can be intercepted by a TUN/TAP device running on a different machine (a NIC). Any time you have a need to do network virtualization it is most likely you would have to use GENEVE encapsulation as it is the standard (supercedes VXLAN).

The included figure of GENEVE on IPv4 shows that your payload can be anything you want it to be or what the receiver expects. If you’re encapsulating a regular TCP packet you would just write the IPv4 header and TCP frame here.

Importantly the details of how these packets are used are up to the network you’re connecting to

Since GENEVE operates within the context of UDP it has a natural analogue to the connectionless semantics of Ethernet and IP which is why it lends itself so well to be used to directly pipe in data from a TUN device (as opposed to something like TCP).

GENEVE encapsulation is super important whenever you want to build networks (overlay networks) on top of another network (an underlay) which is why it’s widely adopted among cloud computing companies. They have the raw hardware (underlay) and the overlay (to oversimplify a bit) is the virtualized network that they create for you (eg a VPC with subnets). The following section will describe this in more detail

Real world application with flow hashing

A lot of the time you’ll have to deal with some form of encapsulation during which there’s a network boundary transition. I dealt with this on my team where our traffic transitions the network boundary from our service VPC (IPv6) into a customer VPC (IPv4). A large part of our service offering is tunneling traffic into customer’s VPCs.

In this case the outer packet was IPv6 going over UDP and the inner (GENEVE encapsulated) packet was IPv4.

This meant that any reverse traffic was always IPv4 and had to be translated to IPv6 to be sent back to the originating client.

You cannot simply encode the client’s IPv6 address in an IPv4 space because it won’t fit, so instead you have to perform translation by hashing the originating flow 5-tuple, mapping this to an IPv6 address and use this information to map back onto the originating flow for reverse traffic.

For example, if your service operates in an IPv6 space, that is to say every client that ingresses to your service sends it with an IPv6 header and you want to egress this traffic to a machine that operates exclusively in IPv4 space (over GENEVE or not) this becomes a significant hurdle. On the ingress path for return traffic you must identify the correct forward initiating IPv6 flow from an IPv4 packet, and are forced to store additional information (state) to identify originating flows for reverse traffic. This can get quite complicated the more flows you have, or if your system is multi tenant, and it has to be done per packet without degrading performance. Additionally, since flow hashing is generally the way we identify what flows are which, we have to make sure that whatever form of hashing that is being used is resistant to collisions and/or have adequate solutions for dealing with them. This is a real problem that services have to solve when sending traffic from one machine to another when transitioning over network boundaries.

Flow hashing isn’t perfect and definitely has some tradeoffs, including handling cache misses/latency, but if you constrain the number of flows your system can support and have a large enough hash space all of these problems are solvable.

Conclusion

This post was brief but something that I felt was necessary to capture as I haven’t found any other articles about it online.

Thanks for reading!

I would like to write about the main constraint I encountered trying to accomplish this and how this constraint can be circumvented around while porting this service to Lambda.

Context

The short and simple of it is that the drift detector just clones a bunch of repositories, finds all of its terraform directories with some tree searching, and then invokes terraform plan a bunch (after doing some massaging of some module paths) to figure out whether or not that service has drift.

This service is really interesting from a technical perspective because there’s some cool things it does to get this work since

• it doesn’t actually use the git cli to clone repos (it does super shallow cloning using the GHE api and only gets terraform directories)
• it rewrites custom terraform module URIs that go over HTTPs/ssh unauthed by rewriting them to strictly HTTPs ones with your personal access token, to allow you to pull modules from other internal repos

Before moving it to lambda the drift detector was running in an EKS cluster and doing perfectly fine until it eventually would consume all of the disk on the node it was scheduled on. It was only ran as a single pod in our cluster and would result in a lot of disk pressure errors, eventually resulting in a whole node being evicted from the cluster which would trigger a variety of fun alarms that were hard to diagnose.

It’s not clear exactly why there was a disk resourcing issue with the service and I even tried limiting its ability to write to a limited-capacity Kubernetes volume but it didn’t resolve the issue :). My guess is terraform commands generate artifacts that aren’t being cleaned up, possibly something related to plugins.

Either way, in the meantime the problem would be solved if we just moved the service to lambda. The other reason besides the node evictions was that it would be part of a wider initiative to reduce the number of services running in our legacy EKS infrastructure which we were offboarding. Fixing the disk leak in the first place would’ve been a great solution too, but it was proving more difficult than simply porting it to lambda. Plus, even if we fixed the disk issue we couldn’t guarantee that it simply wouldn’t re-emerge or have to write a lot of code that did manual cleaning.

Porting the drift detector to lambda

Since the drift detector was designed to run as a Go service in a docker container and use all of the binaries in that container (terraform, tfenv (for terraform versioning) and git) it made the most sense to use AWS Lambda with ECR for containerization. Previously the service had been running in an EKS cluster.

The setup for this change was pretty simple. All that was necessary was to stand up an ECR repo, set up a build process script (or whatever you prefer) for building the main service binary locally, then create a Dockerfile that pulls that locally built binary and once the image is build pushes it to ECR so that the image can execute that binary.

The only real “gotcha” here is that you have to make sure you are configuring the lambda’s network configuration properly, host machine (in this case x86_64 which is amd64, aka not ARM) and that it has enough disk space/memory and you should be good to go.

The vast majority of the source code could stay the same but with minor modifications to some of the alarms (we can use the cloudwatch lambda invocation metric as a heartbeat check as opposed to emitting one ourselves like before).

Difficulty arose whenever I tried to actually get binaries to run on the machine. The AWS documentation isn’t great about this but the only writable directory on a dockerized lambda is /tmp. This becomes an issue when you run a binary and it stores local data in the current working directory of where the binary is, or /usr/local/bin on amazonlinux with a lambda function.

It turns out that tfenv writes its current version state in $PATH/tfenv/version which isn’t writable if it’s not in /tmp so you have to make sure that your tfenv is installed with respect to /tmp.

You also can’t just move everything on container launch to /tmp and point the $PATH to /tmp–it won’t work because /tmp is an ephemeral volume that only gets mounted when the lambda container loads.

The solution for this ends up to be setting an environment variable called TFENV_CONFIG_DIR to /tmp/tfenv/ so that tfenv thinks its going to configure itself in /tmp.

Fortunately, this workaround was very simple.

Conclusion

So that’s how I ended up getting everything working on lambda as it did before despite the constraint that /tmp is the only writable volume in a lambda container.

After invoking the lambda with the repository name and setting up EventBridge to invoke it repeatedly everything works just as before. The only other thing that is notable is that EventBridge rules are limited to at most 5 target per rule, so if you have more than 5 repos you need to keep that in mind (I ended up making 1 rule per repo).

After deploying the lambda our graphs started to look a lot healthier, specifically the disk usage and the number of node evictions. The disk usage dropped to from 80% to 60% and the number of evictions to 0.

I was happy with this result. Since making this change on-call has been a lot less noisy and we’ve seen less alarms go off overall which is a result of having zero node evictions.

So on Day 24 of Advent of Code they gave us this problem where it’s basically: write a program that takes in a program as assembly and runs it against a 14 digit number (with no zeros) and finds the answer where z=0.

The assembly is pretty straight forward, there’s just ops for multiplying, adding, dividing, modulo, equality and taking in input that set x,z,y,w registers.

inp w
mul x 0
add x z
mod x 26
div z 1
add x 12
eql x w
eql x 0
mul y 0
add y 25
mul y x
add y 1
mul z y
mul y 0
add y w
add y 7
mul y x
add z y
inp w
mul x 0
add x z
mod x 26
div z 1
add x 13
eql x w
eql x 0
mul y 0
add y 25
mul y x
add y 1
mul z y
...

You can try to write a bruteforcer that does this (like I did) but it’s pretty impractical since the input w is a 14 digit number and that’s too big for all the possible values.

So what I found out (perusing reddit lol) was you can just parse the input, convert it into C code and then compile it with as many compiler flags as possible (I guess) and then put it into Ghidra, get the simplified source code and then put it into z3. You could probably skip the “put it into Ghidra step” but I did this anyway.

import z3

def to_val(inp):
    res = 0
    for d in inp:
        res = res*10 + d
    return res

def add_constraints(s, inp):
    for digit in inp:
        s.add(digit > 0)
        s.add(digit < 10)

    uVar1 = (inp[0] + 7) * (inp[0] != 12)
    uVar2 = (uVar1 % 26 + 13 != inp[1])
    uVar1 = uVar1 * (uVar2 * 25 + 1) + (inp[1] + 8) * uVar2
    uVar2 = (uVar1 % 26 + 13 != inp[2])
    uVar1 = uVar1 * (uVar2 * 25 + 1) + (inp[2] + 10) * uVar2
    uVar2 = (uVar1 % 26 - 2 != inp[3])
    uVar1 = (uVar1 / 0x1a) * (uVar2 * 0x19 + 1) + (inp[3] + 4) * uVar2
    uVar2 = (uVar1 % 26 - 10 != inp[4])
    uVar1 = (uVar1 / 0x1a) * (uVar2 * 0x19 + 1) + (inp[4] + 4) * uVar2
    uVar2 = (uVar1 % 26 + 13 != inp[5])
    uVar1 = uVar1 * (uVar2 * 25 + 1) + (inp[5] + 6) * uVar2
    uVar2 = (uVar1 % 26 - 14 != inp[6])
    uVar1 = (uVar1 / 0x1a) * (uVar2 * 0x19 + 1) + (inp[6] + 0xb) * uVar2
    uVar2 = (uVar1 % 26 - 5 != inp[7])
    uVar1 = (uVar1 / 0x1a) * (uVar2 * 25 + 1) + (inp[7] + 0xd) * uVar2
    uVar2 = (uVar1 % 26 + 15 != inp[8])
    uVar1 = uVar1 * (uVar2 * 25 + 1) + (inp[8] + 1) * uVar2
    uVar2 = (uVar1 % 26 + 15 != inp[9])
    uVar1 = uVar1 * (uVar2 * 25 + 1) + (inp[9] + 8) * uVar2;
    uVar2 = (uVar1 % 26 - 14 != inp[10])
    uVar1 = (uVar1 / 26) * (uVar2 * 25 + 1) + (inp[10] + 4) * uVar2
    uVar2 = (uVar1 % 26 + 10 != inp[11])
    uVar1 = uVar1 * (uVar2 * 25 + 1) + (inp[11] + 0xd) * uVar2
    uVar2 = (uVar1 % 26 - 14 != inp[12])
    uVar1 = (uVar1 / 26) * (uVar2 * 25 + 1) + (inp[12] + 4) * uVar2
    uVar2 = (uVar1 % 26 - 5 != inp[13]);
    res = (uVar1 / 0x1a) * (uVar2 * 25 + 1) + (inp[13] + 14) * uVar2

    s.add(res == 0)
    return to_val(inp)

def solve(inp):
    s = z3.Optimize()
    value = add_constraints(s, inp)

    s.maximize(value) # maximize, you can change this for part 2

    assert s.check() == z3.sat
    m = s.model()
    return m.eval(value)

inp = [z3.Int('x{}'.format(i)) for i in range(14)]

print(solve(inp))

def prog_to_python(prog):
    s = ""
    for p in prog:
        instr, ops = p
        if instr == "add":
            s += ops[0] + "+=" + str(ops[1]) + ";"
        elif instr == "mod":
            s += ops[0] + "%=" + str(ops[1]) + ";"
        elif instr == "div":
            s += ops[0] + "/=" + str(ops[1]) + ";"
        elif instr == "mul":
            s += ops[0] + "*=" + str(ops[1]) + ";"
        elif instr == "eql":
            s += ops[0] + "=" + ops[0] + "==" + str(ops[1]) + "?1:0;"
        elif instr == "inp":
            s += "w=bla;"
        else:
            print(instr,ops)
            assert(False)
        s += "\n"
    print(s)


prog = []
for line in lines:
    sp = line.split(" ")
    instr = sp[0]
    ops = sp[1:]
    if len(ops) > 1 and ops[1] not in ["x","y","z","w"]:
        ops[1] = int(ops[1])
    prog.append((instr,ops))

Note that the above SAT code is taken from my input which I took from Ghidra to form the z3 eqns

uVar1 = (param_1 + 7) * (uint)(param_1 != 12);
uVar2 = (uint)(uVar1 % 26 + 13 != param_2);
uVar1 = uVar1 * (uVar2 * 25 + 1) + (param_2 + 8) * uVar2;
uVar2 = (uint)(uVar1 % 26 + 13 != param_3);
uVar1 = uVar1 * (uVar2 * 25 + 1) + (param_3 + 10) * uVar2;
uVar2 = (uint)(uVar1 % 26 - 2 != param_4);
uVar1 = (uVar1 / 0x1a) * (uVar2 * 0x19 + 1) + (param_4 + 4) * uVar2;
uVar2 = (uint)(uVar1 % 26 - 10 != param_5);
uVar1 = (uVar1 / 0x1a) * (uVar2 * 0x19 + 1) + (param_5 + 4) * uVar2;
uVar2 = (uint)(uVar1 % 26 + 13 != param_6);
uVar1 = uVar1 * (uVar2 * 25 + 1) + (param_6 + 6) * uVar2;
uVar2 = (uint)(uVar1 % 26 - 14 != param_7);
uVar1 = (uVar1 / 0x1a) * (uVar2 * 0x19 + 1) + (param_7 + 0xb) * uVar2;
uVar2 = (uint)(uVar1 % 26 - 5 != param_8);
uVar1 = (uVar1 / 0x1a) * (uVar2 * 25 + 1) + (param_8 + 0xd) * uVar2;
uVar2 = (uint)(uVar1 % 26 + 15 != param_9);
uVar1 = uVar1 * (uVar2 * 25 + 1) + (param_9 + 1) * uVar2;
uVar2 = (uint)(uVar1 % 26 + 15 != param_10);
uVar1 = uVar1 * (uVar2 * 25 + 1) + (param_10 + 8) * uVar2;
uVar2 = (uint)(uVar1 % 26 - 14 != param_11);
uVar1 = (uVar1 / 26) * (uVar2 * 25 + 1) + (param_11 + 4) * uVar2;
uVar2 = (uint)(uVar1 % 26 + 10 != param_12);
uVar1 = uVar1 * (uVar2 * 25 + 1) + (param_12 + 0xd) * uVar2;
uVar2 = (uint)(uVar1 % 26 - 14 != param_13);
uVar1 = (uVar1 / 26) * (uVar2 * 25 + 1) + (param_13 + 4) * uVar2;
uVar2 = (uint)(uVar1 % 26 - 5 != param_14);
return (uVar1 / 0x1a) * (uVar2 * 25 + 1) + (param_14 + 14) * uVar2; // z value

And that’s about it. I got 79197919993985 for part 1 and 13191913571211 for part 2, best of all the solver can do this within 20-30 seconds on a 2019 mbp.

Conclusion

I think this was a really clever way of solving this problem that is totally reasonable since you may not want to reverse engineer all of that C code into figuring out what the max/min value you can give the input is. Definitely keep knowledge of z3/SAT in your arsenal. Plus, this is in theory a solution that could be done SUPER quickly if you were fast enough.

I figured I’d post about something I did today that I thought was interesting/fun. This post probably won’t make sense unless you have a basic understanding of regex.

Problem description

Let’s say you have a bunch of old urls that are hosting resources which follow an old format and point to an old s3 bucket/have a different uniform resource locator.

In the case of s3, there’s two ways of representing urls: http://s3.amazonaws.com/[bucket_name]-accountnumber/ or http://[bucket_name].s3.amazonaws.com/.

Your task is to convert these to a consistent format so when they point to a “consistent” structure–you get something like s3://bucketname/dir/dir2/resource.ext that you can convert to the proper url for locating the resource.

It sounds convoluted, I know, because it sort of is. The reason this is desirable is because if you can convert all these urls to a different format you can serve a different format that’s easier to look at/parse and is more “consistent”.

In my case I have two url formats similar to above https://coolhost.s3.us-west-2.amazonaws.com/files/bucketname/dir1/dir3/file.jpeg and https://s3-us-west-2.amazonaws.com/bucket2/dir/filename.png that I wanted to pass into a function to get a consistent resource location scheme. For the first case it would be s3://coolhost/files/bucket1/dir3/file.jpeg and for the second case s3://otherbucket/dir/filenamepng. Essentially the bucket name has to follow before all the key information.

Regexing

The way I decided to do this was to use groups and MatchAll in go’s regexp standard library.

Here’s the code which includes the regex

// Convert converts the s3 url into a digestible s3 path
func (s *S3URLConverter) Convert(url string) string {
    return s.re.ReplaceAllString(url, "s3://$2$3$4/$7")
}

// NewS3URLConverter creates a new s3 url converter
func NewS3URLConverter() S3URLConverter {
    return S3URLConverter { 
        re: regexp.MustCompile(`https:\/\/((coolhost)\.s3\.us-west-2\.amazonaws\.com(\/)|s3-us-west-2\.amazonaws\.com\/)(bucket1|bucket2)(-([0-9]+)|)\/([\w\/\-?=%.]+)`)
    })
}

Working backwards, you can view the regex here you’ll notice that both URLs get matched, and groups 2, 3, 4 and 7 can be appended to form a consistent url (hence Convert).

The idea is to just generate groups that can be appended from left to right no matter what the url is so that they form a consistent url. Groups are all the stuff that are included in the ().

A few things:

• the problem boils down to extracting the bucket’s name in both cases and appending it with its subdirectories
• the (-([0-9]+)|) is a group that just says “a hyphen appended with multiple numbers or nothing at all” (I added this to handle the bucketname-accountnumber format. You can probably make this \d+ if you want to be more correct (since this catches account numbers that start with 0)
• the group ((coolhost)\.s3\.us-west-2\.amazonaws\.com(\/)|s3-us-west-2\.amazonaws\.com\/) is the most important. note how it catches the backslash in the first case, but not the second case after the OR. this is needed so that the next group is a backslash when we match coolhost in group 2, but not in the other url format. with the coolhost url format where the host is a subdomain of s3.us-west-2.amazonaws.com the value coolhost is actually the bucketname, so it should prepend everything. in the second format it’s not and the first subdirectory is the bucketname, thus we do want to capture a slash in the other case hence why the OR branch is so large to evaluate these two these cases separately.
• the \/([\w\/\-?=%.]+) group just represents “everything after the backslash” aka the bucket’s subdirectories
• the converter is wrapped in a struct because it wraps around the regexp.Regexp object and we don’t want to re-compile it (that’s slow)
• the unit tests aren’t included in the above code, but they’re incredibly important for verifying the converter behavior and do exist/are exhaustive

And that’s about it! Hopefully this gave some people some new ideas for how to do “string conversion” because I feel like it’s rare that we actually get to “use regex” to solve problems. Keep in mind that this is a basic problem and isn’t meant to highlight anything other than the premise behind “conversion problems” which require you to use information in the string to build new strings.

You can catch me streaming at twitch.tv/cub.

The future of mcidle

Currently mcidle can actually imitate the game’s state fairly well, that is almost every relevant packet is processed (inventory, gamemode, health, position, packets, loaded entities) except block/chunk state. The reason the old version of mcidle didn’t have chunk processing originally is because I was too lazy to read the docs on how to deserialize chunks.

The major problems are

• We can’t easily implement protocol agnostic packet serialization/deserialization
• Processing chunks is extremely slow and cannot be done in Python (blocks the entire program, too much CPU usage)
• Packaging python is brittle, even with pipx you have to compile the cryptography library with mcidle which doesn’t work on a lot of Windows machines for some reason even with VS tools installed

mcidle-cpp

Remember mcidle-cpp? That post was a little misleading. I basically ended up realizing that there’s no way to design mcidle without virtual inheritance/relying purely on templates and not pull my hair out. So instead I decided to borrow an implementation of protocol agnostic packet processing like mclib does.

The basic idea is to take a packet id and then call a lambda function on a protocol object which returns the correct instance of the packet we want for that id then call serialize or deserialize functions respectively which are also implemented using virtual inheritance. Sure there are some complications since we still have to be careful about protocol versions during the serialization process, but in general it works.

So far mcidle-cpp contains a full cmake build system, tests for all the primitive data types, working client->server minecraft connection, byte buffers, all the primitive data types, zlib compression/decompression, and packet encryption/decryption. The only code that I did not write myself was the Yggdrasil auth API which is basically just boilerplate from mclib.

Results so far

Today I was able to test chunk section processing. When I controlled for how slow print statements are in C++ if you use streams and being especially careful of different optimizations and avoiding excessive copies I was able to get a 15-20x speed improvement for chunk processing (0.37 seconds c++ versus 7.1 seconds in python for 14mb of compressed chunk data or 441 chunk packets). Chunk processing means deserializing the entire chunk sections, palette information, etc. I think this is sufficient for my port to be successful. Honestly, testing is sort of a nightmare but I can definitely say the C++ version is at least one order of magnitude faster.

In fact, when I tried to measure how long it took to process chunks it had zero impact on the speed of the program (it was fast). It seems like my actual bottleneck is the rate at which the server could send packets (latency), which works out to chunks being read and processed in 0.36-0.4 seconds upon initially connecting! Blazing fast. These chunks when uncompressed and deserialized amount to ~50mb of data in memory.

The following code, written in Python, to read chunk sections drops the performance of mcidle-python to 20x slower than it’s C++ equivalent

@staticmethod
def read(stream):
    # In the latest protocol we have to read a short here (block count)
    # block_count = Short.read(stream)
    bits_per_block = UnsignedByte.read(stream)

    palette_len = VarInt.read(stream)
    if bits_per_block < 4:
        # Indirect palette
        bits_per_block = 4
    if bits_per_block > 8:
        # Direct palette, ignore
        bits_per_block = 13

    if palette_len > 0:
        while palette_len > 0:
            VarInt.read(stream)
            palette_len -= 1

    mask = (1 << bits_per_block) - 1

    data_len = VarInt.read(stream)
    data = []
    num_read = 0
    while num_read < data_len:
        data.append(UnsignedLong.read(stream))
        num_read = num_read + 1

    SECTION_HEIGHT = 16
    SECTION_WIDTH = 16

    for y in range(0, SECTION_HEIGHT):
        for z in range(0, SECTION_WIDTH):
            for x in range(0, SECTION_WIDTH):
                block_number = ((y * SECTION_HEIGHT) + z) * SECTION_WIDTH + x
                start_long = (block_number * bits_per_block) // 64
                start_offset = (block_number * bits_per_block) % 64
                end_long = ((block_number + 1) * bits_per_block - 1) // 64

                val = 0
                if start_long == end_long:
                    val = data[start_long] >> start_offset
                else:
                    end_offset = 64 - start_offset
                    val = (data[start_long] >> start_offset) | (data[end_long] << end_offset)
                val &= mask

    block_light = ByteArray.read(stream, 4096 // 2)

When I write the exact same thing in C++ with some niceties like using std::copy or memcpy to read a bunch of 64 bit longs it’s very fast. The loop over each block in the chunk section is extremely slow in Python and dominates the time (removing this loop makes the python code run in max 1 second). It boils down to how slow loops are in python due to it being an interpreted language and not being able to optimize to down to bare metal very well.

This was a major win for me and the main issue that blocked mcidle-python from working performantly: per chunk processing was bottlenecked by Python itself and not hardware.

Ultimately I was able to break the game and accomplish feats like alter my player’s stats, get infinite ammo, make my punches instantly kill and buy items from stores for free (items that take weeks if not months to grind for). I would like to say that I had no malicious intent in doing this and didn’t abuse this power at all. It was merely out of curiosity. I did notify the administrators of all the vulnerabilities I found.

How I broke it

Graal is actually not that easy to hack on if you want to do anything complicated. It’s packed with Themida and makes it basically impossible to do static analysis. If you try to attach a debugger with CheatEngine the game will automatically close. I actually unsuccessfully tried to get anything better than a teleport hack out of this game for 4 years where every year I’d make some attempt at hacking it but then fail because of how difficult it was to get a debugger attached (the TP hack is EXTREMELY easy to do). The game purposefully makes it very difficult to cheat in and does take a lot of precautions to dissuade hacking. However, there is no real anti-cheat except preventing the game from being easily reversed.

The way I got around both these security protections is by just by using VEH debugging and never really doing much static analysis. But still, this is only the beginning of an attempt at breaking this game. We need to understand how exactly the game works because otherwise it’s going to be very difficult to attack.

Graal contains many layers of abstraction than in your traditional game. That is to say, the core aspects of a server you’re connected to aren’t actually contained in the binary when you install the game but are loaded once the game is started up and sit on top of the actual engine. In Graal there’s not a lot of hard coded stuff in the binary outside of accomplishing this. All the game logic can be found in clientside scripts written in something called gscript which interact with the game server.

I believe the files are .code files in the weblevels/ directory, but I have no idea how exactly the clientside script .code file format works because I can’t find the routines that handle this in the binary–nor am I exactly certain that there is a way to access this information. Give me a few more days and I bet I could do it though.

This lead me to explore attacking Graal by getting into all the data that the client/server send back and forth to each other. Most of the testing was done on Era but I also found just as many bugs on Zodiac.

Attacking zlib

The big weakness of Graal is that it relies on zlib for compressing a lot of its data. I figured the game was doing some compression/decompression of data using it, so I attached breakpoints on Graal.inflate and Graal.deflate.

I was able to figure out that this was zlib inflate because when I looked at the first argument in eax and compared it to the first argument of zlib.inflate I saw that it was basically a form of zstream* except with ints instead of longs for some of the data types. So, all the data that gets sent to the server is in deflate (compress) and otherwise in inflate if it’s coming to us.

I was able to find data like this in the inflate output buffer, so I knew I was on the right track

which surprises me because it appears the game sends its data in plaintext more or less!

Writing a hook

To attack the game’s netcode I had to write hooks for inflate and deflate. Basically used the technique described in this article except I had to use cdecl__ which took me 3 hours to figure out. The only difference between the hooking technique I used here and my previous hooks is that it involves writing a jmp to your hook, writing back the original bytes, executing the original function, writing back the jump and returning the original function output. Before I was trying to some nightmare mode thing where I was writing raw assembly but then I couldn’t call functions that threw exceptions, had to manually allocate stack space for variables and though it worked it was too messy. To this day I can’t figure out how to build detours on Windows, so this would do.

In the end the code wasn’t super interesting, but it’s crucial for being able to interact with the input/output bytes of the buffers during compression.

int  _cdecl HookedDeflate(z_stream* stream, int flush)
{
	memcpy(origDeflate, oldBytesDeflate, SIZE);

	std::string str;
	// Too big of messages are just blobs of large data and not packet related
	if (stream->avail_in > 0 && stream->avail_in < 1048) {
		int j = 0;
		while (j < stream->avail_in && stream->avail_in > 0) {
			char c = *(char*)(stream->next_in + j);
			str += c;
			j++;
		}

		if (str.find("HitNoPk,Punch") != std::string::npos) {
			// ...
		}
		else if (str.find("removeweapon,1") != std::string::npos) {
            int idx = str.find("removeweapon,1");
            stream->next_in[idx + 13] = '0';
        }

..

In the above excerpt preventing the removal of an item from the player’s inventory in some cases is as simple as changing a 1 to a 0.

If we look at the messages we send to the server it’s pretty interesting

The game uses a weird string-like format that isn’t exactly string like because there are some binary parts, but most of the message/RPC-like protocols just involve sending data like this as strings. Eventually I stumbled upon the way items are bought, for instance it’s typically comma delimited and contains store name, action, quantity, cost and just set the price to 0 (someone didn’t do bounds checks).

I noticed that each shop seems to use a different store script, so I would guess that some shops are vulnerable while others aren’t with different scripts. It was pretty cool being able to spend 0 spar tickets on a gun instead of 3500, and quite the dopamine rush! I couldn’t believe that worked.

The other thing I tried doing was making my punches instantly kill, which was as easy as changing the damage parameter. I also tried getting infinite money by sending floating points in, but it just rounded up the values. At no point could I overflow any data though or send negative values (some scripts did do bounds checking). I’m sure there are TONS more vulnerable scripts that I didn’t get time to test, but I could only inspect a small amount of messages.

On Zodiac (the above image) I was able to make my fireballs do MAX damage very easily since the damage of a spell is controlled by the player for some weird reason. Normally they only do 251 or so at level 1.

Conclusion

That about wraps it up. I left out some of the stuff about what the client receives since I didn’t think it was super interesting, but you can’t really prevent yourself from taking damage in the game or anything. All in all I had an insane amount of fun doing this!

I believe the game tried security through obscurity since they didn’t think anyone could actually change the data that gets sent to the server.

Future work would include figuring out the .code file format and being able to not just packet edit but construct packets.

The first rule is if player 1 has a fibonacci sequence player 1 wins, otherwise player 2 wins and if there’s a tie we evalute the second rule. If need be, we evaluate the third rule if the second rule results in a tie also.

The second rule is the player with the highest # wins.

And the third rule is the player with a higher count of prime numbers wins.

The solution

Initially my solution was to simply write a function that evaluated each of the 3 rules one by one, resolving ties if needed. This solution basically breaks down into a tree and a sort of messy if/else branch. We can think of evaluating a rule as returning 3 values PLAYER1_WON, PLAYER2_WON, or TIE. We can use separate functions like count_primes(list), isFibonacci(list) and max_card(list) to avoid redundant code and simplify everything. I coded it and then my interviewer asked me to do better for arbitrary N rules.

As I was coding the worse solution that worked, I realized we could just simplify each rule into a list of them and evaluate them one by one top down much like you would in a directed-acylic graph of dependencies or a decision tree. This approach is nice because in essence we only have to write code for each rule, then evaluate them sequentially instead of writing very branchy/bug-prone and difficult to read code. With this approach we think of our rules as a pipeline and just evaluate the next rule if the current rule evaluates to a tie.

class Rule {
    virtual RULE_RESULT evaluate(vector<int>& list1, vector<int>& list2);
}

and then have each rule override accordingly using virtual inheritance

class FibonacciRule : public Rule {
    virtual RULE_RESULT evaluate(vector<int>& list1, vector<int>& list2) {
        // implement logic for determining who won the fib sequence rule
    }
}

class MaxHandrule : public Rule {
    virtual RULE_RESULT evaluate(vector<int>& list1, vector<int>& list2) {
        // implement this
    }
}

class PrimeRule : public Rule {
    virtual RULE_RESULT evaluate(vector<int>& list1, vector<int>& list2) {
        // implement this
    }
}

When we want to evaluate any number of N rules we can just pass in a list of rules and go through to evaluate a hand like so, continuing if any ties occur

RULE_RESULT evaluateLists(vector<int>& list1, vector<int>& list2, vector<Rule>& rules) {
    for (Rule& rule: rules) {
        RULE_RESULT result = rule.evaluate(list1, list2);
        if (result != TIE)
            return result;
    }
    // All the rules were ties
    return TIE;
}

So we simplified the problem, reduced the amount of work to do and improved the maintainability of our code by taking this approach to solve this problem.

Best of all, it took only 30 minutes to do. I simply went to the bttv website and monitored the XHR requests to get the URI to their API and then wrote a very simple script to dump the emotes to a dictionary for me.

import requests
import json

emoteTable = {}

def updateEmoteTable(offset, limit):
    url = "https://api.betterttv.net/3/emotes/shared/top?offset=%d&limit=%d" % (offset, limit)

    r = requests.get(url)
    
    js = r.json()

    for i in range(0, len(js)):
        emote = js[i]['emote']
        code = emote['code']
        if code not in emoteTable:
            id = emote['id']
            emoteTable[code] = id
        
        
for i in range(0, 1000):
    updateEmoteTable(i * 100, 100)
        
print(emoteTable)

The way we do something like output DOM elements for a string of text in react is pretty simple too

renderTextLine(message) {
    let tokens = message.split(' ')
    tokens = tokens.map(token => {
      if (token in emoteTable)  {
        return <span><img class="emote" src={ "https://cdn.betterttv.net/emote/" + emoteTable[token] + "/1x"}/> </span>
      }
       return (<span>{token} </span>)
    })
    return tokens
}

and then we can call renderTextLine on any message to map emote tokens to our table. Pretty straight forward.

Additionally, it’s not hard to support Youtube videos too

renderTextLine(message) {
    let tokens = message.split(' ')
    let vids = []
    tokens = tokens.map(token => {
      if (token in emoteTable)  {
        return <span><img class="emote" src={ "https://cdn.betterttv.net/emote/" + emoteTable[token] + "/3x"}/> </span>
      }
      if (token.indexOf('youtu') != -1) {
          let vid = this.parseYoutubeURL(token)
          // Delete the video, add it to the list
          if (vid) {
              vids = vids.concat((<iframe width="560" height="315" src={"https://www.youtube.com/embed/" + vid} frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>))
              return
          }
      }
       return (<span>{token} </span>)
    })
    
    // Videos go at the end
    tokens = tokens.concat(vids)
    return tokens
}

parseYoutubeURL(url) {
    var regExp = /^.*((youtu.be\/)|(v\/)|(\/u\/\w\/)|(embed\/)|(watch\?))\??v?=?([^#\&\?]*).*/;
    var match = url.match(regExp);
    return (match&&match[7].length==11)? match[7] : false;
}

And that’s about it!

Batch normalization

Batch normalization is a layer you can add to your neural network which normalizes the inputs you give it with respect to the batch it is in. It was introduced in 2015 and has been shown to reduce overfitting in networks as well as produce better generalization accuracy. The output of a BN layer with respect to input x is the normalized output of x with respect to the mean and variance of the batch sample you found it in.

Why it works

But why does it help to train a network better? Let’s think of this in terms of how a NN learns.

Whenever you feed data to your neural network from some sample x in a training set it has an underlying inherent distribution associated with it. That is, all of the samples can be thought of as coming from some distribution. For the sake of generalization, let’s say this distribution can be approximated using a Gaussian function parameterized by a mean and variance (something you will no doubt be familiar with if you have ever taken a STATS 101 class).

If you take an arbitrary sample from the distribution of your data and call this x, we can think of x as a transformation (specifically affine) on the dataset it came from with respect to the (approximate) mean/variance of this dataset to get x'.

But why is this useful? The reason is because the data we’re actually dealing with that is x in fact actually depends on the mean and variance of the sample it’s in, and this is very useful since it can be used to normalize x so that no matter which distribution it comes from the relative meaning of each sample stays the same. That is with respect to a Gaussian, when we normalize x directly at the mean it represents 0 and gets increasingly larger further away from the mean.

For instance, take two distributions of male and female heights. Both distributions will be Gaussian (this is a well known fact), but we can expect the distribution of male heights to have a higher mean average while the distribution of female heights has a lower average. Yet, if we fed this to our neural network to determine if someone is “tall” or not we should treat this data irrespective of distribution because at the mean we consider two people with the average, regardless of gender to be at the average.

We can explain why our neural network prefers learning this signal if we understand what happens when we pass data through our neural network WITHOUT applying batch norm. When you do this on some input, you can think of your data as undergoing the affine transformation (see above) before it is inserted which depends on as well as . This is a HUGE problem because this changes for each input sample x! Mathematically this just means our gradients will depend on MORE parameters during training which include the mean and variance of the dataset each sample comes from which they come from (cough covariate shift cough). This reduces generalization accuracy significantly since out in the wild since you can imagine that your training data distribution may not match your testing set distribution. We would benefit from normalizing our input data with respect to the underlying distribution of the dataset we expect to feed it since this would reduce this phenomenon (though not get rid of it entirely).

In this case our network would have a hard time learning without batch norm if we fed it the untransformed data signals because when gradients propagate it would actually have to take into account that the data undergoes this affine transformation with respect to the distribution it came from, which is the assumption we make if we assume it comes from a Gaussian distribution. To conclude, put even more simply: our network without batch norm is learning based on the un-normalized input we gave it, which undergoes a transform f(x) that DEPENDS on the distribution x came from, but it is in fact way simpler for the network to learn on x itself. However, this explanation doesn’t capture the rich nature of what is actually going on when you train a neural network which I have described above.

From the equation above we can see that we approximate the batch with respect to the entire datasaet. The reason this works is that with random shuffling we can assume that the batch should approximate the entire dataset in theory.

Conclusion

I feel like this topic is explained poorly in general. I didn’t learn it in my machine learning class, though I feel like it can be explained very simply with respect to an understanding of statistics. Hopefully this post has helped anyone still struggling with understanding why batch norm is used. I believe there are a ton of ways of looking at this topic, including the aspect of “covariate shift”, but I feel like I explained it better without using too much statistical terminology.