![\"\"](\"https://blog.mnke.org/img/0kBwxslUrU-480.jpeg\")
Recently, I designed and implemented the foundational blocks of a system at my company, inspired from my personal version of the system, with the help of a small team of coworkers. Although I laid the groundwork, most of the code that runs today is not written by me anymore and it's diverged from my personal version, especially with regards to softwarre design in a way that I find degrading to the long-term quality and maintainability of the system. I have a difficult time expressing this, partly because I'm not a very confrontational person plus I don't want to be a prick about controlling every minute detail on how the project should look like, partly because restructuring code for higher code quality doesn't align with corporate interests, and partly because I don't even know how to precisely codify my own standards.
\nI've written a lot of software in various languages, various frameworks, and various subjects. I don't claim to be an expert in everything that I write, but I believe that I've written enough software in a wide enough area that I've developed an aesthetic for good software. I feel out good software from bad software even if I'm not able to explain why, and I have strong reason to trust these instincts because following them has generally lead me to create more maintainable and well-written software over the course of thousands of hours. Yet, if I'm unable to express this private knowledge, no matter if my instincts are correct or not, I'm locked by myself in a Wittgensteinein box of unintelligibility. So I want to do some exercise to explain some of my practices and standards by starting a series of blog posts documenting software design patterns I've picked up. In doing so, I hope to free myself of long-held dogmas and gain insight into the limits of my beliefs.
\nBefore I do that, I want to issue a disclaimer that I'm suspicious of general software design principles. There is a time and a place for everything, even the most hackiest code. In fact, I despised the university course I took on software design. "Clean code"? "SOLID principles"? "Liskov substitution principle"? To hell with that. I'll be the judge of what works and what doesn't. Moreover, I'm no stranger to hacking my way around programs. If I need to do some dirty patch of an upstream library to get something working, I'll do it. Or if I need to fix an urgent and otherwise hard-to-resolve bug on production, of course I'll throw out these stupid principles.
\nYet, I find it hard to deny the recurring and overarching themes I find myself running into when designing software. I don't claim that my principles are the only way to write good software. My more modest statement is simply that, in my experience, these principles have lead me to write what I regard as better software. Or, an even more modest statement is simply that these principles have prevented my own stupidity. As I see it, this is the ultimate aim of software design.
\nTake a look at this function signature. What do you think the function does?
\nasync function gitClone(repository: string): Promise<void>;It's pretty clear to me that it will do something along the lines of "spawn a git subprocess and clone a repository". Given the void return, a consumer wouldn't have access to whether the subprocess ran successfully, so I might expect that the function will throw an error if something went wrong in the subprocess. Let's see what the function actually does.
\nasync function gitClone(repository: string): Promise<void> {\n const sql = new SqlConnection(process.env.DATABASE_URL);\n const res = await exec(\"git\", [\"clone\", repository]);\n await sql.query(`\n insert into git_clones (repository, timestamp, exit_code)\n values (${repository}, now(), ${res.exitCode});\n `);\n if (res.exitCode !== 0) {\n throw new Error(\"git clone failed\");\n }\n}...oh.
\nWell, we were partially right, but it seems to do a lot more than what we expected and it took us having to open up and read the function to figure this out. I guess... that's fine? I just have to be cautious about every function that this author wrote though. Seeing this, who knows what gitCheckout is doing?
async function gitCheckout(branch: string): Promise<void> {\n const sql = new SqlConnection(process.env.DATABASE_URL);\n const res = await exec(\"git\", [\"checkout\", branch]);\n await sql.query(\"truncate table git_clones;\");\n if (res.exitCode !== 0) {\n throw new Error(\"git clone failed\");\n }\n}Okay, what the f**k is this? Sure, maybe there is legitimately a good reason why we're truncating git_clones every time we gitCheckout. Maybe there was some incident that necessitated this. But as a consumer of these functions, I've lost all trust! For all I know, calling a function turns the lights off in the next room or enters the nuclear launch codes! I have no way to reason about what anything in the code does anymore without going and opening through every single function and seeing exactly what they do. On a small scale, fine, I can read gitClone and gitCheckout, but on the scale of even a medium-sized codebase, reading through every single line of code is simply not realistic.
I know there will be some who would respond to the tune of "oh, but you can just get an LLM to read the lines for you and figure out what it does".
\nBut why would you even need to rely on that in the first place? The synthesizing power of LLMs is not an excuse to write shitty code. It's like poking yourself with a knife and expecting your body to heal the wound.
\nTaking a step back, let's be charitable and assume we actually do need to interface with the DB for the git operations for whatever reason. My current frustration isn't that we're doing DB queries alongside git operations, it's that we're secretly doing DB queries alongside git operations. There's no way to tell what gitClone and gitCheckout is doing without going in and auditing the code closely because it severely violated our expectations. Poor Alice could call gitCheckout on her first day of her internship, not understanding its full consequences and cause another incident!
Let's propose a minimal redesign.
\nasync function gitClone(\n repository: string,\n databaseUrl: string,\n): Promise<void>;Instead of reading the database URL from within the function by reaching for environment variables, it's now required as a parameter. If consumers so wish, they can read the environment variable themselves and pass it into this function.
\nIt's helpful to think that these simply are the function's true parameters and all we've done is do a better job of surfacing them**.** After all, it's getting the database URL from somewhere. It was just previously hiding it. Sure, I still wouldn't touch this function with a ten-foot pole, but at least I can come to an informed judgement to refrain just by reading the function signature.
\nSo, again, would you call this function?
\nasync function gitClone(\n repository: string,\n databaseUrl: string,\n): Promise<void>;The trap of having implicit global dependencies that can be promiscuously and magically materialized from anywhere in the code – "from thin air" – is one that I see even well-rounded programmers fall into. The example I gave was admittedly contrived but it serves to show the general pattern and the harmful extremes of its nature. In practice, it takes mindfulness about the dependency graph that the program forms as well as careful judgement on the desired topology of the dependency graph to avoid this.
\nNote that it may, in fact, be desirable to have an implicit global dependency if it should be treated as a primitive to all entities in the relevant domain. For instance, it may be desirable for all entities to have access to a deltaTime() function within a game engine framework so each entity can do time-based physics calculations. When we allow for a global dependency, we're allowing a "depends-on" edge from every single module in the program to that dependency in the dependency graph. If these edges are not obvious, it causes confusion and leads to a broken domain model. Furthermore, stripping these edges or making them explicit is tedious, albeit fairly mechanical.
I feel the need to reiterate my stance on software design before closing off. Materializing hidden dependencies as I described in this post is neither necessary nor sufficient for good software. I'm skeptical "good software" is even well-defined as an objective concept. I simply want to share patterns I've found and I ask that the reader is mature enough to come to their own conclusions, both for this post and upcoming software design posts.
\n", "date_published": "2026-06-06T19:23:40Z" } , { "id": "https://blog.mnke.org/reflections-on-ai-coding/", "url": "https://blog.mnke.org/reflections-on-ai-coding/", "title": "Reflections on AI Coding", "content_html": "A few years ago, OpenAI released ChatGPT, bringing mainstream attention to the otherwise still niche area of LLM research and development. The world was captivated. Investors flocked to funding LLMs and we've since seen an explosion in the presence of LLMs that's rippled through the world: creating new markets, driving research, and sparking discussion on the ethics of how LLMs acquire their knowledge, on the consequences of LLMs on the economy and even its consequences on human dignity. Whatever your opinions on LLMs are, there's no dispute it's permanently left its mark in the cultural zeitgeist.
\nWhat I want to focus on in this post is the use of LLMs as it relates to software engineering and my personal experiences with using it. For the rest of the post, I'll be reluctantly referring to LLMs as AI, since that's the more common name that society's given it, despite LLMs being only a small subset of what AI traditionally meant in computer science.
\nSince its launch a few years ago, providers have been trying to push ChatGPT and other AI models to help with programming. In theory, AI models should do well with programming. After all, LLM stands for large language model, programming languages are a kind of language, even if it's artificially constructed and highly structured compared to natural language. People began "vibe coding", a term coined by Andrej Karpathy referring to a practice where code is created by prompting AI in natural language and guiding it to accomplish your needs.
\nBefore the term was even coined, I used to refer to programming sessions where I'm just vibing and coding as "vibe coding". For example, I might've had a sudden urge to write an HTTP server in C to get more familiarity with some of the Linux kernel APIs allow nginx to be so efficient. When doing that, I would be vibing by going with the flow of exploration, and I would be coding. I would be vibe coding! Now this term has been co-opted by the AI people. Gah!
\nWhen the term was coined and it initially gained its popularity, I treated it as a joke. "Seriously? We're throwing out rigorous computer science and software design in favor of slop generated from natural language?" was my sentiment. I thought it was utterly insane. Smart people have created a way to interface with computers in a precise way via something called programming languages and compilers. By adding another compiler from natural language to programming languages, we're adding another layer of indirection and losing precision.
\nThat said, I wasn't dogmatically against AI. In fact, I found myself using it when exploring new topics. For example, when I was setting up a streaming infrastructure, I consulted AI to figure out what technologies were out there, some of their high level summaries, and interfacing capabilities. Similarly, I'd ask AI about starting points when researching new topics, like asking for references about programming language design.
\nFor a while, AI was a useful tool to me that aided in software engineering and made it more enjoyable. Purists may loathe this, but I prefer asking AI about some weird syntax error in a programming language I'm trying to learn rather than digging up some StackOverflow post from 12 years ago with answers that are outdated. During this time, my use of AI left the fun part of programming intact for me.
\nFast forward to some time in 2025. Throughout the software industry, there had been wide scale adoption of AI in professional development, particularly in agentic coding. I treat agentic coding as a branch of vibe coding: rather than needing to relay information between your system and the AI (e.g. by copying around snippets of code), you install a program that interfaces with an AI and is able to read and write files and run commands on your system directly autonomously (with safeguards of course).
\nOnly as I write out the description of agentic coding do I remember how crazy it sounded to me initially. An untrusted LLM with a dubious understanding of security running commands on your system? Sure, there are guardrails, but you have to trust that the guardrails even work and you have to be diligent and knowledgeable enough to audit what the agent is doing. I imagined that the only people who would use this technology would be people who didn't meet those requirements in the first place!
\nI was skeptical. I didn't trust that AI could write high quality code. I refused to use AI in any capacity for coding (apart from what was mentioned before, as a fancy search engine with a chat interface). In fact, I had worked with people that did vibe code and had seen first-hand how terrible the code was.
\nThat was until around the start of 2026. I had coworkers who were trying to push the limits of AI in their daily workflow. Moreover, there had been a cultural shift at my company to be more AI-driven.
\nAgain, I wasn't dogmatically anti-AI. My chief concern was the quality of code that AI produces. Beyond that, I didn't have a principled reason against using it. So when coworkers started asking me to review AI generated code and seeing a general trend in improvement in the code quality since I last remembered, I decided to give it a try. My company gave me access to Claude so I installed Claude Code and started playing around with it.
\nAt first, I started using it for only small, focused work like refactoring or writing tests or asking questions to the agent like tracing a variable up a long and confusing call stack or searching how something was done idiomatically in a codebase. I found this to be remarkably successful and as of writing, I still feel positively about this use of agentic coding. It removed some of the frustrating parts of programming, especially on a large and unfamiliar codebase (I was still relatively new to my company at the time).
\nI kept experimenting with using AI in my daily programming and pushing the limits of tasks I deferred to it, even outside of work. At first, I was impressed. When I glanced through the code it generated, it believably looked like decent quality and to be fair, it really did generate passable code when working with popular tooling.
\nAt some point, I was writing maybe 80% of my code at work with AI agents. I was measurably writing a larger quantity of code and in an industry with endlessly increasing demands, it felt difficult to refuse a magic tool that boosted productivity. It was steroids for the software industry: even if it's not explicitly forced upon anyone, people will feel like they're falling behind if they don't use it. Damned if you do, damned if you don't.
\nIs this truly a fair analogy to draw though? After all, steroids have known adverse health effects long-term and is generally frowned upon. We should investigate whether this comparison is justified.
\nI was asked recently to patch a fork of a 10+ year old C++ library we maintain. Even getting the library to run is a rite of passage, needing to deal with an antiquated CMake build that only works on very specific configurations, and I was one of the few that had passed the ritual, had sufficient C++ knowledge, and had done work on it recently so I was first in line for duty.
\nI know, I'm so unique. One on hand, writing C++ for my own projects is... not unpleasant, to be honest. I have only a few minor C++ projects here and there so I'm definitely not an expert, but in the times I've used it, my complaints were mainly around C++ being opaque with respect to the underlying machine code it produced compared to C, where every detail is explicit. This mattered to me because I was writing a game hacking library where I wanted precision down to the bytes in assembly instructions.
\nWhat I hate is how odd some of the C++ language/stdlib design decisions are and how easy it is to abuse C++ features that make it absolutely alien to anyone else reading it. It feels like everyone creates their own version of C++ and the knowledge transfer between codebases is thus heavily limited.
\nI had a great time when I last worked on the library. Well, it was a struggle while working through the problem, but it was satisfying to solve it, like any good problem.
\nThe program was segfaulting when sending it certain data. I was working on macOS laptop and I had very limited familiarity with the LLVM toolchain so I couldn't even get a debugger attached to the process at first, which made debugging painful, especially since this was dealing with a segfault. I got desperate enough to start placing print statements at every line along the critical path of failure. Eventually, I wisened up and just built the project on my Linux desktop with the GNU toolchain I was familiar with and got a debugger attached.
\nThe segfault was occurring within some STL function that accessed invalid memory. That was odd. I should surely be able to trust the STL to competently handle its own memory, right?
\nWeirdly, I also noticed the program was segfaulting at a different line on my desktop. I triple checked on my laptop (still without a debugger) to make sure this was the case.
\nstd::cout << \"OK GOT HERE 2 REAL (ACTUAL) FR\" << std::endl;\n// crash\nstd::cout << \"DO NOT SEE!!!\" << std::endl;This was informative. The fact that there was a discrepancy between the two platforms hinted that the underlying issue was likely due to some memory corruption introduced earlier in the program rather than the offending lines themselves initiating the invalid memory access.
\nIndeed, I found in that in a function earlier in the call stack, there was some odd-looking memory allocation patterns they did with their inhouse dynamic array implementation (why even use an inhouse implementation if we end up using the STL anyways...?). In any case, it looked like an object wasn't properly being initialized with their inhouse implementation and sure enough, adding a single line to initialize it properly fixed the segfaults on both platforms.
\nWhat a joy! Each step in the process was soaked with blood and tears, but I felt so alive when working on the problem. I was proud of myself for being able to use these subtle hints to deduce the underlying problem and it culminated in a sweet, sweet victory.
\nAt this point, you might be wondering whether you're reading the right post. This is a post about AI, not about some esoteric C++ library and its internal mechanisms. So why am I talking about this? I promise I'll answer this soon. First, let's talk about the more recent patch I did for this library.
\nI was told that this was an important patch and that a deal with a customer was sitting on the line for this. I was reluctant to give estimates because the nature of this work is inherently unpredictable and I was in the middle of other work at the time. The importance of the patch was emphasized when I was invited to a channel "urgent-XYZ" and prompted to fix it though.
\nThe thing is, this was an issue in the application code in a domain I had absolutely no knowledge of. Realistically, I'd need to take at least a few dedicated days to understand the relevant parts of the domain well enough to give a well-informed assessment of the situation. So I turned to AI.
\nI turned on the most advanced and expensive model and asked it to help work through the problem. Understanding the domain could take too much time and it wasn't clear what the benefits would be if this problem could be solved with a blackbox approach. I fed the agent more data, suggesting it to look through potentially relevant files, relaying information from domain experts to it, praying that it'd magically fix the problem.
\nIt did.
\nThe executives were happy. I was praised for being able to get the fix in. Yet I felt nothing. What was my role in this process? What even was the underlying problem? What did I even do?
\nAt this point, I was a glorified CI runner. All I really did was feed a magic text box some funny-sounding words and run builds to see if the magic text box made the code work and reported back with a green checkmark when it did.
\nThis particular instance made me realize the effects AI was having on me as a programmer. I had a conversation with a coworker a while back about the use of AI in programming and I had agreed with his sentiment that AI takes the joy out of programming, but now I felt it at a visceral level. Programming with AI is just... not fun.
\nUnmistakably, I am more productive with it. There's no way I could have made the patch in the time I did without the use of AI. There's no way I can push out the volume of changes I do without the use of AI. But I've realized using to the extent that I feel like I have to to be competitive as a programmer has taken the joy out of programming.
\nIn my mind, there's little doubt that AI is revolutionizing the software industry and it will continue to do so, provided the economics of running LLMs can stabilize.
\nJunior engineers equipped with AI tooling are transformed to a human-machine hybrid capable of pumping out medium quality changes at a faster rate than more experienced programmers without AI.
\nEven if this isn't completely accurate right now, I believe advancements in LLMs will make this generally true, and the quality of LLM-generated code will increase. It likely won't replace the artistry and craftsmanship of hand-crafted code, but as far as corporations see, I believe it will produce code at a quality indistinguishable from human code.
\nSo where does that leave us?
\n...I don't know. I'm trying to answer this question myself. I enjoy puzzles. I enjoy problem-solving. I enjoy meticulously crafting line-by-line a cohesive answer to a problem in the same way I enjoy crafting an elegant mathematical proof. I enjoy programming.
\nThe way that the software industry seems to be heading is increasing demands on productivity in a way that I can't keep up without the use of AI, yet using AI in the ways to keep up in the ways I need to is fundamentally at odds with what made working in the software industry appealing to begin with though.
\nThis is a rather bleak outlook.
\nI suppose I should just embrace this. This predicament is surely not unique to the software industry, and it may not be the first time a predicament of this nature has happened even within the software industry. Maybe in the late twentieth century, I would have been lamenting about compilers taking the joy out of those who enjoyed writing machine code.
\nOnly time will tell if we're able to preserve our dignity.
\n", "date_published": "2026-05-03T07:33:15Z" } , { "id": "https://blog.mnke.org/taking-the-parser-combinator-pill/", "url": "https://blog.mnke.org/taking-the-parser-combinator-pill/", "title": "Taking The Parser Combinator Pill", "content_html": "I recently discovered the elegance and expressive power of parsing using parser combinators recently and I immediately fell in love with it, wanting to use it to solve new problems using it.
\nIn fact, I used parser combinators to parse a mini programming language, Core, following in close step to "Implementing functional languages: a tutorial" by Simon Peyton Jones, one of the founders of Haskell. I also used parser combinators to implement a parser for the UCI protocol in NodeJS while writing an adapter for another project. And then I implemented the same parser in Gleam, which I'll link to once I make the repo public.
\nIt's just so satisfying to break down a seemingly complicated language specification down to atoms in a disciplined manner and reconstruct the full language through the power of parser combinators.
\nBut I'm getting ahead of myself.
\nWe'll talk a bit about the theory behind parser combinators and build up to writing a fully-functional parser combinator in Haskell to parse JSON.
\nDon't worry if you don't know much Haskell. I'll try to keep things simple and explain things as we go. However, I expect readers to be a bit more mature than my previous post on monads. There will be less hand-holding throughout this post for sake of brevity, and I expect readers to be able to piece things together themselves.
\n...This joke doesn't really work here, does it?
\nSo what even are parser combinators? Well, in general, a combinator in the context of programming is just something that... combines things. So a parser combinator is just something that combines parsers.
\nThe idea is that if we can parse, say, the literal string "true", then we should be able to parse multiple "true"s. And if we can parse a comma and whitespaces, then we should also be able to parse "true" separated by commas and whitespaces. The way we do this is by combining parsers. Parser combinators!
This is really vague if you're new to parser combinators, so we'll dive into code examples immediately. Just keep the idea of being able to "combine" parsers in the back of your head.
\nLet's get you set up with a comfortable Haskell workflow so you're able to follow along with these examples.
\nFirst, you need to install Haskell.
\nOnce that's done, create a new project and create a file called Main.hs with the following contents:
module Main where\n\nmain :: IO ()\nmain = do\n putStrLn \"Hello, World!\"In a terminal, navigate to the directory containing Main.hs and run ghci, which will spawn a new interactive environment to run Haskell code.
Run :l Main.hs to load the file, and then run main. You should see Hello, World! printed to the terminal.
Now, let's add another function:
\nmodule Main where\n...\ndouble :: Int -> Int\ndouble x = x * 2In the existing GHCi session, you can just type :r to reload the file. Now you can actually call double 5 to execute the code you just wrote.
This is my preferred Haskell development workflow, where you make some changes to a file, reload, and then immediately start executing code interactively.
\nThe basic building block of parser combinators – the thing that's being combined – is, of course, the parser. Let's define a parser.
\ntype Parser a = String -> Maybe (a, String)Equivalently, in TypeScript,
\ntype Parser<A> = (x: string) => [A, string] | nullA Parser a is something that accepts an input (in our case, we are restricting ourselves to string inputs) and produces an a along with the rest of the input if successful, or it produces nothing if it fails.
So we might define a function to create a parser to parse a single character like so:
\nchar :: Char -> Parser Char\nchar c = f\n where\n f (x:xs)\n | x == c = Just (c, xs)\n | otherwise = Nothing\n f [] = NothingIn English, char is a function that accepts a character c and returns a function that checks if the first character is c. If it is, then return the successfully parsed c and the rest of the string. Otherwise, if c didn't match or the input was empty, then return an error result.
If you reload GHCi with :r and run char 'a' "abcd", you should get Just ('a', "bcd"). We've successfully created a parser to parse a character!
Now, we'll set our sights on parsing a literal string.
\nAn obvious way of parsing strings might be to write something like this:
\nmodule Main where\n\nimport Data.List\n...\nstr :: String -> Parser String\nstr s = f\n where\n f input\n | s `isPrefixOf` input = Just (s, drop (length s) input)\n | otherwise = NothingYou can verify that it works by running str "foo" "foo bar baz" and getting Just ("foo", " bar baz").
However, this is not a very combinatorial approach. If we're able to parse individual characters, surely we should be able to combine a parser that parses 'f', a parser that parses 'o', and a parser that parses 'o' to create a parser that parses "foo", right?
Indeed, we can! But before we do that, we need to upgrade our parser slightly and prove to Haskell that it's a monad. In doing this, we unlock a ton of features for our Parser that we'll use anyway, so we're not preemptively overengineering.
If you look at the reference for Monads on Haskell, you'll find:
class Applicative m => Monad (m :: Type -> Type) where\n ...This just tells us that in order to be a Monad, our parser has to be an Applicative first. Okay, clicking into Applicative, we see this:
class Functor f => Applicative (f :: Type -> Type) where\n ...Alright, so our parser has to be a Functor first. Looking at the reference for Functor, we have to prove that our parser supports the fmap, or functor map operation.
To do this, we need to define our parser using the newtype syntax. I won't get into why, but using newtype to prove typeclasses is very common in Haskell.
newtype Parser a = Parser\n { runParser :: String -> Maybe (a, String)\n }This is essentially the same definition as before, but we just have to put Parser in front of the parsing function. So our char function just has to be changed to this:
char :: Char -> Parser Char\nchar c = Parser f\n where\n f (x:xs)\n | x == c = Just (c, xs)\n | otherwise = Nothing\n f [] = NothingAdditionally, if you want to run the parser now, you have to call runParser (char 'a') "abcd".
Alright, now let's prove Parser is a Functor.
instance Functor Parser where\n fmap f (Parser p) = Parser $ \\input -> do\n (a, input') <- p input\n return (f a, input')fmap is a function that maps the "inner" value of Parser. What that amounts to is: we run a Parser a on the input to get an (a, input'), where input' is the rest of the input after parsing the a, and then we simply transform a into f a before returning it.
Let's see it in action:
\nghci> runParser (char 'a') \"abcd\"\nJust ('a',\"bcd\")This is expected. But what if we wanted to get the character code of the character that we parsed instead? Try this:
\nghci> import Data.Char (ord)\nghci> runParser (fmap ord (char 'a')) \"abcd\"\nJust (97,\"bcd\")We transformed the "inner" value 'a' into its character code using ord by using fmap. Another way to write this would be:
ghci> runParser (ord <$> char 'a') \"abcd\"\nJust (97,\"bcd\")Here, the <$> acts like the regular $ function in Haskell, except the function gets to "inject" itself into a functor. It's really just an infix version of fmap.
If we really want our Parser to behave properly as a Functor, there are functor laws we should also prove (this proof takes place outside of code). I'll leave proving the functor laws as an exercise to the reader.
We're finished with one-third of our side quest. Now we just have to prove Parser is an Applicative. According to the reference, we need to implement two functions: pure and (<*> or liftA2).
Let's start by implementing pure.
instance Applicative Parser where\n pure a = Parser $ \\input -> Just (a, input)There's basically no other way we could implement pure. After all, pure is supposed to be a function that accepts an a and returns a Parser a. The only sane choice is to create a Parser that always returns a, regardless of the input.
Moving on to liftA2,
instance Applicative Parser where\n ...\n liftA2 g (Parser pa) (Parser pb) = Parser $ \\input -> do\n (a, input') <- pa input\n (b, input'') <- pb input'\n return (g a b, input'')liftA2 in general allows us to apply functions to values within functors. For example, if we had a Maybe a and a Maybe b and a function a -> b -> c, I should be able to unwrap the a and b out of the Maybes and then run a -> b -> c to get a Maybe c right? Or if one of the Maybes was a Nothing, then we should end up returning Nothing. The same logic applies to liftA2 on Parsers.
To verify it works, let's check it in GHCi:
\nghci> runParser (liftA2 (\\x y -> [x,y]) (char 'a') (char 'b')) \"abcd\"\nJust (\"ab\",\"cd\")What's happening is that liftA2 (\\x y -> [x, y]) is "lifting" the lambda expression into the world of Parsers so that we can apply it to values inside char 'a' and char 'b'.
Commonly, you'll also see this being written as
\nrunParser ((\\x y -> [x, y]) <$> (char 'a') <*> (char 'b')) \"abcd\"This is because it allows you to chain more arguments easily. If you also wanted to parse a 'c', then you could just just add to the expression:
ghci> runParser ((\\x y z -> [x, y, z]) <$> (char 'a') <*> (char 'b') <*> (char 'c')) \"abcd\"\nJust (\"abc\",\"d\")Final stretch. Monads only require the >>= operation, or bind to be implemented.
instance Monad Parser where\n (Parser pa) >>= f = Parser $ \\input -> do\n (a, input') <- pa input\n let (Parser pb) = f a\n pb input'We can more or less play "type tetris" to implement this. We have that pa :: String -> Maybe (a, String), f :: a -> Parser b and we need to return a Parser b. The obvious choice is to run the pa on the input to get an a, and then get a Parser pb from f a, and then return pb on input', the input after parsing the a.
Now we can define a parser to parse the string "true"!
true :: Parser Bool\ntrue = do\n _ <- char 't'\n _ <- char 'r'\n _ <- char 'u'\n _ <- char 'e'\n return TrueOr you can even write it like this:
\ntrue :: Parser Bool\ntrue =\n char 't' >>\n char 'r' >>\n char 'u' >>\n char 'e' >>\n return TrueLet's check that it works:
\nghci> runParser true \"true false\"\nJust (True,\" false\")Pretty neat, huh? And don't worry if this is still too cumbersome. We're on our way to creating a function that accepts any string and creates a parser that parses that string. By then, we should be able to define true = str "true" >> return True.
Similar to the functor laws, we should check that our monad satisfies the monad laws, which I'll leave as an exercise to the reader.
\nWhile we're at it, let's implement Alternative on our Parser, which will allow us to do things like pa <|> pb, which is a parser that tries to parse pa first, and if it fails, then parse pb.
instance Alternative Parser where\n empty = Parser (const Nothing)\n (Parser p1) <|> (Parser p2) = Parser $ \\input ->\n p1 input <|> p2 inputThis one's easy. empty needs to give a Parser a. Well, how can we even create a Parser a? We can't. So we have to create a Parser that always fails.
Since Maybe is itself an Alternative, we can just run each parser on the input and then use the implementation for <|> on Maybe to implement <|>.
Let's test it out by writing a parser for False and then trying to parse "true" or "false".
false :: Parser Bool\nfalse =\n char 'f' >>\n char 'a' >>\n char 'l' >>\n char 's' >>\n char 'e' >>\n return FalseLet's check that it works in GHCi:
\nghci> runParser (true <|> false) \"false\"\nJust (False,\"\")\nghci> runParser (true <|> false) \"true\"\nJust (True,\"\")Let's finally re-define str now that we have the power of the monad. The idea is that, since String is literally type-aliased as a [Char], we can map each Char into a Parser Char using our char function, and then do a fold. Roughly,
-- str \"true\"\n-- Turns into a [Parser Char]\n-- [char 't', char 'r', 'char u', 'char e']\n-- Turns into a Parser [Char]\n-- char 't' `(liftA2 (:))` (char 'r' `(liftA2 (:))` ...)We can translate that into:
\nstr :: String -> Parser String\nstr s = foldr (liftA2 (:)) (pure \"\") $ map char sBut it turns out this sort of operation is common enough that there are shortcuts to writing them.
\nghci> :t traverse\ntraverse\n :: (Traversable t, Applicative f) => (a -> f b) -> t a -> f (t b)\nghci> :t mapM\nmapM :: (Traversable t, Monad m) => (a -> m b) -> t a -> m (t b)The general pattern is that we're able to turn an [Char] into an [Parser Char], but what we really wanted was an Parser [Char]! traverse or mapM allows us to "turn the applicative/monad inside out of a list" as long as we know how to turn each element of the list into a monad/applicative. traverse is more general because it only requires that f be an Applicative, which Monad must satisfy already. In fact mapM is literally defined to be traverse. So let's use it.
str :: String -> Parser String\nstr s = traverse char sWith an eta reduction, we can get:
\nstr :: String -> Parser String\nstr = traverse charYup, this is our entire function. If you're newer to Haskell and functional programming, you might accuse me of making this unreadable compared to what we had before, either the version with foldr or the version with isPrefixOf, but as you get to understand Haskell, you'll realize this is a much more natural and elegant solution compared to the previous iterations. Again, the key to traverse is recognizing that we're just "turning the applicative inside out of a list" (more generally, any Traversable).
Let's rewrite true and false with our upgraded str function:
true :: Parser Bool\ntrue = str \"true\" >> return True\n\nfalse :: Parser Bool\nfalse = str \"false\" >> return FalseAnd we may as well write a parser for a Bool in general:
bool :: Parser Bool\nbool = true <|> falseI mentioned before that we should be able to take existing parsers and "combine" them to parse bigger things, right? Well, we actually already have one for free: many. We get to use it for free because we proved Parser was an Alternative!
Try it out, it works exactly like you think:
\nghci> runParser (many (str \"foo\")) \"foofoofoobar\"\nJust ([\"foo\",\"foo\",\"foo\"],\"bar\")\nghci> runParser (many (str \"foo\")) \"\"\nJust ([],\"\")I want to remind the reader that the end goal is to parse JSON. So let's define combinators to help us do that.
\nLet's start by defining many1, which is exactly like many, except it requires parsing at least one item.
many1 :: Parser a -> Parser [a]\nmany1 pa = do\n a <- pa\n as <- many pa\n return (a:as)Put more succinctly,
\nmany1 :: Parser a -> Parser [a]\nmany1 pa = (:) <$> pa <*> many pa(If you feel uncomfortable with notation, don't use it! Stick with what works for you.)
\nsepBy will take two parsers and give us a parser that alternates between using each parser. We should end up with something like this:
ghci> runParser (bool `sepBy` (str \", \")) \"true, false, true, true foo bar\"\nJust ([True, False, True, True], \" foo bar\")Let's try this:
\nsepBy :: Parser a -> Parser b -> Parser [a]\nsepBy pa sep = do\n a <- pa\n as <- many (sep >> pa)\n return (a:as)We first have to parse using pa, and then we parse a bunch of seps, ignore the result (that's what >> is doing), and parse using pa.
Giving it a go in GHCi:
\nghci> runParser (bool `sepBy` (str \", \")) \"true, false, true\"\nJust ([True,False,True],\"\")\nghci> runParser (bool `sepBy` (str \", \")) \"\"\nNothingAh, but we should be parsing an empty string as [], not failing. Let's make it so that if we failed to parse, then we just return an [], using our <|> operator.
sepBy :: Parser a -> Parser b -> Parser [a]\nsepBy pa sep =\n ( do\n a <- pa\n as <- many (sep >> pa)\n return (a:as)\n ) <|> return []Now it's fixed!
\nghci> runParser (bool `sepBy` (str \", \")) \"\"\nJust ([],\"\")Written a bit more succinctly:
\nsepBy :: Parser a -> Parser b -> Parser [a]\nsepBy pa sep = ((:) <$> pa <*> many (sep >> pa)) <|> return [](Again, don't use this if you're not comfortable with this.)
\nWe want a combinator like this:
\nghci> runParser (between (char '[') (char ']') bools) \"[true, false, true]\"\nJust ([True,False,True],\"\")This one's pretty simple and I won't even bother explaining it.
\nbetween :: Parser a -> Parser b -> Parser c -> Parser c\nbetween pl pr pc = do\n _ <- pl\n c <- pc\n _ <- pr\n return cPut more succinctly,
\nbetween :: Parser a -> Parser b -> Parser c -> Parser c\nbetween pl pr pc = pl *> pc <* prWe've built all the ingredients we need to parse JSON. To make life a bit easier, let's first build a couple of common parsers we'll be using. These are also parsers that you'll likely take along with you to parse other things.
\nsatisfy :: (Char -> Bool) -> Parser Char\nsatisfy pr = Parser f\n where\n f (x:xs)\n | pr x = Just (x, xs)\n | otherwise = Nothing\n f [] = Nothing\n\noneOf :: String -> Parser Char\noneOf s = satisfy (`elem` s)\n\nnoneOf :: String -> Parser Char\nnoneOf s = satisfy (not . (`elem` s))\n\ndigit :: Parser Char\ndigit = oneOf \"0123456789\"\n\nwhitespaces :: Parser ()\nwhitespaces = many (oneOf \"\\r\\n\\t \") >> return ()\n\ncommaSep :: Parser a -> Parser [a]\ncommaSep pa = pa `sepBy` (char ',' >> whitespaces)\n\ndoubleQuoted :: Parser a -> Parser a\ndoubleQuoted = between (char '\"') (char '\"')\n\nbrackets :: Parser a -> Parser a\nbrackets = between (char '[') (char ']')\n\ncurlies :: Parser a -> Parser a\ncurlies = between (char '{') (char '}')Let's define an AST for JSON to store our data into:
\ndata JSON\n = JSONNull\n | JSONBool Bool\n | JSONNumber Float\n | JSONString String\n | JSONArray [JSON]\n | JSONObject [(String, JSON)]\n deriving (Eq, Show)Great, let's knock out the easy ones right off the bat:
\njsonNull :: Parser JSON\njsonNull = JSONNull <$ str \"null\"\n\njsonBool :: Parser JSON\njsonBool = JSONBool <$> ((True <$ str \"true\") <|> (False <$ str \"false\"))Again, don't be scared of the notation. We already discussed how to parse Bools, so if you're uncomfortable with this, fall back to using that.
Parsing numbers is a little bit tougher because we have to account for the fact that numbers can be written with or without the .. We can easily split it into two cases though:
jsonNumber :: Parser JSON\njsonNumber = JSONNumber <$> (float <|> int)\n where\n float = read <$>\n ((++) <$> many1 digit <*> ((:) <$> char '.' <*> many1 digit))\n int = read <$> many1 digitIt's a little bit ugly because we end up using read to turn the String into a Float, but it works:
ghci> runParser jsonNumber \"1.23\"\nJust (JSONNumber 1.23,\"\")\nghci> runParser jsonNumber \"111\"\nJust (JSONNumber 111.0,\"\")Now, JSON strings are a bit harder to parse because they might contain escape characters like \\" or \\n. We're going to keep things simple and ignore escape characters.
jsonCapableString :: Parser String\njsonCapableString = doubleQuoted (many (noneOf \"\\\"\"))\n\njsonString :: Parser JSON\njsonString = JSONString <$> jsonCapableStringWe have a separate jsonCapableString because we need a Parser String to parse keys in JSON object the same way we parse regular strings.
For the recursive definitions, let's first write a parser for any JSON value:
json :: Parser JSON\njson =\n jsonNull <|>\n jsonBool <|>\n jsonNumber <|>\n jsonStringA JSONArray is just a bunch of JSONs, separated by commas.
jsonArray :: Parser JSON\njsonArray = JSONArray <$> brackets (commaSep json)Let's upgrade our main parser:
\njson :: Parser JSON\njson =\n jsonNull <|>\n jsonBool <|>\n jsonNumber <|>\n jsonString <|>\n jsonArrayTesting it out in GHCi, it seems to work:
\nghci> runParser json \"[true, false, null, 1, 1.1, \\\"foo\\\", []]\"\nJust (JSONArray [JSONBool True,JSONBool False,JSONNull,JSONNumber 1.0,JSONNumber 1.1,JSONString \"foo\",JSONArray []],\"\")Finally, let's implement our parser for the JSONObject.
jsonObject :: Parser JSON\njsonObject = JSONObject <$> curlies (commaSep entry)\n where\n entry = do\n key <- jsonCapableString\n whitespaces >> char ':' >> whitespaces >> return ()\n value <- json\n return (key, value)Let's upgrade our main parser for the last time:
\njson :: Parser JSON\njson =\n jsonNull <|>\n jsonBool <|>\n jsonNumber <|>\n jsonString <|>\n jsonArray <|>\n jsonObjectAnd to make sure it all parses, let's try something in GHCi:
\nghci> runParser jsonObject \"{\\\"foo\\\": [true, false, null, 1, 1.1, \\\"foo\\\", []], \\\"bar\\\": \\\"baz\\\"}\"\nJust (JSONObject [(\"foo\",JSONArray [JSONBool True,JSONBool False,JSONNull,JSONNumber 1.0,JSONNumber 1.1,JSONString \"foo\",JSONArray []]),(\"bar\",JSONString \"baz\")],\"\")Glorious. A functional JSON parser all within 126 lines of code.
\nAs an exercise, you should try to parse JSON strings with escape characters to complete the spec.
\nIf you wanted to turn this into a proper parser combinator library, there's one main thing we're lacking: error reporting. Currently, if we fail to parse, we just return a Nothing, but this isn't very helpful.
You might want to make the Parser return an Either so that if something failed to parse, you can report the error.
newtype Parser a = Parser\n { runParser :: String -> Either ParseError (a, String)\n }\n\ntype ParseError = StringAdditionally, for even better reporting, you should consider keeping track of the position in the original string: things like the line number and character in the column would be a lot of help.
\nnewtype Parser a = Parser\n { runParser :: Input -> Either ParseError (a, Input)\n }\n\ntype ParseError = String\ndata Input = Input\n { inputLine :: Int,\n inputColumn :: Int,\n inputStr :: String\n }We've been parsing Strings as a stream of characters using our parser combinators, but it's easy to imagine parsing other things. Other streams are trivially easy. But what about parsing something like command line flags, where order doesn't matter?
In this case, we would have to change our strategy slightly.
\nnewtype ArgParser a = ArgParser\n { runArgParser :: Args -> Maybe (a, Args)\n }\n\ntype Args = [String]The function signature looks the same, but the strategy to "consume" something is slightly different. While String represented a continuous stream of data to consume top to bottom, Args represented an unordered bag of items we try to pick from. Here's what an ArgParser for a "--verbose" flag might look like:
findPop :: (a -> Bool) -> [a] -> Maybe (a, [a])\nfindPop _ [] = Nothing\nfindPop p (x:xs)\n | p x = Just (x, xs)\n | otherwise = case findPop p xs of\n Just (y, ys) -> Just (y, x:ys)\n Nothing -> Nothing\n\nverbose :: ArgParser Bool\nverbose = ArgParser f\n where\n f args = case findPop (\"--verbose\" ==) args of\n Just (_, args') -> Just (True, args')\n Nothing -> Just (False, args)Of course, if we do this type of parsing, we'd have to rework a lot of things, and we might not be able to make our parser monadic, depending on how you approach it.
\nIf you're interested in this style of parsing, you might want to see optparse-applicative.
\nLearning to use parser combinators has sparked a renaissance in parsing as well as in how I approach writing code in general. I had been familiar with functional programming for a long while, but I never realized how amazingly expressive and powerful code could be if the core tenet of composability was upheld. It's not hard to imagine creating a combinatory library for other domains in a similar way so long as your functions remain composable.
\nI can't imagine a world without parser combinators now. I've eaten the forbidden fruit. And if you understood this post, you have too.
\nAnyway, here's the final code for the entire JSON parser. It is missing some things like being able to parse negative numbers, numbers in the form 10e6, escape characters in string, etc., but it's got the core of a legitimate JSON parser. It's only 126 lines long and I don't want to even bother creating an entire repository for it.
module ParserCombinators where\n\nimport Control.Applicative\n\ndata JSON\n = JSONNull\n | JSONBool Bool\n | JSONNumber Float\n | JSONString String\n | JSONArray [JSON]\n | JSONObject [(String, JSON)]\n deriving (Eq, Show)\n\nnewtype Parser a = Parser\n { runParser :: String -> Maybe (a, String)\n }\n\ninstance Functor Parser where\n fmap f (Parser p) = Parser $ \\input -> do\n (a, input') <- p input\n return (f a, input')\n\ninstance Applicative Parser where\n pure a = Parser $ \\input -> Just (a, input)\n liftA2 g (Parser pa) (Parser pb) = Parser $ \\input -> do\n (a, input') <- pa input\n (b, input'') <- pb input'\n return (g a b, input'')\n\ninstance Alternative Parser where\n empty = Parser $ const Nothing\n (Parser p1) <|> (Parser p2) = Parser $ \\input ->\n p1 input <|> p2 input\n\ninstance Monad Parser where\n (Parser pa) >>= f = Parser $ \\input -> do\n (a, input') <- pa input\n let (Parser pb) = f a\n pb input'\n\nchar :: Char -> Parser Char\nchar c = satisfy (c ==)\n\nstr :: String -> Parser String\nstr = traverse char\n\nsatisfy :: (Char -> Bool) -> Parser Char\nsatisfy pr = Parser f\n where\n f (x:xs)\n | pr x = Just (x, xs)\n | otherwise = Nothing\n f [] = Nothing\n\noneOf :: String -> Parser Char\noneOf s = satisfy (`elem` s)\n\nnoneOf :: String -> Parser Char\nnoneOf s = satisfy (not . (`elem` s))\n\ndigit :: Parser Char\ndigit = oneOf \"0123456789\"\n\nwhitespaces :: Parser ()\nwhitespaces = many (oneOf \"\\r\\n\\t \") >> return ()\n\ncommaSep :: Parser a -> Parser [a]\ncommaSep pa = pa `sepBy` (char ',' >> whitespaces)\n\ndoubleQuoted :: Parser a -> Parser a\ndoubleQuoted = between (char '\"') (char '\"')\n\nbrackets :: Parser a -> Parser a\nbrackets = between (char '[') (char ']')\n\ncurlies :: Parser a -> Parser a\ncurlies = between (char '{') (char '}')\n\nsepBy :: Parser a -> Parser b -> Parser [a]\nsepBy pa sep = ((:) <$> pa <*> many (sep >> pa)) <|> return []\n\nmany1 :: Parser a -> Parser [a]\nmany1 pa = (:) <$> pa <*> many pa\n\nbetween :: Parser a -> Parser b -> Parser c -> Parser c\nbetween pl pr pc = pl *> pc <* pr\n\njsonNull :: Parser JSON\njsonNull = JSONNull <$ str \"null\"\n\njsonBool :: Parser JSON\njsonBool = JSONBool <$> ((True <$ str \"true\") <|> (False <$ str \"false\"))\n\njsonNumber :: Parser JSON\njsonNumber = JSONNumber <$> (float <|> int)\n where\n float = read <$>\n ((++) <$> many1 digit <*> ((:) <$> char '.' <*> many1 digit))\n int = read <$> many1 digit\n\njsonCapableString :: Parser String\njsonCapableString = doubleQuoted (many (noneOf \"\\\"\"))\n\njsonString :: Parser JSON\njsonString = JSONString <$> jsonCapableString\n\njsonArray :: Parser JSON\njsonArray = JSONArray <$> brackets (commaSep json)\n\njsonObject :: Parser JSON\njsonObject = JSONObject <$> curlies (commaSep entry)\n where\n entry = do\n key <- jsonCapableString\n whitespaces >> char ':' >> whitespaces >> return ()\n value <- json\n return (key, value)\n\njson :: Parser JSON\njson =\n jsonNull <|>\n jsonBool <|>\n jsonNumber <|>\n jsonString <|>\n jsonArray <|>\n jsonObjectWhenever I bring up the topic of monads with my peers in a programming context, I get the impression that others would like to dismiss the topic immediately, possibly because they think monads are completely out of reach for them, or maybe they think I'm using obscurantist language to sound fancy.
\nThis is hardly the case though. In practice, monads very elegantly encapsulate certain programming patterns and I think it would be a shame to reject understanding them out of principle.
\nIn this article, I hope to give some motivation for monads by solving seemingly unrelated real-life problems in programming, and then stepping back to recognize and capture the underlying pattern and distilling it into the monad.
\nI'll assume intermediate familiarity with programming, and I'll use JavaScript/TypeScript in my code examples.
\nStripe's API allows you to "expand" subobjects when retrieving objects. For example, a Subscription has this type signature:
type Subscription = {\n ...\n latest_invoice: Invoice | string | null\n ...\n}Normally, latest_invoice will be a string | null, but if you request for latest_invoice to be "expanded", then latest_invoice will take on a shape of Invoice. Similarly, Invoices has properties that can be expanded itself,
type Invoice = {\n ...\n payment_intent: PaymentIntent | string | null\n ...\n}...and so on:
\ntype PaymentIntent {\n ...\n payment_method: PaymentMethod | string | null\n ...\n}\n\ntype PaymentMethod {\n ...\n billing_details: {\n ...\n name: string\n ...\n }\n ...\n}The trouble with these type signatures is that it's really difficult to traverse nested properties. For example, if I had a subscription: Subscription and I wanted to get the name that paid for this subscription, I'd have to write something like this, if I wanted to be type-safe and runtime-safe:
const subscription = await stripe.subscriptions.retrieve(...)\n\nconst latest_invoice = subscription.latest_invoice\nif (latest_invoice === null ||\n typeof latest_invoice === 'string') {\n throw new Error()\n}\n\nconst payment_intent = latest_invoice.payment_intent\nif (payment_intent === null ||\n typeof payment_intent === 'string') {\n throw new Error()\n}\n\nconst payment_method = payment_intent.payment_method\nif (payment_method === null ||\n typeof payment_method === 'string') {\n throw new Error()\n}\nconst name = payment_method\n .billing_details\n .name;What a chore!
\nI know that at each step, the object I want to index may not actually be expanded: namely the case where it has a type of string | null. Let's build a wrapper around these objects and do that logic in there:
class MaybeExpanded<A> {\n private inner:\n | {tag: \"Expanded\", value: A}\n | {tag: \"NotExpanded\"}\n\n private constructor(mx) {\n this.inner = inner\n }\n\n static of(x: A | string | null) {\n if (x === null ||\n typeof x === 'string') {\n return new MaybeExpanded({\n tag: \"NotExpanded\"\n })\n } else {\n return new MaybeExpanded({\n tag: \"Expanded\",\n value: x\n })\n }\n }\n}All we did was move the expanded check into this MaybeExpanded.of function. We still don't have a way of actually interacting with the expanded object even if we verified that it's expanded though. So let's add a method that allows us to transform the inner value if it's expanded, otherwise do nothing.
class MaybeExpanded<A> {\n ...\n map<B>(f: (a: A) => B): MaybeExpanded<B> {\n if (this.inner.tag === \"NotExpanded\") {\n return this\n } else {\n return new MaybeExpanded({\n tag: \"Expanded\",\n value: f(this.inner.value)\n })\n }\n }\n ...\n}Ah, but this isn't quite what we need! Take a look:
\nMaybeExpanded.of(subscription)\n .map(subscription => MaybeExpanded.of(subscription.invoice))This will give us a type of MaybeExpanded<MaybeExpanded<Invoice>>, when we really wanted MaybeExpanded<Invoice>!
Let's upgrade map into flatMap, similar to how Array.flatMap "flattens" the types for us.
class MaybeExpanded<A> {\n ...\n flatMap<B>(f: (a: A) => MaybeExpanded<B>): MaybeExpanded<B> {\n if (this.inner.tag === \"NotExpanded\") {\n return this\n } else {\n return f(this.inner.value)\n }\n }\n ...\n}Before bringing it back to our example, let's define one more function to make life easier:
\nconst mindex =\n key => obj => MaybeExpanded.of(obj[key])All this function does is create a curried function that accesses an object and wraps it in MaybeExpanded. Don't worry if this doesn't make sense; it's much easier to understand it in action:
const name = MaybeExpanded.of(subscription)\n .flatMap(mindex(\"latest_invoice\"))\n .flatMap(mindex(\"payment_intent\"))\n .flatMap(mindex(\"payment_method\"))\n .map(payment_method => payment_method.billing_details.name)This looks a lot better, right? This is runtime-safe and with a little bit of type plumbing, this is actually type-safe too.
\nTo wrap this problem up, we need a way to unwrap the value out of the MaybeExpanded wrapper. (There had to be a pun there, but the joke would be wrapped in too many layers of complexity. (Did I really choose the laziest way to make this pun? Maybe. (Or perhaps, by writing this, the pun promises to fulfill itself. (Either way, this is getting too far now.))))
class MaybeExpanded<A> {\n ...\n unwrapOrError(): A {\n if (this.inner.tag === \"NotExpanded\") {\n throw new Error()\n } else {\n return this.inner.value\n }\n }\n ...\n}(This example is taken from by Bartosz Milewski's example in his series on category theory.)
\nSuppose your manager tells you that they need to audit the program. Specifically, they want every function in the program to log that it was called. A quick-and-dirty approach might be to add a console.log at the start of every function:
function double(x: number): number {\n console.log(\"double called\")\n return x * 2\n}Ah, but your manager tells you that the program should actually log to a file sometimes. Other times, the program should accumulate the logs and send them over email. Okay, fine. How about this?
\nlet log = \"\"\n\nfunction double(x: number): number {\n log += \"double called\"\n return x * 2\n}\n// Before the program exits, send the log somewhereAs a good programmer yourself, you can feel that there's something... very wrong about this.
\nPart of what's wrong is that this code violates a principle I call "locality of behavior", which states that code should only be able to affect code "close" to it. In other words, there should be no "spooky action at a distance", so to speak. If I call double, I should be able to expect that double will not act on something "far" from it!
So let's refactor this a little bit:
\ntype Logged<A> = [A, string]\n\nfunction double(x: number): Logged<number> {\n return [x * 2, \"double called\"]\n}Now, we're forced to pass in the log so that locality of behavior is preserved, and we return the new log after this function is done writing to it.
\nHowever, the return type of the function was previously a number, but we now return a Logged<number>, which is a tuple of [number, string]! If we had some code like double(square(double(x))), we'd have to change it to something like this:
function compute(x: number): Logged<number> {\n const log0 = \"compute\"\n const [x1, log1] = double(x)\n const [x2, log2] = square(x1)\n const [x3, log3] = double(x2)\n return [x3, log0 + log1 + log2 + log3]\n}This is pretty hard to work with and it's easy to imagine this getting out of hand really fast. So let's write some tooling to make this easier.
\nThe idea is that we want to way to "upgrade" functions of the form (a: A) => Logged<B> into a function with the signature (la: Logged<A>) => Logged<B> and concatenate the logs automatically through the upgrade.
Let me demonstrate:
\nconst upgrade =\n (f: (a: A) => Logged<B>) =>\n (la: Logged<A>): Logged<B> => {\n const [a, log1] = la\n const [b, log2] = f(a)\n return [b, log1 + log2]\n}Now, we can rewrite our computation like so:
\nconst upgradedDouble = upgrade(double)\nconst upgradedSquare = upgrade(square)\n\nfunction compute(x: number): Logged<number> {\n const x0 = [x, \"compute\"]\n const x1 = upgradedDouble(x0)\n const x2 = upgradedSquare(x1)\n const x3 = upgradedDouble(x3)\n return x3\n}This looks a lot better.
\nBy now, you might be thinking that there aren't any patterns to the examples I've presented so far, and may be worried that this is a waste of your time or that you aren't capable of abstracting the pattern.
\nI will not fully assuage you of your concerns because part of the effect I want to create is the "aha!" moment where everything comes together gloriously. However, to plant the seed in your mind, I want you to direct your attention to the idea of "background" and "foreground" code.
\nIn the Stripe example, the "background" code was checking whether something was expandable and the "foreground" code was indexing the object at each step.
\nIn the logging example, the "background" code was concatenating the logs and the "foreground" code was doing the computations like we did before being forced to do logging.
\nIn general, put very informally, the "background" code is what you don't really care about: it describes how to achieve your goal. The "foreground" code is what you actually care about: it describes what your goal is.
\nKeep this in mind as we move onto our final example.
\nSuppose we need to implement a parser for a protocol our company created. The protocol is fairly simple, consisting of discrete plaintext messages containing commands to interact with a shopping cart.
\nYou're presented with examples of messages you may receive:
\nclearcartmodifycart increment item 103 by 2modifycart decrement item 883 by 1Your goal is to parse these commands into this structure for another team to handle:
\ntype CartCommand =\n | {tag: \"Clear\"}\n | {tag: \"Modify\", action: CartAction}\n\ntype CartAction =\n | {tag: \"Increment\", itemId: number, by: number}\n | {tag: \"Decrement\", itemId: number, by: number}Our strategy will be to use a parser combinator approach to parsing. If you don't know what that is, don't worry. We won't go into too much depth on the theory of parser combinators in this post.
\nThe basic building block of the strategy is a Parser. For our purposes, a Parser<A> is something that reads an A from a string and returns A and the rest of the string. If the Parser failed to read an A, then it should return an error result, which will be null in our example.
type Parser<A> = (x: string) => [A, string] | nullFor example, we might implement a Parser<true> like so:
const ptrue = (x: string): [true, string] | null => {\n if (x.startsWith('true'))\n return [true, x.slice('true'.length)]\n else\n return null\n}\n\nptrue(\"true\") // [true, \"\"]\nptrue(\"tru\") // null\nptrue(\"true bar\") // [true, \" bar\"]The idea is that we can build parsers for each "atom" of our protocol and then compose them into a big parser that can then parse the entire protocol.
\nconst integer: Parser<number> = ...\nconst str: (x: string) => Parser<string> = ...\nconst whitespaces: Parser<string> = ...\n\nstr(\"bar\")(\"bar baz\") // [\"bar\", \" baz\"]\nwhitespaces(\" baz\") // [\" \", \"baz\"]\nwhitespaces(\"baz\") // nullBy having the primitive parsers, we can compose them into a bigger parser!
\nconst cartActionIncrement: Parser<CartAction> = (x: string) => {\n const r1 = str(\"increment\")(x)\n if (r1 === null) return null\n\n const [, rest1] = r1\n const r2 = whitespaces(rest1)\n if (r2 === null) return null\n\n const [, rest2] = r2\n const r3 = str(\"item\")(rest2)\n if (r3 === null) return null\n\n const [, rest3] = r3\n const r4 = integer(rest3)\n if (r4 === null) return null\n\n const [itemId, rest4] = r4\n ...\n\n return {\n tag: \"Increment\",\n itemId: itemId,\n by: by\n }\n}Okay, this is just awful to look at. All I want to do is parse the string "increment", some whitespaces, the string "item", some whitespaces, an item id, etc.
\nLet's write a way to chain these parsers easily.
\nconst chain =\n <A, B>\n (pa: Parser<A>,\n f: (a: A) => Parser<B>): Parser<B> => \n (x: string) => {\n const ra = pa(x)\n if (ra === null) return null\n\n const [a, rest1] = ra\n const pb = f(a)\n return pb(rest1)\n}chain takes a Parser<A>, runs it on the input, and then if it was successful, it feeds the result to create a Parser<B> and then returns the result of Parser<B>.
Now we can rewrite our code easily like so:
\nconst cartActionIncrement: Parser<CartAction> =\n chain(str(\"increment\"), () =>\n chain(whitespaces, () => \n chain(str(\"item\"), () => \n chain(integer, itemId => \n chain(whitespaces, () => \n ...\n )))))Ah. Maybe not.
\nThis is where we start to run up against the limits of the language. If you're familiar with older JavaScript, this might be giving you flashbacks to "pyramids of doom". The solution to Promise-based code is to use async/await, but of course, we don't have that at our disposal for Parsers.
There are actually workarounds provided by libraries like fp-ts, but let's just imagine a world where we had similar keywords parser/parse, where we could transform our code to look like this:
const cartActionIncrement: Parser<CartAction> = parser {\n parse str(\"increment\")\n parse whitespaces\n parse str(\"item\")\n parse whitespaces\n const itemId = parse integer\n parse whitespaces\n parse str(\"by\")\n parse whitespaces\n const by = parse integer\n return {\n tag: \"Increment\",\n itemId: itemId,\n by: by\n }\n}Now this is starting to look really nice. In fact, if we were working in Haskell, we'd be able to do something like this for free using its do notation, which produces some incredibly elegant code.
As for parsing the rest of the protocol, I'll leave that as an exercise to the reader in our pseudo-JavaScript language 😃
\nYou might be wondering where the monads are still. Well, we've been working with them the entire time!
\nI'll give a formal definition of a monad and you, dear reader, should take this opportunity to go back to the previous examples and see how it applies. Afterwards, we'll have a little more discussion on monads before wrapping it up.
\nA monad is a generic type Monad<A> supporting the following operation:
type Bind<A, B> =\n (ma: Monad<A>, f: (a: A) => Monad<B>) => Monad<B>Technically, if we're following Haskell's definition of a monad, there needs to be a little bit more, and the operations should satisfy certain monad laws, but we won't complicate things in this post.
\nSo in our case, MaybeExpanded, Logged, and Parser all formed monads with their respective flatMap, upgrade, and chain operations (modulo classes implicitly inserting the this argument and modulo operation order).
This probably feels incredibly unsatisfying. We picked some seemingly unrelated property that happened to be common to each example and called it a monad. I mean, we very well could have defined a "SuperMonad" to be a generic type that has the letter "e" in its name, and all of the examples would trivially have formed SuperMonads. How is this even helpful?
\nMonads are a very abstract concept, and it's really hard to pin down what ties all monads together without losing generality or being too abstract. However, like I hinted at before, one way to understand monads is through the "background" and "foreground" idea: the Bind operation is precisely where we put all the "background" work, allowing us to focus on the "foreground" work "inside" the monad, operating on the A in Monad<A>.
Another way to think of it may be clearer if we translate our examples into pseudo-Haskell using its do notation. Don't worry if you don't understand Haskell, we're mostly focusing on the overall shape of the code, not the details, and I'll modify some syntax to feel more familiar to JavaScript developers.
-- MaybeExpanded\ngetSubscriptionPayer ::\n Subscription ->\n MaybeExpanded String\ngetSubscriptionPayer subscription = do\n latestInvoice <- of subscription.latestInvoice\n paymentIntent <- of latestInvoice.paymentIntent\n paymentMethod <- of paymentIntent.paymentMethod\n return paymentMethod.billingDetails.name-- Logged\nupgrade ::\n String ->\n (a -> b) ->\n a -> Logged b\nupgrade msg f a = do\n tell msg\n return (f a)\n\ndouble' :: Logged Int\ndouble' = upgrade \"double\" double\n\nsquare' :: Logged Int\nsquare' = upgrade \"square\" square\n\ncompute :: Logged Int\ncompute x = do\n tell \"compute\"\n double' x >>= square' >>= double'-- Parser\nstring :: Parser String\ninteger :: Parser Int\nwhitespaces :: Parser String\n\ncartActionIncrement :: Parser CartAction\ncartActionIncrement = do\n str \"increment\"\n whitespaces\n str \"item\"\n whitespaces\n itemId <- integer\n whitespaces\n str \"by\"\n whitespaces\n by <- integer\n return (Increment itemId by)The point of writing it this way is to really make the foreground pop out. Within the do block, we forget that when we "bind" by using integer, what we're doing under the hood is actually parsing a string for digits and if we fail, then we short circuit the rest of the computation.
Put another way, within the context of a monad, we get to focus our attention on the important "foreground" values inside the monad, letting the monadic Bind operation take care of the underlying nasty "background" work. This lets us do magical things like saying itemId <- integer. integer has the type Parser Int, but within the monadic context, we get to "penetrate" into the monad and pretend as if we're with the underlying Int.
For example,
\nsubscription.latestInvoice, and then getting the latestInvoice.paymentIntent, and then...cartActionIncrement is a str "increment" and then some whitespaces, and then a str "item" and then an integer that we "bind" to itemId, and then...I can sympathize with concerns the imperative programmer may be feeling right now. "I want to know exactly what my code is doing! These darn monads are obscuring all the work!", or "This is so confusing! You people are just trying to make things complicated on purpose! I just want to go back to my good ol' clear, imperative code!"
\nHush, hush. Yes, to some degree, the "darn monads" are obscuring the work, and this is confusing at first sight.
\nHowever, the work that's being obscured is well-defined and can easily be translated by essentially placing the contents of the monadic Bind operation between every line of code. Is that so desirable? If so, I can make no further of an aesthetic argument for my case.
Regarding the code looking confusing, it totally can seem that way. However, this is not a good reason against learning or adopting monads. It can take some work to speak the language of monads, but the payoff is well worth it, in my humble opinion.
\nIn general, that sort of mindset is horribly defeatist. Imagine if you made a similar argument when learning how to program for the first time: "This is so confusing! What are these 'variables' and 'functions'? I just want to go back to speaking good ol' clear, natural language!". (Funnily enough, some programmers seem to actually believe this these days by overusing LLMs.) Or imagine if you applied this mindset to learning a new subject in general: "Protons? Neutrons? Electrons? Screw that, I'm only going to keep working with my atoms!" But I digress.
\nI want to leave you, the JavaScript developer, with something surprising to think about. Remember how I earlier talked about how JavaScript solved their "pyramid of doom" problem with async/await on Promises? It's a bit of a coincidence that async/await ended up with pretty similar syntax to Haskell's do notation with monads in general, isn't it? It's almost as if...
Promise<A> was a monad all along?! 😱
That's right. Promise.then has an (implicit) signature of:
then: (pa: Promise<A>, f: (a: A) => Promise<B>): Promise<B>This is exactly the signature for the monadic Bind operation we required. (Well, Promise.then is overloaded with other signatures, and again, there's a bit more monads formally require that Promises do satisfy but we haven't proven, but bah!)
We can now recontextualize all our async/await code as "working within the monadic context of Promises", or a way to "ignore the background work" and "focusing on the foreground work". In fact, this is literally true; there's a lot of work going on behind Promises that's put into the background through the Promise monad.
There's another common monad in vanilla JavaScript people use regularly, probably without realizing it's a monad. This one requires you to bend your mind a little bit to appreciate its monadic structure, which is why I haven't talked about it. I'll leave it as an exercise for the reader to figure out what it is.
\nThis was a bit of a long read, but I'm glad you made it all the way to the end. Hopefully I've convinced you that monads aren't so scary.
\nReally, they're just a programming pattern to separate the background from the foreground.
\nPut another way, working in a monadic context lets you focus on the values within the monad – the what_,_ while letting the monad handle the how with its Bind operation.
So really, monads are nothing special. The fact that the average JavaScript developer has worked with monads their entire JavaScript career without realizing it shows that monads can be understood by anyone and the mystique around monads is really fueled by a self-fulfilling prophecy.
\nWell, kind of. There is certainly more to monads and category theory that I didn't mention. Rather than an end-all-be-all post on monads, this post should serve as a jumping board to dive into the wonderful wacky world of functors, monads, monoids, and category theory in general.
\nI'll leave some further resources for wanting more:
\nIf you're wondering if you should read more mathematically-focused books on category like Saunder MacLane's classic "Categories for the Working Mathematician", or Emily Riehl's "Category Theory in Context", the answer is NO!. These aren't bad books by any means, but these are geared for the mathematician and are simply not suitable for the programmer wanting to figure out how to apply category theory to their programming.
\n\n\n", "date_published": "2025-05-13T04:16:59Z" } , { "id": "https://blog.mnke.org/homelab-tour-2025/", "url": "https://blog.mnke.org/homelab-tour-2025/", "title": "Homelab Tour (2025)", "content_html": "The recognition of the coming and going of things is a first step in training and practice.
\n
\n– Dogen
I ran into a lot of trouble figuring out how to write this post. On one hand, I plan to use this post to demonstrate some of the skills and techniques I've accrued over years of homelabbing, especially to recruiters or managers. On the other hand, I want to write faithfully to my usual rambling and colorful writing style for the small group of friends that may read my posts – and to myself, who truly is the primary intended audience of these posts.
\nIn particular, writing this post has made me realize how far I've developed my skills in systems administration, system infrastructure design, DevOps, etc. I really wanted to reflect with great sentiment on my long journey: where I started and the arduous paths I took. I wanted to pay respects to everything that lead me to where I am today.
\nAfter much deliberation, I have decided to reserve such sentiments for a separate post. I want to dedicate this post to being as technical and straight-to-the-point as possible, as if I were writing a project specification or documentation on my systems for future generations to maintain. That being said, I will not stop to explain each and every concept. To do so would be to document my history in homelabbing, systems design, networking, etc., which is much outside the scope of this post. I will assume the reader to be already familiar in a wide variety of areas so that I can cover as much ground as possible.
\nIn addition, for those unfamiliar with my writing style, I annotate sections of text that are intended as asides to provide additional context or exist simply for flavor with the word Chaff, alluding to the popular idiom, "to separate the wheat and chaff". This is inspired from Leary and Kristiansen's A Friendly Introduction to Mathematical Logic (2015) textbook, where they use the same label in a similar fashion.
\nUltimately, I hope to appeal to the recruiters or technical representatives or perhaps even novice homelabbers that will read this post. In this post, I hope to demonstrate and transfer much knowledge to you, dear reader.
\nMy primary Proxmox virtualization host is a Dell Poweredge R730XD, with a 48-core Intel Xeon 4116 processor running at 2.10GHz over 2 sockets with 256GB of RAM and ~10TB of logical storage onboard, connected to a Perc H730P adapter on RAID6, all encased in the 2U chassis the R730XD comes in.
\nI also have a separate TrueNAS machine with a Intel Xeon E5-2650 processor running at 2.60GHz with 64GB of RAM and ~40TB of logical storage using a M5015 RAID card flashed into HBA mode, connected to a JBOD enclosure, hosting 15 hard drives. This machine additionally has a GTX 1650 connected to it for transcoding media and a 4-port 1Gb/s NIC.
\nConnecting these two machines is a HP Procurve 2900 switch. While this topic exceeds the present discussion on hardware, I find nowhere more appropriate than here to mention that both the Proxmox host and the TrueNAS host are network-bonded (Linux bond) to the switch in LACP (802.3ad link aggregation) mode, for a total bandwidth of 4Gb/s – though unfortunately still limited to 1Gb/s per TCP connection due to the limitations of LACP bottlenecking each TCP connection to the speed of a physical link.
\nStarting with the simpler TrueNAS host, this machine (mostly) acts as a dumb storage controller for my Proxmox host to store large files and for cold storage (e.g. filesystem backups of my old machines). Historical factors have lead me to creating my ZFS pool with 2 data vdevs in RAIDZ2 with differing capacities: one with 6 physical disks, and one with 9 physical disks. While obviously not optimal, this was set up in a time when I was less familiar with ZFS. I have enough data in this pool at this point that it would be impractical to re-optimize in-place. I hope to build an SSD storage array in the future and migrate my data off this pool in that time so I can re-balance the vdevs.
\nYou may be wondering why this TrueNAS host isn't also running Proxmox, so that I can form a cluster with the other Proxmox host. This, too, is a matter of historical contingency rather than a carefully-planned decision. While I knew I could pass through necessary PCI slots on QEMU+KVM so that a TrueNAS VM had direct access to the HBA controller (probably, assuming my IOMMU groups were valid), I thought that TrueNAS would be much heavier than it actually was and needed the full power of the host, given that I had previously run TrueNAS on a severely bottlenecked machine. This turned out to be very incorrect, and I am now unfortunately stuck with TrueNAS instance that does basically nothing else except export storage shares to the network, wasting a ton of resources.
\nThis isn't entirely true though, because I ended up running another VM in TrueNAS and passing through a spare GTX 1650 I had. Given that my media was stored on this host and that I wasn't making full use of its power, I decided to run my Jellyfin server in a VM under TrueNAS for lower latency to the block storage (well kind of, the VM still unfortunately has to connect to the media ZFS dataset on the TrueNAS host through a network share, bleh!) as well as being able to easily access another x16 PCIE slot for transcoding. I've pinned about half of the CPUs to the VM according to the L1 cache topology to further optimize the VM.
\nGiven that I was still far from making full use of the TrueNAS host, I also put some databases/storage on it through TrueNAS Scale's built-in applications: MySQL, Postgres, Redis, Minio. These act as general-purpose databases that I use in other applications I deploy. In fact, this Ghost deployment is backed by a MySQL database on the TrueNAS host.
\nWhile I could deploy more applications on the TrueNAS host, I've decided to limit it to just these. Deploying applications on the TrueNAS host – especially those not officially supported – feels fairly finnicky and I dislike the fact that these applications are not stored as IaC (infrastructure as code). There are just better options for deployments, which we'll touch on later. I only chose the TrueNAS host for the database deployments because it had direct access to the fast ZFS pool and I was fine with having my databases outside of IaC.
\nThis is where the bulk of my infrastructure lies. Almost all of my configuration can be found in my Gitea repo. To offer original content, this post will summarize the configuration, describe the configuration on a higher-level, as well as document the configuration missing from the repo.
\nI will first describe my infrastructure in the rough before explaining how it's managed.
\nBefore a large migration carried out a few months ago, I used PiHole as my primary internal DNS server, allowing me to split DNS and allowing SNI for private, local services. During the migration, I decided to upgrade to a different DNS server, Technitium, which offered much more powerful DNS management capabilities.
\nI couldn't quite abandon my PiHole immediately though, because it contained a ton of DNS records that would be cumbersome to import and I had hard-coded my DNS server IP in multiple places. I had simply never imagined the DNS server IP ever changing. At the time, I also didn't know much about virtual IPs, so that was out of the question. If I shut down the PiHole, I didn't know what services would go down.
\nInstead, I created my new Technitium server, set my PiHole as a forwarding server to maintain backwards compatibility with my old records, and changed the PiHole server IP to the Technitium in wherever I could find.
\nEven so, I'm somewhat paranoid to fully destroy the PiHole, and I don't want to reconfigure all the blocklists, so the PiHole still sits alive to this day, as a companion to the Technitium DNS server. The DNS server(s) sit as a fundamental pillar of my internal networking stack.
\nI have a VM running Docker standalone that runs basic workloads, workloads that require fast, persistent storage – particularly with high random R/W like databases – or workloads that other infrastructure softly depend on, like my Gitea server that I use for GitOps. I call it Jumper because it acts to "jumpstart" the rest of my infrastructure without the rest of my infrastructure strictly relying on it to run. Apart from the Gitea server, I run Portainer for GitOps and I run my media stack on Jumper because the SQLite database some of the applications in the media stack use are best bound to the locally-attached disk rather than a remote share.
\nStingray is the name I give to my Docker swarm, with a 3-node manager and 3-node worker topology.
\nI don't have a particular reason for this naming, unlike my reasons for Jumper. Really, I just wanted to avoid naming things exactly for what they are because I've found such a naming scheme to be surprisingly confusing. For example, my previous Docker swarm was just named Swarm, with VMs in the swarm being named swarm-master-1, swarm-slave-1, etc. The problem arises if I want to introduce another Docker swarm. What would the naming convention be? Something like swarm-2-<role>-<n>? I find it harder to remember what Swarm-2 does, as opposed to just Swarm or Swarm-1. Instead, I find it much easier to remember what things do when I give them proper names, like Jeremy, or Joel. Sure, someone new to these resources may find the naming scheme confusing at first, and wouldn't be able to tell what these resources are, but I find this tradeoff more than worth it.
Stingray is in a bit of a strange spot. It's doomed to its fate as a Docker swarm, being an awkward middleground between a single-node Docker instance and a full-blown Kubernetes cluster. Currently, Stingray runs workloads that don't require fast, persistent storage. Notably, it runs a phpMyAdmin and pgAdmin instance as a UI to my databases, and Infisical as a secrets management replacement to Hashicorp Vault. It does run some other workloads like Portainer, Traefik, TrueCommand and small apps I write, but nothing really interesting.
\nI originally wanted Stingray to act as my Jumper: a lightweight resource that could help jumpstart the rest of my infrastructure as well as run general workloads that aren't suitable for Kubernetes, like my media stack. Unfortunately, Stingray became heavier than I thought, and Docker swarm still has no great CSI (container storage interface), even after all these years, making it difficult to justify putting persistent, IO-heavy workloads on it. Yes, I've tried GlusterFS. Yes, I've tried storage plugins. For different reasons, these solutions are simply not as convenient as running the workload on the single-node Docker instance or using Kubernetes' CSI to allocate storage.
\nI believe the fact that Stingray is in such an awkward spot really speaks to the state of Docker swarm as a feature. It seems like not many people want to keep improving Docker swarm when Kubernetes exists as a better large-scale solution and when a single-node Docker instance exists as a better small-scale solution, making Docker swarm forever doomed to be left in the dust by both parties.
\nMoirai (pronounced "mee-reh") is the name given to a group of nodes used in my Pterodactyl cluster. The Pterodactyl cluster could have an entire blog post dedicated to it, so I won't go into too much detail here. Put simply, I have a Pterodactyl panel container running on Jumper that interfaces with agents installed on the Moirai nodes to manage game servers like Minecraft, Don't Starve Together, Project Zomboid, etc. The Pterodactyl panel allows me to securely give access to some of my computing resources to friends who want to manage game servers. If you're curious on how these Moirai nodes are accessible via the internet, see my other post.
\nDolo is my premier resource, a (currently) 9-node k3s cluster, including my 3-node HA control plane. It runs with a flannel CNI, kube-vip in ARP mode for the HA control plane, and MetalLB on L2 mode for LoadBalancer support. This has been made easily automate-able through Techno Tim's k3s Ansible playbook.
\nDolo runs all of my other workloads, including, but not limited to:
\nAdditionally, Dolo serves as the main entrypoint to all of my HTTP services from the internet. All ingress HTTP traffic passes through Traefik via the cloudflare tunnel and ends up getting logged into Loki, making it easy to audit access. The cloudflare tunnel (and other tunnels) have allowed me to avoid opening any port on my router and very strictly and explicitly control and audit access to my home network. Read about how I exposed non-HTTP services in my other post.
\nWhile I could tirelessly expound on all of these services and their configurations, it would be easier if I just showed them to you. Fortunately, I can do just that, now that I've adopted a GitOps workflow for my homelab. So, instead of going through all of the configuration, I'll leave you to check out the Git repo yourself, while we talk about the structure of the repo as a whole.
\nGit is the primary method by which I manage my configuration. This gives me a centralized location to reason about all my infrastructure as well as a way to reproduce it, should anything happen to it.
\nLet us dissect the anatomy of the Git repo to gain insight into how this is done.
\nThe tf directory is where I keep my Terraform (OpenTofu) configuration. That's right, Proxmox can be interfaced with through Terraform: I manage the lifecycle of all of my VMs through the Terraform configuration now. The details of how to get this set up are provided in the README and Andreas' post and I shall not repeat them here. I created Terraform modules for my architecture components, notably the clusterized components like Docker Swarm and Kubernetes. This allows me to scale up and down my infrastructure easily, by simply changing a number and running a script, and even create test clusters if I so choose.
If you are asking whether the Terraform configuration is applied automatically, the answer is no.
\nMaybe this is violating some GitOps principle, but like any idea, we should take what we find useful from it. In my case, I find that the most valuable part of GitOps is the versioned, centralized, and declarative management approach to infrastructure, not necessarily the automation part.
\nI prefer leaving vital configuration like Terraform infrastructure outside of automation and given that I'm the only one working on this, I won't have any problems with state management.
\nOpenTofu is a FOSS fork of Terraform maintained by the Linux Foundation. You can read more on OpenTofu's FAQ for why they forked Terraform. For the end user, OpenTofu provides a much less restrictive license while doing a great job of remaining compatible with Terraform. As an avid FOSS enjoyer myself, making the move to OpenTofu was obvious.
\nThat being said, since Terraform is still more "official", I'll be referring to all the work I do with OpenTofu as "Terraform". The reader should keep in mind that in actuality, I have completely replaced Terraform with OpenTofu though.
\nWhile Terraform does a great job of managing the lifecycle of VMs, I use Ansible to provision the VMs in my ansible directory. This connection is made incredibly easy with Terraform's Ansible provider as well as Ansible's Terraform inventory plugin. Simply put, I can define a VM resource and add it to an Ansible group via the Ansible provider in Terraform, and then use Ansible's Terraform inventory plugin to generate an inventory dynamically based on the Terraform resources.
Some of the Ansible roles I employ are ones to set up a k3s cluster and set up a Docker swarm. I try to keep Ansible automation to a minimum, preferring to use a GitOps-based approach for as much as I can, so most of my Ansible playbooks are simply for initial provisioning of VMs.
\nThe dns directory is really simple. I manually version control the records by saving them to the Git repo and committing whenever I make a change on the Technitium server.
I thought about using a Terraform provider to apply DNS configuration, using some scripts to automatically synchronize DNS configuration to and from Technitium, or using something like bind9 as my DNS server, which works with .zone files directly to make this easier, but I ultimately decided against it. I definitely could not use a Terraform provider to apply DNS configuration in the same Terraform project, because I'd end up creating a circular dependency. Scripts to synchronize the DNS configuration seemed overkill.
The docker directory in my repository simply contains a repository of Docker compose files (for a standalone Docker instance) or Docker stacks (for a Docker swarm). Portainer uses these manifests to keep my configuration up-to-date, following a GitOps approach. Not all of the manifests are used, but I let them remain in case I ever want to bring them back up in Portainer.
My current workflow deploying applications is admittedly a bit awkward:
\n.env fileIf I chose to go with the "pure" GitOps approach and commit a potentially broken initial compose file and iterate fixing over commits, I would be left with a really ugly commit history unless I did some squashing.
\nI've read from some sources that the solution is to just thoroughly test the configuration, use things like pre-commit hooks or some sort of validation before committing, but the same fundamental problem remains in that I can't actually know if some configuration works until I actually try applying it.
\nThe way I approach this currently is to iteratively work on the configuration (perhaps on a different environment initially) and deploy it outside of GitOps to ensure it works before committing it and letting it be managed by GitOps. In the case of Portainer, I have to have the additional step of first deleting the configuration because Portainer doesn't know how to manage stacks deployed outside of Portainer.
\nLast, but certainly not least, we have the k8s directory where I keep my Kubernetes manifests. As mentioned earlier, I use FluxCD as my GitOps tool for my Kubernetes cluster. The reason why I chose FluxCD and not another GitOps tool like ArgoCD is frankly because FluxCD was a lot easier to set up, learn about, and FluxCD seemed a lot more lightweight than ArgoCD.
The workflow I employ to manage my cluster is now:
\nAdmittedly, there is still much detail of my homelab left out of the tight confines of this post. I haven't talked about my snapshotting and backup strategy, ZFS dataset layout, subnet configuration, etc. Unfortunately, covering the ground fully is impossible in a single blog post, in part due to the legitimate complexity of the years-old homelab, and in part due to the unrelenting passion of the homelabber, possessing them to proudly overindulge in their lengthy explanations. Blame them not though, for how could one similarly blame a child when they present a realized product of their imagination, excitedly babbling senselessly about it? But I digress.
\nI hope to have presented a small taste of my homelabbing experience: the technologies I've come into contact with, some skills I employ, etc. As I have incessantly reminded the reader throughout this post, much has been left out – certainly all the pain and suffering learning and debugging, despite this process being arguably the most valuable. If you're curious about certain aspects of the homelab – whether you're a representative seeking to ascertain my ability or an aspiring homelabber to understand it for yourself – I encourage you to ask.
\nHappy hacking!
\n", "date_published": "2025-03-20T01:31:54Z" } , { "id": "https://blog.mnke.org/no-compromises-how-i-exposed-services-securely-through-a-secondary-router/", "url": "https://blog.mnke.org/no-compromises-how-i-exposed-services-securely-through-a-secondary-router/", "title": "No compromises: How I exposed services securely through a secondary router", "content_html": "💡
\nDo you not care about my background, motivation, stories, etc.? Fine, skip to the Diving In section, you heartless monster.
\nFor the better part of the past month, I've been working on migrating my homelab infrastructure to be much more scalable, maintainable, secure, and fault-resistant (look forward to a post on this once the migration is fully complete). One of the security goals I set for this migration was to forward zero ports on my home router.
\nIn the past, I didn't have any problem with opening a few ports on my router for things like HTTP/HTTPS. I only posted my domain (pointing to my router) on trusted or private places, like posting a blog post in my Discord server, or advertising my blog on resumes I'd only hand out privately.
\nEventually, the services I wanted to expose changed, most notably with a public-facing media server I host locally. Although I only advertised it in my Discord server, the nature of the service meant that it was more likely to be shared around and more susceptible to landing in the hands of malicious actors, who could easily do a dig to find my home IP.
This is when I decided to switch to Cloudflare. I had already been on the hunt for a new domain registrar for the past while because I wanted wildcard certificates and I somehow came under the impression that Namecheap, the registrar I used at the time, didn't support wildcard certificates (they do, I was just dumb). So I came across Cloudflare, which offered a better price for my domain than Namecheap and offered a free HTTP proxying service on top of that!
\nWell, color me impressed. I switched right away and posted my domain around much more freely, now that my mind was freed about securing my IP.
\nI had been using Cloudflare for a little while at this point when I decided to host a Minecraft server for my friends. As I was testing connectivity, I realized that the Cloudflare proxy wouldn't work for Minecraft. After all, Cloudflare proxies based on some hostname identification mechanism, and presumably, Minecraft didn't present hostnames, or at least Cloudflare didn't support Minecraft's hostname identification. At the time, I didn't really know of any solutions other than being forced to open a router port and disabling the proxy on a subdomain, which is what I ended up doing.
\nThis has been a thorn in my side for the longest time. Now that my domain was much more widely accessible, there would be a lot more potential malicious actors being able to find my IP. While I did limit open ports on my router as much as I could and realistically, there aren't gonna be that many people targeting me with advanced enough tooling to compromise my network or otherwise cause harm, I knew this was against best practices and it bugged me. But again, I didn't have a good idea of what to do about it.
\nA while later, I came across ssh's port-forwarding mechanism, where you could open a port locally and have ssh open a port remotely for you and forward the traffic over the SSH tunnel. So for example, if you had a Minecraft server listening locally on 127.0.0.1:25565 and you wanted to open it on a remote port 25566, then you could do something like ssh -R 25566:127.0.0.1:25565 user@example.com, allowing you to connect to example.com:25566 and forwarding the traffic to you locally.
💡
\nIf you didn't know already, there's an accompanying flag to do the reverse: ssh -L <local_port>:<remote_ip>:<remote_port> user@example.com, which will let you connect to <remote_ip>:<remote_port> via localhost:<local_port> through the SSH tunnel.
\nThis is a killer command for managing remote servers.
I was thinking of renting a cheap VPS to forward ports for once I found out about this, but to be frank, I was too lazy and pushed it off until a few months ago, when I felt like my homelab needed a facelift. Given that this feature was so easily available, I wondered if there were any more managed solutions so I didn't have to write system services to establish tunnels myself. Sure enough, I found Fast Reverse Proxy (FRP, for short), which lets you declaratively do remote port-forwarding.
\nChaff: How have I never heard of this? It's such an amazing tool for exposing your services securely, and it even comes with a feature to connect STUN clients for P2P connectivity. I searched for FRP on YouTube and only a few videos with only a couple thousand views covering this. Maybe I ought to cover it myself.
\nSo I set up a VPS to test around with FRP and sure enough, it really was as amazing as it looked. Except, of course, that FRP won't preserve true client IPs.
\nThis isn't really FRP's fault. The same "problem" will happen if you do an SSH remote port-forward. In case you don't know what I'm talking about, let's go through an example scenario.
\n127.0.0.1:25565, which I'm forwarding to my VPS at 101.2.3.4:25565.201.4.5.6, connects to my VPS on port 25565.src=201.4.5.6:9879, dst=101.2.3.4:25565 and pushes them through the tunnel to me.src=127.0.0.1:7389, dst=127.0.0.1:25565.The "problem" is that when the packet reaches my server, the source IP has been NAT'd by FRP/SSH so the server no longer has Candace's public IP, 201.4.5.6.
Most of the time, this doesn't matter. I'd be willing to bet over 90% of game servers (which is my primary use case) would run perfectly fine with the NAT. I can only imagine servers using things like STUN/ICE might run into some problems – and even then, they really shouldn't because STUN/ICE is supposed to solve NAT traversal. The only thing that might be worrying is that a STUN binding with an ICE exchange might end up revealing your public IP over the ICE exchange (realistically, no one's going to be reading ICE exchanges to notice this, of course).
\nAdditionally, the reason I've been quoting "problem" is because this isn't actually a problem. This is intended behavior. To understand why, we have to think about the return trip of a packet.
\nIf Candace's public IP were preserved, my local server would send return packets to 201.4.5.6:9879, completely bypassing FRP/SSH. A packet like src=127.0.0.1:25565, dst=201.4.5.6:9879 would be:
202.6.7.8.src=202.6.7.8:4723, dst=201.4.5.6:9879.This would expose my public IP 202.6.7.8 to Candace, but more importantly, Candace is expecting a reply from 101.2.3.4:25565, my VPS address, not from my public IP. As such, even a TCP handshake wouldn't complete if we preserved Candace's IP through the FRP/SSH tunnel to the server. The FRP/SSH NAT isn't actually a problem because it's what allows reversing the traffic to work at all.
Even though an FRP/SSH solution would probably have been fine for me, I wanted to do better, for no better reason than to simply satisfy my intellectual urge on finding out if and how preserving source IPs could be done securely. I didn't want NAT'd source IPs or being forced to open ports on my router to receive source IPs.
\nI was looking for no compromises.
\n(Get it? "No compromises" is a double entendre for not reducing my standards and doing it securely so that I wouldn't be compromised. Ha.)
I'd say my knowledge and practical skills in computer networking has become fairly mature for my age, but one of the things I'd always been afraid of touching was Linux iptables.
Playing around with networking for many years, I've seen iptables commands here and there on StackOverflow and other forums as I was looking for solutions to networking problems, but I've avoided them if possible because they looked so daunting and it felt like I wouldn't be able to decipher what they did without spending a few days reading the manpages for iptables and its related entries.
However, I knew enough theory on L2-L4 networking to know what should be possible. In fact, I knew in theory what could be done to meet my requirements. I had hoped that there were existing solutions that didn't involve me messing with iptables, but I couldn't find any. Knowing that iptables was likely the best way for me to turn theory into practice, I took the plunge and dived into iptables documentations for, surprisingly, just a few hours.
I won't explain iptables in this post. There are plenty of other articles that do a better job than I possibly could. A lot of guides focus on using iptables for firewalling rather than for general-purpose networking, but here are articles that do focus on the latter:
Here's another diagram I found on ServerFault that I find extremely helpful:
\n
A flowchart of the iptables packet flow
\nWith the lengthy preamble out of the way, let's get down to business. First, we define what we want. We should be able to run a server (or servers) in a private home network (e.g. Minecraft) such that:
\nLet's remind ourselves of some of the options that would normally work and why they don't fully meet our requirements.
\nWhile this meets requirements 1-3, a fundamental feature of this is that they effectively SNAT the client's public IP – which we discussed in detail in the preamble – and violates requirement 4. Without the SNAT, reversing the traffic path would not work.
\nThis option would require us to set up a VPN server that's connected to the private network and ask clients to activate a VPN whenever they want to connect to our server. This definitely violates requirement 2, and kind of violates requirement 4. While we wouldn't be getting the client's IP as it would be seen by the internet, we could have a VPN topology where the client's VPN IP effectively acts as the client's public IP, for all intents and purposes.
\nWe might also violate requirement 3, depending where we place the VPN server. If we don't want to violate it, we could place a VPN server out of the private network that relays and acts as a router between peers external to the network and internal to the network.
\nAs I mentioned before, I knew in theory what could be done to achieve the requirements. The theory is actually really simple: if you understand how your home router works, you already know the theory.
\nThe router's role in a typical home network is to connect devices on the private network to the internet through a shared public IP. If we want others to be able to connect to a device in the private network, we port-forward a public port on the router to an address-port pair in the private network. Our requirements dictate that we don't do this because of security concerns, primarily that of exposing the router's public IP.
\nBut what if we just had another router with disposable public IPs? Really, that's all this solution boils down to. We have a secondary router, Router B, that select devices on the private network connect to that does NAT for us and forwards traffic inbound to the router to devices on the private network. For security, we limit the devices that connect to Router B, limit the inbound traffic Router B forwards, and limit the inbound traffic to Router B in general. Since Router B has a different IP than our primary router, we never end up exposing our primary router's public IP. Additionally, since Router B is disposable, we can easily just toss it away and get a new public IP.
\nSound too good to be true? Well, it kind of is. To illustrate the problems, we need to be more specific.
\nSuppose Router B has a public IP of 172.232.188.40 and we have a server that need to be exposed running on 10.4.4.32:25565 in the local network. The typical round-trip traffic would look like:
src=80.3.4.5:7890, dst=172.232.188.40:25565.10.4.4.32:25565 and forwards the packet to the local server, sending a packet like src=80.3.4.5:7890, dst=10.4.4.32:25565.src=10.4.4.32:25565, dst=80.3.4.5:7890.172.232.188.40:25565 and sends a packet like src=172.232.188.40:25565 back to the internet, completing the round-trip.This works fine, but what if we had multiple servers listening on the same port? For example, I might have a Minecraft server listening on port 25565 on a rack-mounted server, and another Minecraft server listening on the same port on a Raspberry Pi. When Router B receives a packet on port 25565, is it supposed to send it to the rack-mounted server or the Raspberry Pi?
One solution would be to just avoid port collisions globally within the home network for services that I want to expose. In fact, this is what's required if I want to port-forward on my fairly dumb home router. For most use cases, this will be fine, since you probably won't have that many ports to manage in the first place.
\nIf you aren't able to avoid port collisions for whatever reason, here are some solutions:
\nOption 1: Port mapping
\nSupposing the rack-mounted server listens on 10.4.4.32:25565 and the Raspberry Pi listens on 10.4.4.33:25565, you could make it so that 172.32.188.40:25565 forwards to 10.4.4.32:25565 like usual, and then have 172.32.188.40:25566 forward to 10.4.4.33:25565.
The downside of this is that you have to maintain a manual port mapping, which can be tricky. For example, what if your Raspberry Pi needed to open up a service at 10.4.4.33:25566? It can't occupy 172.32.188.40:25566 because that's already forwarded to 10.4.4.33:25565, so you'd have to choose another port to listen on.
If you decide to go with this solution, I recommend "namespacing" port allocations so that for every IP device that you might want to expose, you allot a port range. So for example, my rack-mounted server might get the port range 25000-25100 on the public IP, and the Raspberry Pi might get the port range 25100-25200. Whenever an IP device needs to expose a service publicly, you'd assign a port in its port range to map to its private port.
Option 2: Multiple IP Addresses
\nAnother solution is to have have a public IP address for each device you want to expose, giving you an entire port space for each device. For example, 172.232.188.40:25565 could forward to my rack-mounted server at 10.4.4.32:25565 and 172.232.188.41:25565 could forward to my Raspberry Pi at 10.4.4.33:25565. This is nice because it completely gets rid of port management, leaving you to only manage an IP mapping, which should be much easier. Additionally, a small side benefit of having multiple port spaces available is that it gives you a better chance of letting clients connect to services on their default ports.
However, there's a pretty big problem with this option: where the hell are you gonna get all these IP addresses?
\nWe're running out of IPv4 addresses and a lot of work has been put into place to slow IPv4 address exhaustion down, including NAT between private and public networks. You might convince a cloud provider to give you an extra IPv4 address or two per VPS, but I'd be really surprised if they'd lend out any more than that for this use case. Sure, you can always rent more VPS's, but that can obviously cost quite a bit of money, alongside being harder to maintain since you'd have more routers.
\nIPv6, on the other hand, has a gargantuan address space. Cloud providers like Linode are willing to just hand out 64-bit wide IPv6 networks per Linode instance.
\nChaff: Interestingly, it was actually Linode's support team that pointed me in the IPv6 direction. I'd always known a bit about IPv6 but I've pretty much always stuck with IPv4 for most of my networking career because I was just more familiar with it. When I opened a support ticket describing my use case and asking them if I could assign many IPv4 addresses per Linode instance, they politely declined and pointed me to some options I had, IPv6 being one of them. Overall, Linode's support team has been amazing to me and they have servers at very competitive pricing near me, which is why I settled with them. I'd post a referral link here but I haven't yet spent the $25 quota needed to create one.
\nEven though I had somewhat limited experience in IPv6 and was a bit skeptical because I wasn't sure how well-supported it was – even in 2025 – I decided I'd try option 2, managing multiple IPv6 addresses.
\nWith a firm grasp on the theory and well-thought considerations on network topology, it's time to implement this.
\nOne of the things I left out in my explanation is how my devices in the home network are going to connect to Router B, which will live in the cloud.
\nWe need a virtual network that spans across the home network to the cloud, and it needs to be private.
\nWell, that's what VPNs are for! Specifically, we'll use Wireguard, a relatively new VPN technology that has support in the Linux kernel and boasts to be more secure and faster than predecessors like OpenVPN.
\nThis is what our relevant network topology will look like:
\n![\"\"](\"https://blog.mnke.org/img/ENjQ4XUYgz-480.png\")
The VPS performs static NAT by listening on multiple IPv6 addresses and forwarding them to a VM and forwarding packets from a VM to the internet.
\nNote that many of the IPv6 addresses have been truncated for brevity. Also note the similarity in topology to your home network and the differences in NAT to the NAT on your home network.
\nTo set this up, we need:
\nI've already set up a few VMs using OpenTofu with my Proxmox server and I chose Linode as my cloud provider. Spinning up a Nanode instance on Ubuntu 24.04 with 1vCPU and 1GB of memory was more than enough for me. Here's Terraform/OpenTofu configuration if you want to follow my setup. Make sure to note the IPv6 range that was assigned to you for later.
\nvariable \"ssh_public_key_file\" {\n # example = \"~/.ssh/id_rsa.pub\"\n type = string\n}\n\n# Get this from Linode:\n# https://cloud.linode.com/profile/tokens\nvariable \"linode_pat\" {\n type = string\n sensitive = true\n}\n\nterraform {\n required_providers {\n linode = {\n source = \"linode/linode\"\n version = \"2.34.1\"\n }\n }\n}\n\nprovider \"linode\" {\n token = var.linode_pat\n}\n\nresource \"linode_sshkey\" \"mykey\" {\n label = \"mykey\"\n ssh_key = chomp(file(var.ssh_public_key_file))\n}\n\nresource \"linode_instance\" \"embassy\" {\n image = \"linode/ubuntu24.04\"\n label = \"embassy\"\n region = \"us-sea\" # Choose whichever's closest to you\n type = \"g6-nanode-1\"\n authorized_keys = [linode_sshkey.mykey.ssh_key]\n}\n\n# This assigns an IPv6 range to the instance, but not any\n# addresses. We add addresses later.\nresource \"linode_ipv6_range\" \"embassy\" {\n linode_id = linode_instance.embassy.id\n prefix_length = 64\n}\n\noutput \"ipv6_subnet\" {\n value = \"${linode_ipv6_range.embassy.range}/${linode_ipv6_range.embassy.prefix_length}\"\n}For 3., DigitalOcean recommends basing the IPv6 network off of a hash of the current time and the server's machine ID, which is what I did. For your convenience, I've shortened the steps into a single command to generate an IPv6 subnet.
\nprintf \"$(date +%s%N)$(cat /var/lib/dbus/machine-id)\" |\\\n sha1sum |\\\n awk '{ print $1 }' |\\\n cut -c 31- |\\\n sed \\\n -e 's/^/fd/' \\\n -e 's/\\(.\\{4\\}\\)/\\1:/g' \\\n -e 's/:$/::\\/64/'You should get an output like this:
\nfdb4:f371:2e6c::/64DigitalOcean recommends to run this on your server for its machine ID, but it shouldn't really matter.
\nFor the rest of post, we'll assume that:
\neth0fdab::/642600::/64# Install wireguard\napt update && apt install -y wireguard\n\n# Create public-private key pair\nwg genkey | tee /etc/wireguard/private.key\nchmod go= /etc/wireguard/private.key\ncat /etc/wireguard/private.key | wg pubkey | tee /etc/wireguard/public.keyCreate a file in /etc/wireguard/wg0.conf:
[Interface]\n# First address on the IPv6 Wireguard network you generated earlier\nAddress = fdab::1/64\nPrivateKey = <output of /etc/wireguard/private.key>\nListenPort = 51820\n\n# Enable IP forwarding. Not sure if the ipv4/ip_forward is necessary,\n# but I have it as an artifact from when I was testing with IPv4 and\n# I now have a superstition that I need this.\nPreUp = echo 1 > /proc/sys/net/ipv4/ip_forward\nPreUp = echo 1 > /proc/sys/net/ipv6/conf/all/forwarding\n# Disable autoconf while running. We assign some IP addresses that\n# get removed by autoconf otherwise.\nPreUp = echo 0 > /proc/sys/net/ipv6/conf/eth0/autoconf\n\n# replace with a public IPv6 address you own\nPostUp = ip -6 addr add 2600::10/64 dev eth0\n# we'll forward ports 10000-10100 for demonstration, but\n# **you should limit this**.\n# additionally, if you want UDP, copy these lines and replace\n# tcp with udp\nPostUp = ip6tables -t nat -A PREROUTING -p tcp -d 2600::10 --dport 10000:10100 -j DNAT --to-destination fdab::10\nPostUp = ip6tables -t nat -A POSTROUTING -p tcp -s fdab::10 -j SNAT --to-source 2600::10\n\n\n# these should be the reverse of the commands above.\n# so if you add/remove commands above, mirror them below, except replace\n# \"-A\" with \"-D\" on the ip6tables commands and \"add\" with \"del\" on the\n# ip commands.\nPreDown = ip6tables -t nat -D POSTROUTING -p tcp -s fdab::10 -j SNAT --to-source 2600::10\nPreDown = ip6tables -t nat -D PREROUTING -p tcp -d 2600::10 --dport 10000:10100 -j DNAT --to-destination fdab::10\nPreDown = ip -6 addr del 2600::10 dev eth0\nPostDown = echo 1 > /proc/sys/net/ipv6/conf/eth0/autoconf\nPostDown = echo 0 > /proc/sys/net/ipv6/conf/all/forwarding\nPostDown = echo 0 > /proc/sys/net/ipv4/ip_forward\n\n[Peer]\nPublicKey = <fill this in later>\nAllowedIPs = fdab::10/128\n# recommended if this peer is behind a NAT (it probably is)\n# consider lowering this if the tunnel breaks every once in a while\nPersistentKeepalive = 30Again, I won't go into too much detail explaining the iptables commands. You can refer to the resources I linked earlier if you're interested. On a high level, all this does is:
2600::10.2600::10 to a Wireguard peer on fdab::10 (we'll set the peer up later).fdab::10 to the public IPv6 address on 2600::10Similar to the VPS, we need to install Wireguard and generate some keys.
\n# Install wireguard\napt update && apt install -y wireguard\n\n# Create public-private key pair\nwg genkey | tee /etc/wireguard/private.key\nchmod go= /etc/wireguard/private.key\ncat /etc/wireguard/private.key | wg pubkey | tee /etc/wireguard/public.keyCreate a file in /etc/wireguard/wg0.conf:
[Interface]\nAddress = fdab::10/64\nPrivateKey = <output of /etc/wireguard/private.key>\nListenPort = 51820\n# Outbound IPv6 traffic will by default by routed through the Wireguard\n# tunnel through the AllowedIPs configuration below.\n# In my case, I seemed to have a route already added that\n# routes local traffic properly, but if you don't, consider adding\n# these.\n# PreUp = ip route add <local network> dev eth0 proto static\n# PostDown = ip route del <local network> dev eth0 proto static\n\n[Peer]\nPublicKey = <the public key on your *VPS*>\nAllowedIPs = ::0/0\nPersistentKeepalive = 30\n# You can use an IPv4 address here if you want\nEndpoint = <VPS IP>:51820Finally, don't forget to copy the public key you generated on your local VM and set it on the VPS configuration:
\n...\n[Peer]\nPublicKey = <you can fill this in now>\nAllowedIPs = fdab::10/128\n# recommended if this peer is behind a NAT (it probably is)\nPersistentKeepalive = 30If you have multiple VMs, you'd have to add more entries on the VPS's /etc/wireguard/wg0.conf:
...\nPreUp = echo 1 > /proc/sys/net/ipv4/ip_forward\nPreUp = echo 1 > /proc/sys/net/ipv6/conf/all/forwarding\n\n# === Copy and paste this block for each VM ===\n# === Don't forget to replace addresses ===\nPostUp = ip -6 addr add 2600::10/64 dev eth0\nPostUp = ip6tables -t nat -A PREROUTING -p tcp -d 2600::10 --dport 10000:10100 -j DNAT --to-destination fdab::10\nPostUp = ip6tables -t nat -A POSTROUTING -p tcp -s fdab::10 -j SNAT --to-source 2600::10\n# =============================================\n\n# =========== Same with this block ============ \nPreDown = ip6tables -t nat -D POSTROUTING -p tcp -s fdab::10 -j SNAT --to-source 2600::10\nPreDown = ip6tables -t nat -D PREROUTING -p tcp -d 2600::10 --dport 10000:10100 -j DNAT --to-destination fdab::10\nPreDown = ip -6 addr del 2600::10 dev eth0\n# =============================================\n\nPostDown = echo 0 > /proc/sys/net/ipv6/conf/all/forwarding\nPostDown = echo 0 > /proc/sys/net/ipv4/ip_forward\n...\n\n# =========== Same with this block ============ \n[Peer]\nPublicKey = <fill this in later>\nAllowedIPs = fdab::10/128\nPersistentKeepalive = 30\n# =============================================On each of your hosts, run systemctl start wg-quick@wg0. For a post-flight checklist, I recommend doing the following tests:
Try pinging between your clients and servers: ping6 fdab::10. If this doesn't work, then Wireguard likely has not established a tunnel, which you can check with wg show.
On VPN clients, curl -6 ifconfig.me. It should match a public IPv6 address of your VPS (e.g. 2600::10). If this doesn't work but Wireguard has established the tunnel, then something is wrong with the iptables configuration.
On a VPN client, continually listen on a TCP port with:
\nwhile true; do nc -vnlp 10000; doneOn different machine (ideally one not in your home network to prevent potential routing issues),
\nnc -vn <public IPv6 matching that VPN client> 10000If all goes well, you should be able to establish a TCP connection and start communicating with netcat.
\nIf you're unable to establish a connection, I don't have a quick solution other than to start monitoring traffic on both your VPS and on your VM with tcpdump -ni wg0 -p tcp port 10000 and even tcpdump -ni eth0 -p tcp port 10000 on your VM, in case it turns out that outbound packets on the VM are being routed incorrectly. You'll have to read the traffic flow and diagnose what's going wrong based on which packets are arriving on which interfaces – or if they aren't.
Otherwise, if all of these checks succeed, you should be able to open arbitrary TCP ports on your VMs and expose it via the VPS! Of course, you should limit what's exposed on your VMs and limit what's forwarded/accepted on the VPS. If you want UDP (many games will probably need this), the /etc/wireguard/wg0.conf configuration on the VPS needs to be adjusted accordingly (I've already marked what needs to be changed).
Finally, if you're sure that this configuration works, then you can run systemctl enable wg-quick@wg0 on all your hosts. We didn't do this earlier in case somehow turning on the service blocked us from connecting to a host. By not enabling the service, we'd still be able to just restart the machine to wipe the ephemeral configuration in case things went wrong.
Of course, if you plan to scale up/down the number of local VMs, you should probably write a script to do this for you. Personally, I've created a Terraform/OpenTofu module that chains into an Ansible playbook making use of a Wireguard role that Githubixx wrote, which I may release in the future.
\nWe've configured a VPS as a secondary router that NATs traffic for us over a Wireguard network, which is pretty awesome. Of course, what I set up is way overkill for the average person who just wants to run a Minecraft server for their friends. At the same time though, this really wasn't that hard to implement, nor is it actually that complicated in theory once you break it down. At any rate, FRP would be a solid option for most people. Seriously, FRP looks awesome and I want to give it a try and write another blog post on it or even make a video giving it much-needed spotlight.
\nThe configuration I gave is mostly secure, but I'd definitely tighten up the ports that the VPS forwards for, as well as making sure a firewall is enabled – either by your cloud provider or on the VPS itself through something like ufw (or even through iptables 😉). I'd also be absolutely sure that the range of ports that the VPS forwards for on each VM are closed, except for the ones that explicitly should be open. Even better, you could forward individual ports, one for each that you need to open, rather than forwarding port ranges.
It might be worth looking into traffic limiting as well. Linode only gives a certain amount of bandwidth per month, meaning an attacker could flood the VPS and take down networking for the remaining month. I currently don't know any great solutions to this. Whitelisting IPs is too inflexible and manual for my needs, and limiting bandwidth per connection seems like it would be either too restrictive per connection, or too loose to prevent any competent attacker from flooding the VPS. Fortunately, this isn't a problem I've run into yet.
\nYou could also set up some monitoring on the traffic that's being forwarded on the VPS with something like Promtail, Loki, and Grafana. You could even send it back into your home network, which shouldn't be too hard since you're already connected through the Wireguard tunnel.
\nYou could also set up dynamic policy-based routing, where your VMs will only send packets through the Wireguard tunnel as a response, rather than blindly sending all traffic through the tunnel. This may be advantageous in cases where you want your VMs to use your primary router by default, but only respond to client requests to your server. I personally wasn't able to get this working completely, only being able to support TCP, not UDP. It might be fun to do this as a challenge using iptables and conntrack, but I'm not sure I'd recommend this, since you're probably going to throw off STUN binding results and possibly expose your primary router's IP through STUN/ICE exchanges if you have this split policy.
For me, I'm gonna be monitoring how well this works for the next while, focusing on the IPv6 aspect of it. While I haven't run into any problems so far, I can't be sure of its stability just yet.
\nIn any case, I hope you enjoyed this post and got something out of it. It was a blast playing around with networking like this and discovering the sheer power of iptables.
Recently, some of my friends decided to hold an online tournament, and the main tournament organizer wanted everyone to be able to stream their screens so the production crew could grab live feeds and multiplex/composite them, ultimately feeding an "official" production tournament broadcast to a streaming service like Twitch.
\nOriginally, they were planning to use a service like VDO.Ninja for everyone to stream to. VDO.Ninja has inbuilt compositing capabilities, allowing someone to select an individual stream and show it, show two streams side-by-side, etc. Not only that, but contestants wouldn't have to set up any streaming software since VDO.Ninja uses WebRTC on the browser for streaming, making streaming really easy for contestants with less technical skills.
\nThe only thing VDO.Ninja lacked was recording capabilities. Part of the reason is that WebRTC is fundamentally P2P, meaning streams aren't first delivered to a central server, where they can be saved, before delivering to clients. While there was some tooling to export VDO.Ninja broadcasts into another server (built by a VDO.Ninja maintainer, I think), I found it kind of finnicky to work with. We really wanted every stream to be recorded though since we might miss important highlights live, and we want the ability to dig through individual streams to compile a tournament highlights video after.
\nI happened to have recently watched an old video by Nerd on the Street where he set up a simple HLS streaming server with NGINX, powered by an RTMP module written for NGINX. Setting it up locally was extremely simple, taking me only 15 minutes or so to get a POC up. I got an NGINX instance compiled with arut's NGINX RTMP module running and I used OBS to send an RTMP stream locally and used OBS again to pull the RTMP stream. This demonstrated that we could pretty easily use an RTMP server for clients to send their streams to and then use OBS to pull streams from the RTMP server and use OBS to multiplex and composite the streams.
\nDigging it into the documentation for the RTMP module more, I found that it supported directives for recording. Awesome! This looks like exactly what we needed.
\nEmboldened, I started writing some configuration to deploy the RTMP server on a cloud service. I ended up chose AWS because it had servers fairly close to me, and I wanted to minimize latency as much as possible. I wrote some basic Terraform configuration for server provisioning, and an Ansible playbook to install Docker and deploy a compose file for an NGINX + RTMP image, alongside the NGINX configuration I needed.
\nWe deployed the server on a tiny 1vcpu 1GB server and I was able to send a stream for a friend to pull. However, while I was waiting for packages to install on the low-powered server, I realized there might be a problem. We were going to have up to 40 contestants participating in the tournament. Granted, it wasn't certain that everyone was going to stream, but I wanted to prepare for the worst. If all 40 contestants streamed to the server, would a single server be able to handle it? Just forwarding a stream shouldn't be too intensive, but recording 40 streams at once sounded like it might be a problem. While we may be able to get away with one really powerful server for just 40 streams, I wanted a solution that could scale as much as needed.
\nMy first move was deploying a Docker swarm, which allowed horizontal scaling really easily. I rewrote the Terraform and Ansible configuration to create a Docker swarm with an arbitrary amount of worker nodes and converted the compose file for the RTMP server into a stack with variable number of replicas. I deployed the infrastructure, sent some streams, and verified the load was being spread properly. However, I was only able to pull some of the streams. What was happening was that the streams I send (e.g. to /stream-1, /stream-2, ... were being load balanced on L4 by Docker's mesh network to some arbitrary RTMP container (as expected), but when I try to pull a stream, they might not be load balanced to the same container. Obviously, Docker's L4 load balancing isn't aware of the L7 stream paths. For some reason, I completely neglected to think about this. I'm too spoiled by HTTP, with its powerful L7 reverse-proxy ecosystem, I guess.
Taking a step back, I did some more research on the NGINX RTMP module and found some useful directives. The push and pull directives gave the server powerful relaying capabilities, allowing me to easily compartmentalize and topologize the infrastructure as desired. I still wanted to go with Docker Swarm instead of Kubernetes because Kubernetes seemed overkill and honestly, I was rusty on my Kubernetes. It had been over half a year since I administrated my last cluster. Additionally, I wanted to play around with Docker Swarm a bit more since it seems like a good middle-ground between a single node VM serving Docker containers and an entire Kubernetes cluster.
After some careful planning, I came up with this infrastructure:
\nHere's a diagram of the infrastructure:
\n![\"\"](\"https://blog.mnke.org/img/01_9medzmP-480.png\")
Diagram of streaming infrastructure. Multiple overlaid squares indicates variable number of nodes.
\nYou might be wondering why there's only one playback node. As mentioned earlier, we need a L7 reverse proxy to load balance between playback nodes. While we could actually write a L7 reverse proxy by writing a small HTTP server that's called using the RTMP module's on_* directives alongside Docker Swarm's service discovery to poll internal nodes for whether they have a stream, I decided that would be overkill for this project. A single somewhat powerful playback node should be able to handle only forwarding streams for our purposes.
However, since I did mention scalable infrastructure in the title and I want to stay true to my word, I'll go into a bit more depth on how this would be done.
\n![\"\"](\"https://blog.mnke.org/img/8aM7-H4CR_-480.png\")
Diagram of revised streaming infrastructure. Multiple overlaid squares indicates variable number of nodes.
\nLet's break the changes down:
\n/stream-k from the production crew, we use the on_play directive to make an HTTP request to a controller node requesting for /stream-k./stream-k is available on them and responds with an HTTP redirect to the origin node with /stream-k available./stream-k available and send it to the requester.With these changes, every component of the architecture can be horizontally scaled arbitrarily.
\nAnyway, I didn't do that because it was too high effort 😛. Instead, I added some authentication to the ingestion servers by using the on_play directive and writing a simple HTTP server that authenticated the stream path (what contestants would enter as the stream key on OBS) against a Redis database that mapped their stream key to a human-friendly path. For example, if I (Tony) streamed to the path/333fe3cc19f4a6c80cefb146a28883eb, I'd have a mapping from 333fe3cc19f4a6c80cefb146a28883eb to tony on the Redis database, which would make the ingestion node relay the stream to playback and recording nodes on the path /tony. This made naming recordings easier as well as give the production crew human-friendly stream paths to pull from.
Additionally, I added some monitor tooling with Grafana, Prometheus, cadvisor, node-exporter as well as a Redis admin panel to mess around with stream keys on-the-fly and Traefik as an HTTP reverse proxy for basic authentication. I also wrote a small web application that connected to a mod for the we were playing to expose an admin interface on the browser as well as gave live updates to the web application, but I won't go into detail for that in this post.
\nThe remainder of this post is about how the tournament turned out and my stories from it. If you're only interested in streaming architecture, there's no discussion of that beyond this point.
\nWe set a date to host a mini-tournament with a quarter of the contestants to test out the streaming infrastructure. I ran a few commands to set up the swarm, we distributed streaming keys, and it worked like a charm. I monitored the server loads on my Grafana dashboard and they were holding up amazingly with 10 concurrent streams. None of the servers were even hitting 50% load per CPU. I was feeling confident we'd be able to handle 4x the load as long as I scaled accordingly. After destroying the swarm, all there was left to do was wait for the day of the tournament.
\nBut of course I didn't do that. I knew there were some improvements to be made. Namely, the playback server didn't have any authentication mechanism, meaning malicious actors could potentially publish and override a contestant's stream. Obviously, no one was going to do that, but I had the programmer's itch to achieve perfection, against my best judgement.
\nSo, after making sure to freeze the configuration that worked during the mini-tournament in Git, I started hacking at a solution. Part of what made it nontrivial was that through Docker's ingress network, an SNAT is applied that masquerades the original client IP, making it difficult to deny publishing using IP ranges. Without going into too much detail, I found a solution that seemed to work. After some more testing, I was happy with what I had and settled with this version for the tournament.
\nThe day of the tournament finally arrived. Two hours before the start of the tournament, I spun up the streaming infrastructure and the production crew started setting up their software. A few contestants arrived early and started streaming to the server. Some streams weren't showing up on the production crew's feed, but I gave it little thought because I knew it could take a while to connect and sometimes you just have to refresh the streams.
\nThere was one contestant who said he had been streaming to the server the entire time it was up, yet the production feed wasn't picking up on it. Strange. I started deeper investigation by trying to pull feeds from all the contestants that were streaming. Some of them weren't showing up. At this point, I was starting to get anxious, wondering if I should roll back to the version that worked during the pilot run or whether I could diagnose the problem before the stream started.
\nMore and more contestants were showing up and starting to stream now. Some streams still were not showing up and I hadn't found a diagnosis, much less a solution. Spinning up the entire infrastructure took about 15 minutes. There was no time left and the situation was getting dire. I decided to allot 15 minutes to finding and fixing the issue, otherwise it was time to roll back. I muted and deafened myself, blocking out the noise from the growing number of contestants joining the call and focused all my attention on making sure things worked. There were 40 contestants and a handful of production crew, all of who were depending on the streaming infrastructure. I had to do whatever I could to get it running.
\n15 minutes of furiously issuing commands still left my empty-handed. There was no other choice than to roll things back. I sent a message to a tournament organizer telling them to notify the dozen or so contestants streaming that the server would be going down temporarily. I destroyed the infrastructure and reverted the configuration to the stable version. I ran the commands to spin up the new infrastructure and waited in apprehension, watching each command fly by. What if even rolling back doesn't work? I wondered. I shook the thought off. It will work. I reassured myself.
\nThe streaming server was back up. I began receiving notifications that some contestants started publishing to the server, while others waited for DNS records to propagate. I checked in with production. We were receiving feeds for all publishing contestants. I breathed a big sigh of relief. It seems like at least this problem was solved. But there were more problems to come.
\nThere were still a few contestants unable to connect to the server, and I was sent to diagnose the issue. After going through all the standard procedures like making sure the URL was used, the right stream key was inputted, etc., they still weren't able to connect. It couldn't be a DNS issue, could it? I mean, I set a TTL of 600s on the records, the lowest my registrar allows, and it's been well over 600s. I thought to myself. To make sure, I guided a contestant to do an nslookup on my domain. She was still hitting the cache for the old records, and I assume this was the case for the other few contestants that were experiencing similar issues. Frantically, I searched up how to flush the DNS cache on Windows and relayed the information to them. No luck. Maybe the records were still cached externally. My last resort was to ask them to edit their hosts file, which is seriously a pain on Windows. Yet it still didn't work!
At this point, I should have asked them to stream to the IP address directly. RTMP servers don't do any hostname identification, after all. However, I was too caught up in the chaotic atmosphere, so focused on making sure the domain resolved correctly as the solution that I didn't think of this. Out of ideas, I could only apologize to the contestants and ask them to try again in a few minutes.
\nWe were past the planned start time, but between contestants showing up late, tournament organizers having their hands full dealing with questions from the contestants and a general lack of preparation, we were running late on schedule. Fortunately, I had a chance to breathe because the technical issues seemed to have been resolved. I check up on my Grafana dashboards I had prepared to see how the server was holding up. 400% system load per CPU on one of the servers was the first thing I saw. My face went white. We were 15 minutes past the start time. Scaling horizontally doesn't solve the issue because it seemed to be just the playback container generating this load. The only way to handle this load was scaling vertically, which meant destroying the infrastructure and rebuilding it again.
\nMy mind was racing. The importance of making the correct decision fast was not lost upon me. I didn't have time to think about why there was so much load compared to the test run. I thought hard about what I should do for a minute. If we continue under this extreme load, there could be problems with playback, causing problems with the broadcast. Since I also didn't distribute the placement of the containers well, there seemed to also be a recording container scheduled on the server with high load, meaning there could be recording issues too. However, If I vertically scale, this would cause the infrastructure to be rebuilt, causing everyone to disconnect and possibly never reconnect, given the DNS issues we had earlier.
\nI made the hard decision to vertically scale. My plan was to scale up first, and then start re-balancing the containers after. There was no chance I could re-balance the containers with the current infrastructure, with just one container being under so much pressure. I also didn't configure a re-balance before I spun up the new infrastructure because I needed to get the new infrastructure up ASAP. So I destroyed the infrastructure and started rebuilding it with higher spec'd VMs. I didn't notify anyone of this decision since there was so little time to lose and contestants were left wondering why they disconnected.
\nAfter a painful 10 minutes, the servers were up and contestants were connecting back to them. However, there was one server that just failed to start and the command to deploy the Docker stack was hanging, even though contestants were connecting. In retrospect, my guess is that all the contestants and the production crew constantly trying to connect to the servers at once affected the startup, causing it to behave strangely. As for why one server failed to start, I still have no idea. I guess I got unlucky.
\nEven though almost all of the contestants connected back, I was still anxious. Looking at the dashboard, I saw that the server load per CPU was still around 220%. More importantly, the command to deploy the stack was still hanging. The second part of my plan to re-balance the containers after the initial deployment hadn't been executed yet, and this was vital for further reducing the load. The only thing I could do as I waited for the command to finish was read Docker documentation to see what else I could tune.
\nThe command still hadn't terminated. Even though the stream seemed to be going smoothly, I knew I had to apply the configuration to re-balance the containers. The tournament had officially started now and the production crew was broadcasting to Twitch. I held my breath as I CTRL+C'd out of the hanging command, waiting to see if any streams died.
After a minute of waiting, nothing happened. The streams were continuing like normal. Alright, there's just one last step of this plan. I thought. I just have to apply this configuration that should help re-balance the containers.
\nI ssh into the manager node and carefully edit the compose file, making sure not to make any changes to the playback container. I was fine with ingestion and recording containers being rescheduled, since a new replica should take its place before the old one shuts down, meaning there would only be a brief interruption, but the playback container could not be touched since there could only be one of them.
After double-checking and triple-checking the configuration, I run a docker stack deploy. The playback container goes down. The production crew's streaming software crashes from this interruption, and a fallback broadcast is sent to Twitch. I still don't really get why the playback container went down. Normally, if you don't make any changes to a service's configuration and re-deploy the stack, nothing happens to it. I think it must be because the service didn't start up properly earlier, as indicated by the command hanging.
In any case, we were on code red. One of the casters had to take over and make some small talk during this downtime, and I couldn't do anything other than wait for the stack to come up.
\nAfter an agonizing few minutes, it fortunately did. We re-launched the crashed software and the streams came back online one-by-one. We redirected the broadcast to the contestant's screens and things were back to normal.
\nOne of the servers was still under fairly heavy load (around 150% system load per CPU, spiking occasionally to 180%). Additionally, one of the containers still didn't seem to be rescheduled correctly (likely a misconfiguration on my end), but seeing that the streams were running smoothly after a few minutes, I relaxed. Plus, I was way too scared of making any changes, even if they should be safe, given how catastrophic it was earlier.
\nAfter a turbulent start, the rest of the stream went relatively according to plan. People seemed to have fun and the tournament turned out pretty well. Well, except for the fact that we never actually recorded the main broadcast. We had made a new Twitch account for this and forgot to enable recording broadcasts. So while we recorded the contestant's individual streams, we didn't have the main broadcast, which included at least the commentary of the casters, the stream chat, and the general atmosphere of the stream. 🤦
\nSo, what went wrong that day?
\nTo start, I shouldn't have deployed the unstable configuration. Even though I thought I tested it well and even though I still don't know why it didn't work, I can't really justify "securing" the playback servers against unauthorized publishing by using the unstable configuration. This didn't end up being that big of a factor though. After all, I did have my backup plan of reverting to the stable version, which was executed almost perfectly. The only thing I didn't account for was DNS taking too long to propagate, which did end up causing some issues.
\nReally, the biggest factors were poor container balancing and the single playback container being a bigger bottleneck than I thought. While it's somewhat unreasonable to redesign the architecture to handle multiple playback containers as suggested above, I totally could've vertically scaled the node handling playback to have much higher specs as well as done a better job making sure no other containers were scheduled on the playback node.
\nEven with a few hiccups here and there, this was a great experience. I learned about streaming architecture, tournament planning, and it was great to see everything come together. In preparation for the next tournament, I've ported the architecture to use MediaMTX containers, which has richer features than just an NGINX-RTMP container.
\nCheck out the repository on the stable commit if you'd like.
\nThe MediaMTX port is still a WIP as the next tournament date is still in limbo, but you can check out my current version here.
\n", "date_published": "2025-02-13T03:40:11Z" } , { "id": "https://blog.mnke.org/maplestory-hax-1337/", "url": "https://blog.mnke.org/maplestory-hax-1337/", "title": "Maplestory Hax 1337", "content_html": "I started playing Maplestory in around 2006 upon the recommendation of my friend and my childhood is filled with fond memories of it. I played it on and off in my adolescent years but grew increasingly frustrated with the direction of the game so I quit permanently. In high school, I knew that Maplestory hacks existed and even how to get them running, but I had always wondered how they worked and aspired to create my own hacks.
\nWell, after completing an entire degree in computer science, I came back to this question with renewed interest and newfound expertise. I had some experience in reverse engineering, but basically none in reverse engineering low-level binaries and I wasn’t confident I could crack Maplestory’s anti-cheat right off the bat, so I honed my skills on other singleplayer games without anti-cheat first. Then, with some practice under my belt, I directed my aim to Maplestory.
\nThe first part to hacking a game is to do research on the available hacks. How do they work? What can I learn from them? Unfortunately, Maplestory is a pretty unpopular game by today’s standards and the hacking resources out there were really out of date. Even worse, Maplestory switched from 32-bit binaries to 64-bit binaries relatively recently, which invalidated a ton of resources. Still, I pieced together what I could in the hopes of crossing the first great barrier: disabling their anti-cheat.
\nI don’t think I can legally put most of this information online (hell, hacking the game privately may even be illegal — I mean… allegedly hacking the game privately). Although it’s unlikely Nexon would ever come after me for this especially since this information exists in other, more well-known websites, and I think in general, trying to hack games and publishing reverse engineered information online is morally permissible, I still feel some moral pull to not contribute to it here (at least in detail). Also, this project happened months ago and I’m too lazy to dredge up the details 😛
\nAfter I compiled Cheat Engine to hopefully prevent detection by the anti-cheat, one of the first things I tried was to modify some instructions in memory. Of course, an integrity check caught this and the game exited immediately. Integrity checks must scan the memory I modified, so I made some changes and tried to find what was reading that memory, which lead me to some obfuscated code in the Maplestory binary. I could only see a repe movsb instruction followed by a jmp, and instructions before were clearly mangled, so it seems like the instruction was reached through another jmp. In any case, I broke on that instruction to find the start and end region of the copy and found that it was just the region occupied by the Maplestory module. So, I copied that region into somewhere else in memory and then hooked the instruction to change the region to the copied one. I changed some instructions here and there and… it worked!
Next, I knew that as part of the NGS (Nexon Game Security) anti-cheat suite, another process called BlackCipher64 was spawned. I wasn’t too sure of what that did, but I figured I needed to hijack it, considering I had seen some forum threads talking about how to disable it in previous versions. I guessed that it would scan through the Maplestory process for an extra integrity check and for possibly unknown modules. When I was looking for integrity checks earlier, I couldn’t find anything indicating BlackCipher was doing this though, nor could I find any calls to ReadProcessMemory to remotely read the Maplestory process’s memory though so I tried to catch BlackCipher64 trying to scan for modules in Maplestory. Sure enough, I found a call to Module32Next, which is used to enumerate modules. I tried to place a sleep in Module32Next, but BlackCipher64 and Maplestory would both exit after a while, which indicated that BlackCipher was subject to integrity checks as well. I knew that integrity checks might be difficult to perform across modules, so I guessed that the module containing Module32Next had an integrity check separate from the module running BlackCipher64.aes. I traced the Module32Next call back into the BlackCipher64.aes module and tried to place the sleep in there, and the game didn’t exit! It seems like the main module didn’t have any integrity checks that forced an immediate exit upon failure.
I didn’t know how well these bypasses really worked. I mean, there could be another mechanism that detects my bypasses and silently reports it. The BlackCipher bypass, for example, really shouldn’t have worked: why wasn’t there an integrity check for the main module? I was too excited that my bypasses seemed to work that I didn’t think about this and continued onward.
\nAll of my work had been prototyped using Cheat Engine, but I wanted to transfer it into a DLL I could inject for ease of use. There was the potential that the DLL would be detected still, but this was a challenge I wanted to overcome. So, after a lot of head-scratching in C++, I transferred my work into DLLs to inject into the BlackCipher process and the Maplestory process. Sure enough, they worked.
\nThe pinnacle of a hacked MMO client, as I see it, is a headless client sending and receiving packets conforming to the network protocol of the game. This demonstrates complete mastery over what can be known about the client by the server and is likely to reveal attack vectors that would be otherwise difficult to find through simple client modifications. In practice, this is immensely difficult. Reverse engineering the network protocol is already a daunting task, but trying to do it through obfuscated code makes it difficult enough that it’s just not worth it. I was doing this for sport, not out of a desire to create competent hacks, though. So I set my eyes on understanding the network protocol.
\nThe first matter is that packets are encrypted, meaning we can’t MITM spy on packets past the application layer with Wireshark or something. We have to find hook and log the packets before they’re encrypted and after they’re decrypted. Finding the encryption and decryption functions took me almost a week. The decryption function isn’t too bad because I can just break on recv and monitor reads at the memory that the recv populates and follow it from there. The encryption function is really difficult because by the time the send function is called, you have no idea where the memory came from or how it was constructed. I’d normally backtrace by breaking on the send and tracing up the call stack, but some regions of the code is extremely obfuscated, so the stack during send looks completely messed up and I can’t reasonably follow it that way. As I mentioned before, it’s been months since I started this project and honestly, I have no idea how I even backtraced the encryption function. I’m looking through my notes and I think I pieced together a global structure I called NetworkManager which contained structures I called MessageHolder and used information on how they relate and how MessageHolders are created/destroyed/modified in NetworkManager to find the encrypt function. I remember feeling like pulling my hair out doing this and jumping out of my chair in excitement when I finally found the encrypt function though.
After hooking onto the encrypt and decrypt functions, I started logging the unencrypted packets as an array of bytes, hoping to maybe just eyeball some information. Yeah, that was incredibly naive. Packets were flying by so fast I couldn’t even read them before the next batch came in. I figured that there had to be a sort of opcode for the packets somewhere. I’d usually expect them as the first few bytes, but they were actually 2-byte opcodes starting on the 4th byte. I found this out later, but the first 4 bytes actually contained a 2-byte packet length that had to be decoded in a certain way. I’m not actually sure why they did this. Possibly for redundancy?
\nFor a while, I was grouping the packets by opcode and trying to connect them to an action I did, like opening a menu, moving, going to a map, etc. I could connect opcodes to actions, but the packet body was basically a complete black box. Great, I know that upgrading a skill send a packet with opcode 0x17c, but what were the 6 bytes after? Skill ID? Amount of points invested? Maybe I could hazard I guess with such a small packet body, but I had no idea where to even start for longer bodies or variable length bodies.
I was doing this inefficient, stupid method until I was reading the decompiled source of the binary and just happened to stumble upon a constant that matched an opcode I knew. I searched the surrounding functions and sure enough, I found the code for constructing the packet for that opcode. I didn’t find this earlier because I didn’t even know what to look for. I mean, sure I could try searching the entire binary for an opcode, but there would be way too many hits. I didn’t know what packet construction looked like structurally either and I couldn’t backtrace it easily from before the encrypt function because the code was passing through an obfuscation layer during the encrypt function. From there, I noticed a function that looked suspiciously like an initializer for something like a Packet struct, so I hooked onto it and verified that I was correct. Since the initializer was called without obfuscation, I was able to hook onto it and print out the opcode it was initialized with and the call stack to see the rest of the packet creation. Using this, I was able to efficiently reverse engineer many packets. Unfortunately the creation of many of the sensitive packets, like heartbeat packets and packets to send client information, were called through an obfuscation layer, but there still were a ton of useful packets that weren’t obfuscated.
This applies to only outbound packets, of course. For inbound packets, I was never able to figure out how they were processed. I could find them being decrypted and I could make sense of some of the packets from just black-box reverse engineering, but after a considerable effort, I concluded that all of the inbound packet handling code on the client lived in obfuscation and there wasn’t any analog to the packet creation code for packet handling.
\nEventually, I got a bit bored of playing with the network protocol and decided to play around a bit with writing useful hacks. This is where I ran into a lot of trouble. When I was scanning for pointers to things like HP, MP, player position, etc., I would frequently get disconnected for “suspicious behavior.” I wasn’t sure if my bypass was failing or if they had server-side checks with heuristics for suspicious client behavior. I couldn’t reasonably make anything useful when I was frequently getting disconnected and sometimes banned. So instead, I decided I’d try to update some of the cheats people had previously published. I did this by downloading older versions of the game and finding what the cheats were modifying in the old version and then trying to find the analogous code on the current version and mimicking it. This was pretty successful! With only some effort, I was able to dodge every attack, disable collision, jump multiple times, etc.
\nUnfortunately, I would get banned a day or two after doing these more invasive cheats. Previously, I was just logging packets and I hadn’t been banned after I perfected the code. I was pretty confused at how I was getting caught because I figured that the packet logging should have been caught already. At a later time, I got into contact with another hacker who told me there was another integrity check I failed to bypass. I guess the basic bypass I had only allowed me to do certain things.
\nAs I was trying to figure out what was still getting me banned, my computer died, which lead to a wild goose chase lasting over a week trying to diagnose the problem through ordering new parts, testing them, failing, and repeating. The last thing I thought it would be was a CPU failure because I was booting into the BIOS fine and I could even boot into memtest. I replaced almost everything else — motherboard, power supply, GPU. The only things left of my original computer were my CPU and RAM. As I was testing my RAM to find out which ones I could keep in the new build and still crashing in the memtest, I noticed I was crashing on the same test. I searched up what that test did and it said it was a multi-core test. That’s when I finally realized it could be possible that my test results were consistent with one or more, but less than all CPU cores failing. Indeed, when I ran the memtest in single core mode, I didn’t crash.
\nI have never even heard of this happening and it caused me such a headache because I couldn’t figure out what was going wrong with the computer. I was about to make the Computer of Theseus!
\nDuring this period, I couldn’t continue on the project and I lost all of my momentum from the break. Later on, I got into contact with another hacker which is how I found out about what my bypass lacked. Unfortunately, I haven’t had the motivation to continue the project, so this is all I can write about for now. I wasn’t even going to write about this, but I figured I should complete the reverse engineering article series I started and give an update on where I was in regards to that. Be tuned for a part 2 in another… I don’t know, whenever I get bored enough again.
\n", "date_published": "2024-09-03T15:19:37Z" } , { "id": "https://blog.mnke.org/reversing-wings-of-vi-clickteam-fusion/", "url": "https://blog.mnke.org/reversing-wings-of-vi-clickteam-fusion/", "title": "Reversing Wings Of Vi/Clickteam Fusion", "content_html": "In my last post, I mentioned how I had been reverse engineering Wings Of Vi for the past month. Part of this included digging into the decompilation output from Ghidra and figuring out how the game works. I left it out of my previous post because there was too much to discuss and postponed it for another post. Well, here’s that post. I was going to talk about my workflows and my overall journey to get where I am today, but I think that’d still be way too long. More importantly, I don’t even remember some of the earlier steps of my journey. Instead, I’ll be splitting this post into two parts: the first part will talk about what I discovered myself and the second part will talk about discoveries after I had some help.
\nFor the first couple of weeks, I spent almost all of my free time banging my head against the wall, making hypotheses on what some code did based on Ghidra decompilation output and iteratively refining my understanding. I went through a ton of the code and annotated what I thought things were doing, labelled function names, etc. until I was nearing my limit for what I could figure out. I’ll present what I found as I was nearing that limit. Keep in mind, I had no idea if any of this was really correct. After all, it’s not like there was any way I could truly verify any of this.
\nFirstly, the game logic is run by what I called UpdateObjects. The game would iterate over these UpdateObjects and do various things commanded by them. Here’s what a typical UpdateObject looked like:
typedef struct {\n short size; // add this to the address of the current UpdateObject to get the next\n short code1; // index into a jump table. usually it's 2\n short code2; // index into a second jump table.\n char unknown[2];\n short index; // index into an Entity[]\n unsigned char flags1;\n unsigned char flags2;\n // rest is unknown. no defined size for this struct because it's variable length.\n // minimum size is unknown, but is at least bigger than size up to flags2.\n // often, there are "Expression"s at either offset 0x14 or offset 0x16 though.\n} UpdateObject;\nLet’s go through how a typical UpdateObject might’ve been used. Using code1, a function from a jump table is called and the UpdateObject is passed in. Using code2, another function from another jump table is called and the UpdateObject is passed in. Let’s take a specific case where code1 = 2 and code2 = 3. The resulting function after the two jump tables will:
Entity using the indexExpressionEntity->y to that valueActually, it’s a bit more complicated than that, but let’s focus on the UpdateObjects for now. The idea is that code1 and code2 encode a certain procedure for the UpdateObject and the index is what’s acted upon the procedure. The rest of the UpdateObject contains data on values (either directly stored or computed) that the procedure needs to act.
Furthermore, UpdateObjects may not always be run. Let me give some pseudocode for what was happening:
void fun1(int, int) {\n while (someCondition) {\n UOChainHeader* header = ...;\n UpdateObject* uo = header->updateObjects[0];\n bool shouldExecute = ShouldExecuteJumpTable[uo->code1](uo);\n if (shouldExecute) {\n Globals->numUOInChain = header->numUOInChain;\n ProcessUOChain(uo);\n }\n }\n}\n\nvoid ProcessUOChain(UpdateObject* uo) {\n int i = 0;\n UpdateObject* curr = uo;\n while (i++ < Globals->numUOInChain) {\n UOJumpTable1[uo->code1](uo);\n uo = (UpdateObject*)((int)uo + uo->size);\n }\n}\nBasically, fun1 would iterate through UOChainHeaders, which contained contiguous chains of UpdateObjects. The first UpdateObject contained a condition (usually dynamically computed) that signals whether to continue executing the rest of the chain. It turns out this is actually slightly wrong, but it’s accurate enough for now.
Let’s go back to how values are dynamically computed in UpdateObjects because this is fairly important. An UpdateObject may contain what I eventually called an ExpressionHeader. To compute the result of an ExpressionHeader, a global variable is set to the pointer of the start of the ExpressionHeader and then a function that I called getMagicNumber is called. getMagicNumber iterates through the payload of the ExpressionHeader and maintains a stack in a another global variable and operates on the stack. The payload within an ExpressionHeader contained Expressions which were variable length and iterated through a size field contained in the first few bytes. Using codes in these Expressions, we’d index into jump tables and operate on the stack. For example, an Expression with certain codes may contain an additional payload with information on how to index into the Entity array and find a certain value like an x coordinate and push it onto the stack. An Expression with another code may have a payload to index into a list of global values and push it onto the stack. Another Expression may compute the succeeding Expression, which will push a value onto the stack and then pop two values from the stack, and then add them. Other Expressions may even be used to compute sub-Expressions.
That was maybe a bit confusing, so I think it would be really useful to illustrate with by transforming the Expressions into something more legible. We don’t want to think about the variable length aspect of Expressions and we’d like to transform the codes into something more semantically meaningful. Let’s represent some Expressions in JSON:
[\n// Get a variable from this entity, push it onto the stack\n{ name: "GetEntityVar", entityIndex: 56, varName: "health" },\n// Compute the next Expression. Then, subtract the top two values on the stack\n{ name: "Subtract" },\n// Get this global variable, push it onto the stack\n{ name: "GetGlobalVar", index: 12 }\n]\nMuch clearer, don’t you think? Much of reverse engineering software is transforming data into something with meaningful semantics and this is no exception. Obviously, this is a lot easier said than done. You should’ve seen me bang my head against the wall for days trying to figure this single function out.
\nThere are a lot more details on not only what I’ve talked about, but also things I haven’t talked about. I’m choosing to omit them because I think what I’ve discussed gives a clear enough picture of how the game operates at a high level: the game is actually like an interpreter. It processes directives through the form of UpdateObjects and Expressions and updates the game using those directives. This naturally leads us to ask: “Where are these directives?”
So the hunt for where the UpdateObjects were stored began. The executable for the game was pretty large (90MB) and there wasn’t anything else in the installation directory, so I figured the data should be within the executable itself. Furthermore, I should be able to see where in the code this data is being loaded. I did find it, but it was part of a huge function that Ghidra had trouble decompiling and it seemed to do other things from a DLL that was also packed in the executable, extracted into a temp directory, loaded, and then used to transform the data.
Long story short, I was unable to figure out how to take the data from within the executable and transform it to what’s used during the game’s runtime. During my weeks reversing the game, I had realized that the game was made with a tool called Clickteam Fusion. Desperate, I searched for tools to help with reversing Clickteam Fusion games and I found a tool called Anaconda. Honestly, I had my doubts about this tool because it was really poorly organized: copies of files, executables directly in the Git tree, build artifacts floating around, etc. After wrangling with getting it installed and changing some settings, I actually managed get some output though. It gave me a file that to open in the Clickteam editor but it wouldn’t load correctly. I wasn’t planning on continuing my reversing efforts in the Clickteam editor, but I wanted to confirm that this tool wasn’t giving me completely bad output so I got into contact with one of the authors of Anaconda and he very graciously gave me some tips to get it loaded. And… it actually worked. Well, I was still getting some errors when loading the file and there were clearly artifacts, but it was recognizable. Great, now that I had confirmation it mostly worked, I started digging through the Anaconda source code to see how it worked its magic.
\nWow. I was right about so many things. I mean, of course we had different names for the same thing, but it was clear that their Event referred to the UOChainHeader I talked about and their Actions corresponded to UpdateObjects. Hell, we both settled on the same name for Expressions! It’s hard to put into words how I felt reading that code. I had spent the better part of a month working blindly, unable to truly verify if any of my ideas were correct. Seeing confirmation made me so happy.
In order to make sure I understood how the game was loaded (at least the Events), I wrote a C program to read the file that closely followed the Anaconda code. I think my code does a pretty good job of documenting how Clickteam games pack their data, so go read my code if you’re interested in it.
The most trouble I had was figuring how to decoding and decompressing some parts of the data. In fact, I still don’t really understand how to decode some of the data, and neither does Anaconda. It just calls some C++ function that’s clearly the output of some decompiler (probably IDA). I couldn’t even manage to do that though. The decoding algorithm required a key that I was too lazy to figure out how to generate. I knew it was generated deterministically based on the contents of the executable, but I just hooked onto the key generation function and saved its output. Then, I tried to naively copy and paste the decoding algorithm from Ghidra’s output and use it in my function, but it just didn’t work. I tried to use Compiler Explorer and modifying the algorithm bit by bit to match the right assembly, but this was still really difficult because I don’t know what compiler was used, flags, etc. It would be really difficult to make it match. So I said screw it and just decided to copy/paste the assembly code as inline GCC assembly.
\n![\"\"](\"https://blog.mnke.org/img/1jpYvUmvp--480.png\")
After wrestling with translating Intel assembly to AT&T assembly and compiling my program in 32-bit, I checksummed my data and confirmed this actually worked.
\nAnaconda contained another tool called Chowdren that was supposed to take a Clickteam game and transform it into C++. It didn’t work for Wings Of Vi even after I tried patching the errors that were popping up, but there was clearly something to it. I began to wonder how the author knew enough about the Clickteam runtime to build such a tool. I asked the author of Anaconda, but he said that he didn’t actually write Chowdren. I ended up finding another tool called CTFAK which seemed to be an updated rewrite of Anaconda alongside an active community working to decompile Clickteam games. I asked them if there was any documentation on the Clickteam runtime or if they knew anything about it and it turns out, the source code for the runtime is just sitting in the Clickteam Editor’s install directory.
\nWhat the fuck? Are you telling me I’ve been working tirelessly on decoding bytecode piece by piece when there was perfectly legible source code out there?
\nWell, okay. It’s not actually the source code for my runtime exactly, but it’s pretty damn close. It contained source code for the Android runtime in Java and XNA runtime in C#, so it wasn’t exactly the Windows runtime I was using, but all of the logic was clearly preserved when I compared the decompilation to the high level source. Furthermore, Clickteam published header files for some of their runtime structs. It’s not super easy to tell what the low-level structs look like from Java or C# code, so that supplementary material was extremely helpful. Furthermore, the person who gave this information even gave a CE structure file that represented most of the runtime headers!
\nWell, I’ll be damned. That person also told me that I should’ve immediately asked around communities when I start reverse engineering projects. I probably will take that advice in the future but for this project in particular, I’m glad I didn’t do that. Staring at the disassembly and decompilation for hours and figuring things out that way is a skill I wouldn’t have learned otherwise.
\nNow that I have the source code, how right was I about how the game ran? Pretty fuckin’ right! I’m a BEAST! Almost all of what I described above (before reading the source) was accurate. I didn’t describe a lot of other things, but those were accurate as well. Literally the only thing that I was wrong about was the conditions and it was a relatively minor mistake. It turns out, an Event (a.k.a a UOChainHeader) has multiple Conditions and Actions. I mistook Conditions and Actions for both being UpdateObjects. To be fair, they actually are the exact same, except a Condition has an extra short id at the end. That’s why I thought the Expression was either at offset 0x14 or 0x16. Additionally, I thought only a single Condition was checked before running the Actions. It turns out, there can be multiple Conditions within a single Event. The Ghidra output made that really unclear because there were tons of nesting, loops and gotos in the function.
Out of the things I regarded as having figured out, I was almost always correct. There were plenty of things that I just didn’t get around to figuring out though, mostly because it took so much time. Either way, this gave me a lot of confidence that my methods for reversing binaries were effective and that the biggest optimization to my methods currently is to get faster at it.
\nSo what now? I’ve more or less satiated my desire to figure the game out. I mean, I basically have access to the source code.
\nI’ve been spending some time creating a modding API by allowing hooking onto certain functions (e.g. action handlers, condition evaluators, etc.), and giving access to functions in the game like looking up an entity or getting a global variable, but I don’t think this is a great way to mod Clickteam games ultimately. Sure, I can get things done like this, but intercepting an action handler and deciding whether it should be run is not that easy. You have to figure out if you have the right action, which requires you to understand what the action does. Doing anything more than just blocking an action from running can be really difficult. Since the game is essentially an interpreter, the best way to mod the game will probably be to modify what’s being interpreted. My interest ends here though. I’m interested in reverse engineering, not making a modding tool for an out-of-date, obscure game engine.
\nThis post isn’t meant to be as informative as the previous post, so sorry if you made it all the way here and didn’t get much out of it. I just wanted to share my progress with low-level software reverse engineering. I’ve made strides in this area over the past month and a half, and I’m proud of how far I’ve come.
\n", "date_published": "2024-05-18T17:31:33Z" } , { "id": "https://blog.mnke.org/reverse-engineering-binary-applications/", "url": "https://blog.mnke.org/reverse-engineering-binary-applications/", "title": "Reverse Engineering Binary Applications", "content_html": "I love taking things apart, playing around with what’s inside of them, and figuring out how it works. It’s like working on a giant puzzle, where you only know bits and pieces of the final picture based on how the assembled object behaves, and you have to fill in the missing parts based on deduction. Although my expertise in reverse engineering lies mainly in computer science, I think my love of the craft extends universally.
\nJust last summer, I was taking apart my electric piano to restore color to its yellowed keys, and in doing so, I gained a pretty good understanding of how it worked. The piano could play notes softer or louder based on how hard I struck the key, and I’d wondered what the physical mechanism behind that was. Turns out, it was incredibly simple! Underneath the key, there were two ridges placed at difference offsets from the fulcrum with heights such that one ridge would strike a button before the next. These signals are sent to a controller that calculates the delta between the signal timings to calculate an amplitude, which is then sent to an audio interface.
\n![\"\"](\"https://blog.mnke.org/img/A48il3XIPP-480.png\")
Well, presumably. It’s not like I pulled the machine code out of the controller and analyzed its disassembly. That might’ve been possible, but seeing the ridges and buttons underneath each key gave me enough confidence in my hypothesis.
\nAnyway, I’m not actually here to talk about the mechanisms of an electric piano. I want to talk about my last month and a half disassembling a binary for an old game I used to play: Wings Of Vi. I’ll be talking about… well, a lot. I learned a lot. Methods to reliably find important addresses, how Windows loads and allocates memory in its virtual address space, assembly, function hooking, disassembling and decompiling, etc.
\nMy background in binary analysis had been pretty limited before this. The reverse engineering I’d done in the past involved looking at C# IL (intermediate language), minified JavaScript, or sometimes even code that other people had already reversed to a high-level language like C++. All of these were relatively easy to read and reason about. Reading disassembly is a huge step from that. But first, let’s go back to the start of my project.
\nI’d like to say I had a concrete goal at the beginning like creating a cheat interface or a modding API for the game or something like that, but I really didn’t. At the very least, my goals fluctuated throughout the project.
\nI began with a mini-goal to give myself infinite health. I launched Wings Of Vi, attached CE (Cheat Engine) to it, and started looking for my health address. This is done by:
\nI had a ton of trouble with this in practice. See, those steps assume that the memory address for your health doesn’t change throughout the iterative procedure. After all, if your health address changes from 0x12345 to 0x23456 during this, CE has no idea and is going to continue comparisons at 0x12345. The problem was that enemies hit too hard and there was so little healing that I couldn’t stay alive long enough. Once I respawned, I couldn’t be confident that my health address was still in the same place. How can we find the health address then?
\nTo answer this, we need to talk briefly about how things are actually stored in memory. As a programmer, I may define a struct like
\nstruct Entity {\n int x;\n int y;\n int health;\n}\nWhen an Entity is instantiated, a region of contiguous memory is allocated to it, say at address 0x800. The x value is offset 0 from that address because it’s the first value in the struct. Then, since ints are 4-bytes long (yeah, this is technically decided by the compiler, but 4-byte ints are pretty typical), the y value is going to by at offset 4 from that address. Lastly, the health value is going to be at offset 8 from that address.
Okay, so how does that help us find the health address? Well, we can hypothesize that the programmer might’ve defined the player entity in a similar way, meaning that the x and y coordinates of the player entity are close to the health address. If we can find the x or y coordinate of the player entity, we might be able to find the health address nearby, and the coordinates of the player entity are a lot easier to manipulate than the health!
\nDoing this can still be a little bit tricky in practice though, and I want to give a few quick tips.
\nSee, if you’re playing a 2D game, the y coordinate may increase as the player entity goes down because that coordinate system is often easier for computers to understand. This means you have to make some assumptions on the coordinate system if you’re trying to get the y coordinate. However, it’s almost always the case (from my experience) that x coordinates increase as the player goes right. For that reason, I recommend trying to find the x coordinate (which will often lead you to the y coordinate) for 2D games.
\nFor 3D games, I recommend the opposite. For convention, I’ll refer to the y coordinate to define the up and down of an object while the x and z coordinates define the planar position (the plane parallel to the ground). I’ve found that the y coordinate tends to increase as the player entity moves up in 3D games while x and z coordinates… well, there are too many degrees of freedom to begin making solid hypotheses. You can’t even figure out what directions are aligned with the planar axes most of the time, much less make any deductions on which direction increases the x or z coordinate.
\nSince Wings Of Vi is a 2D platformer, I did a little dance left and right and eventually found my x coordinate. Browsing the memory region where the x coordinate lived on CE while moving around and taking damage, sure enough, my y coordinate and health were pretty close. They were less close than expected though. All of these values were defined as doubles, which took 8 bytes, but the values were spaced out 16 bytes instead, meaning there were an unknown 8 bytes in between each value. It was a bit odd and I took note of this, but didn’t think too much of it at the time (though I now understand why this is the case).
I “locked” my health with CE (really, it’s just CE rewriting the value over and over again really fast) and tried taking some damage. Nice! I wasn’t taking any damage. I’d already confirmed that my health address changes every time I respawn, meaning that I’d have to go through this whole ordeal every time to become invincible. Or do I?
\nLet’s talk theory again. The addresses for player entity values does change on respawn, but the game clearly knows where these values live. This means the address for the player entity values itself has to be stored somewhere. Does that mean we should search in memory for the health address? Not quite. See, x86 (and probably other instruction sets) allow instructions to write at a memory address plus an offset. In the Entity example above, where the start of the struct is stored at 0x800, the instruction to set the entity health to 50 might look something like mov [0x800+8], 50 rather than mov [0x808], 50. Furthermore, the 0x808 address for health is likely never stored anywhere and only stores the 0x800 address for the start of the Entity. If you think about it, this makes a lot of sense. The game knows that the Entity struct is at 0x800 and the health is at offset 8 from it. The game doesn’t need to remember the health address, only where the start of the Entity is, and that the health address is 8 bytes away from it.
So, the idea is that we identify where our Entity struct starts and where the address to the Entity struct is stored, and hopefully this address holds the Entity address even after respawning. What if it doesn’t though? It could be that the player entity address is stored in a dynamically allocated array that’s reallocated on respawn. Okay, well something must hold the address to that array. We keep going up these level of pointers until we hopefully find a static address.
Static addresses are exactly like they sound: static. Well, up to a certain point. Windows loads modules like an executable file or DLL by giving them a memory region for their data and code. Let’s say I have a program MyApp.exe that loads the User32.dll file to draw a window. When MyApp.exe is run, Windows will allocate a certain region of memory for it in the virtual address space, usually specified by a field in the PE (portable executable) headers of the executable. Then, it’ll load the User32.dll module in another region of memory. Static addresses refer to a fixed offset from the start address of a module, usually located in the .data section of the code. Often, this also translates to a completely static address because modules are loaded in the same way, but we shouldn’t rely on that. Fortunately, we don’t have to because the Windows API provides a function to get the start address of a module.
This gives us a plan: follow the pointers and offset all the way up until we find a static address. Doing so will allow us to get the address of what we want through respawns and even through closing and reopening the game. Let’s take an example before we continue.
\ncmp [eax+8], 0 where eax is 0x140000. I conclude there is a struct at address 0x140000.mov eax, [ebx+0x100] where ebx is 0x78000. I conclude there is a struct at 0x78000.Then, we can find the health address by “undoing” the pointer chain we found. Here’s some C-style pseudocode to illustrate:
\nchar* baseAddr = GetModuleBase();\nchar* firstLevel = *(char**)(baseAddr + 0x5500); // dereference 0x45500, get 0x78000\nchar* secondLevel = *(char**)(firstLevel + 0x100); // dereference 0x78100, get 0x140000\nint health = *(int*)(secondLevel + 0x8); // dereference 0x140008, get health\nPutting this in practice, CE gives us a wonderful utility to see what instructions access a memory address, allowing us to figure out the offset, and CE lets us find exact values to see where addresses are being stored. It also highlights static addresses in green for us so we know when we’re at the end.
\nSome tips around getting these multi-level pointers:
\naddr2 + offset2 = &addr1. Seriously, it helps.mov [eax+0x14], ebx and another may be mov [eax+0x38], ecx. Note this and choose one offset to branch into. If it doesn’t work, come back and branch the other ways.mov [eax+ebx], ecx. CE will tell you what was stored at eax and ebx on the first time the instruction was detected. The smaller number of eax and ebx will be the offset and the larger one will be the memory address. Usually, the first operand of the + is the memory address.In reality, CE actually has this “pointermap” and “pointerscan” thing that’s supposed to help in this otherwise tedious process, but I haven’t really figured out how to use it. If this is something I’m going to do often, I’d probably look into that.
\nCE has built-in functionality to deal with these multi-level pointers so we can save the multi-level pointer to our health address and recover it upon closing/reopening the game/CE. This is kind of lame though: I want to be able to press a button to give myself +100 health. We can do this by creating an external or internal hack. An external hack is a program that uses the Windows API to read/write memory of another process. Comparatively, internal hacks run in the same process as the game and directly access the process memory. The advantage of internal hacks is that memory read/writes are a lot faster and hooking is a lot easier. I think a disadvantage of internal hacks is that games with anti-cheat can easily detect internal hacks. I haven’t written any hacks for games with anti-cheats yet, so I’m not sure. Either way, let’s talk about how to create an internal hack.
\nThe idea behind the internal hack I’m about to discuss is that, we make the game process call a function to load our code in another thread by loading a library — our DLL. Our DLL runs on its own thread and does whatever it wants to the game. This method is called DLL injection. The Windows API exposes the two important functions:
\nCreateRemoteThread — Creates a thread that runs in the virtual address space of another process.LoadLibrary — Loads the specified module into the address space of the calling process. The specified module may cause other modules to be loaded.All we have to do is make CreateRemoteThread call LoadLibrary within the remote process and load our DLL. There’s a little bit of complication in that LoadLibrary needs to accept the path to our DLL as an argument, and we can’t just use a string that’s allocated in the program we’re injecting from because LoadLibrary only has access to the remote process’ memory space. A call to VirtualAllocEx to allocate some memory in the remote process and WriteProcessMemory to write our string to that remote process’ memory makes it work though. Altogether, our injector might look something like:
char* dll = "path\\\\to\\\\dll.dll";\nunsigned length = strlen(dll) + 1;\n\n// find process\nDWORD procId = GetProcId("Game.exe");\nHANDLE hProc = OpenProcess(PROCESS_ALL_ACCESS, 0, procId);\n\n// write the argument to pass to LoadLibrary in remote process memory\nvoid* remoteStr = VirtualAllocEx(hProc, 0, length, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);\nWriteProcessMemory(hProc, remoteStr, length, 0);\n\nHANDLE hThread = CreateRemoteThread(hProc, 0, 0, (LPTHREAD_START_ROUTINE)LoadLibraryA, remoteStr, 0, 0);\nCloseHandle(hThread);\nCloseHandle(hProc);\nI don’t actually use this because CE has a DLL injector built in, but I think it’s important to understand how it works. I’m pretty sure it’s more advanced than what I wrote, but it should follow the same idea: call LoadLibrary within the remote process.
From within the DLL, we’d create a thread to run on, likely an AllocConsole in case we need to print any debug information, and likely some GetAsyncKeyState calls to respond to the keyboard. Then, we can use the code above to unravel the multi-level pointers to read/write whatever we want.
All of this can probably get you decently far in modifying the game at runtime, but I wanted to really dig in deep. I wanted to figure out how the main game loop worked. I wanted to know what entities existed, where in memory they live, and how they run. To do this, we need a decompiler.
\nThe only decompilation options I know of are IDA and Ghidra. IDA is a proprietary software made by HexRays and it comes in a free and a pro version, and some others. The free version is apparently lacking in tons of important features but the pro version costs literally thousands of dollars, which is way too expensive for a hobbyist like me. Ghidra is an open source decompiler created by the NSA. It’s apparently nowhere near as polished as IDA Pro though. I did try IDA Free for a little while, but honestly, I just didn’t know how to use it well and I was somewhat discouraged to learn because I knew I was missing important features and tutorials were usually made in IDA Pro. So I switched to Ghidra, and despite people’s criticisms, I found it incredibly easy to pick up.
\nTo get started, I decided to break on the instructions that wrote my coordinates in CE and follow the stack traces up until I could no longer step out of the stack because I was in a while loop. With the Ghidra decompilation on the side, I began to figure out the code by making hypotheses from more readable Ghidra decompilation output, test them in the game with CE and repeat. Ghidra’s auto-struct function, re-labelling, and re-typing started making the code more and more readable. Over the course of the next couple of weeks, I would read the disassembly/decompilation and figure out much of the code.
\nSome of the tools that were incredibly helpful during this time were CE’s structure dissector and a tool called ReClass to let me reverse structs. It would attach into the game and you could enter in memory addresses and give addresses types like declaring that offset 0x4 from an address is a short, or offsets 0x20 to 0x40 contained a string. Importantly, it let me visually see the data in multiple forms and made guesses to what type it thought an address was which might’ve been difficult in static analysis.
Seeing how long this post is getting, I’m deciding to dedicate another post to purely my decompilation findings. Stay tuned for the next post. It’ll be packed to the brim with how the game is implemented.
\nAnother tool I picked up during this time was function hooking. This tool extends far beyond just helping me figure out the code, so I’m going to spend a bit of time talking about it. You’ve probably at least heard of function hooking and have some idea of it might be: we “hook” onto a function A by executing our own function B whenever function A runs. This is the high level idea. This is really not too difficult to do in higher level languages: you’d probably just write a line at the start of the target function that calls your function:
\nvoid Target(int a) {\n MyFunc(a);\n // original function\n}\nWhen you’re working with bytecode, you can’t just do this though. Let’s think this through on a x86 32-bit (this is what I was working with, but obviously everything translates pretty easily into 64-bit). Suppose we’ve created a valid assembly function within the process’ memory at address 0x12345. The start of the target function looks like this:
\n1: push ebp\n2: mov ebp, esp\n3: mov eax, dword ptr [ebp+0x8]\n4: cmp word ptr [eax+0x10], 0x18\n5: ...\nWe can’t just insert before the first line of the function to call 0x12345 or jmp 0x12345! We’d have to shift the entire rest of the code below this function down! As if that wasn’t impractical enough, certain jmp instructions may rely on some code being at a certain offset, which a shift like this would completely destroy.
Okay, let’s be a little smarter. How about we just overwrite the first instruction to jump to our code? So something like:
\n1: jmp 0x12345\n2: mov ebp, esp\n3: mov eax, dword ptr [ebp+0x8]\n4: cmp word ptr [eax+0x10], 0x18\n5: ...\nThere are multiple major problems with this. Firstly, x86 assembly instructions are variable length. A push ebp instruction is only 1 byte long while the jmp 0x12345 is at least 5 bytes long. I’ll annotate the instruction lengths to demonstrate:
1: push ebp // 1 byte\n2: mov ebp, esp // 2 bytes\n3: mov eax, dword ptr [ebp+0x8] // 3 bytes\n4: cmp word ptr [eax+0x10], 0x18 // 5 bytes\n5: ...\nLet’s fix that first by writing exactly 5 bytes for the jmp instruction.
1: jmp 0x12345\n2: or byte ptr [esi-0x7d], ah\n3: js 0x1a\n4: ...\nWait, what happened? Well, we can’t write only 5 bytes! By writing exactly 5 bytes, we mangled the 3rd instruction! We have to write at least 5 bytes, plus however more bytes aligns us to the next instruction. Fortunately, a nop (no-operation) instruction is 1 byte, so we can use it as padding. If we write the 6 bytes correctly, we should get:
\n1: jmp 0x12345 // 5 bytes\n2: nop // 1 byte\n4: cmp word ptr [eax+0x10], 0x18 // 5 bytes\n5: ...\nAlright, but this still has one minor problem and one major problem. The minor problem is that we have to make sure that we jump back to the correct address after executing our function. Suppose we do that. Then, the major problem is that the stack is usually very delicately and meticulously handled by the compiler and the instructions we overwrote modify the stack. If we naively just ran our function we’d destroy certain stack invariants and crash the process almost immediately. Trust me, I know from experience. So before we transfer control back to the target function, we have to be absolutely sure that the stack is in the condition it would’ve been in if we didn’t have this jmp instruction. The easiest way to do that is (assuming the stack is as the same as when we initially jumped to our code) to execute the overwritten instructions.
How do we actually write code for our function though? Sure, we can write raw assembly and make our program jump to that raw assembly, and the MSVC compiler even supports pretty decent inline assembly:
\nvoid __declspec(naked) FakeFunction() {\n __asm {\n mov eax, [ebx]\n cmp eax, 1\n // ...\n // run the original instructions\n jmp 0x98789 // jump back to where we came from\n }\n}\nThis kind of sucks though. It’s super hard to get anything meaningful done with pure assembly. Not only that, it’s still easy to just mess up the stack! If we call a normal, non-naked function, chances are, we’re going to ruin the stack. After a bit of research and seeing how the MS Detours library does it, this is what I came up with:
\n0x10: call TargetFunction\n0x15: mov [eax], ebx\n\n// Get the function signature through Ghidra decompilation. Usually reliable enough\n// int __cdecl TargetFunction(int, int)\n0x50: jmp 0x12345 // we wrote this. jumps to code we wrote\n0x55: nop\n0x56: cmp word ptr [eax+0x10], 0x18\n...\n0x99: ret\n\n// below is our code\n\n// match the original function signature\ntypedef int(__cdecl* TargetFunctionT)(int, int);\n\nTargetFunctionT RealFunction = 0x12349;\n\n// match it here too\nint __cdecl FakeFunction(int a, int b) {\n printf("hooked before. args are %d, %d\\n", a, b);\n int ret = RealFunction(a, b);\n printf("hooked after. args are %d, %d\\n", a, b);\n return ret;\n}\n\n// "trampoline"\n0x12345: jmp &FakeFunction\n0x12349: push ebp\n0x1234A: mov ebp, esp\n0x1234D: jmp 0x56 // go back to where we came from\nWait, it might be better if I explained with a picture.
\n![\"\"](\"https://blog.mnke.org/img/hlriwxN9VW-480.png\")
… don’t look at me like that.
\nSeriously, I do think this visualisation makes a good trade-off between showing what’s actually happening and minimizing the complexity. I’ve simplified the jmp and calls to be the near variants for illustration, but this is actually what’s happening.
The idea is that we “hijack” the target function by jumping to our function immediately through a “trampoline”. The function we jump to immediately must match the target function’s calling convention exactly, including the calling convention so we don’t mess the stack up. The rest of the trampoline is dedicated to pretending to be the real function. If you follow address 0x12349 through, you’d be running the exact same code as the original target function. That’s what’s stored into RealFunction.
In the codebase I was reversing, there are many instances where a function is called through a jump table. I don’t actually know if this is a widely accepted term for what I’m going to describe, but it makes sense to me so I’m rolling with it. What often happens is that a “type” is specified in the first few bytes of a struct. This “type” is used to index into an array of function pointers located at a static address, and the appropriate function is called using the type. Naturally, each of these functions has the exact same function signature. I want to hook onto each of these functions. Sure, I could write FakeFunction1, FakeFunction2, etc., but I’d ideally like to have a single FakeFunction that handles all of these functions and calls the original function. How would FakeFunction know which one to call though? In some cases, the struct pointer with the type was passed into the function, so I could use the type and figure it out, but I didn’t want to rely on that in cases where I couldn’t do that.
So the goal is to inject an extra argument for our FakeFunction to receive. This is actually super hard in practice and I haven’t pulled it off, but in theory, you’d obviously need to add another argument to your function signature, and then you’d have to do something like this at the start of the trampoline (assuming a __cdecl call with originally 1 argument):
push esp // in case we mess up\npush eax // save eax on the stack so we can use it worry-free.\nmov ebx, [esp+0x8] // this will be the first argument to the function (32-bit stack entries)\npush something // this is the argument that will be injected. second argument\npush ebx // add the original argument. first argument\ncall FakeFunction\nadd esp,0x8 // get rid of the two arguments we used\npop eax // restore eax\npop esp // restore esp just in case\n... // run original code here\njmp returnAddr // go back to where we came from\nThe downside of this is that we’re calling our FakeFunction instead of jumping to it, meaning we can’t just skip running the target function (jumping basically completely hijacks the target function and allows us to early return). Yes, we can set up the trampoline in a way that we rearrange the stack and push an extra argument and jump to our FakeFunction, but the call to TargetFunction (and thus FakeFunction) may expect the stack pointer to be at some location, and the extra push instruction causes it to be a couple bytes higher than expected and destroys stack invariants. Of course there are still other ways to do an early return like storing a boolean that the trampoline reads when it returns from the call, but this is already getting out of hand.
Not only that, but handling different function signatures is a huge pain. I don’t know if that code works in the first place, but it definitely wouldn’t work if the target function had 2 arguments, or 3 arguments, or anything other than 1!
\nUltimately, the best solution I came up with was to:
\nint extra;FakeFunction is called so it can call the real functionThis actually works really well. I created a mini-framework for this, and it’s been holding up.
\nAside from this, I’ve also messed around with mid-function hooking, but this is generally really difficult to get right and not even that useful most of the time. I originally wrote a hooking library tailored to mid-function hooking while still allowing start-of-function hooking because I thought it’d be cool, but I just didn’t find myself using the mid-function hooking functionality that often. If you want to do something like skipping an instruction mid-function or saving a variable to a global list you manage, you don’t get much privileges of being able to write high-level code because you risk mangling the stack — and even register values (although this can usually be made a bit safer with pushad and popad and pushing/popping FP registers before and after transferring control flow, respectively). Most of the time, start-of-function hooking is just good enough.
Well, I don’t know what to put here. I don’t really have a conclusion since this post was just an assortment of learnings I’ve had over the past month and a half. I’ve had a ton of fun with playing around with this stuff at such a low level. This knowledge unlocks so many options and I’m excited to put it to use. If it’s not clear, I don’t condone abusing this knowledge to cheat in online games. I think the tone and content of my discussion hinted toward my technical interest rather than an interest to cheat. Cheating in singleplayer games is totally okay (and I like doing it to understand the game), but if you cheat in online games to pretend like you’re good, you’re super lame.
\n", "date_published": "2024-05-16T16:57:03Z" } , { "id": "https://blog.mnke.org/problem-solving-with-vim-advent-of-code-day-1/", "url": "https://blog.mnke.org/problem-solving-with-vim-advent-of-code-day-1/", "title": "Problem Solving with Vim: Advent of Code Day 1", "content_html": "A while back, I had seen someone do some insane things with Vim: things that really should be done in a programming language but is somehow done in Vim. I was inspired, but not enough to go out of my way to find problems to solve in Vim. Fast forward to yesterday, when a friend messaged me his solution to the day 1 Advent of Code problem in an esolang. That gave me the idea to try solving it in Vim, trying to avoid deferring to vimscript or executing outside programs as much as possible — I wanted to solve things almost purely by manipulating the text.
\nToday, I gave day 1 a shot. The first part was really simple, even in Vim. You were given a newline-separated file of alphanumeric strings and all you had to do was take each line, form the first and last numbers of the line into a two digit number and then add up all the numbers. In Vim, that just meant
\n:%s/\\D//g0v"ay$v"by and then paste it with "ap"bp+ using :%s/\\n/+/g+, cut the entire line into the register with 0C, and then use expression mode to evaluate the register content with <C-R>=<C-R>"Where things got a bit more difficult was in part 2. First, you had to convert spelled out digits like one into 1 before doing the same thing as part 1. I wanted to just run %s/one/1/g, %s/two/2/g, ..., but the example showed that words could overlap: you could get words like xtwone3four, which should get converted into 24, but obviously doesn’t with the naive substitutions I wanted to do. Additionally for a more illustrative example, something like 1twone should come out as 121, and then 11, which I didn’t realize until a while later.
The first problem I was thinking about was how can we even know in Vim that we should substitute the two in xtwonefour first before the one. I didn’t end up exactly figuring that out, but it was no matter because I realized that ultimately, all we need to do is resolve the first and last spelled out word and we could ignore everything in between. This is what we do to get xtwonefour into x2ne4:
yy and paste 8 copies with 8P:s/one/1/e)So now we’re at x2nefour. Resolving the last spelled out word is very similar to doing it for the first:
yy and paste 8 copies with 8P:s/.*\\zsone/1/e)!rev here)Now we have x2ne4 and so we can just delete non-digits and do what we did for part 1.
This works for almost all lines, but there’s a special case. For lines with only a single set of two overlapping number words, this fails. Let’s take a look at 3twonep for example, which should convert into 31. With the first set of steps, we’d get 32nep and then… oh we lost the one to replace to 1. After scratching my head a bit on how to resolve this, I came up with this.
3twonep\\n3twonep)32nep\\n3twonep)32nep\\n3tw1p)32nep3tw1p)32nep3tw1p -> 3231 -> 31)This works! It works because we don’t care about what’s happening in the middle of the line. The first digit will always be either already a digit or resolved by us from the first set of steps, and we’re still guaranteeing that the last digit of the combined line will also either already be a digit or resolved by us from the second step of steps, and it doesn’t matter that we have all that junk in the middle.
\nI haven’t felt so happy after solving a puzzle in a long time. From realizing the trick with sorting to extract the first replacement of a digit to realizing the trick with reversing to extract the last replacement to realizing I could just parallelize and then combine the two tricks, I was really forced to think out of the box in a dramatically different way due to my unusual and limited toolset, and I managed to do it. Here’s my full solution with some annotations.
\nGo^[ " add a trailing line to the file. \n " For some reason, the macro below fails if there isn't another line\n " below the one it's operating on.\ngg0\nqqyy9P " First set of steps\n:s/one/1/e " Replace first occurrences\nj:s/two/2/e\nj:s/three/3/e\nj:s/four/4/e\nj:s/five/5/e\nj:s/six/6/e\nj:s/seven/7/e\nj:s/eight/8/e\nj:s/nine/9/e\nV8k:sort " Sort\njV7jd " Delete copies\nyy8P " Second set of steps\n:s/.*\\zsone/1/e " Replace last occurrences\nj:s/.*\\zstwo/2/e\nj:s/.*\\zsthree/3/e\nj:s/.*\\zsfour/4/e\nj:s/.*\\zsfive/5/e\nj:s/.*\\zssix/6/e\nj:s/.*\\zsseven/7/e\nj:s/.*\\zseight/8/e\nj:s/.*\\zsnine/9/e\nV8k!rev " Reverse\nV8j:sort " Sort\njV7jdkV:!rev " Delete copies, reverse remaining line back to original\nkJr,jq " Combine lines, end macro\nVGk:norm @q " Run macro on rest of lines\n:%s/\\D//g " Delete digits\ngg0\nqw " Record macro into w\nv"ay$v"by " Copy first character into register a, last character into register b\n0D"ap"bp " Delete the entire line, paste from register a, paste from register b\nj0q " Go to start of next line, end macro\n:%norm @w " Run on all lines\n:g/^\\s*$/de " Remove all empty lines, ignore errors\n:%s/\\n/+/g " Replace newlines with +\n$x " Delete trailing newline\n0C " Cut line into default register\n<C-r>=<C-r>" " Evaluate expression in default register \n<ESC>\n\nR28bZ2cwcXF5eTlQOnMvb25lLzEvZQ1qOnMvdHdvLzIvZQ1qOnMvdGhyZWUvMy9lDWo6cy9mb3Vy\nLzQvZQ1qOnMvZml2ZS81L2UNajpzL3NpeC82L2UNajpzL3NldmVuLzcvZQ1qOnMvZWlnaHQvOC9l\nDWo6cy9uaW5lLzkvZQ1WOGs6c29ydA1qVjdqZHl5OFA6cy8uKlx6c29uZS8xL2UNajpzLy4qXHpz\ndHdvLzIvZQ1qOnMvLipcenN0aHJlZS8zL2UNajpzLy4qXHpzZm91ci80L2UNajpzLy4qXHpzZml2\nZS81L2UNajpzLy4qXHpzc2l4LzYvZQ1qOnMvLipcenNzZXZlbi83L2UNajpzLy4qXHpzZWlnaHQv\nOC9lDWo6cy8uKlx6c25pbmUvOS9lDVY4ayFyZXYNVjhqOnNvcnQNalY3amRrVjohcmV2DWtKcixq\ncVZHazpub3JtIEBxDTolcy9cRC8vZw1nZzBxd3YiYXkkdiJieTBEImFwImJwajBxOiVub3JtIEB3\nDTpnL15ccyokL2RlDTolcy9cbi8rL2cNJHgwQxI9EiINGwoK\nIt’s base64 encoded because I can’t paste things like ESC (^[). To run it, save that encoded file somewhere, run base64 -d file > solution and then vi -s solution input.txt.
I had an insane Minesweeper phase where I could not stop binge-playing the game. I would literally see Minesweeper patterns in my head and imagine solving them when I wasn’t playing. I’ve since become less fanatical and now only play Minesweeper on occasion. In fact, my personal best (69 seconds on expert) was accomplished during a short return to the game, which tells me I’m far from reaching my true limit, so I’ll be back for more.
\nIn any case, during my Minesweeper escapades, I started playing on a site called Minesweeper Online due to its exclusive “No Flag” mode, where you’re only given boards that you can always deterministically solve. The site is simple with regards to the modern web, but maybe a bit much for Minesweeper with all the collectibles and gems and events and whatnot. I actually wanted to implement my own Minesweeper with my own No Flag mode and solver for fun. I tried doing that until it was no longer fun. Also, shoutout to Minesweeper Online for giving me premium for life after reporting a minor bug. I appreciate you guys.
\nA couple of years ago, I had an idea to shock myself as punishment when I failed at something in video games, inspired from a video I watched. I thought it would be pretty funny to strap myself into a figurative electric chair while I played Minesweeper and shock myself whenever I clicked a bomb, and frame the entire thing as a “science experiment” on whether shock punishment was effective for as a conditioning mechanism for improving at Minesweeper. My friend and I jerry-rigged a dog shock collar with a Raspberry Pi to allow shocks to be delivered through an HTTP request on the Pi. I needed a Minesweeper client to hack around with so I could hook up clicking bombs to sending the request to the Pi, but the ones publicly available and easily findable and open source had some peculiarities that made it function slightly different from the way Minesweeper Online did, which I had gotten very used to. So, I figured I’d just mess with the Minesweeper Online browser client.
\nI found that the Minesweeper Online client communicated with the server via a websocket, but the message formats they used were incredibly cryptic. Fortunately, it wasn’t too hard to identify the packets that signified the game was over, but I still had to figure out how to intercept their message and tell the Pi to shock me. I was still kind of new to web development at the time, so after a good amount of searching and head scratching, I found a way to override the default WebSocket class in JavaScript by loading some code before the page loaded using Tampermonkey and forcibly added custom event listeners to every new instance of WebSocket to handle message received events. I must have been really proud of the code because I posted this code into a PasteBin and named it “fuck shit piss taser” for some reason. You don’t even have to know anything about programming to understand how cursed this code is just from seeing these lines:
![\"\"](\"https://blog.mnke.org/img/bx1fA0yris-426.png\")
This actually worked. With much reluctance from me and much goading from my friend, I did a few test runs with the dog shock collar on my leg. It didn’t take many fails to come to a conclusion for the science experiment. By shocking yourself when you fail, you actively want to avoid failure at all costs. I mean, we can joke about shocking yourself for fun, but if you’ve ever had a shock device strapped to you and you have the controls to it, your body actively refuses to press the button to actually go through with it, and it takes a moment of pure resolve — and idiocy — to override your survival instincts and press the button. So, in terms of improving performance, your pace grinds to a complete halt as you double-check and triple-check every single tile before you make a move. Maybe you could say that this improves your accuracy at the cost of your speed, but frankly, it just made me scared to play Minesweeper because I knew I’d still have to make guesses on some boards. Did I need to tell you all that? Hopefully not. But now you know 🙂
\nAll of that was a little side story to bring us to where we are today. I’ve been learning iOS development with Swift recently and also wanted to improve my reverse engineering skills so I figured it might be a neat little project to make a mobile client for Minesweeper Online. While I haven’t made the mobile client yet, I have made very solid progress on reverse engineering the messages and I’ll be detailing them for my own reference as well as for educational purposes.
\nTo start, the client requests a session id with a GET /authorize?session=. The server replies with JSON message with the following format:
\n{\n "authKey": "xxxx", // 128 bit hex string\n "session": "xxxx", // 128 bit hex string\n "userId": 1234,\n "user": {\n "createdAt": "2023-11-21T19:46:10.730Z",\n "country": "CA",\n "id": 1234,\n "hidden": 0,\n "premiumTo": null,\n "patronState": null,\n "storage": null,\n "hasVIP": 0\n },\n "geolocation": [\n lat,\n lng\n ],\n "serverTime": 1700595970731,\n "isNewUser": true\n}\nThe session id is then stored on the browser. If we wish to retrieve these details again, we can just call the same endpoint and fill in the session id query.
\nNext, I believe the client uses the geolocation from the /authorize endpoint as well as a list of servers in some JavaScript to determine the closest server, usually with a name like los1 or atl1. You can find how the server is determined in their JavaScript, but we may as well hardcode it. After the server name is obtained, we call GET https://${server}.minesweeper.online/mine-websocket/?authKey=${authKey}&session=${session}&userId=${userId}&transport=polling. The response will be something like
94:0{"sid":"xxx","upgrades":["websocket"],"pingInterval":300,"pingTimeout":5000}2:40\nWe need the sid, which we can get here, or from the io cookie set from this endpoint. I’m guessing this is asking the server to create a new socket, hence sid. We call the endpoint again, with &sid=${sid} appended to the query, and we have to do this within 5000 milliseconds, as per the pingTimeout field. If all is good, we’ll get a response back like:
19:42["authorized",[]]\nAfterwards, we may initiate a websocket connection to wss://${server}.minesweeper.online/mine-websocket/?authKey=${authKey}&session=${session}&userId=${userId}&transport=websocket&sid=${sid}. Their protocol requires a 3 way handshake upon websocket initialization otherwise the connection will terminate.
2probe3probe5After the handshake is complete, we must ping the server with 2 according to pingTimeout, which is 5 seconds, to which the server acknowledges with 3. If the server does not receive any pings in 5 seconds from us, it will terminate the connection with code 1005.
Great, we managed to establish a connection after all of this! I think Minesweeper Online uses a library, socket.io, to manage these socket messages and I considered learning it to help this process, but I decided against it. I wanted to do this blind, to simulate reverse engineering a completely foreign network protocol. So, we carry on to encoding and decoding messages for playing a game.
\nFrom what I can tell, all (outgoing and incoming) messages regarding Minesweeper games start with a 42, followed by a JSON array with the following format:
[\n direction, // "request" or "response"\n [\n opcode/function, // see below\n [payload], // depends on opcode/function\n something // the client always seems to include "494" here, while the server\n // doesn't have anything here. maybe version number?\n ]\n]\nopcode is represented as a string when logging it to console, but when converting it back to binary, it seems to be 4 bytes long for outgoing messages and 7 bytes for incoming messages, which is weird because it’s not a multiple of 2. I don’t think this should be a byte alignment issue from converting it to a string and back, because the byte before opcode should correspond to " in the JSON array.
It doesn’t really matter how to make sense of these opcodes since you can just treat them as magic strings, but I kind of wanted to. I thought that maybe there would be a table of opcodes in the JavaScript or that it was actually some binary code converted to a string, but no, I could literally search the JavaScript for the opcode string and find results. Furthermore, the opcodes that were sent by the server kind of looked like symbols in the minified JavaScript, e.g. G68.t18. Sure enough, I found an object called G68 with a function t18 with the number of parameters matching the length of the payload array for the G68.t18 opcode.
Wow, what a discovery! So in fact, opcode isn’t really an opcode, but a function, which is what we’ll be calling it from now on. This makes referencing how to handle incoming messages so much easier. And to think I just stumbled upon this while satiating my curiosity. After thinking about it though, what I really should have done was to figure out where and how incoming messages were being handled in the client code, and I probably would have gotten to the same conclusion, though with much more headache while navigating the minified code.
Anyway, back to business. To make things easier to read, I’ll only write the function name and payload, but you should keep in mind the full message format, especially that clients must send the additional "494".
// function: gn16\n[\n gameType, // beginner=1, intermediate=2, expert=3\n null, null, null, null,\n number, // not sure\n 1, // seems to always be 1\n null,\n countryCode, // 2 letters, e.g. \"CA\"\n null, null, null, null\n]// function: G69.i41\n[\n {\n id, // game id\n originalId,\n server,\n level,\n sizeX,\n sizeY,\n mines,\n state,\n nf,\n exp,\n diff,\n hints,\n userId,\n ping,\n clickType,\n mobile,\n timeGameCreated, // Unix timestamp, milliseconds\n timeStart, // same format\n requests, // array of numbers for when requests were made\n startX,\n startY,\n user {\n username,\n country\n },\n // other stuff\n },\n // board state. each is a key of numbers from 0 to sizeX*sizeY-1\n // goes down and then right\n {\n t: {}, // picture id — 0-11\n o: {}, // opened\n f: {}, // flagged\n g: {}, // not sure\n g2: {}, // not sure\n },\n [click history]\n // other stuff\n]// function: \"gj4\"\n[\n gameId,\n null,\n countryCode,\n 0\n][\n \"request\",\n [\n \"gu57\",\n [\n numClicks,\n gameId,\n clickType, // probe=0, flag=1, chord=2\n x, // 0 indexed, y increases going down\n y,\n timeElapsed, // since the start time, in milliseconds\n [ [x,y] ], // array of coordinates to be changed\n blackMagicString, // no idea, only non-empty when flagging. likely binary.\n null, // usually, not sure what it is\n null, // usually, not sure what it is\n ],\n \"494\"\n ]\n]// function: \"G68.t18\"\n[\n numClicks,\n gameId,\n {\n x, y,\n timeElapsed,\n type,\n touchCells: [\n x, y,\n numBombs, // picture id — 0-11\n isRevealed, // 0 or 1\n isFlagged, // 0 or 1\n ]\n },\n null, // usually, not sure what it is\n bool, // whether the server sent this message, I think\n]The game complete message from the server is fairly similar to the initiate/restore game message, but I’ll omit it here because your eyes have probably already glazed over. If you’re interested, check out the repository at the end of the post.
\nWith this information, it’s not too difficult to build a custom client for the game. In fact, I built a simple terminal interface as a proof of concept.
\n![\"\"](\"https://blog.mnke.org/img/gW5vLrzMdr-480.gif\")
![\"\"](\"https://blog.mnke.org/img/lyM9agwHLh-406.gif\")
To wrap things up, here are some of the things I learned.
\nFirstly, to reverse engineer any protocol, it’s incredibly important to set up a good sniffing tool. Without some way to sniff, filter, and match packets to actions, none of this would be possible. For websockets, this was fortunately very easy since I could just override the browser’s default WebSocket class by loading some code early on, and so I don’t even have to figure out how to decrypt messages. Thank you, JavaScript for having the jank to allow this to happen so easily.
After the sniffer is set up, we have the difficult job of figuring out the client message format. I don’t think there’s any step-by-step process that will work every time, but generally, we want to repeat actions and see what changes or doesn’t change in the corresponding messages. After some baseline is established for the message format of an action, we should slightly change the action to see how the message responds. For example, what happens to the message format if I left click instead of right click an unopened tile? What will happen if I left/right click an opened tile? I was lucky enough to have messages be encoded as a JSON object, but I can imagine tools like xxd would be very helpful to view the message as hex/binary.
For decoding server messages, it’s still a similar process. Repeat actions that you expect should produce similar results, find the kernels of similarity, etc. With server messages, however, you have access to the code that handles them, unlike client messages. Most of the time, the client code will be obfuscated so it’ll be difficult to read, but this is where we can break out our debugging skills and step through the code line by line. Even with JavaScript, it’s usually minified to the point where it’s completely unreadable and you have to trace call stacks and pause execution often to get any understanding of the code.
\nAnd really, that’s it. Afterwards, you might want to build some tools to let you send and receive messages like I did with the terminal interface, but it can be a lot of work, especially for more complex protocols. I had a lot of fun with this project, and I hope you learned a thing or two. Happy hacking!
\nRepository link — contains all the tools I wrote for this project.
\n", "date_published": "2023-11-17T21:54:50Z" } , { "id": "https://blog.mnke.org/bang-bang/", "url": "https://blog.mnke.org/bang-bang/", "title": "bang bang!!", "content_html": "For years now, I’ve been using POSIX shells as one of my primary interfaces to my computers, alongside a browser. In fact, for the past 3 years, I’ve probably spent over 80% of my time on computers using one or the other. Spending so much time with these programs, I try to master them to be as efficient as possible. While mastery over browsers is somewhat limited, there’s nearly boundless possibilities with the shell (really, the assortment of programs accessed through the shell). I continually find new things about programs I use everyday — anything from little tips and tricks to eye-opening, paradigm-shifting features. In fact, just yesterday, I learned about sessions in vim. How have I not known this for years?! And less than a month ago, I learned about brace expansion in bash. Now I can just do mkdir /path/{dir1,dir2,dir3} instead of running three separate commands. It really makes me wonder what incredible knowledge could be gleaned from a thorough read of entire documentations from programs I use often. Interesting as that is, what I want to talk about today is bangs (!) in bash.
Let me first introduce to you the utility of bangs. Ever try to run a program that requires elevated privileges and forget to prepend the obligatory sudo in front? Me too, and too many times. I’d have to scroll up the history with ↑ or Ctrl+p and then shift the cursor to the start of the line with, say, Ctrl+a, and then type the sudo. What a waste of time! Instead of doing that, you can just be angry and tell the shell sudo !!. This is the exact same as if you had ran the command again, except prepending the sudo. For example, if I had ran pacman -Sy abc and then sudo !! right after, what I’d really be running is sudo pacman -Sy abc.
If this isn’t already life-changing, I don’t know what I can tell you. Actually, I’ve known about this for years and it’s been cemented into my workflow. For every few times I remember to add the sudo to a package install command, there’s another time that I forget and sudo !!. Before we go any further though, let’s talk about this weird expression. !! is pronounced “bang bang”, with each exclamation mark being referred to as a “bang”. Why? I’m not sure, but this stackexchange answer points to Unix nerds re-popularizing the name originating from the printing industry. I think it’s a pretty good name and much easier to vocalize than “exclamation point”, which is the same reason why the printing industry ostensibly used it. Enough with the interlude on etymology though, let’s get back to business.
Since learning the !! trick a while ago, I wondered why !! expanded into the previous command and if there were a family of expansions that !! was part of. Sure enough, there is. Broadly, ! is a special bash character for history expansion. From reading the GNU bash docs, ! commands are meant to reference your command history. The way I like to think about it is that it’s an interface to select a previous command, or parts of a previous command. The interface needs to know which command to select and optionally, which parts of the command to select. I won’t go into detail on command selection, especially since the official docs are linked right above, but I’ll give some of the more useful ones.
!-n refers to the nth most recent command.!! refers to the last command and is what powers the sudo !!. I use this all the time.!string refers to the most recent command starting with string.!?string? refers to the most recent command containing string. Omit the trailing ? if you want to match string at the end of the command, rather than anywhere in the command.Word selection is where it gets interesting. Normally, you’d have to delimit the word selection part with a :, but there are exceptions, which we’ll talk about. For this part, let’s say we want to select our previous command, using !!. Keep in mind !! can be replaced with whatever command you want to select (e.g. !-2 or !ls).
!!:n selects the nth word of the command, with the delimiting character being whitespaces, 0-indexed.!!:n-m selects the nth to mth word, inclusive, 0-indexed.!!:^ selects the first word (same as !!:1).!!:$ selects the last word.!!:* selects all but the zeroth word.Again, I’ve omitted a few word selection commands, but I think these are the most important ones. For special characters like $ and *, we can remove the : delimiter entirely, abbreviating to !!$ and !!*. In fact, these two word selections are incredibly useful. Ever had to do a mkdir -p /a/very/very/long/path and then cd /a/very/very/long/path right after? Well, you can just do cd !!$ now! Or, have you ran something like ffmpek --there --are --so --many --options only to realize you spelt ffmpeg as ffmpek? No problem, just do ffmpeg !!*.
Lastly, we should talk about modifiers. Add another delimiter : and modify your bang command with a p to just print, instead of execute the command. For example, ls !!*:p will just print the command that would have been run had you not added the :p. The modifier s/old/new is string substitution, where / can be replaced with any other character (obviously, choose one that doesn’t appear in old or new). For example, if I had ran ls /path/to/long/path/with/spelligg/mistake/, I can make a quick fix with !!:s,spelligg,spelling,.
Whew, that was a lot. And there’s still so much more that I haven’t talked about. If you want to know more, I’ll kindly ask you to RTFM at the official docs 🙂
\nFor me, the biggest takeaways are !!*, !!$, and !!:s/old/new/. The other stuff is powerful, sure, but honestly, I probably wouldn’t notice the difference between taking the slightly less efficient path by Ctrl+r’ing into the previous command and moving by word to edit the command most of the time. Not only that, but remembering the more niche tricks and then learning applying them efficiently isn’t free! In fact, I must confess that much of the reason for writing this post is so I don’t forget it myself…
Cheers, and happy hacking.
\nPhoto by Jake Walker on Unsplash.
\n", "date_published": "2023-09-27T14:25:23Z" } , { "id": "https://blog.mnke.org/storage-issues/", "url": "https://blog.mnke.org/storage-issues/", "title": "Storage Issues", "content_html": "A while back, I purchased a Dell EMC KTN-STL3 storage enclosure so that I could switch off my janky storage setup. I’m embarrassed to even talk about it now, but it was seven 4TB SAS HDDs connected to an IBM M5015 on RAID0, directly passed into ZFS. If that wasn’t bad enough, I didn’t have enough 15 pin SATA power cables coming off the PSU connected to my motherboard, so I had to run another PSU to connect the rest. When I set it up, I didn’t understand much about storage configurations, and I was antsy to just get the system up and running, especially after dealing with much trouble with the RAID controller, but I knew it wasn’t great so I bought the enclosure, but I still didn’t have the equipment to interface with it correctly so I just let it sit there and collect dust for a while.
\nA couple of months after the initial setup, I was alerted by my friend of just how bad my configuration was. I wanted to change it, but the storage wasn’t physically available to me at the time because I was studying away from home. At the same time, the setup seemed to be working fine at the moment, and with no other choice, I just let it be. Half a year later, the day of reckoning arrived. The TrueNAS instance the storage was connected to had suddenly powered off.
\nI guided my family to diagnose the problem and restart the computer, and it did so just fine. I remotely verified that the data was intact and breathed a sigh of relief. I had documents and pictures from my old phones on there, backed up to nowhere else and I would’ve been very upset to lose it. This incident made me realize how urgent the upgrade and backup was. I was planning to at least back up my data over the next few days (I couldn’t do so immediately because I had to take care of my school duties), but the very next day, the computer had shut off again. I asked my family to do the same thing, and I remotely checked the data again. The ZFS pool was offline.
\nMonths later, I returned home and started digging at the issue. In the RAID controller interface, I found that a single drive had failed. There was no chance of recovery because RAID0 had striped everything across all the disks. I cursed my own stupidity. What I should have done months ago was to immediately ask my family to completely shut down the system to prevent any further damage. Alas, the damage was already done and I just had to accept my losses and move on.
\nI got to work researching and consulting my friend on how to get the storage enclosure up and running. Months ago, the same friend had recommended to me and bought for me a HBA controller — an HP H221, flashed with some firmware that I didn’t note down — to pass through each disk separately for a proper ZFS setup. The back plane of the enclosure had four SFF 8088 ports, two on the top half and two on the bottom half. The controller had two SFF 8088 ports, so I bought two male to male connectors and hooked it up and loaded the disks into the enclosure.
\nAfter some stupidity (I didn’t realize you had to unscrew some parts of the SFF cable to fit it correctly!), I connected the cables and tried to boot. I booted into the controller’s BIOS, like normal and waited on the “Initializing…” screen… and I waited… and I waited some more. I ate a meal and came back. Still no change. I was puzzled because the system was able to boot past the controller’s BIOS and even into the controller’s configuration interface just fine before I plugged the cables in. I unplugged the cables and tried booting again. Yup, for some reason, plugging in the cables was causing problems. Maybe I was plugging in to the wrong ports, I thought. After all, there were four ports on the enclosure. I played around with plugging into different combinations of the enclosure. Still no luck.
\nI had tried searching up EMC KTN-STL3 manuals, guides, or just any relevant information that might be able to help, including general information about storage controllers and about the HP H221 card I had, but the information was incredibly sparse. Of the information I could find, it seemed like my setup should have worked. I further probed my friend for consultation, but we came up short. For the next few days, I did more research and fiddled around with settings. I tried disabling BIOS booting through the controller interface as per this Reddit thread (take a moment to look at the upvotes and realize how uncommon my problem was). I was able to skip the controller’s BIOS and boot straight into TrueNAS, but it was failing to start up properly, giving me errors like “Root mount waiting for: CAM”. I could only find a few threads on this with much information I couldn’t even understand, but there was one TrueNAS thread that gave me a lead before delving deeper to learn enough to parse the information in the other threads. It mentioned that this error was associated with an HBA failing to initialize under UEFI.
\nWhen I came into this, I was hoping I wouldn’t have to flash my controller. Things can very easily go wrong if you’re not careful or knowledgeable, and I had neither of those attributes — horror stories of irreversibly bricking a controller were flooding through my mind. However, I soon realized flashing the controller to upgrade to UEFI support was the only lead. My friend had told me of the tools — megarec and sas2flash — but these were fairly specialized tools that I couldn’t easily find digestible documentation for, so I had been scared away in the past. I found a very comprehensive thread to update LSI SAS controllers with an UEFI motherboard, and an even more detailed thread on the (cross)flashing process, and with the go-ahead signal of my friend, I started hacking away.
\nI was told to use megarec to wipe the BIOS of the controller before using sas2flash, so I did that after backing it up with the same tool. I then switched off the DOS shell containing megarec and onto the EFI shell containing sas2flash, created by following the first guide, and tried to follow it. sas2flash -listall was giving me a controller with malfunctioning firmware, which was not what the guide expected, but I wasn’t fazed because I knew I had just wiped the firmware with megarec. I followed the rest of the guide, but I was getting errors trying to run sas2flash -o -f xxx.bin -b x64sas2.rom -b mptsas2.rom, telling me there was a mismatch in the chipset or something. Ah, I’m flashing with the wrong firmware, I realized. After a couple of trips up and down the stairs between my workstation and the storage location trying to figure out the right firmware, I was still getting mismatch errors. I scratched my head in confusion because I was fairly certain I had gotten it right, especially after pausing to understand the crossflashing process a bit more. As I read the second guide more closely, I found out that in newer versions of sas2flash, it doesn’t like it when you crossflash, even when you’re crossflashing a compatible card. I sat down and fully digested the second guide and found a hacked version of sas2flash given in the minimal USB files at the bottom of the guide. I made sure to find compatible firmware files with the HP H221 in this thread, downloaded them off of broadcom, and copied over the minimal USB files and the firmware files onto my USB and used the knowledge from the first guide to find and copy the right firmware files.
Back into the EFI shell, I used the hacked sas2flash to run sas2flash -o -f 9205-8e.bin -b x64sas2.rom -b mptsas2.rom like the first guide mentioned and… the flash succeeded! I updated the SAS ID (I just made up a random ID, which was fine according to the second thread) and exited the EFI shell. Booting back up, the familiar text of the controller BIOS greeted me and I held my breath as the “Initializing…” text showed up again. One mississippi, two mississipi, … I counted in my head. After a short, but nervewracking pause, the screen flashed with new text listing the drives detected by the controller and it booted into TrueNAS with no problems. I checked on the web interface that the drives were indeed being detected — they were!
This whole process of trial and error took over a week and a half, and I’m writing this only hours after the success. It was exhilarating seeing the pieces finally fall into place after much perseverance, and traces of the joy I felt can still be found as I write this. I wanted to write this post to document my struggles and serve as a guiding light for the few who find themselves in a similar situation and stumble upon this post. I omitted some details for sake of comprehensibility — none that I think are relevant — but for those who use this post to troubleshoot their own problems and are confused on some of the steps I took, contact me at tonydu121@hotmail.com and include a reference to this post in the subject so I can find it (something like “Storage Help Required” will do). I’m by no means an expert and cannot guarantee that I’ll even be helpful, but I’ll be happy to provide the full details of what I did.
\nCheers.
\n", "date_published": "2023-09-05T18:02:00Z" } , { "id": "https://blog.mnke.org/death-roll/", "url": "https://blog.mnke.org/death-roll/", "title": "Death Roll", "content_html": "I thought of some interesting questions based off of this game I saw. I don’t know what the game is called, but I call it death roll now. The rules of the game are simple:
\nThis turns into a game between friends by taking turns rolling the number. I figured I might be able to pull this out as a drinking game, but that made me wonder if there was any way to rig the game by choosing the initial number in a certain way. For example, if there are $k$ players, could I choose $n$ inconspicuously so that I can expect the game to end before the $k$th player even has a chance to roll? Or in general, how much shenanigans would I be able to get away with?
\nSo I began messing around with the numbers. Let $T(n)$ be the expected number of turns by starting with $n$. If we roll $n$, then the expected number of turns is $T(n)+1$, because we ended where we started, except we wasted a turn. In general, if we roll $i$, then the expected number of turns is $T(i)+1$. By properties of expectation, $T(n)$ follows this recursive formula:
\n$$T(n) = \\mathbb{P}(n)(T(n)+1)+\\mathbb{P}(n-1)(T(n-1)+1)+\\ldots+\\mathbb{P}(1)$$
\nwhere $\\mathbb{P}(i)$ is the probability to roll $i$, which should be uniformly distributed but is noted for generalization.
\nDoing some algebra will yield that $T(n+1)=T(n)+\\frac{1}{n}$. I’d like to remark that this is an exceptionally simple and elegant formula to capture this game.
\nI also wondered if there was a non-recursive definition for $T(n)$. I tried solving the recurrence by using advancement functions, but I had no luck. I turned to Wolfram Alpha to look for a computerized solution, and I found that $T(n)=H_{n-1}$, where $H_n$ are the harmonic numbers. I’m not very familiar with the harmonic numbers, so I’m not sure why it shows up here, but it was a curious connection to the uninformed me.
\nI also have to look at the variance of the number of turns as a function of $n$. I haven’t done that yet, but it seems to have quite high of a variance, which is not surprising. I wrote a script to simulate the game and after a million samples of $n=50,000$, we get that the expectation is around $12.40$ and the variance is $13.03$. The variance is way too high for me to have any hope of rigging the game unfortunately. I’m too lazy to calculate the variance by hand, but if I ever do, I’ll come back and update the post to show the actual formula.
\nDeath roll will eventually complete, and my questions above were in regards to how long it would take. However, what if we modify death roll a little bit? Let $f: \\mathbb{N}\\to\\mathbb{N}$ and modify death roll into $f$-death roll by keeping the same rules as death roll except in step 4, set $n=f(m)$. I’ll also introduce another variance $f2$-death roll, where $f:\\mathbb{N}\\times\\mathbb{N}\\to\\mathbb{N}$ and in step 4, set $n=f(m,n)$. I’ll mainly be concerned about the first variant for now, but the second is something to think about.
\nI wonder what are necessary conditions such that $f$-death roll doesn’t last forever? For example, I would guess that if $f(n)=n!$, then the game would never end. What’s the boundary between when the game ends and when it doesn’t though? Certainly a sufficient condition would be that $f(n)\\leq n$. I also posited of a necessary condition that $f(n)>n$ for only finitely many $n$.
\nUpon discussion with a colleague, we discovered that the posit is incorrect. Consider the function
\n\\[
\nf(n) = \\begin{cases}
\nn+1, &n \\text{ odd}\\\\
\n1, &n \\text{ even}
\n\\end{cases}
\n\\]
Certainly, $f(n)>n$ for infinitely many $n$, but we’ve proven that $T(n)$ with respect to this $f$ is finite. It’s not quite constant, but taking $n\\to\\infty$ gives a bounded $T(n)$.
\nOf course, $f$ lives among a family of functions where
\n\\[
\nf(n) = \\begin{cases}
\nn+1, &n\\not\\equiv0\\mod m\\\\
\n1, &n \\text{ otherwise}
\n\\end{cases}
\n\\]
I’ll call this family of functions the ideal functions, given by the behavior of “absorbing” elements from the ideals of ring theory, and of course by the fact that $f$ acts on the ring generated by modding out an ideal of the integers. It’s not hard to see that ideal functions do violate $f(n)>n$ for infinitely many $n$, but also that $T(n)$ is finite and $\\lim_{n\\to\\infty}T(n)$ is finite (and although not proven yet, it seems likely that this limit is equal to $m$, the element generated by the principle ideal $f$ acts on).
\nI think what makes the ideal functions special is not the fact that they act on the ideal generated, but that the limit of $T(n)$ is finite. Knowing the existence of ideal functions, it’s a simple exercise to construct a function living outside of the ideal functions that violate $f(n)\\leq n$ for infinitely many $n$ but still have finite expectation for all $n$. We’ll call functions with finite limiting $T(n)$ the limited functions, of which the ideal functions are a subset of. It may not be such a simple task to construct a non-limited functions violating $f(n)\\leq n$ for infinitely many $n$ with finite $T(n)$ for all $n$. Naturally, this is my next posit:
\nGiven $f:\\mathbb{N}\\to\\mathbb{N}$, if $T(n)$ is finite, then $f(n)\\leq n$ for infinitely many $n$ or $f$ is a limited function.
\nIf any more progress is made, I will update this post again.
\n", "date_published": "2023-03-13T01:00:00Z" } , { "id": "https://blog.mnke.org/dns-splitting/", "url": "https://blog.mnke.org/dns-splitting/", "title": "DNS Splitting", "content_html": "I had started working on migrating off of cloud services to self hosting everything a couple of months ago by getting some new gear to set up my server on (shoutout to [REDACTED] for the hookup). I started off by setting up a stub webserver and general environment for me to deploy games servers for my friends which is pretty much the exact same process as if you were to do it on an external hosting service, except I was a little bit more careful to not open myself up to security vulnerabilities since I’m hosting this in my own home.
\nAfter I had set up a Minecraft server, I added a record to my domain provider pointing to my home network and tried to connect to it using that domain. No connection. That was a bit strange but maybe I forgot some configuration so I went through a checklist:
\nWell, what the hell was the issue then??
\nTurns out, it’s a much broader networking issue that’s usually already resolved by someone somewhere.
\nThis answer to a stackexchange question explains a similar problem much better than I can, but I want to dumb it down and over-explain for myself in the future and for those who may stumble upon this and may find it helpful.
\nWhen your computer wants to send a message to someone, it attaches an address to whom the message is sent to (the destination) and one for whom the message was sent from (the source). After your computer sends the message, it expects to get a response from the receiver with the source and destination addresses swapped.
\nYour computer may sometimes know exactly where to send a message to, like in the case of sending a message to localhost (which should be set up for you automatically in almost all scenarios), but most of the time, it may need some help. This is done through a gateway. This is a device (likely your modem/router) that, figures out where to send messages among other things. So, when you connect to a local IP address, your computer doesn’t know where to send the message to, so it forwards it to the gateway. The gateway knows where to send the message, so it relays it to the receiver.
\nOkay, but what if you try to connect to a non-local IP address? Well, it’s still a similar process. Your computer forwards the message to your gateway and your gateway realizes that this IP address belongs to none of the locally connected addresses in your network. It wants to send it to the internet, but it can’t quite do so yet. This is because the local IP address of the source is not unique among the internet IP addresses, and so your gateway rewrites the source IP address to your public IP address, which is unique, and sends it off. When your gateway eventually receives a message back, the destination IP address is your public IP address, as per the protocol described earlier, and to finally send the message back to the original computer, it rewrites the destination IP address as the local IP address that originally sent the message. This process is called network address translation, or NAT for short.
\nSending messages through a domain like www.public.com just adds one more simple step to the process (simple for our sake). When your gateway receives the message, it doesn’t know who www.public.com is. It only knows how to send a message to IP addresses. The extra step is to contact a DNS server to find out the IP address of www.public.com. Once it does that, the rest is the same.
\nNow that we understand this, we’ll work through what happened when I tried to connect to my server through the public domain.
\nSuppose my public domain is www.public.com, my public IP address is 1.2.3.4, and the local IP address of the computer my Minecraft server on is 192.168.1.2.
\nNow that we know how it fails, how can we solve this?
\nThe method I used is called DNS splitting. Really, all it is is telling your computer that when a message is supposed to be sent to X, send it to Y instead. So when we want to connect to www.public.com, we just tell it, “We know where www.public.com is already. It’s at 192.168.1.2″. How this can be accomplished is by modifying your /etc/hosts file to include a line like
\n192.168.1.2 www.public.com\nWith that, it was fixed!
\nThis should work on all *nix systems (I’ve tried with Linux and macOS at least). If you’re on Windows, apparently doing something with the system drivers(?) might work. See this article by yunohost for more details.
\nI think there are other ways of solving this problem like NAT reflection as mentioned in the stackexchange thread earlier, but this solution is dead simple and I’m happy with it.
\nI think for some routers, this is configured for you by default in the gateway itself, or at least it’s configurable in the DNS settings in your gateway, so you don’t have to DNS split on every single machine that needs it, but my router, provided by Shaw, sucks and so I have to do this workaround instead.
\nHopefully this helps someone out and hopefully you learned a thing or two reading this and I wish you luck in your networking adventures!
\n", "date_published": "2022-06-25T00:36:00Z" } ] }