An Ode to Mastery
Constructing Complexity, Part 1
Aspiring programmers receive a range of stories from the zeitgeist. Previously, they were promised an easy path to senior engineer at Netflix earning $500K USD/year. Just complete this coding bootcamp, solve these medium LeetCodes, make some GitHub commits, get a green graph, you're done - you’re basically hired. This narrative was further pushed by non-technical founders who, after generating their landing pages, declared "coding is easy" and even further by real senior engineers who - with a shrug - claim that they "don't really know anything" and the newcomer thinks, "they've made it and they’re just like me". Today however, the story has flipped, AI has progressed slightly, and newcomers are now being warned of the impending AI takeover, and the subsequent futility of learning to code; "give up now, don’t start, don’t waste your time because you’ll never catch up". They’re told to surrender their agency to the agents, to simply give into the vibes and let the language models carry us into the future, "you don’t need to understand how things work - that’s what the AI is for, silly!".
This essay will show you what is wrong with these stories. The too-easy story is wrong because it blinds newcomers to what lies ahead, because it romanticises "skill issues" and because it falsely conveys the sky-high skill ladder as a small climb. The too-late, too-hard story is wrong because it assumes AI contains "genius level" intelligence when in reality they are merely useful tools, and because their blog-reading, text-traversing algorithms grant them only a poor facsimile of expert reasoning, and lastly, because AI is simply overhyped and overestimated; LLMs are a game-changer but not a role taker.
This essay is my defence of mastery. This essay is my ode to mastery. Becoming "good" at any craft is difficult; it takes a lot of time. However, if you can find it, the journey contains much joy. I found it, and I hope you can too.
I love programming. I have been writing software personally and professionally for 16 years, and in this time, I have accumulated about 30,000 hours of mostly deliberate practice, which averages out to about 5 hours everyday. By stars, I am currently ranked 309th of 100M on GitHub. I have a degree in computer science; and although it was an excellent program - which gifted me a wide base of knowledge - unfortunately, my degree did not teach what it is to be "good" at writing software. I learnt the theory, SOLID principles, design patterns, discrete mathematics, big O notation, graph theory, etc., which helped me arrive at point solutions, but none of these explicitly made me better. By the end of my degree, still, I was not a "good" programmer, nor software engineer.
The software engineers whom I consider to be "good" on the other hand, have all attained a modicum of mastery. They all have a T-shaped base of knowledge that extends both broadly and deeply, and their mastery lies in the centre of this T. They are not master blacksmiths; they are master sword and dagger smiths. Their mastery covers just a few areas, not the entire field. They are jacks of many trades but only masters of one - two, or three (depending on the shape of the T).
There is one exception to this diversity of talent, one skill that is required to become “good”, one skill that is universal in the programming universe: the ability to design "good" software. Software design is programming in the abstract, it is modularisation across multiple layers of the complexity stack, it is the act of making deep trade-offs. This vague generality is why it both difficult to learn and extremely useful. This difficulty stems from the necessary unbroken knowledge path to ground truth. When you're asked in an interview "what happens when you type google.com in your browser?", you're actually being asked to demonstrate your sympathy for the machine, to lay out your knowledge path. This experience-imbued path is central to why stages of mastery must be progressed in order. A junior engineer can't skip over the mid-level topics and land straight into a senior engineer level of skill. They must walk the path; do the time. They must sequentially lay each brick atop another.
In my first few years of programming, I organically (re)discovered most of the useful software patterns by simple brute force, via sheer strength of hours. I fell into every pit. I shot myself in the foot so many times there's now just a stump below the knee. I could have optimised my time better. If someone had told me where to go, I could have spent less time down dead ends. However, you must find the balance; you must write software with your own hands. If you meet no dead ends, then solutions are being spoon fed to you, and you're not writing your own software, and you're not truly learning.
I have come to accept that diving into the wrong rabbit hole is just part of the process; instead of optimising for time, I optimise for improvement. From the very beginning, and still to this day, I only have one goal: to ensure the code I write today is better than the code I wrote yesterday. That's it. My coding career is simply the sum of many many tiny increments.
I take apart my workflow often, and I question the value of each part. I trial new steps to see what might improve the whole. I try to understand something new, a new tool, a new approach to an old problem. I found improvements on my own and through others. When I was starting out, learning from others was critical (thanks TJ!). The goal must be improvement, no matter how minor, because daily improvement yields compounding returns, which is critical if you're in it for the long haul.
All upward-aiming long-haul programmers have travelled this long road, and although this road was sometimes a gruelling gauntlet, it was mostly a meandering exploration; and this exploratory route is found by discovering the joy within the craft. This enjoyment of the process however, is difficult to attain. In my experience, you must first become a builder, and then you may discover the joy.
A programmer is someone who cares about what they build; they care about the destination. A builder on the other hand, is a programmer who - regardless of tenure - alsocares about how they build. They care about their process, and most importantly how they might improve. Organisations decline when they amass programmers who just don’t care; those who skim just over the low bar, who unthinkingly clock in and out every day. The solution is not to replace them with "A players", but to replace them with builders; people who simply care. It's this builder's focus on improvement that will raise the bar within organisations; and individually, will set you on your own path to mastery.
Personally, I define mastery as the endless pursuit of excellence, so I lied earlier, you do not become a master sword smith and you’re done. No; the blade can always be sharper. You will forever approach the limit of mastery - forever making "50% more" progress but never quite getting there; forever honing your skills, forever striving.
You need to find your muse, the thing that makes you run - not just walk - to your destination; and more importantly, the thing that brings you back tomorrow. You need to find "product-market-fit" for your brain in the vast world of software. Finding this fit often requires a diversity of experience, but once found, you'll be ready. Your desire for improvement will grow, and will slowly evolve into enjoyment of the craft itself. As a mere programmer, you carelessly scramble towards the destination. As a builder, you are aware of the mountain ahead; you climb carefully for competence.
For without this builder's mentality, you face only the gauntlet, and the entire length of this gruelling path cannot be travelled through gritted teeth. It's too arduous and holds too many lessons, and these lessons can only be received with deliberate openness.
With only the destination in mind, you’ll become frustrated, tempted into trading off the long term for the short. An irresistible exchange; and after repeatedly taking on this short term debt, the pain accumulates. The weight of this debt then traps you in a downward spiral of yet more short term trade-offs. Instead, you must stop; don’t be tempted. Take the time to understand, to truly learn. If you don’t have the time, find ways to better use the time you have - because slow is smooth, and smooth is fast.
To become "good", you must care about the process, you must become a builder, you must travel this distance yourself - which need not take 30,000 hours; it’s really not about hours at all. The engine of mastery is driven by improvement iterations. Growth comes from how many times you’ve tried, succeeded or failed, learnt from the result, and then tried again. This process demands time; a lot of time, and this is unavoidable, because you must traverse some minimum distance to attain the required experience. You won’t cover it by watching talks and reading essays like this. The input you consume does not matter. Only your personal attempts, only the output you produce truly matters.
In my first year of university, I was averaging 60%. Programming was hard, and it appeared to come so easily to others. But the answer was simple, they were just further up the "mastery stack" than I was. The mastery stack is series of abstraction layers, where each layer contains a collection of ideas, dependant on the prior layer. The mastery stack is not the Dreyfus Model, with its 5 levels of skills, the mastery stack varies per craft, and some crafts are "deeper" than others.
To make this concrete, take chess for example, we can approximately equate the first layer to the rules of the game: how each piece moves and what is a valid and invalid move. The second layer might be tactics, forks, skewers, etc. The third might be openings. The fourth might be board position and movement. The fifth might be deeper strategy and end-games. An actual chess expert would likely come up with better examples, but you get the idea.
Whereas in programming, we might say the first layer is about language syntax, valid tokens, and the practical steps in providing your text input to the compiler. The second layer might be about language semantics, the basic idea of a program counter, how loops work, what "goto" does, etc. The third could be higher level composition, like functions, project structure, code "tricks", design patterns. The forth could be higher still, optimising the below layers for humans, for mental cohesion and comprehensibility. The fifth might be making the entire codebase "easy to change" with all that this entails. The sixth might be attaining simplicity.
The layers are sequential sets of ideas, where the diversity and depth of the abstractions increase as you rise up the stack; and their sequential nature is key – you can't truly understand language semantics before understanding language syntax, and so on.
So like everyone, I began on the first layer of the mastery stack; the base level of knowledge and experience required for the craft. At this layer, everything is conscious, all your thinking is slow. Frustration is at its peak and the peak of the mountain is far away. But, as you practice here, action progressively moves into the subconscious. At some threshold of performing subconsciously, some percentage of thinking fast, your conscious mind - your slow thought - frees up enough to focus to the next layer. Whether you’re a writer, dancer, chess player, programmer, carpenter, with enough time at this layer, you acquire fingerspitzengefühl "finger tips feel", you can now play with the pieces in front of you, to freestyle, to improvise. You no longer think about individual steps at this first layer anymore. Your slow focus is now aimed at improving your actions within the second layer.
This process repeats. The second layer moves into the subconscious too, and your fast thinking now handles it by default. You move up again. You dance. You play some more. You explore. You experience free flow of action. You find yourself in the egoless void, a judgement free zone. You’re in the zone, within flow. You find more layers. You can always abstract further, always move higher. The human brain cannot fast-track this process, and so, you simply need to put in the quality time.
It appears that present AI cannot fast track this process either; it appears to be stuck at the first layer. At best, the LLMs of today, can be viewed as junior programmers, with faux reasoning, and a massive, mostly-accurate static memory of the internet. The stagnation of these virtual "minds" at the junior layer explains much of what we see today. It explains why non-technical founders gush about AI, because they truly do make them 100x better at coding - AI raised them from nothing to something. It explains why juniors absolutely love using AI - it lets them skip over the hard part - at the cost of future mastery. It explains why AI agents hit a limit with code bases, because the garden is growing, it now demands quality tending, and all the AI can do is spray more seeds and water. It explains why AI cannot do basic refactoring. It explains why senior engineers use AI like any other tool; as multi-line autocomplete for what they were already going to type – they remain the pilot. It explains why AI doesn’t write "beautiful" code; simple, linear, multi-layer, multi-purpose solutions. It explains why AI has no "taste" in programming; the ability to appreciate beauty. It explains why AI needs expert guidance, and consequently, why you need to become both AI guide and expert.
AGI will happen. AI - at some point in the future - will be generally intelligent enough to pass the "Turing Machine Test" and be indistinguishable from your talented remote colleagues. The question is: when will it happen? and if it’s in the next few years, is mastery worth it? I still say yes, if you want agency in this world; if you don’t want to be the AI’s passenger, then it’s worth learning to be the pilot. At present, our competence reigns supreme and so we remain. Text-traversing AI has feeble abstract reasoning, and an intelligence ceiling far below ours; but, when we discover that new algorithm, the one after language models, it will surpass us, whether that is in 1 or 100 years, it will. However, our relative incompetence will only relegate us to the back of the plane if we go willingly. It seems we will either have a job controlling AI, or there will be no jobs and AI will control us. We can't control the future, but we control today. Either way, mastery will be your best chance at freedom. Today, I have my muse, I continue on the path, I build, and where it helps, I build with AI too.
And tomorrow, at last, when you too have found your muse, when you become a builder and commit yourself to this journey of incremental improvement; when you commit to invest this slow time, you will gradually train your meat-based neural network to perform the multi-layered dance within and around your code. You will learn to construct complexity in the small, medium, and large - all at once; and although you must start small, the better you get, the larger your dance will become, and the further your intuition will take you. Your abstractions will become meta-abstractions, and around again. The depth of your vision will increase. And in time - a lot of time - you will slowly construct complexity with evermore elegance. Your software will approach being both beautiful, where it is as simple as it should be, and effective, where it is mostly correct now, and easy to change later; and therefore, sustainable across both time and people.
In this essay series, I will offer my foundational view on software engineering, an overview of my knowledge; and to me, the most valuable parts, in the hopes that - while you invest the time into the craft - you can focus your attention on what's important and stay away from what's not.
If you liked this essay, please subscribe to Constructing Complexity to follow along with the series.
You can also follow me on Twitter/X @jpillora. I don’t tweet much but this may change as I do have many hot takes in Obsidian…
Disclaimer: While I work in the technology industry, the opinions and views expressed on this blog are solely my own.
I have many essays planned, and to give you a sneak peek of what's to come, among other topics, I will attempt to convey:
Why the commandment to "reduce overall complexity" is at best confusing, and at worst, a fool's errand; and what we should strive for instead
Why simplicity should be your north star, and your primary weapon
Why the clean code movement is misguided, and leads to unnecessary software bloat
Why you should generally build first and buy second; and why you often never get to the second step
Why our processes and code bases are overflowing with unnecessary complexity, and how it got there in the first place
Why the 10x engineer is real, and what actually yields this 10x gain
Why Go is an excellent tool for building good software, and how Go embodies the ideas in these essays
Why gaming often flows nicely into programming, and what programmers and gamers have in common (and surprisingly, it's not hours sitting at a computer)
Why the "box and arrow" diagram is all you need, and why everything else just gets in the way
Why developer platforms are pivotal for large organisations, and why their success depends on an excellent developer experience
Why your "LeetCode" and micro-optimisation skills are orders of magnitude less important than your software design skills
Why designing with vision, planning with architectural stages in mind is critical to software engineering scalability
Why the ship must be captained by a visionary BDFL, and why without one, you'll be lost at sea
Why the fog-of-code, the gap between the stacks, instils fear in the hearts of software engineers - and how it leads to the "frameworker" and the "React engineer" who has never written a JSON API
Why print-debugging is more effective than step-debugging, and why "good" code tends to favour print debugging
Why trunk-based development, paired with release channels, is the right way to deploy software to production
Why LLMs in their current form will not replace software engineers, and make a prediction about what it would take
And for the next essay, the second part in this series, I'll attempt to answer:
What is "good" software, and what exactly makes it good
Subscribe now to stay tuned!