These 2 things are often confused in our industry: experience and institutional knowledge. Too often I see companies evaluating engineers based on the fact they know where to go and make changes quickly and put out fires, rather than looking at the impact long term an individual has, on the platform, on its stability, on its maintainability, ease of deployment and ease of on-boarding new team members, as well as the impact they have on the team itself. This often leads to wrongly evaluating the talent and experience in the team, which of course leads to making wrong technical decisions, as the right people are not heard — and this leads to technical failures in products, which can heavily interfere with company plans and scaling up.
If you are working with engineering teams and trying to identify (maybe promote?) talent in the team, or perhaps have just inherited a new team and trying to figure out who are the people who can help you advance a product, the following points should help.
Institutional knowledge
What is institutional knowledge? According to wikipedia “Institutional knowledge is gained by organizations translating historical data into useful knowledge and wisdom.” In other words: creating wisdom from history.
Translating this into engineering, institutional knowledge is gained by a team by translating various events (platform incidents, outages, upgrades, changes etc — historical data) into knowledge (“when doing X be aware that Y might happen, so you must do Z to prevent it” for instance). Every time an event occurs — be it major (such as an outage) or minor (e.g. updating a component configuration somewhere in the system) — there is some data that surfaces from it; institutional knowledge is the sum of all this data and all the learnings the team has acquired having gone through these events.
Every system out there requires some level of institutional knowledge — for instance, for most of us, when writing documents, we are probably in the habit of constantly pressing Cmd+S key combination to save the document regularly, so we don’t lose content in case of a system crash. This is institutional knowledge: chances are at some point we or someone around us has encountered a crash and found out lost some data — so we now apply this knowledge by automatically using the keystroke combination to prevent it.
Tech debt and institutional knowledge
In engineering this is perhaps even more so present due to ongoing tech debt accumulation. (This is subject to a future post, but tech debt is unavoidable; at any point, leaders have to make trade-offs which require consciously accumulating tech debt — normally this tech debt gets cleaned up in time and hopefully regularly, but it is unavoidable.) Every time we accumulate some tech debt, we have created some institutional knowledge — or at least the opportunity for it.
Think of this scenario: we have a component which we need to deploy in a very tight timeline; the team decides that this component ideally needs a “dashboard” to control its behavior, so we can change in real time certain aspects (e.g. things like cache size, TTLs etc). However, because of the tight timeline a decision is reached consciously that in version 1, rather than a UI to control this component we will have a configuration file shared somewhere on the network; when we need to change any behavior in this component someone goes and edits the file with the desired changes. (And also we agree that version 2 will include the above-mentioned dashboard, at which point we can get rid of the configuration file.) Having executed the above approach, we now have introduced some institutional knowledge! Anyone who wants to change any behavior in this component now has to “know” 2 things: the fact that the behavior is controlled by a file somewhere on the network, as well as the location of this file. This might NOT be institutional knowledge right away, and simply be tech debt — however, when someone needs to go and change the behavior of this component they will go through an event which will generate this institutional knowledge for them: they have to go and “learn” the above and find out where the file is and go and change it. That’s why tech debt either introduces institutional knowledge upfront — or creates the opportunity for institutional knowledge.
Even more, the main way to reduce institutional knowledge in engineering is by reducing and pruning the tech debt. Adhering to standards, applying standard patterns, architecture, etc reduces the need for institutional knowledge — in favor of “generic” engineering knowledge. (See right below for the difference.)
Why “institutional”?
How is “institutional knowledge” different from just “knowledge”? After all both result from a process of learning! That aspect is correct, and it is true that we learn when we assimilate institutional knowledge. However, the difference is that this learning, this knowledge typically cannot be applied anywhere else, but in the team/company/institution we acquired it in — it is very specific knowledge to a particular case in a particular institution.
This means that this type of knowledge will only help you in the context of the particular team or company or department you are in — any changes to this environment will render this knowledge unusable and not useful. (And typically this is why this sort of knowledge often gets forgotten in time.)
The “generic” knowledge we acquire — through training courses, books, education etc — can be generally applied everywhere, or at least in a wide variety of cases / companies / teams / institutions. This is the sort of knowledge you would expect in engineering from any new recruit — things like a programming language for example; by contract, institutional knowledge is the sort of knowledge that normally people who have been through some events in your company have acquired. (This is why typically people who have been with a company for a while possess the most institutional knowledge — and sadly, that’s why some companies still promote based on history within the company: because of the institutional knowledge individuals who have been with the company for a while possess!)
As a consequence of the above, institutional knowledge is probably the biggest component of the ramp-up period your new hires need to go through — in fact, without the need for institutional knowledge the onboarding process you have implemented will probably be reduced to a few hours or less.
Institutional knowledge is unavoidable
Wherever we work, no matter how well managed and structured the team is, we will always encounter a level of institutional knowledge — and that is not necessarily a bad thing. Think about your current job: knowing where the source repositories are kept in Github for instance is an institutional knowledge; the URL for the expense system, or whether you submit your expenses to your boss or to the accounting department, that is institutional knowledge too. And as I said, this is not a bad thing — learning all these small details allow us to move quicker through certain tasks, and that is why we learn these things: to be able to execute certain things fast.
No matter how much a company will standardize their processes and operations, there will always be a certain level of institutional knowledge they require from their employees so they can operate efficiently. The trick here is though to keep that level of institutional level requirements to very low, so they do not create friction during onboarding and also they do not interfere with the normal run of the business.
Perhaps even more so in engineering, institutional knowledge is unavoidable: if you recall from my comments above, this goes hand in hand with tech debt which is also unavoidable. And again, this can be kept low through constant tech debt pruning and addressing.
Experience
We have looked above at how institutional knowledge differs from “generic” knowledge. So what is “experience” then? Experience normally translates to a large amount of knowledge — both empirical and practical — that someone has accumulated. In engineering in particular, an experienced engineer is someone who has both acquired a lot of knowledge as well as had the opportunity to apply this knowledge. Typically that knowledge would be a mix of institutional and “generic” (non-institutional) knowledge — however, as we saw above, the institutional knowledge is mostly non-transferrable, so the one component normally referred to when talking about experience is the non-institutional part.
Experienced engineer
You heard probably often engineers being referred to as “very experienced Java engineer” or “experienced database admin” and so on — as per above, this means the engineer referred to has a lot of generic knowledge as well as a lot of practice with a certain language or technology , and you can benefit directly from this knowledge by hiring them and letting them apply this knowledge in your organization.
One of the many (good) traits of an experienced engineer is that they are trying constantly to “standardize” things — this, simply put, means adjusting systems to a state where there is very little institutional knowledge needed to understand it. Often this is achieved by adhering to industry standards, applying standard patterns or employing generic or standard tools and frameworks and processes. As such, in most cases, an experienced engineer sits at the polar opposite of institutional knowledge and their actions are often in the direction of minimizing this.
Conflict: experience vs institutional knowledge
This however, cannot be done alone — see my previous article on working with senior engineers also; an experienced engineer can only make the impact they are empowered to make, and it is the job of the leadership to facilitate this. The challenge for the leadership group arises typically from the confusion in between experience and institutional knowledge: very often someone with a lot of institutional knowledge is described as a person with a lot of experience in the company, which, as I’ve explained above, is actually correct. However, this is where is very easy to make the leap and assume that “experience in the company” / “experience with the company system” / “experience with our platform” is the same as “experience”.
Why is this a problem?
Institutional knowledge presents first of all a higher barrier for new members of the team: anyone who will join the team will only become fully efficient once they assimilate all of this knowledge so they can effectively operate changes in your platform. So your onboarding time will be longer.
Secondly, institutional knowledge creates single points of failure: typically this knowledge is stored in people’s heads, and more often than not in a single person’s memory. And this makes that person a single point of failure.
Sure, you can organize knowledge transfer sessions and use documentation to combat this, but even that requires additional institutional knowledge: because now your team has to know there is a documentation somewhere which instructs them how to deal with certain situations, failures etc.
Also, as I said institutional knowledge is a result of tech debt. And a lot of institutional knowledge is a sign of a huge tech debt — which is bad as tech debt holds you back.
Another important issue here is the Stockholm syndrome; when applied to tech and institutional knowledge, this means that a person or a group of people are getting so used to something that is uncomfortable and not standard but they know it so well that they are going to hold on to it and resist change. In other words, people end up using this institutional knowledge so often that it becomes second nature and when presented with the opportunity to make things “right” and change it they will resist it because any change will push them out of what has become now a comfort zone.
And this is probably one of the aspects where I see a lot of leaders failing: too often an experienced engineer would advocate for standardizing things and getting rid of tech debt and institutional knowledge, and they find themselves in conflict with the ones possessing this knowledge who will oppose the change; as a leader meant to help the team advance from a situation like this, if you don’t understand the differences laid out above, it is very easy to make the wrong decisions.
I’ve seen many cases where such people who realized they have become essential to the team due to their institutional knowledge get promoted, because the manager has realized the single point of failure and they are trying to provide incentives to the individuals so they don’t leave and check-mate the team in doing so. The problem in doing this, is that coupled with what I labelled above as the Stockholm syndrome, this now applies even more breaks on your changing and improving the platform: any experienced engineer who will try to change things will be opposed by someone who is perceived as equally (or perhaps even more) senior, due to the recent promotion, who will oppose change. I’ve seen systems built like a house of cards where all of the “quirks” (another term used often for “institutional knowledge”) of the system are with one person, be them Technical Lead or Chief Architect or some other title denoting seniority — and these people will do everything to make sure their job stays relevant by ensuring their institutional knowledge stays relevant; and the only way to do that is to not remove it and oppose change.
Conclusion
When hearing the term “experience” in engineering, a leader must always dig deep to find out if this is in fact institutional experience and knowledge; while it is easy to confuse one with the other, the difference is huge as well as the implications. Listen to the voice of generic experience and try to minimize the need for institutional knowledge.
