The Breakthrough How-To, Part 2

The previous post gave some background on Kuhn's theories about scientific development and breakthrough, and extended those ideas to industry. In this post we'll talk about how to find breakthroughs.

1. What sort of problems lend themselves to breakthroughs?
Let's start with the problem. There's a reason this blog is called "Seeking Questions". Breakthroughs tend to start with an open, unsolved, hard problem. But not just any sort of hard problem; certain kinds of problems lend themselves to breakthrough solutions.

Problem 1: Build a deep-sea oil rig. Breakthrough? Optional.

Problem 2: P vs NP. Breakthrough? Required.

Consider building a deep-sea oil rig. This is a very hard problem. Deep-sea oil rigs are very complicated, with dozens of subsystems and hundreds of thousands of parts. On the other hand, each component of a deep-sea oil rig is straightforward. The drill, the casing, the motor, the circulation, the stabilization... each of these is a previously solved problem. To build a deep-sea oil rig, each subsystem is assigned to a small group of engineers, and each group can design their part. The end result is complicated, but it is complicated only because of the number of parts, not because of the complexity of any single part. Deep-sea oil rigs do not require fundamentally new insights; they do not require any major breakthrough.

For a breakthrough, we don't want a problem which is hard only because it has a large number of straightforward pieces. So what kind of "hard" do we want?

Let's consider a particular open problem: P-NP. P-NP is a problem in computer science which asks whether or not two particular large classes of problems (P problems and NP problems) are equivalent. At this point, a solution or even any significant progress on P-NP would certainly be a breakthrough. Alas, we have little idea of how to approach the problem. It's hard to even find a starting point (a useful starting point, anyway). P-NP is not a problem with lots of straightforward pieces; it is a problem with a single large, cloudy piece which we don't understand. It requires fundamentally new insight.

So we have two examples of hard problems: a problem which is hard because it has lots of pieces, and a problem which is hard because it requires new insight. The former requires hard work and a large team to solve. The latter requires insight, and lends itself to breakthrough. I believe that most if not all hard problems fall into one (or sometimes both) of these categories. For breakthroughs, we want to look for the latter type of problem: problems which require new insight.

There's still variety within problems which require insight. There are insights which take five minutes and there are insights which take five years. Presumably, most of the time, more difficult insights are needed for harder problems and correspond to bigger breakthroughs. I might have a small breakthrough in a project at work in five minutes; I might have a large breakthrough on a major open problem in five years.

2. What sort of skillsets lend themselves to breakthroughs?
A big takeaway from the last section is that breakthrough-type problems involve new insight. At first, that makes it hard to anticipate what sort of skillset will be useful. If nobody's had the key insight yet, then presumably the key insight will not be included in any extant skillset. But that's not quite true...

For starters, the insight need only be new to the people working on the problem, not necessarily original. For example, both physicists and chemists have a long history of regularly invading biology (sort of like China's relation with the steppes). These invasions tend to produce major breakthroughs in biology, including bacterial locomotion and the birth of molecular biology. Taking knowledge from one field and applying it in another is about as close as we can get to a reliable recipe for producing breakthroughs. It's certainly not the only way, but it's probably the most reliable.

This suggests a generalist skillset as ideal for finding breakthroughs. Contrast to a specialist skillset: a specialist gains lots of practice within a particular paradigm, able to quickly and efficiently apply the tools of that paradigm. But when the specialist's tools fail, they have nothing to fall back on. When a different paradigm is needed, the specialist can make no headway. A generalist, on the other hand, has many more tricks to try when one paradigm fails. The more fields the generalist knows, the better. Of course, generalization has its tradeoffs: a generalist will usually be slower and more error-prone with any particular tool than a specialist. For problems which the specialist can solve, the specialist will produce a better solution faster. But the generalist shines when the specialist's tools fail altogether.

There's more to the story. A general skillset is preferable for breakthrough-type problems, but not all fields are equal. The tools of some fields are far more general than the tools of other fields. Mathematics, in particular, offers the most flexible and powerful tools for technical problems. Within mathematics, the tools of applied math tend to prove useful in new problems. After mathematics, computer science is a close second, especially in today's environment. That said, the generalist rule still applies: the more different tools you have, the more likely one of them will have the right insight for a new problem.

Next post we'll look at what sort of environment lends itself to breakthroughs.