Math vs Science vs Engineering is (mostly) Orthogonal to Benchtop vs Production

1

During @kgostic’s talk, I asked how CFA was thinking about what distinguishes their “more engineered” mindset from whatever it is infectious disease modeling for public health currently is. I also qualified my question by saying that what it sounded like they are using the term to mean is “production” as opposed to “benchtop”. (aside: I should emphasize that I do take seriously the caveat about not speaking on behalf of USG, HHS, CDC, or CFA - I understand that the answer during the talk is not the official position of any of those entities)

@samabbott goaded me into opining on the topic, so I’ll try to summarize my position here, and hopefully folks will find that interesting and clear enough to argue with.

I’ve already done a very short summary of my position via the post title. To rephrase in complete sentences: I think Math, Science, and Engineering are different activities - and that difference manifests in usefully different institutions and practices around those activities. I also think that benchtop vs production is a different axis. I’m going to define all of those words for the purpose of this argument; feel free to @ me that those definitions are nonsense.

First, the M-S-E category is distinguished by their fundamental questions:

  • Math: what do the rules mean?
  • Science: what are the rules?
  • Engineering: what do we do with these rules?

Or put into activity terms, Mathematicians might try to answer “what does this set of rules for differentiation mean for the relationship between postion, velocity, acceleration, jerk, ad infinitum?” while Scientists pursue “Is Impetus or Energy a better model of projectile motion?” and Engineers “How can I hit that target with enough rounds, as quickly and cheaply as possible?”

These different motivating tasks lead to different institutions around the disciplines and their practictioners, despite the fact that all three groups have a lot of surface overlap in language and tools. There’s no such thing as mathematics license, while there is a professional engineers license - if you’re cutting corners in math, your result is just wrong, but a poorly engineered design can kill people. Scientists are taught to get the best possible measurement, engineers the best practical one. And so on.

I posit that we can find example after example of how those three groups view a problem in different ways, traceable to those fundamentally different telos informing how they have been taught to work, what to value, the incentives they experience and on and on. And when I say view in different ways - I don’t just mean “approach differently”, I mean “define differently”.

Despite my assertion that the talk implied mostly the mindset that Engineering == better Science – i.e. that its production vs benchtop distinction – I do think there were points that acknowledged some of these deep distinctions. Most notably, the accuracy versus timeliness point. But the were others that hinted at not yet shifting into the engineering mindset - e.g. the notion that we need to aggressively propogate all the CIs. Engineers frequently use point estimates, just not the central ones! A first pass, engineering mindset calculation uses whatever tolerance boundary is worst for the outcome you’re seeking to avoid. Even using tolerances on parts that are mutually-exclusively bad - dealing with that sort of thing is a second pass version of the calculation, if the first doesn’t get the job done.

On to defining Benchtop vs Production: something that works on the proverbial benchtop entails a lot of handwork and is generally quite sensitive to … well, everything, and most particularly “who” is doing the work. Something that’s Production can be turned out within tolerance (n.b. that doesn’t mean identically) by most anyone under pretty mean conditions – that means a simple-to-follow recipe, but often with tools that are very robust (to chew through user and material input flaws).

But that distinction has nothing to with whether an activity is fundamentally a scientific one (what are the rules?) or an engineering one (what do we do?). Now that said: science does generally lean “benchtop” and engineering generally leans “production”, but that tendency is downstream of the fundamental distinctions between the disciplines, not itself one of the fundamental characteristics.

That seems like enough opening rant - return salvos, please!

1 Like
2

I want to come back and comment more substantially, but this discussion of “an engineering orientation” reminds me of my chemical engineering undergraduate–nearly every course would start with images of the Bhopal disaster with the instructor saying “this is what happens if you get it wrong, so propagate uncertainty all the way through, then add a backup pump.”

1 Like
3

My engineering heritage was focused on the USS Thresher and Chernobyl, so yeah, big same energy.

But to be clear: accounting for uncertainty isn’t precisely the same as literally projecting the uncertainty distributions all the way through the calculation. That’s the kind of stuff we did as a last resort (e.g. when one extreme tolerance is worst for a certain kind of failure, while the other extreme is worse for a different kind failure, but the crisis condition is the combination of those two failures, and assuming both of those makes for device that is unmanufacturable to pass muster).

2 Likes
4

HAEC SUNT DRACONES: THIS IS A DRAFT IN PROGESS

I’ve made enough progress here that I’m happy to expose myself to critique, but this is incomplete thoughts. I think I’m more likely to actually finish it with a more live version of that threat.

More Thoughts

Since @samabbott decided to paint a twitter shaped target on this, feels like I should do a bit more honing on the argument here.

Not-Science Doesn’t Mean I Don’t Like You

I’ll start by clarifying what might come across as an attack: if I opine something is e.g. “not science” that isn’t a value judgment. I’m offering a mis-categorization assessment, among equally useful categories. That assessment is intended to help the recipient best accomplish their aim, by either adjusting that aim (because they actually want to do e.g. science) or by adjusting their activity (dropping irrelevant science-y behaviors, adding relevant other-category-y practices). I used “not science” example because that’s the one most confused in my immediate vicinity (possibly most generally - think “follow the science” sloganeering), but I also use “not math” and “not engineering” as well (the instigating post here being a bit of a “not engineering” example).

Rules, Rules, Rules

Revisiting how I made those category distinctions: I frame all of the definitions in terms of how they related to “the rules”. That’s deliberate; all of these fields use formal representations, descriptions, systems, &c to do their business - in short, “rules”. These rules are frequently (but not exclusively) expressed as mathematical relationships and/or computational algorithms.

Aside (1/3): I’m treating mathematical language is distinct from Mathematical activity - if that seems a bit odd, think about it in terms of normal language. The grammatical conventions and vocabulary of English are a separate thing from, say, writing fiction or rhetorical debate, and further distinct still from the linguistical study of the evolution of English. If my job is to write opinionated blog posts for an English-speaking audience, clearly I need to use relevant conventions and words, and rely on shared idiom. Were my job to study English itself, I would likewise need to know those, but in a different way to a different end.

Aside (2/3): I distinguished mathematical and computational representation. I’m aware that, in some sense, mathematical == computational. I think they’re practically distinct enough to invoke both for example purposes.

Aside (3/3): Implicitly, trying to do any of these three activities without any reference to “the rules” is … well, close to impossible. I’m aware of some of the philosophical arguments to the contrary. I don’t find them particularly compelling. Moreover, if I see someone claiming to science with no attempt to have theory or denying various guardrails on data collection and interpretation apply, then my assumption is that they are frauds (possibly true-believers, but frauds nonetheless). If y’all want to rely upon “other ways of knowing”, fine, but let’s not muddle vocabulary. On the flipside, yes, scientism is a thing. Also, if a practice / belief / assumption descends from non-scientific evidence, that doesn’t make it wrong, just lacking in scientific justification.

More About Definitions

Regarding the my crude category summaries, I think it’s also useful to understand the what-it’s-not perspective. I’ll also add my pitch for “equally useful” distinctions.

Maths

For mathematicians, “what do the rules mean” implies that, in some sense, they don’t care what the rules are or how they are used. Perhaps, dear reader, you are recall the great “2 + 2 = 5” kerfluffle of … 2020? 2021? Feels like it was pandemic times. Anywho: many matheticians weighed in that this was, indeed, just fine - having the rules arranged such that this equation is valid (instead of the more common convention) is a reasonable thing to think about as a mathematician. Probably the simplest version is just to swap the (arbitrary) meaning of 4 and 5 symbols. Scientists might reply “sure, but if you measure how people use those symbols, you’re making the wrong definition”. Engineers might reply “Okay, but that’s not the spec, and this spec change doesn’t get anything built better / faster / cheaper - probably the opposite.”

Likely mathematicians can cook up more interesting versions of self-consistent systems where “2 + 2 = 5”, and considering those rule sets for their own interesting sake (without worrying about assigning any meaning or correctness to variables / operations therein) might well yield interesting generalizations, even if the particular “2 + 2 = 5” system isn’t itself userful. Maths as a discipline is primarily concerned with finding insights about working with rules, rather than directly about whatever those equations might signify.

So if folks doing mathematics’ core activity aren’t fussed with the concerns of the “real world”, what use are they? Certainly some of them would make aesthetic, mystical arguments - finding beauty, higher truths, and so on. Fair enough, and probably great for motivation people within the field. But for the skeptical, truth-and-beauty-hating outsider, there are indeed practical benefits. Insights about how various rules relate to each other mean that we can confidently take shortcuts - for example, math let’s us know that the Fast Fourier Transform is equivalent to the plain expression of the algorithm. Thus, we can apply it to get the same result, just fast(er).

Sciences - WIP

Pivoting to to the task of “what are the rules?”, I’d argue that Scientific activity should be disinterested in the to-what-ends / so-what / who-cares questions (despite assorted guidance about publication to the contrary). That way lies advocacy, which justifies process shortcuts and undermines the likelihood of getting closer to truth. Similarly, focus on working with the rules (as opposed to focusing on which rules are correct) will lead to a preference for systems that are convenient, elegant, interesting, etc in terms of rules - and those are, again, not necessarily the correct ones.

Scientists, being human, of course have trouble with the advocacy problem. We shouldn’t imagine it possible to eliminate that problem - but rather that our duty is to combat it (in ourselves, our team, others)

Engineering…s - WIP

Engineers: “what do we do with these rules?” At last we come to the somewhat focus on my complaint in this post. Unlike Science, Engineering is a matter of advocacy. The state of affairs is a particular way, we’d prefer it to be another way, what are the means to get from here to there?

Placeholder Asides

Is-vs-Ought - or positive vs normative distinction. Maths concerned with Is, Science mostly Is, though analysis can be used to evaluate Ought assertions (i.e. it ought to be this way, so we should do this - scientific activity can assess whether do intervention => achieve outcome Is). Engineers start from Ought, and use Is to acheive that.

5

I mostly agree with @pearsonca , so I have a few questions and maybe examples to provide.

Engineering as seeking to change the world

In the Existential Pleasure of Engineering, the author points out that engineering is fundamentally about changing the state of the world, though it is rooted firmly in science. This ties back to Carl’s “what to do with these rules.” More concretely, engineers are often asked to change something about the world under generally fixed constraints where decisions have to be made outside of the traditional scientific process.

The engineering process

I think I good example of this is thinking through a traditional design engineering workflow. A customer/client (internal, external, whoever) writes a Functional Specification[1]. As implied, this document lays out what the client wants, functionally (not how to do it). For example, “I want a machine to make 50 widgets per day, run with 1 person, and break down no more than 5% of the time, be able to change from making red widgets to green widgets in less than 10 minutes, fit in the available space, cost less than $500k, make parts within +/- 5 mm, and follow standard safety rules.”

The engineering team then responds with the Technical Specification which provides the how these functions will be met. Continuing with the above example, the engineering team will say document the technical approach to making the machine–it will be made out of steel, it will have quick change parts, we will have multiple spray nozzles that trigger when a proximity switch is hit after being manually loaded by the machine operator with containers for red and green ink, it will have emergency stop buttons and pull lines, we’ll use ControlLogix controllers, with etc. These design decisions are based in science, but also fundamentally represent decision-making in the context of limited resources. I could make the machine out of titanium, but it would cost a fortune. I could make the machine really large to allow easier maintenance, but now it doesn’t fit within the building. This detailed response of the “how” is rooted in science (e.g., you have to understand materials, you need to know which measures of central tendency to use, you need to know how to write good code with food logic). Fundamentally, the role of the engineer is to design/build something that meets the functional requirements and produces outputs that meet the tolerances for the job at hand[2].

Tolerances

I think a term that @pearsonca uses that needs some unpacking is “tolerance.” Tolerance differs from measurement error/statistical uncertainty in that a the latter is inherent in the process while the former is generally related to meeting the functional demands of the situation. If a bolt is out of tolerance it won’t fit (but each bolt will vary in size). If a beam is out of tolerance, the bridge collapses. To meet a tolerance, you have to be aware of your source of uncertainty (and everything else that could happen) and hold it in tension with the functional demand (which is where it differs from a scientific ethos).

Individuals exist on a continuum

Any one individual may do any one of the activities at a given time. For example, a scientist in realizing that the device measuring something phenomena realizes that they need a better sensor, may generate a functional specification ("I need an apparatus that holds this sensor and flexes less than 1mm laterally), generate a technical spec (“I will make a reinforced steel sensor holder that is bolted to the bench”), and then build and test it–and thus performed engineering activities.

Objectivity of science

I agree here with Carl that science qua science should be rational and objective[3]. This radical objectivity can make it look like it has the moral high ground, but describing the world as it is or modelling as it could be doesn’t in itself constitute a recommendation.

Benchtop vs Production

Every July I listen to 13 minutes to the moon. In season 2, they discuss Apollo 13 where there was a failure on the spacecraft causing a moon exploration mission to become a rescue mission. One engineering problem they had to overcome was with the CO2 scrubbers. Because one company designed the lunar lander and another the command module, one filter was round and the other was square. So the ground engineers had to design something that would fit a square filter into a circular hole. It was definitely an engineered solution–however when you look at the pictures it was certainly not “production” quality as it was built of book covers, a sock, duck tape, etc. Just another example of orthogonality of engineering vs production/benchtop.

  1. So the question becomes how does one get a functional specification. There are all kinds of exercises that are used to develop these from market research, “voice of the customer” exercises, the down stream process tolerances. I sometimes wonder what this part would look like in infectious disease modelling. We would have to identify our customers (likely in “market segments” like front-line public health, hospitals, local officials, other government officials, the general public). Then they would need some elicitation on what they would like to see and what their tolerances are which differ at different levels. Missing the hospitalization projection by 20 beds might mean that a hospital can’t staff a ward vs 20 beds on a country level is noise. ↩︎

  2. I sometimes thing about this in the context of funding. When you look at research grants vs research contracts. The latter has much more of an engineering approach. Make something that does X. ↩︎

  3. The German philosopher in “What is thinking” has some line about science being unable to tell “man” what to do. Heidegger was not a nice man, but I think his point holds. Science fundamentally lacks the grounds for telling someone what to do–those decisions are constrained by more human questions of ethics and finite resources (including our own). ↩︎

6

“voice of the customer” exercises, the down stream process tolerances. I sometimes wonder what this part would look like in infectious disease modelling. We would have to identify our customers (likely in “market segments” like front-line public health, hospitals, local officials, other government officials, the general public). Then they would need some elicitation on what they would like to see and what their tolerances are which differ at different levels. Missing the hospitalization projection by 20 beds might mean that a hospital can’t staff a ward vs 20 beds on a country level is noise.

I really like the idea of this + formalising it. Partly getting closer to the customer is what interests me about working at the CFA as they are both the customer and also have access to other levels of public health.

It was definitely an engineered solution–however when you look at the pictures it was certainly not “production” quality as it was built of book covers, a sock, duck tape, etc. Just another example of orthogonality of engineering vs production/benchtop.

Great example!

7

Thanks for this Carl. I think that it’s important to do both the engineering and the science well, especially for public health organizations whose mission is to produce answers quickly for decision support.

Sometimes there can be tension between the science side and the engineering side. E.g. if there isn’t yet a production-level pipeline in place, or if the pipeline isn’t sufficiently modular, then this approach can slow down the science.

Instead, our long-term goal is to build production-level codebases that enable scientific experimentation. E.g. with enough person-power directed at the engineering side of a problem like forecasting or nowcasting, we can build mature test suites and standard evaluation protocols for our models. In turn, this production-level code can make it faster to build and test new modeling approaches.

I think this will take a while to achieve, but this how ML experimentation is often done in industry, so with a large enough staffing footprint, including professional software engineers who can help us build and maintain these pipelines, it should be possible to bring this approach to public health.

1 Like