The odds that odd things will happen: The Improbability Principle (Hand, 2014)

   

David J. Hand. The Improbability Principle (2014) Say you go to Timmie’s and meet a person you haven’t seen for 50 years. “That’s a one in a million chance!” people say. Which means that in Canada it will happen to about 37 people. If you mean one in a million per year, that’s 37 this year, and about 900 or more during the 20-odd years since I turned old enough to meet someone for the 1st time in 50 years. “One in a million” isn’t such overwhelming odds for or against after all.

     And that’s Hand’s point. We are bad at estimating odds. Very bad. We react with Wows! to many things that we should expect to happen pretty often. Take the meeting a long-ago friend or neighbour: Since we travel much more than we used to, the odds are far better than they used to be that we will come across someone we knew some decades ago, and even better that that we will meet people who know people we know.

     Explaining why the improbable happens much more often than we expect, Hand provides an excellent introduction to probability and statistics. He writes clearly, with occasional glimmers of a pleasantly dry wit. Anyone who gambles should read this book. His discussion of drug testing will make the reader skeptical of pretty well all news about medical breakthroughs. Which reminds me that reporters of dramatic rate increases in something or other almost never give us the actual numbers. Reporting a 100% increase in some rare disease is much more exciting than reporting that 2 more deaths are expected this year in Ontario.

     And it’s much more thrilling to read about a traveller who changes his flight plans, and so isn’t one of the 300-odd who die when the plane crashes into a mountain. We don’t read about the hundreds of thousands of travellers who have changed their flight plans every day, but have never missed being killed in a plane crash.

     One aspect of our innumeracy that Hand doesn't mention is our difficulty imagining large numbers. That's certainly one reason we have trouble estimating odds. John Paulos says some entertaining things about that in his Innumeracy.

     Well done, recommended. ***½

 

 Another review of this book: https://kirkwood40.blogspot.com/2018/01/the-improability-pirnciple-why-we-dont.html

Oddities in print

     Russell Ash & Brian Lake, eds. Bizarre Books (1985) Ash and Lake share a fascination with publishing oddities: titles that suggest oddball subjects, oddball subjects as such, and books that prompt the question What Were They Thinking? This bibliography, with occasional summaries of excerpts, should satisfy anyone with a similar interest, and will provide mild amusement for anyone needing to fill a few idle minutes here and there. Here are few samples:
     The Hookers of Kew (1967; about a family of botanists)
     How to be Plump (1878)
     The Fangs of Suet pudding (1944; a thriller)
     Frog Raising for Pleasure and Profit (1950; includes recipes)
     How to Fill Mental Cavities (1978)
This book itself should of course be included. **½

Scientific ideas we should forget

 

    John Brockman, ed. This Idea Must Die (2015) A compilation of answers to the question, “What scientific idea is ready for retirement?”, posed on https://www.edge.org/ in 2014. Brockman arranges the answers, starting with general ones, then roughly by topic, such as quantum physics, neurology, evolution, etc, and ending with math and statistics. Often, a short sequence of essays reads like a dialogue.

     Most answers are directed at a general audience, which of course includes scientists in other fields. The writers try to explicate how the target concept causes mistakes or worse, what a better understanding would look like, and sometimes what concept should replace the target. A handful read like part of an ongoing dispute between the writer and the other specialists in the field.

     I was pleased to see that many of my objections, puzzlements, and exasperations were confirmed or clarified in these essays. One of these is the wave-particle duality interpretation of some experiments in quantum physics, which I think is a holdover from the days when observations and models made a nice clean distinction between things that rippled through, and things that bumped into, each other. QM equations show that this distinction isn’t much use. It’s nonsense to say that entities are both waves and particles. It would be like arguing that because people sometimes exhibit fear and at other times exhibit joy, that human beings are somehow both fearful and joyful all at once.

     Another of my annoyances is Schrödinger’s Cat. I’m glad to see that Freeman Dyson notes that the wave function isn’t a thing, so it doesn’t collapse. It’s statement of probabilities in some specific context. (Or conversely, it’s a context defined by a distribution of probabilities). An observation measures one of the probable states. At another time, another state will be observed. To argue that somehow all probable states exist at once is like arguing that because Jack is sometimes angry and sometimes happy when he goes to a baseball game, that therefore Jack is both angry and happy until he goes to the game.

     I found some of the best entertainment in the essays dealing with psychology. One writer attacks a concept, another assumes that same concept in order to attack another one. So what’s an non-expert to do?

     However, the overall effect of reading these essays is the somewhat depressing reminder that we all hold erroneous or misunderstood scientific ideas. They appear in news reports and TV punditry hourly, and many of them have very bad effects on public understanding and thereby on public opinion, which in turn limits politicians’ beliefs about what can and should be done.

     Misunderstanding of basic math is nowhere more obvious than in news about statistics. Case in point: This morning, I heard a report on rising rates of STDs in Alberta, a roughly 40% increase overall in the last ten years, with the highest rate increases among the young and the old, and the lowest among the middle aged. Well, without the actual numbers, rate increases are pretty well meaningless. An increase of, say, from 10 to 20 per 10,000 young would be a 100% increase, while from from 100 to 150 per 10,000 middle-aged people would be only a 50% increase. 50% sounds a lot better than 100%, right? But in this example, 50% is worse, since 50 extra cases will cost five times as much as 10 extra cases.

     The final essay, by Paul Saffo, reminds the reader that the more we know, the more unknowns we encounter. Saffo refers to Teilhard de Chardin’s noosphere, the sphere of knowledge. As it expands into the unknown, its surface increases, the contact between known and unknown increases. I developed this idea on my own many years ago, when I thought of the known as an expanding circle. 2D instead of 3D, but otherwise the same. Either way, there will never be an end to the questions we can ask. Even better, there will always be more questions to ask than have already been answered. But Socrates said as much 2,500 years ago. History echoes.

     Highly recommended, as is the website. ****

The Harlequin Tea Set: More Christie short stories

 

    Agatha Christie. The Harlequin Tea Set... (1997) Except for the last two, these stories were published in various magazines and not collected until now. All except one were written in the 1920s and 30s; they cater to the taste of that time. Christie shows she can write romance, often with a touch of the supernatural, psychological studies, and of course whodunits. Christie had mastered the craft long before she became the Queen of Crime.
     The plots generally have a twist calculated to surprise the reader, such as a star actress who doubts her skills, yet uses them to trick a would-be blackmailer into leaving the country, thus proving to herself that she’s a real actress. All are written to create an ambience, a mood: a small statue of some pagan god displayed in a museum becomes the occasion of two lonely people meeting and falling in love. Poetic justice figures in most of them, as in the story where a woman marries well after losing her first love, who reappears, and then kills himself so that she will not be tempted to leave her husband. But she will now forever know that he died not realising that she had become a morally lazy lover of creature comforts who would never have left her rich husband.
   And so on. Christie fans will be happy, and the casual reader may be entertained.  Perhaps useful for anyone wondering about their ancestors' taste. * to ***

2026-06-02: Edited.

The Mean Streets: Private Eye Stories.

     Bill Pronzini & Martin Greenberg. The Mammoth Book of Private Eye Stories (1988) The PI story is quintessentially American. Since about 1950, writers in other countries have tried their hand at it, but my reading suggests that they capture the bleak existentialism of the PI tale best with police officers, perhaps because the PI in most of the US has a larger scope of action than in Europe.
     The editors offer these tales in chronological order by author birth dates. A couple are duds, not because of the plots but because of the writing. Whether first or third person, it’s the PI’s off-hand observations about the weather, the characters’ faces, the smells and sounds of morning-after bars, that create the ambience which convinces us that these dark fantasies are true. That’s a style that’s easy to parody and difficult to do well.
     All the expected authors are here, Chandler, Hammett, Ross MacDonald, Sue Grafton, and so on, as well as some I know only from reviews. A wonderful potato-chip book, you read one story, and you want to read the next one just to taste that dark and near-despairing view of life again.
     Recommended, if you can find a copy. Numerous typos mar an otherwise near-perfect collection: the effect of early use of spell-checking, no doubt. ***

Oil Cartel, anyone?

Today's New York Times reports that OPEC and Russia will be meeting (probably) to discuss a freeze on pumping oil, and perhaps exercise "market discipline."

What is market discipline? An agreement among sellers to control prices. Which happens to be illegal everywhere. Well among the G20, anyway. So why is OPEC allowed to get away with it?

The Limits of Knowledge

 Theory, Model, Algorithm, and the Limits of Knowledge

Three terms that are often used interchangeably. They do have something in common, we’ll see what it is after an attempt to differentiate them, by describing how what they refer to differs.

Framework: The world we live in is “reality”. We interact with it in various ways. As we grow from infancy to adulthood, we develop various methods of predicting how reality works so that we can get what we need and want. Explicit ideas about how reality works are the theories on which we base our actions. We reason about the state of reality right now so that we can change it to suit ourselves. For example, we plant seeds when we figure the weather is favourable so that we get tomatoes a couple of months later. We add fertiliser and soil conditioners and water to ensure that the tomatoes will grow. Those actions are based on a bundle of ideas and observations that form a more or less coherent theory about how tomatoes grow from seeds.

Theory: An explanation of how something works the way it does. It’s what you get when you test a hypothesis, which is a more or less speculative explanation of some observation(s). A good hypothesis links the observation(s) to some existing explanation, and predicts additional observation(s). If those predictions are proved true, then the hypothesis is confirmed and becomes a theory. A good theory implies or suggests further hypotheses, which in turn imply new observations. When a theory is applied to some practical problem, we get a model. That, and the desire to just figure things out, are what drives science and engineering.

Model: An explanation that can be used to predict how some part of reality will work. We use this term because a conceptual model about growing tomatoes is analogous to a physical model of, say, a steam locomotive. A scale model is not a replica, it is something that looks like, and in a limited way works like its prototype. The model locomotive may operate on steam as the prototype does, but even so, there will be compromises. E.g., the thickness of the boiler shell will not be to scale for that would make it too weak to contain the necessary steam pressure. And so on.

We use both models and theories to plan what to do so as to get some desired result. The difference is subtle. We test a theory’s predictions in order to discover its limits, so that if necessary we can modify it or even replace it. We use a model within its limits to control some aspect of reality as much as possible. We may use a model to test a theory; an experiment is a model constructed from that part of a theory that we wish to test. It’s not easy to derive a model from a theory: models also have to be tested.

Both models and theories are true insofar as they work. When a model becomes a precise set of rules, it becomes an algorithm.

Algorithm: A set of procedures applied to some inputs that will produce outputs in a predictable way. Thus, “long division” is an algorithm because it describes how to manipulate the input numbers (divisor and dividend) to get the answer (quotient). A recipe for a toasted cheese sandwich is an algorithm because it describes how to manipulate the inputs (ingredients and  heating device) so as to get an output (tasty sandwich). And so on.

Algorithms are everywhere. They are especially handy for determining future values of present states. In this sense, an algorithm is a knowledge machine: input information about “this thing here and now”, turn the crank, and you get information about “this thing somewhere, somewhen, somehow else”.

If the above comments make sense, we may see a model as a set of interrelated algorithms, and a theory then becomes a set of validated and interconnected models.

And that brings us to what they have in common: All three are modes of gaining new knowledge. All three operate on the same fundamental principle: “If you do this, you will find out that”. None of them “describe reality”. They describe only how we may observe certain aspects of reality. Which ones? Those that the theory or model or algorithm “is about.” What “is about” means is not easy to say. An example will explain (as far as the example applies, that is):

We may use Newton’s laws of motion to build a model that calculates the course of a rocket launched towards Jupiter. If we know its mass and its velocity, the varying gravitational forces of the Moon and Mars, the trajectory of the rocket, etc, we can calculate, and recalculate, its course to whatever precision we like. But the model will tell us nothing about the health of the crew, if any. If we want to know that, we need another (and more complicated and less certain) model. The model cannot tell us what the rocket “really is”, only how it interacts with gravitational fields and the reaction forces of its engines. If we want to know other things about it, we must use other models. What’s more, even to monitor the course of the rocket, we have to use other models.

Thus all theories, all models, all algorithms are knowledge engines. They are epistemological devices. But they are limited. They can’t tell us what some entity really is, only how we can interact with it, and what will happen when we do so. Even the notion of “entity” is fundamentally epistemological: An entity is a more or less consistent bundle of expected interactions. If any of them are missing or unexpected, we doubt that we are interacting with that entity. It may be an hallucination, or a dream, or a fake, or merely an image of the entity

Kant was right, I think: There is no way to know reality in itself. That doesn’t mean there is no reality “out there”. It just means that we can know only our interactions with it. That we can know even that much is, I think, an even greater puzzle than what it is that we can’t know.

(c) 2016-04-07 Edited 2025-10-20