Table of Contents

Preface to the Third Edition xiii

Acknowledgments xix

About the Author xxi

Part I The Measurement Solution Exists 1

Chapter 1 The Challenge of Intangibles 3

The Alleged Intangibles 4

Yes, I Mean Anything 5

The Proposal: It’s about Decisions 7

A “Power Tools” Approach to Measurement 10

A Guide to the Rest of the Book 11

Chapter 2 An Intuitive Measurement Habit: Eratosthenes, Enrico, and Emily 15

How an Ancient Greek Measured the Size of Earth 16

Estimating: Be Like Fermi 17

Experiments: Not Just for Adults 20

Notes on What to Learn from Eratosthenes, Enrico, and Emily 25

Notes 27

Chapter 3 The Illusion of Intangibles: Why Immeasurables Aren’t 29

The Concept of Measurement 30

The Object of Measurement 37

The Methods of Measurement 40

Economic Objections to Measurement 48

The Broader Objection to the Usefulness of “Statistics” 52

Ethical Objections to Measurement 55

Reversing Old Assumptions 58

Notes 65

Part II Before You Measure 69

Chapter 4 Clarifying the Measurement Problem 71

Toward a Universal Approach to Measurement 73

The Unexpected Challenge of Defining a Decision 74

If You Understand It, You Can Model It 80

Getting the Language Right: What “Uncertainty” and “Risk” Really Mean 83

An Example of a Clarified Decision 84

Notes 90

Chapter 5 Calibrated Estimates: How Much Do You Know Now? 93

Calibration Exercise 95

Calibration Trick: Bet Money (or Even Just Pretend To) 101

Further Improvements on Calibration 104

Conceptual Obstacles to Calibration 106

The Effects of Calibration Training 111

Notes 118

Chapter 6 Quantifying Risk through Modeling 123

How Not to Quantify Risk 123

Real Risk Analysis: The Monte Carlo 125

An Example of the Monte Carlo Method and Risk 127

Tools and Other Resources for Monte Carlo Simulations 136

The Risk Paradox and the Need for Better Risk Analysis 140

Notes 143

Chapter 7 Quantifying the Value of Information 145

The Chance of Being Wrong and the Cost of Being Wrong: Expected Opportunity Loss 146

The Value of Information for Ranges 149

Beyond Yes/No: Decisions on a Continuum 156

The Imperfect World: The Value of Partial Uncertainty Reduction 159

The Epiphany Equation: How the Value of Information Changes Everything 166

Summarizing Uncertainty, Risk, and Information Value: The Pre-Measurements 171

Notes 172

Part III Measurement Methods 173

Chapter 8 The Transition: From What to Measure to How to Measure 175

Tools of Observation: Introduction to the Instrument of Measurement 177

Decomposition 180

Secondary Research: Assuming You Weren’t the First to Measure It 184

The Basic Methods of Observation: If One Doesn’t Work, Try the Next 186

Measure Just Enough 188

Consider the Error 189

Choose and Design the Instrument 194

Notes 196

Chapter 9 Sampling Reality: How Observing Some Things Tells Us about All Things 197

Building an Intuition for Random Sampling: The Jelly Bean Example 199

A Little about Little Samples: A Beer Brewer’s Approach 200

Are Small Samples Really “Statistically Significant”? 204

When Outliers Matter Most 208

The Easiest Sample Statistics Ever 210

A Biased Sample of Sampling Methods 214

Experiment 226

Seeing Relationships in the Data: An Introduction to Regression Modeling 235

Notes 243

Chapter 10 Bayes: Adding to What You Know Now 247

The Basics and Bayes 248

Using Your Natural Bayesian Instinct 257

Heterogeneous Benchmarking: A “Brand Damage” Application 263

Bayesian Inversion for Ranges: An Overview 267

The Lessons of Bayes 276

Notes 282

Part IV Beyond the Basics 285

Chapter 11 Preference and Attitudes: The Softer Side of Measurement 287

Observing Opinions, Values, and the Pursuit of Happiness 287

A Willingness to Pay: Measuring Value via Trade-Offs 292

Putting It All on the Line: Quantifying Risk Tolerance 296

Quantifying Subjective Trade-Offs: Dealing with Multiple Conflicting Preferences 299

Keeping the Big Picture in Mind: Profit Maximization versus Purely Subjective Trade-Offs 302

Notes 304

Chapter 12 The Ultimate Measurement Instrument: Human Judges 307

Homo Absurdus: The Weird Reasons behind Our Decisions 308

Getting Organized: A Performance Evaluation Example 313

Surprisingly Simple Linear Models 315

How to Standardize Any Evaluation: Rasch Models 316

Removing Human Inconsistency: The Lens Model 320

Panacea or Placebo?: Questionable Methods of Measurement 325

Comparing the Methods 333

Example: A Scientist Measures the Performance of a Decision Model 335

Notes 336

Chapter 13 New Measurement Instruments for Management 339

The Twenty-First-Century Tracker: Keeping Tabs with Technology 339

Measuring the World: The Internet as an Instrument 342

Prediction Markets: A Dynamic Aggregation of Opinions 346

Notes 353

Chapter 14 A Universal Measurement Method: Applied Information Economics 357

Bringing the Pieces Together 358

Case: The Value of the System That Monitors Your Drinking Water 362

Case: Forecasting Fuel for the Marine Corps 367

Case: Measuring the Value of ACORD Standards 373

Ideas for Getting Started: A Few Final Examples 378

Summarizing the Philosophy 384

Notes 385

Appendix Calibration Tests (and Their Answers) 387

Index 397

About the Author

DOUGLAS W. HUBBARD is the inventor of Applied InformationEconomics (AIE), a measurement methodology that has earned himcritical praise from The Gartner Group, Giga Information Group, andForrester Research. He is an internationally recognized expert inthe field of decision analysis and challenging measurements and isa popular speaker at numerous conferences. He has written articlesfor Information Week, CIO Enterprise, and DBMSMagazine. He is the author of The Failure of RiskManagement: Why It s Broken and How to Fix It andPulse: The New Science of Harnessing Internet Buzz to TrackThreats and Opportunities.