Table of Contents

Foreword xix
Preface xxi
Acknowledgments xxv
About the Author xxvii

Part I: Measurement 1
Chapter 1: My Program Is Too Slow 3
Chapter 2: Measuring CPUs 15
Chapter 3: Measuring Memory 31
Chapter 4: CPU and Memory Interaction 49
Chapter 5: Measuring Disk/SSD 61
Chapter 6: Measuring Networks 85
Chapter 7: Disk and Network Database Interaction 111

Part II: Observation 131
Chapter 8: Logging 133
Chapter 9: Aggregate Measures 141
Chapter 10: Dashboards 157
Chapter 11: Other Existing Tools 167
Chapter 12: Traces 193
Chapter 13: Observation Tool Design Principles 209

Part III: Kernel-User Trace 217
Chapter 14: KUtrace: Goals, Design, Implementation 219
Chapter 15: KUtrace: Linux Kernel Patches 227
Chapter 16: KUtrace: Linux Loadable Module 239
Chapter 17: KUtrace: User-Mode Runtime Control 245
Chapter 18: KUtrace: Postprocessing 249
Chapter 19: KUtrace: Display of Software Dynamics 257

Part IV: Reasoning 267
Chapter 20: What to Look For 269
Chapter 21: Executing Too Much 271
Chapter 22: Executing Slowly 279
Chapter 23: Waiting for CPU 289
Chapter 24: Waiting for Memory 299
Chapter 25: Waiting for Disk 307
Chapter 26: Waiting for Network 319
Chapter 27: Waiting for Locks 337
Chapter 28: Waiting for Time 357
Chapter 29: Waiting for Queues 361
Chapter 30: Recap 383

Appendix A: Sample Servers 387
Appendix B: Trace Entries 391

Glossary 397
References 405
Index 415

About the Author

Richard L. Sites wrote his first computer program in 1959 and has spent most of his career at the boundary between hardware and software, with a particular interest in CPU/software performance interactions. His past work includes VAX microcode, DEC Alpha co-architect, and inventing the performance counters found in nearly all processors today. He has done low-overhead microcode and software tracing at DEC, Adobe, Google, and Tesla. Dr. Sites earned his PhD at Stanford in 1974; he holds 66 patents and is a member of the US National Academy of Engineering.

Reviews

"Complex software often uses threads, events, and I/O to produce results. Even intermittent performance bugs can lead to functional failures when results are time sensitive. Here Sites shares his methods from decades of experience as a real-world performance detective to enable you to approach the deduction skills of an IT Sherlock Holmes."
—Mark D. Hill, Partner Hardware Architect at Microsoft and Professor Emeritus at the University of Wisconsin-Madison "I think that any senior CS student or professional can benefit by reading this book. While all the material in the first half of the book leads up to the use of KUTrace, the first two parts are worth reading on their own by anyone who wants to better understand the systems they are building and using."—Rik Farrow, ;login