Neither. I think. Super Crunchers is a fascinating study of the ascension of data analysis and statistics in decision and policy-making in all realms of life, from business to government to health. Ayres shows us how the ability to collect millions upon millions of data points and number crunch them to study trends, analyze relationships and make predictions, has created a schism between professionals (lawyers, educators and doctors) for example, who use their experience and expertise with intution to come to decisions, and data crunchers, who us the power of computers to do the same. Ayres uses significant case studies to demonstrate how super crunching consistently beats intuition.

The caveat is that data crunchers are also human, ultimately, and may make errors in selecting the inputs and variables that are analyzed. In addition, Ayres warns against the possible misuse of super crunching to, for example, create pricing systems where every consumer would be charge a customized price for a product based on a super crunching analysis that predicts how much each unique consumer is willing to pay.

Nevertheless, Ayres demonstrates the decision making and analysis power of super crunching with examples from medicine, law enforcement and education. The message is that, in sum, super crunching is good and is something that we should all become familiar and comfortable with because it will not go away and will, indeed, be a strong complement to intuition.

The gimmick in the TV show Numbers--and all crime shows have to have some sort of gimmick--is that a genius mathematician is able to help the FBI solve crimes. He particularly does so by finding patterns amongst the haze of large data sets. Ian Ayres's book Super Crunchers is a non-fiction look at a similar idea: the trend to find patterns and make predictions from analysis of large amounts of data.

The two principal ways that this is done--outlined in the first two chapters--is through regression and randomized trials. Simply put, regression sees how things have performed in the past and tries to extrapolate into the future; it's like plotting points in a graph and finding the best line or curve that fits that line. With randomized trials, you take a random sample from a population and see how these sample members react to a certain situation, such as a new drug or advertising slogan.

Of course, as Mark Twain is alleged to have said, there are lies, damn lies and statistics, and the same holds true with this "Super Crunching" of data. It only works properly if you know what you're doing and you're doing it properly. A well-known example (also cited in Freakonomics, which is kind of a companion piece to this book) deals with John Lott, the author of More Guns, Less Crime. According to Ayres (and the Freakonomics writers), Lott's crunching of the numbers had some serious errors, which when corrected, showed that the data actually contradicted his conclusion that the more guns owned in a community, the less crime took place. Did Lott purposely skew information to satisfy his own agenda or were these accidental mistakes (or did Ayres)? The reader can make his own judgment.

Beyond any issue with a particular study, the more relevant issue is the increasing prevalence of Super Crunching data and the effect it is having in determining business decisions and government policy. As cited early in the book, if the numbers are worked properly, a computer can do a better job predicting the performance of a wine or baseball player than a wine expert or a baseball scout.

In fact, often times the databases will provide better insight than experts. What does this mean? Will so-called experts go away? Will more and more decisions be based solely by computer with little or no human input? Is this a bad thing? Once again, the reader can make his own judgment as to whether this is a good or bad trend, though it is certainly an increasing one.

All this would mean little to the reader, however, if the writing is not good; fortunately, Ayres is successful at describing what Super Crunching is and what its possible ramifications are. You do not need to be a math or business expert to understand this book. And if Super Crunching is the future (even more than the present), it makes sense to read this book and get a glimpse of things to come.

The author explains that he originally intended to title the book, "The End of Intuition". I think that would have been the better title. Interestingly, the name change resulted in a clever use of Google's adwords. The problem with the title Super Crunchers is that it makes the impression that the book is about data mining huge data sets. I have read one scathing review in the KDD community that makes this point, but data mining is not the focus of the book. If you are looking for a book about data mining, it is a poor choice.

The theme of the book is how data, and data analysis, is making inroads in industries and firms where it has not been emphasized in the past and how the use of data to make decisions is often fiercely resisted. The techniques will seem mundane to professional analysts. I won't be recommending this to many of my data mining colleagues - there is nothing really new here. However, this book does have its place. I often meet analytically oriented managers that are trying to introduce advanced analytics into departments (or entire firms) that are not accustomed to data based decision making. It doesn't matter that Ayres' examples center on ordinary multiple regression, or standard randomized trials. This is where the battle is being fought and won in many organizations. The reluctance of some doctors to embrace "evidence based medicine" made for compelling reading.

The examples Ayres uses are diverse and unusual: wine rating, medicine, welfare reform, prison sentencing, dating web sites, Capital One and others. Some are much better than others. I didn't learn anything about data mining or analysis while reading this book. But I did learn something about people, and it confirmed some things I have learned about trying to deploy models in organizations. So while this book is not without flaws, if you want to better understand why there is often so much reluctance to use data to drive decisions it is worth a read.

This is an easy and mostly entertaining read. The author uses many anecdotes to
persuade us that statistics can be a useful tool for decision making. Some of
the described applications use lots of data and multiple regression. Those are
easier to do now than they used to be, because more data is collected and kept.
Some are trivial. If your company hurts a customer, apologize. You might get
some ideas of thing to do that might help your organization. You will not get
any detailed help about how to implement the improvement, but there is a good
chance there is enough information that some systems person can figure out what
other skills are needed to make the idea work.

There is some discussion of limitations on the methods, and some warnings about
potential abuse, but not enough. Ayres seems to confuse correlation with causation.
He also frequently assumes the sample is representative of the population.
Even when trying to make the sample representative, it often is not. He also
assumes the answer is in the data. Sometimes it is not. Ayres reports a study
concluding widespread point shaving in college basketball because a distribution
at game end did not match the distribution five minutes earlier when a highly
favored team was ahead by about the spread. I have no opinion about the conclusion,
but the simpler explanation of the coach thinking it was late enough to safely
let the weaker players participate more was not considered.

Regression is a powerful tool, but it is easy to misuse. For an ongoing
survey of misuses, see junkfoodscience dot com, a blog. Many of the entries show
the flaws in statistical claims of medical trials. Also try stats dot org.

The answer is of course: a lot.
And Ian Ayres' book will tell you a little about it.

Supercrunchers are those who use lage datasets
to find patterns in human behaviour, and
predict the future based on these large datasets.

The book informs us that super crunching is on the verge of being
used all over. E.g.
Chess grandmaster Kasparov was no match
for IBMs Deep Blue chess computer,
that stored some 700.000 grandmaster chess games to help find the
winning move.
The IRS could use its data to tell a small business,
if it is spending too much or too little on advertising.
Indeed, the IRS probably has enough data to
make good estimates on whether business, marriages, etc. etc.
will fail - based only on comparison with its existing dataset.

For the paranoid, it is a horror that supermarkets could map your life cycle and predict your next purchases pretty accurately (based on
what other similar customers did).
For the optimist data mining is a good thing and we'll all lead better lives because of it.

Want to write a bestseller about it? Compare your title and some key words with data from a database of books, titlescore.com, containing millions of bestsellers and flops, and you will get your answer.

It all seems pretty straight forward, and the book has some nice examples of what we can expect in the coming years.

-Simon

This is new? The notion that empirical research is useful has been dealt with in book after book. The book not only recycles stories word for word without quote marks from the New York Times and other publications. There are hundreds of books that show that empirical work can help understand the world. What is new? What is interesting that is new here?

It comes on the heals of some really great non-fiction analytical books. Unfortunately, this book is all anecdotal and lacks real substance. It is good for non-mathematical, non-analytical people, but not good for people with solid educations in math, statistics, and data analysis.

Ayres demonstrates how statistical analysis of large datasets is affecting the way the world works in a broad range of applications: credit card companies, sports teams, wine critics, development economists, medical practitioners,* law enforcement agencies, schools, etc. "Freakonomics didn't talk much about the extent to which quantitative analysis is impacting real-world decisions. In contrast, this book is about just that - the impact of number crunching" (p13).

As an economist, some of the work is familiar (for example, the research Ayres and Steve Levitt did on the value of the vehicle-recovery device LoJack or the Poverty Action Lab), but Ayres gives a good introduction for the uninitiated. And he covers such a broad range of applications that I learned a great deal.

Like other research surveys (Freakonomics, The Tipping Point, Blink, Stumbling on Happiness), I view these books mostly as surveys of interesting research. Each has a central thesis (Ayres' is that traditional intuition and expertise will be - or already has been - replaced by computing power and will have to learn to complement that power rather than compete with it) which may or may not be convincing, but the books tend to be good rides because so much of the surveyed research is interesting. (For example, I'll be studying more about Direct Instruction - a scripted way of teaching reading that may be useful in my own work - based on this book; and the model Ayres expounds of how private firms learn from iterative experimental trials may apply well to some of the agencies I engage.)

As far as Ayres' thesis goes, I find him relatively convincing (computers with lots of data do predict many things better than people**) but despite his many caveats, the tone should probably have been more humble. He doesn't - for example - explore the issues brought by Taleb in The Black Swan: The Impact of the Highly Improbable, how traditional statistics may be worse than useless in financial markets where a single, completely unpredictable bad shock can wipe out years of carefully predicted investments.

This book was lots of fun to listen to, not least (unintentionally) because Ayres loves giving irrelevant but amusing descriptions of his researchers. The examples below are all economists:

"Ashenfelter is a tall man with a bushy mane of white hair and a booming, friendly voice... No milquetoast he" (p2).

"Even now, in his forties, Larry [Katz] still looks more like a wiry teenage than a chaired Harvard professor (which he actually is)" (p65).

"Esther [Duflo] has endless energy. A wiry mountain climber..." (p73).

And of course you know this is the Freakonomics family because of the Levitt-love scattered here and there: "There is a new breed of innovative Super Crunchers - people like Steve Levitt - who toggle between their intuitions and number crunching to see farther than either intuitivists or gearheads ever could before" (p17).

I listened the unabridged audiobook narrated by Michael Kramer (not Michael Kremer - quoted in this book on p74), published by Books on Tape (6 CDs). Kramer does a good job except when he tries an Australian or British accent.

* For an excellently written description of evidence-based medicine and more, read Atul Gawande's Better: A Surgeon's Notes on Performance.

** One of the most striking findings comes from the meta-analysis (1996) of two psychologists, Meehl & Grove, who look at 136 studies comparing human judgment to equation-based judgment. In only 8 of the 136 studies was expert prediction found to be appreciably more accurate than statistical prediction." Overall, experts got the predictions right 66% of the time whereas Super Crunchers got them right 73% of the time. And the 8 in which experts did better weren't concentrated in any particular field. From looking at the paper myself, I found that 64 of the studies favored the Super Crunchers whereas 64 found the two methods roughly equal. Noteworthy. [In the book, p111 and p232.]

Super crunchers is a nice little book for those who are statistically challenged, but for those who know about joint probability distributions and copula methods, you miss the "gee whiz!" factor of "can they really figure that out?"

Still, it is a fun read in how large data sets are becoming a norm, and like all residue of science can be used for good or evil depending on the moral skew of those who hold the diagnosis. For the paranoid, it is a horror that Wal-mart can map your life cycle and predict our next purchases pretty accurately. But the major theme of this book is that data mining is a good thing and we'll all lead better lives and corporations will engage in less waste as we refine techniques that are targeted and discard techniques that are blankets (and hence create noise).

The first half of the book is better than the second half, so I suspect this was once really a long Atlantic Monthly article stretched for a mini-book. Still, it is a thinking man's plane ride of a paperback.

A brief glance on areas where data driven decision can be made. It is yet bounderyliness to apply but the art to drive such an initiative is still missing.

Okay, got it, people are crunching large datasets to find answers to questions they didn't necessarily know to ask. After that, the book only provides a few interesting ideas buried in a sea of overly simplified generic stories. I was rather disappointed in finding the target audience was the general public and in finding no real depth in any particular topic. If you have any advanced studies in mathematics, definitely recommend against this book as you'll find yourself frustrated for the majority of the read. Oh well, at least another notch on my bookshelf.

Super Crunchers: Why Thinking-by-Numbers Is the New Way to Be Smart
If you have ever taken a probability and/or statistics class, you won't find much new here.
If you are naïve or paranoid don't read this book. (As if you didn't know the government and business are busy collecting every bit of data about you.)
The main theme is the power of modern computers to crunch vast quantities of information and sometimes produce counter intuitive results. There are too many repetitive examples of data collection and number crunching. All in all, an easy read, but fairly shallow book.

If you're looking for another "Freakanomics," you won't find it here.

SuperCrunchers doesn't deal with details or with the unexpected ways numbers can be used but rather sticks to broad abstract statements about a trend (hint: the use of statistics is on the rise).

Most of the book can be summed up in one sentence: The government and business are increasingly using statistics to predict behavior and increase sales.

Most people, I'm assuming, are already aware of this and won't be too surprised.

Most of the information you've probably already read about in the newspapers.

This book is riding on the popularity of "Freakanomics" and it seems to have been written in a hurry in order to publish while economics is "hot." It's more of chronicling than a well-thought out idea.

Here's some numbers advice: save your money.

Supercrunchers is a very fun introduction to data mining and serious numerical analysis. It is light reading, and is filled with examples of current uses and results obtained throughout the world by churning large amounts of data. The book has examples from all industries, government and NGOs. It includes information on uses of data analysis for HR which is usually skimmed over. It fills you with ideas of further uses of data analysis and has info that is interesting for everyone. I couldn't stop talking about it for a week after I read it

By Ayres' definition, I guess I am a professional Super Cruncher. I really didn't enjoy this book. While some of the anecdotes were interesting, he doesn't go into enough depth with each story and after a while the book just feels like a grab bag of anecdotes. Sure, I would have appreciated a more accurate and less sensational use of technical terms, but I would have walked away from this book with a much better feeling if he fleshed out more context with each anecdote. As a counterpoint, take "Money Ball" by Michael Lewis - this book is a single Super Crunching story and is a fantastic read.

It is a shame that such a recent book fails to discuss any of the serious debate of the issues that arise from data mining techniques in hypothesis formation and testing. For a fairly lay description of these issue in the medical field, track down the paper "Why Most Published Research Findings Are False" by John Ioannidis.

All in all, this text succeeds as a quick run through of various applications of data mining. But as a book, it isn't satisfying.

The author has a two Bachelors (Russian Studies and Economics) from Yale, PhD (Economics) from MIT, JD from Yale. He is a bonifide genius and he sprinkles this book with much of his intellectual horsepower.

He provides multiple historical and empirical examples of how humans are very bad at making decisions. Human are even worse in determining the quality of their decisions (most people tends to exaggerate the correctness of their answers). Statistics, on the other hand, are mathetically sound way of predicting future results. Better yet, it provides a highly reliable way of determining the quality of the prediction. Generally speaking, the better the data, the more predictive statistics can become.

Recently, however, there has been a great convergence of statistics and technology. More specifically, informational technology which allows number crunching of multiple terabytes of data in a relatively short period of time has become available at a practical price.

The author believes whoever can take advantage of this relatively new phenomenon has a great competitive and informational edge. Having reliable data analysis tools that use statistical regression analysis enables an entitity to make better decisions (and also know the probability of making a bad decision).

This has become a game changing convergence. Indeed, it is changing the world and will become even more important in the future.

What should you do to be better prepared?

Learn statistics and get the best data mining/analysis tools and employees your company can buy. As the author states, we are seeing only the beginning of this mostly positive trend.

Ayres, a law professor and econometrician, eschews the subjective for the objective. As he writes, "we are in a historic moment of horse vs. locomotive competition, where intuitive and experiential expertise is losing out time and time again to number crunching."

Most of the book covers dozens of examples where randomized testing provided statistically improved results in fields a diverse as charitable fundraising, internet dating, baseball drafting, and medical diagnosis. In Ayres view of the world, the role of the expert is to inform the process of statistical testing, model building and analysis. Once the original analysis is completed, decision-making becomes an automated process that can be performed by anyone with access to the model or decision tree.

Super Crunchers never teaches us how to "think-by-the numbers." While it provides many valid examples of data mining, it seldom provides the reader with the tools or the background to do any sort of meaningful analysis on his or her own. Even such basic concepts such as probability distribution and standard deviation are saved for the final chapter of the book. And this is perhaps slightly disappointing, given that Ayres provides so many delightful examples. As such, Super Crunchers is perhaps more inspirational than instructive.

This book is a typical example of a simple concept usually found in "Introduction" chapter on statistics book, and blown into a book of its own.

If you squeeze all the water out of Super Crunchers, it will turn into just that: a good one-page answer to "What Statistics is Good for?" question. I would recommend this book to any 19 year old thinking of taking statistics class in college, although he or she could safely stop after about 40 pages.

There is no substance, no story, no conclusion and no "main point" in it. The book reads as glorified hype of something old and trivial. The author never mentions cases when statistics gives incorrect answers due to incomplete data or unknown confounding variables, yet any statistician will tell you it happens all the time. He tells stories with doctors getting it wrong, but doesn't mention hundreds of flawed statistical studies where "super crunchers" got it wrong, except for one case where "the other guy" appears to be his colleague-opponent, and that part reads like a pathetic gossip intended for a limited audience.

This book reminds me of "Long Tail", a very similar bloated "book" about trivial and obvious matter. Every once in a while a man just wants to write a book about something, wants it real bad, but falls short of picking a good "something". Not every subject is worth a book, Ian. Sometimes a short article or a blog post will do better.

"There are three kinds of lies," said Benjamin Disraeli, "lies, damned lies and statistics." But, like it or not, the world is becoming more quantitative every day and no one can afford to be statistically innumerate. If you live in Excel and use quantitative techniques daily, this may come as no surprise. What may be surprising, even to data-heads, is the extent to which statistical methods are illuminating areas of human life hitherto relegated to "experts." Call it the new age of empiricism or the rise of numerical "super crunchers," but, whatever the name, the trend is real. In this book, Yale law professor and econometrician Ian Ayres provides an unbiased sample of entertaining anecdotes showing how quantitative thinkers are taking over and why the trend is unlikely to abate. The caveat: as the world and its feedback loops get increasingly complex, is regression less useful? If so, Ayers is a bit optimistic. Yet, getAbstract finds that his book, as well as being entertaining and vigorously written, offers a painless review of important statistical ideas that even Disraeli would've found hard to challenge.

The book was written at a more hands-on level than Competing on Analytics. Also, unlike Freakonomics which showed many random relationships, this book deals with issues that business people will see everyday. I wrote down several ideas on ways to improve my own company based on concepts mentioned in this book.

I thought this was one of the best books I've read this past year. I found it to be well written, entertaining and insightful. The author's main point is that people often put way too much emphasis on their intuition and ability to predict outcomes when computer models based on historical data analysis are often much more accurate.

One of the main targets of the book is the health care industry in the U.S. and how doctors especially place too much emphasis on their own analytical skills and not enough emphasis on data. I suspect at least some of the negative reviews here are from people in the health care industry. There are good reasons why health care in the U.S. is the most expensive in the world, yet according the World Health Care organization in terms of quality of health care, it ranks at number 37, between Costa Rica and Slovenia. An industry mindset that lacks a history of taking advantage of number crunching and thinking by the numbers may well be part of the problem.

I am self employed and since reading this book I've been more numbers oriented and have been more careful to track my hours and keep very detailed logs of what activities make the most money. It has worked out well, so for me reading this book was a great investment and well worth the price of the book.

Super Crunchers is a new book about data mining by Ian Ayres. Super crunching, according to Ayres is the action of applying data mining algorithms to real situations in order to make better decisions from data. I will make it clear right now: Super Crunchers will not give you examples of complex data mining techniques in real situations. Most of the book shows the use of randomized experiments (there are also a few pages on neural network, but that's all).

This book is nevertheless a very interesting reading for many reasons. First, the author did a very good job in introducing the basic ideas behind data mining for non-specialist readers. In addition, Ayres has collected a bunch of small, and very interesting, stories about people crunching data (wine quality prediction, baseball, etc.). In every situations, the author shows how crunching numbers help people make decisions but also how difficult it is to make non-expert believe in your results. This is, to my opinion, the most interesting aspect of the book.

Super Crunchers is very well written. After giving some examples, Ayres describes the actors of this industry (super crunching). He then introduces the idea of randomized experiments. There is also a nice chapter about the confrontation "Experts Versus Equations". He concludes by explaining why this enthusiasm for super crunching is happening only now and not before.

Finally, although the action of super crunching is certainly more about applying statistic methods (rather than data mining) to real situations, this is a must-have book for anybody interested in predictions.

Ayres' offers example after example of real world number crunching that are challenging conventional wisdom. From the predicting the taste of wine for a given year, to projecting future box office success of potential scripts, data crunching is changing the way people think. The amount of information available is making these kind of regressions possible for the first time. Ayres' book has really changed the way I think about problem solving and coming up with new solutions. The book stays away from politics, while Ayres points out Super Crunchers all over the country that are making changes in education, insurance, and medicine. I highly recommend this book for people who love statistics and intuition based economics.

I'm now inclined to be scared after reading this. Data mining, data "mashups", pervasive digital surveillance...that's what this book is about. I'm no Luddite. I love all my electronic goodies and the benefits of digitization. For example, I like Amazon's ability trecommend fairly good product matches for me based on my reading and similarities to other customers. I like scanning technology to speed me through check out lines. But after reading this book and seeing the potential impacts to our privacy and to our lives, I'm feeling more pessimistic about the unfettered direction the technology is taking us. Marketers, like Amazon and the businesses using scanners, are mining our lives for all they are worth. More than you are aware of.

Decision makers like physicians are very soon to be replaceable with artificial intelligence. Good or bad? Good if you are the patient because of better decision making. Bad if you are the physician. You can be replaced by techs in the near future. Movie producers...your go/no go "greenlight" decisions can better be made by a data mining analysis approach. Authors? Better watch out...the publisher may soon make decisions about your books based on their digital rating as potential best sellers. Almost any job requiring expert decision making is subject to increasing data analysis that is better than the experts can deliver. Scary that even creative professions are vulnerable. Customers probably will benefit from the application of the "super crunching" approach, but something may be lost (such as professional decision making jobs?). And this trend seems to be exponential as data storage becomes commoditized and processor speeds increase. Anything that can be stored as data can be "mashed up" and mined for mathematical predictive relationships. Since most of us are unaware of the acceleration of this approach, we really can't see the implications until they are on us. Where goeth free will if everything is digitized, predicted, and manipulated?

This book is not for mathematicians. It is for the lay reader. Marketers and business people who aren't aware of these trends will be excited by their potential for use in their own industries. The writing is clear but unexceptional. But the book is incredibly thought provoking if you haven't been aware of the trends. The author, who is a data miner himself, is enthusiastic about the "New Way to Be Smart" and the potentials for vastly improved decision making that can enhance our lives, but we non-crunchers need to be aware of potential freedoms unwittingly given away.

I liked this book, I really did. I liked the fact that the author provided examples all of us could understand in a language all of us could understand. Where I thought the book could be better is that I felt the author assumed all data is the same. In this age of web based surveys and telephone surveys, where you never know who is filling out screens, we come to rely on this information. I know in many areas getting good data is hard, and having 150 CFOs respond, or 125 purchasing managers is a big effort. The problem is that too many times it is not the CFO responding, but some subordinate. And so many times the survey design is flawed that what you end up with is a big data set of garbage - or what we call GIGO- garbage in, garbage out. The author did not address this at all - considering data quality a given. I think it was a mistake and would have enhanced the book.

I too was intrigued by this book because of its review in the Economist. Do not waste your money! If one is looking for a well told tale of the validity of data mining read "Moneyball" by Michael Lewis.

While I am definitely a layman with respect to the world of statistics, I would at least expect to see a simple example of how a regression analysis might be calculated (page 2 does not qualify). After multiple stories of the validity of "Super Crunching" one still has not the faintest idea how such data analyses are performed mathematically - yea sure, when performed and plotted they follow a Gaussian curve and 95% of the information generally falls within two standard deviations of the mean - great. However, as the author notes, "If you're hooked, the end-notes contain suggested readings for the future." Yet the endnotes are merely the footnotes to the text, not any actual list of what could be appropriate introductions of statistical analysis which can only be textbooks! As interesting as it may be to hear stories (of which "Moneyball" is vastly superior) one is never going to be more than a raconteur without opening an actual arid text and learning how to apply the contents. If you are looking for stories fine, if you are looking for more than topical information look elsewhere.

A very well written & stimulating book into a new field of research opening up as a result of the enormous increase in our ability to analyse data being generated by the computer. This will enable us to out perform the experts by replacing assumptions with hard factual evidence. The example from the Wine Industry is a classic! I have recommended this book to university students as showing the way forward to a new career possibility.

This book is a good start to the power of statistics in business. I have taken two statistics courses and found the book to be a bit tedious. The first half of the book has some interesting examples but they are far too simplistic for anyone with statistics knowledge. If you're looking for an enjoyable read regardless of your statistics knowledge I would go with Freakanomics. Also, I think Ayres makes some false statements in this book and overlooks the dangers of statistics when they are misused. Overall it was a decent book though.

Because they sit in their ivory towers glibly pontificating down to us why we need to take bad tasting medicine for the good of "the economy" I'm not a big fan of economists. Ian Ayres is one of these numerate knowers. That said, he's written a pretty good book that should be on any informed citizen's reading list.

Super Crunchers surveys the many ways statistically literate number crunchers are using huge databases to make better decisions than traditional intuitive 'gurus'. Right from the start Ayres uses the example of how smart baseball front offices use statistics to select players that fit a team model while down playing talent scouts. Talent scouts' limited observation of a player simply doesn't hold a candle to the track record of a players' numbers. An even more compelling example is the New England Patriots which have been turned into a powerhouse by economics trained coach Bill Belichick who makes trades based partly on the numbers.

Super crunching, the application of statistical analysis, especially regression analysis, to very large datasets, has exploded in the last few decades thanks to the availability of these databases to increasing numbers and types of 'crunchers'. Access to data been accelerated by the internet and the personal computer revolution. Ayres uses his plain and lively style to explain and exemplify some surprising places where regression analysis has had success. Imagine predicting what movie script will make money. How about which reading teaching method is the most effective? Can we improve government using randomized trials? These are just a few case studies touched on by Ayres.

Throughout the book Ayres points out the inevitable resistance by entrenched interests that are threatened by number crunching algorithms. Physicians resist powerful computer assisted diagnosis databases in favor of their intuition. I'm reminded of Gerome Groopman's excellent book HOW DOCTORS THINK, where Groopman acknowledges that physicians tend to latch onto an initial diagnosis. He then mostly dismisses computerized diagnosis. Ironically, the diagnosic tool Isabel does exactly what Groopman recommends a patient should do namely asks what else could it be and offers alternatives.

Teachers object to being forced to teach reading from a strict script despite studies that show it helps very young children learn to read faster and better. Ayres does a nice job using the standard deviation in explaining ex-Harvard president Larry Summers reasoning for the disparity between men and women in math intensive fields. Ayres also touches on how artificial intelligence, namely neural networks is emerging as an alternative to regression analysis.

Super crunching is hardly all good news. Consumers are being super crunched for the benefit of business. Credit card issuers and banks increasingly use supercrunching to rig the unsecured debt game against borrowers. Casinos use supercrunching to determine the pain point of the best "customers" and use this to determine when to cut the gambler off with a nice distracting reward such as dinner and a show.

Where I think Ayres falls back into the glib academic character is when he dismisses teacher creativity in favor of a mind numbing by the script Direct Instruction. The DI method works for the three R's, according to the studies he cites and is seen as 'teacher proof'. However there are other subjects in teaching that educate the whole child and which benefit from teacher creativity; e.g. science, art, music, social studies, etc.

Ayres doesn't completely eschew intuition. The hypothesis needed to formulate a regression analysis is often initiated and driven by intuition. The smart, future decision makers will learn to meld these two ways of thinking according to the author.

I took stats at my MBA course last semester (curiously, with a professor that appears in the book) and months later read SuperCrunchers over winter break. Too bad, because the book really manages to get you excited about the power of statistics for improving decision-making, and I would have seen stats with different eyes if I had read it six months ago in the first place. It's an interesting book, tries to be the new Freakonomics in style, but is not as much fun.

Like "Freakonomics," this book over-relies on a catchy phrase as a substitute for a thorough exploration of the concepts and issues. The list of concerns includes:
1. Vague definition of the term "supercrunching." Is it "super" because the author wants us to think all statistics are super, or (what I had hoped) is there something about the type of statistics to which he refers that are in fact different from statistics in decision making for the last 40 years? All the talk of large datasets implies that supercrunching is a matter of size, but then why does the very first example of regression involve a model that has only 2 predictors? No need for large data sets for this kind of a model, right? Unless the effect size is tiny, but then, what good is the model? Tell us how things really are new and different now.
2. The book reads like a list of (mostly internet) companies and how fabulous and smart they are for using statistics. Actuarial science has been around for many, many years and again we see little discussion of how the actuarial tradition has become more available outside of the insurance industry. The whole book seems more like a stream of consciousness than an organized conceptual framework about the emergence of statistics as a guide to commercial, medical, and policy making over time.
3. While perhaps an excellent lawyer and professor, the author makes so many misleading or inaccurate remarks about statistics that it could be difficult for someone with a statistics background to enjoy the book. For example, regression is discussed as a technique that is different from the "randomized test," when in fact the randomized test is a design, and the regression (more commonly the "general linear model," including regression, analysis of variance, linear and structural modeling) is the inferential statistical technique used to evaluate the results of the test design. Early in the book, the author talks about how amazing regression is, and then gives and example of how a bank evaluates probability of future actions on the phone based on past behaviors on the phone. This very first example after introducing regression does not involve regression as a prediction technique, but rather actuarial base rates! In fact, I found it very disappointing that actuarial science, probability, and Bayes' theorem (all at least as relevant to data-driven decision-making as the randomized trial) were given so little attention in the book.
4. Finally, the great irony--and part of the "this book is so bad I have to finish it" quality--is that the author writes in an incredibly intuitive manner. The book is full of cognitively biased representation of the topic, owing mainly to "availability" heuristics, for example, the authors' excessive attention to the work of his friends, his roommates, his enemies, his daughter, or the companies he shops from. Better scholarship (or at least more rational) would have involved an initial sampling of all the relevant examples and final selection of the ones that would best illustrate the concepts (which I never really understood--see points 1 and 2). As other reviewers have pointed out, there is also "confirmatory bias" all over the place (presenting only the facts that fit with one's idea)--why aren't the counter arguments and counter-evidence better presented? The author must know that people buying a book on statistics will actually be smart enough to weigh the different sides of an issue. Like I said, I read to the end just to see if there was a "punch line" where the author confesses about his unapologetically intuitive approach to writing--that the book was a joke on the reader.
I would recommend this only to people who know very little about statistics and are unaware how companies like amazon.com use statistics to improve business. Such readers will be impressed. For everyone else...there are so many better books out there. Paul Meehl would be super-disappointed in this work.

Ayres does an excellent job of easily conveying many examples of how data and statistics are changing our lives.
Super Crunchers is an easy read for non-numbers folks because he's not referring to statistics and formulas.
For statistics-wonks, Super Crunchers is still valuable to put yourself in the mindset of seeing many varied areas where your expertise could be useful.
*The one caveat about this book is it's political (yes, Ayres is able to fit in a swipe at Cheney and James Dobson). Many of the examples cited in the book are policy-related, from a liberal mindset. And then he will turn around and use super crunching against opponents of liberal viewpoints by saying they use data the wrong way.
But, Ayers maybe is using these points in order to convince liberals that statistics and the use of data are the way of the future. To Ayers credit he does acknowledge that "the stories in this book refute the idea that Super Crunching is part of some flattening right-wing conspiracy" (so clearly this is what his liberal colleagues must complain to him about).
And when thinking about the use of data and statistics, the whole book is essentially a 'market' friendly book.

Some overall good points, but utterly lacking in context. Yes, large data sets do steer us in the right direction, but, as someone with a background in mathematics and statistics will point out, a model is just that--a model.
Statistics is one of those relatively arcane pursuits where, in the hands of a master, it can educate, inspire and inform. However, in the wrong hands, or worse, in the hands of someone with just enough knowledge to be dangerous, it can lead to all sorts of problems. In fact, there are many 'trap-doors' in statistical analysis (for example, heteroscedasticity) which are not even touched upon in this book.

Data mining and analysis is the new science behind customer service, pricing strategies, and just about everything else in our lives, or at least that's the message that Ian Ayres tries to convey in his book. Belonging to the popular science genre, the examples are illustrative and abundant, albeit also over-simplified in my cases. "Supercrunchers" is an accessible book if you want an introduction into what the `actuaries' have been working on for the past several decades, but please treat it as such - an introduction - as there is a lot of overlooked details that have been swept under the rug in the process of writing this book.

I really was disappointed with this book. I ordered it expecting a lot of information on data mining, but it felt more like a long infomercial on all the different companies that use data mining. I didn't really get anything useful from this book and wish I'd never ordered it.

It's like the author just created a long list of companies that use data mining and describes each one, but doesn't really give you any insights on what they learned from doing it or how it changes much. There isn't really any evidence here. You just have to take his word for it that it's all a good thing.

I was disappointed and do not recommend this.

I find this book an excellent commute read. I work in the market research industry and this material come as a morning treat. It is easy to understand (assuming some knowledge of statistics and data-analysis tools) and offers some great insights on consumer behavior.

Ayres' argument that regression analysis can be used to inform opinions is definitely not new, but even worse he has finally admitted to not only recylcing examples from national newspapers but to using the exact words of those reporters (Google the quotes below to get the web links).

1) "The [Yale Daily] News found nine passages in the book similar to or the same as sentences from articles printed in the New York Times, the Los Angeles Times, the Wall Street Journal, the San Diego Union-Tribune and Fast Company magazine. . . . . Ayres said. 'I apologize for these errors . . . .'" Yale Daily News, October 4, 2007

2) David Leonhardt of the New York Times wrote the following: "[Ayres] reproduction of these sources can be quite troubling. I realized this when I came across two sentences about a doctor in Atlanta that were nearly identical to two sentences I wrote in this newspaper last year. The sources show up in the footnotes, but many readers will surely assume that Ayres witnessed some events, like the scene at a call center from an article in Fast Company magazine, when he in fact did not. They will also assume the words are his. "A phone call that might have taken 20 or 30 seconds, or even a minute, now lasts just 10 seconds. Everyone wins," Charles Fishman wrote in Fast Company, describing the center's data-centric customer service. Ayres reproduces the exact words, without quotation marks."

There is very little I could say bad about this book. I though work with databases, so this likely piqued my interested more than many. Still, the author uses an easy-to-understand prose while describing intriguing facts about how now (and in the future) our lives, decisions, and the way the world turns is being changed by the art of data-mining. I rarely find a book worthy of more than 3 stars, but felt this one deserved 5 as I thoroughly enjoyed this work and would recommend Super Crunchers to anyone interested in how data is and will continue to change our lives (and not always for the better).

Super-crunchers explores the way sophisticated decision-makers are running the world. We can either pretend nothing has changed, or strive to understand, as Ian has, the direction the world is going.

Decision makers now collect information, find the important factors, and try improvements. It sounds simple, but requires a paradigm shift to add a slew of new computerized and database-based tools to our intuition.

The book surveys dozens of business decisions, policy decisions, and sports issues, and shows how we can understand to forces at work better. I think the ultimate importance of the changes are great. In the cases for which I know the details, the book is very much on the money. I work at data-mining for science, and found the technical parts all old hat, but the many case studies fascinating.

Ian has a remarkable track record of uncovering facts we should have known, and need to fix, with efficient real-life experiments, and again here his savvy yet lucid exposition reveals the goings-on necessary to see the state of the union.

I am stunned by the negative reviews here -- I can't help but wonder if they are from "competitors" of some sort. This book was outstanding. Far, far better than the trivia of Freakonomics. Packed with great, insightful, apple-falling-on-your-head type info. Interesting, entertaining, educational.

Numbers win every time.

I teach research to graduate students. This is a text I'd love to use to show my students how working with numbers is becoming essential in having options in the their work lives. Ayres' book is an excellent synthesis of the trend to utilize empirical data to backup arguments. Well written, with great stories and accessible examples.

I enjoyed reading this book very much. It was refreshing to find an economist/lawyer who can communicate the idea and usefulness of regression/neural analysis without putting down the reader, and for making it initeresting and convincing.
This book, though, has a very definite target group of consumers in mind. Given that, I think most of the negative comments fall into three camps:
1. it is too simplistic or nothing new: yes, the book explains regression analysis using simple easy to digest examples to drive the point. Regression is not new, but making it understandable is almost unheard of.
2. ethic/moral objections: examples on direct instruction and eharmony tends to antogonize readers who have strong attachment to politically correct positions on teacher's role or race in society. this objection misses the point, regression allows you to tease out race effect (holding other things constant) and the use of evidence or results to drive actions. if you disagree with that, this book will offend you.
3. question about his competency on technical details and statistics: this book purposely omit them to reach wider audience. the reader cna rest assured that he is not math illiterate, the author has a phd in economics from MIT; that degree is probably worth a phd in math and economics. don't take my word for it, check out his website or his scholarly work.
Overall, this is a very well written and accessible book on the use of regression analysis. I teach econometrics to college students and the most difficult part of teaching this course is not deriving the properties of estimators or proving one technique works better than another but providing intuition behind mind numbing mathematical manipulations and formulas. I will be using this book as supplement to my econometrics textbook.

I recently read two books on data crunching or analytics: "Competing on Analytics" by Thomas Davenport and "Super Crunchers" by Ian Ayres. I started reading these books with a bit of skepticism. This is most probably because it was after I read Nassim Nicolas Taleb's "Fooled by Randomness" and "The Black Swan", and Roger Lowenstein's "When Genius Failed", among others, in which I was so intrigued by the author's view of the world that it is full of asymmetries and the highly improbable as well as the spectacular fall/implosion of the quantitative investment strategies taken by LTCM in 1998. Coupled with these are my own experiences in my professional career to date as well as an amateur quasi-quant trader, in both of which I come across many "black swans"/luck/the improbable whatever you call it, including both good and bad.

Professors Davenport and Ayres both addresses basically that there are a number of areas where data crunching can produce lots of insights as well as fairly accurate forecasts against which human brains usually cannot match, and the both books are chock full of real-world examples, with "Competing on Analytics" centering mostly on business with focus on how to build a company with competitive advantage capitalizing on analytics capabilities, while the "Super Crunchers" encompassing broader range examples in legal and governmental areas with focus on statistical methodologies (i.e. randomizations and regressions) used in crunching data. Upon reading these books, I personally found the Super Crunching more interesting due to the width of areas covered and the methodologies explained, but was pleasantly surprised by the fact mentioned in the two books that there are so many real life situations, where data crunching plays a pivotal role in making business decisions and driving institutional policies, and my skepticism about the validity of data crunching based on statistical methodologies are largely gone....but not entirely.

When data is available for crunching and that data has more or less a normal tail (i.e. not too many outliers), relying on the results of crunching should provide reasonably good forecasts/predictions and would be of great use. However, I still tend to believe that investment is a different animal to which super crunching may or may not work effectively. This notion comes from my observations that quant hedge funds collapse here and there when the market acts capriciously or atrociously, as well as from my experience in investment where I have been using extensive statistical modeling but have had quasi-implosions here and there...... The correlation suddenly gets close to almost 1.0 when the market volatility roars and. And what's significant here is that it is not just counts/frequency of outliers that is relevant but it is more the amount involved that should be kept in mind, as the impact is frequency times the amount of money involved. Imagine the consequences when a high degree of leverage is involved. Surprise! ...and you are out. One can say that it does not mean the super crunching does not work in investment modeling but indicates that the modeling is deficient or not robust enough to deal with the unexpected market volatility, and/or coupled with inadequate risk management such as too aggressive leverage. Well, that may well be the case.... Just wondering if there are any quant-funds that have sustained, say, for 15 to 20 years through market volatilities and continue to produce stellar returns....?

Freakonomics was successful because Steven Leavitt is a truly original thinker and Stephen Dubner is a terrific writer. They took a great a great article from the New York Times Magazine and expanded it into a successful book. Here, Ayres tries to follow in the same vein but unfortunately lightning doesn't strike twice.

One stat question for the supercrunchers out there: When Ayres asked his daughter how many times she had hiked a particular trail she answered with "six" and a standard deviation of two, then revised her mean to "eight." He was understandably proud of here quant chops and expounded on that interchange for several pages. But wasn't this use of standard deviations and mean off the mark? If he had asked her how many times she hiked the trail each year on average, it would have fit. But he asked her how many times she had hiked the path in her life, so there was no distribution to be considered. Or am I missing something?

I ask only because it seems like a fundamental error to make in a book about statistical analysis.

Is it a new brand of cereal? Or maybe it's a granola bar, or a chunky peanut butter spread? Then again, could it be the latest infomercial exercise device designed to give you the six pack abs you've always dreamed of but know in your heart of hearts you'll never achieve? Actually, it's a book - the title a product of the very methods the book describes. Here's what SUPER CRUNCHERS says.

(1) Mathematical regression models generated from large datasets often generate better predictions than human experts, and they provide supporting information on the predictive weight and reliability of each explanatory variable.
(2) Well-crafted experiments using randomized trials and control groups provide good market research and behavioral analysis results.
(3) Technological advances - the Internet, massive data storage devices, rapid computation, broadband telecommunication - are making it possible to share more sources of information and create ever-larger databases for analysis.
(4) Today's companies engage in multiple forms of market research by creating and using large databases and large-scale randomized trials.
(5) Many phenomena conform to normal distributions in which 95% of the population will be found within two standard deviations of the mean, the5% balance generally divided evenly in the two tails.

That's it. I just saved you $25.00 U.S. and a half-dozen or more hours learning how a guy from Yale named Ian Ayres collected a bit of information about applied mathematical techniques that have been in practical use for decades, packaged them up, palmed them off as something new, and cooked up the ridiculous name Super Crunching to describe an ostensibly new technological development. Yet "Super Crunching" is nothing more than the author's marketing hype for a couple of standard mathematical methodologies, a creation of nothing from something. There's no new breakthrough here, no new paradigm.

Yes, the anecdotal information about the future prices of wine vintages, Capital One's teaser offerings, and evidence-based medical diagnosis are interesting (hence the two stars rating). The rest, however, is neither prescriptive nor sufficiently critically analytical. Should we go out shopping for a Super Cruncher tomorrow? Should we delight in the increased accuracy of data-driven modeling and prediction, or should we fear the implied manipulation of our desires and the incessant, single-minded drive toward maximum profit at the expense of creativity? Do we really want movies and books to be developed from mathematical models like Epagogix? Do we really want our every keystroke on the Internet to be fodder for market research that manipulates us in response? John Kenneth Galbraith, among others, warned of exogenous, manufactured demand decades ago.

SUPER CRUNCHERS is part business tome, part econometric paean, and part sociology book, but not fully any of the three. No matter how many time the author uses words like "cool" and "humongous" and "amazing," it's still regrettably a "No Sale" even for someone like me who enjoys reading about applied mathematics.

Fundamentally sound exploration of the numbers game,and how it affects our daily lives.
Somewhat technical,but easy reading and informative.

Please ignore all of the reviewers that have a political axe to grind. This book is primarily about the rise of statistical models (primarily predictive models such as regression) to help organizations make sounder, more accurate decisions.

The book is at its best when it discusses the applications of super-crunching algorithms, and how they routinely beat the opinions and decisions of experts and of people who make choices based on experience and instinct. Ayres does a very good job discussing the implications of these models, both good and bad, and how they are funamentally changing the worlds of business, medicine and government.

I tried not to read the book with too critical of an eye for the technical aspects of the models...if you're looking for a better discussion of regression models and how to build them, best to consult a statistics book. And I do take particular issue with the author's lack of understanding of sampling variation and what consitutes a statistically significant result. At the very least he shouldn't make such bold statements about polling data and uniformed reporters when he makes a fundamental mistake about interpreting the margin of error. For the most part, the math is spot on, especially when it comes to the instances of False Positives in medical testing.

Focus on the book for what it is - a exploration of the rise and impact of statistics, mathematics and science into areas where decisions had previously not been data-driven, and what impacts that will have on our daily lives and the organizations around us. On these issues, Ayres' book excels.

I started reading Super Crunchers last night. I got an Advance Reading Copy from somewhere. I only read the first few chapters and it is okay. I think some of the examples he uses makes him sound like he is exaggerating the usefulness of data. In other places he is more accurate about their utility.

One part of the book got me shaking my head and I don't want to read the rest. He accuses Eharmony of discrimination based on race: "Even though it's only acting on the wishes of its clients, matching services that discriminate by race may violate a statute dating back to the Civil Warthat prohibits race discrimination in contracting."

Only a professor at an Ivy School can say something so asinine. But he doesn't stop there.

He goes on to accuse Eharmony of discriminating against gays! He says Eharmony uses a computer algorithm that matches people based on similarity. Then he says, "When it comes to gender, they assume opposites attract." Then he goes on to say how the founder is a self-described "Passionate Christian."

You can see the author didn't say, "Black dating sites discriminate against whites", "Gay dating sites discriminate against straight people" or "Jewish dating sites discriminate against Christians and Muslims". He says sites run by "passionate Christians" discriminate against blacks and gays.

I only now wonder if this author tells his kids they are "homophobic" if they don't date gay people or racist if they don't date out of their race.

If I wanted to be manipulated and hear silly logic like this author's, I would read the NY Times editorials--I don't need it from a data-mining book too.

For those of you that want a good book on data mining, this book is only an okay introduction at best. But if you like it when authors go on tangents and try to manipulate you with their social beliefs, this book is recommended so far.