The Effect of Test Set Selection on Classification Accuracy

I was looking at some prediction results for the UCI Michalski and Chilausky soybean data set and wondered how they depended on test set selection. Some had classification accuracy as high as 93.1% accuracy on a 25% training set and  97.1% on 290 training and 340 test instances.

A few weeks ago I had been asked to find the best classifier for the soybean data set based on prediction accuracy on a test set of 20% of the data. The remaining 80% could be used for training. That gave 306!/(245!x61!) = 1.3 x 10^65 possible splits of the 306 data points into training and test sets. Could some of these splits lead to better results than others for the classifiers I was about to use?

The WEKA data mining package was used for classification. WEKA has many classifiers that can be run on a data set and their performance to be compared.

WEKA also has a programming interface so I used it to write some Jython toolsto explore the performance of a range of classifiers.

One of these tools was run on the soybean data to find the training/test splits with best and worst classification accuracy. The results were

Classifier	Best Accuracy	Worst Accuracy
Naive Bayes	100%	70.5%
Bayes Net	100%	75.4%
J48 (C4.5)	95%	69%
JRip (RIPPER)	98.4%	70.5%
KStar	96.7%	65.6%
Random Forest	95%	62.3%
SMO (support vector machine)	96.7%	82%
MLP (neural network)	100%	77%

Fig1. Best and worst accuracies for selected WEKA classifiers run on different training/test splits

That was quite a range of test set accuracies for different training/test splits. My simple genetic algorithm may not have found the extremes of the distributions so the actual range may have been higher.

When I ran the test set selection scripta second time (Fig 2) it also found a 100% SMO accuracy. The second test was set up to find a single training/test set split that gave best results for all classifiers at once. It also had a slightly different pre-processing. The 4 duplicate instances were removed and the troublesome single 2-4-5-t sample was left in. Therefore I expected it to give worse results than the pre-processing used for the results in Fig 1.

Classifier	Correct (out of 60)	Percent Correct
Naive Bayes	57	95 %
Bayes Net	59	98.3 %
J48	58	96.7 %
JRip	60	100 %
KStar	60	100 %
Random Forest	59	98.3 %
SMO	60	100 %
MLP	60	100 %

Fig2. Best accuracies for selected WEKA classifiers all run on the same training/test split

Both the above results were for the default settings of each of the WEKA classifiers. The WEKA classifiers all have parameters that can be tuned and it is possible to select subsets of attributes so they can give better and much worse results than the defaults. However the default parameters are usually close to the best so they may be good indicators of the best possible accuracies.

It appears that the training/test split of a data set can change classification accuracy by more than 30%. This was observed on a well-known and widely used classification data set.

Watching Percipo

What you should do if you are offered the Nobel Prize for Economics

If Nicolas Taleb is right, you should refuse to accept it because it was proven to be worthless when Robert Merton won it in 1997 for inventing Black-Scholes-Merton . Taleb says this often.  Here is a quote from him that you could use for your Nobel Prize refusal speech.

Then, in 1997, the Royal Swedish Academy of Sciences awarded the prize to Robert Merton and Myron Scholes for their option pricing formula. I (and many traders) find the prize offensive: many, such as the mathematician and trader Ed Thorp, used a more realistic approach to the formula years before. What Mr Merton and Mr Scholes did was to make it compatible with financial economic theory, by “re-deriving” it assuming “dynamic hedging ”, a method of continuous adjustment of portfolios by buying and selling securities in response to price variations.

Dynamic hedging assumes no jumps – it fails miserably in all markets and did so catastrophically in 1987 (failures textbooks do not like to mention).

Later, Robert Engle received the prize for “Arch ”, a complicated method of prediction of volatility that does not predict better than simple rules – it was “successful” academically, even though it underperformed simple volatility forecasts that my colleagues and I used to make a living.

The environment in financial economics is reminiscent of medieval medicine, which refused to incorporate the observations and experiences of the plebeian barbers and surgeons. Medicine used to kill more patients than it saved – just as financial economics endangers the system by creating, not reducing, risk. But how did financial economics take on the appearance of a science? Not by experiments (perhaps the only true scientist who got the prize was Daniel Kahneman , who happens to be a psychologist, not an economist). It did so by drowning us in mathematics with abstract “theorems”. Prof Merton’s book Continuous Time Finance contains 339 mentions of the word “theorem” (or equivalent). An average physics book of the same length has 25 such mentions. Yet while economic models, it has been shown, work hardly better than random guesses or the intuition of cab drivers, physics can predict a wide range of phenomena with a tenth decimal precision.

Every time I have questioned these methods I have been abruptly countered with: “they have the Nobel”, which I have found impossible to argue with. There are even practitioner associations such as the International Association of Financial Engineers partaking of the cover-up and promoting this pseudo-science among financial institutions. The knowledge and risk awareness we are accumulating from the current subprime crisis and its aftermath will most certainly not make it to business schools. The previous dozen crises and experiences did not do so. It will be dying with us, unless we discredit that absurd Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel commonly called the “Nobel Prize”.

The Ed Thorp he mentions is the same Ed Thorp who figured out how to win at blackjack in casinos by counting cards. 

Thorp was one of the first seven inductees chosen to the Blackjack Hall of Fame.

The mathematician who could model risk effectively got into the Blackjack Hall of Fame while the duffers who could not won the Nobel Prize for Economics. 

Most of you readers are not in the Blackjack Hall of Fame but at least you have not won the Noble Prize for Economics, and that has to count for something.

This paper that explains how options trading works in practice is fascinating.

As Albert Einstein said

"As far as the laws of mathematics refer to reality, they are not certain; and as far as they are certain, they do not refer to reality."

Risk Management for Software Projects

Large software projects are risky because

1. They involve people.
2. They are can be complex. 
3. They depend on people and companies outside your direct control, such as OS makers and 3rd party library vendors.
4. There is little flexibility to make big changes at end of big software projects. This was described elegantly in the Mythical Man Month.

Risk Management Strategies

1. Understand your project's final deliverable very well, including your customer's needs and quality requirements. 
2. Manage for an appropriate level of risk, not zero risk, which is unachievable in any case. Learn which risks give the best returns and which lead to the worst problems. 


1. Take the costs of managing risks into account; don't spend more time measuring, avoiding and mitigating risk than you get back from doing so. 
2. The appropriate level of risk depends on the benefit of your project succeeding, the cost of it failing and the probability of it failing.
3. Base schedule estimates on reasonable expectation of things going wrong from time to time. An honest schedule allows you to eliminate non-essential features at the start of a project and improve its chance of success. It also wins you the respect of seasoned developers, sets a tone of realism, and provides the foundation for a successful project.


1. If in doubt then pad your estimated schedule based on the accuracy of your previous schedule estimates. e.g. If your last few projects took 1.5 as long as you expected them to take then multiply your current schedule estimate by 1.5. This is a way of admitting the limitations of your ability to estimate schedules. As you get better at estimating schedules you will be able to pad less. Meanwhile it is critical that you understand and compensate for your limitations. Padding is necessary.
   
2. Take a minimax  approach and minimize your project's reasonable worst case schedule. The reasonable worst case depends on your project's circumstances but it will be something like the 90th percentile, 95th percentile or other high percentile in the distribution of possible schedules.
4. When your project gets started, keep track of your real schedule. Be good at discovering reality. 


1. Work with today's real schedule and expectations. This is similar to the popular Agile  methodology. Even if your company does not use Agile methodology, once development starts you need to deal with the current reality of your project. If the reality differs from the plan then you need to deal with the reality. If you can regain your original schedule that's great but you must deal with the reality while the original plan differs from the current reality.
2. Devise good metrics and use them. Use the metrics to compute a real schedule. E.g. #open bugs may be an accurate metric that reflects the true state of the project, but it probably doesn't predict ship date accurately since bug fixes (like any code change) introduce bugs. Code churn will almost certainly be a better predictor of ship date, but the code churn is meaningful only as long as the bug fixes are being driven to the final (shipping) bug fix number . 
3. Remember that projects need to converge. Doing extra work at the end of the project invariably leads to slippage. 
4. Don't shortchange any critical upstream development activities (Steve McConnell  #20). If something is going to have to be done then do it at the most efficient time. Design is much more effective if it is done before coding. Testing is more effective if code is written with testing in mind. Etc. All incomplete critical activities are risks. 
5. Minimize the number of risks in a project. 


1. Try to avoid introducing multiple new technologies in a separate project.
2. If you cannot one or more major risks (e.g. new technology, new supplier, new market) in a project then avoid non-essential risks. e.g. If you have to introduce a new technology for the project but you can delay the new supplier until the next project then do so. If your company is large enough to support multiple simultaneous projects then spread the risks between projects.
3. People are part of the risk. If you have to take a risk then don't add to the risk with  inexpert or untried developers or developers who are subject to external pressures. Likewise this is a bad time to use newly formed teams, teams whose members don't back each other up or teams going through major issues.
4. Infrastructure and organization are part of the risk. If you have to take a risk then support the risk-taking team with your organization's best infrastructure, including IT, HR, facilities, etc. Shield the risk-taking team from distractions such as re-organizations, moving to new premises, learning a new email system, heavy personnel review processes, non-critical training etc.
6. Break down major risks. Break down complex tasks into several smaller tasks so that one failure won't bring the whole large task down. 


1. In a series of tasks with measurable completion milestones, failures to achieve the milestones will become apparent early while there is still time to recover.
   
2. Breaking a big task down into task a set of concurrent tasks can either increase risk by adding coordination risks or decrease risk if some of the sub-tasks can fail without causing the whole task to fail. If the coordination risk can be minimized and sub-task failures are tolerable then this is a good strategy.
7. Move risky items to the start of the project. If something goes wrong with the risky items this gives you time to address the problem while code and designs can be changed without high risk. Performing this step rigorously will distinguish well risk-managed projects from other projects. 
8. Keep a top N (say N=10) risks list. Risks are everywhere and eternal vigilance is required. However not all people have this mindset. A top N risks list is an easy-to-grasp way of communicating risks to a group.
9. Follow good software development and management practices. Avoid the classic mistakes because they can be easily avoided by reading the list (which I wish I had read before I made most of them). Many standard software engineering principles minimize risk, so don't unlearn them when you work on your first commercial product. In particular don't stop using good development practices when your project is under pressure.
   
10. Expect the unexpected.


1. Never ever forget Murphy's Law. Expect to make mistakes. 
2. Effective risk management requires sensitivity to risk. If you are not emotionally wired in this way then you will need to learn how to think this way .
3. Keep in mind that while the past at best provides an imprecise guide to the future (see the item on schedule padding), at worst it provides no indication of the future at all. This is exemplified by Taleb's Turkey: Imagine that you're a turkey. You've eaten well and lived in safety every day of your life. Everything in your experience tells you that tomorrow will be no different. Then Thanksgiving arrives.
4. As advised above, adapt to the currently reality, even it differs from your plan.

Summary of Risk-Based Schedule Prediction

1. Knowns. Plan these rigorously.
2. Known unknowns. Pad schedule for these. e.g. predicted schedule + 2 std devs
3. Unknown unknowns. Requires eternal vigilence and adapting to the current reality.

What Makes Risk Sensitive Managers Different

Risk management is not just part of software development project management like design, scheduling and presentation skills. A truly risk-sensitive approach to project management requires explicitly managing by risk. This means that if your scheduling and prioritization is not based on realistic risk estimates then your are not managing risk well. For example

1. In a project involving 100 engineers, a group of 3 engineers are working on a completely new piece of code that implements difficult algorithms for a critical deliverable and every other engineer is making incremental changes to the new code, then a risk sensitive manager would focus on this group of 3 engineers.
2. If a project introduced a new software technology, brought in a new hardware supplier and entered a new untested market then a risk sensitive manager would try to break it into 3 projects, each with only one risk.
3. MOST IMPORTANT EXAMPLE. A risk manager assumes that a) many things could go wrong will go wrong and b) there is limited scope to correct mistakes at the end of a project. Therefore a risk sensitive manager will trim features at the start of a project to give the project a reasonable chance of success. 

Risk sensitive managers manage like this even if doing so is at odds with other management techniques.

1. In example 1, the risk sensitive manager would be focusing on the risky group of 3, even if it meant PRDs coming in late and the low-risk 97 engineers getting less than optimal help. The risk sensitive manager would work hard to move work from the group of 3 to other people in the 100, and negotiate with other clients of the group of 3 to lighten their load.
   
2. In example 2, the risk sensitive manager would lobby high and low through his/her company to avoid the dangerous confluence of risks.
   
3. Example 3 illustrates what separates risk sensitive managers from average software managers, the absence of wishful thinking in decision making. In #13 in the previous link Steve McConnell says Wishful thinking isn't just optimism. It's closing your eyes and hoping something works when you have no reasonable basis for thinking it will. Wishful thinking at the beginning of a project leads to big blowups at the end of a project. It undermines meaningful planning and may be at the root of more software problems than all other causes combined.

And good risk managers always attack uncertainty.

Further Reading

Risk Management bared down to one Question
Why MS Project Sucks for Software Development

Premium No Name Brands

Try googling "premium no name brand" . The list of websites is small and probably includes this one.

With good analytics such as pattern recognition, near infrared spectrometry, liquid chromatography, bluetooth and other technologies and good supply chain management it should be possible to retailers to guarantee the quality levels associated with top brands without paying brand premiums to the brand owners.

Bottled water is the opposite of this idea.

Journalism in 2008

If you believe what you read in the news media then take a look at this Marginal Revolution post on the recent stories about a study on gender variations in mathematical ability.

For those of you who don't remember, Larry Summers had to leave Harvard because of what he said on this subject, essentially that the male and female distributions had the same mean but the male distribution had a higher standard deviation. Summers said this was a major factor in the high ratio of men to women in jobs requiring extremely high mathematical ability such as mathematics professors in elite colleges.

Summers' chain of reasoning was hard to argue with. If ability in a field of endeavor is approximately normally distributed and two groups have the same mean ability then the group with higher variation in that ability will be tend to be occupy more of the jobs that require extremely high levels of that ability. Therefore either Summers' conclusion was correct or the data he based it on was wrong. It appears that many journalists did not agree with Summers' conclusion so they were eager to report on the new study that showed that his data were questionable. The only problem was that the new study's data confirmed that variarition in mathematical ability is higher in males than females. From the Marginal Revolution post

• If you do the same type of calculation as the authors but now look at the expected gender ratio at 4 standard deviations from the mean you find a ratio of more than 3:1, i.e. just over 75 men for every 25 women should be expected at say a top-25 math or physics department on the basis of math ability alone .

No-one who posts on this blog claims to be able to predict how the tail-ends of data sets that are expected to be normally distributed will behave in reality. However higher variation within a population usually leads to more people at the top and more at the bottom.

The difference between predicting the future and fitting a model to past behavior ...

... is that the former is extremely difficult and the latter is almost trivially easy. However some people will be surprised by this Freakonomics post on Jim Collins' "Good To Great" . For those of you who don't remember, Good to Great is based on an analysis of some companies who were once just okay then went to outperforming the market over a long period. It compares these companies to other companies working in the same fields and distills what they did differently to some rules, the Good to Great Principles (more on this below). The Freakonomics blog post looks at the performance of the Good to Great companies:

• ...It looks like Fannie Mae is going to need to be bailed out by the federal government. If you had bought Fannie Mae stock around the time Good to Great was published, you would have lost over 80 percent of your initial investment... Another one of the “good to great” companies is Circuit City. You would have lost your shirt investing in Circuit City as well, which is also down 80 percent or more. Best Buy has cleaned Circuit City’s clock for the last seven or eight years....Nine of the eleven companies remain more or less intact. Of these, Nucor is the only one that has dramatically outperformed the stock market since the book came out. Abbott Labs and Wells Fargo have done okay. Overall, a portfolio of the “good to great” companies looks like it would have underperformed the S&P 500.

How did this happen? Leonard Mlodinow describes how random variation is routinely mistaken for the results of people's decisions in his book "The Drunkard's Walk: How Randomness Rules Our Lives".

Of course the fact that the management techniques recommended in Good to Great are based on faulty analysis doesn't make those techniques any less (or more) effective. However it will be interesting to see the reaction of those people who promoted those management techniques because they raised the stock prices of the Good to Great companies relative to the market. Will those fans now champion doing the opposite of what the book recommends? A quick look at the Good to Great recommendations shows they are not harmful enough to make companies underperform the market as badly as the Good to Great companies have done.

2.1 Chapter 1: Good is the Enemy of Great
2.2 Chapter 2: Level 5 Leadership
2.3 Chapter 3: First Who, Then What
2.4 Chapter 4: Confront the Brutal Facts (Yet Never Lose Faith)
2.5 Chapter 5: The Hedgehog Concept (Simplicity Within the Three Circles)
2.6 Chapter 6: A Culture of Discipline
2.7 Chapter 7: Technology Accelerators
2.8 Chapter 8: The Flywheel and the Doom Loop
2.9 Chapter 9: From Good to Great to Built to Last .

This article gives a plausible explanation of how Good to Great works its magic. Here is an excerpt:

• First of all, the good-to-great principles are true in the same way a horoscope is true. They are fairly generic and thus we all apply them from our own viewpoint to make them true. I believe that some "Good to Great" readers that love the book may be suffering from the Barnum effect. The principles Collins proposes aren't bad ones, but they are ambiguous and open to interpretation, which in effect decreases their usefulness. For instance, Collins says good-to-great companies practice "First Who, Then What," which basically means "hire good people." I'm willing to bet no one read the book and said "Eureka! I've been hiring slimy weasels when I should have been hiring top performers. That is why we aren't a great company."

New Facebook Apps Architecture. New Opportunities to Monetize Social Networking?

Tech Crunch posts on facebook's new three tier app system and developers' need for certainty about the app platform.

I put foward some ideas for facebook apps in this post but I never got around to finishing my "How Long Are You Going to Stay Married" facebook app. This company beat me (and people who could actually carry this out) to it on using images for matching people but maybe there is a way to predict the length of marriage as a function of appearance, or people may want to know how likely their potential partners are to end up in jail .

You may be asking "Why predict what people will need? Why not just let them choose?". The reason is that in some cases it is difficult to do so. To para-phrase the Rolling Stones, You don't always know what you want but sometimes you just might find you can deduce what you need. Wikipedia's list of cognitive biases (and the related cognitive bias) enumerates some of the barriers to us understanding the world around us. The relatively straightforward objective measurements and statistical analyses proposed above and in the previous post can help people overcome these biases. Don't let Dunning Kruger effect stop you from exploring this!

Was Anyone Surprised By This?

This recent Freakonomics post shows that forecasters have a done terrible job at predicting the price of oil.

That should not be surprising.

Why should the price of oil be predictable? It depends on factors that are barely measurable by humans such as future consumer sentiment, future government action, future weather and future geo-politics.

Nicolas Taleb has described this in great detail over the last few years. He calls the mis-application of mathematical models to data which they cannot hope to model the Ludic fallacy .

In some cases people make a jump from analyses of recent history to prediction of the future. Businees books such as Good To Great come to mind.

Ronald Fisher who was good at prediction said this about pundits.

In scientific subjects, the natural remedy for dogmatism has been found in research. By temperament and training, the research worker is the antithesis of the pundit. What he is actively and constantly aware of is his ignorance, not his knowledge; the insufficiency of his concepts, of the terms and phrases in which he tries to excogitate his problems: not their final and exhaustive sufficiency. He is, therefore, usually only a good teacher for the few who wish to use their mind as a workshop, rather than a warehouse.- R.A. Fisher. Eugenics, academic and practical. Eugenics Review, 27, 95-100, 1935.

For those who care, here is an attempt at explaning of oil price trends http://europe.theoildrum.com/node/4007

After 30 years of drinking cheap wine and not noticing any problems with it...

... it turns out I was more or less right and could have proven so with some simple experiments. The ancedote is from the Freakonomics blog and the research is here.

And just when I was about to stop adding balsamic vinegar to my beer the Freakanomics follow-up post shows that all I really needed to do was to have someone add it for me and not tell me.

The positive correlation between price and perceived quality is familiar to most of us. When I was growing up on a farm in Australia in the 1960s my great-uncle Fred who owned an audio equipment store in Melbourne told me that if he had items that were not selling he would raise the price to entice buyers.

Why does Freakonomics focus exclusively on the taste of wine? Why not the pleasure derived from the altered state of conciousness or the severity of the after-effects? My remembered perception of wine drinking is that the best wines are the ones that taste best when drunk in large volumes.

In a future post we will subject water to the above analysis.

Monetizing Social Networking

This innovative use of social networking could have a short-term payoff. It turns out to be one of the few successes to date in making money from social networking. When I ran a straw poll on making money from social networking with some software engineers, wannabe entrepeneurs and friends, they all came up with essentially the same idea: Mining the users' personal details and finding some way to make them pay to keep this information confidential. This is outside the ethical bounds of this blog so I am forced to look elsewhere for this post.

Here are some ideas. Many social networks are forums for dating or job searching so an obvious angle is to provide tools that help users in these pursuits. The drawbacks to this approach are

• it is obvious so there will be many competitors
• people don't like to stalked by unwelcome suiters or harassed by recruiters

Possible solutions are to

1. Match people. Find compatible dating partners or employee/employer matches. This suffers from being obvious but it may be difficult enough to deter competitors
2. Help people be found by the type of people they want to find them. This is a variation on 1.
3. Lower people's resistance to being matched.

Anatomy of a Relationship
Relationships with spouses, friends, employers and employees are equilibrium states. Relationships are likely to persist because of

1. Emotional attachment to current partner *
2. Financial cost of breaking relationship *
3. Irrational fear of change
4. Rational risk of change. This includes limited ability to evaluate the new partner
5. Difficulty of finding a new partner
6. Cost of learning the skills needed for the new partnership
7. Inablity to be productive while learning
8. (Current) lack of suitable partners
9. Inability to find a good match
10. Cost of relocation

Some people are not in relationships. The main groups are single people searching for partners and unemployed people searching for jobs. The * items do not apply to these groups
Helping people find partners = finding a partner for whom these barriers are low
Matching partners = is finding pair for whom these barriers are mutually low. It is not clear what mutually low means yet.
Lowering resistance to change = lowering a subset of these barriers.

Bad Strategies

1. Trying to appear what one is not. Copying other people's witty profiles, faking work history and qualifications are strategies that suffer from being easy to replicate, focusing on the means (getting a date/interview) rather the end (getting a sustainable relationship/job) and high risk/reward ratio.
2. Dating/job search advice. Many companies already provide these services. This can also suffer from focusing on the means rather than the end.
3. Competitive analysis. Competition is not the main driving force in the creation of relationships though we tend to focus on it. Monogamy, trust and reliability dominate the relationship sphere. Any winning strategy must embrace these forces.

Remaining Strategies

1. Apply actuarial prediction to people's profiles and social graphs to reveal things about them that they do not know themselves. Use these predictions to provide the matching and introductions services that are currently based on raw profile data. The tools listed on Ian Ayers' web site give an indication of what can be predicted. E.g. Probability of divorce on a given year, income 10 years from now, date of death.
2. Use the stategies from 1 for predictive health care maintenance. Monitor user data to predict disease and injury.
   

REFERENCES
Social Networking Usage
http://blog.compete.com/2007/09/14/facebook-activity-breakdown-application/
http://valleywag.com/tech/advertising/facebook-consistently-the-worst-performing-site-242234.php

Information Mining - The Social Graph

My Way Business The Social Graph: Issues and Strategies in 2008 is an enlightening overview of the social graph.

http://apps.facebook.com/socialistics/ attempts to map social graphs in facebook. When I ran it on myself http://www.facebook.com/profile.php?id=504949372&hiq=peter%2Cwilliams it did not reveal any insights I was not currently aware of. Whether this was due to the weakness analytics in the app or the lack of information in facebook links or my own lack of ability at using the app was not clear.