DoE (Design of Experiments)

Design of experiments is the term used to describe how a series of experiments or tests are to be arranged to be able to determine which are the variables that have the most effect on the result, i.e. the output or characteristic in which we are interested.

The term goes back to the British statistician Ronald Fisher who published a book of that name in 1935. In that book, amongst other topics, he described full factorial experiments and the null hypothesis.

Types of Experiment Design

One Factor at a Time

As a young automotive engineer in the 1970’s and early 80’s, this was the type of experiment. If one wanted to determine which aspects of a vehicle made the biggest difference to the lap time around a circuit, then one selected different possible variables to change and changed them one at a time. Therefore if one had 10 different variables, each at 2 potential settings or levels e.g. front tyre pressures, rear tyre pressures, front spring rate, rear spring rate, front damper rate, rear damper rate, front ant-roll bar rate, rear anti roll bar rate, front wing angle, rear wing angle, one would do a baseline run and then change each one in turn, and then possibly do a repeat baseline run at the end to see if anything else had changed during that time that affected the lap time, and one would perhaps adjust the results accordingly.

This is a 10 factor experiment with 2 levels, and so it would take 11 experiments (baseline with everything at level 1, plus 10 runs, each with a different variable at level 2, while all the others were at level 1), not counting any repeat runs. Lets say it takes 10 minutes to do each test (out lap, timed lap, in lap and then time to change the variable). Thus this set of tests would take 110 minutes, or nearly 2 hours.

The advantage is that this is the quickest set of experiments to run and then analyse, because it combines the minimum number of runs, with the quickest analysis, because no fancy statistical analysis is required, you just look at the raw results. And in the motor racing world test time and practice time on the track is limited, so the number of tests must be kept to a minimum or they won’t get done in time.

A disadvantages are that external factors e.g. temperature, wind direction, tyre wear, surface grip etc. may mess with the results and there is no easy way of assessing this other than keeping an eye on those extraneous (noise) factors, and maybe repeating some of the tests if there is time. However if you are looking for which variable had the biggest effect, you may be OK, as big effects usually shine through. Small effects may get swamped by noise factors.

Another disadvantage is that this experimental design does not include testing any combinations of variables, which is fine if they are all independent effects, but may not be so fine if they are not. For instance in the example above fitting stiffer springs without changing the damping, probably means that the vehicle is now underdamped, and it may be a good idea to stiffen the springs and the dampers at the same time.

In this type of situation, in my experience the basic one at a time type of test is just done to establish broad sensitivity to each variable, and then a further set or sets of experiments are done to fine tune things, and knowledge of the system and engineering judgement starts to come into play, and 2 or more factors may be changed at the same time if it seems sensible to do so. This is perhaps the difference between developing or tuning a system in order to achieve a practical result, and pure scientific experimentation to prove or disprove a theory in an academic study.

Full Factorial Experiments

In a full factorial experiment then every possible combination is tested, so that all the interactions between variables are assessed. The formula for calculating how many tests N are required for F factors each at L levels is,

The number of tests = the number of levels raised to the power of the number of factors (variables), i.e. N = L^F

So in our example above the number of tests required to complete a full factorial experiment would be

N = 2^10 = 1024 tests. So this would take 10,240 minutes, i.e. 7 days, each of 24 hours, or 3 weeks if one assumes an 8 hour day, if one only had one prototype test vehicle, or one would need 21 prototypes to get it all done in a day, but that would bring in the extra variables of different vehicles and different drivers.

This is why in nearly 40 years of being an engineer I have never ever done a full factorial experiment. They take too much time and hence cost too much money, and halfway through something else in the design would have changed anyway, invalidating the whole caboodle.

Fractional Factorial Experiments (Orthogonal Arrays – Taguchi Method)

In the early 1950s Genichi Taguchi visited the Indian Statistical Institute where work was being done on orthogonal arrays. From that he designed a new more efficient kind of experimental design based on a fractional factorial approach that could incorporate some interactions if desired and used specific orthogonal arrays and ANOVA (analysis of variance ) techniques to analyse the results, thus potentially combining the best of both worlds in a practical approach – the speed of one at a time experiments with the interaction effects of a factorial and statistical analysis approach.

Only certain size arrays are amenable to this treatment. In our example above which has 10 factors, each at 2 levels, the nearest Taguchi orthogonal array would be an L12, which handles 11 2-level variables in a total of 12 experiments. Thus in our example we would have to and another run (another variable (which could be a ‘dummy’ variable)), or a particular combination (deliberately changing two factors together). Thus the test time would only take 10 minutes longer than a one at a time experiment, but more information could be extracted once the statistical analysis had been done on the results. If you want more interactions and noise factors to be included one could choose a larger array, such as an L16 (16 experiments with 15 2-level factors). It is also possible to look at multi-level factors as a way of examining a non-linear response.

Comment

Since the late 1980’s Taguchi’s methods have gained a lot of ground, initially in the manufacturing departments of the automotive industry where they became a standard tool in the quality control armoury of manufacturing and quality engineers, and from there as part of the Six Sigma toolbox.

There is a lot more to Design of Experiments than this brief and very simplified overview, both in the depth of the techniques described, other techniques not mentioned here, and different industries favouring different approaches.

We do not claim to be world experts on this subject, but are nevertheless happy to discuss the pros & cons of different methods with regards to a particular project if you contact us.