From DDWiki

The phrase design of experiments (also DOE or experimental design) refers to the use of statistical techniques to create an efficient, systematic set of controlled experiments for collecting data efficiently in order to estimate relationships between independent and dependent variables through measurement. DOE is used in engineering to design physical experiments to determine physical relationships (e.g.: effect of pressure and temperature on yield in a manufacturing process). DOE is also used in conjoint analysis as a market research tool to design efficient surveys, where each survey question is an "experiment" tested on the respondent.

In DOE, the experimenter wishes to determine the impact of several independent variables (called factors or attributes) on one or more dependent variables (output). The experimenter will measure the effect of each independent variable by systematically varying its level. In most DOE frameworks, each factor takes on one of several discrete levels as its value in each experiment (for example, temperature may be tested at {90° and 100°} and pressure may be tested at {10MPa, 20MPa, and 30MPa}). By testing each attribute at different levels and measuring the resulting output, it is possible to measure the effect of changing levels of each factor on the output.

Contents 1 Full Factorial Designs 1.1 Properties of a Full Factorial Design 1.1.1 Balance 1.1.2 Orthogonality 2 Effects 2.1 Main Effects 2.2 Interaction Effects 2.3 Summary of Effects 3 Fractional Factorial Designs 3.1 Example 3.1.1 Design 1 3.1.2 Design 2 3.1.3 Design 3 3.1.4 Comparison 3.2 Justification for Allowing Aliasing with Interaction Effects 3.3 Factors with Multiple Levels 3.4 Constructing Fractional Factorial Designs 4 Uncertainty and Statistical Significance 5 Conjoint: DOE in Survey Design 5.1 References

Full Factorial Designs

A full factorial experimental design contains all possible combinations of attribute levels. The figure at right shows a cube representing all possible combinations for an experiment with three factors (a, b and c) at two levels each (- and +), therefore with 2×2×2 = 2³ = 8 combinations. Generally, the number of combinations in a full factorial experimental design with n factors and m levels each is mⁿ. If each factor has m_i levels, the number of combinations is

The figure and the table below show the naming convention to describe each possible combination (experiment) and its output.

	Attribute a	Attribute b	Attribute c	Dependent variable y	Name
1	-	-	-	y1 = f(a − ,b − ,c − )
2	-	-	+	y2 = f(a − ,b − ,c + )
3	-	+	-	y3 = f(a − ,b + ,c − )
4	-	+	+	y4 = f(a − ,b + ,c + )
5	+	-	-	y5 = f(a + ,b − ,c − )
6	+	-	+	y6 = f(a + ,b − ,c + )
7	+	+	-	y7 = f(a + ,b + ,c − )
8	+	+	+	y8 = f(a + ,b + ,c + )

Properties of a Full Factorial Design

Full factorial designs are balanced and orthogonal, which are desirable properties for obtaining unbiased estimates.

Balance

In a balanced design, all attribute levels appear an equal number of times across the experiments. A full factorial design is balanced. In the example above, each level of each attribute appears in 4 of the 8 experiments.

Orthogonality

In an orthogonal design, all pairs of attribute levels appear together an equal number of times for each pair of attributes. A full factorial design is orthogonal. In the example above, each pair of attribute-level combinations appears in 2 of the 8 experiments.

Effects

The purpose of running DOE experiments is to determine the effect that changing the levels of each independent variable will have on the dependent variable. There are two categories of effects: main effects and interaction effects

Main Effects

A main effect measures the influence on the output of a single factor's change of level averaged over all levels of all other factors. Since the effect of changing the level of a factor may vary depending on the levels of other factors (e.g. the effect of increasing temperature is different depending on the pressure), the main effect measures the average effect over the possible levels of the other factors. For the example above,

In the cube figure, this can be seen as subtracting the average value on the left face of the cube from the average value on the right face of the cube. The main effects of factors b and c are calculated similarly.

Interaction Effects

Interaction effects measure how much the influence of one factor's level change varies from its main effect when the level change occurs at specific levels of other factors. A two-way interaction is the interaction between two factors:

A three-way interaction is between three factors:

Summary of Effects

For the 2³ example described above, the sign of each component in calculating each effect is:

Effect	I	A	B	AB	C	AC	BC	ABC
ME(a)	-	+	-	+	-	+	-	+
ME(b)	-	-	+	+	-	-	+	+
ME(c)	-	-	-	-	+	+	+	+
INT(ab)	+	-	-	+	+	-	-	+
INT(bc)	+	+	-	-	-	-	+	+
INT(ac)	+	-	+	-	-	+	-	+
INT(abc)	-	+	+	-	+	-	-	+

Fractional Factorial Designs

In many practical applications the number of factors and/or the number of levels for each factor creates a full factorial design that is too large for all possible combinations to be tested due to resource constraints. In such a case, an experimenter would like to conduct some fraction of the full factorial of experiments and still gain meaningful information. By carefully selecting which subset of the full factorial to test, it is possible to obtain unbiased estimates for main effects and low-order interactions if the experimenter is willing to assume that high-order interactions are negligible.

Example

To understand the relationship between selection of a fractional factorial design and its implication on the ability to estimate main and interaction effects, take the 2³ example design

Attribute a	Attribute b	Attribute c	Name
-	-	-
-	-	+
-	+	-
-	+	+
+	-	-
+	-	+
+	+	-
+	+	+

and consider if you were to conduct any four of the eight experiments, which would you choose? Consider these alternative choices:

1. Choose I, C, AB, and ABC
2. Choose A, B, AC, and BC
3. Choose A, B, C, and ABC

Design 1

In design 1, we have:

Attribute a	Attribute b	Attribute c	Name
-	-	-
-	-	+
+	+	-
+	+	+

Note that this design is balanced because each level {-,+} of each factor {a,b,c} appears in two out of the four experiments, but the design is not orthogonal because some pairs of attributes such as {a,b} = {-,-} appear in two of the four experiments while other pairs of attributes such as {a,b} = {-,+} appear in none of the four experiments. Note also that a and b are confounded or aliased: Whenever a is -, b is also -, and whenever a is +, b is also +. Identically, we can notice that column a and column b are equal. We note this by writing:

• A = B

We can also notice that other effects are aliased. For example, AC=BC: Whenever {a,c} is {-,-}, {b,c} is also {-,-}, etc. We notice that (column a)(column c) = (column b)(column c). It turns out that we can easily find all of the aliased effects by making use of a property of the notation we have adopted: we "multiply" both sides of any of the aliasing "equations" by the same name but rewrite any squared terms as 1:

• A = B
• (A = B)C -> AC = BC
• (A = B)B -> AB = B² -> AB = I
• (A = B)AC -> A²C = ABC -> C = ABC

Design 2

Likewise, in design #2:

Attribute a	Attribute b	Attribute c	Name
-	+	-
-	+	+
+	-	-
+	-	+

we see that the design is balanced but not orthogonal, and we notice that (column a) = -(column b):

• A = -B
• AB = -I
• ABC = -C
• AC = -BC

Design 3

And for design #3:

Attribute a	Attribute b	Attribute c	Name
-	-	+
-	+	-
+	-	-
+	+	+

we see that the design is balanced and orthogonal, and we notice that (column a) = (column b)(column c), so:

• A = BC
• B = AC
• C = AB
• I = ABC

Comparison

So, which of these three fractional factorial designs is better? The answer depends on what we wish to measure. Designs 1 and 2 cannot estimate the main effect of a independently of b because it has no information to distinguish whether changes in the dependent variable were caused by changes of a's level or by changes to b's level. The effects of a and b are confounded since they always change together in the experiment. Design 3 can estimate each main effect independently of the other main effects; however, each main effect is confounded with a two-way interaction (since they always vary together in the experiment). If it were true that two way interaction effects in this system were negligible compared to main effects, then it would be possible to estimate the main effects of a, b and c with only the four questions of design 3 instead of requiring the eight questions in the original full factorial design.

Justification for Allowing Aliasing with Interaction Effects

Louviere, Hensher and Swait (2000) note that (p94)

• main effects typically account for 70-90% of explained variance
• two-way interactions typically account for 5-15% of explained variance, and
• higher-order interactions account for the remaining explained variance.

In addition, higher order interactions are typically more difficult to interpret meaningfully. While scientific experimentation may tend to call for inclusion of more higher order effects in order to eliminate alternative explanations and develop deeper understanding of phenomena, practical engineering and marketing implementations are typically interested only as much data as is needed to support decision-making, and typically higher-order interactions are ignored.

A main effects design is thus a fractional factorial design capable of estimating main effects without confounding any other main effects. In a main effects design, the main effects are often confounded with two-way and higher-order interaction effects, but these interaction effects are assumed to be negligible.

Factors with Multiple Levels

Full factorial, fractional factorial, and main effects designs can also be created for cases where some or all factors have more than two levels. It is possible to examine simple cases by hand; however, it is cumbersome. It is recommended to use software tools for DOE when working with factors that have more than two levels.

Constructing Fractional Factorial Designs

Standard reference tables are available for common experimental designs in the textbook references listed below. Also, a variety of software exists for generating fractional factorial designs, including Sawtooth Software and Kuhfeld's Market Research Methods in SAS. There is also a tutorial for creating a choice based conjoint design in SAS available on DDWiki.

Uncertainty and Statistical Significance

Conjoint: DOE in Survey Design

Conjoint analysis involves the design of surveys using DOE techniques, where the factors are attributes of the product (price, fuel economy, acceleration, color, etc), and the levels are alternative values for each attribute (price = {$10, $20, $30}). Each "experiment" in the DOE design represents a full product profile with each attribute set at one of its levels (e.g.: price = $10, fuel economy = 35mpg, acceleration = 6sec, color=blue). There are three classes of conjoint surveys:

1. Rating-based conjoint involves asking respondents to rate each product profile (for example, on a scale of 1 to 10). Ratings can be used directly as the dependent variable y when estimating main effects and interactions.
2. Ranking-based conjoint involves asking respondents to sort the alternative product profiles in order of preference. Ranking data can be related to rating data.
3. Choice-based conjoint involves asking respondents to select a preferred product profile out of a set of alternatives. The primary advantage of choice-based conjoint is that respondents are asked to perform a task most similar to the task they perform while shopping (shoppers are rarely asked to "rate" the available products), so the task is less artificial and produces more reliable data. However, the observed data is binary choice: Either the respondent chose product A or they did not choose product A. Choice does not function directly as a dependent variable - instead choice modeling methods are needed to infer an intermediate variable (utility) that operates like the dependent variable.

For more information, see choice modeling, and Louviere, Hensher and Swait (2000) or Train (2003).

References

1. Engineering Statistics Handbook
2. Louviere, Hensher and Swait (2000) Stated Choice Methods: Analysis and Application, Cambridge University Press.
3. Wu, C.F.J. and M. Hamada (2000) Experiments: Planning, Analysis, and Parameter Design Optimization, John Wiley and Sons.
4. Kuhfeld Marketing Research Methods in SAS.
5. Train, K. (2003) Discrete Choice Methods with Simulation, Cambridge University Press