Design of Experiments

Which experimental points such a partial factorial experimental design should ideally consist of is the central question of statistical experimental design. The aim is always to achieve the maximum information content with the minimum experimental effort.

Maximum information content means being able to identify the parameters of a certain mathematical model with the smallest possible uncertainties on the basis of the measurement data. Therefore, an optimal experimental design always belongs to a very specific model. If you do not know much about the observed process in advance and therefore do not have a specific mathematical model in mind, the following two general experimental designs are suitable.

Plackett-Burman experimental designs

Plackett-Burman experimental designs are particularly useful if you want to find out at an early stage of research or development which factors have an influence on the result in the first place. They serve to identify the most important influencing factors and can be used as a starting point for further experimental investigations, in which interactions between the factors are then also taken into account.

Plackett-Burman experimental designs assume that the data can be described with a simple linear model without interactions. Since it is a linear model, the optimal experimental design is universal and does not depend on the parameter values that are unknown in advance. Rather, it depends solely on the number of factors to be investigated. For example, 3 factors result in 4 experimental points. This means that thanks to optimal experimental design, it is possible to determine relatively precisely with only 4 experiments which of 3 factors have an influence on a certain result.

Latin Hypercube Experimental Designs

Another frequently used type of experimental design is the Latin Hypercube experimental design. In contrast to the plans described so far, the factors are not divided into fixed levels, but are distributed completely randomly. The aim is an efficient and even exploration of the experimental space. They are particularly suitable for considering a wider range of factors and interactions.

In contrast to Plackett-Burman experimental designs, Latin Hypercube experimental designs can better capture interactions between factors. Due to the uniform and random arrangement of the factors, complex relationships between the variables can be detected.

Latin Hypercube experimental designs are not limited to a specific number of factors or levels. To create them, you simply specify the number of factors and the desired number of measurement points.

In summary, Latin Hypercube experimental designs allow for a broader exploration of the factor space, including possible interactions, while still using comparatively fewer experiments. This makes them a powerful method for complex experiments where a more comprehensive analysis of factors and their interactions is required.

Create experimental design in Python

Python has become the standard for data analysis. With the Pandas library, there is a very powerful tool for dealing with all kinds of data, including experimental data. And the library Seaborn offers very comfortable possibilities to visualize data in the Pandas format.

Create general experimental designs

There is a very useful package for Python to create standard experimental designs: pyDOE2 a fork of the no longer maintained package pyDOE.

A full factorial experimental design for 3 factors with 3 levels each can be created for example like this:

from pyDOE2 import *

design = fullfact([3, 3, 3])

Result is a 2D array with integer values for the factors in the columns. Each row corresponds to an experiment to be performed (experimental point):

It makes sense to convert this array into a Pandas DataFrame. In this way, names can be defined for the individual factors and a clear pair diagram can be created with Seaborn:

import pandas as pd
import seaborn as sns

column_names = ["Faktor A", "Faktor B","Faktor C"]
df = pd.DataFrame(design, columns=column_names)
pair_plot = sns.pairplot(df)

A Plackett-Burman experimental design for 3 factors can also be created with one line:

design = pbdesign(3)

Or a Latin hypercube experimental design for 3 factors with 10 experimental points:

design = lhs(3, 10)

Experimental designs for specific models

In addition to these general experimental designs, Python can also be used to create experimental designs for special application-specific mathematical models. For this purpose, we at TLK have developed our own Python package with the somewhat uncreative name doe_tool. With it, optimal experimental designs can be determined, graphically displayed and compared even for complex physical models. The models can either be in  FMI format (Functional Mock-up Interface) or as Python functions.

As a minimal example, let's look at the experimental design for fitting a straight line to measurement data. First, the model is defined as a Python function:

def linear_function(p, u):
    return p[0]*u + p[1]

From this, a corresponding object for the doe_tool is created:

import doe_tool

model = doe_tool.PythonFittingModel(linear_function)

The model parameters to be identified are defined:

model.fitting_parameters = [{'name': 'p0', 'value': 1.0},
                            {'name':'p1', 'value': 1.0}]

The dependent variable together with its standard deviation as a measure of measurement uncertainties:

model.dependent_variables = [{'name':  'y', 'std': 0.1}]

and the independent variables or factors:

model.independent_variables = [{'name': 'u', 'min': -1.0, 'max': 1.0}]

We now want to determine an optimal experimental design for this model. First, we create a full factorial experimental design with 10 levels per factor and calculate all internally required data:

design = doe_tool.FullFactorialDesign(model)
design.populate(10)
design.compute_data()

Now we want to determine an optimized experimental design with only 2 experimental points. Using a certain algorithm (DETMAX), the 2 best points are selected from the 10 candidates:

>>> d_opt = doe_tool.optimize(design, 2)
>>> print(d_opt.independent)
     u
0 -1.0
9  1.0

And these are precisely the two extreme points. This is also clearly comprehensible. Because when fitting a straight line, the measurement errors have the least effect on the estimated value for the slope if the distance between two measurement points is as large as possible.

What happens now if we want to afford 4 measurements?

>>> d_opt =doe_tool.optimize(design, 4)
>>> print(d_opt.independent)
     u
9  1.0
0 -1.0
9  1.0
0 -1.0

The statistical experimental design suggests that we should rather repeat the two experiments at the extreme points instead of placing additional measuring points in the middle.

These repetitions are a typical result of the experimental design. Let me remind you again: The experimental design always belongs to a very specific model, in this case a straight line equation. We therefore assume that the data can be described by a straight line. Whether this prerequisite is fulfilled in practice or whether it is better to measure a point in the middle to be on the safe side is left to the user of these mathematical methods.

In doe_tool, much more complex models can also be imported from other modelling environments thanks to the FMI interface. We use it, for example, to create test plans for refrigerant compressors. We use physics-based models from the Modelica libraries of the TIL Suite.

Literature on experimental design

The problem with statistical experimental design is that it relies very heavily on mathematics (statistics). Many sources are too theoretical, mathematical and are therefore difficult for most users (e.g. engineers) to understand. Other sources, on the other hand, oversimplify so that the basic concepts and assumptions are not clear. Both lead to experimental design methods not being understood in practice and therefore not being applied.

An exception to this is the very readable article FisherMatrix for Beginners by D. Wittman. It explains the central element of experimental design with a simple example: the Fisher information matrix. The mathematics necessary for understanding it is described in a comprehensible way without diving into too much theory.

Another highly recommended source is the NIST/SEMATECHe-Handbook of Statistical Methods. This is a very comprehensive source on all statistics topics that concern engineers and other users. Here, too, the balancing act between mathematical depth and comprehensibility is achieved very well. This is achieved through many examples and a focus on graphical/visual evaluations instead of formulas and numbers. There is a separate section on the topic of experimental design.