I’ve been working on a Python package to analyze adhesively bonded joints recently. This package will be used to analyze adhesive joints in certain aircraft structure and will be used to substantiate the design of structural repairs, amongst other uses. Because of this, the output of this package needs to be validated against test data. This validation also needs to be documented in an engineering report.

I’ve been thinking about how to do this. On one hand, I’ve been thinking about the types of (mechanical) tests that we’ll need to run to validate the model and the various test configurations that we’ll need to include in the validation test matrix. On the other hand, I’ve also been thinking about change change management of the package and ensuring that validation report stays up to date.

I’m imagining the scenario where we run the validation testing and find that the model and the test results agree within, say, 10%. Maybe that’s good enough for the purpose (depending on the direction of the disagreement). We can then write our validation report and type out the sentence “the test data and the model were found to agree within 10%.” Then, I’m imagining that we make a refinement to the model formulation and release a new version of the package that now agrees with the test data within 5%. Now, we have a validation report for the old version of the package, but no report describing the validation of the new version. We’d need to go back through the validation report, re-run the model for all the validation cases and update the report.

When we update the validation report manually, there’s probably a pretty good chance that some aspect of the update gets missed. Maybe it’s as simple as a one of the model outputs doesn’t get updated in the revised validation report. It’s also potentially rather time consuming to update this report. It would be faster to make this validation report a Jupyter Notebook (which I’ve previously talked about). I haven’t yet written about it here, but it is possible to have a Jupyter Notebook render to a PDF using a corporate report format, so it’s even possible to make this validation report look like it should (Edit: I’ve now written about this here). We could also set up a test in the package to re-run the Jupyter Notebook, and perhaps integrate it into a continuous integration system so that the Notebook gets re-run every time a change is made to the package. This would mean that the validation report is always up to date.

When you write a Jupyter Notebook, it usually has some code that produces a result — either a numeric result, or a graph — and then you have some text that you’ve written which explains the result. The problem is that this text that you’ve written doesn’t doesn’t respond to changes in the result. Sure, there are ways of automatically updating individual numbers inside the text that you’ve written, but sometimes the way that the result of the code changes warrants a change in the sentiment of the text. Maybe the text needs to change from “the model shows poor agreement with experimental results and shouldn’t be used in this case” to “the model shows excellent agreement with experimental results and has been validated.” There’s no practical way that this type of update to the text could be automated. But if the update to the result of the code in the Notebook has been automated, there’s a good chance that the text and the results from the code will end up disagreeing — especially if the report is more than a few pages.

The Solution

So, what can be done to rectify this? We want to have the ease of having the results of the code automatically update, but we want to make sure that those results and the text of the report match. One approach to this problem — and the approach that I intend to use for the adhesive joint analysis package — is to add assert statements to the Notebook. This way, if the assertion fails, the Notebook won’t automatically rebuild and our attention will be drawn to the issue.

As an example, if the text says that the model is conservative, meaning that the strain predicted by the model is higher than the strain measured by strain gauges installed on the test articles from the validation testing, we could write the following assert statement in the Jupyter Notebook:

assert(model_strain > experimental_strain)

Now, if we later make a change to the model that causes it to under-predict strain, we’ll be alerted to this and prompted to update the validation report.

Implementing the Solution

To run a Jupyter Notebook from code (for example in a test suite), I’ve use the following code in the past. This code was based on code found on The Data Incubator Blog

def _notebook_run(self, path):
    kernel_name = "python{}".format(sys.version_info[0])
    file_dir = os.path.dirname(__file__)
    errors = []

    with open(path) as f:
        nb = nbformat.read(f, as_version=4)
        nb.metadata.get("kernelspec", {})["name"] = kernel_name
        ep = ExecutePreprocessor(kernel_name=kernel_name, timeout=3600)

        try:
            ep.preprocess(nb, {"metadata": {"path": file_dir}})

        except CellExecutionError as e:
            if "SKIP" in e.traceback:
                errors.append(str(e.traceback))
            else:
                raise e

        return nb, errors


_notebook_run("file-name-of-my-notebook.ipynb")

This code will run the Notebook file-name-of-my-notebook.ipynb and will raise an error if an error is encountered. If this is inside a unittest2 or NoseTest test suite, this will cause a test failure.

Conclusion

Validating software used in a way that affects an aircraft design is very important in ensuring the safety of that design. Keeping the validation report up to date can be tedious, but can be automated using Jupyter Notebooks. The conclusions drawn in the validation report need to match the results of the software being validated. One approach to ensuring that this is always true is to add assert statements to the Jupyter Notebook that forms the validation report.