The world’s leading publication for data science, AI, and ML professionals.

Unraveling the Design Pattern of Physics-Informed Neural Networks: Part 02

Enhancing PINN training stability through ensemble learning and dynamic solution interval expansion

Photo by Clay Banks on Unsplash
Photo by Clay Banks on Unsplash

Welcome to the 2nd blog of this series on the Design Patterns of physics-informed neural networks (PINN)🙌 Here, we will look at techniques of ensemble learning and dynamic solution interval expansion to enhance PINN training stability and accuracy.

Looking ahead, we will start with the specific problem the proposed strategy is trying to address, ** followed by detailing the proposed strategy, how it is implemented, and why it might work. Afterward, we will take a look at what physical problems are benchmarked, as well as** the strengths & weaknesses of the proposed approach. Finally, we discuss the alternatives and future opportunities for the proposed approach.

As this series continues to expand, the collection of PINN design patterns grows even richer🙌 Here’s a sneak peek at what awaits you:

PINN design pattern 01: Optimizing the residual point distribution

PINN design pattern 03: training PINN with gradient boosting

PINN design pattern 04: gradient-enhanced PINN learning

PINN design pattern 05: automated hyperparameter tuning

PINN design pattern 06: Causal PINN training

PINN design pattern 07: Active learning with PINN

Let’s get started!


1. Paper at a glance:

  • Title: Improved training of physics-informed Neural Networks with model ensembles
  • Authors: K. Haitsiukevich, A. Ilin
  • Institutes: Aalto University
  • Link: arXiv

2. Design pattern

2.1 Problem

Physics-Informed Neural Networks (PINNs) provide a compelling avenue for solving partial differential equations (PDEs) due to their inherent advantages such as flexibility, elegance, and seamless integration of observed data. Despite these advantages, training PINNs in practice often presents a significant challenge: the networks have the tendency to converge to incorrect solutions due to their complexity and sensitivity to hyperparameters.

This issue has a significant impact on the current state of applications in PINNs, as it limits the accuracy and reliability of the solutions they can produce.

Now the question is: How can we improve the training process to ensure that PINNs converge to the correct solutions?

PINN workflow. In practice, PINNs may converge to wrong solutions during training. One promising way to address this issue is by gradually expanding the solution interval within the simulation domain. (Image by this blog author)
PINN workflow. In practice, PINNs may converge to wrong solutions during training. One promising way to address this issue is by gradually expanding the solution interval within the simulation domain. (Image by this blog author)

2.2 Solution

The key ideas are to gradually expand the solution interval and leverage ensemble learning.

Residual points are gradually included to mimic the behavior of solution domain expansion. (Image by this blog author)
Residual points are gradually included to mimic the behavior of solution domain expansion. (Image by this blog author)

At the beginning (a), only residual points (pre-sampled to fill the entire solution domain) that are close to the points sampled at known initial conditions (solid blue rectangles) are deployed for PDE loss computation. This kicks off the PINN training, where multiple PINN models initialized with different weights are trained, thus forming an ensemble of PINN. After training for certain iterations (b), some of the previously deployed residual points are considered "pseudo points", if the variances of PINN ensemble predictions are sufficiently small at those locations. Afterward, the solution domain can be expanded (c), such that a new set of residual points will be deployed if there are close enough to the "pseudo points". The iteration goes on (d) until all residual points are deployed for loss calculation.

The above walk-through assumes only initial conditions are known. For other cases where observation data is available inside of the simulation domain, the solution interval can also expand from there.

According to the authors’ implementation, several new hyperparameters were introduced for this strategy, which determine the threshold of distance to deploy new residual points, the threshold of distance to turn deployed residual points into "pseudo points", as well as the threshold of variance to decide if PINN ensembles agree with each other. Please refer to the original paper for more details.

2.3 Why the solution might work

The gradual expansion of the solution interval mimics the behavior of the classical numerical solvers that "propagate" the solution from the initial and boundary conditions to the entire domain. As a result, the physical causality is respected, which potentially contributes to more stable PINN training.

By leveraging an ensemble of PINNs, the proposed algorithm would have a higher chance to escape from wrong solutions that are often found with traditional, single-PINN training methods. This explains why the proposed algorithm could reach a higher level of accuracy and reliability.

2.4 Benchmark

The paper extensively benchmarked the proposed strategy’s performance on five diverse problems, each representing a unique physical phenomenon:

  • Convection equation: this equation models transport phenomena, which are fundamental in various fields, including fluid dynamics, heat transfer, and mass transfer. In these scenarios, understanding the movement of quantities such as energy, mass, or momentum, primarily due to differential forces or gradients, is crucial.
  • Reaction system: this problem category models chemical reactions. From basic chemistry classes to complex bioengineering processes, understanding reaction kinetics and equilibrium can be the difference between successful experiments or industrial processes and catastrophic failures.
  • Reaction-diffusion equation: this equation models reactions combined with the diffusion of substances. This type of problem is pivotal in areas like biology, where they describe processes like pattern formation in developmental biology, or in chemistry, where they model the spread and interaction of different species in a medium.
  • Diffusion equation (with periodic boundary conditions): this type of equation is central to understanding heat conduction, fluid flow, and Brownian motion, among other things. Periodic boundary conditions mean that the system behavior repeats over time or space, a common assumption in problems dealing with cyclic or repeating systems.
  • Diffusion equation (same equation as shown above, but with boundary conditions of the Dirichlet-type): Dirichlet-type boundary conditions stipulate the values a solution must take on the boundary of the domain.

The benchmark studies yielded that:

  • The proposed algorithm provides stable training and shows competitive performance for all considered problems;
  • The proposed algorithm is generally stable with little sensitivity to the hyperparameter choices.

Additionally, the paper suggested that the following practices can help improve PINN accuracy:

  • Normalizing the inputs (both spatial input x and temporal input t) to [-1, 1];
  • Fine-tuning the PINN training with L-BFGS after the usual Adam optimization

2.5 Strength and Weakness

👍 Strengths

  • Able to stabilize PINNs’ training and yield competitive performance.
  • Do not need a pre-defined schedule of interval expansion (in contrast to the time-adaptive strategies explained in the "Alternatives" section).
  • Treat time and space equally.
  • Flexible enough to allow easy incorporation of measurements in arbitrary locations.
  • Confidence intervals of the predictions are automatically obtained thanks to the ensemble approach.

👎 Weaknesses

  • Computationally expensive compared to vanilla PINNs (common for all ensemble approaches).
  • Introduced extra hyperparameters (although the proposed solution is shown to be insensitive to hyperparameter values).

2.6 Alternatives

Other methods that address a similar problem are the family of time-adaptive techniques, which heavily influenced the current approach. Time-adaptive techniques split the time interval [_T_₁, _T_₂] into multiple sub-intervals and solve the equation on each individual sub-interval sequentially by a separate PINN. The solution from one sub-interval is then used as an initial condition for the subsequent sub-interval.

The current algorithm inherits the merits of time-adaptive techniques, i.e., capable of achieving a more accurate and stable propagation of the solution in time. On top of that, it also removes the constraints of requiring a pre-defined schedule of interval expansion. As a result, the current approach is more flexible to incorporate known measurements in arbitrary locations.


3 Potential Future Improvements

There are several possibilities to further improve the proposed strategy:

  • Efficient ways of creating model ensembles (e.g., dropout).
  • Combine the current algorithm with other PINN best practices (e.g., intelligent sampling of residual points, as discussed in the first blog).

4 Takeaways

In this blog, we looked at enhancing PINN training stability through ensemble learning and dynamic solution interval expansion. This topic is highly relevant as it directly impacts the accuracy of the PINNs.

Here are the takeaways from the design pattern proposed in the paper:

  • [Problem]: How to ensure that PINNs converge to the correct solution?
  • [Solution]: 1. Gradually expand the solution interval, where residual points are gradually deployed until fill the entire domain. 2. Ensemble learning, where multiple PINNs initialized with different weights are.
  • [Potential benefits]: 1. Stabilize PINN training. 2. Uncertain estimates of the PINN predictions

Here is the PINN design card I prepared if you prefer a visual presentation:

PINN design pattern proposed in the paper. (Image by this blog author)
PINN design pattern proposed in the paper. (Image by this blog author)

I hope you found this blog useful😃 If you want to learn more about PINN design patterns, feel free to take a look at other posts in this series:

Looking forward to sharing more insights with you in the upcoming blogs!

Reference

[1] Haitsiukevich et al., Improved training of physics-informed neural networks with model ensembles, arXiv, 2022.


Related Articles