Generalization analyses of deep learning typically assume that the training converges to a fixed point. But, recent results indicate that in practice, the weights of deep neural networks optimized with stochastic gradient descent often oscillate indefinitely. To reduce this discrepancy between theory and practice, this paper focuses on the generalization of neural networks whose training dynamics do not necessarily converge to fixed points. Our main contribution is to propose a notion of statistical algorithmic stability (SAS) that extends classical algorithmic stability to non-convergent algorithms and to study its connection to generalization. This ergodic-theoretic approach leads to new insights when compared to the traditional optimization and learning theory perspectives. We prove that the stability of the time-asymptotic behavior of a learning algorithm relates to its generalization and empirically demonstrate how loss dynamics can provide clues to generalization performance. Our findings provide evidence that networks that “train stably generalize better” even when the training continues indefinitely and the weights do not converge.

Keywords: Computer Science - Machine Learning, Mathematics - Dynamical Systems, Mathematics - Optimization and Control, Statistics - Machine Learning

Ruelle (1997 Commun. Math. Phys. 187 227–41; 2003 Commun. Math. Phys. 234 185–90) (see also Jiang 2012 Ergod. Theor. Dynam. Syst. 32 1350–69) gave a formula for linear response of transitive Anosov diffeomorphisms. Recently, practically computable realizations of Ruelle’s formula have emerged that potentially enable sensitivity analysis of certain high-dimensional chaotic numerical simulations encountered in the applied sciences. In this paper, we provide full mathematical justification for the convergence of one such efficient computation, the space–split sensitivity, or S3, algorithm (Chandramoorthy and Wang 2022 SIAM J. Appl. Dyn. Syst. 21 735–81). In S3, Ruelle’s formula is computed as a sum of two terms obtained by decomposing the perturbation vector field into a coboundary and a remainder that is parallel to the unstable direction. Such a decomposition results in a splitting of Ruelle’s formula that is amenable to efficient computation. We prove the existence of the S3 decomposition and the convergence of the computations of both resulting components of Ruelle’s formula.

The sensitivity of time averages in a chaotic system to an infinitesimal parameter perturbation grows exponentially with the averaging time. However, long-term averages or ensemble statistics often vary differentiably with system parameters. Ruelle's response theory gives a rigorous formula for these parametric derivatives of statistics or linear response. But the direct evaluation of this formula is ill-conditioned, and hence linear response and downstream applications of sensitivity analysis, such as optimization and uncertainty quantification, have been a computational challenge in chaotic dynamical systems. This paper presents the space-split sensitivity (S3) algorithm to transform Ruelle's formula into a well-conditioned ergodic-averaging computation. We prove a decomposition of Ruelle's formula that is differentiable on the unstable manifold, which we assume to be one-dimensional. This decomposition of Ruelle's formula ensures that one of the resulting terms, the stable contribution, can be computed using a regularized tangent equation, similarly as in a nonchaotic system. The remaining term, known as the unstable contribution, is regularized and converted into an efficiently computable ergodic average. In this process, we develop new algorithms, which may be useful beyond linear response, to compute the unstable derivatives of the regularized tangent vector field and the unstable direction. We prove that the S3 algorithm, which combines these computational ingredients that enter the stable and unstable contributions, converges like a Monte Carlo approximation of Ruelle's formula. The algorithm presented here is hence a first step toward full-fledged applications of sensitivity analysis in chaotic systems, wherever such applications have been limited due to lack of availability of long-term sensitivities.

An assumption of smooth response to small parameter changes, of statistics or long-time averages of a chaotic system, is generally made in the field of sensitivity analysis, and the parametric derivatives of statistical quantities are critically used in science and engineering. In this paper, we propose a numerical procedure to assess the differentiability of statistics with respect to parameters in chaotic systems. We numerically show that the existence of the derivative depends on the Lebesgue-integrability of a certain density gradient function, which we define as the derivative of logarithmic SRB density along the unstable manifold. We develop a recursive formula for the density gradient that can be efficiently computed along trajectories, and demonstrate its use in determining the differentiability of statistics. Our numerical procedure is illustrated on low-dimensional chaotic systems whose statistics exhibit both smooth and rough regions in parameter space.

Keywords: Chaotic dynamical systems, Sensitivity analysis, Linear response theory, SRB density gradient function

This paper proves that shadowing solutions can be almost surely nonphysical. This finding invalidates the argument that small perturbations in a chaotic system can only have a small impact on its statistical behavior. This theoretical finding has implications for many applications in which chaotic dynamics plays an important role. It suggests, for example, that we can control the climate through subtle perturbations. It also suggests that numerical simulations of chaotic dynamics, such as turbulent flows and global atmosphere and ocean circulation, may fail to predict the true long-term or statistical behavior.

Keywords: Chaotic dynamics, Numerical simulations, Scientific computing, Shadowing sensitivity analysis

How does long-term chaotic behavior respond to small parameter perturbations? Using detailed models, chaotic systems are frequently simulated across disciplines – from climate science to astrophysics. But, an efficient computation of parametric derivatives of their statistics or long-term averages, also known as linear response, is an open problem. The difficulty is due to an inherent feature of chaos: an exponential growth over time of infinitesimal perturbations, which renders conventional methods for sensitivity computation inapplicable. More sophisticated recent approaches, including ensemble-based and shadowing-based methods are either computationally impractical or lack convergence guarantees. We propose a novel alternative known as space-split sensitivity or S3, which evaluates linear response as an efficiently computable, provably convergent ergodic average. The main contribution of this thesis is the development of the S3 algorithm for uniformly hyperbolic systems – the simplest setting in which chaotic attractors occur – with one-dimensional unstable manifolds. S3 can enable applications of the computed sensitivities to optimization, control theory and uncertainty quantification, in the realm of chaotic dynamics, wherein these applications remain nascent. We propose a transformation of Ruelle’s rigorous linear response formula, which is ill-conditioned in its original form, into a well-conditioned ergodic-averaging computation. We prove a decomposition of Ruelle’s formula, called the S3 decomposition, that is differentiable on the unstable manifold. The S3 decomposition ensures that one of the resulting terms, the stable contribution, can be computed using a regularized tangent equation, similar to in a non-chaotic system. The remainder, known as the unstable contribution, is regularized and converted into a computable ergodic average. The S3 algorithm presented here can be naturally extended to systems with higher-dimensional unstable manifolds. The secondary contributions of this thesis are analysis and applications of existing methods, including those shadowing-based and ensemble-based, to compute linear response. A feasibility analysis of ensemble sensitivity calculation, which is a direct evaluation of Ruelle’s formula, reveals a problem-dependent, typically poor rate of convergence, rendering it computationally impractical. Shadowing-based sensitivity computation is not guaranteed to converge because of atypicality of shadowing orbits. This atypicality also implies that small parameter perturbations can lead, contrary to popular belief, to a large change in the statistics of a chaotic system, a consequence being that numerical simulations of chaotic systems may not reproduce their true long-term behaviors.

Covariant Lyapunov vectors or CLVs span the expanding and contracting directions of perturbations along trajectories in a chaotic dynamical system. Due to efficient algorithms to compute them that only utilize trajectory information, they have been widely applied across scientific disciplines, principally for sensitivity analysis and predictions under uncertainty. In this paper, we develop a numerical method to compute the directional derivatives of the first CLV along its own direction; the norm of this derivative is also the curvature of one-dimensional unstable manifolds. Similar to the computation of CLVs, the present method for their derivatives is iterative and analogously uses the second-order derivative of the chaotic map along trajectories, in addition to the Jacobian. We validate the new method on a super-contracting Smale–Williams Solenoid attractor. We also demonstrate the algorithm on several other examples including smoothly perturbed Arnold Cat maps, and the Lorenz’63 attractor, obtaining visualizations of the curvature of each attractor. Furthermore, we reveal a fundamental connection of the derivation of the CLV self-derivative computation with an efficient computation of linear response of chaotic systems

In this computational paper, we perform sensitivity analysis of long- time (or ensemble) averages in the chaotic regime using the shadowing algorithm. We introduce automatic differentiation to eliminate the tangent/adjoint equation solvers used in the shadowing algorithm. In a gradient-based optimization, we use the computed shadowing sensitivity to minimize different long- time averaged functionals of a chaotic time-delayed system by optimal parameter selection. In combined state and parameter estimation for data assimilation, we use the computed sensitivity to predict the optimal trajectory given information from a model and data from measurements beyond the predictability time. The algorithms are applied to a thermoacoustic model. Because the computational framework is rather general, the techniques presented in this paper may be used for sensitivity analysis of ensemble averages, parameter optimization and data assimilation of other chaotic problems, where shadowing methods are applicable.

Keywords: Mathematics - Dynamical Systems, Nonlinear Sciences - Chaotic Dynamics

ABSTRACT Sensitivity analysis in chaotic dynamical systems is a challenging task from a computational point of view. In this work, we present a numerical investigation of a novel approach, known as the space-split sensitivity or S3 algorithm. The S3 algorithm is an ergodic-averaging method to differentiate statistics in ergodic, chaotic systems, rigorously based on the theory of hyperbolic dynamics. We illustrate S3 on one-dimensional chaotic maps, revealing its computational advantage over naïve finite difference computations of the same statistical response. In addition, we provide an intuitive explanation of the key components of the S3 algorithm, including the density gradient function.

Keywords: Sensitivity analysis, Chaotic systems, Ergodicity, Space-split sensitivity (S3) method

We present a computable reformulation of Ruelle's linear response formula for chaotic systems. The new formula, called Space-Split Sensitivity or S3, achieves an error convergence of the order O(1/sqrt(N)) using N phase points. The reformulation is based on splitting the overall sensitivity into that to stable and unstable components of the perturbation. The unstable contribution to the sensitivity is regularized using ergodic properties and the hyperbolic structure of the dynamics. Numerical examples of uniformly hyperbolic attractors are used to validate the S3 formula against a naïve finite- difference calculation; sensitivities match closely, with far fewer sample points required by S3.

Keywords: Mathematics - Dynamical Systems, Mathematical Physics, Nonlinear Sciences - Chaotic Dynamics

In chaotic systems, such as turbulent flows, the solutions to tangent and adjoint equations exhibit an unbounded growth in their norms. This behavior renders the instantaneous tangent and adjoint solutions unusable for sensitivity analysis. The Lea–Allen–Haine ensemble sensitivity (ES) estimates provide a way of computing meaningful sensitivities in chaotic systems by using tangent/adjoint solutions over short trajectories. In this paper, the feasibility of ES computations is analyzed under optimistic mathematical assumptions on the flow dynamics. Furthermore, upper bounds are estimated on the rate of convergence of the ES method in numerical simulations of turbulent flow. Even at the optimistic upper bound, the ES method is computationally intractable in each of the numerical examples considered.

Chaotic dynamical systems such as turbulent flows are characterized by an exponential divergence of infinitesimal perturbations to initial conditions. Therefore, conventional adjoint/tangent sensitivity analysis methods that are successful with RANS simulations fail in the case of chaotic LES/DNS. In this work, we discuss the limitations of current approaches, including ensemble-based and shadowing-based sensitivity methods, that were proposed as alternatives to conventional sensitivity analysis. We propose a new alternative, called the space-split sensitivity (S3) algorithm, that is computationally efficient and addresses these limitations. In this work, the derivation of the S3 algorithm is presented in the special case where the system converges to a stationary distribution that can be expressed with a probability density function everywhere in phase-space. Numerical examples of low-dimensional chaotic maps are discussed where S3 computation shows good agreement with finite-difference results, indicating potential for the development of the method in more generality.

Keywords: Computer Science - Computational Engineering, Finance, and Science, Nonlinear Sciences - Chaotic Dynamics

It is well-known that linearized perturbation methods for sensitivity analysis, such as tangent or adjoint equation-based, finite difference and automatic differentiation are not suitable for turbulent flows. The reason is that turbulent flows exhibit chaotic dynamics, leading to the norm of an infinitesimal perturbation to the state growing exponentially in time. As a result, these conventional methods cannot be used to compute the derivatives of long-time averaged quantities to control or design inputs. The ensemble-based approaches and shadowing-based approaches to circumvent the problems of the conventional methods in chaotic systems, also suffer from computational impracticality and lack of consistency guarantees, respectively. We introduce the space-split sensitivity, or the S3 algorithm, which is a Monte-Carlo approach to the chaotic sensitivity computation problem. In this work, we derive the S3 algorithm under simplifying assumptions on the dynamics and present a numerical validation on a low-dimensional example of chaos.

Lubrication problems at lengthscales for which the traditional Navier–Stokes description fails can be solved using a modified Reynolds lubrication equation that is based on the following two observations: first, classical Reynolds equation failure at small lengthscales is a result of the failure of the Poiseuille flowrate closure (the Reynolds equation is derived from a statement of mass conservation, which is valid at all scales); second, averaging across the film thickness eliminates the need for a constitutive relation providing spatial resolution of flow profiles in this direction. In other words, the constitutive information required to extend the classical Reynolds lubrication equation to small lengthscales is limited to knowledge of the flowrate as a function of the gap height, which is significantly less complex than a general constitutive relation, and can be obtained by experiments and/or offline molecular simulations of pressure-driven flow under fully developed conditions. The proposed methodology, which is an extension of the generalized lubrication equation of Fukui and Kaneko to dense fluids, is demonstrated and validated via comparison to molecular dynamics simulations of a model lubrication problem

Classical lubrication theory is unable to describe nanoscale flows due to the failure of two of its constitutive components: a) the Newtonian stress-strain rate relationship and b) the no-slip boundary condition. In this thesis, we present a methodology for deriving a modified Reynolds equation (referred to as the Molecular Dynamics-based Equation for Lubrication, or the MODEL) which overcomes these limitations by introducing a Molecular Dynamics-based constitutive relationship for the flow rate through the lubrication gap, that is valid beyond the range of validity of the Navier-Stokes constitutive models. We demonstrate the proposed methodology for the flow of a simple lubricant, n-hexadecane, between smooth Iron walls and show that the MODEL is able to predict flow rates with good accuracy even in nanochannels that are only a few atomic layers wide. The MODEL constitutive relationship for the flow rate used in this work is a slip-corrected Poiseuille model with the slip length and viscosity derived from Molecular Dynamics (MD) simulations of pressure-driven flow in nanochannels sufficiently large that the Navier-Stokes description is valid. Although more general expressions for the flow rate can certainly be used, for the lubricant-solid system modeled here, the slip-corrected Poiseuille flow was surprisingly found to be sufficient. We validate the MODEL by comparing MD results for the pressure distribution in a barrel-drop lubrication configuration with the analytical solution for the pressure obtained by solving the MODEL. The excellent agreement obtained between the dynamic pressure in the fluid measured from these MD simulations and the MODEL results suggests that it is possible to extend pde-based hydrodynamic modelling of lubrication problems even to nanoscale films beyond the validity of the Navier-Stokes description. In other words, once the flow rate constitutive relation is obtained, lubrication problems in nanoscale films can be solved without resorting to expensive particle methods like MD. We demonstrate that slip cannot be neglected in the boundary lubrication regime by considering various lubrication problems of practical interest. Using a simple barrel-drop lubrication model for the top two rings in an internal combustion engine, we show that for lubrication gaps with a minimum thickness that is ten times the size of the slip length, the normal force and the frictional force are overestimated by a factor of 1.5 when assuming no-slip. By modifying the Twin Land Oil Control Ring (TLOCR)-liner interface model to include slip, we find significant reduction in the hydrodynamic pressure and the friction when compared to the original model; the oil flow rate does not change appreciably. Finally, we chalk out a procedure for the inclusion of slip in the methodology for developing correlations for the pressure, friction and the flow rate in the TLOCR-liner system.