158 Chapter 7. Maintenance Strategies for Sewer Pipes with Multi-State Deterioration and Deep Reinforcement Learning intentional, to emphasise details and behaviours that various deterioration models express across severity levels. Gray circles represent the frequency per severity level from the inspection dataset. Jimenez-Roa, Heskes, Tinga, et al., 2022 details how these frequencies are computed. Vertical black lines in Figure 7.2 mark the last available data point for each severity level. Additionally, Figure 7.2 presents the Turnbull non-parametric estimator (see Section 6.2.3), which assumes no specific distribution for survival times (Turnbull, 1976). In our context, this estimator represents the ground truth of stochastic deterioration behaviour in sewer mains. Tables 7.1 presents the Root Mean Square Error (RMSE) computed with respect to the Turnbull estimator, for each MSDM assumption, for cohorts CMW. These results show that models employing Gompertz and Weibull distributions yield smaller RMSEs compared to the one using the Exponential distribution. Table 7.1: RMSE with respect Turnbull estimator, per severity level k and total RMSE, cohort: CMW. Exponential Gompertz Weibull pk=1(t) 3.38E-02 3.27E-02 3.34E-02 pk=2(t) 7.04E-02 3.70E-02 3.57E-02 pk=3(t) 6.27E-02 2.81E-02 4.38E-02 pk=4(t) 4.28E-03 1.13E-02 5.06E-03 pk=5(t) 8.33E-03 1.09E-02 3.04E-02 pk=F(t) 9.19E-03 1.17E-02 3.62E-03 Total 4.13E-02 2.45E-02 2.96E-02 These MSDMs serve two crucial roles within our environment: first, they drive the stochastic deterioration behaviour of sewer mains, e!ectively emulating how sewer mains degrade over time. Second, the output from the MSDMs is incorporated as prognostic information, available to the agent to support decisions at any time point. This latter aspect is considered a novel feature of our framework. Details on the MDP are provided in the section below. 7.5 Definition of Markov Decision Process for Maintenance Policy Optimisation of a sewer main considering deterioration over the pipe length Figure 7.3 provides the workflow that the RL agent uses to learn maintenance policies for sewer mains, considering deterioration along the pipe length. In the
RkJQdWJsaXNoZXIy MjY0ODMw