7 7.5. Definition of Markov Decision Process for Maintenance Policy Optimisation of a sewer main considering deterioration over the pipe length 161 7.5.2 Action space A Our action space Ais discrete with dimensionality |A| =3. At each time step t, the agent selects an action a. If the decision at time t is do nothing, at is set to 0. To performmaintenance, at is set to 1, and to replace the pipe, at is set to 2. The outcomes of these actions are discussed in Section 7.5.3. 7.5.3 Transition probability function P Our transition function P(st+1|st, a) is stochastic, dependent on time t, and considers both the actions a ↓Aand the current st and next state st+1 dynamics described by the MSDM. We illustrate the behaviour of P with the following example. For a 30-year-old pipe with length L=40 meters and discretised in segments of length ∆L=1, let the current state space be st=30 ↓S: st=30 =↔30, 0.60, 0.35, 0.025, 0.025, 0.0, 0.0, 0.475, 0.436, 0.069, 0.010, 0.005, 0.005↗. st=30 indicates the age of the pipe is 30 years. From Eq. 7.5, the number of segments at severity k is determined by multiplying the health vector (hk): hk = [0.60, 0.35, 0.025, 0.025, 0.0, 0.0] by 40 meters, yielding 1k = [24, 14, 1, 1, 0, 0], indicating that, out of the 40 meters of pipe length, 24 segments of 1 meter are at severity k =1, 14 at severity k =2, and so forth. The distribution pk(t =30.0) predicts the probability of being in a severity level k at age t =30. This is achieved by evaluating t =30.0 in the corresponding MSDM. pk(t =30.0)= [0.475, 0.436, 0.069, 0.010, 0.005, 0.005] Assuming the agent takes an action every half year, we illustrate the e!ect of each action in Abelow. • If at =0: the agent decides to “do nothing”, the pipe’s degradation evolves in line with the MSDM progression. Here the new state space becomes sa=0 t=30.5. sa=0 t=30.5 =↔30.5, 0.575, 0.35, 0.05, 0.025, 0.0, 0.0, 0.470, 0.439, 0.071, 0.010, 0.05, 0.05↗ Notice that the pipe age increased to 30.5, and 1k = [23, 14, 2, 1, 0, 0], where a segment with severity k =1 progressed to k =2, and one segment with k =2 advanced to k =3. Additionally, pk(t) is updated by evaluating t =30.5.
RkJQdWJsaXNoZXIy MjY0ODMw