solver_ctmc_reward.h File Reference

Port of solver_ctmc_reward.m and the @@SolverCTMC reward surface (runRewardAnalyzer, getAvgReward, getTranReward). More...

#include <cmath>
#include <cstddef>
#include <string>
#include <vector>
#include "line/lang/qn/network_struct.h"
#include "line/solvers/ctmc/solver_ctmc_analyzer.h"
#include "line/solvers/ctmc/solver_ctmc_transient.h"
#include "line/util/error.h"
#include "line/util/matrix.h"

Include dependency graph for solver_ctmc_reward.h:

Go to the source code of this file.

Classes
struct	line::ctmc::CtmcReward< T >
	What the reward analyzer produces, per declared reward. More...

Namespaces
namespace	line
namespace	line::ctmc

Functions
template<class T>
CtmcReward< T >	line::ctmc::solver_ctmc_reward (const NetworkStruct< T > &sn, const CtmcOptions &opt, std::size_t tmax=1000)
	Port of solver_ctmc_reward.m.
template<class T>
std::vector< std::vector< T > >	line::ctmc::solver_ctmc_tran_reward (const NetworkStruct< T > &sn, const CtmcOptions &opt, const T &t0, const T &t1, std::vector< T > *tout=nullptr, std::vector< std::string > *names=nullptr)
	Port of @@SolverCTMC/getTranReward: E[r(X(t))] = sum_s pi_t(s) r(s).
template<class T>
std::vector< T >	line::ctmc::solver_ctmc_avg_reward (const NetworkStruct< T > &sn, const CtmcOptions &opt, std::vector< std::string > *names=nullptr)
	Port of @@SolverCTMC/getAvgReward: the steady-state expected rewards.

Detailed Description

Port of solver_ctmc_reward.m and the @@SolverCTMC reward surface (runRewardAnalyzer, getAvgReward, getTranReward).

TWO DIFFERENT QUANTITIES SHARE THE WORD "REWARD", and conflating them is the trap this file exists to avoid:

steady state E[r] = sum_s pi(s) r(s), a RATE – the long-run average reward earned per unit time. value function V^k(s), the reward ACCUMULATED over k uniformized steps starting from s. It grows without bound in a recurrent chain, because it is a total and not an average.

V is therefore not "the transient version of E[r]" and does not converge to it; its SLOPE does. The reference reports both and so does this port.

UNIFORMIZATION IS WHAT MAKES THE VALUE ITERATION A CTMC ANSWER. The embedded chain P = Q/q + I with q = max|diag(Q)| has the same stationary law as Q and a uniform step of mean duration 1/q, so iteration index k maps to time k/q. Any q at least as large as the maximum exit rate is valid; taking the maximum is the tightest, hence the fastest-mixing, choice.

Definition in file solver_ctmc_reward.h.