LINE Solver (C++)
Templated C++ port of the LINE queueing solver
Loading...
Searching...
No Matches
infer_smp.h
Go to the documentation of this file.
1/*
2 * Copyright (c) 2012-2026, QORE Lab, Imperial College London
3 * All rights reserved.
4 */
5#ifndef LINE_API_INFER_INFER_SMP_H
6#define LINE_API_INFER_INFER_SMP_H
7
8/**
9 * @file
10 * @ingroup api_infer
11 * Maximum-likelihood fit of a parametric semi-Markov multi-state model.
12 *
13 * Port of `matlab/src/api/infer/infer_smp.m` and its four companions,
14 * `infer_smp_family.m`, `infer_smp_env.m`, `infer_smp_hazard.m` and
15 * `infer_smp_nelson_aalen.m`, which this header carries together.
16 *
17 * DATA. One row per observed sojourn: the process entered state `from[k]`,
18 * stayed `dur[k] > 0`, and then moved to state `to[k]`, or was still in
19 * `from[k]` when observation stopped, coded `to[k] = 0` (right censoring).
20 * States are 1..E, as in the reference, since 0 is the censoring code.
21 *
22 * THE TWO APPROACHES of Asanjarani, Liquet and Nazarathy (arXiv:2005.14462,
23 * 2020). `intensity` (Approach II, the default) fits each transition i -> j as
24 * a latent clock with its own law, so the likelihood splits into one censored
25 * two-state likelihood per transition, every sojourn out of i that did not end
26 * in j counting as censored for i -> j. `sojourn` (Approach I) fits the
27 * embedded chain p_ij and the law F_ij of a sojourn that ends in j; a censored
28 * sojourn contributes S_i(U) = sum_j p_ij S_ij(U), which couples the
29 * parameters of a state, so a state with censored rows is fitted jointly (p by
30 * a softmax over the logits of destinations 2..m) and one without in closed
31 * form for p.
32 *
33 * FAMILIES, in unconstrained coordinates THETA (`infer_smp_family_*`):
34 * exp THETA = log(rate)
35 * weibull THETA = [log(shape), log(scale)]
36 * gamma THETA = [log(shape), log(scale)]
37 * lognormal THETA = [mu, log(sigma)]
38 * Covariates enter as a proportional-hazards factor exp(beta' z).
39 *
40 * THE MAXIMIZER is the reference's deterministic Nelder-Mead (the Lagarias et
41 * al. variant of fminsearch, restarted from its own optimum until it stops
42 * improving), step for step, so the four codebases walk the same simplex; an
43 * exponential transition without covariates has its closed-form MLE. Standard
44 * errors come from the central-difference Hessian of the negative
45 * log-likelihood in THETA coordinates, NaN when it is not positive definite.
46 */
47
48#include <algorithm>
49#include <cctype>
50#include <cmath>
51#include <cstddef>
52#include <functional>
53#include <limits>
54#include <string>
55#include <vector>
56
63#include "line/util/error.h"
64#include "line/util/matrix.h"
65
66namespace line {
67namespace infer {
68
69// ---- infer_smp_family -------------------------------------------------------
70
71/** Lower-case copy of a family name. */
72inline std::string smp_family_name(const std::string& f) {
73 std::string s = f;
74 std::transform(s.begin(), s.end(), s.begin(),
75 [](unsigned char c) { return static_cast<char>(std::tolower(c)); });
76 return s;
77}
78
79/** `infer_smp_family(F, 'npar')`: the number of parameters, validating the name. */
80inline std::size_t infer_smp_family_npar(const std::string& family) {
81 const std::string f = smp_family_name(family);
82 if (f == "exp") return 1;
83 if (f == "weibull" || f == "gamma" || f == "lognormal") return 2;
84 throw InputError("infer_smp_family: Unknown family '" + f +
85 "': use 'exp', 'weibull', 'gamma' or 'lognormal'.");
86}
87
88/**
89 * `infer_smp_family(F, 'logfs', THETA, T, ETA)`: log density and log survival
90 * at `t` under the proportional-hazards factor exp(eta): H = H0 exp(eta),
91 * log h = log h0 + eta, log f = log h - H. `eta` holds one value per instant,
92 * or a single value for all of them, or nothing for the baseline. The survival
93 * of 'gamma' and 'lognormal' is floored at 1e-300 before the log.
94 */
95inline void infer_smp_family_logfs(const std::string& family, const std::vector<double>& th,
96 const std::vector<double>& t, const std::vector<double>& eta,
97 std::vector<double>& logf, std::vector<double>& logS) {
98 const std::string f = smp_family_name(family);
99 const std::size_t n = t.size();
100 logf.assign(n, 0.0);
101 logS.assign(n, 0.0);
102 if (!(eta.empty() || eta.size() == 1 || eta.size() == n))
103 throw InputError("infer_smp_family: ETA must be a scalar or hold one value per instant");
104 const double log2pi = std::log(2.0 * 3.14159265358979323846);
105 for (std::size_t k = 0; k < n; ++k) {
106 const double tk = t[k];
107 double H0 = 0.0, logh0 = 0.0;
108 if (f == "exp") {
109 const double lam = std::exp(th[0]);
110 logh0 = std::log(lam);
111 H0 = lam * tk;
112 } else if (f == "weibull") {
113 const double kk = std::exp(th[0]), s = std::exp(th[1]);
114 H0 = std::pow(tk / s, kk);
115 logh0 = std::log(kk) - std::log(s) + (kk - 1.0) * (std::log(tk) - std::log(s));
116 } else if (f == "gamma") {
117 const double a = std::exp(th[0]), b = std::exp(th[1]);
118 const double lf0 = (a - 1.0) * std::log(tk) - tk / b - std::lgamma(a) - a * std::log(b);
119 const double lS0 = std::log(std::max(pfqn::detail::gammainc(a, tk / b).q, 1e-300));
120 H0 = -lS0;
121 logh0 = lf0 - lS0;
122 } else if (f == "lognormal") {
123 const double mu = th[0], sg = std::exp(th[1]);
124 const double z = (std::log(tk) - mu) / sg;
125 const double lf0 = -std::log(tk) - std::log(sg) - 0.5 * log2pi - z * z / 2.0;
126 const double lS0 = std::log(std::max(0.5 * std::erfc(z / std::sqrt(2.0)), 1e-300));
127 H0 = -lS0;
128 logh0 = lf0 - lS0;
129 } else {
131 }
132 const double e = eta.empty() ? 0.0 : (eta.size() == 1 ? eta[0] : eta[k]);
133 const double H = H0 * std::exp(e);
134 logS[k] = -H;
135 logf[k] = logh0 + e - H;
136 }
137}
138
139/** `infer_smp_family(F, 'natural', THETA)`: rate | shape, scale | mu, sigma. */
140inline std::vector<double> infer_smp_family_natural(const std::string& family,
141 const std::vector<double>& th) {
142 const std::string f = smp_family_name(family);
143 if (f == "exp") return std::vector<double>{std::exp(th[0])};
144 if (f == "weibull" || f == "gamma") return std::vector<double>{std::exp(th[0]), std::exp(th[1])};
145 if (f == "lognormal") return std::vector<double>{th[0], std::exp(th[1])};
147 return std::vector<double>();
148}
149
150/**
151 * `infer_smp_family(F, 'theta0', TEV, TEXP)`: the starting point from the
152 * event times `tev` and the total exposure `texp`.
153 */
154inline std::vector<double> infer_smp_family_theta0(const std::string& family,
155 const std::vector<double>& tev, double texp) {
156 const std::string f = smp_family_name(family);
157 const double d = static_cast<double>(tev.size());
158 if (f == "exp") return std::vector<double>{std::log(d / texp)};
159 if (f == "weibull" || f == "gamma") return std::vector<double>{0.0, std::log(texp / d)};
160 if (f == "lognormal") {
161 std::vector<double> lt(tev.size());
162 double mean = 0.0;
163 for (std::size_t k = 0; k < tev.size(); ++k) {
164 lt[k] = std::log(tev[k]);
165 mean += lt[k];
166 }
167 mean /= d;
168 double sg = 1.0;
169 if (tev.size() >= 2) {
170 // MATLAB's std: the sample standard deviation, n - 1 in the denominator.
171 double ss = 0.0;
172 for (double v : lt) ss += (v - mean) * (v - mean);
173 const double sd = std::sqrt(ss / (d - 1.0));
174 if (sd > 0.0) sg = sd;
175 }
176 return std::vector<double>{mean, std::log(sg)};
177 }
179 return std::vector<double>();
180}
181
182/**
183 * `infer_smp_family(F, 'dist', THETA, ETA)`: the LINE distribution of the law
184 * with hazard h0(t) exp(eta). Only 'exp' and 'weibull' are closed under a
185 * proportional-hazards factor; the other two are refused for eta != 0.
186 */
187inline lang::Distrib<double> infer_smp_family_dist(const std::string& family,
188 const std::vector<double>& th, double eta) {
189 const std::string f = smp_family_name(family);
190 const std::vector<double> p = infer_smp_family_natural(f, th);
191 if (f == "exp") return lang::Distrib<double>::exp_rate(p[0] * std::exp(eta));
192 // MATLAB's Weibull(shape, scale) is the C++ Distrib::weibull(scale, shape).
193 if (f == "weibull") return lang::Distrib<double>::weibull(p[1] * std::exp(-eta / p[0]), p[0]);
194 if (eta != 0.0)
195 throw InputError("infer_smp_family: A '" + f +
196 "' law scaled by a proportional-hazards factor is not a '" + f +
197 "' law, so it has no LINE distribution: build the environment at the "
198 "baseline (z = 0), or fit 'exp' or 'weibull'.");
199 if (f == "gamma") return lang::Distrib<double>::gamma_dist(p[0], p[1]);
200 return lang::Distrib<double>::lognormal(p[0], p[1]);
201}
202
203// ---- infer_smp -------------------------------------------------------------
204
205/** Options of `infer_smp`, the reference's name/value pairs. */
207 /** 'intensity' (Approach II, default) or 'sojourn' (Approach I). */
208 std::string approach = "intensity";
209 /** The family of every transition: 'weibull' (default), 'exp', 'gamma' or 'lognormal'. */
210 std::string dist = "weibull";
211 /**
212 * Per transition, `dist_table[i][j]` for i -> j (0-based states), when not
213 * empty; it must then be E x E, and an empty entry takes 'weibull'.
214 */
215 std::vector<std::vector<std::string>> dist_table;
216 /** n x p covariates Z, one row per sojourn; an empty matrix means none. */
218 /**
219 * `covmask[i][j][c]`: whether covariate c enters transition i -> j (0-based
220 * states); empty means all, otherwise E x E x p.
221 */
222 std::vector<std::vector<std::vector<bool>>> covmask;
223 /** E; 0 means max(from, to). */
224 std::size_t nstates = 0;
225 double tolx = 1e-9;
226 double tolf = 1e-12;
227 /** Nelder-Mead iteration cap per run; 0 means 2000 per parameter. */
228 std::size_t maxiter = 0;
229};
230
231/** One fitted transition. States are 1-based. */
232struct SmpArc {
233 std::size_t from = 0, to = 0;
234 std::string family;
235 std::vector<double> theta; ///< unconstrained parameters
236 std::vector<double> params; ///< natural parameters
237 std::vector<double> beta; ///< p covariate effects, zero where masked out
238 std::vector<double> se; ///< standard errors of [theta; beta]
239 /** Approach I: p_ij; Approach II: the embedded probability at z = 0. */
240 double prob = std::numeric_limits<double>::quiet_NaN();
241 std::size_t nevents = 0;
242};
243
244/** What `infer_smp` returns. */
245struct SmpFit {
246 std::string approach;
247 std::size_t nstates = 0;
248 std::size_t ncov = 0;
249 std::vector<SmpArc> arcs;
250 double loglik = 0.0;
251 std::size_t nparams = 0;
252 double aic = 0.0;
253 Matrix<double> P; ///< E x E embedded jump chain at z = 0
254};
255
257
258using Objective = std::function<double(const std::vector<double>&)>;
259
260/** MATLAB's `max` over a vector: NaNs ignored, NaN when every entry is one. */
261inline double nanmax(const std::vector<double>& v) {
262 double m = std::numeric_limits<double>::quiet_NaN();
263 for (double x : v) {
264 if (std::isnan(x)) continue;
265 if (std::isnan(m) || x > m) m = x;
266 }
267 return m;
268}
269
270/** One run of the reference's Nelder-Mead (`nmRun`). */
271inline void nm_run(const Objective& f, const std::vector<double>& x0, double tolx, double tolf,
272 std::size_t maxit, std::vector<double>& xb, double& fb) {
273 const std::size_t n = x0.size();
274 std::vector<std::vector<double>> v(n + 1, x0);
275 std::vector<double> fv(n + 1, 0.0);
276 fv[0] = f(x0);
277 for (std::size_t j = 0; j < n; ++j) {
278 std::vector<double> y = x0;
279 if (y[j] != 0.0)
280 y[j] = 1.05 * y[j];
281 else
282 y[j] = 0.00025;
283 v[j + 1] = y;
284 fv[j + 1] = f(y);
285 }
286 // MATLAB's sort is stable, so ties keep their order.
287 const auto sort_simplex = [&]() {
288 std::vector<std::size_t> ix(n + 1);
289 for (std::size_t k = 0; k <= n; ++k) ix[k] = k;
290 std::stable_sort(ix.begin(), ix.end(), [&](std::size_t a, std::size_t b) {
291 const double fa = fv[a], fb2 = fv[b];
292 if (std::isnan(fb2)) return !std::isnan(fa);
293 if (std::isnan(fa)) return false;
294 return fa < fb2;
295 });
296 std::vector<std::vector<double>> v2(n + 1);
297 std::vector<double> f2(n + 1);
298 for (std::size_t k = 0; k <= n; ++k) {
299 v2[k] = v[ix[k]];
300 f2[k] = fv[ix[k]];
301 }
302 v.swap(v2);
303 fv.swap(f2);
304 };
305 sort_simplex();
306 std::size_t it = 0;
307 while (it < maxit) {
308 std::vector<double> df(n), dx;
309 dx.reserve(n * n);
310 for (std::size_t j = 1; j <= n; ++j) {
311 df[j - 1] = std::fabs(fv[0] - fv[j]);
312 for (std::size_t a = 0; a < n; ++a) dx.push_back(std::fabs(v[j][a] - v[0][a]));
313 }
314 if (nanmax(df) <= tolf && nanmax(dx) <= tolx) break;
315 std::vector<double> xbar(n, 0.0);
316 for (std::size_t a = 0; a < n; ++a) {
317 double s = 0.0;
318 for (std::size_t j = 0; j < n; ++j) s += v[j][a];
319 xbar[a] = s / static_cast<double>(n);
320 }
321 std::vector<double>& vw = v[n];
322 std::vector<double> xr(n);
323 for (std::size_t a = 0; a < n; ++a) xr[a] = 2.0 * xbar[a] - vw[a];
324 const double fr = f(xr);
325 if (fr < fv[0]) {
326 std::vector<double> xe(n);
327 for (std::size_t a = 0; a < n; ++a) xe[a] = 3.0 * xbar[a] - 2.0 * vw[a];
328 const double fe = f(xe);
329 if (fe < fr) {
330 vw = xe;
331 fv[n] = fe;
332 } else {
333 vw = xr;
334 fv[n] = fr;
335 }
336 } else if (fr < fv[n - 1]) {
337 vw = xr;
338 fv[n] = fr;
339 } else {
340 bool shrink = false;
341 if (fr < fv[n]) {
342 std::vector<double> xc(n);
343 for (std::size_t a = 0; a < n; ++a) xc[a] = 1.5 * xbar[a] - 0.5 * vw[a];
344 const double fc = f(xc);
345 if (fc <= fr) {
346 vw = xc;
347 fv[n] = fc;
348 } else {
349 shrink = true;
350 }
351 } else {
352 std::vector<double> xcc(n);
353 for (std::size_t a = 0; a < n; ++a) xcc[a] = 0.5 * xbar[a] + 0.5 * vw[a];
354 const double fcc = f(xcc);
355 if (fcc < fv[n]) {
356 vw = xcc;
357 fv[n] = fcc;
358 } else {
359 shrink = true;
360 }
361 }
362 if (shrink)
363 for (std::size_t j = 1; j <= n; ++j) {
364 for (std::size_t a = 0; a < n; ++a)
365 v[j][a] = v[0][a] + 0.5 * (v[j][a] - v[0][a]);
366 fv[j] = f(v[j]);
367 }
368 }
369 sort_simplex();
370 ++it;
371 }
372 xb = v[0];
373 fb = fv[0];
374}
375
376/** `nelderMead`: restart `nm_run` from its own optimum, at most four runs. */
377inline std::vector<double> nelder_mead(const Objective& f, const std::vector<double>& x0,
378 const SmpOptions& opt) {
379 const std::size_t maxit = opt.maxiter > 0 ? opt.maxiter : 2000 * x0.size();
380 std::vector<double> x = x0;
381 double fx = f(x);
382 for (int restart = 0; restart < 4; ++restart) {
383 std::vector<double> xn;
384 double fn = 0.0;
385 nm_run(f, x, opt.tolx, opt.tolf, maxit, xn, fn);
386 const bool improved = fx - fn > opt.tolf;
387 if (fn <= fx) {
388 x = xn;
389 fx = fn;
390 }
391 if (!improved) break;
392 }
393 return x;
394}
395
396/**
397 * `hessianSE`: central-difference Hessian with steps 1e-4 max(1, |x|), its
398 * Cholesky factor H = R'R, and se = sqrt(rowsum((R \ I).^2)); NaN everywhere
399 * when H is not finite or not positive definite.
400 */
401inline std::vector<double> hessian_se(const Objective& f, const std::vector<double>& x) {
402 const std::size_t n = x.size();
403 std::vector<double> h(n);
404 for (std::size_t a = 0; a < n; ++a) h[a] = 1e-4 * std::max(1.0, std::fabs(x[a]));
405 Matrix<double> H(n, n, 0.0);
406 const double f0 = f(x);
407 for (std::size_t a = 0; a < n; ++a) {
408 std::vector<double> xp = x, xm = x;
409 xp[a] += h[a];
410 xm[a] -= h[a];
411 H(a, a) = (f(xp) - 2.0 * f0 + f(xm)) / (h[a] * h[a]);
412 for (std::size_t b = a + 1; b < n; ++b) {
413 std::vector<double> pp = xp, pm = xp, mp = xm, mm = xm;
414 pp[b] += h[b];
415 pm[b] -= h[b];
416 mp[b] += h[b];
417 mm[b] -= h[b];
418 H(a, b) = (f(pp) - f(pm) - f(mp) + f(mm)) / (4.0 * h[a] * h[b]);
419 H(b, a) = H(a, b);
420 }
421 }
422 const std::vector<double> nan(n, std::numeric_limits<double>::quiet_NaN());
423 for (std::size_t a = 0; a < n; ++a)
424 for (std::size_t b = 0; b < n; ++b)
425 if (!std::isfinite(H(a, b))) return nan;
426 // chol(H): the upper factor R, reading the upper triangle as MATLAB does.
427 Matrix<double> R(n, n, 0.0);
428 for (std::size_t j = 0; j < n; ++j) {
429 double s = H(j, j);
430 for (std::size_t k = 0; k < j; ++k) s -= R(k, j) * R(k, j);
431 if (!(s > 0.0)) return nan;
432 R(j, j) = std::sqrt(s);
433 for (std::size_t i = j + 1; i < n; ++i) {
434 double t = H(j, i);
435 for (std::size_t k = 0; k < j; ++k) t -= R(k, j) * R(k, i);
436 R(j, i) = t / R(j, j);
437 }
438 }
439 // Ri = R \ I by back substitution, column by column.
440 Matrix<double> Ri(n, n, 0.0);
441 for (std::size_t c = 0; c < n; ++c)
442 for (std::size_t ii = n; ii-- > 0;) {
443 double s = (ii == c) ? 1.0 : 0.0;
444 for (std::size_t k = ii + 1; k < n; ++k) s -= R(ii, k) * Ri(k, c);
445 Ri(ii, c) = s / R(ii, ii);
446 }
447 std::vector<double> se(n, 0.0);
448 for (std::size_t a = 0; a < n; ++a) {
449 double s = 0.0;
450 for (std::size_t c = 0; c < n; ++c) s += Ri(a, c) * Ri(a, c);
451 se[a] = std::sqrt(s);
452 }
453 return se;
454}
455
456/** `Z(rows, cov) * beta`, one value per row. */
457inline std::vector<double> linpred(const Matrix<double>& Z, const std::vector<std::size_t>& rows,
458 const std::vector<std::size_t>& cov, const double* beta) {
459 std::vector<double> eta(rows.size(), 0.0);
460 for (std::size_t r = 0; r < rows.size(); ++r) {
461 double s = 0.0;
462 for (std::size_t c = 0; c < cov.size(); ++c) s += Z(rows[r], cov[c]) * beta[c];
463 eta[r] = s;
464 }
465 return eta;
466}
467
468/** `fitIntensityArc`: one transition of Approach II, its events `ev` among the sojourns `t`. */
469inline void fit_intensity_arc(const std::string& family, const std::vector<double>& t,
470 const std::vector<char>& ev, const Matrix<double>& Z,
471 const std::vector<std::size_t>& rows,
472 const std::vector<std::size_t>& cov, const SmpOptions& opt,
473 std::vector<double>& th, std::vector<double>& be,
474 std::vector<double>& se, double& ll) {
475 const std::size_t q = cov.size();
476 const std::size_t npar = infer_smp_family_npar(family);
477 std::vector<double> tev;
478 double tsum = 0.0;
479 for (std::size_t k = 0; k < t.size(); ++k) {
480 tsum += t[k];
481 if (ev[k]) tev.push_back(t[k]);
482 }
483 const Objective nll = [&](const std::vector<double>& x) {
484 std::vector<double> eta;
485 if (q > 0) eta = linpred(Z, rows, cov, x.data() + npar);
486 std::vector<double> lf, lS;
487 infer_smp_family_logfs(family, std::vector<double>(x.begin(), x.begin() + npar), t, eta,
488 lf, lS);
489 double sev = 0.0, scen = 0.0;
490 for (std::size_t k = 0; k < t.size(); ++k) {
491 if (ev[k])
492 sev += lf[k];
493 else
494 scen += lS[k];
495 }
496 double v = -(sev + scen);
497 if (!std::isfinite(v)) v = std::numeric_limits<double>::infinity();
498 return v;
499 };
500 std::vector<double> x;
501 if (smp_family_name(family) == "exp" && q == 0) {
502 x.push_back(std::log(static_cast<double>(tev.size()) / tsum));
503 } else {
504 std::vector<double> x0 = infer_smp_family_theta0(family, tev, tsum);
505 x0.resize(npar + q, 0.0);
506 x = nelder_mead(nll, x0, opt);
507 }
508 th.assign(x.begin(), x.begin() + npar);
509 be.assign(x.begin() + npar, x.end());
510 ll = -nll(x);
511 se = hessian_se(nll, x);
512}
513
514/** `expandBeta`: the effects of the selected covariates, zero elsewhere. */
515inline std::vector<double> expand_beta(const std::vector<double>& be,
516 const std::vector<std::size_t>& cov, std::size_t p) {
517 std::vector<double> b(p, 0.0);
518 for (std::size_t c = 0; c < cov.size(); ++c) b[cov[c]] = be[c];
519 return b;
520}
521
522/**
523 * `fitSojournState`: state i of Approach I. The parameter vector is laid out
524 * [logits of destinations 2..m; theta_1; beta_1; theta_2; beta_2; ...].
525 */
526inline void fit_sojourn_state(std::size_t i, const std::vector<std::size_t>& dests,
527 const std::vector<std::vector<std::string>>& fam,
528 const std::vector<double>& t, const std::vector<std::size_t>& to,
529 const Matrix<double>& Z, const std::vector<std::size_t>& rows,
530 const std::vector<std::vector<std::vector<bool>>>& mask,
531 std::size_t p, const SmpOptions& opt, std::vector<SmpArc>& arcs,
532 double& ll, std::size_t& np) {
533 const std::size_t m = dests.size(), n = t.size();
534 std::vector<std::vector<std::size_t>> cov(m);
535 std::vector<std::size_t> npar(m, 0);
536 std::vector<std::string> famk(m);
537 for (std::size_t k = 0; k < m; ++k) {
538 for (std::size_t c = 0; c < p; ++c)
539 if (mask[i - 1][dests[k] - 1][c]) cov[k].push_back(c);
540 famk[k] = fam[i - 1][dests[k] - 1];
541 npar[k] = infer_smp_family_npar(famk[k]);
542 }
543 std::vector<char> cens(n, 0);
544 bool anycens = false;
545 for (std::size_t r = 0; r < n; ++r)
546 if (to[r] == 0) {
547 cens[r] = 1;
548 anycens = true;
549 }
550 std::vector<std::size_t> off(m, 0);
551 std::size_t pos = m - 1;
552 for (std::size_t k = 0; k < m; ++k) {
553 off[k] = pos;
554 pos += npar[k] + cov[k].size();
555 }
556 // Per destination, the rows that ended there.
557 std::vector<std::vector<std::size_t>> evidx(m);
558 for (std::size_t k = 0; k < m; ++k)
559 for (std::size_t r = 0; r < n; ++r)
560 if (to[r] == dests[k]) evidx[k].push_back(r);
561
562 const auto nll_dest = [&](std::size_t k, const std::vector<double>& y) {
563 std::vector<double> tev(evidx[k].size()), eta;
564 std::vector<std::size_t> erows(evidx[k].size());
565 for (std::size_t q = 0; q < evidx[k].size(); ++q) {
566 tev[q] = t[evidx[k][q]];
567 erows[q] = rows[evidx[k][q]];
568 }
569 if (!cov[k].empty()) eta = linpred(Z, erows, cov[k], y.data() + npar[k]);
570 std::vector<double> lf, lS;
571 infer_smp_family_logfs(famk[k], std::vector<double>(y.begin(), y.begin() + npar[k]), tev,
572 eta, lf, lS);
573 double s = 0.0;
574 for (double v : lf) s += v;
575 double v = -s;
576 if (!std::isfinite(v)) v = std::numeric_limits<double>::infinity();
577 return v;
578 };
579 const auto softmax = [&](const std::vector<double>& x) {
580 std::vector<double> l(m, 0.0);
581 for (std::size_t k = 1; k < m; ++k) l[k] = x[k - 1];
582 const double mx = *std::max_element(l.begin(), l.end());
583 double tot = 0.0;
584 for (std::size_t k = 0; k < m; ++k) {
585 l[k] -= mx;
586 tot += std::exp(l[k]);
587 }
588 std::vector<double> pk(m);
589 for (std::size_t k = 0; k < m; ++k) pk[k] = std::exp(l[k]) / tot;
590 return pk;
591 };
592 std::vector<std::size_t> crow;
593 for (std::size_t r = 0; r < n; ++r)
594 if (cens[r]) crow.push_back(r);
595 const Objective nll_state = [&](const std::vector<double>& x) {
596 const std::vector<double> pk = softmax(x);
597 double v = 0.0;
598 std::vector<double> Sc(crow.size(), 0.0);
599 for (std::size_t kk = 0; kk < m; ++kk) {
600 const std::vector<double> th(x.begin() + off[kk], x.begin() + off[kk] + npar[kk]);
601 std::vector<double> eta(n, 0.0);
602 if (!cov[kk].empty()) eta = linpred(Z, rows, cov[kk], x.data() + off[kk] + npar[kk]);
603 std::vector<double> lf, lS;
604 infer_smp_family_logfs(famk[kk], th, t, eta, lf, lS);
605 double s = 0.0;
606 const double lp = std::log(pk[kk]);
607 for (std::size_t r : evidx[kk]) s += lp + lf[r];
608 v = v - s;
609 for (std::size_t q = 0; q < crow.size(); ++q) Sc[q] = Sc[q] + pk[kk] * std::exp(lS[crow[q]]);
610 }
611 double s = 0.0;
612 for (double c : Sc) s += std::log(std::max(c, 1e-300));
613 v = v - s;
614 if (!std::isfinite(v)) v = std::numeric_limits<double>::infinity();
615 return v;
616 };
617
618 std::vector<double> x(pos, 0.0);
619 std::vector<std::size_t> cnt(m, 0);
620 for (std::size_t k = 0; k < m; ++k) {
621 cnt[k] = evidx[k].size();
622 std::vector<double> tev(cnt[k]);
623 double texp = 0.0;
624 for (std::size_t q = 0; q < cnt[k]; ++q) {
625 tev[q] = t[evidx[k][q]];
626 texp += tev[q];
627 }
628 std::vector<double> xk = infer_smp_family_theta0(famk[k], tev, texp);
629 xk.resize(npar[k] + cov[k].size(), 0.0);
630 if (!(smp_family_name(famk[k]) == "exp" && cov[k].empty()))
631 xk = nelder_mead([&](const std::vector<double>& y) { return nll_dest(k, y); }, xk, opt);
632 for (std::size_t q = 0; q < xk.size(); ++q) x[off[k] + q] = xk[q];
633 }
634 for (std::size_t k = 1; k < m; ++k)
635 x[k - 1] = std::log(static_cast<double>(cnt[k]) / static_cast<double>(cnt[0]));
636 if (anycens) x = nelder_mead(nll_state, x, opt);
637 ll = -nll_state(x);
638 np = pos;
639 const std::vector<double> Sfull = hessian_se(nll_state, x);
640 const std::vector<double> pr = softmax(x);
641 for (std::size_t k = 0; k < m; ++k) {
642 const std::size_t b0 = off[k], b1 = off[k] + npar[k] + cov[k].size();
643 SmpArc a;
644 a.from = i;
645 a.to = dests[k];
646 a.family = famk[k];
647 a.theta.assign(x.begin() + off[k], x.begin() + off[k] + npar[k]);
648 a.params = infer_smp_family_natural(famk[k], a.theta);
649 a.beta = expand_beta(std::vector<double>(x.begin() + off[k] + npar[k], x.begin() + b1),
650 cov[k], p);
651 if (anycens) {
652 a.se.assign(Sfull.begin() + b0, Sfull.begin() + b1);
653 } else {
654 a.se = hessian_se([&](const std::vector<double>& y) { return nll_dest(k, y); },
655 std::vector<double>(x.begin() + b0, x.begin() + b1));
656 }
657 a.prob = pr[k];
658 a.nevents = cnt[k];
659 arcs.push_back(a);
660 }
661}
662
663} // namespace smp_infer_detail
664
665/**
666 * `infer_smp(from, to, dur, ...)`: the fit, under `opt.approach`.
667 *
668 * `from` holds states 1..E and `to` states 1..E or 0 for a censored sojourn;
669 * `dur` the sojourn durations. See the file comment for the two approaches.
670 */
671inline SmpFit infer_smp(const std::vector<std::size_t>& from, const std::vector<std::size_t>& to,
672 const std::vector<double>& dur, const SmpOptions& opt = SmpOptions()) {
673 using namespace smp_infer_detail;
674 const std::size_t n = from.size();
675 if (to.size() != n || dur.size() != n || n == 0)
676 throw InputError("infer_smp: FROM, TO and DUR must be nonempty and of equal length.");
677 std::size_t smax = 0;
678 for (std::size_t k = 0; k < n; ++k) {
679 if (from[k] < 1)
680 throw InputError("infer_smp: FROM must hold states 1..E and TO states 1..E, or 0 for "
681 "a censored sojourn.");
682 if (to[k] == from[k])
683 throw InputError("infer_smp: A sojourn cannot end in the state it started from.");
684 if (!std::isfinite(dur[k]) || !(dur[k] > 0.0))
685 throw InputError("infer_smp: Every sojourn duration must be finite and positive.");
686 smax = std::max(smax, std::max(from[k], to[k]));
687 }
688 const std::size_t E = opt.nstates > 0 ? opt.nstates : smax;
689 if (smax > E) throw InputError("infer_smp: A state exceeds 'nstates'.");
690 Matrix<double> Z = opt.covariates;
691 if (Z.rows() == 0 && Z.cols() == 0) Z = Matrix<double>(n, 0, 0.0);
692 if (Z.rows() != n)
693 throw InputError("infer_smp: The covariates must be a finite n x p matrix, one row per "
694 "sojourn.");
695 for (std::size_t r = 0; r < Z.rows(); ++r)
696 for (std::size_t c = 0; c < Z.cols(); ++c)
697 if (!std::isfinite(Z(r, c)))
698 throw InputError("infer_smp: The covariates must be a finite n x p matrix, one row "
699 "per sojourn.");
700 const std::size_t p = Z.cols();
701 std::vector<std::vector<std::vector<bool>>> mask = opt.covmask;
702 if (mask.empty()) {
703 mask.assign(E, std::vector<std::vector<bool>>(E, std::vector<bool>(p, true)));
704 } else if (p > 0) {
705 bool ok = mask.size() == E;
706 for (std::size_t i = 0; ok && i < E; ++i) {
707 ok = mask[i].size() == E;
708 for (std::size_t j = 0; ok && j < E; ++j) ok = mask[i][j].size() == p;
709 }
710 if (!ok) throw InputError("infer_smp: The covariate mask must be E x E x p.");
711 }
712 // familyTable
713 std::vector<std::vector<std::string>> fam(E, std::vector<std::string>(E));
714 if (opt.dist_table.empty()) {
715 for (std::size_t i = 0; i < E; ++i)
716 for (std::size_t j = 0; j < E; ++j) fam[i][j] = smp_family_name(opt.dist);
717 } else {
718 bool ok = opt.dist_table.size() == E;
719 for (std::size_t i = 0; ok && i < E; ++i) ok = opt.dist_table[i].size() == E;
720 if (!ok)
721 throw InputError("infer_smp: The 'dist' option must be a family name or an E x E cell "
722 "of names.");
723 for (std::size_t i = 0; i < E; ++i)
724 for (std::size_t j = 0; j < E; ++j)
725 fam[i][j] = opt.dist_table[i][j].empty() ? std::string("weibull")
726 : smp_family_name(opt.dist_table[i][j]);
727 }
728 for (std::size_t i = 0; i < E; ++i)
729 for (std::size_t j = 0; j < E; ++j) infer_smp_family_npar(fam[i][j]);
730 const std::string approach = smp_family_name(opt.approach);
731 if (approach != "intensity" && approach != "sojourn")
732 throw InputError("infer_smp: The approach must be 'intensity' or 'sojourn'.");
733
734 SmpFit fit;
735 fit.approach = approach;
736 fit.nstates = E;
737 fit.ncov = p;
738 double loglik = 0.0;
739 std::size_t nparams = 0;
740 for (std::size_t i = 1; i <= E; ++i) {
741 std::vector<std::size_t> rows;
742 for (std::size_t k = 0; k < n; ++k)
743 if (from[k] == i) rows.push_back(k);
744 if (rows.empty()) continue;
745 std::vector<std::size_t> dests;
746 for (std::size_t r : rows)
747 if (to[r] > 0) dests.push_back(to[r]);
748 std::sort(dests.begin(), dests.end());
749 dests.erase(std::unique(dests.begin(), dests.end()), dests.end());
750 if (dests.empty()) continue;
751 std::vector<double> t(rows.size());
752 std::vector<std::size_t> tor(rows.size());
753 for (std::size_t q = 0; q < rows.size(); ++q) {
754 t[q] = dur[rows[q]];
755 tor[q] = to[rows[q]];
756 }
757 if (approach == "intensity") {
758 for (std::size_t j : dests) {
759 std::vector<std::size_t> cov;
760 for (std::size_t c = 0; c < p; ++c)
761 if (mask[i - 1][j - 1][c]) cov.push_back(c);
762 std::vector<char> ev(rows.size(), 0);
763 std::size_t nev = 0;
764 for (std::size_t q = 0; q < rows.size(); ++q)
765 if (tor[q] == j) {
766 ev[q] = 1;
767 ++nev;
768 }
769 SmpArc a;
770 std::vector<double> be;
771 double ll = 0.0;
772 fit_intensity_arc(fam[i - 1][j - 1], t, ev, Z, rows, cov, opt, a.theta, be, a.se,
773 ll);
774 a.from = i;
775 a.to = j;
776 a.family = fam[i - 1][j - 1];
778 a.beta = expand_beta(be, cov, p);
779 a.nevents = nev;
780 fit.arcs.push_back(a);
781 loglik += ll;
782 nparams += a.theta.size() + cov.size();
783 }
784 } else {
785 double ll = 0.0;
786 std::size_t np = 0;
787 fit_sojourn_state(i, dests, fam, t, tor, Z, rows, mask, p, opt, fit.arcs, ll, np);
788 loglik += ll;
789 nparams += np;
790 }
791 }
792 fit.loglik = loglik;
793 fit.nparams = nparams;
794 fit.aic = 2.0 * static_cast<double>(nparams) - 2.0 * loglik;
795 fit.P = Matrix<double>(E, E, 0.0);
796 if (approach == "sojourn") {
797 for (const SmpArc& a : fit.arcs) fit.P(a.from - 1, a.to - 1) = a.prob;
798 } else {
799 for (std::size_t i = 1; i <= E; ++i) {
800 std::vector<lang::Distrib<double>> stage(E);
801 bool any = false;
802 for (const SmpArc& a : fit.arcs)
803 if (a.from == i) {
804 stage[a.to - 1] = infer_smp_family_dist(a.family, a.theta, 0.0);
805 any = true;
806 }
807 if (!any) continue;
808 const mc::SmpStageQuad q = mc::smp_stage_quad(stage, std::vector<double>());
809 for (std::size_t j = 0; j < E; ++j) fit.P(i - 1, j) = q.P[j];
810 }
811 for (SmpArc& a : fit.arcs) a.prob = fit.P(a.from - 1, a.to - 1);
812 }
813 return fit;
814}
815
816// ---- infer_smp_env ---------------------------------------------------------
817
818/** Options of `infer_smp_env`. */
820 std::string name = "SMPFit";
821 std::vector<std::string> names; ///< default S1..SE
822 std::vector<std::string> types; ///< default 'stage'
823 std::vector<double> z; ///< covariate vector, default the baseline 0
824};
825
826/**
827 * `infer_smp_env(fit, models, ...)`: the Environment of a fit, `models[e]` the
828 * network of stage e. An 'intensity' fit becomes competing arcs
829 * (`add_transition`), each the fitted latent clock; a 'sojourn' fit becomes
830 * kernel arcs (`add_kernel_transition`) with the fitted embedded probability and
831 * conditional sojourn law. Either is solved by SolverENV method 'smp'.
832 */
834 const std::vector<qn::NetworkStruct<double>>& models,
835 const SmpEnvOptions& opt = SmpEnvOptions()) {
836 const std::size_t E = fit.nstates;
837 if (models.size() != E)
838 throw InputError("infer_smp_env: MODELS must be a cell of " + std::to_string(E) +
839 " networks, one per state.");
840 std::vector<std::string> names = opt.names, types = opt.types;
841 if (names.empty())
842 for (std::size_t e = 1; e <= E; ++e) names.push_back("S" + std::to_string(e));
843 if (types.empty()) types.assign(E, "stage");
844 if (names.size() != E || types.size() != E)
845 throw InputError("infer_smp_env: 'names' and 'types' must have " + std::to_string(E) +
846 " entries.");
847 std::vector<double> z = opt.z;
848 if (z.empty()) z.assign(fit.ncov, 0.0);
849 if (z.size() != fit.ncov)
850 throw InputError("infer_smp_env: 'z' must have " + std::to_string(fit.ncov) +
851 " entries, one per covariate.");
852 env::Environment<double> envm(opt.name, E);
853 for (std::size_t e = 0; e < E; ++e) envm.set_stage(e, names[e], types[e], models[e]);
854 for (const SmpArc& a : fit.arcs) {
855 double eta = 0.0;
856 for (std::size_t c = 0; c < z.size(); ++c) eta += a.beta[c] * z[c];
858 if (fit.approach == "sojourn")
859 envm.add_kernel_transition(a.from - 1, a.to - 1, a.prob, d);
860 else
861 envm.add_transition(a.from - 1, a.to - 1, d);
862 }
863 return envm;
864}
865
866// ---- infer_smp_hazard ------------------------------------------------------
867
868/** Both hazard functions of one transition, rows aligned with the instants asked for. */
869struct SmpHazard {
870 std::size_t from = 0, to = 0;
871 std::vector<double> intensity; ///< a~_ij(t), the transition intensity
872 std::vector<double> sojourn; ///< alpha_ij(t), the hazard of the conditional sojourn law
873 std::vector<double> cumulative; ///< A~_ij(t) = int_0^t a~_ij(s) ds
874 std::vector<double> cif; ///< CIF_ij(t) = P(sojourn <= t, next state j)
875};
876
877/**
878 * `infer_smp_hazard(fit, t, 'z', z)`: both hazard functions of every arc of a
879 * fit on the ascending instants t > 0 (Asanjarani et al., Eqs. 11-15), related
880 * by a~_ij(t) = p_ij f_ij(t) / S_i(t). An 'intensity' fit has a~ and A~ in
881 * closed form and the CIF by `mc::smp_stage_cif`, so alpha_ij =
882 * a~_ij S_i / (p_ij - CIF_ij). A 'sojourn' fit has alpha and the CIF in closed
883 * form, a~ from the relation, and A~ as a midpoint sum on the grid
884 * [0, logspace(log10(tmax)-8, log10(tmax), 2001)] joined with
885 * linspace(0, tmax, 20001) and t. The result is aligned with `fit.arcs`.
886 */
887inline std::vector<SmpHazard> infer_smp_hazard(const SmpFit& fit, const std::vector<double>& t,
888 const std::vector<double>& zin = std::vector<double>()) {
889 std::vector<double> z = zin;
890 if (z.empty()) z.assign(fit.ncov, 0.0);
891 if (z.size() != fit.ncov)
892 throw InputError("infer_smp_hazard: 'z' must have " + std::to_string(fit.ncov) +
893 " entries, one per covariate.");
894 bool ok = !t.empty();
895 for (std::size_t k = 0; ok && k < t.size(); ++k) {
896 if (!std::isfinite(t[k]) || !(t[k] > 0.0)) ok = false;
897 if (k > 0 && !(t[k] - t[k - 1] > 0.0)) ok = false;
898 }
899 if (!ok) throw InputError("infer_smp_hazard: T must be ascending, finite and positive.");
900 const std::size_t E = fit.nstates, nt = t.size();
901 std::vector<SmpHazard> hz(fit.arcs.size());
902 std::vector<std::size_t> froms;
903 for (const SmpArc& a : fit.arcs) froms.push_back(a.from);
904 std::sort(froms.begin(), froms.end());
905 froms.erase(std::unique(froms.begin(), froms.end()), froms.end());
906 const auto eta_of = [&](const SmpArc& a) {
907 double s = 0.0;
908 for (std::size_t c = 0; c < z.size(); ++c) s += a.beta[c] * z[c];
909 return s;
910 };
911 for (std::size_t i : froms) {
912 std::vector<std::size_t> idx;
913 for (std::size_t a = 0; a < fit.arcs.size(); ++a)
914 if (fit.arcs[a].from == i) idx.push_back(a);
915 const std::size_t m = idx.size();
916 std::vector<std::vector<double>> lf(m), lS(m);
917 for (std::size_t k = 0; k < m; ++k) {
918 const SmpArc& a = fit.arcs[idx[k]];
919 infer_smp_family_logfs(a.family, a.theta, t, std::vector<double>{eta_of(a)}, lf[k],
920 lS[k]);
921 }
922 if (fit.approach == "intensity") {
923 std::vector<lang::Distrib<double>> stage(E);
924 for (std::size_t k = 0; k < m; ++k) {
925 const SmpArc& a = fit.arcs[idx[k]];
926 stage[a.to - 1] = infer_smp_family_dist(a.family, a.theta, eta_of(a));
927 }
928 const mc::SmpStageQuad q = mc::smp_stage_quad(stage, std::vector<double>());
929 const mc::SmpStageCif C = mc::smp_stage_cif(stage, std::vector<double>(), t);
930 std::vector<double> Si(nt, 0.0);
931 for (std::size_t j = 0; j < nt; ++j) {
932 double s = 0.0;
933 for (std::size_t k = 0; k < m; ++k) s += lS[k][j];
934 Si[j] = std::exp(s);
935 }
936 for (std::size_t k = 0; k < m; ++k) {
937 const SmpArc& a = fit.arcs[idx[k]];
938 SmpHazard& h = hz[idx[k]];
939 h.from = i;
940 h.to = a.to;
941 h.intensity.resize(nt);
942 h.sojourn.resize(nt);
943 h.cumulative.resize(nt);
944 h.cif.resize(nt);
945 for (std::size_t j = 0; j < nt; ++j) {
946 const double hh = std::exp(lf[k][j] - lS[k][j]);
947 const double cif = C.C(a.to - 1, j);
948 h.intensity[j] = hh;
949 h.sojourn[j] = hh * Si[j] / std::max(q.P[a.to - 1] - cif, 0.0);
950 h.cumulative[j] = -lS[k][j];
951 h.cif[j] = cif;
952 }
953 }
954 } else {
955 std::vector<double> pk(m);
956 for (std::size_t k = 0; k < m; ++k) pk[k] = fit.arcs[idx[k]].prob;
957 std::vector<double> Si(nt, 0.0);
958 for (std::size_t j = 0; j < nt; ++j)
959 for (std::size_t k = 0; k < m; ++k) Si[j] += std::exp(lS[k][j]) * pk[k];
960 const double tmax = t.back();
961 std::vector<double> g(1, 0.0);
962 const std::vector<double> ls =
963 mc::smp_detail::linspace(std::log10(tmax) - 8.0, std::log10(tmax), 2001);
964 for (double v : ls) g.push_back(std::pow(10.0, v));
965 const std::vector<double> lin = mc::smp_detail::linspace(0.0, tmax, 20001);
966 g.insert(g.end(), lin.begin(), lin.end());
967 g.insert(g.end(), t.begin(), t.end());
968 std::sort(g.begin(), g.end());
969 g.erase(std::unique(g.begin(), g.end()), g.end());
970 const std::size_t ng = g.size();
971 std::vector<double> gm(ng - 1);
972 for (std::size_t j = 0; j + 1 < ng; ++j) gm[j] = (g[j] + g[j + 1]) / 2.0;
973 std::vector<std::vector<double>> lfm(m), lSm(m);
974 for (std::size_t k = 0; k < m; ++k) {
975 const SmpArc& a = fit.arcs[idx[k]];
976 infer_smp_family_logfs(a.family, a.theta, gm, std::vector<double>{eta_of(a)},
977 lfm[k], lSm[k]);
978 }
979 std::vector<double> Sim(ng - 1, 0.0);
980 for (std::size_t j = 0; j + 1 < ng; ++j)
981 for (std::size_t k = 0; k < m; ++k) Sim[j] += std::exp(lSm[k][j]) * pk[k];
982 std::vector<std::size_t> at(nt);
983 for (std::size_t j = 0; j < nt; ++j)
984 at[j] = static_cast<std::size_t>(std::lower_bound(g.begin(), g.end(), t[j]) -
985 g.begin());
986 for (std::size_t k = 0; k < m; ++k) {
987 const SmpArc& a = fit.arcs[idx[k]];
988 std::vector<double> cum(ng, 0.0);
989 for (std::size_t j = 0; j + 1 < ng; ++j)
990 cum[j + 1] = cum[j] + pk[k] * std::exp(lfm[k][j]) / Sim[j] * (g[j + 1] - g[j]);
991 SmpHazard& h = hz[idx[k]];
992 h.from = i;
993 h.to = a.to;
994 h.intensity.resize(nt);
995 h.sojourn.resize(nt);
996 h.cumulative.resize(nt);
997 h.cif.resize(nt);
998 for (std::size_t j = 0; j < nt; ++j) {
999 h.intensity[j] = pk[k] * std::exp(lf[k][j]) / Si[j];
1000 h.sojourn[j] = std::exp(lf[k][j] - lS[k][j]);
1001 h.cumulative[j] = cum[at[j]];
1002 h.cif[j] = pk[k] * (1.0 - std::exp(lS[k][j]));
1003 }
1004 }
1005 }
1006 }
1007 return hz;
1008}
1009
1010// ---- infer_smp_nelson_aalen ------------------------------------------------
1011
1012/** One Nelson-Aalen estimate, on the distinct event times of its transition. */
1014 std::size_t from = 0, to = 0;
1015 std::vector<double> time; ///< the distinct event times of i -> j, ascending
1016 std::vector<double> A; ///< the estimate there (right-continuous step function)
1017 std::vector<double> var; ///< its Greenwood-type variance sum d/Y^2
1018};
1019
1020/**
1021 * `infer_smp_nelson_aalen(from, to, dur, 'nstates', E)`: the Nelson-Aalen
1022 * estimate of the cumulative transition intensity of every observed
1023 * transition, on the clock reset at each jump. The risk set at t of i -> j is
1024 * the number of sojourns in i of duration at least t, so this is the
1025 * nonparametric benchmark of the 'intensity' approach (Asanjarani et al.,
1026 * Sec. 4). `nstates` 0 means max(from, to).
1027 */
1028inline std::vector<SmpNelsonAalen> infer_smp_nelson_aalen(const std::vector<std::size_t>& from,
1029 const std::vector<std::size_t>& to,
1030 const std::vector<double>& dur,
1031 std::size_t nstates = 0) {
1032 const std::size_t n = from.size();
1033 if (to.size() != n || dur.size() != n || n == 0)
1034 throw InputError("infer_smp_nelson_aalen: FROM, TO and DUR must be nonempty and of equal "
1035 "length.");
1036 std::size_t smax = 0;
1037 for (std::size_t k = 0; k < n; ++k) {
1038 if (!std::isfinite(dur[k]) || !(dur[k] > 0.0) || from[k] < 1 || to[k] == from[k])
1039 throw InputError("infer_smp_nelson_aalen: Invalid sojourn data: see INFER_SMP for the "
1040 "coding.");
1041 smax = std::max(smax, std::max(from[k], to[k]));
1042 }
1043 const std::size_t E = nstates > 0 ? nstates : smax;
1044 std::vector<SmpNelsonAalen> na;
1045 for (std::size_t i = 1; i <= E; ++i) {
1046 std::vector<double> ti;
1047 std::vector<std::size_t> di;
1048 for (std::size_t k = 0; k < n; ++k)
1049 if (from[k] == i) {
1050 ti.push_back(dur[k]);
1051 di.push_back(to[k]);
1052 }
1053 std::vector<std::size_t> dests;
1054 for (std::size_t d : di)
1055 if (d > 0) dests.push_back(d);
1056 std::sort(dests.begin(), dests.end());
1057 dests.erase(std::unique(dests.begin(), dests.end()), dests.end());
1058 for (std::size_t j : dests) {
1059 std::vector<double> tev;
1060 for (std::size_t q = 0; q < ti.size(); ++q)
1061 if (di[q] == j) tev.push_back(ti[q]);
1062 std::sort(tev.begin(), tev.end());
1063 tev.erase(std::unique(tev.begin(), tev.end()), tev.end());
1065 r.from = i;
1066 r.to = j;
1067 r.time = tev;
1068 r.A.resize(tev.size());
1069 r.var.resize(tev.size());
1070 double a = 0.0, v = 0.0;
1071 for (std::size_t k = 0; k < tev.size(); ++k) {
1072 double d = 0.0, Y = 0.0;
1073 for (std::size_t q = 0; q < ti.size(); ++q) {
1074 if (ti[q] == tev[k] && di[q] == j) d += 1.0;
1075 if (ti[q] >= tev[k]) Y += 1.0;
1076 }
1077 a = a + d / Y;
1078 v = v + d / (Y * Y);
1079 r.A[k] = a;
1080 r.var[k] = v;
1081 }
1082 na.push_back(r);
1083 }
1084 }
1085 return na;
1086}
1087
1088} // namespace infer
1089} // namespace line
1090
1091#endif // LINE_API_INFER_INFER_SMP_H
InputError(const std::string &what)
Definition error.h:105
std::size_t cols() const
Definition matrix.h:90
std::size_t rows() const
Definition matrix.h:89
void add_kernel_transition(std::size_t e, std::size_t h, double prob, const lang::Distrib< T > &d, const ResetMarginal &reset=ResetMarginal())
addKernelTransition: a KERNEL arc of Q(e,h,t) = prob * F(t).
void set_stage(std::size_t e, const std::string &nm, const std::string &type, const qn::NetworkStruct< T > &model)
addStage: name the stage and give it its network.
void add_transition(std::size_t e, std::size_t h, const lang::Distrib< T > &d, const ResetMarginal &reset=ResetMarginal())
addTransition: enable e -> h with a distribution and a reset policy.
A network plus its refreshed NetworkStruct.
A random environment: a port of matlab/src/lang/Environment.m, restricted to what SolverENV reads out...
The exception types the port throws.
Enumerations and the minimal distribution descriptor shared by the model layer of the C++ port.
Dense matrix and non-owning view.
void fit_sojourn_state(std::size_t i, const std::vector< std::size_t > &dests, const std::vector< std::vector< std::string > > &fam, const std::vector< double > &t, const std::vector< std::size_t > &to, const Matrix< double > &Z, const std::vector< std::size_t > &rows, const std::vector< std::vector< std::vector< bool > > > &mask, std::size_t p, const SmpOptions &opt, std::vector< SmpArc > &arcs, double &ll, std::size_t &np)
fitSojournState: state i of Approach I.
Definition infer_smp.h:526
std::vector< double > expand_beta(const std::vector< double > &be, const std::vector< std::size_t > &cov, std::size_t p)
expandBeta: the effects of the selected covariates, zero elsewhere.
Definition infer_smp.h:515
double nanmax(const std::vector< double > &v)
MATLAB's max over a vector: NaNs ignored, NaN when every entry is one.
Definition infer_smp.h:261
std::function< double(const std::vector< double > &)> Objective
Definition infer_smp.h:258
std::vector< double > nelder_mead(const Objective &f, const std::vector< double > &x0, const SmpOptions &opt)
nelderMead: restart nm_run from its own optimum, at most four runs.
Definition infer_smp.h:377
std::vector< double > linpred(const Matrix< double > &Z, const std::vector< std::size_t > &rows, const std::vector< std::size_t > &cov, const double *beta)
Z(rows, cov) * beta, one value per row.
Definition infer_smp.h:457
void fit_intensity_arc(const std::string &family, const std::vector< double > &t, const std::vector< char > &ev, const Matrix< double > &Z, const std::vector< std::size_t > &rows, const std::vector< std::size_t > &cov, const SmpOptions &opt, std::vector< double > &th, std::vector< double > &be, std::vector< double > &se, double &ll)
fitIntensityArc: one transition of Approach II, its events ev among the sojourns t.
Definition infer_smp.h:469
void nm_run(const Objective &f, const std::vector< double > &x0, double tolx, double tolf, std::size_t maxit, std::vector< double > &xb, double &fb)
One run of the reference's Nelder-Mead (nmRun).
Definition infer_smp.h:271
std::vector< double > hessian_se(const Objective &f, const std::vector< double > &x)
hessianSE: central-difference Hessian with steps 1e-4 max(1, |x|), its Cholesky factor H = R'R,...
Definition infer_smp.h:401
std::string smp_family_name(const std::string &f)
Lower-case copy of a family name.
Definition infer_smp.h:72
std::size_t infer_smp_family_npar(const std::string &family)
infer_smp_family(F, 'npar'): the number of parameters, validating the name.
Definition infer_smp.h:80
std::vector< SmpHazard > infer_smp_hazard(const SmpFit &fit, const std::vector< double > &t, const std::vector< double > &zin=std::vector< double >())
infer_smp_hazard(fit, t, 'z', z): both hazard functions of every arc of a fit on the ascending instan...
Definition infer_smp.h:887
std::vector< double > infer_smp_family_natural(const std::string &family, const std::vector< double > &th)
infer_smp_family(F, 'natural', THETA): rate | shape, scale | mu, sigma.
Definition infer_smp.h:140
void infer_smp_family_logfs(const std::string &family, const std::vector< double > &th, const std::vector< double > &t, const std::vector< double > &eta, std::vector< double > &logf, std::vector< double > &logS)
infer_smp_family(F, 'logfs', THETA, T, ETA): log density and log survival at t under the proportional...
Definition infer_smp.h:95
env::Environment< double > infer_smp_env(const SmpFit &fit, const std::vector< qn::NetworkStruct< double > > &models, const SmpEnvOptions &opt=SmpEnvOptions())
infer_smp_env(fit, models, ...): the Environment of a fit, models[e] the network of stage e.
Definition infer_smp.h:833
lang::Distrib< double > infer_smp_family_dist(const std::string &family, const std::vector< double > &th, double eta)
infer_smp_family(F, 'dist', THETA, ETA): the LINE distribution of the law with hazard h0(t) exp(eta).
Definition infer_smp.h:187
std::vector< double > infer_smp_family_theta0(const std::string &family, const std::vector< double > &tev, double texp)
infer_smp_family(F, 'theta0', TEV, TEXP): the starting point from the event times tev and the total e...
Definition infer_smp.h:154
SmpFit infer_smp(const std::vector< std::size_t > &from, const std::vector< std::size_t > &to, const std::vector< double > &dur, const SmpOptions &opt=SmpOptions())
infer_smp(from, to, dur, ...): the fit, under opt.approach.
Definition infer_smp.h:671
std::vector< SmpNelsonAalen > infer_smp_nelson_aalen(const std::vector< std::size_t > &from, const std::vector< std::size_t > &to, const std::vector< double > &dur, std::size_t nstates=0)
infer_smp_nelson_aalen(from, to, dur, 'nstates', E): the Nelson-Aalen estimate of the cumulative tran...
Definition infer_smp.h:1028
std::vector< double > linspace(double a, double b, std::size_t n)
linspace(a, b, n), with the last point exactly b.
SmpStageQuad smp_stage_quad(const std::vector< lang::Distrib< T > > &arcs, const std::vector< double > &probs)
SmpStageCif smp_stage_cif(const std::vector< lang::Distrib< T > > &arcs, const std::vector< double > &probs, const std::vector< double > &t)
CIF and holding-time CDF of a stage at the ascending instants t.
Conservation laws of a layered queueing network, enumerated from its structure.
Definition aoi_dist2ph.h:52
A queueing network and its refreshed NetworkStruct.
Closed networks with non-preemptive shortest-job-next (SJN/SJF) stations.
Cumulative incidence functions of one semi-Markov stage.
Embedded jump probabilities and mean holding time of a semi-Markov stage.
One fitted transition.
Definition infer_smp.h:232
std::size_t from
Definition infer_smp.h:233
std::vector< double > beta
p covariate effects, zero where masked out
Definition infer_smp.h:237
std::string family
Definition infer_smp.h:234
std::vector< double > theta
unconstrained parameters
Definition infer_smp.h:235
std::vector< double > se
standard errors of [theta; beta]
Definition infer_smp.h:238
std::size_t nevents
Definition infer_smp.h:241
std::vector< double > params
natural parameters
Definition infer_smp.h:236
double prob
Approach I: p_ij; Approach II: the embedded probability at z = 0.
Definition infer_smp.h:240
Options of infer_smp_env.
Definition infer_smp.h:819
std::vector< std::string > types
default 'stage'
Definition infer_smp.h:822
std::vector< std::string > names
default S1..SE
Definition infer_smp.h:821
std::vector< double > z
covariate vector, default the baseline 0
Definition infer_smp.h:823
What infer_smp returns.
Definition infer_smp.h:245
std::size_t nstates
Definition infer_smp.h:247
std::size_t nparams
Definition infer_smp.h:251
std::string approach
Definition infer_smp.h:246
Matrix< double > P
E x E embedded jump chain at z = 0.
Definition infer_smp.h:253
std::vector< SmpArc > arcs
Definition infer_smp.h:249
std::size_t ncov
Definition infer_smp.h:248
Both hazard functions of one transition, rows aligned with the instants asked for.
Definition infer_smp.h:869
std::vector< double > sojourn
alpha_ij(t), the hazard of the conditional sojourn law
Definition infer_smp.h:872
std::vector< double > cumulative
A~_ij(t) = int_0^t a~_ij(s) ds.
Definition infer_smp.h:873
std::vector< double > intensity
a~_ij(t), the transition intensity
Definition infer_smp.h:871
std::vector< double > cif
CIF_ij(t) = P(sojourn <= t, next state j).
Definition infer_smp.h:874
One Nelson-Aalen estimate, on the distinct event times of its transition.
Definition infer_smp.h:1013
std::vector< double > time
the distinct event times of i -> j, ascending
Definition infer_smp.h:1015
std::vector< double > var
its Greenwood-type variance sum d/Y^2
Definition infer_smp.h:1017
std::vector< double > A
the estimate there (right-continuous step function)
Definition infer_smp.h:1016
Options of infer_smp, the reference's name/value pairs.
Definition infer_smp.h:206
std::vector< std::vector< std::string > > dist_table
Per transition, dist_table[i][j] for i -> j (0-based states), when not empty; it must then be E x E,...
Definition infer_smp.h:215
std::string dist
The family of every transition: 'weibull' (default), 'exp', 'gamma' or 'lognormal'.
Definition infer_smp.h:210
std::vector< std::vector< std::vector< bool > > > covmask
covmask[i][j][c]: whether covariate c enters transition i -> j (0-based states); empty means all,...
Definition infer_smp.h:222
Matrix< double > covariates
n x p covariates Z, one row per sojourn; an empty matrix means none.
Definition infer_smp.h:217
std::size_t maxiter
Nelder-Mead iteration cap per run; 0 means 2000 per parameter.
Definition infer_smp.h:228
std::string approach
'intensity' (Approach II, default) or 'sojourn' (Approach I).
Definition infer_smp.h:208
std::size_t nstates
E; 0 means max(from, to).
Definition infer_smp.h:224
static Distrib exp_rate(const T &r)
Definition lang_types.h:955
static Distrib weibull(const T &scale, const T &shape)
static Distrib gamma_dist(const T &shape, const T &scale)
Gamma(shape, scale), Weibull(scale, shape) and Lognormal(mu, sigma).
static Distrib lognormal(const T &logmean, const T &logsigma)
What smp_stage_cif returns: C(h, j) = CIF_h(t_j) and Fhold[j].
Matrix< double > C
What smp_stage_quad returns.
std::vector< double > P
the embedded jump probabilities, one per arc