Port of matlab/src/io/WfCommonsLoader.m (and jline.io.WfCommonsLoader / WfCommonsOptions): a WfCommons workflow trace (https://github.com/wfcommons/workflow-schema, schema 1.3 to 1.5) read into a workflow::Workflow<T>. More...

#include <algorithm>
#include <cctype>
#include <cmath>
#include <cstdlib>
#include <deque>
#include <fstream>
#include <iostream>
#include <iterator>
#include <map>
#include <sstream>
#include <string>
#include <vector>
#include "json.hpp"
#include "line/lang/dist_fitters.h"
#include "line/lang/lang_types.h"
#include "line/lang/workflow/workflow.h"
#include "line/num/number.h"
#include "line/util/error.h"
#include "line/util/http.h"
#include "line/util/subprocess.h"
#include "line/util/tempdir.h"

Include dependency graph for wfcommons_loader.h:

Classes
struct	line::io::WfCommonsOptions
	The loader options; the defaults are the reference's parseOptions. More...

Namespaces
namespace	line
	Conservation laws of a layered queueing network, enumerated from its structure.
namespace	line::io
namespace	line::io::wfcommons_detail

Typedefs
using	line::io::wfcommons_detail::json = nlohmann::json

Enumerations
enum class	line::io::WfDistributionType { line::io::EXP , line::io::DET , line::io::APH , line::io::HYPEREXP }
	options.distributionType, the JAR's WfCommonsOptions.DistributionType. More...

Functions
WfDistributionType	line::io::wf_distribution_type_from_string (const std::string &s)
	lower(options.distributionType) as the reference switches on it: exp, det, aph, hyperexp.
bool	line::io::wfcommons_detail::supported_version (const std::string &v)
	WfCommonsLoader.SUPPORTED_SCHEMA_VERSIONS.
bool	line::io::wfcommons_detail::has (const json &o, const char *key)
bool	line::io::wfcommons_detail::nonempty (const json &v)
	~isempty(x) on a decoded JSON value.
std::string	line::io::wfcommons_detail::as_text (const json &v)
	A JSON scalar as the text strcmp compares: strings verbatim, numbers as written.
void	line::io::wfcommons_detail::validate_schema (const json &data)
	Port of validateSchema.
std::string	line::io::wfcommons_detail::file_stem (const std::string &s)
	[~, name, ~] = fileparts(s).
std::string	line::io::wfcommons_detail::extract_name (const json &data, const std::string &default_name)
	Port of extractName: the document's name, else the default's stem, sanitized.
std::string	line::io::wfcommons_detail::task_id (const json &task, std::size_t idx1)
	Port of getTaskId.
template<class T>
lang::Distrib< T >	line::io::wfcommons_detail::fit_distribution (double runtime, const WfCommonsOptions &opt)
	Port of fitDistribution.
std::map< std::string, std::string >	line::io::wfcommons_detail::extract_metadata (const json &task, const std::string &id, const json *exec)
	Port of extractMetadata; values are the JSON text of each field.
std::vector< std::size_t >	line::io::wfcommons_detail::reachable_nodes (std::size_t start, const std::vector< std::vector< std::size_t > > &adj)
	Port of getReachableNodes: BFS order, the start excluded.
std::size_t	line::io::wfcommons_detail::find_common_join (const std::vector< std::size_t > &children, const std::vector< std::vector< std::size_t > > &adj, const std::vector< std::size_t > &indeg)
	Port of findCommonJoin.
template<class T>
workflow::Workflow< T >	line::io::wfcommons_detail::build_workflow (const json &data, const std::string &wf_name, const WfCommonsOptions &opt)
	Port of buildWorkflow (with buildAdjacency and addPrecedences).
json	line::io::wfcommons_detail::parse_file (const std::string &path)
	Parse a whole file as JSON, naming the file on failure.
template<class T>
workflow::Workflow< T >	line::io::wfcommons_load_from_json (const nlohmann::json &data, const WfCommonsOptions &options=WfCommonsOptions())
	Port of WfCommonsLoader.loadFromStruct(data, options), on a parsed document.
template<class T>
workflow::Workflow< T >	line::io::wfcommons_load (const std::string &path, const WfCommonsOptions &options=WfCommonsOptions())
	Port of WfCommonsLoader.load(jsonFile, options).
std::string	line::io::wfcommons_detail::which_on_path (const std::string &name)
	The first executable name on PATH, empty when there is none.
std::string	line::io::wfcommons_detail::fetch_https (const std::string &url, int timeout_ms)
	GET an https:// URL with curl, else wget, into memory.
template<class T>
workflow::Workflow< T >	line::io::wfcommons_load_from_url (const std::string &url, const WfCommonsOptions &options=WfCommonsOptions(), int timeout_ms=60000)
	Port of WfCommonsLoader.loadFromUrl(urlString, options).
bool	line::io::wfcommons_validate (const std::string &path)
	Port of WfCommonsLoader.validateFile(jsonFile): true when the file parses and passes the schema checks.

Detailed Description

Port of matlab/src/io/WfCommonsLoader.m (and jline.io.WfCommonsLoader / WfCommonsOptions): a WfCommons workflow trace (https://github.com/wfcommons/workflow-schema, schema 1.3 to 1.5) read into a workflow::Workflow<T>.

WHAT IS BUILT. One activity per task, named by the task's id (its name when there is no id, task_<i> when neither), with a host demand fitted to the task's runtimeInSeconds from the execution data: embedded in the task for schema 1.4 and earlier, under workflow.execution.tasks for 1.5+. The edges are the union of the children (task -> child) and parents (parent -> task) lists, each pair once and children-derived edges first, so a trace that lists parents only (Montage dss) gives the same graph. They become precedences in two passes, exactly as the reference: a task with several children whose children ALL lead directly to one common descendant becomes an AND-fork / AND-join pair, and every edge left over becomes a serial precedence.

CHILDREN THAT NAME A TASK BY name WHEN THE TASKS CARRY AN id. The Pegasus traces of schema 1.4 do this: task mProject_ID0000001 has "id": "ID0000001" and lists its children as "mBackground_ID0000013". The reference keys the lookup on the id alone, so EVERY edge of such a trace is dropped without a word and the workflow reduces to its tasks in series (the JAR on montage-chameleon-2mass-005d-001.json: 58 activities, 0 precedences). WfCommonsOptions::resolve_children_by_name (default true) looks a child that matches no id up among the task names; a child matching neither is dropped, as in the reference. Parent references resolve by the same rule, and one warning counts the unresolved references of both kinds. Set it false for the id-only lookup. On a document whose children are ids, which is every 1.5 document, the two agree.

loadFromUrl fetches http:// over line/util/http.h, the port's own HTTP/1.1 client, which has no TLS and follows no redirects. An https:// URL, which is how the WfCommons repositories are served and what MATLAB's webread takes, is handed to curl (else wget) on PATH, as SolverJMT fetches JMT.jar: no TLS library becomes a build dependency, and a host with neither tool is told so by name. The fetch stays on https across redirects.

WARNINGS go to stderr with the [LINE] Warning: prefix the io layer uses; the port has no line_warning channel.

Definition in file wfcommons_loader.h.

Classes

Namespaces

Typedefs

Enumerations

Functions

Detailed Description