The NestedScale object

The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

This vignette takes one object apart. The example is a small fuel classification with every awkward feature: a code reused at two frames, a frame that cross-cuts another, an atom with no code at one frame, and attributes stored at three different levels.

lt <- data.frame(
  group   = c("Fossil", "Fossil", "Fossil", "Fossil", "Fossil",
              "Bio", "Bio", "Bio", "Power"),
  phase   = c("solid", "solid", "liquid", "liquid", "gas",
              "solid", "liquid", "gas", NA),
  carrier = c("Coal", "Coal", "Oil", "Oil", "Gas",
              "Biomass", "Biomass", "Biomass", "Electricity"),
  fuel    = c("Anthracite", "Lignite", "Diesel", "Gasoline", "NaturalGas",
              "Wood", "Biodiesel", "Biogas", "Electricity")
)
at <- data.frame(
  frame     = c("fuel", "fuel", "carrier", "group", "fuel", "carrier"),
  unit      = c("Anthracite", "Lignite", "Oil", "Bio", "Anthracite", "Gas"),
  attribute = c("lhv", "lhv", "lhv", "lhv", "ef_co2", "ef_co2"),
  kind      = c(rep("property", 4), "factor", "factor"),
  value     = c(26.7, 11.9, 42.5, 15.0, 98300, 56100),
  unit_of_measure = c(rep("GJ/t", 4), "kg/TJ", "kg/TJ"),
  source    = "example"
)
al <- data.frame(
  frame  = c("fuel", "carrier", "fuel", "carrier", "carrier"),
  unit   = c("Anthracite", "Coal", "Electricity", "Electricity", "Electricity"),
  alias  = c("hard coal", "coal", "elc", "elc", "power"),
  source = "example"
)
x <- scale_from_leaftable(
  lt, frames = c("group", "phase", "carrier", "fuel"), key = "fuel",
  attributes = at, aliases = al, name = "fuels"
)
x
#> NestedScale: fuels
#> Frames (4, coarsest first):
#>   - group (3)
#>     - phase (3)  [1 unit unassigned]
#>       - carrier (5)
#>         - fuel (9)
#> Atoms: 9
#> Attributes: lhv, ef_co2 (sources: example) -- 1 at group, 2 at carrier, 3 at fuel
#> Aliases: 5 (example)

The object

A NestedScale is an S7 object with seven properties:

S7::prop_names(x)
#> [1] "leaftable"  "frames"     "members"    "key"        "meta"      
#> [6] "attributes" "aliases"

property	type	holds
`leaftable`	`data.frame`	one row per atom: one column per frame, the key column, weight columns
`frames`	`character`	the hierarchy, ordered coarsest first
`members`	named `list`	the ordered code vocabulary at each frame
`key`	`character`	the name of the atom column (`"unit"` by default)
`meta`	`list`	`name`, `desc`, `weights`, `default_weight`, and whatever the builder adds
`attributes`	`data.frame`	values keyed by `(frame, unit)`: `attribute`, `kind`, `value`, `lower`, `upper`, `unit_of_measure`, `source`, `note`
`aliases`	`data.frame`	alternative labels keyed by `(frame, unit)`: `alias`, `source`

Everything else – parent/child tables, ancestry, shares, resolved attributes – is derived on demand; nothing is cached on the object. The two tables are empty unless the builder fills them; a scale saved by an earlier version, which had no such properties, loads and validates unchanged.

The leaftable

scale_leaftable(x)
#>    group  phase     carrier        fuel
#> 1 Fossil  solid        Coal  Anthracite
#> 2 Fossil  solid        Coal     Lignite
#> 3 Fossil liquid         Oil      Diesel
#> 4 Fossil liquid         Oil    Gasoline
#> 5 Fossil    gas         Gas  NaturalGas
#> 6    Bio  solid     Biomass        Wood
#> 7    Bio liquid     Biomass   Biodiesel
#> 8    Bio    gas     Biomass      Biogas
#> 9  Power   <NA> Electricity Electricity

Each row is one atom. The key column (fuel here) is unique and complete. Frame columns may hold NA: Electricity has no physical state, so it does not participate at the phase frame, and na_action= decides what recasts do about it. The key may share its name with the finest frame, as here and in the shipped carriers: there the frame column is the key column.

Frames, members, ranks

scale_frames(x)
#> [1] "group"   "phase"   "carrier" "fuel"
scale_atom_level(x)              # the finest frame, or the key
#> [1] "fuel"
scale_units(x, "carrier")        # the members at one frame
#> [1] "Coal"        "Oil"         "Gas"         "Biomass"     "Electricity"
scale_units(x)                   # the atoms
#> [1] "Anthracite"  "Lignite"     "Diesel"      "Gasoline"    "NaturalGas" 
#> [6] "Wood"        "Biodiesel"   "Biogas"      "Electricity"
scale_rank(x, "carrier")
#> [1] 3

Codes are not unique across frames: Electricity is a group and a carrier here, and Nuclear, Heat, Storage, Other are each a group and a carrier in carriers. Every function that takes a code therefore takes its frame as well.

Navigation is derived from the leaftable and routed through the atoms, so a frame that cross-cuts another reports overlap rather than invented containment:

scale_nests(x, "carrier", "fuel")    # a tree
#> [1] TRUE
scale_nests(x, "phase", "carrier")   # Biomass is solid, liquid and gas
#> [1] FALSE
#> attr(,"offenders")
#> [1] "Biomass"
scale_parents(x, "carrier", "Biomass", target = "phase")
#> [1] "solid"  "liquid" "gas"
scale_ancestors(x, "fuel", "Biogas")
#>     frame    unit
#> 1   group     Bio
#> 2   phase     gas
#> 3 carrier Biomass
scale_descendants(x, "group", "Fossil")
#>      frame       unit
#> 1    phase      solid
#> 2    phase     liquid
#> 3    phase        gas
#> 4  carrier       Coal
#> 5  carrier        Oil
#> 6  carrier        Gas
#> 7     fuel Anthracite
#> 8     fuel    Lignite
#> 9     fuel     Diesel
#> 10    fuel   Gasoline
#> 11    fuel NaturalGas
head(scale_ancestry(x), 4)
#>   parent_frame parent child_frame   child
#> 1        group    Bio       phase     gas
#> 2        group    Bio       phase  liquid
#> 3        group    Bio       phase   solid
#> 4        group    Bio     carrier Biomass

The attribute table

Physical attributes live in one long table, and a row may sit on any member, not only on an atom:

scale_attributes(x)
#>     frame       unit attribute     kind   value lower upper unit_of_measure
#> 1    fuel Anthracite       lhv property    26.7    NA    NA            GJ/t
#> 2    fuel    Lignite       lhv property    11.9    NA    NA            GJ/t
#> 3 carrier        Oil       lhv property    42.5    NA    NA            GJ/t
#> 4   group        Bio       lhv property    15.0    NA    NA            GJ/t
#> 5    fuel Anthracite    ef_co2   factor 98300.0    NA    NA           kg/TJ
#> 6 carrier        Gas    ef_co2   factor 56100.0    NA    NA           kg/TJ
#>    source note
#> 1 example <NA>
#> 2 example <NA>
#> 3 example <NA>
#> 4 example <NA>
#> 5 example <NA>
#> 6 example <NA>

Reading with a frame resolves one value per member, walking to the nearest coarser frame when a member has none of its own, and says where the value came from:

scale_attributes(x, "lhv", frame = "fuel")[
  c("unit", "value", "unit_of_measure", "origin_frame", "origin_unit")]
#>          unit value unit_of_measure origin_frame origin_unit
#> 1  Anthracite  26.7            GJ/t         fuel  Anthracite
#> 2     Lignite  11.9            GJ/t         fuel     Lignite
#> 3      Diesel  42.5            GJ/t      carrier         Oil
#> 4    Gasoline  42.5            GJ/t      carrier         Oil
#> 5  NaturalGas    NA            <NA>         <NA>        <NA>
#> 6        Wood  15.0            GJ/t        group         Bio
#> 7   Biodiesel  15.0            GJ/t        group         Bio
#> 8      Biogas  15.0            GJ/t        group         Bio
#> 9 Electricity    NA            <NA>         <NA>        <NA>

Attributes are not weights. A weight says how big an atom is and drives aggregation; an attribute is a per-member conversion factor and is consumed only by convert_scale(). The constructor promotes numeric leaftable columns to weights, so a calorific value must be passed as attributes=, never as a leaftable column. Every attribute name carries a kind: property (an invertible ratio) or factor (one-way).

SCALE_ATTRIBUTE_KINDS
#> [1] "property" "factor"

vignette("attributes") has the reasoning and the full account of inheritance, sources and conversion.

The alias table

scale_aliases(x)
#>     frame        unit     alias  source
#> 1    fuel  Anthracite hard coal example
#> 2 carrier        Coal      coal example
#> 3    fuel Electricity       elc example
#> 4 carrier Electricity       elc example
#> 5 carrier Electricity     power example
scale_match(x, c("hard coal", "elc", "Lignite"))
#>       label frame       unit    method                            candidates
#> 1 hard coal  fuel Anthracite     alias                                  <NA>
#> 2       elc  <NA>       <NA> ambiguous fuel/Electricity; carrier/Electricity
#> 3   Lignite  fuel    Lignite     exact                                  <NA>

One alias may point at several members (elc above); matching reports that as ambiguous unless frame= narrows it. Matching is exact and case-insensitive, with no fuzzy stage.

Weights and meta

scale_weights(x)                        # none declared here
#> character(0)
S7::prop(x, "meta")[c("name", "desc")]
#> $name
#> [1] "fuels"
#> 
#> $desc
#> [1] ""
carriers <- read_scale("carriers")
scale_weights(carriers)
#> character(0)
S7::prop(carriers, "meta")[c("name", "source", "palette_version")]
#> $name
#> [1] "carriers"
#> 
#> $source
#> [1] "energypal carriers.yml"
#> 
#> $palette_version
#> [1] "0.1.1"
head(S7::prop(carriers, "meta")$labels, 4)
#>     frame        unit label_short   label_long
#> 1   group FossilFuels      Fossil Fossil Fuels
#> 2 carrier  FossilCoal        Coal         Coal
#> 3 carrier   FossilOil         Oil          Oil
#> 4 carrier   FossilGas         Gas  Natural Gas

meta$name is load-bearing: join_scale() prefixes the columns it attaches with "<name>.", scale_map_between() names its label columns after the objects, and write_scale() needs a name for the definition. The shipped carriers keeps energypal’s short and long labels in meta$labels; colours are not stored.

What the validator guarantees

scale_from_leaftable() and every operation that rebuilds a scale enforce:

the key column is present, unique and complete; every frame is a leaftable column and members are exactly its non-NA values, in a stable order;
weights are numeric and finite;
every attribute and alias row names a (frame, unit) that is a member; value is finite; lower <= value <= upper where given; kind is one of SCALE_ATTRIBUTE_KINDS and one attribute name has one kind; unit_of_measure and source are non-empty; no duplicate (frame, unit, attribute, source) rows and no duplicate alias rows.

The order of frames is a convention, coarsest first; the direction-dependent rules (share, residuals, reconciliation) follow it.

Subsetting keeps the tables consistent: filter_scale() and [ keep the attribute and alias rows of every surviving member at every frame, and prune_scale() keeps the rows at the frames that survive the collapse.

fossil <- x["group", "Fossil"]
scale_attributes(fossil)[c("frame", "unit", "attribute", "value")]
#>     frame       unit attribute   value
#> 1    fuel Anthracite       lhv    26.7
#> 2    fuel    Lignite       lhv    11.9
#> 3 carrier        Oil       lhv    42.5
#> 4    fuel Anthracite    ef_co2 98300.0
#> 5 carrier        Gas    ef_co2 56100.0

prune_scale() needs the frames above the new atom level to nest, so the cross-cutting phase has to go first:

tree <- scale_from_leaftable(
  lt[c("group", "carrier", "fuel")], frames = c("group", "carrier", "fuel"),
  key = "fuel", attributes = at, aliases = al, name = "tree"
)
scale_aliases(prune_scale(tree, "carrier"))
#>     frame        unit alias  source
#> 1 carrier        Coal  coal example
#> 2 carrier Electricity   elc example
#> 3 carrier Electricity power example

Definition files

A scale is stored as a definition, not as an R object: a JSON header with the three tables inline, or, for large scales, a <name>.scale/ bundle directory holding the header and the tables as Arrow IPC or CSV files. The Python binding reads and writes the same files.

path <- file.path(tempdir(), "fuels.json")
write_scale(x, path)
cat(head(readLines(path), 12), sep = "\n")
#> {
#>   "schema_version": 1,
#>   "nestedscales": "0.2.0",
#>   "class": "nestedscales::NestedScale",
#>   "name": "fuels",
#>   "key": "fuel",
#>   "frames": ["group", "phase", "carrier", "fuel"],
#>   "members": {
#>     "group": ["Fossil", "Bio", "Power"],
#>     "phase": ["solid", "liquid", "gas"],
#>     "carrier": ["Coal", "Oil", "Gas", "Biomass", "Electricity"],
#>     "fuel": ["Anthracite", "Lignite", "Diesel", "Gasoline", "NaturalGas", "Wood", "Biodiesel", "Biogas", "Electricity"]
back <- read_scale(path)
identical(scale_frames(back), scale_frames(x))
#> [1] TRUE
identical(scale_attributes(back), scale_attributes(x))
#> [1] TRUE

write_scale(format = "yaml") writes the same header as YAML, and read_scale() accepts either. Installed packages register directories of definitions; scale_catalog() lists them and read_scale(id) opens one by its id:

scale_catalog()[c("id", "name", "kind", "frames", "n_atoms", "package")]
#>         id     name        kind             frames n_atoms      package
#> 1 carriers carriers NestedScale group/carrier/fuel      71 nestedscales

The registry

One package-level registry changes behaviour explicitly. Rules map value-column names to their aggregation rule; there is no fallback, and an unregistered column without rule= is an error:

SCALE_RULES
#> [1] "sum"           "weighted_mean" "mean"          "copy"         
#> [5] "sd"            "share"         "logshare"
register_scale_rule("tj", "sum")
register_scale_rule("price", "weighted_mean", weight = "tj")
get_scale_rule("price")
#> $rule
#> [1] "weighted_mean"
#> 
#> $weight
#> [1] "tj"
#> 
#> $scope
#> NULL
list_scale_rules()
#>   param          rule weight scope
#> 1 price weighted_mean     tj  <NA>
#> 2    tj           sum   <NA>  <NA>
clear_scale_rules()

A rule may be scoped to one dimension with scope=, so a model’s tj column aggregates the same way over fuels, regions and months without being registered three times – or differently, when it should.

Where to next?

vignette("nestedscales") – the tour of the main functions.
vignette("attributes") – attributes, aliases and conversion on the shipped carriers.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.