The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.
Nothing is written outside tempdir() unless the user
asks. The Chicago full-dataset cache lives under tempdir()
by default and moves to a persistent directory only through
options(rmoriedata.cache_dir = );
clear_chicago_cache() empties it.
load_chicago_data(as = "parquet_path") also writes under
tempdir(): it wrote the bundled sample into the user’s
R_user_dir() cache, and a bounded full = TRUE
fetch could land on the full-dataset cache path. The schema names the
language column of siu_directors_reports and
siu_drid_manifest _language, as the files do;
it carried the X_language that check.names had
made of it, so the typed store and load_siu_reports()
disagreed on the name. load_siu_reports() documents that it
returns the corpus as text ("" for empty cells) while
morie_data_load("siu_directors_reports") returns the typed
frame.
Every bundled table ships as Parquet again, next to its CSV
(inst/extdata/parquet/<slug>.parquet, plus
_catalog.parquet and
siu_directors_reports_corpus.parquet).
morie_data_load(format = "parquet") and
load_siu_reports(format = "parquet") read those copies; the
new morie_data_path(slug, format) returns the verified path
of either copy, which is the bridge to
pandas.read_parquet(). The catalog gains a
parquet_path column and the signed manifest covers the new
files.
The Parquet writer no longer mangles non-ASCII text in a C locale
(enc2utf8() on an unmarked string turned each byte into its
<c3><a9> display form, and
load_chicago_data(full = TRUE) wrote that into the
cross-session cache). load_siu_reports() reads its CSV as
UTF-8 like the store does, so both copies agree in every locale; a data
page v2 file is reported as such before any decompression; and CI runs
the check in a C locale. Every CSV reader in the package now declares
UTF-8, so a live Chicago fetch and its cached copy are
identical() in any locale. A string that is not valid UTF-8
and carries no encoding mark is refused by column name (a Parquet string
is UTF-8; escaping lost the data, passing the bytes through made a file
no other reader opens), and duplicate column names are refused on write
and kept apart on read. The cross-session Chicago cache carries a
SHA-256 sidecar checked on every read.
load_cihi_data_tables() rejects an
archived_only that is not TRUE or FALSE instead of ignoring
it.
morie_data_checksums() gains a path
column relative to the extdata root; the bare file
basenames it returned were not unique and 20 of them did not resolve
from the root (morie_data_verify() already returned
path).
ask() quotes every argument it hands to the
rmorie CLI. The preamble contains a parenthesis, so with
the CLI installed every call had died in the shell before the binary
ran; a ; in the question would have run as a command.
model and backend are validated, and a blank
or NA question is rejected like "" and
NULL. A stub binary on PATH now exercises the
branch the tests never reached.
morie_data_load() on a dictionary slug says to use
morie_data_dictionary() instead of pointing back at the
catalog.
load_chicago_data(full = TRUE) no longer caches a
response that is not the dataset. A 200 carrying an outage page, a
header-only body or an export shorter than the row count the service
reports is refused with the reason; before, an outage was written to the
cross-session cache and every later session read the empty frame back.
The cache is written to a temporary file and renamed, so a concurrent
reader never sees a half-written file; a cache that cannot be read is
discarded and refetched; and a new refresh = TRUE argument
refetches on demand. When the service’s count endpoint is down the data
is returned but not cached across sessions.
fetch_cihi_table() checks the downloaded file’s size
and signature against the catalogue’s declared format and validates
which (a whole number within the catalogue, or a non-empty
title substring) and timeout before any request. A live URL
that answers 200 with an outage page now falls through to the Wayback
copy.
The verifier pins the signing key’s XMSS root and public seed in package code. A store re-signed with another key, even with a consistent manifest and shipped public key, no longer verifies; the signature now means more than a checksum against local tampering.
CSV tables are read with encoding = "UTF-8" instead
of fileEncoding = "UTF-8-BOM": in a C locale the latter
truncated 22 of the 99 tables (5 to zero rows); the bytes are now left
alone and marked as UTF-8 in every locale.
The data store is signed. _checksums.csv lists every
shipped file with its SHA-256, and the manifest’s own SHA-256 carries an
XMSS (RFC 8391, SHA-256) signature whose public key ships as
_signing_key.json. The signature is checked once per
session and each file when first read, so a modified or corrupted
install errors instead of returning altered data;
morie_data_verify() checks the whole store at once. Needs
rmoriebricklayer 0.4.0 or newer for the verification.
load_siu_reports(format = "parquet") pointed at the
parquet copy this release removed; the option now errors with a note
that the gzip CSV is the same corpus. rmorie reads the corpus through
the default "csv".
morie_dp_gaussian_mean() now rejects infinite
lower or upper with the error its message
already promised, instead of returning NaN with a warning
from rnorm().
The data store ships every table once.
morie_data_load() reads the table’s CSV (the file rmorie
reads by name) and applies a bundled column schema, so each table comes
back with exactly the names and classes the retired parquet copies had
(verified identical for all 99 tables), and keeps it in a session cache:
repeated loads are instant. The parquet directory, a duplicate
samples/ directory and a second copy of the SIU reports are
gone, which takes the source tarball from 6.8 MB to under the 5 MB CRAN
cap. The ten Victoria tables that existed only as parquet now ship as
CSV under vic/. morie_data_catalog() has
n_rows and n_cols for every table;
morie_data_dictionary() returns the shipped JSON.
Documentation is generated with markdown roxygen; backticks in
the help pages are now \code{} (they rendered as stray
opening quotes in the PDF manual).
The narrative corpus behind morie_describe() was
excluded from the 0.3.1 tarball to get this package under 5 MB, which
left rmorie carrying the only shipped copy. rmorie has no room for it
either, so the corpus is back where it belongs, and both packages sit in
the 5-10 MB range CRAN accepts with a written justification.
The package shipped 9.2 MB, and most of the excess was duplication:
rmoriedata.sqlite (15 MB uncompressed) was retired by
the SQLite-to-Parquet migration and no shipped code reads it. Kept in
the repository, excluded from the tarball..csv.gz. load_siu_reports(format = "parquet")
now reads the store copy that morie_data_load() already
serves, so the top-level duplicate is gone. The CSV remains: it is the
default format and a genuinely different encoding, not a third copy of
the same one.describe_corpus.Rds (1.6 MB, and it does not compress)
is referenced by no R code, Rd page or test. Excluded.Verified after the change: the Parquet path, the CSV path and the
siu_directors_reports store slug all return the same 5,157
x 65 corpus.
nanoparquet is gone from Imports. The bundled store is
now read and written by this package’s own codec
(R/aaa_parquet.R), so a plain install no longer pulls a
compiled Parquet dependency, and
load_siu_reports(format = "parquet") works on a machine
that never had one – previously it turned a missing optional package
into a hard stop, despite the corpus being bundled in exactly that
format.morie_data_load("vic_<key>"); they appear in
morie_data_catalog() like any other slug.data-raw/build_vic_tables.R.morie_datasets_vic_table() defaults to
offline = TRUE, and with nothing bundled it returned a
0-row frame on any machine that had not already downloaded the
workbook.load_chicago_data(full = TRUE) gains
limit (exact row cap) and fraction (share of
the live dataset, total looked up first); bounded fetches never touch
the full-dataset cache. Network examples are now
\donttest{} with try() so they degrade
gracefully offline.
SIU corpus also bundled as Parquet;
load_siu_reports(format = "parquet") reads it via
nanoparquet (columnar / SQL-friendly). CSV remains the default (no extra
dependency).
panel_reviewed flag column (65 cols).inst/extdata/rmoriedata.sqlite (retired by the
sqlite-to-parquet migration; no shipped code read it — the loader API is
Parquet-only via nanoparquet, and the file remains backed
up outside the package). Tarball drops from 5.5 MB to 4.25 MB.AGPL (>= 3) (the LICENSE
file was a verbatim copy of the standard license, which CRAN
flags).CITATION.cff, NOTICE, and
LICENSE are excluded from the built package
(.Rbuildignore), clearing the top-level-files NOTE.LinkingTo: rmoriebricklayer (>= 0.2.0) and links the
ecosystem’s shared compiled core instead of duplicating any C code.morie_core_sha256() and
morie_core_mean() call the shared kernels directly (fast
data-integrity hashing + summaries for the integrated fixtures, with no
dependency on rmorie). Tests assert they are byte-identical to
rmoriebricklayer’s own core_sha256() /
core_mean().Six small, base-R-only helpers for analysts releasing aggregate statistics from the integrated fixtures (or any other dataset) without re-identification risk:
morie_dp_laplace_count() — (epsilon, 0)-DP count via
the Laplace mechanism (sensitivity 1).morie_dp_gaussian_mean() — approximate (epsilon,
delta)-DP mean of a bounded numeric vector via the analytic Gaussian
mechanism.morie_dp_laplace_histogram() — per-bin Laplace noise on
a histogram.morie_k_anonymity_verify() — checks k-anonymity over a
set of quasi-identifiers and returns the offending equivalence
classes.morie_l_diversity_verify() — per-class distinct-count
check on a sensitive attribute.morie_cell_suppress() — small-cell suppression with
optional complementary suppression so primary-suppressed cells can’t be
reconstructed from row/column marginals (StatCan-style).All helpers use only base R + the existing stats import;
no new runtime dependencies. Inputs are validated and edge cases (empty
vectors, NAs, out-of-bounds inputs, invalid privacy parameters) throw
informative errors.
tests/testthat/test-dp.R — Monte-Carlo convergence +
variance-scaling sanity tests for the DP mechanisms; input-validation
coverage.tests/testthat/test-k-anonymity.R — hand-built fixtures
with known equivalence-class structure; complementary-suppression
correctness.These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.