The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.
This package provides functions for quickly writing (and reading
back) a data.frame
to file in sqlite
format.
The name stands for Store Tables using SQLite, or alternatively
for Quick Store Tables (either way, it could be pronounced as
Quest). For data.frames
containing the supported
data types it is intended to work as a drop-in replacement for the
write_*()
and read_*()
functions provided by
packages such as fst
, feather
,
qs
, and readr
packages (as well as the
writeRDS()
and readRDS()
functions).
The core functions are read_qst()
and
write_qst()
, which read/write a data.frame
from/to a SQLite database. The database contains a single table named
data
, which contains the data from the
data.frame
.
The package wraps the functionality of RSQLite
and
dbplyr
, which do the heavy lifting. The resulting file is
reasonably small and the read/write process for a complete file is
reasonably fast, although no claims are made of superiority to the above
packages that focus on these issues.
This packages shines in the ability to quickly and easily switch from
loading a complete data.frame
to using
dbplyr
/SQLite
to subset data on disk before
loading it into memory. This is done simply by setting the
lazy
parameter of read_qst()
to
TRUE
:
write_qst(dat_org, "dat.qst") # A reasonably large data set
<- read_qst("dat.qst") # Reasonably quick
dat_full <- read_qst("dat.qst", lazy=TRUE) # Near instanteous
dat # Prints out nicely using dbplyr
dat %>% filter(state=="AZ") # Filtered from disk using SQLite dat
Filtering from disk (as shown in the last line) is reasonably fast
(typically much faster than loading the whole data set before then
filtering in memory), even without indexes. With indexes, which can be
created on write with the indexes
and
unique_indexes
arguments to write_qst()
, it
becomes even faster.
The following features are planned in future releases:
If you are using qst
and would like a specific feature
to be implemented, either from the road map or another feature, the
simplest way to request it is by opening an issue.
These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.