Importing data.table

This document is focused on using data.table as a dependency in other R packages. If you are interested in using data.table C code from a non-R application, or in calling its C functions directly, jump to the last section of this vignette.

Importing data.table is no different from importing other R packages. This vignette is meant to answer the most common questions arising around that subject; the lessons presented here can be applied to other R packages.

Why to import data.table

One of the biggest features of data.table is its concise syntax which makes exploratory analysis faster and easier to write and perceive; this convenience can drive package authors to use data.table. Another, perhaps more important reason is high performance. When outsourcing heavy computing tasks from your package to data.table, you usually get top performance without needing to re-invent any of these numerical optimization tricks on your own.

Importing data.table is easy

It is very easy to use data.table as a dependency due to the fact that data.table does not have any of its own dependencies. This applies both to operating system and to R dependencies. It means that if you have R installed on your machine, it already has everything needed to install data.table. It also means that adding data.table as a dependency of your package will not result in a chain of other recursive dependencies to install, making it very convenient for offline installation.

DESCRIPTION file

The first place to define a dependency in a package is the DESCRIPTION file. Most commonly, you will need to add data.table under the Imports: field. Doing so will necessitate an installation of data.table before your package can compile/install. As mentioned above, no other packages will be installed because data.table does not have any dependencies of its own. You can also specify the minimal required version of a dependency; for example, if your package is using the fwrite function, which was introduced in data.table in version 1.9.8, you should incorporate this as Imports: data.table (>= 1.9.8). This way you can ensure that the version of data.table installed is 1.9.8 or later before your users will be able to install your package. Besides the Imports: field, you can also use Depends: data.table but we strongly discourage this approach (and may disallow it in future) because this loads data.table into your user’s workspace; i.e. it enables data.table functionality in your user’s scripts without them requesting that. Imports: is the proper way to use data.table within your package without inflicting data.table on your user. In fact, we hope the Depends: field is eventually deprecated in R since this is true for all packages.

NAMESPACE file

The next thing is to define what content of data.table your package is using. This needs to be done in the NAMESPACE file. Most commonly, package authors will want to use import(data.table) which will import all exported (i.e., listed in data.table’s own NAMESPACE file) functions from data.table.

You may also want to use just a subset of data.table functions; for example, some packages may simply make use of data.table’s high-performance CSV reader and writer, for which you can add importFrom(data.table, fread, fwrite) in your NAMESPACE file. It is also possible to import all functions from a package excluding particular ones using import(data.table, except=c(fread, fwrite)).

Be sure to read also the note about non-standard evaluation in data.table in the section on “undefined globals”

Usage

As an example we will define two functions in a.pkg package that uses data.table. One function, gen, will generate a simple data.table; another, aggr, will do a simple aggregation of it.

gen = function (n = 100L) {
  dt = as.data.table(list(id = seq_len(n)))
  dt[, grp := ((id - 1) %% 26) + 1
     ][, grp := letters[grp]
       ][]
}
aggr = function (x) {
  stopifnot(
    is.data.table(x),
    "grp" %in% names(x)
  )
  x[, .N, by = grp]
}

Testing

Be sure to include tests in your package. Before each major release of data.table, we check reverse dependencies. This means that if any changes in data.table would break your code, we will be able to spot breaking changes and inform you before releasing the new version. This of course assumes you will publish your package to CRAN or Bioconductor. The most basic test can be a plaintext R script in your package directory tests/test.R:

library(a.pkg)
dt = gen()
stopifnot(nrow(dt) == 100)
dt2 = aggr(dt)
stopifnot(nrow(dt2) < 100)

When testing your package, you may want to use R CMD check --no-stop-on-test-error, which will continue after an error and run all your tests (as opposed to stopping on the first line of script that failed) NB this requires R 3.4.0 or greater.

Testing using testthat

It is very common to use the testthat package for purpose of tests. Testing a package that imports data.table is no different from testing other packages. An example test script tests/testthat/test-pkg.R:

context("pkg tests")

test_that("generate dt", { expect_true(nrow(gen()) == 100) })
test_that("aggregate dt", { expect_true(nrow(aggr(gen())) < 100) })

If data.table is in Suggests (but not Imports) then you need to declare .datatable.aware=TRUE in one of the R/* files to avoid “object not found” errors when testing via testthat::test_package or testthat::test_check.

Dealing with "undefined global functions or variables"

data.table‘s use of R’s deferred evaluation (especially on the left-hand side of :=) is not well-recognised by R CMD check. This results in NOTEs like the following during package check:

* checking R code for possible problems ... NOTE
aggr: no visible binding for global variable 'grp'
gen: no visible binding for global variable 'grp'
gen: no visible binding for global variable 'id'
Undefined global functions or variables:
grp id

The easiest way to deal with this is to pre-define those variables within your package and set them to NULL, optionally adding a comment (as is done in the refined version of gen below). When possible, you could also use a character vector instead of symbols (as in aggr below):

gen = function (n = 100L) {
  id = grp = NULL # due to NSE notes in R CMD check
  dt = as.data.table(list(id = seq_len(n)))
  dt[, grp := ((id - 1) %% 26) + 1
     ][, grp := letters[grp]
       ][]
}
aggr = function (x) {
  stopifnot(
    is.data.table(x),
    "grp" %in% names(x)
  )
  x[, .N, by = "grp"]
}

The case for data.table’s special symbols (e.g. .SD and .N) and assignment operator (:=) is slightly different (see ?.N for more, including a complete listing of such symbols). You should import whichever of these values you use from data.table’s namespace to protect against any issues arising from the unlikely scenario that we change the exported value of these in the future, e.g. if you want to use .N, .I, and :=, a minimal NAMESPACE would have:

importFrom(data.table, .N, .I, ':=')

Much simpler is to just use import(data.table) which will greedily allow usage in your package’s code of any object exported from data.table.

If you don’t mind having id and grp registered as variables globally in your package namespace you can use ?globalVariables. Be aware that these notes do not have any impact on the code or its functionality; if you are not going to publish your package, you may simply choose to ignore them.

Care needed when providing and using options

Common practice by R packages is to provide customization options set by options(name=val) and fetched using getOption("name", default). Function arguments often specify a call to getOption() so that the user knows (from ?fun or args(fun)) the name of the option controlling the default for that parameter; e.g. fun(..., verbose=getOption("datatable.verbose", FALSE)). All data.table options start with datatable. so as to not conflict with options in other packages. A user simply calls options(datatable.verbose=TRUE) to turn on verbosity. This affects all data.table function calls unless verbose=FALSE is provided explicitly; e.g. fun(..., verbose=FALSE).

The option mechanism in R is global. Meaning that if a user sets a data.table option for their own use, that setting also affects code inside any package that is using data.table too. For an option like datatable.verbose, this is exactly the desired behavior since the desire is to trace and log all data.table operations from wherever they originate; turning on verbosity does not affect the results. Another unique-to-R and excellent-for-production option is R’s options(warn=2) which turns all warnings into errors. Again, the desire is to affect any warning in any package so as to not miss any warnings in production. There are 6 datatable.print.* options and 3 optimization options which do not affect the result of operations. However, there is one data.table option that does and is now a concern: datatable.nomatch. This option changes the default join from outer to inner. [Aside, the default join is outer because outer is safer; it doesn’t drop missing data silently; moreover it is consistent to base R way of matching by names and indices.] Some users prefer inner join to be the default and we provided this option for them. However, a user setting this option can unintentionally change the behavior of joins inside packages that use data.table. Accordingly, in v1.12.4 (Oct 2019) a message was printed when the datatable.nomatch option was used, and from v1.14.2 it is now ignored with warning. It was the only data.table option with this concern.

Troubleshooting

If you face any problems in creating a package that uses data.table, please confirm that the problem is reproducible in a clean R session using the R console: R CMD check package.name.

Some of the most common issues developers are facing are usually related to helper tools that are meant to automate some package development tasks, for example, using roxygen to generate your NAMESPACE file from metadata in the R code files. Others are related to helpers that build and check the package. Unfortunately, these helpers sometimes have unintended/hidden side effects which can obscure the source of your troubles. As such, be sure to double check using R console (run R on the command line) and ensure the import is defined in the DESCRIPTION and NAMESPACE files following the instructions above.

If you are not able to reproduce problems you have using the plain R console build and check, you may try to get some support based on past issues we’ve encountered with data.table interacting with helper tools: devtools#192 or devtools#1472.

License

Since version 1.10.5 data.table is licensed as Mozilla Public License (MPL). The reasons for the change from GPL should be read in full here and you can read more about MPL on Wikipedia here and here.

Optionally import data.table: Suggests

If you want to use data.table conditionally, i.e., only when it is installed, you should use Suggests: data.table in your DESCRIPTION file instead of using Imports: data.table. By default this definition will not force installation of data.table when installing your package. This also requires you to conditionally use data.table in your package code which should be done using the ?requireNamespace function. The below example demonstrates conditional use of data.table’s fast CSV writer ?fwrite. If the data.table package is not installed, the much-slower base R ?write.table function is used instead.

my.write = function (x) {
  if(requireNamespace("data.table", quietly=TRUE)) {
    data.table::fwrite(x, "data.csv")
  } else {
    write.table(x, "data.csv")
  }
}

A slightly more extended version of this would also ensure that the installed version of data.table is recent enough to have the fwrite function available:

my.write = function (x) {
  if(requireNamespace("data.table", quietly=TRUE) &&
    utils::packageVersion("data.table") >= "1.9.8") {
    data.table::fwrite(x, "data.csv")
  } else {
    write.table(x, "data.csv")
  }
}

When using a package as a suggested dependency, you should not import it in the NAMESPACE file. Just mention it in the DESCRIPTION file. When using data.table functions in package code (R/* files) you need to use the data.table:: prefix because none of them are imported. When using data.table in package tests (e.g. tests/testthat/test* files), you need to declare .datatable.aware=TRUE in one of the R/* files.

data.table in Imports but nothing imported

Some users (e.g.) may prefer to eschew using importFrom or import in their NAMESPACE file and instead use data.table:: qualification on all internal code (of course keeping data.table under their Imports: in DESCRIPTION).

In this case, the un-exported function [.data.table will revert to calling [.data.frame as a safeguard since data.table has no way of knowing that the parent package is aware it’s attempting to make calls against the syntax of data.table’s query API (which could lead to unexpected behavior as the structure of calls to [.data.frame and [.data.table fundamentally differ, e.g. the latter has many more arguments).

If this is anyway your preferred approach to package development, please define .datatable.aware = TRUE anywhere in your R source code (no need to export). This tells data.table that you as a package developer have designed your code to intentionally rely on data.table functionality even though it may not be obvious from inspecting your NAMESPACE file.

data.table determines on the fly whether the calling function is aware it’s tapping into data.table with the internal cedta function (Calling Environment is Data Table Aware), which, beyond checking the ?getNamespaceImports for your package, also checks the existence of this variable (among other things).

Further information on dependencies

For more canonical documentation of defining packages dependency check the official manual: Writing R Extensions.

Importing data.table C routines

Some of internally used C routines are now exported on C level thus can be used in R packages directly from their C code. See ?cdt for details and Writing R Extensions Linking to native routines in other packages section for usage.

Importing from non-r Applications

Some tiny parts of data.table C code were isolated from the R C API and can now be used from non-R applications by linking to .so / .dll files. More concrete details about this will be provided later; for now you can study the C code that was isolated from the R C API in src/fread.c and src/fwrite.c.

How to convert your Depends dependency on data.table to Imports

To convert a Depends dependency on data.table to an Imports dependency in your package, follow these steps:

Step 0. Ensure your package is passing R CMD check initially

Step 1. Update the DESCRIPTION file to put data.table in Imports, not Depends

Before:

Depends:
    R (>= 3.5.0),
    data.table
Imports:

After:

Depends:
    R (>= 3.5.0)
Imports:
    data.table

Step 2.1: Run R CMD check

Run R CMD check to identify any missing imports or symbols. This step helps:

• Automatically detect any functions or symbols from data.table that are not explicitly imported.
• Flag missing special symbols like .N, .SD, and :=.
• Provide immediate feedback on what needs to be added to the NAMESPACE file.

Note: Not all such usages are caught by R CMD check. In particular, R CMD check skips some symbols/functions in formulas and will completely miss parsed expressions like parse(text = "data.table(a = 1)"). Packages will need good test coverage to detect these edge cases.

Step 2.2: Modify the NAMESPACE file

Based on the R CMD check results, ensure all used functions, special symbols, S3 generics, and S4 classes from data.table are imported.

That means adding importFrom(data.table, ...) directives for symbols, functions, and S3 generics, and/or importClassesFrom(data.table, ...) directives for S4 classes as appropriate. See ‘Writing R Extensions’ for full details on how to do so properly.

Blanket import

Alternatively, you can import all functions from data.table at once, though this is generally not recommended:

import(data.table)

Justification for Avoiding Blanket Imports:

1. Documentation: The NAMESPACE file can serve as good documentation of how you depend on certain packages.
2. Avoiding Conflicts: Blanket imports leave you open to subtle breakage. For example, if you import(pkgA) and import(pkgB), but later pkgB exports a function also exported by pkgA, this will break your package due to conflicts in your namespace, which is disallowed by R CMD check and CRAN.

Step 3: Update Your R code files outside the package’s R/ directory

When you move a package from Depends to Imports, it will no longer be automatically attached when your package is loaded. This can be important for examples, tests, vignettes, and demos, where Imports packages need to be attached explicitly.

Before (with Depends):

# data.table functions are directly available
library(MyPkgDependsDataTable)
dt <- data.table(x = 1:10, y = letters[1:10])
setDT(dt)
result <- merge(dt, other_dt, by = "x")

After (with Imports):

# Explicitly load data.table in user scripts or vignettes
library(data.table)
library(MyPkgDependsDataTable)
dt <- data.table(x = 1:10, y = letters[1:10])
setDT(dt)
result <- merge(dt, other_dt, by = "x")

Benefits of using Imports

• *User-Friendliness: Depends alters your users’ search() path, possibly without their wanting to do so.
• Namespace Management: Only the functions your package explicitly imports are available, reducing the risk of function name clashes.
• Cleaner Package Loading: Your package’s dependencies are not attached to the search path, making the loading process cleaner and potentially faster.
• Easier Maintenance: It simplifies maintenance tasks as upstream dependencies’ APIs evolve. Depending too much on Depends can lead to conflicts and compatibility issues over time.