Getting Started with memtoc

The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Introduction

memtoc provides simple start/stop memory tracking for R, inspired by the tictoc package for timing. Wrap any code block with tic_mem() and toc_mem() to measure RAM usage.

library(memtoc)

Basic Usage

The simplest use case is tracking memory for a single operation:

tic_mem("load data")
data <- read.csv("large_file.csv")
toc_mem()
#> ✔ load data: 142.3 MB peak | 142.3 MB current | 1.24 sec | 2 samples

The output shows:

Peak memory: Maximum RSS (Resident Set Size) during the operation
Current memory: RSS at the end of the operation
Elapsed time: Wall-clock time
Samples: Number of memory measurements taken

Background Polling

By default, memtoc spawns a background process that continuously samples memory. The reported peak is the maximum observed sample. Short-lived allocations between samples can be missed:

tic_mem("matrix operations", interval = 0.5)  # Sample every 0.5 seconds

# Create a large temporary matrix
x <- matrix(rnorm(1e8), ncol = 1000)  # ~800 MB
y <- colMeans(x)                        # x can be garbage collected
rm(x)
gc()

result <- toc_mem()
#> ✔ matrix operations: 812.4 MB peak | 45.2 MB current | 3.21 sec | 7 samples

Without background polling, you would only see the final memory (45.2 MB), missing the 800 MB peak. Access the full trajectory with result$trajectory.

For very quick operations, disable polling to avoid startup overhead:

tic_mem("quick op", interval = NULL)  # Snapshot mode
y <- 1:100
toc_mem()

Nested Tracking

Track an entire pipeline while also measuring individual steps:

tic_mem("full pipeline")

  tic_mem("step 1: load")
  data <- read.csv("data.csv")
  toc_mem()
  #> ✔ step 1: load: 50.2 MB peak | 50.2 MB current | 1.2 sec

  tic_mem("step 2: transform")
  features <- transform(data)
  toc_mem()
  #> ✔ step 2: transform: 125.8 MB peak | 98.3 MB current | 2.4 sec

  tic_mem("step 3: model")
  model <- train(features)
  toc_mem()
  #> ✔ step 3: model: 512.1 MB peak | 201.5 MB current | 45.2 sec

toc_mem()
#> ✔ full pipeline: 512.1 MB peak | 201.5 MB current | 48.8 sec

Logging Results

Collect results for later analysis:

mem_clearlog()

for (i in 1:10) {
  tic_mem(paste("iteration", i))
  # ... do work ...
  toc_mem(log = TRUE, quiet = TRUE)
}

# Get all results as a data frame
results <- mem_log()
summary(results$mem_peak)

Parallel Worker Monitoring

When using the future package for parallel processing, memtoc can monitor memory across all workers:

library(future)
library(future.apply)

# Set up parallel workers
plan(multisession, workers = 4)

# Check that workers are detected
mem_parallel_info()
#> ── Parallel Backend Info
#> • Main process PID: 12345
#> • Current plan: multisession
#> • Workers configured: 4

# Monitor parallel job
tic_mem("parallel computation", workers = "auto")
result <- future_lapply(1:100, function(i) {
  x <- rnorm(1e6)
  mean(x)
}, future.seed = TRUE)
mem_result <- toc_mem()
#> ✔ parallel computation: 1.2 GB peak | 245 MB current | 5.4 sec | 4 workers

# View per-worker breakdown
mem_result$worker_stats

# Clean up
plan(sequential)

Worker options:

workers = "auto": Auto-detect future workers (default)
workers = "none": Only monitor main process
workers = "children": Monitor main process and child processes
workers = c(pid1, pid2): Explicit PID list

System Memory Warnings

memtoc warns you when system RAM is running low:

tic_mem("memory intensive")
# ... allocate lots of memory ...
toc_mem()
#> ✔ memory intensive: 12.4 GB peak | 11.2 GB current | 45.2 sec
#> ⚠ System RAM high: 87.3% used

Warnings appear at 80% usage; critical alerts at 95%.

Crash Recovery

Checkpoints are stored in R’s session-specific temporary directory. Within a session, list available checkpoints or recover the outer block by PID. After a restart, use mem_recover(path = ...) with the actual surviving checkpoint path from the previous session. If the temporary directory was removed, the samples cannot be recovered. Normal completion removes checkpoints.

# List checkpoints in this R session
mem_recover()
#> ℹ Found 1 recovery file:
#>   • PID 12345: 15.2 KB, 152 samples

# Recover the data
recovered <- mem_recover(pid = 12345)
head(recovered)

Diagnostics

If background polling isn’t working, run diagnostics:

mem_capabilities()
#>     memory_queries background_polling 
#>               TRUE               TRUE

# Detailed troubleshooting
mem_diagnose()

Tips

Use labels: Always pass a message to tic_mem() for easier tracking
Adjust interval: Use shorter intervals (0.1-0.5s) for fast operations, longer intervals (1-5s) for long-running jobs
Disable polling for quick ops: Use interval = NULL for sub-second operations
Monitor workers: Set workers = "auto" when using future for parallelism
Log results: Use log = TRUE when running benchmarks or comparisons

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.