Detailed example workflow: Data loading and preparation
Load packages to work with this vignette, including
forceR
library(magrittr)
library(dplyr)
library(stringr)
library(purrr)
library(ggplot2)
library(readr)
library(forceR)Download raw measurement files
From hereon, we will use data stored in a local folder. Please
download and unzip the content of the example_data.zip file from
Zenodo onto your hard drive, unzip the files and store the folder
location in the variable data.folder. This folder now
contains every file that is needed and has been produced during the
creation of this vignette, so you can always restore the files from
Zenodo in case you have accidentally overwritten anything
you wanted to keep.
If you decide to use your own files, consider that
forceR generally expects file names to start with leading
digits specifying the measurement number (E.g.
0001_G_maculatus.csv). The number (in this case
0001) is used to keep data files, log files, and PDF files
of the same measurement associated with each other. If not already the
case, we strongly recommend renaming the raw data files accordingly
before continuing with this forceR vignette.
First, we set the folder containing the example data as the
data.folder (make sure this fits your local path,
e.g. "./example_data"):
data.folder <- "./example_data"In case you want to use your own measurements, and if they were
recorded by the LJStream software (LabJack Corporation, Lakewood,
Colorado, US; tested for v1.19) as suggested in the forceX
instructions, you can use the convert_measurement()
function of forceR to convert your files and save them in
the data.folder. See ?convert_measurement()
for details.
Plot raw measurement
The measurement file should have the time steps (preferably in m.secs, since all code of this package expects this) in the first column, and the measurements in the second column. Column names do not matter, and everything but the first two columns will be ignored.
file <- file.path(data.folder, "0982.csv")
plot_measurement(file,
columns = c(1:2))
Crop raw measurement
The function crop_measurement() can be used to crop a
measurement. Since we want to work with the cropped file later, we
define path.data when calling the function to store the
result as a new file. Before running the function, create the folder on
your hard drive, e.g. at "./example_data/cropped".
cropped.folder <- "./example_data/cropped"If a measurement contains two distinct regions of bites with a lot of unnecessary data and/or measurement artefacts in-between (such as user-made peaks), I recommend to manually copy the RAW data files, give the copy a new measurement number (as if it was actually a separate measurement), and crop the distinct parts containing actual bites separately from the two copies of the file. For more distinct regions, create more copies.
I recommend to not crop the files too much in case baseline
corrections are needed later, because then the
baseline_corr() function will not be able to figure out
where the actual baseline might be. Leaving several seconds before and
after the first and last bite of a series will prevent such
problems.
file.cropped <- crop_measurement(file,
path.data = cropped.folder)Select two points at the beginning and end of the part of the
measurement we want to keep. If more than two points are selected, only
the last two points will be considered. This allows for corrections of
erroneous clicks. The position of the points on the y-axis is
irrelevant. Make sure that the start and end points are placed on the
graph between peaks, i.e. on the baseline. The y-values of the graph at
these positions will be used as reference points in the following
amp_drift_corr() function.
The resulting file will be saved with the suffix
"_cropped". We can plot it again:
cropped.folder <- "./example_data/cropped"
file <- file.path(cropped.folder, "0982_cropped.csv")
plot_measurement(file)
Amplifier drift correction
To remove the systemic, asymptotical drift characteristic for charge
amplifiers with resistor–capacitor (RC) circuits, we can use the
amp_drift_corr() function. This function requires a few
parameters. ?amp_drift_corr brings up the full explanation
of all values.
Basically, we only have to decide how we want to plot the data, define the correct folder with our measurement data, provide the time constant (tau, \(\tau\)) of our charge amplifier, and define a folder where we want to store the results. This function, in contrast to the previous functions, takes a folder as input because when the same charge amplifier was used for all measurements, the same settings can be used for all measurement files.
# create file list with all cropped files that need amplifier drift corrections.
# Here we use the cropped.folder with the cropped measurements.
file.list <- list.files(cropped.folder,
pattern = "csv$",
full.names = TRUE)
# define folder where to save the amplifier drift corrected file.
# If this folder does not yet exist, it will be created.
ampdriftcorr.folder <- "./cropped/ampdriftcorr"
for(filename in file.list){
print(filename)
amp_drift_corr(filename = filename,
tau = 9400,
res.reduction = 10,
plot.to.screen = FALSE,
write.data = TRUE,
write.PDFs = TRUE,
write.logs = TRUE,
output.folder = ampdriftcorr.folder,
show.progress = FALSE)
print("***********")
}Because we set the arguments write.data,
write.PDFs and write.logs to
TRUE, this creates the output.folder
("./ampdriftcorr") and two new folders inside the
output.folder:
"./ampdriftcorr/logs/"where log files containing info on theamp_drift_corrused are stored"./ampdriftcorr/pdfs/"where the PDFs of the time series before and after the corrections are stored.
The resulting graph of one corrected measurement may look like this:
In gray, we see the raw measurements, and in green the corrected
data. A linear drift in the corrected data may occur in case the first
value of the measurement is not exactly 0. This drift is
also corrected for by the function. The final result for each file is,
since write.data was set to TRUE, also saved
in a new CSV file in the "./ampdriftcorr/" folder.
Automatic or manual baseline correction of time series
If the baseline (zero-line) of a measurement is unstable (e.g. due to
temperature fluctuations, wind, …), it needs to be continually adjusted
throughout the measurement. Ideally, this is done automatically by the
function baseline.corr(). This automatic adjustment of the
baseline invokes a sliding window approach, during which the ‘minimum’
within each window is stored. A ‘minimum’ is defined by the parameter
quantile.size. If quantile.size is set to
0.05, the value below which 5% of the measurement data
within the current window lies, is treated as the current window’s
‘minimum’. Not taking the actual minimum within each window prevents the
treatment of short undershoots (artifacts created during the
measurement) as minima. In a second iteration, another sliding window
calculates the average of the ‘minima’ within each window. The resulting
‘minimal’ values are subtracted from the original time series.
This approach works well for time series with relatively short peaks.
For longer peaks, the window.size.mins should be increased.
However, the greater window.size.mins gets, the smaller
becomes the baseline-correcting effect of the function.
For the file
"./example_data/cropped/ampdriftcorr/1068_ampdriftcorr.csv",
we choose a window.size.mins of 2000:
filename = file.path(ampdriftcorr.folder, "1068_ampdriftcorr.csv")
plot_measurement(filename)
baselinecorr.folder <- "./cropped/ampdriftcorr/baselinecorr"
file.baselinecorr <- baseline_corr(filename = filename,
corr.type = "auto",
window.size.mins = 2000,
window.size.means = NULL,
quantile.size = 0.05,
y.scale = 0.5,
res.reduction = 10,
Hz = 100,
plot.to.screen = TRUE,
write.data = TRUE,
write.PDFs = TRUE,
write.logs = TRUE,
output.folder = baselinecorr.folder,
show.progress = FALSE)Again, new folders, PDFs and log files are created. The resulting plot shows the original data in gray, the ‘minima’ of the sliding windows in blue, the sliding mean of the ‘minima’ in bright green and the corrected time series in dark green:
If the automatic approach does not yield acceptable results, an
interactive manual approach to correct the baseline can be performed
instead. We do the manual correction for the file
"./example_data/cropped/ampdriftcorr/1174_ampdriftcorr.csv",
which contains measurements with long, plateau-like peaks where the
automatic baseline correction would fail.
filename = file.path(ampdriftcorr.folder, "1174_ampdriftcorr.csv")
plot_measurement(file)
file.baselinecorr <- baseline_corr(filename = filename,
corr.type = "manual",
plot.to.screen = TRUE,
write.data = TRUE,
write.PDFs = TRUE,
write.logs = TRUE,
output.folder = baselinecorr.folder,
show.progress = FALSE)
The green line represents a spline along the manually defined points and is subtracted from the original data (gray). The corrected data is plotted in dark green again.
We have used baseline_corr() on the files
1041_ampdriftcorr.csv, 1068_ampdriftcorr.csv,
1174_ampdriftcorr.csv,
1440_ampdriftcorr.csv.
File sorting
After all original files have been processed, we need to separate the
resulting files that we want to keep from files that have been created
along the way. The function sort_files() looks through a
vector of folders (data.folders) to identify files with the
same measurement name (i.e., the same character string in the file names
before the first underscore (_) and keeps the version of
the file that is found in the folder that is most closely to the end of
the list of data.folders. Thus, it is important that the
folders are parsed in the correct order. In our example, this order
would look like this:
"./example_data/""./example_data/cropped/""./example_data/cropped/ampdriftcorr/""./example_data/cropped/ampdriftcorr/baselinecorr"
If a file is present in
"./example_data/cropped/ampdriftcorr/baselinecorr", this
will be copied (move = FALSE) or moved
(move = TRUE) to a folder specified in
path.corrected, e.g.,
"./example_data/corrected/". Versions of the same
measurement in the other folders will be ingored. This is iteratively
repeated for the rest of the folders in reverse order of the
data.folders vector.
data.folders <- c(data.folder,
file.path(data.folder, "/cropped"),
file.path(data.folder, "/cropped/ampdriftcorr"),
file.path(data.folder, "/cropped/ampdriftcorr/baselinecorr"))
results.folder <- file.path(data.folder, "/corrected/")
sort_files(data.folders = data.folders,
results.folder = results.folder,
move = FALSE)File loading
Now we will load the first file in the results folder
using load_single(). The function removes all columns
except the one defined in columns = c(1:2) and renames them
to t and y. Then, it adds a
filename column based on the file name of the data.
file.list <- list.files(results.folder, pattern = "csv", full.names = TRUE)
df.1 <- load_single(file = file.list[1],
columns = c(1:2))df.1
#> # A tibble: 129,001 x 3
#> t y filename
#> <dbl> <dbl> <chr>
#> 1 0 0.00447 0979_ampdriftcorr
#> 2 1 0.00954 0979_ampdriftcorr
#> 3 2 0.0146 0979_ampdriftcorr
#> 4 3 0.0146 0979_ampdriftcorr
#> 5 4 0.00951 0979_ampdriftcorr
#> 6 5 0.00951 0979_ampdriftcorr
#> 7 6 0.00950 0979_ampdriftcorr
#> 8 7 0.00949 0979_ampdriftcorr
#> 9 8 0.00948 0979_ampdriftcorr
#> 10 9 0.00948 0979_ampdriftcorr
#> # ... with 128,991 more rowsunique(df.1$filename)
#> [1] "0979_ampdriftcorr"Alternatively, We can rapidly load all files in the
results folder by using load_mult():
df.all <- load_mult(folder = results.folder,
columns = c(1:2),
show.progress = TRUE)df.all
#> # A tibble: 615,724 x 3
#> t y filename
#> <dbl> <dbl> <chr>
#> 1 0 0.00447 0979_ampdriftcorr
#> 2 1 0.00954 0979_ampdriftcorr
#> 3 2 0.0146 0979_ampdriftcorr
#> 4 3 0.0146 0979_ampdriftcorr
#> 5 4 0.00951 0979_ampdriftcorr
#> 6 5 0.00951 0979_ampdriftcorr
#> 7 6 0.00950 0979_ampdriftcorr
#> 8 7 0.00949 0979_ampdriftcorr
#> 9 8 0.00948 0979_ampdriftcorr
#> 10 9 0.00948 0979_ampdriftcorr
#> # ... with 615,714 more rows
unique(df.all$filename)
#> [1] "0979_ampdriftcorr"
#> [2] "0980_ampdriftcorr"
#> [3] "0981_ampdriftcorr"
#> [4] "0982_ampdriftcorr"
#> [5] "1041_baselinecorr"
#> [6] "1068_baselinecorr"
#> [7] "1174_baselinecorr"
#> [8] "1440_baselinecorr"
The new tibble contains the measurement of all files in the selected folder.
Note that the peak in the figures above looks differently from the
one in your plots, but the principles are the same.
