--- title: "Creating a `BGF` from a Custom Fermentation System" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Creating a `BGF` from a Homebrew Fermentation System} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) ``` ## Whats this vignette is about This vignette is the second of three, that focus the topic of how to get data into a `BGF`. Assuming the reader has read `vignette("Introducing-BGF",package="bgfanalyzer")` and `vignette("BGF-commercial",package="bgfanalyzer")` and is familiar with the general concepts of `BGF`'s and how to build a `BGF` from a report generated by a distinct commercial fermentation system (see `vignette("BGF-commercial",package="bgfanalyzer")` for details). In this vignette, we will discuss how to build a `BGF` from one or more external files, that are in a standardized text format. Such files are expected to contain at least cumulative exhaust gas volume measurements and a time log of a single fermentation, but can include additional sensor readings such as pH or gas quality measurements. In the following sections we will see how to build a `BGF` from multiple such standardized files, which store the data of several fermentations. ## Background The data used during this vignette was acquired from a custom fermentation system. It consisted of a 2l stirred and heated batch reactor. This reactor was equipped with several sensors, such as a pH-, temperature- and RedOx-electrodes, allowing continuously monitoring of essential fermentation parameters. In addition, the cumulative exhaust gas volume was recorded and the exhaust gas composition (= gas quality) was sequentially analyzed. The data files used in the following, can be grouped in two groups: The *Fermentation_\**-files store cumulative exhaust gas volumes measurements together with other sensor readings of each one fermentation. The *gasq_\**-files store gas quality measurements of the respective fermentations. The fermentations were carried out in individually heated batch reactors and sensor readings within *Fermentation_\**-files were recorded by third party software. The data in *gasq_\**-files originates from a gas chromatograph sequentially analyzing the gas composition within the exhaust gas of the same fermentations. Sensor readings in *Fermentation_\**-files were manually converted into the standard format using an R script specifically developed to convert text reports generated by the third party software. Gas quality measurements were collected using instrument specific software and uploaded into a database afterwards. The *gasq_\**-files were extracted from that database and were already in the standardized format upon database extraction. ## Object creation Now, let's start building a simple `BGF` form an external file in standard format. This `BGF` will not contain any 'Blank' fermentations. We will use the function `from_standard_record()` to create a new `BGF`. ```{r setup} # load library library(bgfanalyzer) # We build a BGF from myBGF<-from_standard_record(ReactorLayout = "Substrat A", ProcessTemp = 80, InocToSubRatio = 0.1, path = system.file("extdata","Fermentation_A.tsv",package = "bgfanalyzer"), time_col = 1, product_col = 3, BlankLabel = "Blank", name = "myBGF", units = "days") # inspect object myBGF # have a closer look at its 'metaData'-layer myBGF$metaData ``` A new `BGF` was created with only one row in the `metaData`-layer! How about the `BioGasData`-layer? Let's take a look at its `head()` and `tail()` output: ```{r headTail} # look at the first six rows of the 'BioGasData'-layer head(myBGF$BioGasData) # look at a snippet of the last six rows of the 'BioGasData'-layer tail(myBGF$BioGasData[c(1,2,3,(length(myBGF$BioGasData)-c(2,1,0)))]) ``` As we can see, `reactor`, `time` and `product` columns were already filled with data from the external file and in addition eight new columns with data were added to the `BioGasData`-layer of 'myBGF' through the call to `from_standard_record()` ### Adding additional data to `BioGasData`-layer Now we will add the gas composition measurements stored in *gasq_A.tsv*. First, we import the file, next we add it to the `BGF`. ```{r gasq_A} # import the gas quality measurements stored in a separate file gasq_A <- import_standard_record(ipath = system.file("extdata","gasq_A.tsv",package = "bgfanalyzer"), dec = ".", sep = "\t", header = TRUE, mkFRTime = "2025-05-11 14:08:35", FRTime_col = 1, units = "days") # add gas quality measurements to BGF myBGF <- add_BG_parameter(myBGF, parameter = gasq_A, reactor = "R1", time = 3, value = 2, name = "H2", cut_zero = TRUE, interpolate_missing = FALSE) # look at a snippet of the first six rows of the 'BioGsaData'-layer to see the results head(myBGF$BioGasData[c(1,2,3,(length(myBGF$BioGasData)-c(3,2,1,0)))]) ``` A new column, `H2`, storing the amount of the target gas in the exhaust gas (in this example the target gas is Hydrogen, *H~2~*) was added to the `BioGasData`-layer. However, most of the entries in this new column are `NA` and insertion of data into the existing `BioGasData`-layer created additional `NA`'s whenever a value was inserted: ```{r insertion-gaps} # data insertion created gaps myBGF$BioGasData[c(97:99),c(1,2,3,(length(myBGF$BioGasData)-c(3,2,1,0)))] ``` ### `NA` removal through data inter- and extrapolation Such 'gaps' in the data can be closed using the function `na_correction()`. By default, this function will close the gaps in all numeric columns of the `BioGasData`-layer, as long as there is a value before and a value after the gap. ```{r na_correction} # close the gaps myBGF <- na_correction(myBGF) # look at a snippet of the first six rows head(myBGF$BioGasData[c(1,2,3,(length(myBGF$BioGasData)-c(3,2,1,0)))]) # look at the gap that arrose from data insertion myBGF$BioGasData[c(97:99),c(1,2,3,(length(myBGF$BioGasData)-c(3,2,1,0)))] # look at a snippet of the last six rows tail(myBGF$BioGasData[c(1,2,3,(length(myBGF$BioGasData)-c(3,2,1,0)))]) ``` The data is almost free of `NA`'s now. Only in the end of the `BioGasData`-layer, `NA`'s in `H2` still exist. We can handle this by a second call to `na_correction()` with some adjustments. We will specifically correct column `H2` and will not end correction if no value after the gap exist. To avoid that this `NA`-correction yields in negative concentrations, we add argument `sub_zero = 0` to the function call: ```{r sepcific_correction} # correct remaining NA's in H2 myBGF <- na_correction(myBGF, which = "H2", end=F, sub_zero=0) # insepct results tail(myBGF$BioGasData[c(1,2,3,(length(myBGF$BioGasData)-c(3,2,1,0)))]) ``` As we can see, the terminal `NA`'s in `H2` have been replaced by interpolated values. ### Further data processing Now, we have build a `BGF` from data stored within two external files in a standardized data format. This `BGF` still has missing values in `production`, `rel_production`, `net_product` and `yield` of its `BioGasData`-layer, which we need to take care of. First, we calculate the values for `production` using `calculate_flow_from_volume()`: ```{r flow_from_volume} # cumulative exhaust gas volume mesurements in 'product' can be used to calculate # the 'production', a.k.a the biogas flow myBGF <- calculate_flow_from_volume(myBGF) # inspect standard columns of 'BioGasData'-layer head(myBGF$BioGasData[c(1:7)]) ``` Similarly, we can calculate values for `rel_production` using `relative_production()`: ```{r rel_prod} # calculate relative production myBGF <- relative_production(myBGF) # inspect standard columns of 'BioGasData'-layer head(myBGF$BioGasData[c(1:7)]) ``` Let's proceed with the calculation of `net_product`. Therefore, we use the function `netGasGC()`, which takes the values of `production` and multiplies it with the target gas concentration stored in `H2`, to calculate the net amount of target gas produced in between observations. Next, this function will cumulate these 'net production' values which yields in the final `net_product` values. ```{r netGasGC} # calculate net_product myBGF <- netGasGC(myBGF, purity = "H2", substract_blank = FALSE) # inspect standard columns of 'BioGasData'-layer tail(myBGF$BioGasData[c(1:7)]) ``` We successfully calculated `net_product`. Before we continue calculating the `yield`, we must first add some information to the `metaData`-layer. The `yield` reflects the amount of `net_product` produced per 'input organics'. This example fermentation was conducted in a 3L reactor with 3.23 wt.\% organic total solutes (oTS), which makes 96.89 g oTS in the fermentation. We add this value like this: ```{r add-oTS} # add the oTS to 'metaData'-layer myBGF <- add_metaData(myBGF,98.89,lab = "oTS") # inspect change in 'metaData'-layer myBGF$metaData ``` Now, we can calculate the `yield` like this: ```{r yield_cal} # calculate 'yield' myBGF <- calc_yield(myBGF,pos = 4) # inspect standard columns of 'BioGasData'-layer tail(myBGF$BioGasData[c(1:7)]) # get the yield summary myBGF <- summarize_yield(myBGF) # inspect change in 'metaData'-layer myBGF$metaData ``` Afterwards, all standard columns of the `BGF` were calculated and we can use `bgf_plot()` to visualize the data: ```{r bgf_plot-type-all,fig.width=7,fig.height=5} # build all standard plots bgf_plot(myBGF) ``` In the line plots, we can see some disturbance in the beginning of the fermentation. We can 'correct' this by moving the '0' value in the observation time to the right, and afterwards delete all data with a 'negative' observation time. A suitable new '0' value can be found interactively: ```{r interactive_product,fig.width=7,fig.height=5} # print 'product_curve' interactively bgf_plot(myBGF,type = "product",interaction=TRUE) ``` Now we can hover with the courser over the line and identify 'time=0.104' as the first value after the disturbance. Alternatively, we could have use `` `View(myBGF\$BioGasData)` `` (or `` `print(myBGF\$BioGasData)` ``) and search for a suitable time there. So, data clean up is possible by deleting the first 0.11 days of this 6 days fermentation. By doing this, 98 \% of the data will be kept. ```{r trim_FRA,fig.width=7,fig.height=5} # trim the fermentation myBGF<-trim_FR_time(myBGF,0.11) # inspect results by ploting bgf_plot(myBGF,type = "product") ``` When shifting the '0' value to the right, it is advised to run `update_BGF()` to ensure the internal logic of the `BGF` and to rerun data processing functions (`calculate_flow_from_volume()`, `relative_production()`, `netGasGC()`, `calc_yield()` and `summarize_yield()`). So: ```{r rerun-data-proc,fig.width=7,fig.height=5} # ensure internal logic of the 'BGF' myBGF<-update_BGF(myBGF) # re-calculate 'production' myBGF <- calculate_flow_from_volume(myBGF) # re-calculate 'rel_production' myBGF <- relative_production(myBGF) # re-calculate 'net_product' myBGF <- netGasGC(myBGF,"H2",substract_blank = FALSE) # re-calculate 'yield' myBGF <- calc_yield(myBGF,pos = 4) # plot 'BGF' again bgf_plot(myBGF) ``` Let's try printing the BGF: ```{r full_hombrew_BGF} # print the BGF myBGF ``` Finally, we have a complete `BGF` with data of a single fermentation. The input data of that `BGF` was split over two external data files. In the following, we will see, how to add further fermentations with the same, or a different reactor layout to that `BGF`. ## Adding further fermentations Now, that a final `BGF` is build let's see how to add another fermentation with the same reactor layout. ### Same reactor layout Like the first fermentation, also the input data of the second is split in two external files. The first step is importing these files and adding the new data to the existing `BGF`: ```{r input-second-fr} # Import the new data directly from the input file with cumulative exhaust gas volume measurements myBGF <- add_standard_record(myBGF, path = system.file("extdata","Fermentation_B.tsv",package = "bgfanalyzer"), RName = "R2", time_col = "UTC", units = "days", product_col = "GCounter..ml.") # updating internal logic is highly recommended myBGF <- update_BGF(myBGF) # correct 'metaData$Layout' for new fermentation myBGF <- alter_whatever(myBGF,layer = "metaData",what = "Layout",value = "Substrate A") # already add 'metaData$oTS' at this point myBGF <- alter_whatever(myBGF,layer = "metaData",what = "oTS",value = 98.89,ID = "R2") # import respective gas quality measurements gasq_B <- import_standard_record(ipath = system.file("extdata","gasq_B.tsv",package = "bgfanalyzer"), dec = ".", sep = "\t", header = TRUE, mkFRTime = "2025-05-19 22:00:00", FRTime_col = 1, units = "days") # add gas quality measurements to BGF myBGF <- add_BG_parameter(myBGF, parameter = gasq_B, reactor = "R2", time = 3, value = 2, name = "H2", makeCol = FALSE, cut_zero = TRUE, interpolate_missing = TRUE) # look at a snippet of the first six rows of the 'BioGsaData'-layer to see the results tail(myBGF$BioGasData[c(1,2,3,(length(myBGF$BioGasData)-c(3,2,1,0)))]) ``` Now that we have added the raw data of a second fermentation we can interactively plot the product curve to see if (and how) we need to trim the added data: ```{r interactive_product_2,fig.width=7,fig.height=5} # plot interactive product curve of BGF plot_product_curve(myBGF,interaction=TRUE) ``` The data of 'R2' look strange from 0 to 0.61 d and after 2.49 d. We can adjust this like before using ```{r trim_FRB,fig.width=7,fig.height=5} # remove data later than 2.49d myBGF <- trim_FR_time(myBGF, value = 2.49, mode = "R2", left_end = FALSE) # remove data before 0.61d myBGF <- trim_FR_time(myBGF, value = 0.61, mode = "R2") # plot product curve again to inspect results plot_product_curve(myBGF) ``` The data looks much better now! Let's continue with calculating `production`, `rel_production`, `netGas` and `yield` for the newly added fermentation. We can use the same functions as before: ```{r rerun-data-proc_2,fig.width=7,fig.height=5} # ensure internal logic of the 'BGF' myBGF<-update_BGF(myBGF) # close gaps myBGF <- na_correction(myBGF) # re-calculate 'production' myBGF <- calculate_flow_from_volume(myBGF) # re-calculate 'rel_production' myBGF <- relative_production(myBGF) # re-calculate 'net_product' myBGF <- netGasGC(myBGF,"H2",substract_blank = FALSE) # re-calculate 'yield' myBGF <- calc_yield(myBGF,pos = 4) # summarize yield myBGF <- summarize_yield(myBGF) # print the new BGF myBGF ``` In the `print()` we see, the `BGF` now has 2 fermentations. In the 'yield summary' standard deviations for 'yield', 'production' and 'time_production' are calculated as the fermentations both have 'Layout' 'Substrate A'. Interestingly, although the observation times of these example fermentations differ, the 'yield' at the final observation time is quite similar as seen in the low 'sd_yield' value. ### Different reactor layout Now that we have build a `BGF` with two fermentations of the same reactor layout from several external files, it is time to add further fermentations, that have a different reactor layout. At first we create a second `BGF` with two fermentations of reactor layout 'Substrate B', and next we merge the two `BGF`'s into a single object. Let's start with creating the new `BGF`. It will have 2 fermentations in the `metaData`-layer, and the raw data of the first fermentation will be directly imported to the `BioGasLayer`-layer. ```{r create_myBGF2} # create a new BGF directly from the record of a third fermentation myBGF2<-from_standard_record(ReactorLayout = c("2*Substrate B"), ProcessTemp = 80, InocToSubRatio = 0.1, path = system.file("extdata","Fermentation_C.tsv",package = "bgfanalyzer"), time_col = 1, product_col = 3, BlankLabel = "Blank", name = "myBGF2", units = "days") # import respective gas quality data gasq_C <- import_standard_record(ipath = system.file("extdata","gasq_C.tsv",package = "bgfanalyzer"), dec = ".", sep = "\t", header = TRUE, mkFRTime = "2024-10-27 05:30:00", FRTime_col = 1, units = "days") # add gas quality measurements to BGF myBGF2 <- add_BG_parameter(myBGF2, parameter = gasq_C, reactor = "R1", time = 3, value = 2, name = "H2", cut_zero = TRUE, interpolate_missing = TRUE) # print the new BGF myBGF2 ``` Before, further processing the data, let's add the raw data from the last fermentation: ```{r adding_FRD} myBGF2 <- add_standard_record(myBGF2, path = system.file("extdata","Fermentation_D.tsv",package = "bgfanalyzer"), RName = "R2", time_col = "UTC", units = "days", product_col = "GCounter..ml.") # updating internal logic is highly recommended myBGF2 <- update_BGF(myBGF2) # import respective gas quality data gasq_D <- import_standard_record(ipath = system.file("extdata","gasq_D.tsv",package = "bgfanalyzer"), dec = ".", sep = "\t", header = TRUE, mkFRTime = "2024-10-30 23:00:00", FRTime_col = 1, units = "days") # add gas quality measurements to BGF myBGF2 <- add_BG_parameter(myBGF2, parameter = gasq_D, reactor = "R2", time = 3, value = 2, name = "H2", makeCol = FALSE, cut_zero = TRUE, interpolate_missing = TRUE) # print the BGF myBGF2 ``` As we can see, the number of observations in the `BioGasData`-layer has increased indicating a successful data addition. We continue with processing the data. First, we check if the data needs trimming and afterwards we can head towards `yield` calculation! ```{r interactive_product_myBGF2,fig.width=7,fig.height=5} # interactively plot product curve plot_product_curve(myBGF2,interaction=TRUE) ``` We see that both fermentations need adjustments in the beginning. 'R1' will be trimmed by 0.42d and 'R2' will be trimmed by 0.56d counting from observation start. Furthermore, 'R1' needs to be cut after 4.08d, which will be our first step: ```{r trimming_myBGF2,fig.width=7,fig.height=5} # trim 'R1' fermentation myBGF2 <- trim_FR_time(myBGF2,4.08,"R1",left_end = FALSE) # updating internal logic is highly recommended myBGF2 <- update_BGF(myBGF2) # trim 'R1' fermentation myBGF2 <- trim_FR_time(myBGF2,0.42,"R1") # updating internal logic is highly recommended myBGF2 <- update_BGF(myBGF2) # trim 'R2' fermentation myBGF2 <- trim_FR_time(myBGF2,0.56,"R2") # updating internal logic is highly recommended myBGF2 <- update_BGF(myBGF2) # plot the product curve again to see results plot_product_curve(myBGF2) ``` Now we can start the data processing loop: ```{r processing_myBGF2} # updating internal logic is highly recommended myBGF2 <- update_BGF(myBGF2) # close gaps in data myBGF2 <- na_correction(myBGF2,end=F) # calculate production myBGF2 <- calculate_flow_from_volume(myBGF2) # calculate relative production myBGF2 <- relative_production(myBGF2) # calculate net gas myBGF2 <- netGasGC(myBGF2,"H2",substract_blank = FALSE) # add a oTS column at the metaData-layer myBGF2<-add_metaData(myBGF2,c(85.5,85.5),lab="oTS") # calculate yield myBGF2<-calc_yield(myBGF2,4) # summarize yield myBGF2<-summarize_yield(myBGF2) # print BGF myBGF2 ``` Finally, we can merge the two `BGF`'s into a new `BGF` containing all 4 fermentations using `merge_BGF()` : ```{r merge_BGF} # merge the two BGFs mergedBGF<-merge_BGF(myBGF,myBGF2,name = "merged BGF") # print the merged BGF mergedBGF ``` Now we have successfully created a single `BGF` with 4 fermentations of two reactor layouts! The raw data for this `BGF` was extracted from 8 external files. Note that it would have been possible to create the same object without merging two `BGF`'s. To do this one could set e.g. '`ReactorLayout = c("2*Substrate A","2*Substrate B")`' in the initial call to `from_standard_record()` to set up a `BGF` for 4 fermentations and than successively add further data from external files like described e. g. for `myBGF2` in the section above.