The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

A practical marcxmlr workflow

Lorenzo Isella

Overview

marcxmlr reads MARC 21 XML into a fixed 11-column canonical table that preserves record structure, source order, repeated fields, repeated subfields, and indicators. The same representation can be queried with ordinary R tools, checked after editing, written back to MARCXML, or stored as Parquet for larger collections.

This vignette follows one complete workflow:

  1. read a MARCXML collection;
  2. inspect repeated MARC structure;
  3. query and derive analytical data with dplyr;
  4. edit a repeated subfield;
  5. diagnose and serialize the canonical representation; and
  6. optionally move the same schema to a Parquet-backed workflow.

All examples use the same synthetic Orwell MARCXML record used in the README. It is bundled with the package, so the vignette is offline and quick to build.

Read MARCXML

library(marcxmlr)
library(dplyr)

source_xml <- system.file(
  "extdata",
  "orwell-marcxml.xml",
  package = "marcxmlr"
)

marc <- read_marcxml(source_xml)
marc
#> # A tibble: 16 × 11
#>    record_id field_type   tag   subfield_code value field_order field_occurrence
#>        <int> <chr>        <chr> <chr>         <chr>       <int>            <int>
#>  1         1 leader       LDR   <NA>          0000…           0                1
#>  2         1 controlfield 001   <NA>          12345           1                1
#>  3         1 datafield    100   a             Orwe…           2                1
#>  4         1 datafield    100   d             1903…           2                1
#>  5         1 datafield    245   a             Nine…           3                1
#>  6         1 datafield    245   c             Geor…           3                1
#>  7         1 datafield    264   a             Lond…           4                1
#>  8         1 datafield    264   b             Seck…           4                1
#>  9         1 datafield    264   c             1949.           4                1
#> 10         1 datafield    650   a             Tota…           5                1
#> 11         1 datafield    650   v             Fict…           5                1
#> 12         1 datafield    650   a             Dyst…           6                2
#> 13         1 datafield    650   v             Fict…           6                2
#> 14         1 datafield    856   u             http…           7                1
#> 15         1 datafield    856   y             Full…           7                1
#> 16         1 datafield    856   y             Mirr…           7                1
#> # ℹ 4 more variables: ind1 <chr>, ind2 <chr>, subfield_order <int>,
#> #   subfield_occurrence <int>

The result is an ordinary tibble with the canonical 11-column schema:

names(marc)
#>  [1] "record_id"           "field_type"          "tag"                
#>  [4] "subfield_code"       "value"               "field_order"        
#>  [7] "field_occurrence"    "ind1"                "ind2"               
#> [10] "subfield_order"      "subfield_occurrence"

The columns are record_id, field_type, tag, subfield_code, value, field_order, field_occurrence, ind1, ind2, subfield_order, and subfield_occurrence.

record_id, field_order, and subfield_order locate values in the represented MARC structure. field_occurrence and subfield_occurrence are derived analytical coordinates that make repetitions easier to address.

Inspect repeated structure

The first record contains an 856 field with two $y subfields. Their source order and separate occurrences remain explicit:

marc |>
  filter(record_id == 1L, tag == "856") |>
  select(
    subfield_code,
    value,
    subfield_order,
    subfield_occurrence
  )
#> # A tibble: 3 × 4
#>   subfield_code value                    subfield_order subfield_occurrence
#>   <chr>         <chr>                             <int>               <int>
#> 1 u             https://example.org/1984              1                   1
#> 2 y             Full text                             2                   1
#> 3 y             Mirror                                3                   2

Repeated fields are preserved in the same way. The first record contains two 650 fields:

marc |>
  filter(record_id == 1L, tag == "650", subfield_code == "a") |>
  select(record_id, field_order, field_occurrence, value)
#> # A tibble: 2 × 4
#>   record_id field_order field_occurrence value          
#>       <int>       <int>            <int> <chr>          
#> 1         1           5                1 Totalitarianism
#> 2         1           6                2 Dystopias.

This is why marcxmlr does not flatten MARCXML directly to one row per bibliographic record: repeated field and subfield relationships remain explicit until the analyst chooses how to simplify them.

Query with dplyr

Because the canonical representation is a tibble, ordinary dplyr operations can be used directly. For example, title fields can be selected by MARC tag:

marc |>
  filter(tag == "245") |>
  select(record_id, subfield_code, value)
#> # A tibble: 2 × 3
#>   record_id subfield_code value                 
#>       <int> <chr>         <chr>                 
#> 1         1 a             Nineteen eighty-four /
#> 2         1 c             George Orwell.

A condition can also select a complete field instance rather than only the row that matched. Rows with the same record_id and field_order belong to the same field instance:

marc |>
  group_by(record_id, field_order) |>
  filter(
    any(
      tag == "650" &
        subfield_code == "a" &
        value == "Totalitarianism"
    )
  ) |>
  ungroup() |>
  select(record_id, field_order, tag, subfield_code, value)
#> # A tibble: 2 × 5
#>   record_id field_order tag   subfield_code value          
#>       <int>       <int> <chr> <chr>         <chr>          
#> 1         1           5 650   a             Totalitarianism
#> 2         1           5 650   v             Fiction.

Derive a simpler analytical table

The canonical representation preserves MARC structure first. A simpler table can then be derived for a particular analysis.

titles <- marc |>
  filter(tag == "245", subfield_code == "a") |>
  transmute(record_id, title = value)

subjects <- marc |>
  filter(tag == "650") |>
  group_by(record_id, field_order) |>
  summarise(
    subject = paste(value, collapse = " ; "),
    .groups = "drop"
  ) |>
  group_by(record_id) |>
  summarise(
    subjects = paste(subject, collapse = " | "),
    .groups = "drop"
  )

left_join(titles, subjects, by = "record_id")
#> # A tibble: 1 × 3
#>   record_id title                  subjects                                     
#>       <int> <chr>                  <chr>                                        
#> 1         1 Nineteen eighty-four / Totalitarianism ; Fiction. | Dystopias. ; Fi…

This simplification is deliberately one-way. Once field membership or repetition has been collapsed, reconstructing the original MARC structure may no longer be possible.

Edit a repeated subfield

A specific repeated value can be addressed with the occurrence coordinates. Here only the second 856$y value in the first record is changed:

edited <- marc |>
  mutate(
    value = if_else(
      record_id == 1L &
        tag == "856" &
        subfield_code == "y" &
        subfield_occurrence == 2L,
      "Backup access",
      value
    )
  )

edited |>
  filter(record_id == 1L, tag == "856") |>
  select(subfield_code, value, subfield_occurrence)
#> # A tibble: 3 × 3
#>   subfield_code value                    subfield_occurrence
#>   <chr>         <chr>                                  <int>
#> 1 u             https://example.org/1984                   1
#> 2 y             Full text                                  1
#> 3 y             Backup access                              2

Only value has changed; the structural coordinates remain intact.

Check before writing

After transformations that may affect MARC structure, diagnose_canonical() can be used to inspect whether the representation is still safe to serialize.

diagnostics <- diagnose_canonical(edited)
diagnostics
#> # A tibble: 0 × 6
#> # ℹ 6 variables: severity <chr>, code <chr>, message <chr>, record_id <int>,
#> #   field_order <int>, subfield_order <int>

A zero-row diagnostics tibble means that no issue was found. Errors identify structural problems that prevent safe serialization. Warnings can identify stale or gapped analytical occurrence coordinates that can be regenerated on reread.

diagnose_canonical() checks the structural contract used by marcxmlr; it is not a complete MARC 21 content validator and does not replace MARCXML schema validation.

Write MARCXML and read it back

write_marcxml() serializes coherent canonical data back to MARCXML. Structural errors are rejected by the writer; with the default check = TRUE, warning-level diagnostics are also emitted before writing.

out_xml <- tempfile(fileext = ".xml")
write_marcxml(edited, out_xml)

roundtrip <- read_marcxml(out_xml)
identical(edited, roundtrip)
#> [1] TRUE

A related workflow is to select complete records and export the result:

selected <- marc |>
  filter(record_id == 1L)

selected_xml <- tempfile(fileext = ".xml")
write_marcxml(selected, selected_xml)

identical(selected, read_marcxml(selected_xml))
#> [1] TRUE

The round trip preserves the represented MARC structure and values. It does not promise byte-for-byte reproduction of incidental XML serialization details such as indentation or namespace-prefix spelling.

Recover from malformed MARC records

By default, parsing stops when a structurally malformed MARC record is encountered. Alternatively, on_marc_error = "skip" allows processing to continue while skipping complete malformed records:

recovered <- read_marcxml(
  "catalogue-with-malformed-records.xml",
  on_marc_error = "skip",
  error_report = "marc-errors.csv"
)

An error report is required in skip mode. It identifies skipped records, explains the problem, and preserves their original XML. The same recovery options are available in marcxml_to_parquet().

Skipped records leave gaps in the original record_id sequence. Such gaps may subsequently be reported as warnings by diagnose_canonical().

Recovery applies only to MARC-level structural problems. XML syntax and encoding errors remain fatal.

Move the same schema to Parquet

For collections that are inconvenient to materialize as one R object, marcxml_to_parquet() writes the same canonical schema as a directory of Parquet files in bounded record batches. Arrow support is optional, so the following examples run only when the optional packages needed for this workflow are available.

parquet_dir <- tempfile("marcxmlr-parquet-")

marcxml_to_parquet(
  source_xml,
  output_dir = parquet_dir,
  workers = 1L,
  verbose = FALSE
)

catalogue <- arrow::open_dataset(parquet_dir)

catalogue |>
  summarise(rows = n()) |>
  collect()
#> # A tibble: 1 × 1
#>    rows
#>   <int>
#> 1    16

The Parquet dataset can be queried lazily with Arrow and dplyr before a small result is collected into R:

catalogue |>
  filter(tag == "650", subfield_code == "a") |>
  count(value, sort = TRUE) |>
  collect()
#> # A tibble: 2 × 2
#>   value               n
#>   <chr>           <int>
#> 1 Totalitarianism     1
#> 2 Dystopias.          1

Opening the dataset is lazy; collecting the complete dataset would still bring all rows into R memory.

The same structural diagnostics can run directly on an Arrow dataset:

diagnose_canonical(catalogue)
#> # A tibble: 0 × 6
#> # ℹ 6 variables: severity <chr>, code <chr>, message <chr>, record_id <int>,
#> #   field_order <int>, subfield_order <int>

Validation proceeds incrementally without collecting the entire dataset. Records must appear in non-decreasing record_id order.

Choosing a workflow

For ordinary analysis, start with read_marcxml() and work with the returned tibble. Use diagnose_canonical() after transformations that may affect structure, and write_marcxml() when the result needs to return to MARCXML.

Use marcxml_to_parquet() when keeping the complete canonical table in memory is undesirable or when a persistent Arrow/Parquet dataset is useful for repeated analytical queries. The logical 11-column representation is the same in both workflows.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.