Classify, compare, and resume

Preserve every membership

path <- tempfile(fileext = ".csv")
write.csv(data.frame(
  domain = c("example.com", "example.com", "university.example"),
  category = c("news", "shopping", "education"),
  usage = c("black", "black", "white")
), path, row.names = FALSE)
results <- ut1_cat(c("https://example.com/article", "missing.example"), use_file = path)
results[c("input_id", "input", "provider", "status", "category")]

## # A tibble: 2 × 5
##   input_id input                       provider status   category
##      <int> <chr>                       <chr>    <chr>    <chr>   
## 1        1 https://example.com/article ut1      ok       <NA>    
## 2        2 missing.example             ut1      no_match <NA>

results$labels[[1]]

## # A tibble: 2 × 3
##   category score vendor
##   <chr>    <dbl> <chr> 
## 1 news        NA <NA>  
## 2 shopping    NA <NA>

fetch_report(results)

## # A tibble: 1 × 8
##       n    ok failed no_match retryable by_stage         by_reason missing  
##   <int> <int>  <int>    <int>     <int> <list>           <list>    <list>   
## 1     2     1      0        1         0 <tibble [0 × 2]> <tibble>  <chr [0]>

The first domain has two memberships, so no single category is selected. The second has no match, which is different from a failed download. UT1 usage and raw records remain in details. The fixture has no verified metadata, so its dates are unknown.

Preserve identifiers when resuming

pending <- which(results$retryable)
retried <- ut1_cat(results$input[pending], use_file = path)
retried$input_id <- results$input_id[pending]
results[pending, ] <- retried
stopifnot(nrow(results) == 2L, nrow(retried) == 0L)
unlink(path)

For a paid provider, save completed chunks with saveRDS() and retry only failed inputs using the same provider, model, and vocabulary. Keep original identifiers when combining chunks. A result’s no_match state does not call for a retry.

Distinguish page state from content classification

page_signals("<html><body>This domain is for sale. Inquire now.</body></html>")

## # A tibble: 1 × 8
##   page_state blocked block_vendor block_reason parked unavailable thin  n_tokens
##   <chr>      <lgl>   <chr>        <chr>        <lgl>  <lgl>       <lgl>    <int>
## 1 parked     FALSE   <NA>         <NA>         TRUE   FALSE       TRUE         7

news_url_signals(c("https://example.com/sport", "https://example.com/politics"))

## # A tibble: 2 × 4
##   input_id input                        soft_news hard_news
##      <int> <chr>                        <lgl>     <lgl>    
## 1        1 https://example.com/sport    TRUE      FALSE    
## 2        2 https://example.com/politics FALSE     TRUE

Page-state detection is heuristic. URL keyword flags are neither mutually exclusive nor a substitute for reading the page. The content classifier uses Python’s own HTML extraction and its pinned model weights. Its documented quality limitations still apply.

reticulate::py_require("piedomains==0.15.0", python_version = ">=3.12")
pages <- collect_content("https://example.com/article")
classified <- pie_cat(pages = pages)
classified$details[[1]][c("fetched_at", "source", "snapshot_timestamp", "fetch_status")]

Historical collection uses the exact URL and the most recent capture on or before archive_date. Current collection does not use archived fallback unless requested.