The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

Package {safeconduct}


Type: Package
Title: Fetch Files Through a Real Browser When a Plain Request is Refused
Version: 0.1.0
Description: Some servers refuse any request that does not come from a web browser, with a browser check or a block page instead of the file, 'Cloudflare' being the common case. This package opens a real 'Chrome' with a throwaway profile, lets any check run, and fetches the file or page: through 'curl' carrying the clearance the browser was given where the server accepts that, and through the browser itself where it does not. Nothing is simulated; where a check asks a person to click, a window waits for them. Built on 'chromote'.
License: MIT + file LICENSE
Encoding: UTF-8
Language: en-CA
SystemRequirements: Google Chrome or another Chromium-based browser (<https://www.google.com/chrome/>).
Depends: R (≥ 4.1)
Imports: chromote (≥ 0.3.0), curl (≥ 5.0), jsonlite, processx
Suggests: knitr, rmarkdown, testthat (≥ 3.0.0), withr, xml2
VignetteBuilder: knitr
URL: https://mountainmath.github.io/safeconduct/, https://github.com/mountainMath/safeconduct
BugReports: https://github.com/mountainMath/safeconduct/issues
Config/testthat/edition: 3
Config/roxygen2/version: 8.1.0
NeedsCompilation: no
Packaged: 2026-11-02 04:23:37 UTC; jens
Author: Jens von Bergmann [aut, cre]
Maintainer: Jens von Bergmann <jens@mountainmath.ca>
Repository: CRAN
Date/Publication: 2026-11-20 11:50:02 UTC

safeconduct: Fetch Files Through a Real Browser When a Plain Request is Refused

Description

logo

Some servers refuse any request that does not come from a web browser, with a browser check or a block page instead of the file, 'Cloudflare' being the common case. This package opens a real 'Chrome' with a throwaway profile, lets any check run, and fetches the file or page: through 'curl' carrying the clearance the browser was given where the server accepts that, and through the browser itself where it does not. Nothing is simulated; where a check asks a person to click, a window waits for them. Built on 'chromote'.

Options

Behaviour switches are function arguments, so that nothing leaks between packages using safeconduct in one R session; the options are these. safeconduct.browser = FALSE never opens a browser. safeconduct.window sets the window's size, c(width, height) in pixels (1000 by 800 by default) or "maximized". Experimentally, safeconduct.focus, a CSS selector, pins the element holding the check to the window's top left corner, enlarged by safeconduct.zoom (1.5), and hides the rest of the page.

Author(s)

Maintainer: Jens von Bergmann jens@mountainmath.ca

Authors:

See Also

Useful links:


Can this package open a browser here?

Description

Can this package open a browser here?

Usage

browser_available()

Value

TRUE, or FALSE with a reason attribute: the safeconduct.browser option is FALSE, or no Chrome or Chromium was found (the CHROMOTE_CHROME environment variable points at one).

Examples

browser_available()
attr(browser_available(), "reason")

Keep a browser open on a host

Description

Opens a Chrome with a throwaway profile on the host's root, lets any check run, and keeps it open for read_page() and fetch() through their session argument. Close it with close(); otherwise it lives until the R session ends. A browser that has gone away, its window closed, is reopened the same way at the next call, which says so.

Usage

browser_session(
  url,
  quiet = FALSE,
  wait = NULL,
  headed = "auto",
  useragent = TRUE
)

Arguments

url

A URL on the host. headed and useragent hold for the session's life.

quiet

Suppress progress messages.

wait

Seconds to wait for the check to pass: by default two minutes with a window, so that a person can click, and thirty seconds headless.

headed

How to run the browser when one is needed. "auto", the default, runs headless and opens a window only when the headless run ends on a check or a block and there is a display to show it on; TRUE opens a window at once; FALSE never does. A second check, at the file's URL, takes headless Chrome about two minutes to pass unclicked, where a window is through as soon as its box is clicked: "auto" waits those minutes before opening a window, TRUE skips the wait.

useragent

The browser's user-agent string. TRUE, the default, drops the word Headless from Chrome's own string, which some hosts refuse on that word alone, and changes nothing else; NULL leaves Chrome's own; a string is used as given.

Value

A safeconduct_session, whose clearance is what clearance() would return. Errors as fetch()'s.

Examples

## Not run: 
s <- browser_session("https://results.vancouver.ca/")
r <- read_page("https://results.vancouver.ca/data/arearesults.json",
               session = s)
close(s)

## End(Not run)

Get a clearance for a host from a browser

Description

Opens a Chrome on the host, lets any check run, and returns the browser's user-agent string and cookies, which clearance_handle() puts on a curl request. Whether the server then serves curl is the server's call: after a block page it usually does not, since the rule is about the client, and fetch() then downloads in the browser instead. Clearances are cached per host for the R session.

Usage

clearance(
  url,
  refresh = FALSE,
  quiet = FALSE,
  wait = NULL,
  headed = "auto",
  useragent = TRUE
)

Arguments

url

The file's URL.

refresh

Get a new clearance even when one is cached.

quiet

Suppress progress messages.

wait

Seconds to wait for the check to pass: by default two minutes with a window, so that a person can click, and thirty seconds headless.

headed

How to run the browser when one is needed. "auto", the default, runs headless and opens a window only when the headless run ends on a check or a block and there is a display to show it on; TRUE opens a window at once; FALSE never does. A second check, at the file's URL, takes headless Chrome about two minutes to pass unclicked, where a window is through as soon as its box is clicked: "auto" waits those minutes before opening a window, TRUE skips the wait.

useragent

The browser's user-agent string. TRUE, the default, drops the word Headless from Chrome's own string, which some hosts refuse on that word alone, and changes nothing else; NULL leaves Chrome's own; a string is used as given.

Value

A safeconduct_clearance: a list with host, useragent, cookie (one ⁠Cookie:⁠ header value) and time. Errors as fetch()'s.

Examples

## Not run: 
cl <- clearance("https://www12.statcan.gc.ca/")
curl::curl_fetch_memory("https://www12.statcan.gc.ca/", clearance_handle(cl))

## End(Not run)

A curl handle that carries a clearance

Description

The user-agent string and the cookies go on a request together: Cloudflare ties the clearance cookie to the user agent that earned it, and the cookie alone is answered with the check again.

Usage

clearance_handle(clearance = NULL, ...)

Arguments

clearance

A clearance from clearance(), or NULL for a plain handle.

...

Further options for curl::new_handle().

Value

A curl handle with followlocation set, and the clearance's useragent and cookie when one was given.

Examples

h <- clearance_handle(list(useragent = "Mozilla/5.0", cookie = "cf_clearance=x"))

Download a file in a browser

Description

Opens a Chrome with a throwaway profile on the host, lets any check run, and downloads url in it, with no plain request first. fetch() does this only when it has to.

Usage

download_in_browser(
  url,
  destfile,
  quiet = FALSE,
  timeout = 3600,
  headed = "auto",
  useragent = TRUE
)

Arguments

url

The file's URL.

destfile

Where to write it. A failed transfer leaves nothing there.

quiet

Suppress progress messages.

timeout

Seconds a transfer in the browser may take.

headed

How to run the browser when one is needed. "auto", the default, runs headless and opens a window only when the headless run ends on a check or a block and there is a display to show it on; TRUE opens a window at once; FALSE never does. A second check, at the file's URL, takes headless Chrome about two minutes to pass unclicked, where a window is through as soon as its box is clicked: "auto" waits those minutes before opening a window, TRUE skips the wait.

useragent

The browser's user-agent string. TRUE, the default, drops the word Headless from Chrome's own string, which some hosts refuse on that word alone, and changes nothing else; NULL leaves Chrome's own; a string is used as given.

Value

destfile, invisibly. Errors as fetch()'s.

Examples

## Not run: 
download_in_browser("https://example.org/large.zip", "large.zip")

## End(Not run)

Download a file, through a browser if the server insists on one

Description

Fetches url to destfile. A plain request is made first, and if the file comes no browser is involved. If the server answers with a browser check (is_challenge()) or a block page (is_block()), a Chrome with a throwaway profile is opened on the host, any check runs, and the file is downloaded: through curl carrying the browser's clearance where the server accepts that, and in the browser itself where it does not. A clearance curl can use is kept for the rest of the R session, so later files from the host need no browser.

Usage

fetch(
  url,
  destfile,
  quiet = FALSE,
  timeout = 3600,
  session = NULL,
  headed = "auto",
  useragent = TRUE
)

Arguments

url

The file's URL.

destfile

Where to write it. A failed transfer leaves nothing there.

quiet

Suppress progress messages.

timeout

Seconds a transfer in the browser may take.

session

A browser_session(). The file is then downloaded in that browser without probing, with the session's headed and useragent, and the browser stays open; one that has gone away is reopened first.

headed

How to run the browser when one is needed. "auto", the default, runs headless and opens a window only when the headless run ends on a check or a block and there is a display to show it on; TRUE opens a window at once; FALSE never does. A second check, at the file's URL, takes headless Chrome about two minutes to pass unclicked, where a window is through as soon as its box is clicked: "auto" waits those minutes before opening a window, TRUE skips the wait.

useragent

The browser's user-agent string. TRUE, the default, drops the word Headless from Chrome's own string, which some hosts refuse on that word alone, and changes nothing else; NULL leaves Chrome's own; a string is used as given.

Details

Nothing is simulated. A check that shows a "Verify you are human" box waits in a window, for up to two minutes, for a person to click it. The window is brought to the front when a check shows in it, and wants to be left there until it is through: a check run with the window behind something else is reissued on every click. A file downloaded in the browser lands under tempdir() and is moved to destfile when complete.

Value

destfile, invisibly. Errors are classed under safeconduct_error: safeconduct_unavailable when no browser can be opened here, safeconduct_not_passed when the check was not passed in time or the server blocks the browser too, with a reason field, and safeconduct_download_error when the transfer failed.

Requirements

Chrome or Chromium, found by chromote::find_chrome(); the CHROMOTE_CHROME environment variable points at one somewhere unusual, and browser_available() says whether one can be opened here. options(safeconduct.browser = FALSE) never opens a browser; the window's size and the other options are described in safeconduct-package.

Tips

At the console, once a host has been found to need a window or the browser for the transfer, a tip says how to skip the finding out next time: headed = TRUE, or a browser_session(). Tips come once per host and R session, not when quiet, and not for calls made from another package. They are messages of class safeconduct_hint.

Examples

## Not run: 
url <- paste0("https://www12.statcan.gc.ca/census-recensement/2021/",
              "geo/sip-pis/rnf-frr/files-fichiers/lrnf000r21a_e.zip")
fetch(url, "lrnf000r21a_e.zip")

## End(Not run)

Is a response a block page that a browser would not be shown?

Description

Cloudflare's firewall can refuse a client that does not look like a browser outright, with a 403 and its own block page, while a real browser is served.

Usage

is_block(status, headers, body = NULL)

Arguments

status

The HTTP status code.

headers

The response headers, as the raw header block (one string, lines separated by ⁠\r\n⁠) or as a named list such as curl::parse_headers_list() makes of it.

body

The response body, as a string.

Value

TRUE for that page: a 403 from server: cloudflare whose body says "Sorry, you have been blocked".

Examples

is_block(403, "HTTP/2 403\r\nserver: cloudflare\r\n\r\n",
         "<html><title>Attention Required! | Cloudflare</title>...")
is_block(403, "HTTP/2 403\r\nserver: nginx\r\n\r\n", "Forbidden")

Does a response carry a browser check instead of the resource?

Description

Does a response carry a browser check instead of the resource?

Usage

is_challenge(headers)

Arguments

headers

The response headers, as the raw header block (one string, lines separated by ⁠\r\n⁠) or as a named list such as curl::parse_headers_list() makes of it.

Value

TRUE for a check a real browser would pass: Cloudflare's, by its cf-mitigated: challenge header.

See Also

is_block() for the other refusal a browser gets past.

Examples

is_challenge("HTTP/2 403\r\ncf-mitigated: challenge\r\n\r\n")
is_challenge("HTTP/2 200\r\ncontent-type: application/zip\r\n\r\n")

Parse a response from read_page()

Description

page_text() decodes the body to a UTF-8 string, page_json() parses it with jsonlite::fromJSON(), and page_html() with xml2::read_html().

Usage

page_text(resp, encoding = NULL, check = TRUE)

page_json(resp, ..., check = TRUE)

page_html(resp, encoding = NULL, check = TRUE)

Arguments

resp

A response from read_page(), or a URL, which is then read with read_page()'s defaults, so that page_html(url) stands in for rvest::read_html(url).

encoding

The body's charset, overriding the response's.

check

Whether an HTTP status outside 200–299 is an error.

...

Passed on to jsonlite::fromJSON().

Details

The charset comes from the response's content-type; without one, page_text() and page_json() assume UTF-8 and page_html() leaves it to the page's own ⁠<meta charset>⁠. ISO-8859-1 is read as windows-1252, as browsers do. A body that is not valid in its charset is an error. A "not found" page served with status 200 passes the status check; page_json() refuses an HTML body, and for a page, compare resp$url with the URL asked for.

Value

page_text(): a UTF-8 string. page_json(): what jsonlite::fromJSON() returns. page_html(): an xml_document whose relative links resolve against resp$url. Errors are classed under safeconduct_error: safeconduct_http_error for a status outside 200–299, carrying status and url, and safeconduct_parse_error for a body that cannot be decoded, or HTML where JSON was asked for.

Examples

## Not run: 
page_json("https://results.vancouver.ca/data/arearesults.json")
page_html("https://www12.statcan.gc.ca/census-recensement/index-eng.cfm")

## End(Not run)

Ask a server what it serves at a URL

Description

Sends a GET for the first byte, so that the answer carries the resource's headers without the resource; a HEAD is not used because some servers answer it differently.

Usage

probe(url, clearance = NULL)

Arguments

url

The URL.

clearance

A clearance from clearance(), to probe as the browser that earned it; NULL probes as a plain client.

Value

A list: reachable, status (NA when unreachable), challenged (is_challenge()), blocked (is_block()), type (the content-type, lower case), size (bytes, from content-range or content-length, NA when unknown) and headers (a named list). A browser is called for when challenged or blocked is TRUE.

Examples

## Not run: 
probe("https://www12.statcan.gc.ca/census-recensement/index-eng.cfm")

## End(Not run)

Read a page or a feed into memory, through a browser if the server insists

Description

fetch() for a page or a feed: the response is returned rather than written to disk, shaped like curl::curl_fetch_memory()'s, by the same route; where only the browser is served, the page the browser holds on the host reads it. For a feed read every few minutes, pass a browser_session() instead of opening a browser per call; a check the host asks for again meanwhile is passed and the read repeated.

Usage

read_page(
  url,
  session = NULL,
  quiet = FALSE,
  timeout = 120,
  headed = "auto",
  useragent = TRUE
)

Arguments

url

The file's URL.

session

A browser_session() on the host. The page is then read in that browser without probing.

quiet

Suppress progress messages.

timeout

Seconds a read in the browser may take.

headed

How to run the browser when one is needed. "auto", the default, runs headless and opens a window only when the headless run ends on a check or a block and there is a display to show it on; TRUE opens a window at once; FALSE never does. A second check, at the file's URL, takes headless Chrome about two minutes to pass unclicked, where a window is through as soon as its box is clicked: "auto" waits those minutes before opening a window, TRUE skips the wait.

useragent

The browser's user-agent string. TRUE, the default, drops the word Headless from Chrome's own string, which some hosts refuse on that word alone, and changes nothing else; NULL leaves Chrome's own; a string is used as given.

Value

A list: url (after redirects), status_code, type (the content-type, lower case), headers (a named list) and content (raw). An HTTP error status is returned, not raised. Parse it with page_text(), page_json() or page_html(). Errors as fetch()'s.

Examples

## Not run: 
r <- read_page("https://results.vancouver.ca/data/arearesults.json")
page_json(r)

## End(Not run)

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.