The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

rdomains 0.5.0

A label has a date, and now the package says so

Two of the four lookup sources are no longer published: DMOZ closed in March 2017 and Shallalist stopped in 2022. Their labels were correct when assigned, but domains expire and change hands, so a lookup today can return the previous registrant’s category. Until now that answer was presented identically to one from a list updated last week.

• New source_vintage() reports every category source, when it was last published, whether it is still maintained, and its successor where one exists.
• shalla_cat(), dmoz_cat() and stevenblack_cat() now return a source_last_published column. For the two dead lists this is a constant; for Steven Black’s actively-maintained hosts file it is the fetched file’s own date.
• Looking up against a discontinued source warns once per session — not once per call, which is a warning people learn to filter out.

The sibling project piedomains measured what this confusion costs: its worst class disagreed with its own page content 71% of the time, and the cause was not bad annotation but roughly a decade between the label and the page. Only 60% of the domains it trained on still resolve.

Fetch page content, not just look domains up

New collect_content() fetches homepage HTML and text, so a domain can be classified on what it says today rather than on what a list said years ago. It returns one row per requested domain, never dropped, each carrying status, stage, error_code and retryable – so a transient failure is distinguishable from a permanent one and only the right rows get retried. fetch_error_codes() documents the closed set of reasons; fetch_report() summarises a run.

Supporting functions, all usable on their own if you already hold HTML:

• page_signals() reports whether a page is an anti-bot interstitial, a domain-parking placeholder, a server’s “nothing here” page, or too thin to classify. Vendor presence alone is not a block: reddit, walmart and quora all serve real pages while embedding reCAPTCHA.
• html_text_content() extracts text, title, description and language.

The crawler identifies itself as rdomains/<version> with a contact URL, obeys robots.txt including Crawl-delay, spaces requests to the same host, caps the response body, follows redirects by hand so every hop is re-validated, and refuses hosts resolving to private or link-local addresses.

Static HTML only – no headless browser, so a JavaScript-rendered page comes back thin and says so.

Bug fixes

• get_dmoz_data() built its output path with paste0(), so the documented default outdir = "." produced a hidden .dmoz_domain_category.csv that dmoz_cat() would then fail to find. It now uses file.path(), matching its two siblings.
• stevenblack_cat(use_file = NULL) re-downloaded roughly 4 MB on every call. The file is now cached for the session.

Testing

• The Shallalist and DMOZ tests download real files; they now carry skip_on_cran() and skip_if_offline() guards, as CRAN policy requires.

rdomains 0.4.0

Breaking Changes

• Removed get_alexa_data() function (Alexa service discontinued by Amazon)

Major Changes

• Removed unused aws.alexa dependency
• Removed devtools from Imports (incorrect usage)
• Added modern tidyverse-style API with comprehensive input validation
• Significant code deduplication through shared helper functions

API Updates

• Updated virustotal_cat() to use VirusTotal API v3 (previously v2.0)
• Updated documentation references to v3 API endpoints
• Fixed virustotal_cat() implementation to properly extract categories from v3 API response structure

Improvements

• All categorization functions now validate inputs with helpful error messages using cli package
• Standardized parameter naming (virustotal_cat now uses ‘domains’ instead of ‘domain’)
• Better error messages with clear guidance on how to fix issues
• Modernized code style (pipes, purrr, tibble internally with data.frame output for compatibility)
• Improved file path handling with informative errors
• Enhanced rate limiting in LLM functions
• Cleaner domain preprocessing logic shared across all functions

Internal Changes

• Added helper functions for common operations:
  • clean_domains() - standardized domain cleaning
  • validate_domains() - comprehensive input validation
  • validate_data_file() - consistent file validation
  • get_api_key() - unified API key retrieval
  • build_categorization_prompt() - LLM prompt construction
  • apply_rate_limit() - rate limiting logic
• Refactored to use purrr instead of for-loops where appropriate
• All functions now return tibbles for modern data handling
• Added checkmate for robust input validation
• Added readr for faster CSV reading
• Extracted domain cleaning logic to single function
• Improved string operations with stringr
• Removed redundant :: notation for imported functions (cleaner code, consistent with @importFrom)

Breaking Changes

• All categorization functions now return tibbles instead of data.frames
• get_alexa_data() has been removed (service discontinued)
• Input validation is now stricter (NULL and empty strings are properly rejected)
• virustotal_cat() parameter renamed from domain to domains for consistency

rdomains 0.3.0

• NEW: Added LLM-based domain classification with openai_cat() and claude_cat() functions
• Support for OpenAI GPT models and Anthropic Claude models for domain categorization
• Flexible custom category schemas - users can specify their own categories or use defaults
• Consistent API design matching existing *_cat() functions for seamless integration
• Built-in rate limiting and error handling for API calls
• REMOVED: BrightCloud support due to service unavailability
• Updated documentation URLs from HTTP to HTTPS where applicable
• Fixed Shallalist references to reflect service discontinuation

rdomains 0.2.1

• shallalist stopped its service so downloaded latest shalla db and changed the URL from which we fetch the shallalist file

rdomains 0.2.0

• URL fixes. in resubmission now because site from which data was downloaded went down which broke some tests

rdomains 0.1.9

• R package supporting headless browsing has been abandoned. So removing trusted_cat. Sigh.

rdomains 0.1.8

• Function for checking if domain a university domain using https://github.com/Hipo/university-domains-list

rdomains 0.1.7

• Changes due to move to a new repo.
• Basic brightcloud function added

rdomains 0.1.6

• Adds not_news classifier that classifies not news based on published work.
• passes expect_lint_free

rdomains 0.1.5

• Shallalist and DMOZ data read in with stringAsFactors as FALSE.
• Swapped the DMOZ data to domain level category data, included English translations of non-English categories, quote protection of multiple categories.
• Accounting for changes in RSelenium — startServer() for instance is deprecated. But currently only allow for passing of log for trusted_cat.
• Fixed bug in shalla_cat for multiple domain names arguments
• Fixed small issue with adult_ml1_cat() whose returned data.frame had a column that was a named list. The column is now a vector.
• If an unknown domain is passed to virustotal, it will return an empty data.frame rather than throw an error.

rdomains 0.1.0

• Initial release

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.