The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

tokenizers.bpe - R package for Byte Pair Encoding

This repository contains an R package which is an Rcpp wrapper around the YouTokenToMe C++ library

• YouTokenToMe is an unsupervised text tokenizer focused on computational efficiency
• It currently implements fast Byte Pair Encoding (BPE) [Sennrich et al.]
• YouTokenToMe is available at https://github.com/VKCOM/YouTokenToMe
• Note that the flat_hash_map used in YouTokenToMe was replaced by parallel-hashmap

Features

The R package allows you to

• build a Byte Pair Encoding (BPE) model
• apply the model to encode text
• apply the model to decode ids back to text

Installation

• For regular users, install the package from your local CRAN mirror install.packages("tokenizers.bpe")
• For installing the development version of this package: remotes::install_github("bnosac/tokenizers.bpe")

Look to the documentation of the functions

help(package = "tokenizers.bpe")

Example

• As an example, let’s take some training data containing questions asked in Belgian Parliament in 2017 and focus on French text only.

library(tokenizers.bpe)
data(belgium_parliament, package = "tokenizers.bpe")
x <- subset(belgium_parliament, language == "french")
writeLines(text = x$text, con = "traindata.txt")

• Train a model on text data and inspect the vocabulary

model <- bpe("traindata.txt", coverage = 0.999, vocab_size = 5000)
model

Byte Pair Encoding model trained with YouTokenToMe
  size of the vocabulary: 5000
  model stored at: C:/Users/Jan/Dropbox/Work/RForgeBNOSAC/BNOSAC/tokenizers.bpe/youtokentome.bpe

str(model$vocabulary)

'data.frame':   5000 obs. of  2 variables:
 $ id     : int  0 1 2 3 4 5 6 7 8 9 ...
 $ subword: chr  "<PAD>" "<UNK>" "<BOS>" "<EOS>" ...

• Use the model to encode text

text <- c("L'appartement est grand & vraiment bien situe en plein centre",
          "Proportion de femmes dans les situations de famille monoparentale.")
bpe_encode(model, x = text, type = "subwords")

[[1]]
 [1] "▁L'"     "app"     "ar"      "tement"  "▁est"    "▁grand"  "▁"       "&"       "▁v"      "r"       "ai"      "ment"    "▁bien"   "▁situe"  "▁en"     "▁plein"  "▁centre"

[[2]]
 [1] "▁Pro"        "por"         "tion"        "▁de"         "▁femmes"     "▁dans"       "▁les"        "▁situations" "▁de"         "▁famille"    "▁mon"        "op"          "ar"          "ent"         "ale." 

bpe_encode(model, x = text, type = "ids")

[[1]]
 [1]  421  327   98  554  178 1521    4    1  117   11  101   99  679 4599  113 3702 2126

[[2]]
 [1] 1529 4878   92   76 2321  162  108 4099   76 3218  791  312   98   87 2546

• Use the model to decode byte pair encodings back to text

x <- bpe_encode(model, x = text, type = "ids")
bpe_decode(model, x)

[[1]]
[1] "L'appartement est grand <UNK> vraiment bien situe en plein centre"

[[2]]
[1] "Proportion de femmes dans les situations de famille monoparentale."

Support in text mining

Need support in text mining? Contact BNOSAC: http://www.bnosac.be

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.