The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.
Text classification assigns documents to predefined categories. In organizational research, documents might be vacancy sentences, employee comments, reports, or interview excerpts. This tutorial develops a transparent workflow from raw HTML to predictions.
The package includes the nursing-vacancy page used by the original tutorial.
html_file <- system.file(
"extdata", "sample_nursing_vacancy.html",
package = "textclassificationtutorial"
)
vacancy_text <- extract_html_text(html_file)
substr(vacancy_text, 1, 200)
#> [1] "Als einer der führenden privaten Träger im Bereich der stationären Pflege bietet CASA REHA Ihnen zukunfts- und krisensichere Arbeitsplätze. Über 6500 Mitarbeiter versorgen rund um die Uhr unsere Bewoh"To process a folder, use extract_html_dir(). The result
keeps a document ID, the source path, and extracted text together.
CSS and XPath selection are available when xml2 is
installed:
The unit of analysis should follow the research question. Here, each sentence is treated as one document.
sentences <- split_sentences(vacancy_text)
head(sentences)
#> [1] "Als einer der führenden privaten Träger im Bereich der stationären Pflege bietet CASA REHA Ihnen zukunfts- und krisensichere Arbeitsplätze."
#> [2] "Über 6500 Mitarbeiter versorgen rund um die Uhr unsere Bewohner und leben die CASA REHA-Philosophie – \"von Mensch zu Mensch\"."
#> [3] "Bereits zum zweiten Mal in Folge wurden wir als einer der besten Arbeitgeber Deutschlands im Bereich \"Gesundheit & Soziales\" ausgezeichnet."
#> [4] "Jede unserer fast 70 Einrichtungen hat einen einzigartigen Stil, der die Besonderheiten des Standorts widerspiegelt."
#> [5] "Allen gemein sind eine familiäre Atmosphäre, ein modernes Arbeitsumfeld und hohe Qualitätsstandards."
#> [6] " "Preprocessing choices are analytical decisions, not housekeeping. Removing numbers may discard years of experience, and removing stopwords may discard meaningful negation. Make each choice explicit.
german_stopwords <- c(
"der", "die", "das", "den", "dem", "des", "und", "oder", "mit",
"für", "von", "zu", "im", "in", "auf", "ein", "eine"
)
clean <- preprocess_text(
sentences,
lowercase = TRUE,
remove_punctuation = TRUE,
remove_numbers = TRUE,
stopwords = german_stopwords,
min_token_length = 2
)
clean <- clean[nzchar(clean)]
head(clean)
#> [1] "als einer führenden privaten träger bereich stationären pflege bietet casa reha ihnen zukunfts krisensichere arbeitsplätze"
#> [2] "über mitarbeiter versorgen rund um uhr unsere bewohner leben casa reha philosophie mensch mensch"
#> [3] "bereits zum zweiten mal folge wurden wir als einer besten arbeitgeber deutschlands bereich gesundheit soziales ausgezeichnet"
#> [4] "jede unserer fast einrichtungen hat einen einzigartigen stil besonderheiten standorts widerspiegelt"
#> [5] "allen gemein sind familiäre atmosphäre modernes arbeitsumfeld hohe qualitätsstandards"
#> [6] "seniorenpflegeheim rosenpark hemmingen bieten wir bewohnern zuhause"dtm <- document_term_matrix(
clean,
min_doc_freq = 2,
max_doc_prop = 0.95
)
dtm
#> <text_dtm> 41 documents x 45 terms
#> als an andreas auch baumert bei bereich betreuung bewohner bewohnern
#> doc1 1 0 0 0 0 0 1 0 0 0
#> doc2 0 0 0 0 0 0 0 0 1 0
#> doc3 1 0 0 0 0 0 1 0 0 0
#> doc4 0 0 0 0 0 0 0 0 0 0
#> doc5 0 0 0 0 0 0 0 0 0 0
#> doc6 0 0 0 0 0 0 0 0 0 1
#> doc7 1 0 0 1 0 0 0 0 0 0
#> doc8 0 0 0 0 0 0 0 0 0 0
#> doc9 1 0 0 0 0 0 0 0 0 0
#> doc10 0 0 0 0 0 0 0 0 0 0
#> doc11 0 1 0 0 0 0 0 1 1 0
#> doc12 0 0 0 0 0 0 0 0 0 0
#> doc13 0 0 0 0 0 0 0 0 0 0
#> doc14 0 0 0 0 0 0 0 1 1 0
#> doc15 0 0 0 0 0 0 0 0 0 0
#> doc16 0 1 0 0 0 0 0 0 0 0
#> doc17 0 0 0 0 0 0 0 0 0 0
#> doc18 0 0 0 0 0 0 0 0 0 0
#> doc19 0 0 0 0 0 0 0 0 0 0
#> doc20 0 0 0 0 0 0 0 0 0 0
#> doc21 0 0 0 0 0 0 0 0 0 1
#> doc22 0 1 0 1 0 1 0 0 0 0
#> doc23 0 0 0 0 0 0 0 0 0 0
#> doc24 1 0 0 0 0 0 1 0 0 0
#> doc25 0 0 0 0 0 0 0 0 0 0
#> doc26 0 0 0 0 0 0 0 0 0 0
#> doc27 0 0 0 0 0 0 0 0 0 0
#> doc28 0 0 0 0 0 0 0 0 0 0
#> doc29 0 0 0 0 0 0 0 0 0 0
#> doc30 0 0 0 0 0 0 0 0 0 0
#> doc31 0 0 0 0 0 0 0 0 0 0
#> doc32 0 1 0 0 0 1 0 0 0 0
#> doc33 0 0 0 0 0 0 0 0 0 0
#> doc34 0 0 0 0 0 1 0 0 0 0
#> doc35 0 0 0 0 0 0 0 0 0 0
#> doc36 0 0 0 0 0 0 0 0 0 0
#> doc37 0 0 0 0 0 0 0 0 0 0
#> doc38 0 0 0 0 0 0 0 0 0 0
#> doc39 0 0 1 0 1 0 0 0 0 0
#> doc40 0 0 0 0 0 0 0 0 0 0
#> doc41 0 0 1 0 1 0 0 0 0 0
#> bieten casa dann durch einen einer einrichtung freuen gerne hemmingen
#> doc1 0 1 0 0 0 1 0 0 0 0
#> doc2 0 1 0 0 0 0 0 0 0 0
#> doc3 0 0 0 0 0 1 0 0 0 0
#> doc4 0 0 0 0 1 0 0 0 0 0
#> doc5 0 0 0 0 0 0 0 0 0 0
#> doc6 1 0 0 0 0 0 0 0 0 1
#> doc7 0 0 0 0 0 0 1 1 0 0
#> doc8 0 0 0 0 0 0 0 0 0 0
#> doc9 0 0 0 0 0 0 0 0 0 0
#> doc10 0 0 0 0 0 0 0 0 0 0
#> doc11 0 0 0 0 0 0 0 0 0 0
#> doc12 0 0 0 0 0 0 0 0 0 0
#> doc13 0 0 0 0 0 0 0 0 0 0
#> doc14 0 0 0 1 0 0 0 0 0 0
#> doc15 0 0 0 0 0 0 0 0 0 0
#> doc16 0 0 0 0 0 0 0 0 0 0
#> doc17 0 0 0 0 0 0 0 0 0 0
#> doc18 0 0 0 0 0 0 0 0 0 0
#> doc19 0 0 0 0 0 0 0 0 0 0
#> doc20 0 0 0 0 0 0 0 0 0 0
#> doc21 0 0 0 0 0 0 0 0 0 0
#> doc22 0 0 0 0 0 0 0 0 1 0
#> doc23 0 0 0 1 0 0 0 0 0 0
#> doc24 0 0 0 0 0 0 0 0 0 0
#> doc25 0 0 0 0 0 1 1 0 0 0
#> doc26 0 0 0 0 0 0 0 0 0 0
#> doc27 0 0 0 0 0 0 0 0 0 0
#> doc28 0 0 0 0 0 0 0 0 0 0
#> doc29 0 0 0 0 0 0 0 0 0 0
#> doc30 1 0 0 0 0 0 0 0 0 0
#> doc31 1 0 0 0 0 0 0 0 0 0
#> doc32 0 0 0 0 0 0 0 0 0 0
#> doc33 0 0 0 0 1 0 0 0 0 0
#> doc34 0 2 1 0 0 0 0 0 0 0
#> doc35 0 0 0 0 0 0 0 1 0 0
#> doc36 0 0 0 0 0 0 0 0 0 0
#> doc37 0 0 0 0 0 0 0 0 0 1
#> doc38 0 0 0 0 0 0 0 0 0 0
#> doc39 0 0 0 0 0 0 0 0 0 0
#> doc40 0 0 0 0 0 0 0 0 0 0
#> doc41 0 0 1 0 0 0 0 0 1 0
#> hohe ihnen ihre ihrer mensch mitarbeiter neben pflege philosophie reha
#> doc1 0 1 0 0 0 0 0 1 0 1
#> doc2 0 0 0 0 2 1 0 0 1 1
#> doc3 0 0 0 0 0 0 0 0 0 0
#> doc4 0 0 0 0 0 0 0 0 0 0
#> doc5 1 0 0 0 0 0 0 0 0 0
#> doc6 0 0 0 0 0 0 0 0 0 0
#> doc7 0 0 0 0 0 1 0 0 0 0
#> doc8 0 0 0 0 0 0 0 0 0 0
#> doc9 0 0 0 0 0 0 0 0 0 0
#> doc10 0 0 1 0 0 0 0 0 0 0
#> doc11 0 0 0 0 0 0 0 1 0 0
#> doc12 0 0 0 0 2 0 0 0 1 0
#> doc13 0 0 1 0 0 0 0 0 0 0
#> doc14 0 0 0 0 0 0 0 1 0 0
#> doc15 0 0 0 0 0 0 0 0 0 0
#> doc16 0 0 0 0 0 0 0 1 0 0
#> doc17 0 0 0 0 0 0 0 0 0 0
#> doc18 0 0 0 0 0 0 0 0 0 0
#> doc19 0 0 0 0 0 0 0 0 0 0
#> doc20 0 0 0 0 0 0 0 0 0 0
#> doc21 0 0 0 1 0 0 0 0 0 0
#> doc22 0 0 0 0 0 0 0 0 0 0
#> doc23 0 0 0 1 0 0 1 0 0 0
#> doc24 0 0 0 0 0 0 0 0 0 0
#> doc25 0 0 0 0 0 0 0 0 0 0
#> doc26 0 0 0 0 0 0 0 0 0 0
#> doc27 1 0 0 0 0 0 0 0 0 0
#> doc28 0 0 0 0 0 0 0 0 0 0
#> doc29 0 0 0 0 0 0 0 0 0 0
#> doc30 0 0 0 0 0 0 0 0 0 0
#> doc31 0 1 0 0 0 0 1 0 0 0
#> doc32 0 0 1 0 0 0 0 0 0 0
#> doc33 0 0 0 0 0 0 0 0 0 0
#> doc34 0 0 0 0 0 0 0 0 0 2
#> doc35 0 0 0 0 0 0 0 0 0 0
#> doc36 0 0 0 0 0 0 0 0 0 0
#> doc37 0 0 0 0 0 0 0 0 0 0
#> doc38 0 0 0 0 0 0 0 0 0 0
#> doc39 0 0 0 0 0 0 0 0 0 0
#> doc40 0 0 0 0 0 0 0 0 0 0
#> doc41 0 1 0 0 0 0 0 0 0 0
#> rosenpark seniorenpflegeheim sich sie sind sowie sozialkonzept
#> doc1 0 0 0 0 0 0 0
#> doc2 0 0 0 0 0 0 0
#> doc3 0 0 0 0 0 0 0
#> doc4 0 0 0 0 0 0 0
#> doc5 0 0 0 0 1 0 0
#> doc6 1 1 0 0 0 0 0
#> doc7 0 0 1 1 0 0 1
#> doc8 0 0 0 0 0 0 0
#> doc9 0 0 0 0 0 0 0
#> doc10 0 0 0 0 1 0 0
#> doc11 0 0 1 1 0 0 0
#> doc12 0 0 1 1 0 0 0
#> doc13 0 0 0 0 1 0 0
#> doc14 0 0 0 0 0 0 0
#> doc15 0 0 0 0 0 0 0
#> doc16 0 0 0 0 0 0 0
#> doc17 0 0 0 0 0 1 0
#> doc18 0 0 0 0 0 0 0
#> doc19 0 0 0 0 0 0 0
#> doc20 0 0 0 1 1 0 0
#> doc21 0 0 0 1 0 0 0
#> doc22 0 0 0 1 1 0 0
#> doc23 0 0 1 1 0 0 0
#> doc24 0 0 0 0 0 0 0
#> doc25 0 0 0 0 0 0 0
#> doc26 0 0 0 0 0 0 0
#> doc27 0 0 0 0 0 0 0
#> doc28 0 0 0 0 0 0 0
#> doc29 0 0 0 0 0 0 0
#> doc30 0 0 0 0 0 0 0
#> doc31 0 0 0 0 0 1 0
#> doc32 0 0 2 1 0 0 0
#> doc33 0 0 0 1 0 0 0
#> doc34 0 0 1 1 0 0 0
#> doc35 0 0 0 1 0 0 0
#> doc36 1 1 0 0 0 0 1
#> doc37 0 0 0 0 0 0 0
#> doc38 0 0 0 0 0 0 0
#> doc39 0 0 0 0 0 0 0
#> doc40 0 0 0 1 0 0 0
#> doc41 0 0 0 0 0 0 0
#> stationären täglich uns unsere unserer unter wir über
#> doc1 1 0 0 0 0 0 0 0
#> doc2 0 0 0 1 0 0 0 1
#> doc3 0 0 0 0 0 0 1 0
#> doc4 0 0 0 0 1 0 0 0
#> doc5 0 0 0 0 0 0 0 0
#> doc6 0 0 0 0 0 0 1 0
#> doc7 0 0 0 1 0 0 0 0
#> doc8 0 0 0 0 0 0 0 0
#> doc9 0 0 0 0 0 0 0 0
#> doc10 0 0 0 0 0 0 0 0
#> doc11 0 1 0 2 0 0 0 0
#> doc12 0 0 0 0 1 0 0 0
#> doc13 0 0 0 0 0 0 0 0
#> doc14 0 0 0 0 1 0 0 0
#> doc15 0 0 0 0 0 0 0 0
#> doc16 0 0 0 0 0 0 0 0
#> doc17 0 0 0 0 0 0 0 0
#> doc18 0 0 0 0 0 0 0 0
#> doc19 0 0 0 0 0 0 0 0
#> doc20 0 0 0 0 0 0 0 0
#> doc21 0 1 0 0 0 0 0 0
#> doc22 0 0 0 0 0 0 0 0
#> doc23 0 0 0 0 0 0 0 0
#> doc24 0 0 0 0 0 0 0 0
#> doc25 1 0 0 0 0 0 0 0
#> doc26 0 0 0 0 0 0 0 0
#> doc27 0 0 0 0 0 0 0 0
#> doc28 0 0 0 0 0 0 0 0
#> doc29 0 0 0 0 0 0 0 0
#> doc30 0 0 0 0 0 0 1 0
#> doc31 0 0 0 0 0 1 1 0
#> doc32 0 0 1 0 0 0 1 0
#> doc33 0 0 0 0 0 0 0 0
#> doc34 0 0 0 0 0 1 0 1
#> doc35 0 0 1 0 0 0 1 0
#> doc36 0 0 0 0 0 0 0 0
#> doc37 0 0 0 0 0 0 0 0
#> doc38 0 0 0 0 0 0 0 0
#> doc39 0 0 0 0 0 0 0 0
#> doc40 0 0 0 0 0 0 0 0
#> doc41 0 0 0 0 0 0 0 0
#> attr(,"binary")
#> [1] FALSERows represent documents, columns represent terms, and cells contain
counts. Use binary = TRUE when presence is more appropriate
than frequency.
TF-IDF increases the weight of terms that are frequent in a particular document but uncommon across the collection.
weighted <- tf_idf(dtm)
keywords <- extract_keywords(dtm, n = 3)
head(keywords, 12)
#> document rank term weight
#> 1 doc1 1 stationären 0.4548822
#> 2 doc1 2 bereich 0.4189219
#> 3 doc1 3 casa 0.4189219
#> 4 doc2 1 mensch 0.8086794
#> 5 doc2 2 mitarbeiter 0.4043397
#> 6 doc2 3 philosophie 0.4043397
#> 7 doc3 1 bereich 0.8378438
#> 8 doc3 2 einer 0.8378438
#> 9 doc3 3 als 0.7364775
#> 10 doc4 1 einen 1.8195287
#> 11 doc4 2 unserer 1.6756876
#> 12 doc5 1 hohe 1.8195287Cosine similarity compares the orientation of two feature vectors while reducing the influence of document length.
For a compact illustration, use synthetic documents with known labels.
training_text <- c(
"analyze data statistical model",
"build predictive model data",
"create dashboard analyze metrics",
"provide nursing care patient",
"support patient clinical care",
"coordinate nurse patient treatment"
)
training_labels <- c("data", "data", "data", "care", "care", "care")
training_dtm <- document_term_matrix(training_text)
model <- fit_naive_bayes(training_dtm, training_labels, laplace = 1)
model
#> <text_nb> Multinomial Naive Bayes
#> Classes: data, care
#> Terms: 18
predicted <- predict(model, training_dtm)
classification_metrics(training_labels, predicted, positive = "data")
#> n true_positive false_positive true_negative false_negative accuracy
#> 1 6 3 0 3 0 1
#> balanced_accuracy precision recall specificity f1
#> 1 1 1 1 1 1This training-set result demonstrates mechanics, not generalization. The next vignette shows out-of-sample evaluation.
These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.