Korean text analysis with RmecabKo

The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

RmecabKo is a Korean text-analysis layer on top of the MeCab morphological analyzer. The heavy lifting - the native engine and dictionary compilation - comes from RcppMeCab; RmecabKo adds tokenizers that follow the tokenizers contract, curated Korean data, user-dictionary tools, and a handful of analysis helpers. This vignette walks through a full tidy workflow.

Setup

Install the engine and a Korean dictionary once per machine:

install.packages("RcppMeCab")
RcppMeCab::download_dic("ko")
RcppMeCab::set_dic("ko")

Normalizing text

text_normalize() is a pure, dependency-light cleanup step. It composes Unicode (NFC), folds full-width characters, and squashes repeated characters - all of which help the analyzer and downstream matching. It needs no backend:

text_normalize("한국어 분석 ㅋㅋㅋㅋ 정말 재밌어요!!!!")
#> [1] "한국어 분석 ㅋㅋ 정말 재밌어요!!"

A tidy tokenization

The tokenizers take a character vector and return a list of character vectors, so they slot directly into tidytext::unnest_tokens(). The package ships a small demonstration corpus, demo_ko:

demo_ko[1:2]
#>                                                   d1 
#> "한국어 형태소 분석은 텍스트 마이닝의 첫걸음입니다." 
#>                                                   d2 
#>      "오늘 날씨가 정말 좋아서 공원을 오래 걸었어요."

corpus <- tibble(doc = names(demo_ko), text = demo_ko)
tokens <- corpus |>
  unnest_tokens(word, text, token = token_nouns)
head(tokens, 8)
#> # A tibble: 8 × 2
#>   doc   word  
#>   <chr> <chr> 
#> 1 d1    한국어
#> 2 d1    형태소
#> 3 d1    분석  
#> 4 d1    텍스트
#> 5 d1    마이닝
#> 6 d1    걸음  
#> 7 d2    날씨  
#> 8 d2    공원

Even without a working backend, the pre-computed tokenization bundled with the package lets us continue:

tidy_tokens <- if (backend && exists("tokens")) {
  tokens
} else {
  readRDS(system.file("extdata", "demo_ko_tokens.rds", package = "RmecabKo"))
}
count(tidy_tokens, word, sort = TRUE) |> head(6)
#> # A tibble: 6 × 2
#>   word      n
#>   <chr> <int>
#> 1 분석      2
#> 2 사람      2
#> 3 거리      1
#> 4 걸음      1
#> 5 것        1
#> 6 결과      1

Removing stopwords

stopwords_ko is a curated table of Korean function morphemes. Filter by surface form with an anti_join(), or strip whole part-of-speech classes at the tag level with drop_pos:

tidy_tokens |>
  anti_join(data.frame(word = stopwords_ko_words()), by = "word") |>
  count(word, sort = TRUE) |>
  head(6)
#> # A tibble: 6 × 2
#>   word      n
#>   <chr> <int>
#> 1 분석      2
#> 2 사람      2
#> 3 거리      1
#> 4 걸음      1
#> 5 결과      1
#> 6 공원      1

# drop every particle and ending directly during tokenization
token_morph(demo_ko[[2]], drop_pos = stopwords_ko_tags(c("josa", "eomi")),
            simplify = TRUE)
#> [1] "오늘" "날씨" "정말" "좋"   "공원" "오래" "걸"   "."

Keywords and TF-IDF

With a document column in hand, tidytext::bind_tf_idf() gives per-document keyword weights; keywords_tfidf() offers the same without the tidy detour:

tidy_tokens |>
  count(doc, word) |>
  bind_tf_idf(word, doc, n) |>
  arrange(desc(tf_idf)) |>
  head(6)
#> # A tibble: 6 × 6
#>   doc   word       n    tf   idf tf_idf
#>   <chr> <chr>  <int> <dbl> <dbl>  <dbl>
#> 1 d2    공원       1 0.5    2.30  1.15 
#> 2 d2    날씨       1 0.5    2.30  1.15 
#> 3 d9    아이       1 0.5    2.30  1.15 
#> 4 d9    운동장     1 0.5    2.30  1.15 
#> 5 d10   글         1 0.333  2.30  0.768
#> 6 d10   마음       1 0.333  2.30  0.768

keywords_tfidf(demo_ko, div = "nouns", top_n = 2) |> head(6)
#>   doc   word n        tf      idf    tf_idf
#> 1  d1   걸음 1 0.1666667 2.302585 0.3837642
#> 2  d1 마이닝 1 0.1666667 2.302585 0.3837642
#> 3  d2   공원 1 0.5000000 2.302585 1.1512925
#> 4  d2   날씨 1 0.5000000 2.302585 1.1512925
#> 5  d3   도움 1 0.2000000 2.302585 0.4605170
#> 6  d3 자연어 1 0.2000000 2.302585 0.4605170

Sentiment

lexicon_knu() downloads and caches the KNU Korean sentiment lexicon (polarity from -2 to 2). Joining it against tokens yields a per-document sentiment score. The lexicon is distributed under CC BY-NC-SA (Kyungpook National University), so it is fetched on demand rather than bundled; note the NonCommercial clause and review its terms before use.

senti <- lexicon_knu()
tidy_tokens |>
  inner_join(senti[senti$n_words == 1, ], by = "word") |>
  group_by(doc) |>
  summarise(score = sum(polarity))

N-grams, lemmas, and concordances

Morpheme n-grams never bridge a removed stopword:

token_ngrams(demo_ko[[1]], n = 2, div = "nouns", simplify = TRUE)
#> [1] "한국어 형태소" "형태소 분석"   "분석 텍스트"   "텍스트 마이닝"
#> [5] "마이닝 걸음"

token_lemma() recovers the dictionary form of predicates, which keeps inflected verbs and adjectives from scattering in a frequency count:

token_lemma(c("아침을 먹었다", "날씨가 좋았다"))
#> [[1]]
#> [1] "먹다"
#> 
#> [[2]]
#> [1] "좋다"

kwic() shows a keyword in its morpheme context:

kwic(demo_ko, "분석")
#>   doc position             left keyword                  right
#> 1  d1        3    한국어 형태소    분석 은 텍스트 마이닝 의 첫
#> 2  d8        9 하 게 정리 하 면    분석  이 훨씬 쉬워 집니다 .

Teaching the analyzer new words

When MeCab splits a name or neologism you care about, register it once and activate it for the session. This writes to your user data directory, so it is not run here:

dict_add_words(c("은전한닢", "카비봇"), tag = "NNP")
dict_use()
pos("카비봇 출시 소식")
dict_words()

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.