Worldlex: Twitter and blog word frequencies for 66 languages

Behav Res Methods. 2016 Sep;48(3):963-72. doi: 10.3758/s13428-015-0621-0.

Abstract

Lexical frequency is one of the strongest predictors of word processing time. The frequencies are often calculated from book-based corpora, or more recently from subtitle-based corpora. We present new frequencies based on Twitter, blog posts, or newspapers for 66 languages. We show that these frequencies predict lexical decision reaction times similar to the already existing frequencies, or even better than them. These new frequencies are freely available and may be downloaded from http://worldlex.lexique.org .

Keywords: Blogs; Cross-language frequency; Twitter; Word frequency.

MeSH terms

  • Blogging
  • Humans
  • Language*
  • Psycholinguistics / statistics & numerical data*
  • Social Media / statistics & numerical data*