this is the text in my dataframe df which has a text column called 'problem_note_text'
SSCIssue: Note Dispenser Failureperformed checks / dispensor failure / asked the stores to take the note dispensor out and set it back / still error message says front door is open / hence CE attn reqContact details - Olivia taber 01159063390 / 7am-11pm
df$problem_note_text <- tolower(df$problem_note_text)
df$problem_note_text <- tm::removeNumbers(df$problem_note_text)
df$problem_note_text<- str_replace_all(df$problem_note_text, " ", "") # replace double spaces with single space
df$problem_note_text = str_replace_all(df$problem_note_text, pattern = "[[:punct:]]", " ")
df$problem_note_text<- tm::removeWords(x = df$problem_note_text, stopwords(kind = 'english'))
Words = all_words(df$problem_note_text, begins.with=NULL)
Now have a dataframe which has a list of words but there are words like
"Failureperformed"
which needs to be split into two meaningful words like
"Failure" "performed".
how do I do this, also the words dataframe also contain words like
"im" , "h"
which do not make sense and have to be removed, I do not know how to achieve this.
Given a list of English words you can do this pretty simply by looking up every possible split of the word in the list. I'll use the first Google hit I found for my word list, which contains about 70k lower-case words:
This sometimes works:
But also sometimes fails when the relevant words aren't in the word list (in this case "sensor" was missing):