Search Sqlite Full Text Search Arabic without diacritics

Question

I have a sqlite db with FTS4 search function. It has a lot of arabic text with diacritics, which are important for viewing and readability but not for search.

I wish for the search to be diacritics agnostic but it seems to only find matches exactly as I type them and ignores everything that even has a single diacritical mark.

How do I prevent it from doing so. I know custom sqlite functions are a no go in android, and the tokenizers are very limited.

Answer 1

In my experience the shortest solution is to store both forms of Arabic strings, with diacritics for display and without diacritics for search. So you can use simple database search functions.

Only if your application uses some kind of advanced search engine so you can implement an Arabic diacritics stemming function to filter such diacritics. (Some already have such function)

Search engines uses stemming functions while indexing.

Answer 2

this an example

SELECT field
from myTable
where replace (replace(replace (replace(replace(replace(replace(replace(
      field 
      ,char(1612) ,"") ,char(1611) ,""),char(1614) ,""),char(1615) ,""),char(1616) ,""),char(1617) ,""),char(1618) ,""),char(1648) ,"") 
      like '%طائفة%'