indexing code documentation sites (sphinx, postgres full-text, xapian...?)

Question

I was wondering which of the indexing libraries out there would be suitable for a site of code documentation where you can't just ignore "punctuation" as insignificant. (In some of the languages I'm interested in, punctuation can be part of significant "words", like some standard method names in Raku such as "^methods", "^mro", etc.)

Full unicode/utf-8 support is also important, but I suspect that's a given in modern packages.

Answer 1

Manticore Search would be suitable since:

• it supports low-level tokenization fine-tuning which is important for can't just ignore "punctuation" as insignificant
• searchdcode.com uses Manticore Search as a backend which proves it can work in your case too