You make very good points. The quality of the results from this system when indexing source code is pretty poor. However, it should perform fine for non-English European languages which have similar phonics to English.

As PlayNice.ly grows it is becoming clear that we are going to need to replace this search infrastructure with something more robust. It is a way off yet, but it is on the horizon.

Adam

Maybe I’m missing something, but how do you provide support for searching terms and phrases that are *not* in the english language? Sometimes source code and error message references seem like they contain *mostly* non-english words, and there are symbols and other special characters to consider. Do you just punt on these? Supporting them seems like it would grow the potential list of keys dramatically. On the other hand, not supporting them seems like it’d kill any usefulness of the search, since a really common use case for search is “I got this weird error…” and another is “I think the function name is ‘def x_this_for_me’”

You could use a sorted zset to assign weightings to different fields. For example you could increment score by 2 when indexing if the term is in the title and 1 if the word is in any other field. You could also increment score several times if it’s found more than 1 time.

I will take a stab in the dark and point out that Redis will optimise the storage of sets of integers, which can greatly reduce memory usage. However, by the look of Spinn3r I guess that you are dealing with a much greater data volume than we initially expect.