Finding the balance between what to keep to index is hard! The attention mechanism in transformers should be pretty good at ranking results. The idea is to feed into context titles, top answers, etc in bulk along with a search query. The attention heatmap relative to the search gives you a general rank for how good each result is. Ironically enough, this is probably the most powerful indexer, yet no big tech uses it and instead has the model generate answers instead of ranking them. The best part is, this system is tunable and can be adjusted to user preference with little data. The overall goal should be to minimize the number of results a user checks. (This should be what other engines are doing in the first place)