• system may return irrelevant documents in the interest of learning but they must come after relevant documents ALWAYS

Does this mean that the result is appended at the very end of the search results? Studies show most users don't really look beyond the first few results. Link to post

USP: system self corrects based on usage

What parts does the system correct? Seems that we can improve on ranking with some LTR, but is there anything else I'm missing?

• I don't understand which results you are talking about when you say append at the end of the "results". Do you have two different "results" in mind?
• What is LTR?
• Building the relevant document set. That's what improves

• results is whatever ordered list of units you're planning to append to. As you put it, where ".. they must come .."
• LTR is Learning to rank

• we define a document as a unit. Two sets are created, relevant and irrelevant. Ranking within set is not yet decided upon. Since the algorithm will make mistakes, those mistakes must come after all the relevant documents is what I meant. There is no appending business. You generate a set, impose an order on it and that's your search result.
• If any subsystem defines inputs it requires and can learn from them, there's no reason why only ranking can be learned. Other parts of the system can improve too.

Since the algorithm will make mistakes, those mistakes must come after all the relevant documents

How do you guarantee this? This means that the algorithm has to "know" that it is making a mistake?

We cannot guarantee this. However there are two ways an approximation can be obtained:

• Built the penalty/loss to include uncertainty in ranking of documents. This then let's the system learn to place uncertain documents in the end.
• Bias the training mechanism to favour true positives