Expose search quality metrics like CTR (click-through rate) in the Search Traffic Analytics dashboard
That would require collecting information about user clicks. The dashboard could also show the number of unique users over time and MRR (mean reciprocal rank).
Please take a look at the search traffic analysis and let us know if this suits your needs.
Janusz Lembicz commented
Search Traffic Analytics : https://azure.microsoft.com/en-us/documentation/articles/search-traffic-analytics/