When we perform a search for challenges with page=100 and perPage=100, I see that the number of total pages is 583
See below:
But when I try to search with page=101 and perPage=100, there are no results returned by the API and the values of total-pages and total number of records are set to 0 which is wrong:
Root cause:
The issue here is due to the pagination parameters used to get the data from Elasticsearch es-query-pagination
According to ES documentation By default, you cannot use from and size to page through more than 10,000 hits. This limit is a safeguard set by the [index.max_result_window](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-max-result-window) index setting. If you need to page through more than 10,000 hits, use the [search_after](https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after) parameter instead.
See es-paginate-search-result
cc @ThomasKranitsas
When we perform a search for challenges with page=100 and perPage=100, I see that the number of total pages is 583
See below:
But when I try to search with page=101 and perPage=100, there are no results returned by the API and the values of total-pages and total number of records are set to 0 which is wrong:
Root cause:
The issue here is due to the pagination parameters used to get the data from Elasticsearch es-query-pagination
According to ES documentation
By default, you cannot use from and size to page through more than 10,000 hits. This limit is a safeguard set by the [index.max_result_window](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-max-result-window) index setting. If you need to page through more than 10,000 hits, use the [search_after](https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after) parameter instead.See es-paginate-search-result
cc @ThomasKranitsas