Tuesday, November 7, 2017

Retrieve data from a large list via REST

The infamous ´5000´ listview threshold, we all have encountered that limit at least once in utilizing SharePoint as data backend. This time I was consulted by a business user that utilized SharePoint´s data management capabilities for storage of above 135,000 listitems. Wrt storage this amount is not an issue, but for retrieving it can be due the listview threshold. The advised approach to deal with that is via indexed columns, and tabbed/indexed views. That is for retrieving + viewing the big amount of listitems in the standard SharePoint UI. But what about requesting the data via SharePoint REST service? The REST protocol promises to support a similar navigation/tabbed experience via $top and $skip parameters. However, here SharePoint (2010) demonstrates to be not a fully compliant REST citizen. The $top parameters works fine on indexed large list, but usage of $skip results in an HTTP 500; and in ULS the error "Throttled:Big list slow query. List item query elapsed time: 0 milliseconds" is logged.
Also here it turns out that the '5000 threshold' is such a common encountered SharePoint issue. Internet search within a few hits leads to the helpful Stackoverflow resource: SharePoint 2010 REST top, skip fails on large list:
$skip query option is not supported, but $top is. Workaround, use a mix of $orderby on ID, $filter on ID and $top x items, and loop
Pseudo-code to loop through the entire big SharePoint List:
var nextId = 0;
WHILE TRUE DO
    var getData = $.getJson(“<site-collection url>/_vti_bin/listdata.svc/LargeList"
          + "?$select=Id,Name&$top=1000&&orderby=id&$filter=Id gt " + nextId);
    if (getData is not empty) {
        nextId = getData(last)[id];
    } else {
        break;
    }
END DO

No comments:

Post a Comment