Wednesday, October 13, 2021

Beware: Issues in Hybrid Search Crawling might not be flagged

We have a mixed SharePoint setup with parts still on-prem, while for others we moved to Microsoft cloud. We try as much as possible to abstract our end-users from this complexity in the SharePoint landscape. Therefore we utilize SharePoint Hybrid Search to allow the end-users finding from one single search entrance all SharePoint stored content, whether stored on-prem or online.
From time to time we encounter issues in this setup, most significant cause being that the required trust on the on-prem search crawlers to online index is lost due change of certificate on Microsoft Online side. We are automated flagged if such issue occurs, as we have an automated health monitoring + notification on the state of the crawl logs. Too bad, due a recent Microsoft change we can no longer trustworthy rely on the automated signalling. For (design) reasons so far only known to Microsoft development team, they decided to no longer flag in the crawl log in case an error occurs on propagating crawled content to the online index. We noticed as we started missing recent content in the search results; while on inspection of the crawl log it was reported to be successful crawled: full green. Yet by doing a query on 'k=isExternalContent:1' the most recent on-prem crawled items were not included in the search result, a clear deviation from the healthy report of the on-prem to online crawling.
Note: the actual rootcause of the missed crawled items in the online index was that in the user policy of the on-prem webapplication some accounts were configured, that in online world no longer had a match. This resulted in failing to index in SharePoint Online with error "Hybrid ACL Mapping: Deny ACE with unknown/unmappable hybrid claim", but it turns out that such error is no longer notified back to the on-prem crawler context. As Office 365 customer you do not have visibility on the Online index; we only came to know of this rootcause after we involved Microsoft Support. Once cause known, the problem was easily fixed: we removed the stale accounts also from the on-prem user policy.