Architecture Pattern XML-Funneling of complex SAP structures for Duet Enterprise interoperability

This blog is earlier published on SAP Community Network Blogs

Via Duet Enterprise it is possible to expose SAP data to a SharePoint UI in short timeframes (less than an hour). This agility of SAP/SharePoint interoperability is made possible by combining existing strengths of SAP NetWeaver Gateway (Design Time + Runtime support) and SharePoint BCS. To utilize these strengths in optimum way, your scenario must obey to the following list of prerequisites:

• Usable RFCs or BORs are available in the SAP backend
• Flatten SAP data structure
• SharePoint ExternalList UI metaphore is good enough / adequat for the User Experience (note that you can alter it considerable via a custom XSLT stylesheet)

Duet Enterprise leans on the power of SAP NetWeaver Gateway 2.0 to publish SAP data from backend to external UI channels. Gateway design time tools enable you to quickly generate a Gateway DataModel, next expose this via a generated SAP webservice, and import the SAP data into SharePoint BCS.
Althought Gateway itself allows exchange of complex SAP data structures, the GW tools current do not support that. If you need to expose a complex SAP structure through Gateway, you’re on your own with handcrafting the Backend Operation Proxy, GSDO mappers, Gateway Service Proxy. This is not very attractive; it is laborsome, errorprone, and for every change in the data or service signature you must repeat some or even the whole effort.

An alternative is an approach in which you can still use and rely on the powers of the GW Design Time tools to generate the mappers. As said, the GW Design Tools demand the mapped SAP data is in a flattened structure. This conflicts when your source SAP structure is complex, with nested tables. But you can align them by introducing an intermediair component on SAP side; which semi-flattens the complex structure while maintaining in metadata the complex structure. Outline of this architecture pattern is that you build a custom RFC on the SAP backend that retrieves the complex structure by invoking the existing RFC/BOR, or even invoke that RFC/BOR in a loop or invoke multiple RFCs and/or BORs. The received SAP data is collected into a runtime XML document instance, in which you (re)build + maintain the complex data(object) structure. The custom RFC outputs as single string parameter the OuterXML of the runtime XML document. Such a single string parameter can without constraints be handled by the GW Design Tools to generate the entire Gateway + Duet Enterprise data exchange components: GW Model, GW Service, BDC Model. At runtime the complex structure is derived at SAP business side, funneled into a ‘flattened’ string-representation, and exchanged via Gateway + Duet Enterprise to and from SharePoint UI. Within SharePoint context the received string-representation is interpretated as serialized format of a complex data transfer object (DTO). Either via XSLT-transformation in the standard BCS controls, or via a custom UI-control, the complex structure is rendered on a SharePoint page.

The Xml-funneling pattern can be applied at 2 levels. You can use it to flatten the collection of child SAP entities of a parent SAP entity (returned as nested tables in the RFC output). And you can use it to flatten an entire functional SAP entity; which may be runtime composed by issuing multiple RFC calls.

Impossible to use custom implementation of public BCS interface IEntityInstanceProvider

SharePoint Business Connectivity Services contains out-of-the-box webparts to visualize external business data. All the standard BCS business data renderers utilize the pattern that data selection is delegated to a data provider control. The 2 controls are associated via consumer/provider connectionpoints. BusinessDataDetailsWebPart is one of them. Its standard usage is for BCS profile pages. However, usage of this webpart on itself is not necessarily limited to that context alone. The webpart is selectable in the webpart gallery, and can be added to arbitrair SharePoint webpart page. The challenge is how to connect it to an external BCS entity. The trivial approach is to do it similar as on profile pages; feed it via the provider BusinessDataItemBuilder. The working of BusinessDataItemBuilder depends on an ‘ID’ querystring parameter. However, when that parameter is included in the url of a page with the 2 BCS webparts on it, SharePoint faults with an error message:

System.NullReferenceException: Object reference not set to an instance of an object.
  at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapDataSource.OnInit(EventArgs e)

This occurs because PortalSiteMapDataSource also reacts on the presence of ‘ID’ querystring parameter, but assigns another and in this context incorrect meaning to the value. When the ‘ID’ querystring parameter is renamed, the error is prevented. However, then the standard BusinessDataItemBuilder no longer reacts on it.

A proper solution seemed to build a custom provider that relies on another named querystring parameter to retrieve the BCS entity item. The BCS architecture has the promise to support this: it heavily decouples via interfaces from the specific BCS implementation classes. But this appears to be a false promise. Microsoft already points you to that via the warning accompanying the public (!!) interface IEntityInstanceProvider

This class and its members are reserved for internal use and are not intended to be used in your code.

Still, since the interface is public; and also the connectionpoints are public; it sounded safe to expect it must work. But when I added both my own IEntityInstanceProvider implemenation class and a BusinessDataDetailsWebPart on a page; it was not possible to connect them; not in the GUI, and also not explicit in custom code. Via Reflector I reverse engineered the cause, to ultimately end up with the following:

Microsoft.SharePoint.WebPartPages.WebPartPageUserException was unhandled
Message=The connectionpoint BDWP Item on g_1e1e7d15_652d_44cd_bc7d_9026382ff33b is disabled.
Source=Microsoft.SharePoint
StackTrace:
at Microsoft.SharePoint.WebPartPages.SPWebPartManager.CanSPConnectWebPartsCore(WebPart provider, ProviderConnectionPoint providerConnectionPoint, WebPart consumer, ConsumerConnectionPoint consumerConnectionPoint, WebPartTransformer transformer, Boolean throwOnError)

and digging deeper:

if (!providerConnectionPoint.GetEnabled(connectionAgentByInterfaceName))
{
    if (throwOnError)
    {
        throw new WebPartPageUserException(WebPartPageResource.GetString("SPWebPartConnection_DisabledConnectionPoint", new object[] { providerConnectionPoint.ID, provider.ID }));
    }
    return false;
}

public override bool GetEnabled(Control control)
{
    bool flag = false;
    BusinessDataWebPart part = control as BusinessDataWebPart;
    if (part != null)
    {
        try
        {
            return part.CanConnect(true);
        }
        catch
        {
            return false;
        }
    }
    if (control is BusinessDataItemBuilder)
    {
        flag = true;
    }
    return flag;
}

Thus: although at the external front decoupled via interface IEntityInstanceProvider, internally BusinessDataDetailsWebPart demands it's provider to be either a BusinessDataWebPart or BusinessDataItemBuilder webpart!! So much for interface decoupling.
Confine to this internal constraint by inheriting either of these classes is also not possible: BusinessDataItemBuilder is sealed, and BusinessDataWebPart is public abstract, but with 4 internal abstract methods.

BusinessDataDetailsWebPart also allows to explicit connect it in code with a BCS entity. Less elegant as via the connectionpoint, but workable. The standard BusinessDataDetailsWebPart can then still be used to render the external data item; no need to custom build an own one (or ‘copy’).

BusinessDataDetailsWebPart busDetailsWp = wp as BusinessDataDetailsWebPart;
if (busDetailsWp.BdcEntity != null && busDetailsWp.BdcEntity.Equals(entity))
{
    string entityInstanceId = ((IEntityInstanceProvider)this).GetEntityInstanceId();
    busDetailsWp.ItemID = entityInstanceId;
}