Duet Enterprise BDC Publisher requires 'Z' prefix

Beware that when using the BDC Publisher tool to generate the SAP Gateway service (in SPROXY) and associated SharePoint BDC Model, to enter a prefix that starts with a 'Z'. Otherwise the BDC Publisher will immediately abort, with error message 'No license to edit object R3TR FUGR...'. I did not immediately link this error message to the apparent SAP obligation of starting 'Z'. It cost me some elapse time (a.o. checking our systems license, my own profile, ...) before I rather coincidently detected that starting the BDC Publisher prefix with 'Z' fixes the issue:

Apparent this Z-obligation is common in SAP development; so common that consulted SAP experts didn’t even think of this as cause. The displayed error message also doesn't immediate puts you on the right direction.
I notified the Duet Enterprise product development team of this issue, including suggestion to prevent it by auto-fill in the 'Z' in the BDC Publisher screen.

BusinessDataItemBuilder not usable on publishing site

SharePoint BCS out-of-the-box delivers a number of useful webparts to render external data. The BusinessData webparts consistently apply the architecture pattern of data provider and data consumer. A provider-consumer example is that of the BusinessDataItemDetails webpart to render an external data item, connected to a BusinessDataItemBuilder webpart that provides the external data item. The latter applies the 'ID' querystring parameter in the page URL to determine which entity to retrieve from the external administration. A standard example where the 2 webparts are used in this combination are the BCS profile pages.

It appears a sensible approach to use the same webparts combination for rendering an external entity item on a publishing page. However, when opening the page with the 'ID' querystring parameter referencing a BCSidentity:

http://tnvduetenterprise/subsite/Pages/TestSAP.aspx?ID=0aab

SharePoint throws the following error:

PortalSiteMapProvider was unable to fetch current node, request URL: /subsite/Pages/TestSAP.aspx, message: Value cannot be null. Parameter name: key, stack trace: at System.Web.SiteMapNode..ctor(SiteMapProvider provider, String key, String url, String title, String description, IList roles, NameValueCollection attributes, NameValueCollection explicitResourceKeys, String implicitResourceKey) at System.Web.SiteMapNode..ctor(SiteMapProvider provider, String key, String url, String title, String description) at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapNode..ctor(PortalSiteMapProvider provider, PortalWebSiteMapNode webNode, String key, NodeTypes type, String url, String title, String description) at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapProvider.get_CurrentNode()

The cause is that the masterpage of publishing site includes PortalSiteMapProvider and PortalSiteMapDataSource. And the latter also reacts on the presence of the 'ID' querystring parameter, but expects a page identification as value instead of the BCS identity.

Justification of the Duet Enterprise XML-Funneling pattern

XML-Funneling pattern as answer for dealing with complex SAP data structures

In the blogpost ‘Architecture Pattern XML-Funneling of complex SAP structures for Duet Enterprise interoperability’, I introduced the XML-Funneling pattern as pragmatic approach to handle the Duet Enterprise limitation regarding complex SAP data structures. This Duet Enterprise limitation has 2 origins:

• The SAP NetWeaver Gateway generator tools are not capable of preservering hierarchical structure; their simple execution model is to flatten the SAP structure in the Gateway DataModel.
• The SharePoint BCS UI-controls are not capable of rendering hierarchical structure; their default rendering model is list/row based.

An often-heard misunderstanding is that Duet Enterprise itself, via the foundational layers SAP NetWeaver Gateway and SharePoint BCS, cannot be used to exchange complex data structures between SAP and SharePoint. This is not true. Gateway can expose complex data structures, but currently then requires you to model the interface and data signatures in ESR Builder, and next manually develop the Gateway Mappers. Although in FP1 less cumbersone as in Duet Enterprise 1.0, it is by far not the preferred approach to expose SAP building blocks as Gateway services. SharePoint BCS as integration layer can consume complex external data structures, but the rendering via standard controls is limited to the root level. In case the child levels also must be visualized, you need to develop a custom view that interoperates with the BCS API. This is doable, although a painpoint is that the SharePoint BCS API is weakly typed, and therefore programming against the API is laborsome and error prone.

I invented the XML-Funneling approach to simplify the development process at the SAP side. By mapping the SAP hierarchical structure to an XML string, the Gateway generator tools can be applied: a string is namely a ‘simple’ type.
On the SharePoint side the XML-Funneling approach does not really make developer life easier. A marked disadvantage is that it requires custom development to deserialize the received XML-structure into a strongtyped data object, before you can next use and visualize the received complex SAP data structure in a SharePoint view.

Trade-off decision whether or not to apply XML-Funneling

A trade-off decision is then whether in the total SAP + SharePoint system architecture and development effort, this front-end side / .NET disadvantage outweights the advantage on SAP side? Is it sensible to abandon the XML-Funneling approach in case of complex SAP data structure? My advice is to still apply the XML-Funneling pattern. Justification herefore are the following considerations.

Starting-points

• SharePoint is the target front-end platform.
• Favour out-of-the-box SharePoint above custom development.
• Application of SharePoint BCS; i) for the connectivity handling to external data administrations, ii) for the out-of-the-box UI controls to render external data, iii) and for integration within SharePoint Enterprise Search and FAST.
• Complex data structure is common within SAP business packages

Are there sensible alternatives for XML-Funneling approach?

WCF + DataContracts as alternative for Duet Enterprise to consume SAP data
First, Duet Enterprise is itself a WCF variant: the SAP data is consumed in SharePoint BCS as WCF service endpoint.
Second, on SharePoint side BCS is the limitating factor wrt complex structures, not Duet Enterprise. What if you bypass BCS and directly consume the Gateway Services as WCF endpoints? The result is that you not only loose the added interoperability value of Duet Enterprise: Single Sign-On, Authorization, Logging and Monitoring. But in addition also all of the native values of SharePoint BCS: standard UI-controls to visualize data (not all exposed SAP data is of hierarchical / complex structure), exposal to offline usage, use in SharePoint Search or FAST context.
Conclusion: this alternative is not sensible; it does not comply with the starting-points.

Expose complex SAP data structures via Duet Enterprise into SharePoint BCS
SharePoint BCS is a generic integration layer, usable for arbitrary external administrations: SAP, Siebel, Oracle, SQL-databases, ... This transparant interoperability capability is supported through a script-like, weakly typed BCS programming model. Whenever you can operate with BCS imported external data through any of the available standard UI controls, this doesn’t impose a problem. The controls abstract the weakly typed BCS programming model. But the standard controls are not able to visualize complex external data structures. For them also in this alternative you are required to custom build a view. Due the weakly typed programming model of the BCS API this is laborsome and error-prone.
Conclusion: this alternative is on SharePoint side an hardly better, programmer-friendly approach for consuming complex BCS consumed data; merely different. At SAP side, the consequence is that you cannot use the Gateway Generator tools/pipeline.

Expose complex SAP data structures as multiple associated simple structures
SharePoint BCS supports the concept of associations between two external content types. As parent-child is a relation, you can use this concept to expose complex SAP data structures through Duet Enterprise into SharePoint BCS.
BCS associations certainly has value, e.g. for master-child data model and UI-formats. But it also suffers from limitations that make it less applicable in other scenarios. For one: the net effect is that to retrieve a single complex SAP structure, multiple invocations from SharePoint to SAP must be done to retrieve the entire structure. And the responsibility for retrieving the entire structure is placed on the front-end side; instead of the data provider side. Another, the standard SharePoint UI controls all operate on a single BCS entity. If in the User Interface Design the ‘child’ data is directly mingled with(in) the ‘parent’ data, the standard controls cannot be used.
A Duet Enterprise design time issue is that the SAP Duet Enterprise generators are focussed on a single Gateway datamodel, and have no support for defining associations between 2 datamodels. Usage of BCS associations requires you to afterwards modify the BDC Model that is generated by BDC Publisher to add the associations. Either via SharePoint Designer, but in my experience you will more often than not need to resort to manual modification of the BDC Model. Which is time consuming, and error prone.
Conclusion: associations suffers from limitations and software architecture + performance drawbacks.

Final thought

Given that you want to benefit from the interoperability values of Duet Enterprise, and that within the exposed SAP business package there are entities with complex structure; XML-Funneling is a feasible approach. Consistent application of the pattern results in recognizable code setups for multiple Duet Enterprise scenarios, on SAP and SharePoint side. And in our experience, it is also possible to develop reusable building blocks for SAP and SharePoint to ease in the application of the pattern.