Influence on Duet Enterprise product development + roadmap

The Next View engages in select international comitee with SAP on development and roadmap of Duet Enterprise

The Next View is regarded in the Netherlands as the Duet Enterprise expert. SAP itself also recognizes this. We are therefore SAP Partner on SAP / Microsoft interoperability. SAP Netherlands refers customers that seek for information and guidance on this subject to us, and actively engages us in Duet Enterprise consultations. The Duet Enterprise product team in Walldorf and Bangalore [very much] appreciates the feedback we give based on our experiences with the product. As such we are also asked to participate in the Duet Enterprise Customer Engagement Initiative (CEI). This is a select international group of partners and customers that take the lead with implementations of Duet Enterprise. Participating organizations come from Germany, Australia, England, Denmark, and The Next View as sole and exclusive Dutch partner.

The CEI concept is to provide input for desired developments in the Duet Enterprise product roadmap. Also the Duet Enterprise product team demonstrates (potential) new product features, and inputs new ideas to discuss with and be assessed by the CEI participants. As the disclosed information concerns potential future product features, all CEI participants prior have signed a Non-Disclosure Agreement (NDA).

The recent Duet Enterprise 2.0 Preview announcement proves that via the CEI concept we (The Next View) indeed exert influence on the development of the product. Our main criticism on Duet Enterprise 1.0 is the missing of OData interoperability support. The underlying SAP NetWeaver Gateway integration layer includes OData provider support since version 2.0. The bottleneck here is that SharePoint 2010 Business Connectivity Services does not include OData consumption support. But now using OData sources with BCS is added in the upcoming SharePoint 2013, the next version of Duet Enterprise can utilize this to consume SAP data and processes.

Duet Enterprise is real, effort is in it’s correct application

In the 1st week of May, The Next View participated in the second edition of the Duet Enterprise Velocity event. We already joined the first edition, and thus are familiar with the Velocity concept. As consequence, we could start immediate at the begin of the Velocity week on the realization of our customer’s scenario: Coöperative Learning.

Duet Enterprise is a given; real effort is in it's application AND understanding the business process + SAP business package

Noteworthy finding is that Duet Enterprise knowledge + experience is a necessity to successfully and productively apply this SAP-SharePoint interoperability middleware. However, this alone is not sufficient. Also required is that your customer has, or can reach, a clear vision of the application context they aim to achieve. In a normal project (with aim to deliver business value) this is of course the essential starting point. In our development approach we facilitate this by usage of mock-ups: discuss and derive in workshop(s) together with customer representatives the application functionality and behavior. The visualization format can be as simple and basic as sketches on paper, onto fully operational prototypes the customer can click through. SharePoint GUI + SharePoint Designer are very helpful for achieving the latter.

With the customer vision visualized in mock-up, the next step is to map this User Interface Design (UID) onto the combined SAP and Microsoft landscape. Here it is essential that you understand, and be aware of the behavior of the SAP business package building blocks.

An example to illustrate this

Part of Coöperative Learning scenario is to deliver Training Collaboration Workspace. Each workspace displays information about the training itself, about the participants, qualifications. SAP Enterprise Learning contains standard BAPI Function Modules to retrieve this information. It appears as simple as to identify the proper Function Module, and expose this via Duet Enterprise to SharePoint. But in reality the required insight in the SAP business package goes further. You must also be aware of the proper way of Function Module invocation, depending on the application context. For our Training Workspace, we apply the standard BAPI LSO_TRAINING_GET_DETAIL_C to retrieve training title, description and date. On day X this worked correct. Next morning, I received functional error ‘Course already passed’ from SAP LSO. It appeared as if the standard BAPI FM was not usable to retrieve data on elapsed training data, making it unusable for historical workspace context. Strange enough, a co-worker could still retrieve the training data. Our differences? He was enlisted as participant in this training, in contrary to me. And the standard Function Module responds on this current user context by only allow course participants to view the training data. That is, if you invoke the Function Module in it’s default modus ‘User_Role = Learner’.

When you overrule this for anonymous modus, you can successfully retrieve the training data also for elapsed dates, irrespective of the participation status of current user. A (proper) Duet Enterprise manner to do this is in the Gateway Model mapping.

Blueprint Duet Enterprise scenario's

The Next View recognizes the derivation and mapping from business vision via UID to the combination of SharePoint capabilities and SAP business package functionality + building blocks as essential step to deliver the correct application. We therefore incorporate this structural in our applied Duet Enterprise development process. As soon as we receive the initial business approvement on the UID translation of the business vision, next step in The Next View process is to deliver a Blueprint for the Duet Enterprise scenario’s that can be identified from the UID. Involved roles in delivering the Duet Enterprise Blueprint are a) Duet Enterprise Solution Architect as owner and facilator, b) SAP Business Analist with functional and high-level technical knowledge of the SAP Business Package, and c) SharePoint (Lead) Developer with knowledge of the SharePoint front-end capabilities. The purpose of the Duet Enterprise Blueprint is to derive and describe the high-level Solution Design; identifying the SAP and the SharePoint side. In the Solution Design, also fundamental design decisions on the SAP-SharePoint boundary are made and argumented. And when applicable, for alternative approaches explained why decided not to apply that.

In the process of derivng the Blueprint, it may also become clear that the requested UID cannot, or only with substantial development effort (= costs), by achieved by the SAP + SharePoint combination. Whenever such a consequence is identified, this is discussed with the UID expert and business. Is it feasible to alter the UID to accompany the standard SAP + SharePoint functionality? And if not from either functional or UID perspective, is the business willing to pay the extra development and also maintenance costs? Thus, the Blueprint derivation has an iterative effect and relation towards the UID and the application functionality.

Microsoft Trends 2012

At The Next View we regularly have 'TheNextKnowledge' (pizza)sessions to update each other on subjects that are or will become of interest for us. The subjects can be of pure technical nature (SAP Gateway, SharePoint 2010, Duet Enterprise), architectural (e.g. Gateway vs PI), functional (BI Dashboards).
Our latest session addressed innovation and trends as we (!) recognize them at our major technology suppliers. As 'true trend watchers', our statements are completely on personal notice, and on intention also sometimes 'bold'.

Update: On 18 June, Microsoft announced the Microsoft Surface Tablet. This is a new product, and different from the 'old' Microsoft Surface (Multi)Touch Table product referred in this trends overview.

Duet Enterprise BDC Publisher requires 'Z' prefix

Beware that when using the BDC Publisher tool to generate the SAP Gateway service (in SPROXY) and associated SharePoint BDC Model, to enter a prefix that starts with a 'Z'. Otherwise the BDC Publisher will immediately abort, with error message 'No license to edit object R3TR FUGR...'. I did not immediately link this error message to the apparent SAP obligation of starting 'Z'. It cost me some elapse time (a.o. checking our systems license, my own profile, ...) before I rather coincidently detected that starting the BDC Publisher prefix with 'Z' fixes the issue:

Apparent this Z-obligation is common in SAP development; so common that consulted SAP experts didn’t even think of this as cause. The displayed error message also doesn't immediate puts you on the right direction.
I notified the Duet Enterprise product development team of this issue, including suggestion to prevent it by auto-fill in the 'Z' in the BDC Publisher screen.

BusinessDataItemBuilder not usable on publishing site

SharePoint BCS out-of-the-box delivers a number of useful webparts to render external data. The BusinessData webparts consistently apply the architecture pattern of data provider and data consumer. A provider-consumer example is that of the BusinessDataItemDetails webpart to render an external data item, connected to a BusinessDataItemBuilder webpart that provides the external data item. The latter applies the 'ID' querystring parameter in the page URL to determine which entity to retrieve from the external administration. A standard example where the 2 webparts are used in this combination are the BCS profile pages.

It appears a sensible approach to use the same webparts combination for rendering an external entity item on a publishing page. However, when opening the page with the 'ID' querystring parameter referencing a BCSidentity:

http://tnvduetenterprise/subsite/Pages/TestSAP.aspx?ID=0aab

SharePoint throws the following error:

PortalSiteMapProvider was unable to fetch current node, request URL: /subsite/Pages/TestSAP.aspx, message: Value cannot be null. Parameter name: key, stack trace: at System.Web.SiteMapNode..ctor(SiteMapProvider provider, String key, String url, String title, String description, IList roles, NameValueCollection attributes, NameValueCollection explicitResourceKeys, String implicitResourceKey) at System.Web.SiteMapNode..ctor(SiteMapProvider provider, String key, String url, String title, String description) at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapNode..ctor(PortalSiteMapProvider provider, PortalWebSiteMapNode webNode, String key, NodeTypes type, String url, String title, String description) at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapProvider.get_CurrentNode()

The cause is that the masterpage of publishing site includes PortalSiteMapProvider and PortalSiteMapDataSource. And the latter also reacts on the presence of the 'ID' querystring parameter, but expects a page identification as value instead of the BCS identity.

Justification of the Duet Enterprise XML-Funneling pattern

XML-Funneling pattern as answer for dealing with complex SAP data structures

In the blogpost ‘Architecture Pattern XML-Funneling of complex SAP structures for Duet Enterprise interoperability’, I introduced the XML-Funneling pattern as pragmatic approach to handle the Duet Enterprise limitation regarding complex SAP data structures. This Duet Enterprise limitation has 2 origins:

• The SAP NetWeaver Gateway generator tools are not capable of preservering hierarchical structure; their simple execution model is to flatten the SAP structure in the Gateway DataModel.
• The SharePoint BCS UI-controls are not capable of rendering hierarchical structure; their default rendering model is list/row based.

An often-heard misunderstanding is that Duet Enterprise itself, via the foundational layers SAP NetWeaver Gateway and SharePoint BCS, cannot be used to exchange complex data structures between SAP and SharePoint. This is not true. Gateway can expose complex data structures, but currently then requires you to model the interface and data signatures in ESR Builder, and next manually develop the Gateway Mappers. Although in FP1 less cumbersone as in Duet Enterprise 1.0, it is by far not the preferred approach to expose SAP building blocks as Gateway services. SharePoint BCS as integration layer can consume complex external data structures, but the rendering via standard controls is limited to the root level. In case the child levels also must be visualized, you need to develop a custom view that interoperates with the BCS API. This is doable, although a painpoint is that the SharePoint BCS API is weakly typed, and therefore programming against the API is laborsome and error prone.

I invented the XML-Funneling approach to simplify the development process at the SAP side. By mapping the SAP hierarchical structure to an XML string, the Gateway generator tools can be applied: a string is namely a ‘simple’ type.
On the SharePoint side the XML-Funneling approach does not really make developer life easier. A marked disadvantage is that it requires custom development to deserialize the received XML-structure into a strongtyped data object, before you can next use and visualize the received complex SAP data structure in a SharePoint view.

Trade-off decision whether or not to apply XML-Funneling

A trade-off decision is then whether in the total SAP + SharePoint system architecture and development effort, this front-end side / .NET disadvantage outweights the advantage on SAP side? Is it sensible to abandon the XML-Funneling approach in case of complex SAP data structure? My advice is to still apply the XML-Funneling pattern. Justification herefore are the following considerations.

Starting-points

• SharePoint is the target front-end platform.
• Favour out-of-the-box SharePoint above custom development.
• Application of SharePoint BCS; i) for the connectivity handling to external data administrations, ii) for the out-of-the-box UI controls to render external data, iii) and for integration within SharePoint Enterprise Search and FAST.
• Complex data structure is common within SAP business packages

Are there sensible alternatives for XML-Funneling approach?

WCF + DataContracts as alternative for Duet Enterprise to consume SAP data
First, Duet Enterprise is itself a WCF variant: the SAP data is consumed in SharePoint BCS as WCF service endpoint.
Second, on SharePoint side BCS is the limitating factor wrt complex structures, not Duet Enterprise. What if you bypass BCS and directly consume the Gateway Services as WCF endpoints? The result is that you not only loose the added interoperability value of Duet Enterprise: Single Sign-On, Authorization, Logging and Monitoring. But in addition also all of the native values of SharePoint BCS: standard UI-controls to visualize data (not all exposed SAP data is of hierarchical / complex structure), exposal to offline usage, use in SharePoint Search or FAST context.
Conclusion: this alternative is not sensible; it does not comply with the starting-points.

Expose complex SAP data structures via Duet Enterprise into SharePoint BCS
SharePoint BCS is a generic integration layer, usable for arbitrary external administrations: SAP, Siebel, Oracle, SQL-databases, ... This transparant interoperability capability is supported through a script-like, weakly typed BCS programming model. Whenever you can operate with BCS imported external data through any of the available standard UI controls, this doesn’t impose a problem. The controls abstract the weakly typed BCS programming model. But the standard controls are not able to visualize complex external data structures. For them also in this alternative you are required to custom build a view. Due the weakly typed programming model of the BCS API this is laborsome and error-prone.
Conclusion: this alternative is on SharePoint side an hardly better, programmer-friendly approach for consuming complex BCS consumed data; merely different. At SAP side, the consequence is that you cannot use the Gateway Generator tools/pipeline.

Expose complex SAP data structures as multiple associated simple structures
SharePoint BCS supports the concept of associations between two external content types. As parent-child is a relation, you can use this concept to expose complex SAP data structures through Duet Enterprise into SharePoint BCS.
BCS associations certainly has value, e.g. for master-child data model and UI-formats. But it also suffers from limitations that make it less applicable in other scenarios. For one: the net effect is that to retrieve a single complex SAP structure, multiple invocations from SharePoint to SAP must be done to retrieve the entire structure. And the responsibility for retrieving the entire structure is placed on the front-end side; instead of the data provider side. Another, the standard SharePoint UI controls all operate on a single BCS entity. If in the User Interface Design the ‘child’ data is directly mingled with(in) the ‘parent’ data, the standard controls cannot be used.
A Duet Enterprise design time issue is that the SAP Duet Enterprise generators are focussed on a single Gateway datamodel, and have no support for defining associations between 2 datamodels. Usage of BCS associations requires you to afterwards modify the BDC Model that is generated by BDC Publisher to add the associations. Either via SharePoint Designer, but in my experience you will more often than not need to resort to manual modification of the BDC Model. Which is time consuming, and error prone.
Conclusion: associations suffers from limitations and software architecture + performance drawbacks.

Final thought

Given that you want to benefit from the interoperability values of Duet Enterprise, and that within the exposed SAP business package there are entities with complex structure; XML-Funneling is a feasible approach. Consistent application of the pattern results in recognizable code setups for multiple Duet Enterprise scenarios, on SAP and SharePoint side. And in our experience, it is also possible to develop reusable building blocks for SAP and SharePoint to ease in the application of the pattern.

Execution flow of Duet Enterprise single sign-on

One of the much appreciated Duet Enterprise "plumping" capabilities is single sign-on (SSO) from SharePoint farm into the SAP landscape. This capability is achieved by reusing available capabilities in both SharePoint 2010 and SAP NetWeaver platforms. For SharePoint this is BCS WCF Connector, Secure Token Service application and SAML2 support; at SAP side the existing table for external users 'VUSREXTID' and also SAML2 support.

How does Duet Enterprise authentication and single sign-on work out at runtime moment? The end-user starts at SharePoint front-end, thus authenticated as SharePoint user. The SharePoint authentication can be via any authentication provider. However, if you also intend to use the Duet Enterprise Roles Synchronization capability, this requires Claims-Based authentication. Upon invoking a call via SharePoint BCS through Duet Enterprise connnectivity handling to the SAP NetWeaver Gateway system, Duet Enterprise maps the authenticated SharePoint account onto a SAP accounts. The staps in this user mapping are:

1. The SharePoint authenticated user identity is sent to Microsoft Business Connectivity Services Windows Communication Foundation connector.
2. The BCS WCF connector invokes the SharePoint Security Token Service to retrieve the user token for this SharePoint user identity.
3. The SharePoint Security Token Service returns a token that identifies the SharePoint user.
4. Duet Enterprise SSO handling sends the received token to SAP NetWeaver Gateway, packaged within a SOAP request
5. SAP NetWeaver Gateway uses the received token to identify the external user, and locate in the user mapping table (VUSREXTID) the associated SAP user with this credential
6. The SAP user account that is mapped to the SharePoint user is returned to SAP NetWeaver Gateway.
7. SAP NetWeaver Gateway uses the SAP user account to request access to information in the SAP backend.
8. If the user is authorized to access the information, the requested information is sent to SAP NetWeaver Gateway.
9. SAP NetWeaver Gateway sends the requested information to the Microsoft Business Connectivity Services WCF connector as a SOAP response.
10. The Microsoft Business Connectivity Services connector passes the information to the SharePoint front-end side; e.g. to display via an External List or any of the other BCS Business Data webparts.

For this SSO pipeline to actually work, requires and strongly depends on correct execution of Duet Enterprise configuration of SharePoint and SAP mutual authentication. SAP Gateway must be configured for SAML2 usage, a trust relationship must be configured between SharePoint and SAP, SSL certificates have to be exchanged between SAP and SharePoint. But this is only a one-time, at configuration step. Afterwards you can rely and profit on Duet Enterprise Single Sign-On capability, without need to maintain the supporting code.