OData Channel (ODC) development now preferred for new Duet Enterprise scenarios

As of Duet Enterprise FP1 SP3, the Duet Enterprise product team now clearly positions OData Channel as the recommended approach for new developments of custom Duet Enterprise scenarios:

Feature Pack 1 for Duet Enterprise 1.0 Development Introduction

Generic Channel remains supported, also within Duet Enterprise 2.0; but will not be enhanced with new features and capabilities. Message: OData is the way to go. Main reason is that OData has much more potential for consumption outside the SAP Business Suite boundaries, and therefore the Duet Enterprise team is focusing its resources on supporting that more prospective channel.

Consequence for Duet Enterprise custom development: learn new approach + tools

Although one thus still has the option to utilize the Generic Channel development approach, the signal send by Duet Enterprise team is clear. OData development has the momentum and the future. Consequence for custom Duet Enterprise development is that on SAP side the programmer must transition to develop Gateway OData Services instead of GSDO types. The concepts and steps of the 2 are somewhat comparable, but the tools and execution + coding are different. For GSDO the ABAP programmer uses SE80 and /IWFND/GWO_GEN; OData development is performed mainly within the new Service Builder (SEGW). Within both approaches you have the options to generate the data provider class through mapping to RFC and BOR, or to manual handcraft it. Once the data provider class is ready (OData DPC or Generic Channel GSDO), you expose it as SharePoint 2010 callable entity. For OData DPC, the BDC Browser tool is extended with the capability to convert OData Services into Gateway SOAP web service. At runtime, Gateway applies for this the Duet Enterprise FP1 SP3 OData-SOAP Bridge. The existence of the OData-SOAP Bridge is transparent for the SAP Duet Enterprise developer. One simple utilizes the BDC Browser as before for GSDO type, to now expose the OData Service for consumption within SharePoint 2010.

Noticeable is that for SharePoint 2010 development side, nothing changes. You still import the BDC model generated via BDC Browser containing one or more External Content Types that refer to Gateway SOAP webservice(s). And next utilize the out-of-the-box Business Data UI-controls, and/or the BCS API to develop a custom UI.

See also:

Duet Enterprise 2.0 General Available: improvements on service development and deployment options

After successful completion of Duet Enterprise 2.0 Ramp-Up and with SharePoint 2013 reaching RTM state, SAP and Microsoft have made the new version of Duet Enterprise General Available. Customers can download the software from SAP Market Place (Installations and Upgrades \ Duet Enterprise \ Duet Enterprise 2.0). Duet Enterprise 2.0 documentation can also be found on SAP Market Place, and also on MSDN.

Most significant of the new Duet Enterprise version is that it connects with the latest capabilities of the underlying platforms, respectively SAP Gateway 2.0 and Microsoft SharePoint 2013. In both these platforms a move is made to support cloud and lightweight usage scenarios - typically but not limited to mobility use cases.

Duet Enterprise 2.0 in overview

Key topics:

• Mobility
• Duet Enterprise now fully supports OData protocol; which is THE protocol for mobility because is it light-weight and self-describing, and supported by all mobile platforms
• Cloud
• Connect to SharePoint Online
• Only support for Odata channel; not for SOAP / Generic channel
• Requires a 'light' on premisse SharePoint installation, as intermediate layer ('broker') from internal SAP Gateway to SharePoint online. Includes SSO Generator, via reverse proxy connection with SharePoint Online
• Remark: Duet Enterprise cloud stuff is not yet GA; this will follow in a few month

Facilitate custom Duet Enterprise development

• Design Time improved
• 2 design time approaches supported
• Inside-Out: via the SAP NetWeaver Gateway generators
• Outside-In: start from SharePoint UI --> visually sketch / specify a screen in Visual Studio; export the .EDMX file and import into SAP Gateway (Service Builder) for constructing the Gateway data models
• Both approaches are restricted to flat SAP data entities
• Complex data structures still requires hand-crafting on SAP Gateway side
• Via OData service providers; the handcrafting work is far simpler as with Duet Enterprise 1.0 (FP1).
• Improve the development story:
• Reference architecture guidance
• Setting up local development landscape --> facilitate individual SharePoint developers to quickly start with Duet Enterprise development

Consumption of OData services

• Consequence: drop SAML, as OData does not support SAML. Use oAuth instead
• Generic Channel will still be supported; OData is extra channel possibility

Aligned with SharePoint 2013

• Duet Enterprise 2.0 REQUIRES SharePoint 2013
  • SharePoint 2013 BCS supports Odata consumption; earlier version not
  • SharePoint Online.

Duet Enterprise 2.0 deliberaty remains a SAP / Microsoft interoperability + development framework

In the philosophy of SAP and Microsoft, also Duet Enterprise 2.0 does not include end-user functionality, e.g. order-to-cash. Duet Enterprise remains a customization + development framework for SAP / Microsoft interoperability, that enables end-organizations and partners to compose concrete scenarios and build upon. Primary reason for the Duet Enterprise product team to not include end-user functionality is that partners and end-organizations know best what diverse scenarios are requested, and are therefore better suited to construct the optimal solution. Also, even SAP plus Microsoft simple do not have the development resources to realize and support a total set of end-user capabilities for which there is market demand.

Growing Duet Enterprise eco-system

It is left to the market / Duet Enterprise eco-system to deliver standard / common-of-the-shelf end-user functionalities. Well-known example is the Cordis Suite; for self-service HCM and purchase requisition solutions. More examples can be found on Duet Enterprise Unite Partner Connection. And more partners are known to be developing on additional scenarios.

With Duet Enterprise 2.0 General Availability, it is expected that more companies will find it benefial to develop end-user solutions. This is due the combination of Duet Enterprise 2.0 development improvements, out-of-the-box plus customization capabilities of SharePoint 2013 and Office 365, and deployment extended to cloud and mobile platforms resulting in a larger market. Partners will develop solutions for horizontal and vertical markets; for local to international markets, and from small/partial solutions to complete end-2-end scenarios.

Locale bug in ddwrt:FormatDate not resolved for external data

XsltListViewWebPart is a powerful out-of-the-box SharePoint 2010 capability to render data, using Xslt to format the data display as you like. One of the standard Xslt functions, ddwrt:FormatDate, exhibits an annoying bug when used within XsltListViewWebPart. In case the locale of SharePoint server is other than US, then day and month are switched if day is < 13. The rootcause is – according to Microsoft - that ddwrt:FormatDate expects the datetime value to be in UTC format, but in reality XsltListViewWebPart preformats datetime values to the SharePoint server Locale before the Xslt-transformation takes place.
The SharePoint 2010 February CU 2012 contains an hotfix for this bug, and introduces the ListViewWebPart property ‘EnableOrginalValue’.

<property name="EnableOriginalValue" type="bool">True</property>

The effect of this property setting is that XsltListViewWebPart will include multiple formats per datetime value in the preformatted data. By using the ISO8601 datetime variant, the display bug can be resolved:

<xsl:value-of select="ddwrt:FormatDate(@Name,'.ISO8601')]),1033, 3)" />

However, this does not hold in case of external data. XsltListViewWebPart ignores the ‘EnableOriginalValue’ flag when rendering external data; and does not preformat the 'ISO8601' variant. Inspecting the sourcecode of super class DataFormWebPart proofs this to be true:

Conclusion is that XsltListViewWebPart by design does not support the hotfix for external data. Reasoning is that SharePoint BCS expects datetime values received from external systems to be in UTC format. And if not in UTC (DateTime.Kind unequal ‘UTC’), SharePoint BCS converts the received DateTime value into UTC:

"External systems commonly standardize on Coordinated Universal Time (UTC) as the time zone in which to store data. While this works well for data storage, end users prefer to work with times shown in their local time zone. External lists and the External Data Web Parts tackle this problem by converting time zones before presenting DateTime fields to end users, as well as before submitting DateTime values to the external system.
After retrieving a DateTime field from an external system, the BDC runtime examines the DateTime.Kind property to determine its time zone. If it is UTC or Unspecified, no action is taken. If it is marked as Local, BDC converts it to UTC by applying an offset based on the SharePoint front-end web server’s time zone, as specified by its Windows settings.
Now that the time zone has been standardized to UTC, the external list (aka XsltListViewWebPart) examines the user profile settings for the current user; if a time zone has been specified, the time is converted from UTC to the user's time zone. If a time zone has not been specified (which is the default state), the time zone of the site on which the external list is hosted is used to convert the external data. The converted times are then presented to the user in the UI in a format (for example dd/mm/yyyy hh:mm:ss) based on the site locale."

Source: Professional Business Connectivity Services – Scot Hillier

Thus external data received within XsltListViewWebPart will only contain UTC datetime values; therefore the issue should not occur; and the hotfix is on purpose disabled for external datasource.

But this is a design fault. We have a situation in which we retrieve SAP data [using Duet Enterprise + SharePoint BCS] in UTC format; and then still encounter the issue in case day < 13. Not surpisingly, UCT merely applies an offset in hours to the locale datetime; day and month remain in the same range; with at maximum 1 day earlier or later. Therefore the ddwrt:FormatDate also manifest itself for UTC datetime in case day < 13. As the XsltListViewWebPart hotfix on purpose does not fix that, we need an own Xslt FormatDate function to properly render the UTC datetime received from external system.

Notification + data flow in Duet Enterprise Workflow mobile App

The Duet Enterprise Workflow capability can be used to expose SAP workflow tasks to SharePoint context. Once in SharePoint, it can be further built upon. You can extend the associated SharePoint workflow with additional workflow steps and business processing. Another option is to use the SharePoint Duet Enterprise Workflow site as integration layer to expose and handle the SAP tasks in a mobile application.

System architecture

In such system architecture, following systems/layers are present:

1. SAP backend / workflow engine: the host environment of the SAP workflows
2. SAP Gateway: general purpose SAP integration layer; here used to publish SAP workflows from backend to workflow notification subscribers
3. SharePoint Duet Enterprise Add-On: functions here as workflow notification subscriber; host of Duet Enterprise workflow instances, and task repository for the mobile application
4. Mobile view: retrieve plus display tasks, and propagate task approval decision back to the SharePoint task repository

Note that in this system architecture, the (mobile) view and SAP landscape operate disconnected; SharePoint is the glue between the backend and frontend parts.

Notification and data flow for exposing SAP task to non-SAP view

Sequence steps:

1. The SAP ERP workflow execution arives on a user decision task, that is configured for notification to Gateway (tx SPRO, Gateway workflow customization);
2. SAP Document Publisher composes a XML Payload document with task data: workitem identification plus business data. The addition of the workflow specific business data is optional in Gateway Workflow notification. If applicable, you can achieve this by extanding the workflow customization with the plug-in of a custom OutboundHandler;
3. SAP Document Publisher calls the SharePoint OBAWorkflowServer.asmx::CreateTask operation, with inputparameters the task(type) identifier, workitem identifier and XML PayLoad document;
4. SharePoint OBAWorkflowServer.asmx::CreateTask method:
   1. Queries the TaskLocator List in the Duet Enterprise Workflow rootsite, and determines on basis of tasktype and user language the Duet Enterprise Workflow subsite to which the received SAP task must be routed;
   2. Instantiates a new SharePoint workflow instance in the identified Duet Enterprise Workflow subsite, and associates this instance with the SAP workitem identifier;
   3. And uploads the received XML PayLoad document into the Workflow Business Data DocLibrary in the Workflow subsite;
5. The SharePoint Duet Enterprise Workflow contains by default only 1 step that represents the decision-making for the SAP task. It is possible to extend the SharePoint Workflow with additional steps and functionality. This will operate isolated from the SAP Workflow handling. The default approval task in the SharePoint Duet Enterprise workflow is the only connectionpoint between the SharePoint workflow and the SAP decision workflow task. Upon the Duet Enterprise Workflow arriving at the approval task, it appears in the SharePoint tasklist.
6. User opens the tasklist in the mobile application;
7. The tasklist App call SharePoint List.svc service (DataAccess Layer) to retrieve the open tasks for the logged-on user
8. SharePoint List.svc queries the SharePoint workflow tasklist, to determine and retrieve the open task items for logged-on user
9. SharePoint List.svc returns the task items in Odata format
10. The Duet Enterprise Workflow Approval Task and the XML PayLoad document are in the SharePoint Workflow administration logical associated to each other. If additional business extended properties are notified for the SAP workitem via OutboundHandler, the associated Business Data document must be opened from the SharePoint document library and its contents parsed to retrieve the additional properties.

Control and data flow for handling task decision from non-SAP mobile App to SAP workflow context

Sequence steps:

1. User makes a decision in the task management App;
2. Task management App calls SharePoint Workflow.asmx webservice to propagate the task approval decision to SharePoint task item;
3. SharePoint workflows.asmx updates the task item with approval decision;
4. The update on SharePoint taskitem triggers the TaskItemEventReceiver;
5. Triggered TaskItemEventReceiver::ItemUpdating method checks whether the SharePoint task item is completed; that is a approval decision is made. If completed, TaskItemEventReceiver calls [through SharePoint Business Connectivity Services] the Update method of External ContentType  Duet Enterprise Workflow Task;
6. SharePoint BCS calls for Duet Enterprise Workflow Task Update the wfUpdate method of SAP workflow consumer webservice; with input parameters workitem identifier, decision value and comments;
7. SAP workflow consumer webservice propagates task approval decision to the associated SAP workflow instance, and executes apply decision on the SAP workitem;
8. SAP workflow completes this workitem step, and proceeds to the next step in SAP workflow. If it was the last step in workflow, the SAP workflow is now completed;
9. TaskItemEventReceiver updates the status of SharePoint taskitem to completed. In default (generated) Duet Enterprise Workflow is this the only step/task, and the SharePoint workflow is completed. The SharePoint taskitem is removed from the users open tasks list.

Selective enable ScriptManager presence in SharePoint Publishing Site

It is a well-known symptom of standard SharePoint Publishing that it renders bloathed HTML pages. The cause is that SharePoint publishing pages serve 2 modes: the end-user view, and the editorial view. To serve the latter, considerable extra html and SharePoint javascript files are included that are of no use for the end-user [view].

Approach to mitigate this effect is by separating the editorial from the end-user view. Examples of this approach are Macaw Dual Layout and Mavention Flex Layout. At high level the working is to have a distinct masterpage for end-user view, and another for the content editor. Only the latter must have the means to enable SharePoint WCM functionality; the former can be designed ‘clean’: no Form runat=”server”, ScriptManager, and include of SharePoint javascript files (init.js, core.js, …). Besides loading faster, additional advantage is that pure HTML sites are also more SEO-friendly.

However, there typically remain situations in a SharePoint publishing site in which you do need the presence of ScriptManager and the standard SharePoint client-side scripts. Multiple of the standard SharePoint controls are dependent on ScriptManager: CalendarView, TaskList, External List (XstlListView), Business Data / BCS webparts, to name some. Also custom developed controls might depend on presence of ScriptManager, e.g. rotating banner control using jQuery libraries for client-side user interface behaviour.

You have basically 2 options to enable ScriptManager et al for Publishing Pages on with controls are placed that need the ScriptManager:

1. Duplicate the ‘clean’ end-user masterpage; and augment it with the required entities.
2. Create an additional Publishing PageLayout; and augment that the required entities.

Option 1 suffers from the following disadvantages:

• As consequence that a SharePoint SPWeb can only have 1 masterpage, ‘clean’ content publishing pages and WebControls pages cannot be within the same SharePoint SPWeb. In fact, the type of pages now influences the site structure. This is visible for the end-user. And for the content manager it implies that (s)he is forced to locate new page a priori within the correct SPWeb given the intended type of page: content or control. Also, as standard SharePoint navigation is based on the site structure, it will require custom development to repair the logical site navigation.
• Double maintenance of the site look&feel in the 2 public masterpages.

As in the 2nd option both page types use the same public masterpage, all pages can be within the same SharePoint SPWeb. So with this approach it is non-visible for the end-user, and also the editing experience is far more transparent for the content editor. Not entirely, as (s)he must apply the correct PageLayout given the type of page. But the applied PageLayout is something that in SharePoint publishing can be modified / corrected after initial creation.

From SharePoint technical perspective, the 2nd option is more difficult / challenging to realize as the 1st (this is simple: just copy the public masterpage and add ScriptManager et al). Issues here are following:

• The extended PageLayout must function both in combination with the clean masterpage as with the content-editor enabled masterpage. But the latter also as Form runat=”server” and ScriptManager included; resulting in 2 occurences of them. And this is not allowed in ASP.NET plus SharePoint.
• SharePoint scriptmodel imposes hidden dependencies on the order of including + evaluating/loading SharePoint javascript files (through SOD framework).

You can deal with these issues via following approach and architecture:

• Include Form runat=”server” in the extended PageLayout; but default not visible.
• A server control included in the extended PageLayout to dynamically insert the ScriptManager plus ScriptLink for core.js; but only if no ScriptManager present yet (included from the content-editor masterpage). When runtime determined that the ScriptManager must be inserted, also the invisible server-side Form is made visible; and all siblings of the Form control are moved within the server-side HtmlForm control.
• A server control included in the public masterpage to dynamically insert WpzEmitControl (explanation why needed) if needed; the condition for this is whether the InsertScriptManager control is include in the page context.

Result

Regular content pages still render ‘clean’ html in public view, without the unneeded SharePoint html and javascript. Control pages render in public view including the required ScriptManager and Form runat=”server”; and in editing view also with only occurrence of ScriptManager and server-side HtmlForm.

Deploy custom Duet Enterprise scenario from Dev into SAP landscape

In the case that an organization develops custom business scenarios on top of Duet Enterprise foundation, an essential step is to implement it through to the organizations SAP production environment. Duet Enterprise relies herefore on the standard CTS+ transport mechanism.

Design time - prepare the custom business scenario

It is best practice to include all of the Duet Enterprise services for one scenario within an own isolated and recognizable namespace in the Gateway system. Each scenario will then be recognizable, and not per accident mingled with deliverables of another Duet Enterprise scenario. To achieve this, you create per business scenario an own Software Component in SLD, and use that to create in ESR an own Software Component Version and namespace.

Steps:

1. Open SLD, and create a new Software Component
2. Open ESR, and create a new Software Component Version via import of the newly created Software Component
3. Create a namespace in the newly created Software Component Version
4. Copy the Basic DataTypes from SAP IW TNG to the new Software Component Version and namespace
5. Copy the Fault Message from SAP IW TNG to the new Software Component Version and namespace
6. Activate the resulting changelist(s) in ESR
7. Logon to Gateway system, transaction SE80; and create a Package
8. Logon to Gateway system, transaction SProxy; and create + activate proxies for the Basic DataTypes, ExchangeFaultData and ExchangeLogData in the newly created Package.

With this, the preparation phase is completed. You now have an own business scenario container for all services that will be developed for the scenario.

That is, within the Dev environment. The same container must be created on the counterpart Gateway system into Test, QA and Production environments to implement the custom scenario in these environments. To facilitate this, all of the design time preparation steps performed on the Gateway system are recorded in a CTS+ Transport Workbench request: Create Package (step 7), and Create + Activate Proxies (step 8).

Note that it is not needed to also implement or repeat the SLD and ESR steps into the Test, QA and Production landscapes. For Duet Enterprise design time, the SLD and ESR only have a role in the design time phase; in the development environment. Once the service proxy objects for ESR content are generated (basic / standard ones, and the custom developed), there is no dependency on ESR content. The transport therefore only consists of the service definition(s) and the proxy objects that are part of them.

Design time - create the service definitions for the custom business scenario

Per service this consists of the 2 familiar actions: create and map a Gateway DataModel, and next generate the service definition + BDC Model for this Gateway DataModel (GSDO Type).

Steps:

1. Logon to Gateway system, transaction SE80; and create a new Gateway DataModel; make sure not to select OData channel (Duet Enterprise 1.0 and SharePoint 2010 do not support OData consumption; this will change with Duet Enterprise 2.0 and SharePoint 2013). The system enforces you to include the creation of the Gateway DataModel in a Workbench request. The advice is to not include it in the same Workbench request from the preparation step, but instead use an own Workbench request for the design time deliverables. As result you have the implementation of the Package including the generic basic DataTypes + FaultMessage separated from the implementation of the specific scenario entities. It is typical in a custom realization project that you incremently build new versions of an own service definition, and if a previous version already transported you must be able to replace that while preserving the Package.
2. Map 1 of more operations in the Gateway DataModel; e.g. Query and Read
3. Generate the Gateway DataModel; the system enforces you to include the generation and creation of SAP entities within a Customizing request.
4. Open transaction BDC Browser; if this is the first service definition for the custom business scenario, create a new Business Scenario. Otherwise you may opt to include the 2^nd, 3^rd and so on service definition all within the same Business scenario. I advice this as best practice, you then end up with a single Duet Enterprise Business Scenario that contains the entire set of Service Definitions developed for it. In case you create a new Business Scenario, it is again enforced by the system to include it within a customizing request. You can select the same as earlier created for the Gateway Model generation(s).
5. Select the Business Scenario, and Create BDC Model for GSDO Type; select the Gateway Model earlier created. Also here the system enforces to include within a customizing request.
6. [Optional] Open transaction SProxy, select the generated service interface and test its correct working for the diverse methods in the interface (e.g. Find, Read, Create, )
7. [Optional] Open SOAMANAGER on Gateway system, browse to Business Scenario Configuration. Select your Business Scenario, and export + save into a local XML file.

Implementaton time - transport the custom business scenario

When the custom realization has progressed or is even completed in the Gateway system of the SAP Development environment, next is to propagate it within the SAP landscape.

Steps:

1. Logon to Gateway system of SAP Dev environment, transaction SE03 or SE10; and release all the Workbench and Customizing requests that are related to the custom Duet Enterprise Business Scenario
2. Logon to Gateway system of the transport target environment (SAP Test or QA environment), and open Transport Management System (TMS). Import the released Workbench and Customizing requests from the Development environment.
3. Activate each of the requests. Make sure to do this in the correct logical order: first package, next the business scenario including service definitions.
4. Logon to Gateway system of the transport target environment, transaction BDC Browser. Validate that the Business Scenario(s) is/are present. Also validate that for each of the imported service definitions, there is no end point yet.
5. Release and activate all of the imported service definitions in SOAMANAGER. Browse to Business Scenario Configuration, and import the Business_Scenario XML file that is exported in the Development environment. If the Business Scenario already exists here from a previous import, first delete it. Or otherwise you can also explicitly associate new service definitions within a/the Business Scenario, and assign a profile. After activation of the service definitions, validate in transaction BDC Browser that for each of the service definitions now an end point is available.
6. [Optional] Open transaction SProxy, select the imported service interface and test its correct working for the diverse methods in the interface (e.g. Find, Read, Create, )
7. Transaction BDC Browser, select the imported Business Scenario and click on Export Business Scenario. Hand over the exported SharePoint BDC Model(s) to import within the SharePoint BCS Service Application of the implementation environment.
8. [Optional] In SharePoint webapplication, create an ExternalList for one of the imported External Content Types. Validate that you can retrieve SAP data via Duet Enterprise and SAP Gateway using one of the transported / imported Gateway Service Definitions.

Duet Enterprise troubleshooting pointers

Generic

• Duet Enterprise 1.0 Troubleshooting Guide
• Troubleshooting example
• Holger Bruchelt's Duet Enterprise blogpage

SAP side

• BDC Browser - Troubleshooting pointers
• BDC Browser Configuration - Software Component SAP IW TNG missing
• Workflow tasks not delivered?
  • txn SLG1 on backend
  • txn SRT_UTIL on Gateway
  • txn SBGRFCMON: retry queue contains wf-items failed to deliver to SharePoint
    • use 'Unit Analysis \ Run Unit with Trace' to execute again/debug
• ABAP runtime errors / dumps: txn ST22
• ABAP system log: txn SM21
• Duet Enterprise 1 - Troubleshooting Report Delivery
• GW 2.0 SP5: Which Tools Can Be Used for Troubleshooting
• For Payload tracing:
  • txn SRT_UTIL
  • txn SMICM (ICF trace tools)
• ABAP Function Modules and services: External Debugging, Set External Breakpoints for ABAP Debugging
• Troubleshooting Gateway Development
• SAP system HTTP Logging
• Using Starter Services without EhP4

SharePoint side

• ULS Logviewer
• SharePoint Developer Dashboard
• Troubleshoot Business Connectivity Services using Performance Monitor
• SCOM (System Center Operations Manager
• Overview of monitoring and troubleshooting

Beware with BCS Filter on non-nullable types

Application scenario:

• ‘query’ operation in external system with one or more import parameters; with default values
  • Examples: SQL dynamic query or stored procedure, SAP Remote Function Call, SAP Gateway Service, WCF web service;
• ‘query’ operation exposed via a BCS::Finder method; each of the import parameters mapped to BCS Filter parameters
• SharePoint view on the ‘query’ result; e.g. via ExternalList, Business Data List, custom webpart

In such an application context you might experience that when you (on purpose or by accident) do not enter concrete values for the Filter parameters, SharePoint retrieves + renders a different set of external items as expected. You would expect the native ‘query’ operation to apply its default value(s). However, this is only true when the types of the import/filter parameters are nullable. If not nullable, SharePoint is required to submit a concrete value when invoking the Finder method. If missing from front-end invocation, SharePoint will submit the default value for the non-nullable type. E.g. for a DateTime, this means the value ‘1-1-1900’.

You have 2 options to alter this SharePoint BCS invocation behavior:

1. In the BCS Model, declare the non-nullable types as nullable.
   <TypeDescriptor TypeName="System.Nullable`1[[System.DateTime, mscorlib, Version=1.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]" Name="IV_SEARCH_BEGDA" AssociatedFilter="IV_SEARCH_BEGDA">
2. In the BCS Model, search for a method parameter with name "<filter-import-parameter>Specified". If present, make sure it’s default value is set to ‘false’.
   <TypeDescriptor TypeName="System.Boolean" Name="IV_SEARCH_BEGDASpecified">
       <DefaultValues>
           <DefaultValue MethodInstanceName2="Find_ExternalItems" Type="System.Boolean">false</DefaultValue>
       </DefaultValues>
   </TypeDescriptor>

OData service support in Duet Enterprise - 2.0 and 1.0FP1 SP3

SAP NetWeaver Gateway 2.0 has within the SAP architecture the role of transparent integration layer to the SAP Business Suite:

Duet Enterprise is a consumer of the SAP Gateway layer, Sybase Unwired Platform is another consumer for Mobile devices. The access through Gateway is based on current interoperability standards: W3* (SOAP / WSDL) Web Services and OData services (REST protocol).
In the current Duet Enterprise 1.0 FP1 product OData support iswas missing. The direct explanation is that SharePoint 2010 BCS does not support OData consumption. For Duet Enterprise interoperability it is was therefore necessary to unlock SAP data via the SAP Gateway Generic Channel. Sybase Unwired Platform since version 2.1 does have OData consumption support.
From the market (partners, including The Next View, and customers) is urged to include OData support in Duet Enterprise. Main limitation is, however, SharePoint 2010 BCS.

In SharePoint 2013, BCS supports OData consumption. And Duet Enterprise 2.0 can therefore also support the interoperability via Gateway OData Channel. Integration via Generic Channel continues to be supported, in terms of design-time and runtime.

SAP Gateway includes generator tools for Generic and OData channel. Both are subject to the restriction that SAP hierarchical data structure is not supported (no mapping capability). In case of hierarchical data it is possible to 'handcraft' the Gateway service in ABAP code. The programming model for custom OData providers is thereby a lot simpler compared to that for Generic Channel services.
XML Funneling architectural pattern, via which we still have the opportunity to utilize the benefits of the Gateway generator tools.

SAP and Microsoft have recognized that (potential) Duet Enterprise 1.0 FP1 customers, now want to connect to the future OData option of Duet Enterprise 2.0. For example, to reuse the same Gateway service for Mobile channel and SharePoint / Duet Enterprise. To make this possible, with Duet Enterprise 1.0 SP03 (released today, September 14) an OData-SOAP Bridge is included. In short this Bridge maps OData-based output of Gateway OData services to SOAP services, which then integrate into SharePoint 2010 BCS. The OData-SOAP Bridge is restricted to 'flat' SAP data structure, data-retrieval of complex SAP structure to SharePoint is thus not possible.

In Duet Enterprise 2.0 the OData-SOAP Bridge is not included; SharePoint 2013 BCS itself can directly consume OData.

Summarized:

• Duet Enterprise 2.0 requires SharePoint 2013
• In Duet Enterprise 2.0 the OData + Generic channels are supported; SharePoint 2013 BCS can consume OData services
• For Duet Enterprise 1.0 a OData-SOAP bridge is available via SP03 to bridge Gateway OData services for consumption as SOAP web service in SharePoint 2010 BCS
  • Application of the Bridge is restricted to flattened SAP structure; in reality SAP data often has a hierarchical / complex structure (e.g. SAP LSO)
• SAP advice on the interoperability endpoint decision: OData Channel or Generic Channel?

User-centric Development Process for Duet Enterprise projects

In essence, a ‘Duet Enterprise custom scenario’ development project is not different from other application development projects. The customer – either the real end-user, or product management for package software -, has an idea about what to achieve and functional + non-functional (quality) requirements to underline the idea. Often this idea is still vague, and the requirement set far from complete and on aspects also contradictionary. The waterfall approach to yet start with application development (specify, build, test) typically ends up with wrong or failed end-results: the application does not live up to the real customer needs and expectations, and the budget is largely overrun as consequence of responding on the requirements changes. The agile movement came up to improve on this bad practice. Central is the structural involvement of the user (representative) during the full course of the development project. Continous, the user is in the lead about the specifics and the behaviour of the application functionality to be delivered. In Scrum, this responsibility is formalized within the product owner role.

Application mockup

A proven approach to enable the product owner to validate, correct, and ultimately confirm the application functionality is by make it experiencable, ‘touchable’. Provide the product owner with a realistic impression of the application behavour + feature(s) that the project will build: a clickable prototype / mockup.

Based on multiple positive experiences (also within non-Duet Enterprise project, e.g. BI Dashboards), we strongly believe in the added value and power of application mockup. We regard the role of UX / UID expert crucial for customer involvement, and thus for delivering the correct application functionality.

Meeting of 2 worlds

Yet, a Duet Enterprise project also is very much different from ‘normal’ projects. The cause lies with the meeting of 2 very different environments: IT technology, platform concepts, and human nature. The tradional SAP environment is transaction oriented, with focus on structural business process executions. The Microsoft environment, and in particular SharePoint, is far more loose. Ad-hoc processes are the norm, the individual user decides how to perform a specific (business) action. When these 2 worlds + thinking come together in a Duet Enterprise project, the first outcomes are often dramatic. SAP versus SharePoint architects, business analysts and developers do not understand each other (concepts). Both ‘parties’ disagree on how and where (that is, on SAP or on SharePoint side) custom development should be done. The personal preference on how to achieve the SAP+SharePoint integration is influenced by one’s own technology frames and familiair concepts. This has the risk that the user is forgotten: the user interests must be central, deliver an optimal user experience.

Align through Blueprint Duet Enterprise scenario’s

To break through this, we utilize in our Duet Enterprise development process a blueprint step with the focus exclusive on the Duet Enterprise specific scenarios. This blueprint is derived and agreed by a mixed group of SAP and SharePoint representatives. The goal of this step, and its endresult, is to come to a mutual understanding of the solution direction for each of the Duet Enterprise scenario’s. Self-employed constraints are to stick as close to the standards of both the SAP business package and SharePoint platform. Where feasible, we do allow custom development. Here we apply Duet Enterprise integration patterns that we have identified and defined: Consumer-oriented, Provider-oriented, and XML-Funneling pattern.

Of course, the discussion and preferences about where to make custom adjustments still manifest within the Duet Enterprise blueprinting step. However, as it now has the full focus of all involved, mutual agreement is earlier and more effective reached. This is the so-called ‘workshop effect’. Also, in this focussed approach, comparable application situations are easier to recognize. And where so identified, the earlier agreed solution approach can be reused. Starting point for the blueprint derivation is the User Interface Design as validated and confirmed by the product owner.

In the blueprint we derive and define per identified Duet Enterprise scenario the following pieces:

1. Global descripion of the scenario: screenshot of the mockup screen and textual explanation. The purpose is to facilitate on high level a common understanding of the functionality in the scenario. There is no intend to compete with or replace use cases.
2. Derivation, explanation and justification of solution direction. At minimal explanation of the proposed solution direction. But whenever applicable, also the rationale of why an alternative solution direction is dropped.
3. SAP / SharePoint interface
4. SAP Gateway datamodel(s)
5. SAP building blocks: standard RFC’s, BAPIs; identification of needed custom SAP building blocks
6. SharePoint building blocks: standard and custom webparts, SharePoint Service Applications, …

Blueprint validation by Design Authority / Authorities

Justified mantra nowadays within SAP and Microsoft IT departments is to avoid unnecessary or otherwise avoidable custom development. Instead stick close to the standard delivered functionalities of both landscapes.

It is also evident that fully excluding custom development is not feasible when you are developing company custom scenario’s.

We facilitate the consistency and continuity in the Duet Enterprise solution directions by confirming the blueprint to Duet Enterprise Rules, Conventions and Guidelines. The Design Authorities of both SAP and Microsoft IT departments validate the blueprint against these agreements. Derivation can be allowed, but requires the formal approvement of the respective Design Authority (SAP, Microsoft, and in some cases both).

SAP + SharePoint realizations based on the blueprint

With the blueprint accepted and delivered, from here on SAP and SharePoint teams in the project can start with their own development activities for the Duet Enterprise scenario’s. Since representatives of the teams were involved in deriving the blueprint, the concrete software design + development can be done relatively independent of the other party. Of course the data contract details of each SAP / Duet Enterprise interface are input for the SharePoint consumption. Regular sharing the SAP design with the SharePoint developers can achieve this.

Duet Enterprise Development Process

	Step	Responsible	Consulted	Approach	Input
1	Business analysis	Business Analyst	Business representatives	Interviews Workshops	Business vision
2	Mock-Up	UID / UX expert	Business representatives Business Analyst	Sketching Prototyping	Style guide
3	Blueprint	Duet Enterprise Solution Architect	SAP Business Analyst SharePoint lead	Workshop	UID Duet Enterprise Rules, Conventions, Guidelines
4	FoTo’s	SAP Business Analyst	SAP developer SharePoint developer Duet Enterprise Solution Architect		Blueprint Duet Enterprise Rules, Conventions, Guidelines
5	Build	SAP developer			Blueprint FoTo’s SAP Build – Rules, Conventions, Guidelines
		SharePoint developer	Duet Enterprise Solution Architect		UID Blueprint FoTo’s SharePoint Build – Rules, Conventions, Guidelines
6	Integrate	SAP developer + SharePoint developer			Operations – Rules, Conventions, Guidelines,

Deploy Duet Enterprise in production infra landscape

Duet Enterprise PoC in sandbox environments

Organizations that evaluate Duet Enterprise for the added value it brings typically do so via a proof of concept. Mostly, these PoC’s are executed in an isolated sandbox, a laboratory environment. Naturally also in the sandbox all the minimal prerequisites must be present in order to rightfully test the effect and operation of the SAP-SharePoint interoperability. Thus (an element of) SAP Business Suite, a server hosting SAP NetWeaver Gateway 2.0 with the Duet Enterprise SAP Add-On installed, and a server hosting SharePoint 2010 with the Duet Enterprise SharePoint Add-On installed. But for cost reasons, the sandbox infra is typical less extensive as in the regular production environment. Thus no load-balancer, SharePoint 2010 farm in the single-server setup, no proxy server, no firewall, et cetera. Infrastructure elements that are regurlarly present in nowadays mature infra architectures.

Note: to facilitate organizations that quickly want to inspect and evaluate Duet Enterprise, without spending effort in installing it within own landscape; the Duet Enterprise team has prepared a fully installed Microsoft Hyper-V image: SAP NetWeaver Duet Enterprise Trial System.

Deploy Duet Enterprise in production landscape

At a client organization we also started with a Duet Enterprise PoC, in a sandbox environment. On basis of the succesful outcome, Enterprise Architecture has mandaded the application of Duet Enterprise as the organization standard interoperability approach when it concerns SAP-SharePoint business scenarios. Next logical, and a required step, is to make Duet Enterprise available for generic utilization in the regular IT landscape. Via the Duet Enterprise wizard a lot of the Duet Enterprise deployment effort on SAP side is automated, largely transparent to the specifics of the SAP landscape. For SharePoint side, there is no wizard. But the installation effort there is far less as on the SAP side.

Duet Enterprise deploy differences sandbox versus production landscape

During deployment of Duet Enterprise in the production landscape, we experienced a number of effects resulting from differences in the production versus sandbox landscapes.

SAP side

• Starting from NetWeaver 7.31, the consumer proxies for Duet Enterprise workflow are configured via transaction SOAMANAGER. The proxies configuration in LPCONFIG, although still present and possible, no longer has effect.
• For design-time, the ESR Builder must be associated with the Software Component Version and namespace in the Gateway system where custom development will reside. It is not feasible in production landscape to create Gateway DataModel and business scenarios as Local Objects.

SharePoint side

• At the external boundary from end-user to SharePoint, the infrastructure contains a Load Balancer with SSL-offloading to the internal network. Duet Enterprise however requires SSL enabled on the SharePoint 2010 server that is in direct connection with the SAP NetWeaver Gateway server, hosting the Duet Enterprise SAP Add-on. And the SSL certificate of the SharePoint server must be imported within SAML2 transaction of SAP Gateway system.
• For shared resource usage, and for security, the SharePoint infra is partitioned over 2 separate SharePoint farms: a SharePoint Service Applications farm with also a Central Admin webapplication; and a consuming SharePoint farm that hosts all the end-user SharePoint webapplications. Also SharePoint BCS Service Application is thus on the Service Applications farm. As BCS is the SharePoint consuming side in Duet Enterprise architecture, it seems logical to exchange the Security Token Service (STS) certificate of this farm to Gateway; to configure the SAML based Duet Enterprise Single Sign-On behaviour (see Execution flow of Duet Enterprise Single Sign-On). However, when we configured the Duet Enterprise landscape with the STS certificate of the Service Applications farm, the Single Sign-On derivation at Duet Enterprise connection time failed. In SRT_UTIL the infamous error message ”SSFW_KRN_VERIFY failed with: verification failed, see decrypted digest” was logged. Further analysis of the logged information in SRT_UTIL displayed that the REMOTE_ADDRESS in the runtime SAML connection is that of the consuming farm. After consultation with SAP Support we tried to use the STS certificate of this farm instead. And indeed this resolved the invalid certificate signature issue. The explanation for this behaviour is that the end-user that starts the Duet Enterprise request flow is authenticated within the consuming farm; thus also the secure token identifying the user must be taken from this farm.

Filter on current datetime in BCS::ReadList upon Search Crawling

SharePoint Business Connectivity Services plus SharePoint Enterprise Search and/or SharePoint FAST is a powerful combination. With them you can unlock selective data from external repositories and bring into SharePoint context to realize search-driven applications. The external data is retrieved at Search Crawl time, and administrated in the search index file.

The SharePoint Indexing Engine first invokes the Finder method of the external content type(s) that is tagged as ‘RootFinder’ in the BDC Metadata Model. And second per returned entity the SpecificFinder to retrieve more details. If all required entity details are already returned by the Finder method, the latter can be skipped for better crawl performance and throughput.

Via the Finder method thus the set of relevant entities of external content type is retrieved. The decision what is relevant is made from perspective of the external repository upon the moment of invoking the Finder/ReadList method. A typical filter here is that of current datetime: only return active external entities.

Current datetime is a dynamic value to be determined at Finder invocation time. A consequence is that this filter value cannot be a priori included in the BDC model. This leaves 3 options: derive and specify the dynamic value from either the context of the BCS::Finder consumer, from within the BCS::Finder provider as integration layer, or from the data access operation (web service, ADO.NET or custom connector) within the external repository.

BCS::Finder consumer context

In content crawling context the SharePoint Indexing Engine is the consumer. It is not possible the augment its the standard behaviour of invoking the Finder method.

BCS::Finder provider context

This option requires that the Finder execution invoke the associated access point of the external repository with DateTime.Now as filter parameter. This option is not supported for BCS no-code solutions, but requires the utilization of a custom BCS .Net Assembly Connector. If already a custom .Net Connector is used for retrieving data from the external repository, it is probably a minor change to extend it with this dynamic derived filter value. But if not, introducing the usage of a dedicated BCS .Net connector merely to filter on the current DateTime is somewhat of an overweighted solution approach.

DataAccess operation within external repository context

So that leaves as only option to apply the current datetime filter value within the interoperability point of the external repository. For example in Duet Enterprise scenarios to unlock SAP data to SharePoint: The invoked Query operation of the Gateway webservice must then self apply the system datetime. SAP NetWeaver supports this via the concept of default value for input parameters.
Sadly it is not possible to specify this at the level of the Gateway Data Model, so it must be specified in the invoked ABAP Function Module (RFC).

Note: avoid specifying application-specific default value in the RFC. Systemtime satisfies this guideline, but a concrete default value for DateTime input parameter typically does not.

See also MSDN forum thread on this subject: Invoke BCS::ReadList operation with 'today' as default value for datetime input parameter.

Influence on Duet Enterprise product development + roadmap

The Next View engages in select international comitee with SAP on development and roadmap of Duet Enterprise

The Next View is regarded in the Netherlands as the Duet Enterprise expert. SAP itself also recognizes this. We are therefore SAP Partner on SAP / Microsoft interoperability. SAP Netherlands refers customers that seek for information and guidance on this subject to us, and actively engages us in Duet Enterprise consultations. The Duet Enterprise product team in Walldorf and Bangalore [very much] appreciates the feedback we give based on our experiences with the product. As such we are also asked to participate in the Duet Enterprise Customer Engagement Initiative (CEI). This is a select international group of partners and customers that take the lead with implementations of Duet Enterprise. Participating organizations come from Germany, Australia, England, Denmark, and The Next View as sole and exclusive Dutch partner.

The CEI concept is to provide input for desired developments in the Duet Enterprise product roadmap. Also the Duet Enterprise product team demonstrates (potential) new product features, and inputs new ideas to discuss with and be assessed by the CEI participants. As the disclosed information concerns potential future product features, all CEI participants prior have signed a Non-Disclosure Agreement (NDA).

The recent Duet Enterprise 2.0 Preview announcement proves that via the CEI concept we (The Next View) indeed exert influence on the development of the product. Our main criticism on Duet Enterprise 1.0 is the missing of OData interoperability support. The underlying SAP NetWeaver Gateway integration layer includes OData provider support since version 2.0. The bottleneck here is that SharePoint 2010 Business Connectivity Services does not include OData consumption support. But now using OData sources with BCS is added in the upcoming SharePoint 2013, the next version of Duet Enterprise can utilize this to consume SAP data and processes.

Duet Enterprise is real, effort is in it’s correct application

In the 1st week of May, The Next View participated in the second edition of the Duet Enterprise Velocity event. We already joined the first edition, and thus are familiar with the Velocity concept. As consequence, we could start immediate at the begin of the Velocity week on the realization of our customer’s scenario: Coöperative Learning.

Duet Enterprise is a given; real effort is in it's application AND understanding the business process + SAP business package

Noteworthy finding is that Duet Enterprise knowledge + experience is a necessity to successfully and productively apply this SAP-SharePoint interoperability middleware. However, this alone is not sufficient. Also required is that your customer has, or can reach, a clear vision of the application context they aim to achieve. In a normal project (with aim to deliver business value) this is of course the essential starting point. In our development approach we facilitate this by usage of mock-ups: discuss and derive in workshop(s) together with customer representatives the application functionality and behavior. The visualization format can be as simple and basic as sketches on paper, onto fully operational prototypes the customer can click through. SharePoint GUI + SharePoint Designer are very helpful for achieving the latter.

With the customer vision visualized in mock-up, the next step is to map this User Interface Design (UID) onto the combined SAP and Microsoft landscape. Here it is essential that you understand, and be aware of the behavior of the SAP business package building blocks.

An example to illustrate this

Part of Coöperative Learning scenario is to deliver Training Collaboration Workspace. Each workspace displays information about the training itself, about the participants, qualifications. SAP Enterprise Learning contains standard BAPI Function Modules to retrieve this information. It appears as simple as to identify the proper Function Module, and expose this via Duet Enterprise to SharePoint. But in reality the required insight in the SAP business package goes further. You must also be aware of the proper way of Function Module invocation, depending on the application context. For our Training Workspace, we apply the standard BAPI LSO_TRAINING_GET_DETAIL_C to retrieve training title, description and date. On day X this worked correct. Next morning, I received functional error ‘Course already passed’ from SAP LSO. It appeared as if the standard BAPI FM was not usable to retrieve data on elapsed training data, making it unusable for historical workspace context. Strange enough, a co-worker could still retrieve the training data. Our differences? He was enlisted as participant in this training, in contrary to me. And the standard Function Module responds on this current user context by only allow course participants to view the training data. That is, if you invoke the Function Module in it’s default modus ‘User_Role = Learner’.

When you overrule this for anonymous modus, you can successfully retrieve the training data also for elapsed dates, irrespective of the participation status of current user. A (proper) Duet Enterprise manner to do this is in the Gateway Model mapping.

Blueprint Duet Enterprise scenario's

The Next View recognizes the derivation and mapping from business vision via UID to the combination of SharePoint capabilities and SAP business package functionality + building blocks as essential step to deliver the correct application. We therefore incorporate this structural in our applied Duet Enterprise development process. As soon as we receive the initial business approvement on the UID translation of the business vision, next step in The Next View process is to deliver a Blueprint for the Duet Enterprise scenario’s that can be identified from the UID. Involved roles in delivering the Duet Enterprise Blueprint are a) Duet Enterprise Solution Architect as owner and facilator, b) SAP Business Analist with functional and high-level technical knowledge of the SAP Business Package, and c) SharePoint (Lead) Developer with knowledge of the SharePoint front-end capabilities. The purpose of the Duet Enterprise Blueprint is to derive and describe the high-level Solution Design; identifying the SAP and the SharePoint side. In the Solution Design, also fundamental design decisions on the SAP-SharePoint boundary are made and argumented. And when applicable, for alternative approaches explained why decided not to apply that.

In the process of derivng the Blueprint, it may also become clear that the requested UID cannot, or only with substantial development effort (= costs), by achieved by the SAP + SharePoint combination. Whenever such a consequence is identified, this is discussed with the UID expert and business. Is it feasible to alter the UID to accompany the standard SAP + SharePoint functionality? And if not from either functional or UID perspective, is the business willing to pay the extra development and also maintenance costs? Thus, the Blueprint derivation has an iterative effect and relation towards the UID and the application functionality.

Microsoft Trends 2012

At The Next View we regularly have 'TheNextKnowledge' (pizza)sessions to update each other on subjects that are or will become of interest for us. The subjects can be of pure technical nature (SAP Gateway, SharePoint 2010, Duet Enterprise), architectural (e.g. Gateway vs PI), functional (BI Dashboards).
Our latest session addressed innovation and trends as we (!) recognize them at our major technology suppliers. As 'true trend watchers', our statements are completely on personal notice, and on intention also sometimes 'bold'.

Update: On 18 June, Microsoft announced the Microsoft Surface Tablet. This is a new product, and different from the 'old' Microsoft Surface (Multi)Touch Table product referred in this trends overview.

Duet Enterprise BDC Publisher requires 'Z' prefix

Beware that when using the BDC Publisher tool to generate the SAP Gateway service (in SPROXY) and associated SharePoint BDC Model, to enter a prefix that starts with a 'Z'. Otherwise the BDC Publisher will immediately abort, with error message 'No license to edit object R3TR FUGR...'. I did not immediately link this error message to the apparent SAP obligation of starting 'Z'. It cost me some elapse time (a.o. checking our systems license, my own profile, ...) before I rather coincidently detected that starting the BDC Publisher prefix with 'Z' fixes the issue:

Apparent this Z-obligation is common in SAP development; so common that consulted SAP experts didn’t even think of this as cause. The displayed error message also doesn't immediate puts you on the right direction.
I notified the Duet Enterprise product development team of this issue, including suggestion to prevent it by auto-fill in the 'Z' in the BDC Publisher screen.

BusinessDataItemBuilder not usable on publishing site

SharePoint BCS out-of-the-box delivers a number of useful webparts to render external data. The BusinessData webparts consistently apply the architecture pattern of data provider and data consumer. A provider-consumer example is that of the BusinessDataItemDetails webpart to render an external data item, connected to a BusinessDataItemBuilder webpart that provides the external data item. The latter applies the 'ID' querystring parameter in the page URL to determine which entity to retrieve from the external administration. A standard example where the 2 webparts are used in this combination are the BCS profile pages.

It appears a sensible approach to use the same webparts combination for rendering an external entity item on a publishing page. However, when opening the page with the 'ID' querystring parameter referencing a BCSidentity:

http://tnvduetenterprise/subsite/Pages/TestSAP.aspx?ID=0aab

SharePoint throws the following error:

PortalSiteMapProvider was unable to fetch current node, request URL: /subsite/Pages/TestSAP.aspx, message: Value cannot be null. Parameter name: key, stack trace: at System.Web.SiteMapNode..ctor(SiteMapProvider provider, String key, String url, String title, String description, IList roles, NameValueCollection attributes, NameValueCollection explicitResourceKeys, String implicitResourceKey) at System.Web.SiteMapNode..ctor(SiteMapProvider provider, String key, String url, String title, String description) at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapNode..ctor(PortalSiteMapProvider provider, PortalWebSiteMapNode webNode, String key, NodeTypes type, String url, String title, String description) at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapProvider.get_CurrentNode()

The cause is that the masterpage of publishing site includes PortalSiteMapProvider and PortalSiteMapDataSource. And the latter also reacts on the presence of the 'ID' querystring parameter, but expects a page identification as value instead of the BCS identity.

Justification of the Duet Enterprise XML-Funneling pattern

XML-Funneling pattern as answer for dealing with complex SAP data structures

In the blogpost ‘Architecture Pattern XML-Funneling of complex SAP structures for Duet Enterprise interoperability’, I introduced the XML-Funneling pattern as pragmatic approach to handle the Duet Enterprise limitation regarding complex SAP data structures. This Duet Enterprise limitation has 2 origins:

• The SAP NetWeaver Gateway generator tools are not capable of preservering hierarchical structure; their simple execution model is to flatten the SAP structure in the Gateway DataModel.
• The SharePoint BCS UI-controls are not capable of rendering hierarchical structure; their default rendering model is list/row based.

An often-heard misunderstanding is that Duet Enterprise itself, via the foundational layers SAP NetWeaver Gateway and SharePoint BCS, cannot be used to exchange complex data structures between SAP and SharePoint. This is not true. Gateway can expose complex data structures, but currently then requires you to model the interface and data signatures in ESR Builder, and next manually develop the Gateway Mappers. Although in FP1 less cumbersone as in Duet Enterprise 1.0, it is by far not the preferred approach to expose SAP building blocks as Gateway services. SharePoint BCS as integration layer can consume complex external data structures, but the rendering via standard controls is limited to the root level. In case the child levels also must be visualized, you need to develop a custom view that interoperates with the BCS API. This is doable, although a painpoint is that the SharePoint BCS API is weakly typed, and therefore programming against the API is laborsome and error prone.

I invented the XML-Funneling approach to simplify the development process at the SAP side. By mapping the SAP hierarchical structure to an XML string, the Gateway generator tools can be applied: a string is namely a ‘simple’ type.
On the SharePoint side the XML-Funneling approach does not really make developer life easier. A marked disadvantage is that it requires custom development to deserialize the received XML-structure into a strongtyped data object, before you can next use and visualize the received complex SAP data structure in a SharePoint view.

Trade-off decision whether or not to apply XML-Funneling

A trade-off decision is then whether in the total SAP + SharePoint system architecture and development effort, this front-end side / .NET disadvantage outweights the advantage on SAP side? Is it sensible to abandon the XML-Funneling approach in case of complex SAP data structure? My advice is to still apply the XML-Funneling pattern. Justification herefore are the following considerations.

Starting-points

• SharePoint is the target front-end platform.
• Favour out-of-the-box SharePoint above custom development.
• Application of SharePoint BCS; i) for the connectivity handling to external data administrations, ii) for the out-of-the-box UI controls to render external data, iii) and for integration within SharePoint Enterprise Search and FAST.
• Complex data structure is common within SAP business packages

Are there sensible alternatives for XML-Funneling approach?

WCF + DataContracts as alternative for Duet Enterprise to consume SAP data
First, Duet Enterprise is itself a WCF variant: the SAP data is consumed in SharePoint BCS as WCF service endpoint.
Second, on SharePoint side BCS is the limitating factor wrt complex structures, not Duet Enterprise. What if you bypass BCS and directly consume the Gateway Services as WCF endpoints? The result is that you not only loose the added interoperability value of Duet Enterprise: Single Sign-On, Authorization, Logging and Monitoring. But in addition also all of the native values of SharePoint BCS: standard UI-controls to visualize data (not all exposed SAP data is of hierarchical / complex structure), exposal to offline usage, use in SharePoint Search or FAST context.
Conclusion: this alternative is not sensible; it does not comply with the starting-points.

Expose complex SAP data structures via Duet Enterprise into SharePoint BCS
SharePoint BCS is a generic integration layer, usable for arbitrary external administrations: SAP, Siebel, Oracle, SQL-databases, ... This transparant interoperability capability is supported through a script-like, weakly typed BCS programming model. Whenever you can operate with BCS imported external data through any of the available standard UI controls, this doesn’t impose a problem. The controls abstract the weakly typed BCS programming model. But the standard controls are not able to visualize complex external data structures. For them also in this alternative you are required to custom build a view. Due the weakly typed programming model of the BCS API this is laborsome and error-prone.
Conclusion: this alternative is on SharePoint side an hardly better, programmer-friendly approach for consuming complex BCS consumed data; merely different. At SAP side, the consequence is that you cannot use the Gateway Generator tools/pipeline.

Expose complex SAP data structures as multiple associated simple structures
SharePoint BCS supports the concept of associations between two external content types. As parent-child is a relation, you can use this concept to expose complex SAP data structures through Duet Enterprise into SharePoint BCS.
BCS associations certainly has value, e.g. for master-child data model and UI-formats. But it also suffers from limitations that make it less applicable in other scenarios. For one: the net effect is that to retrieve a single complex SAP structure, multiple invocations from SharePoint to SAP must be done to retrieve the entire structure. And the responsibility for retrieving the entire structure is placed on the front-end side; instead of the data provider side. Another, the standard SharePoint UI controls all operate on a single BCS entity. If in the User Interface Design the ‘child’ data is directly mingled with(in) the ‘parent’ data, the standard controls cannot be used.
A Duet Enterprise design time issue is that the SAP Duet Enterprise generators are focussed on a single Gateway datamodel, and have no support for defining associations between 2 datamodels. Usage of BCS associations requires you to afterwards modify the BDC Model that is generated by BDC Publisher to add the associations. Either via SharePoint Designer, but in my experience you will more often than not need to resort to manual modification of the BDC Model. Which is time consuming, and error prone.
Conclusion: associations suffers from limitations and software architecture + performance drawbacks.

Final thought

Given that you want to benefit from the interoperability values of Duet Enterprise, and that within the exposed SAP business package there are entities with complex structure; XML-Funneling is a feasible approach. Consistent application of the pattern results in recognizable code setups for multiple Duet Enterprise scenarios, on SAP and SharePoint side. And in our experience, it is also possible to develop reusable building blocks for SAP and SharePoint to ease in the application of the pattern.