OData Channel (ODC) development now preferred for new Duet Enterprise scenarios

As of Duet Enterprise FP1 SP3, the Duet Enterprise product team now clearly positions OData Channel as the recommended approach for new developments of custom Duet Enterprise scenarios:

Feature Pack 1 for Duet Enterprise 1.0 Development Introduction

Generic Channel remains supported, also within Duet Enterprise 2.0; but will not be enhanced with new features and capabilities. Message: OData is the way to go. Main reason is that OData has much more potential for consumption outside the SAP Business Suite boundaries, and therefore the Duet Enterprise team is focusing its resources on supporting that more prospective channel.

Consequence for Duet Enterprise custom development: learn new approach + tools

Although one thus still has the option to utilize the Generic Channel development approach, the signal send by Duet Enterprise team is clear. OData development has the momentum and the future. Consequence for custom Duet Enterprise development is that on SAP side the programmer must transition to develop Gateway OData Services instead of GSDO types. The concepts and steps of the 2 are somewhat comparable, but the tools and execution + coding are different. For GSDO the ABAP programmer uses SE80 and /IWFND/GWO_GEN; OData development is performed mainly within the new Service Builder (SEGW). Within both approaches you have the options to generate the data provider class through mapping to RFC and BOR, or to manual handcraft it. Once the data provider class is ready (OData DPC or Generic Channel GSDO), you expose it as SharePoint 2010 callable entity. For OData DPC, the BDC Browser tool is extended with the capability to convert OData Services into Gateway SOAP web service. At runtime, Gateway applies for this the Duet Enterprise FP1 SP3 OData-SOAP Bridge. The existence of the OData-SOAP Bridge is transparent for the SAP Duet Enterprise developer. One simple utilizes the BDC Browser as before for GSDO type, to now expose the OData Service for consumption within SharePoint 2010.

Noticeable is that for SharePoint 2010 development side, nothing changes. You still import the BDC model generated via BDC Browser containing one or more External Content Types that refer to Gateway SOAP webservice(s). And next utilize the out-of-the-box Business Data UI-controls, and/or the BCS API to develop a custom UI.

See also:

Duet Enterprise 2.0 General Available: improvements on service development and deployment options

After successful completion of Duet Enterprise 2.0 Ramp-Up and with SharePoint 2013 reaching RTM state, SAP and Microsoft have made the new version of Duet Enterprise General Available. Customers can download the software from SAP Market Place (Installations and Upgrades \ Duet Enterprise \ Duet Enterprise 2.0). Duet Enterprise 2.0 documentation can also be found on SAP Market Place, and also on MSDN.

Most significant of the new Duet Enterprise version is that it connects with the latest capabilities of the underlying platforms, respectively SAP Gateway 2.0 and Microsoft SharePoint 2013. In both these platforms a move is made to support cloud and lightweight usage scenarios - typically but not limited to mobility use cases.

Duet Enterprise 2.0 in overview

Key topics:

• Mobility
• Duet Enterprise now fully supports OData protocol; which is THE protocol for mobility because is it light-weight and self-describing, and supported by all mobile platforms
• Cloud
• Connect to SharePoint Online
• Only support for Odata channel; not for SOAP / Generic channel
• Requires a 'light' on premisse SharePoint installation, as intermediate layer ('broker') from internal SAP Gateway to SharePoint online. Includes SSO Generator, via reverse proxy connection with SharePoint Online
• Remark: Duet Enterprise cloud stuff is not yet GA; this will follow in a few month

Facilitate custom Duet Enterprise development

• Design Time improved
• 2 design time approaches supported
• Inside-Out: via the SAP NetWeaver Gateway generators
• Outside-In: start from SharePoint UI --> visually sketch / specify a screen in Visual Studio; export the .EDMX file and import into SAP Gateway (Service Builder) for constructing the Gateway data models
• Both approaches are restricted to flat SAP data entities
• Complex data structures still requires hand-crafting on SAP Gateway side
• Via OData service providers; the handcrafting work is far simpler as with Duet Enterprise 1.0 (FP1).
• Improve the development story:
• Reference architecture guidance
• Setting up local development landscape --> facilitate individual SharePoint developers to quickly start with Duet Enterprise development

Consumption of OData services

• Consequence: drop SAML, as OData does not support SAML. Use oAuth instead
• Generic Channel will still be supported; OData is extra channel possibility

Aligned with SharePoint 2013

• Duet Enterprise 2.0 REQUIRES SharePoint 2013
  • SharePoint 2013 BCS supports Odata consumption; earlier version not
  • SharePoint Online.

Duet Enterprise 2.0 deliberaty remains a SAP / Microsoft interoperability + development framework

In the philosophy of SAP and Microsoft, also Duet Enterprise 2.0 does not include end-user functionality, e.g. order-to-cash. Duet Enterprise remains a customization + development framework for SAP / Microsoft interoperability, that enables end-organizations and partners to compose concrete scenarios and build upon. Primary reason for the Duet Enterprise product team to not include end-user functionality is that partners and end-organizations know best what diverse scenarios are requested, and are therefore better suited to construct the optimal solution. Also, even SAP plus Microsoft simple do not have the development resources to realize and support a total set of end-user capabilities for which there is market demand.

Growing Duet Enterprise eco-system

It is left to the market / Duet Enterprise eco-system to deliver standard / common-of-the-shelf end-user functionalities. Well-known example is the Cordis Suite; for self-service HCM and purchase requisition solutions. More examples can be found on Duet Enterprise Unite Partner Connection. And more partners are known to be developing on additional scenarios.

With Duet Enterprise 2.0 General Availability, it is expected that more companies will find it benefial to develop end-user solutions. This is due the combination of Duet Enterprise 2.0 development improvements, out-of-the-box plus customization capabilities of SharePoint 2013 and Office 365, and deployment extended to cloud and mobile platforms resulting in a larger market. Partners will develop solutions for horizontal and vertical markets; for local to international markets, and from small/partial solutions to complete end-2-end scenarios.

Locale bug in ddwrt:FormatDate not resolved for external data

XsltListViewWebPart is a powerful out-of-the-box SharePoint 2010 capability to render data, using Xslt to format the data display as you like. One of the standard Xslt functions, ddwrt:FormatDate, exhibits an annoying bug when used within XsltListViewWebPart. In case the locale of SharePoint server is other than US, then day and month are switched if day is < 13. The rootcause is – according to Microsoft - that ddwrt:FormatDate expects the datetime value to be in UTC format, but in reality XsltListViewWebPart preformats datetime values to the SharePoint server Locale before the Xslt-transformation takes place.
The SharePoint 2010 February CU 2012 contains an hotfix for this bug, and introduces the ListViewWebPart property ‘EnableOrginalValue’.

<property name="EnableOriginalValue" type="bool">True</property>

The effect of this property setting is that XsltListViewWebPart will include multiple formats per datetime value in the preformatted data. By using the ISO8601 datetime variant, the display bug can be resolved:

<xsl:value-of select="ddwrt:FormatDate(@Name,'.ISO8601')]),1033, 3)" />

However, this does not hold in case of external data. XsltListViewWebPart ignores the ‘EnableOriginalValue’ flag when rendering external data; and does not preformat the 'ISO8601' variant. Inspecting the sourcecode of super class DataFormWebPart proofs this to be true:

Conclusion is that XsltListViewWebPart by design does not support the hotfix for external data. Reasoning is that SharePoint BCS expects datetime values received from external systems to be in UTC format. And if not in UTC (DateTime.Kind unequal ‘UTC’), SharePoint BCS converts the received DateTime value into UTC:

"External systems commonly standardize on Coordinated Universal Time (UTC) as the time zone in which to store data. While this works well for data storage, end users prefer to work with times shown in their local time zone. External lists and the External Data Web Parts tackle this problem by converting time zones before presenting DateTime fields to end users, as well as before submitting DateTime values to the external system.
After retrieving a DateTime field from an external system, the BDC runtime examines the DateTime.Kind property to determine its time zone. If it is UTC or Unspecified, no action is taken. If it is marked as Local, BDC converts it to UTC by applying an offset based on the SharePoint front-end web server’s time zone, as specified by its Windows settings.
Now that the time zone has been standardized to UTC, the external list (aka XsltListViewWebPart) examines the user profile settings for the current user; if a time zone has been specified, the time is converted from UTC to the user's time zone. If a time zone has not been specified (which is the default state), the time zone of the site on which the external list is hosted is used to convert the external data. The converted times are then presented to the user in the UI in a format (for example dd/mm/yyyy hh:mm:ss) based on the site locale."

Source: Professional Business Connectivity Services – Scot Hillier

Thus external data received within XsltListViewWebPart will only contain UTC datetime values; therefore the issue should not occur; and the hotfix is on purpose disabled for external datasource.

But this is a design fault. We have a situation in which we retrieve SAP data [using Duet Enterprise + SharePoint BCS] in UTC format; and then still encounter the issue in case day < 13. Not surpisingly, UCT merely applies an offset in hours to the locale datetime; day and month remain in the same range; with at maximum 1 day earlier or later. Therefore the ddwrt:FormatDate also manifest itself for UTC datetime in case day < 13. As the XsltListViewWebPart hotfix on purpose does not fix that, we need an own Xslt FormatDate function to properly render the UTC datetime received from external system.

Notification + data flow in Duet Enterprise Workflow mobile App

The Duet Enterprise Workflow capability can be used to expose SAP workflow tasks to SharePoint context. Once in SharePoint, it can be further built upon. You can extend the associated SharePoint workflow with additional workflow steps and business processing. Another option is to use the SharePoint Duet Enterprise Workflow site as integration layer to expose and handle the SAP tasks in a mobile application.

System architecture

In such system architecture, following systems/layers are present:

1. SAP backend / workflow engine: the host environment of the SAP workflows
2. SAP Gateway: general purpose SAP integration layer; here used to publish SAP workflows from backend to workflow notification subscribers
3. SharePoint Duet Enterprise Add-On: functions here as workflow notification subscriber; host of Duet Enterprise workflow instances, and task repository for the mobile application
4. Mobile view: retrieve plus display tasks, and propagate task approval decision back to the SharePoint task repository

Note that in this system architecture, the (mobile) view and SAP landscape operate disconnected; SharePoint is the glue between the backend and frontend parts.

Notification and data flow for exposing SAP task to non-SAP view

Sequence steps:

1. The SAP ERP workflow execution arives on a user decision task, that is configured for notification to Gateway (tx SPRO, Gateway workflow customization);
2. SAP Document Publisher composes a XML Payload document with task data: workitem identification plus business data. The addition of the workflow specific business data is optional in Gateway Workflow notification. If applicable, you can achieve this by extanding the workflow customization with the plug-in of a custom OutboundHandler;
3. SAP Document Publisher calls the SharePoint OBAWorkflowServer.asmx::CreateTask operation, with inputparameters the task(type) identifier, workitem identifier and XML PayLoad document;
4. SharePoint OBAWorkflowServer.asmx::CreateTask method:
   1. Queries the TaskLocator List in the Duet Enterprise Workflow rootsite, and determines on basis of tasktype and user language the Duet Enterprise Workflow subsite to which the received SAP task must be routed;
   2. Instantiates a new SharePoint workflow instance in the identified Duet Enterprise Workflow subsite, and associates this instance with the SAP workitem identifier;
   3. And uploads the received XML PayLoad document into the Workflow Business Data DocLibrary in the Workflow subsite;
5. The SharePoint Duet Enterprise Workflow contains by default only 1 step that represents the decision-making for the SAP task. It is possible to extend the SharePoint Workflow with additional steps and functionality. This will operate isolated from the SAP Workflow handling. The default approval task in the SharePoint Duet Enterprise workflow is the only connectionpoint between the SharePoint workflow and the SAP decision workflow task. Upon the Duet Enterprise Workflow arriving at the approval task, it appears in the SharePoint tasklist.
6. User opens the tasklist in the mobile application;
7. The tasklist App call SharePoint List.svc service (DataAccess Layer) to retrieve the open tasks for the logged-on user
8. SharePoint List.svc queries the SharePoint workflow tasklist, to determine and retrieve the open task items for logged-on user
9. SharePoint List.svc returns the task items in Odata format
10. The Duet Enterprise Workflow Approval Task and the XML PayLoad document are in the SharePoint Workflow administration logical associated to each other. If additional business extended properties are notified for the SAP workitem via OutboundHandler, the associated Business Data document must be opened from the SharePoint document library and its contents parsed to retrieve the additional properties.

Control and data flow for handling task decision from non-SAP mobile App to SAP workflow context

Sequence steps:

1. User makes a decision in the task management App;
2. Task management App calls SharePoint Workflow.asmx webservice to propagate the task approval decision to SharePoint task item;
3. SharePoint workflows.asmx updates the task item with approval decision;
4. The update on SharePoint taskitem triggers the TaskItemEventReceiver;
5. Triggered TaskItemEventReceiver::ItemUpdating method checks whether the SharePoint task item is completed; that is a approval decision is made. If completed, TaskItemEventReceiver calls [through SharePoint Business Connectivity Services] the Update method of External ContentType  Duet Enterprise Workflow Task;
6. SharePoint BCS calls for Duet Enterprise Workflow Task Update the wfUpdate method of SAP workflow consumer webservice; with input parameters workitem identifier, decision value and comments;
7. SAP workflow consumer webservice propagates task approval decision to the associated SAP workflow instance, and executes apply decision on the SAP workitem;
8. SAP workflow completes this workitem step, and proceeds to the next step in SAP workflow. If it was the last step in workflow, the SAP workflow is now completed;
9. TaskItemEventReceiver updates the status of SharePoint taskitem to completed. In default (generated) Duet Enterprise Workflow is this the only step/task, and the SharePoint workflow is completed. The SharePoint taskitem is removed from the users open tasks list.

Selective enable ScriptManager presence in SharePoint Publishing Site

It is a well-known symptom of standard SharePoint Publishing that it renders bloathed HTML pages. The cause is that SharePoint publishing pages serve 2 modes: the end-user view, and the editorial view. To serve the latter, considerable extra html and SharePoint javascript files are included that are of no use for the end-user [view].

Approach to mitigate this effect is by separating the editorial from the end-user view. Examples of this approach are Macaw Dual Layout and Mavention Flex Layout. At high level the working is to have a distinct masterpage for end-user view, and another for the content editor. Only the latter must have the means to enable SharePoint WCM functionality; the former can be designed ‘clean’: no Form runat=”server”, ScriptManager, and include of SharePoint javascript files (init.js, core.js, …). Besides loading faster, additional advantage is that pure HTML sites are also more SEO-friendly.

However, there typically remain situations in a SharePoint publishing site in which you do need the presence of ScriptManager and the standard SharePoint client-side scripts. Multiple of the standard SharePoint controls are dependent on ScriptManager: CalendarView, TaskList, External List (XstlListView), Business Data / BCS webparts, to name some. Also custom developed controls might depend on presence of ScriptManager, e.g. rotating banner control using jQuery libraries for client-side user interface behaviour.

You have basically 2 options to enable ScriptManager et al for Publishing Pages on with controls are placed that need the ScriptManager:

1. Duplicate the ‘clean’ end-user masterpage; and augment it with the required entities.
2. Create an additional Publishing PageLayout; and augment that the required entities.

Option 1 suffers from the following disadvantages:

• As consequence that a SharePoint SPWeb can only have 1 masterpage, ‘clean’ content publishing pages and WebControls pages cannot be within the same SharePoint SPWeb. In fact, the type of pages now influences the site structure. This is visible for the end-user. And for the content manager it implies that (s)he is forced to locate new page a priori within the correct SPWeb given the intended type of page: content or control. Also, as standard SharePoint navigation is based on the site structure, it will require custom development to repair the logical site navigation.
• Double maintenance of the site look&feel in the 2 public masterpages.

As in the 2nd option both page types use the same public masterpage, all pages can be within the same SharePoint SPWeb. So with this approach it is non-visible for the end-user, and also the editing experience is far more transparent for the content editor. Not entirely, as (s)he must apply the correct PageLayout given the type of page. But the applied PageLayout is something that in SharePoint publishing can be modified / corrected after initial creation.

From SharePoint technical perspective, the 2nd option is more difficult / challenging to realize as the 1st (this is simple: just copy the public masterpage and add ScriptManager et al). Issues here are following:

• The extended PageLayout must function both in combination with the clean masterpage as with the content-editor enabled masterpage. But the latter also as Form runat=”server” and ScriptManager included; resulting in 2 occurences of them. And this is not allowed in ASP.NET plus SharePoint.
• SharePoint scriptmodel imposes hidden dependencies on the order of including + evaluating/loading SharePoint javascript files (through SOD framework).

You can deal with these issues via following approach and architecture:

• Include Form runat=”server” in the extended PageLayout; but default not visible.
• A server control included in the extended PageLayout to dynamically insert the ScriptManager plus ScriptLink for core.js; but only if no ScriptManager present yet (included from the content-editor masterpage). When runtime determined that the ScriptManager must be inserted, also the invisible server-side Form is made visible; and all siblings of the Form control are moved within the server-side HtmlForm control.
• A server control included in the public masterpage to dynamically insert WpzEmitControl (explanation why needed) if needed; the condition for this is whether the InsertScriptManager control is include in the page context.

Result

Regular content pages still render ‘clean’ html in public view, without the unneeded SharePoint html and javascript. Control pages render in public view including the required ScriptManager and Form runat=”server”; and in editing view also with only occurrence of ScriptManager and server-side HtmlForm.

Deploy custom Duet Enterprise scenario from Dev into SAP landscape

In the case that an organization develops custom business scenarios on top of Duet Enterprise foundation, an essential step is to implement it through to the organizations SAP production environment. Duet Enterprise relies herefore on the standard CTS+ transport mechanism.

Design time - prepare the custom business scenario

It is best practice to include all of the Duet Enterprise services for one scenario within an own isolated and recognizable namespace in the Gateway system. Each scenario will then be recognizable, and not per accident mingled with deliverables of another Duet Enterprise scenario. To achieve this, you create per business scenario an own Software Component in SLD, and use that to create in ESR an own Software Component Version and namespace.

Steps:

1. Open SLD, and create a new Software Component
2. Open ESR, and create a new Software Component Version via import of the newly created Software Component
3. Create a namespace in the newly created Software Component Version
4. Copy the Basic DataTypes from SAP IW TNG to the new Software Component Version and namespace
5. Copy the Fault Message from SAP IW TNG to the new Software Component Version and namespace
6. Activate the resulting changelist(s) in ESR
7. Logon to Gateway system, transaction SE80; and create a Package
8. Logon to Gateway system, transaction SProxy; and create + activate proxies for the Basic DataTypes, ExchangeFaultData and ExchangeLogData in the newly created Package.

With this, the preparation phase is completed. You now have an own business scenario container for all services that will be developed for the scenario.

That is, within the Dev environment. The same container must be created on the counterpart Gateway system into Test, QA and Production environments to implement the custom scenario in these environments. To facilitate this, all of the design time preparation steps performed on the Gateway system are recorded in a CTS+ Transport Workbench request: Create Package (step 7), and Create + Activate Proxies (step 8).

Note that it is not needed to also implement or repeat the SLD and ESR steps into the Test, QA and Production landscapes. For Duet Enterprise design time, the SLD and ESR only have a role in the design time phase; in the development environment. Once the service proxy objects for ESR content are generated (basic / standard ones, and the custom developed), there is no dependency on ESR content. The transport therefore only consists of the service definition(s) and the proxy objects that are part of them.

Design time - create the service definitions for the custom business scenario

Per service this consists of the 2 familiar actions: create and map a Gateway DataModel, and next generate the service definition + BDC Model for this Gateway DataModel (GSDO Type).

Steps:

1. Logon to Gateway system, transaction SE80; and create a new Gateway DataModel; make sure not to select OData channel (Duet Enterprise 1.0 and SharePoint 2010 do not support OData consumption; this will change with Duet Enterprise 2.0 and SharePoint 2013). The system enforces you to include the creation of the Gateway DataModel in a Workbench request. The advice is to not include it in the same Workbench request from the preparation step, but instead use an own Workbench request for the design time deliverables. As result you have the implementation of the Package including the generic basic DataTypes + FaultMessage separated from the implementation of the specific scenario entities. It is typical in a custom realization project that you incremently build new versions of an own service definition, and if a previous version already transported you must be able to replace that while preserving the Package.
2. Map 1 of more operations in the Gateway DataModel; e.g. Query and Read
3. Generate the Gateway DataModel; the system enforces you to include the generation and creation of SAP entities within a Customizing request.
4. Open transaction BDC Browser; if this is the first service definition for the custom business scenario, create a new Business Scenario. Otherwise you may opt to include the 2^nd, 3^rd and so on service definition all within the same Business scenario. I advice this as best practice, you then end up with a single Duet Enterprise Business Scenario that contains the entire set of Service Definitions developed for it. In case you create a new Business Scenario, it is again enforced by the system to include it within a customizing request. You can select the same as earlier created for the Gateway Model generation(s).
5. Select the Business Scenario, and Create BDC Model for GSDO Type; select the Gateway Model earlier created. Also here the system enforces to include within a customizing request.
6. [Optional] Open transaction SProxy, select the generated service interface and test its correct working for the diverse methods in the interface (e.g. Find, Read, Create, )
7. [Optional] Open SOAMANAGER on Gateway system, browse to Business Scenario Configuration. Select your Business Scenario, and export + save into a local XML file.

Implementaton time - transport the custom business scenario

When the custom realization has progressed or is even completed in the Gateway system of the SAP Development environment, next is to propagate it within the SAP landscape.

Steps:

1. Logon to Gateway system of SAP Dev environment, transaction SE03 or SE10; and release all the Workbench and Customizing requests that are related to the custom Duet Enterprise Business Scenario
2. Logon to Gateway system of the transport target environment (SAP Test or QA environment), and open Transport Management System (TMS). Import the released Workbench and Customizing requests from the Development environment.
3. Activate each of the requests. Make sure to do this in the correct logical order: first package, next the business scenario including service definitions.
4. Logon to Gateway system of the transport target environment, transaction BDC Browser. Validate that the Business Scenario(s) is/are present. Also validate that for each of the imported service definitions, there is no end point yet.
5. Release and activate all of the imported service definitions in SOAMANAGER. Browse to Business Scenario Configuration, and import the Business_Scenario XML file that is exported in the Development environment. If the Business Scenario already exists here from a previous import, first delete it. Or otherwise you can also explicitly associate new service definitions within a/the Business Scenario, and assign a profile. After activation of the service definitions, validate in transaction BDC Browser that for each of the service definitions now an end point is available.
6. [Optional] Open transaction SProxy, select the imported service interface and test its correct working for the diverse methods in the interface (e.g. Find, Read, Create, )
7. Transaction BDC Browser, select the imported Business Scenario and click on Export Business Scenario. Hand over the exported SharePoint BDC Model(s) to import within the SharePoint BCS Service Application of the implementation environment.
8. [Optional] In SharePoint webapplication, create an ExternalList for one of the imported External Content Types. Validate that you can retrieve SAP data via Duet Enterprise and SAP Gateway using one of the transported / imported Gateway Service Definitions.

Duet Enterprise troubleshooting pointers

Generic

• Duet Enterprise 1.0 Troubleshooting Guide
• Troubleshooting example
• Holger Bruchelt's Duet Enterprise blogpage

SAP side

• BDC Browser - Troubleshooting pointers
• BDC Browser Configuration - Software Component SAP IW TNG missing
• Workflow tasks not delivered?
  • txn SLG1 on backend
  • txn SRT_UTIL on Gateway
  • txn SBGRFCMON: retry queue contains wf-items failed to deliver to SharePoint
    • use 'Unit Analysis \ Run Unit with Trace' to execute again/debug
• ABAP runtime errors / dumps: txn ST22
• ABAP system log: txn SM21
• Duet Enterprise 1 - Troubleshooting Report Delivery
• GW 2.0 SP5: Which Tools Can Be Used for Troubleshooting
• For Payload tracing:
  • txn SRT_UTIL
  • txn SMICM (ICF trace tools)
• ABAP Function Modules and services: External Debugging, Set External Breakpoints for ABAP Debugging
• Troubleshooting Gateway Development
• SAP system HTTP Logging
• Using Starter Services without EhP4

SharePoint side

• ULS Logviewer
• SharePoint Developer Dashboard
• Troubleshoot Business Connectivity Services using Performance Monitor
• SCOM (System Center Operations Manager
• Overview of monitoring and troubleshooting