Recover from SharePoint 2013 Search and CryptoGraphic mismatch - 'Key does not exist'

SharePoint Search Application is applied to crawl external data via Duet Enterprise 2.0. In the Duet Enterprise authentication flow from SharePoint to SAP, the SSA process via Business Connectivity Services invokes Secure Token Service to runtime create an X.509 authentication user certificate for the SharePoint account under which the search crawling is executed. (See 'How authentication works in Duet Enterprise 2.0').

This worked fine, until I upgraded the SharePoint 2013 landscape to the latest released cumulative updates: Sept + Oct 2014. Part of the upgrade steps is to temporarily disable the SharePoint Search services, and restart them after the CU installations. However, afterwards it appeared that the runtime Duet Enterprise SSO behavior was broken. The crawl log on the external content source reported structurally the error Exception in invoking the ODataExtensionProvider of type 'OBA.Server.Canary.ObaOdataServerExtensionProvider'. And the ULS contains on constant basis the error 'The search connector framework caught an exception from BDC: Exception in invoking the ODataExtensionProvider of type 'OBA.Server.Canary.ObaOdataServerExtensionProvider'. (Key does not exist. )'

But this is only when the BCS OData service is invoked from Search Crawling context. Using the same SharePoint user credentials to interactively retrieve SAP data in a SharePoint site still works, and successful retrieves the external SAP data applying Duet Enterprise 2.0 Single Sign-On.

Recovery fix:

Restart the SharePoint Search service (OSearch15), to force a reset of the runtime memory in that process and resync with CryptoGraphic on Windows OS level.

Connect SharePoint 2013 to SAP via Gateway for Microsoft

SAP initially developed Gateway for Microsoft [GWM] as ‘Duet Enterprise beyond SharePoint’. Allow Gateway OData service consumption in other Microsoft front-end formats, like Microsoft Office clients (Outlook, Excel, Word, PowerPoint). In essence GWM can also be regarded as a trimmed-down Duet Enterprise 2.0 variant; provides some of the same basic integration capabilities, but not all. Question that arises: can you also utilize GWM to connect SharePoint to SAP?

The answer is a firm Yes you can! Which is logical, as SharePoint itself is a .Net application. You can apply multiple approaches to consume the SAP data through a GWM generated reference proxy into SharePoint:

1. Connect to the Gateway service proxy from a SharePoint web part context [mind you, Microsoft urges us to step away from this SharePoint server-side model; but it is still possible and supported]
2. Build a custom BCS Connector to connect to the Gateway service proxy, and use that connector from SharePoint to render the received SAP data via standard External List, Business Data WebParts, or a custom build web part that invokes the BCS Api
3. Build a SharePoint WCF RESTful service to connect to the Gateway service proxy, and consume that service in a browser-based front-end; using a databinding library like knockout.js, Angular.js. This is an example of SharePoint 2013 App model, and can also be applied in Office 365 / SharePoint Online.

SharePoint version

SharePoint 2010 has been build “ages ago”, and has a hard dependency on .Net framework 3.5. In the current year, the .Net framework has progressed to .Net framework 4.5, and the latest version of SharePoint [2013] has catched up with that. And the same holds for Duet Enterprise 2.0 [which is actually for SharePoint 2013, if you have SharePoint 2010 you need the Duet Enterprise 1.0 version], and also GWM: both are .Net Framework 4.x dependent. As consequence, I have to relax my firm statement a bit: Yes, GWM can be used to connect SAP and SharePoint 2013; but not for the older SharePoint versions (2003/2007/2010).

Simple example for Proofing

To demonstrate, I made up a sample scenario to retrieve SAP CRM data via GWM into SharePoint. I used the demo Gateway services as data source (see article by Martin Bachmann for instruction how to get access to the SAP Gateway demo system), and our own SharePoint 2013 environment as front-end. Further I have Visual Studio 2013, and necessarily also the latest GWM sp3 (versions preceeding sp3 will not install in Visual Studio 2013; but they do install in Visual Studio 2010/2012).

I applied the guidance provided in the GWM Developer Guide, but instead of Windows Forms project, choose SharePoint Visual Web Part template.

Next, you first need to add references to .NET WCF and GWM libraries. The easiest way to achieve that is by utilizing the GWM Visual Studio Add-In. Click on ‘Add SAP Service Reference’:

fill in as Service Url the url to GWdemo service, and press ‘Go’:

In the Service Explorer you can explore and inspect the OData entities that the GWdemo service exposes. Select one, and click 'Ok'. Direct result is the inclusion of multiple GWM assemblies (note that they are still named with the older GWPAM naming) in the Visual Studio project references:

Next, open the visual webpart and put in some code to consume the GWM created proxy reference and display received data. As this is merely a short demo to proof the connectivity, I simple display the SAP CRM BusinessPartners through a plain GridView control:

Build the Visual Studio project, and deploy the SharePoint solution. Then browse to your SharePoint site, create a new page, and add the GWM-build web part. And voila, we have SAP data in SharePoint:

All in all, building this took me less than an hour. Of course it is only a simple retrieval example; and no effort spend whatsoever on achieving a good looking and behaving UI. Also for this simple example I relied on basic authentication. For a trustworthy enterprise context, this is not usable and either OAuth, X.509 or SAML must be applied. But still, the outcome is very promising with respect to the SAP/SharePoint interoperability capability of GWM.

Fixes needed to GWM generated code

Issue 1: Microsoft.Sharp not included in references

Solution: Add the missing assembly to the Visual Studio project reference

Issue 2: Configuration Reader tries to read from windows form based path

Solution: avoid the attempt to read from windows-forms based path

Issue 3: The used GWDemo system gives error

Solution: invoke another method…

Honestly: this is an example of why in every SAP interoperability project also domain knowledge is required --> contactpersoncollection is connected to a business partner key, you cannot invoke this without one.

Does GWM replace Duet Enterprise?

The answer to this is a clear No: Duet Enterprise as framework has more capabilities as GWM: SAP workflow integration, SAP BW reports publication, roles synchronization, user profile synchronization. Also Duet Enterprise has complete self-contained support within for Single Sign-On between SharePoint and SAP, while with GWM you need additional infra to achieve this (e.g. X.509 certificates infra in your landscape).

The better question is however: in what scenario’s would GWM be sufficient? Well, it might be sufficient in scenarios where all you need is the connectivity from SharePoint to SAP for data CRUDQ actions, and you already have Single Sign-On supporting infra in your IT landscape.

Programmatically set decision of Duet Enterprise task

In our award-winning VIEW (Virtual Integrated Enterprise Workplace) concept we include a generic inbox. In the generic inbox tasks are aggregated from multiple sources: SAP, SharePoint, Oracle, proprietary systems, and so on. The user sees in his/her inbox all outstanding tasks from the diverse task backends. And it is also supported to direct handle the tasks from the generic inbox.

To make the generic inbox real also for Duet Enterprise tasks, it is required to programmatically approve them via the inbox instead of via the standard Duet Enterprise task forms. A good start how to achieve this is article ‘how to create Duet SharePoint webparts’ of Ravi Sharma, in which he uses programmatic Duet Enterprise task approval as an example.

The approach Ravi takes in his setup is to directly update the SAP task by invoke via SharePoint BCS Application Programming Interface (API), explict self the wfUpdate method on the DuetEnterprise WorkflowTask External ContentType. And if the SAP task update is successful, also afterwards alter the SharePoint task that functions as reference or placeholder in SharePoint context to that SAP task.

Although this approach works, it has multiple architectural disadvantages:

• It bypasses the standard Duet Enterprise workflow capability to update the SAP task via the SharePoint task.
• It effectively duplicates some of the code and working of the internals of Duet Enterprise workflow update. As architect, I very much dislike any code duplication.
• Per conceptual task update, 2 tasks must be explicit updated: first the SAP task, next the SharePoint task. This makes the task handling more complex.
• There is a clear difference in the approval of SharePoint tasks that are created via Duet Enterprise workflow capability, versus regular SharePoint tasks (from any SharePoint workflow, e.g. document approval)

I favor the approach to handle all SharePoint tasks, including the ones that are instantiated as result of Duet Enterprise workflow publication, the same. And be ignorant of what happens in the SharePoint context and possible beyond as result of the task decision update. To achieve this, my SharePoint task update code is a variant on the ‘updateSharePointTask’ method of Ravi. And in this approach, I have no need for the ‘updateSAPTask’ method, as this responsibility is handled by the standard Duet Enterprise workflow capability.

There are 2 caveats when programmatically update Duet Enterprise tasks:

1. Generic for SharePoint tasks: extendedproperties field ‘ows_taskstatus’ must have been set for the task update to be accepted. However, you cannot set this direct self, but set it indirect by this code snippet:
2. Specific for Duet Enterprise tasks: In ItemUpdating eventreceiver, multiple checks are done to verify that it is indeed the intention to propagate to SAP. One of these checks is that a string-compare is done on both the set status text as status code. If this string-concattenation is not present in the Duet Enterprise task PossibleOutcomes (you specify this when you in SharePoint ‘Configure a new SAP workflow task type’), then the SharePoint task update succeeds, but the update is silently not propagated onto the connected SAP side. To make sure the programatically set status text is in sync with the configured PossibleOutcomes, I derive the status text based on that same configured PossibleOutcomes:

The complete code example to programmatically set the decision on SharePoint task, which via Duet Enterprise workflow capability then propagate the decision made to the connected SAP task:

Utilize Duet Enterprise 1.0 and 2.0 in parallel in enterprise architecture

Duet Enterprise supports migration scenarios in mixed SharePoint 2010 + 2013 landscape

Deployment of Duet Enterprise is a strategic addition to an organization’s existing infrastructure of SAP Business Suites and Microsoft SharePoint. One of the requirements from enterprise architecture perspective is that the investment in Duet Enterprise is future-proof. The utilization of Duet Enterprise for SAP-SharePoint interoperability must include a roadmap to follow-up on the innovations and developments in the SAP and SharePoint platforms. This is where Duet Enterprise 2.0 comes into place: the first version of Duet Enterprise supports SharePoint 2010, it’s successor supports SharePoint 2013. Duet Enterprise 2.0 is also backwards compatible, Duet Enterprise 1.0 services + BDC Models with Gateway Generic channel can be directly reused in SharePoint 2013 to interoperate via SAP NetWeaver Gateway with the SAP business suites.

In nowadays reality of many SharePoint using organizations, it is typical that for a time they have a mixed presence in their infrastructure of current SharePoint 2010 based applications, and new SharePoint 2013 developments plus migrations from SharePoint 2010. In such situations, enterprise architecture aims to utilize Duet Enterprise for SAP-SharePoint integration on both SharePoint versions, connecting to the same SAP business suites. SAP NetWeaver Gateway in its role of the central gateway to the SAP backend landscape, supports this enterprise architecture demand. The same single Gateway system can serve multiple consumers, including multiple SharePoint farms. The consequence is that investments made in Duet Enterprise 1.0 on SharePoint 2010 platform can be harvested when gradually migrating to SharePoint 2013 context with Duet Enterprise 2.0 deployed.

In our demo landscape we have this mixed landscape operational. On a SharePoint 2010 farm we have Duet Enterprise 1.0 based scenarios operational, and on a SharePoint 2013 farm the same Duet Enterprise 1.0 scenarios and also new Duet Enterprise 2.0 scenarios.

A prerequisite for this parallel SharePoint 2010 + Duet Enterprise 1.0 and SharePoint 2013 + Duet Enterprise 2.0 setup, is that the SharePoint farms must use the same SharePoint STS certificate. The reason for this is that in Duet Enterprise 1.0 scenarios, SAML is used for Single Sign-On handling. Both SharePoint 2010 as also SharePoint 2013 use the SharePoint STS certificate to sign the SAML assertion that is added to Duet Enterprise 1.0 requests originating through SharePoint Business Connectivity Service (BCS) application.

As our landscape is not a production system, I used a self-signed certificate for the SharePoint STS service applications in the 2010 plus 2013 farm. This saves us from the nuisance of having to renew the STS certificate each year. Something to be aware of is that in case of using a self-signed certificate on server A that is generated on another server B, it is also required to import on server A the internal root certificate of server B. This is required to enable server A to verify the full chain of certificates of the self-signed certificate generated on server B.

Duet Enterprise 2.0 - installation troubleshooting

Initial, the installation of the first version (1.0) of Duet Enterprise was cumbersome and labour-intensive, and therefore also error prone. With the Duet Enterprise 1.0 installation wizard at SAP side, the installation already was substantially improved. And this has progressed on to the installation of the new version, Duet Enterprise 2.0. However, as the landscape is inherent complex (minimal one, but typically more SAP backends, a SAP Gateway system, and SharePoint 2013 farm), you still may face issues.

In this blog I describe some of the issues that I encountered during Duet Enterprise 2.0 installations, and share my resolutions.

Issue 1: ‘Add-on IW_DUETE Release 100 can only be installed in client 000’

Import of the Duet Enterprise 2.0 SAP Add-On via transaction SAINT gives the following error:

Resolution: Use client 000 to import the add-on via transaction SAINT.

Issue 2: ‘The remote certificate is invalid according to the validation procedure’.

This error can occur in multiple SharePoint-SAP scenarios: runtime invocation of Duet Enterprise service from SharePoint 2013 side, import of a Duet Enterprise 2.0 BDC Model: Application definition import failed. The following error occurred: The remote certificate is invalid according to the validation procedure.

Resolution: Typical the above error message is caused by mismatch of the SSL certificate used on SAP Gateway side to encrypt the traffic, versus the SAP SSL certificate that has been imported in SharePoint 2013 ‘Manage trust’. To repair, import the SSL certificate again. You can either ask again the SAP BASIS administrator to export from SAP Gateway STRUST. But a more convenient, and in my experiences also one with better results (was already so for Duet Enterprise 1.0) is to open on SharePoint web frontend the SAP service url, and then from SSL certificate warning export the SAP SSL certificate to file; and next add this to ‘Manage trust’.

Issue 3: ‘The root certificate that was just selected is invalid’

Importing in SharePoint 2013 ‘Manage Trust’ the SSL certificate of SAP NetWeaver Gateway results in error:

Resolution: import the SSL certificate via Internet Explorer (11) instead of Firefox.

Issue 4: ‘No service found for namespace /IWWRK/, name DUET_WORKFLOW_CORE

Manifest itself upon configuration of Duet Enterprise 2.0 Workflow solution.

Resolution: Add and activate the Duet Enterprise 2.0 Gateway REST service in MAINT_SERVICE.

Issue 5: DuetConfig: ‘The operation name is missing or invalid’

For instance occurs with the DuetConfig command to configure workflow, command with multiple operation parameters: DuetConfig.exe -importbdc -FeatureName Workflow -LsiUrl "https://tnvsrm.tnv.corp/sap/opu/odata/IWWRK/DUET_WORKFLOW_CORE;mo;c=SHAREPOINT_DE/" -BdcServiceApplication "Business Data Connectivity Service" -UserSubLsiUrl "https://tnvsrm.tnv.corp/sap/opu/odata/IWBEP/SUBSCRIPTIONMANAGEMENT; mo;v=2;c=SHAREPOINT_DE/"

Resolution: careful check the command line that the start character for each of the operation parameter names is indeed the minus (‘-‘) character. In case you copied the command line from the deployment guide, this may be ‘invisible’ the wrong character. To be sure, in the command line explicit (replace and) type the ‘-‘ character:

Issue 6: ‘Lobsystem (External System) returned authentication error'

For instance, occurs upon trying to import a Duet Enterprise 2.0 BDC Model into SharePoint 2013 Business Connectivity Services application:

Resolution 1: Check that the Duet Enterprise 2.0 generated ‘Duet Root Certificate Authority’ is added to SSL Server Standard in STRUST on the SAP Gateway system.

Resolution 2: Check the user mapping of the SharePoint account used to invoke the SAP Gateway service / import the Duet Enterprise 2.0 BDC Model. In particular check the extid that it contains the correct pattern (account name, comma + space, and then the domain name in small capitals):

Issue 7: Import of Duet Enterprise 1.0 BDC Model fails due missing ‘WSDL’ application definition.

Resolution: Add an “Duet Enterprise 1.0 WSDL” application definition in SharePoint 2013 Secure Store.

Issue 8: Invocation of Duet Enterprise 1.0 service fails due missing SSO

Browsing a SharePoint 2013 External List results in error ‘An unsecured or incorrectly secured fault was received from the other party’.

Inspect the SRT_UTIL ErrorLog:

Resolution: Besides the Duet Enterprise 2.0 SSO approach based on X.509 certificate, also enable Duet Enterprise 1.0 SAML2 approach. Use the Duet Enterprise 1.0 Deployment Guide for information how to enable SAML2 in the Duet Enterprise 2.0 (Gateway 2.0 + SharePoint 2013) landscape.

Epilogue

Compared to the excellent troubleshooting guide for Duet Enterprise 1.0, the above list of issues plus resolutions is much smaller. The Duet Enterprise product team of SAP plus Microsoft has clearly improved their delivery on this aspect.

Others that have been deploying Duet Enterprise 2.0 may have encountered other issues as I did so far. Still a general classification of all issues seen so far is that they are caused by manual error, indirect caused by the sometimes unclear, fragmented and even incorrect Duet Enterprise deployment guides.

Winner of SAP Microsoft Unite Partner Connection Customer Impact and Value Award with VIEW solution

I am very proud that The Next View has won the 2014 edition of the SAP Microsoft Unite Partner Connection Award for Customer Impact and Value with our VIEW solution!!

VIEW stands for Virtual Integrated Enterprise Workplace. For us, the VIEW concept is not new, I for instance already defined and published on this blog parts of the Conceptual Solution Architecture back in 2009. But it is only up to now with the advent of the modern integration technologies SAP NetWeaver Gateway, Duet Enterprise, Gateway for Microsoft (GWM), plus the availability of standard functional products from our partner Cordis Solutions, that we are enabled to actually realize the VIEW concept in a cost-effective manner.

So, what does VIEW stand for? VIEW is a new operating concept in which the central concept is that of an employee-centric mindset. In VIEW, we strive to optimally enable organization’s employees to perform their daily work-related activities. In current reality, this work execution often means that one must operate in (and switch to) multiple applications and systems, monitor multiple tasklists in different environments (SAP, SharePoint, Oracle, Outlook, ...), remember login credentials of the diverse systems. With VIEW, we relieve the employees from all this ‘IT landscape’ hashle. Instead that the employees must explicit go to all the different applications, in VIEW we collect all the work execution in a central place: the VIEW landing page.

And this VIEW landing page has multiple appearances: desktop and mobile, to fit in with the nowadays reality that employees are [willing to be] always connected to the business systems, to at minimum monitor and act on urgent matters.

Although the VIEW concept does not mandates this, the typical platform for the desktop appearance is SharePoint; as this is in majority of the organizations the declared [by Information Management, Enterprise Architecture] business webplatform. The VIEW landing page is merely added as new employee business application within the already present SharePoint-based intranet.

Also for the mobile appearance, SharePoint can be the platform [certainly SharePoint 2013 has made some big steps on enabling us to provide a proper mobile appearance, a.o. taking into account the diversity in mobile devices]. But the mobile landing page can also be hosted outside SharePoint, e.g. via SAP Mobile Platform [SMP], an hybrid App [HTML5/CSS/PhoneGap], and other alternatives. Again, the VIEW concept does not put strict restrictions on this.

If you want to learn more about the VIEW solution, check out the SAP Microsoft Unite Partner Collection solution brief.

Corrupt SharePoint account breaks Duet Enterprise workflow publishing

One of our Duet Enterprise customers where I had a.o. configured the workflow capability, requested my support as workflow tasks where no longer published from SAP workflow backend into SharePoint. I performed cause analysis in the combined SAP + SharePoint landscape, starting with inspecting the logs – SAP backend, SAP Gateway and SharePoint systems. The SAP Gateway log contained a recurring error log “logical port not found for routing url”.

I checked in SIMGH on the Gateway system, and found a valid routing url that referred to ‘LOGICALPORTFORWORKFLOW’; and in SOAMANAGER an active logical port with this name. But when I tried to ping the /IWTNG/CO_TASKFLOW_WEB_SERVICE consumer proxy, http error 405 was returned. The actual cause of the malfunctioning Workflow publishing from Gateway into SharePoint was the SharePoint service account that Gateway uses to authenticate against the Duet Enterprise webservice '/_vti_bin/OBAWorkflowService.asmx' on the SharePoint server. It had somehow become corrupted, and lacked the authorization to access the WSDL document of the Duet Enterprise service. After recreating/resetting the new service, account workflow publishing is working fine again.

Note: the logged message ‘No logical port found’ can be misleading; in the ABAP code it is the generic catch of whatever problem is encountered upon trying to propagate SAP task notifications from SAP Gateway to SharePoint.

SAML2 Protocol requires SSL-enabled Duet Enterprise 1.0 endpoints

An organization that is deploying Duet Enterprise in the landscape, default has no SSL enabled in their internal network. They questioned to neither activate SSL on the Duet Enterprise SAP Add-On / SAP NetWeaver Gateway 2.0 system.

However, this request cannot be granted. Reason is the usage of SAML2 for Duet Enterprise 1.0 Single Sign-On between SharePoint 2010 and SAP NetWeaver Gateway system.

Source 1: Duet Enterprise Security Considerations

The user accounts that are used to access SharePoint Server 2010 and Microsoft Office 2010 suites clients cannot be used to access the information in SAP directly. The Duet Enterprise security architecture solves this issue by configuring the Microsoft Business Connectivity Services Windows Communications Foundation (WCF) connector that is included in SharePoint Server 2010. This WCF connector interacts with the Security Token Service in SharePoint Server 2010 and with SAP NetWeaver in the SAP system. The goal of this implementation is to map user identities in SharePoint Server 2010 to user accounts in the SAP system so that a user who logs on to the SharePoint Server 2010 Web site can have access to the external data that is stored in the SAP system without having to log on again in the SAP system.

Duet Enterprise 1.0 applies SAML2 with Symmetric Key for Endorsing Signature.

Source 2: STS Scenario with Symmetric Key for Endorsing Signature

With this scenario, the STS and the WS consumer negotiate a symmetric key. This is used for an endorsing signature for messages between the WS consumer and the WS provider. The WS consumer uses this endorsing signature to prove that it is in possession of the key that the STS signed.

In short (a.o. source 2 contains a complete outline of the SAML2 Protocol steps): in the SAML2 Protocol with Symmetric Key,

• the webservice consumer (in Duet Enterprise scenario: SharePoint BCS) authenticates the logged-on SharePoint user at the Identity Provider (in Duet Enterprise scenario: SharePoint STS),
• the Identity Provider grants a SAML security token for the SharePoint account, generates a short-lived key and signs this with the public key of the X.509 certificate of the webservice provider (in Duet Enterprise scenario: SAP NetWeaver Gateway),
• STS returns the SAML token + signed STS key as SAML assertion to SharePoint BCS as webservice consumer.
• SharePoint BCS adds the signed token as SAML assertion (Holder-of-Key, HoK assertion) to the header of the service request and sends the request to SAP NetWeaver Gateway as webservice provider.
• Gateway uses the private key of its own SSL certificate to decipher the token, and validate it as genuine coming from trusted asserting party.
• If so, the SAML:NameIdentifier in the service request is relied on, and applied via SAP NetWeaver User Mapping to automatically log on the SharePoint account as its mapped SAP named user.

The SAML2 protocol effectively prevents other message recipients, which do not posses the private key, are able to decipher and misuse the security token.

Both the SharePoint 2010 and SAP Gateway participants in the Duet Enterprise SSO handling, uphold to the SAML2 Protocol. This implies that SharePoint expects requires that Gateway as webservice provider exposes https-enabled service endpoints. In case not, on each runtime Duet Enterprise service request, SharePoint STS will throw exception from method System.ServiceModel.ClientCredentialsSecurityTokenManager.CreateServerX509TokenProvider, like: "System.InvalidOperationException: The service certificate is not provided for target 'http://<hostname>/sap/bc/srt/pm/.....' ", as it is not enabled to setup a valid SAML2 Protocol handling with the invoked Gateway relying party.

For this customer organization, as the SSL is only applied in the internal network to uphold the SAML2 Protocol handling between SAP Gateway system and SharePoint 2010 farm, there is no need for a (pricy + period-bound) signed SSL certificate from a Certificate Authority. A self-signed certificate suffices. Note that the Duet Enterprise Configuration Wizard creates and configures a self-signed SSL certificate in case the Gateway system is not yet SSL-enabled.

SharePoint 2010 BCS SAML Assertion too coarse-grained

Results in false positives notification at service providers of message replay

Security Assertion Markup Language (SAML) is an XML standard for exchanging authentication and authorization data between security domains. SAML is an XML-based protocol that uses security tokens containing assertions to pass information about a principal (usually an end user) between a SAML authority, that is an identity provider, and a SAML consumer, that is a service provider. SAML assertions contain statements that service providers use to make access control decisions. SAML enables web-based authentication and authorization scenarios including cross-domain single sign-on (SSO).

Source: http://en.wikipedia.org/wiki/SAML_1.1

SharePoint 2010 supports SAML 1.1 to authenticate as service consumer to external systems. Involved service applications are Secure Token Service as SharePoint internal identity provider, and Business Connectivity Services as service consumer to external systems. Required part of SAML support is that the identity provider must unique assert each identity token. This uniqueness is used by service provider to verify the validity of a SAML assertion: in case already ‘used’, the identity assertion should be interpretated as ‘stolen’ and potential misused in message replays.

Message replays are applied in 2 categories of security attacks:

• Service Availability: overflood the service provider with same requests over and over in a Denial of Service Attack.
• Data Integrity: request same data updates multiple times (e.g. bank transfer).

To protect against message replay attacks, service providers need to verify the idempotency of each received client request. Each unique message should be handled only once, also in the case it is send + received more than once to compensate for unreliable message delivery.

SAML assertion by SharePoint BCS

SharePoint BCS in the role of service consumer supports this by unique tagging each request sent with a SAML Assertion ID added in the message header. SharePoint BCS calculates this SAML SignatureValue based on 2 values to uniquely tag the message: message issuer and message instant id. The issuer value is fixed set to “SharePoint”, the instant value is dynamic set to timestamp at moment of sending:

<saml:Assertion xmlns:saml="urn:oasis:names:tc:SAML:1.0:assertion" AssertionID="_2519fc29-9577-403c-a66d-7926b55760a2" IssueInstant="2013-11-08T08:39:07.274Z" Issuer="SharePoint" MajorVersion="1" MinorVersion="1">

As it turns out this tagging is not guaranteed unique. In case more than 1 message is send by BCS in the same millisecond, the issue instant is equal for all those messages and therefore also the calculated SAML Assertion ID. And the receiving SAML consumer, aka the service provider will thus likely regard all these messages as equal, handle only the first, and refuse to handle the supposed duplicates.

Duet Enterprise: SAP Gateway as SAML consumer of SharePoint BCS

Duet Enterprise 1.0 applies SAML 1.1 to provide Single Sign-On from SharePoint 2010 into SAP NetWeaver Gateway 2.0. In the Duet Enterprise runtime client-service flow, SharePoint BCS is the service client, and SAP NetWeaver Gateway is the service provider. So Gateway must verify requests received from SharePoint BCS on idempotency, aka detect message replays. As result of the vulnerability in the BCS SAML assertion approach, Gateway may report false positives of message replay. This is visible in the NetWeaver WS (WebServices) errorlog via presence of error: CX_WS_ST_CACHE_ERROR / Error when writing the ST to DB.

I raised this issue to Duet Enterprise team for investigation. Their rightful response is that the problem cause is not within Duet Enterprise, but lies within SharePoint. So we contacted Microsoft. Microsoft Support acknowledges the problem cause, but sadly stated that “as it is not critical” no solution will be provided on short notice. Consequence is that you either have to accept the potential occurrence of false positives of message replays, or to ask SAP BASIS to disable the message replay detection mechanism at the receiving SAP side. This boils down to first answering the question whether it is likely that multiple requests are made from SharePoint BCS to SAP within same timestamp. A typical example of this is SharePoint search crawling, which issues a batch of requests into SAP. But it also may occur in a SharePoint webpart, when it uses a composition of SAP data services to render the webpart UI (e.g. customer + order headers).

See also: Explanation + resolution for CX_WS_ST_CACHE_ERROR, on the Duet Enterprise Space on SAP Community Network (SCN).

Duet Enterprise / GWPAM interview at SAP TechEd Amsterdam

Recording of the interview I gave at SAP Tech-Ed Amsterdam, talking about my experiences with SAP NetWeaver Gateway and Duet Enterprise, SAP Customer Engagement Initiative programs, and the recent released NetWeaver Gateway Productivity Accelerator for Microsoft (GWPAM).

Tip: activate Gateway Metadata cache to resolve NullPointer / CX_SY_REF_IS_INITIAL error

For a customer we are installing the myHR suite from Cordis Solutions to deliver ESS/MSS functionality within SharePoint application. The myHR suite is a SAP-certified Duet Enterprise product. As part of the project we first deploy the prerequisite supporting platform elements SAP NetWeaver Gateway 2.0 and Duet Enterprise for SharePoint and SAP. And next install the myHR suite. We do this across the customer landscapes of development/test, acceptance and production environments. Each environment installation ends with technical validation of the environment: Gateway, Duet Enterprise, Cordis myHR.

Per technical validation of the environments, I encounter the issue that the first 2 to 3 invocations for each of the myHR Duet Enterprise services returns the error CX_SY_REF_IS_INITIAL. After 3 service invocation calls, each myHR service successfully returns the requested SAP data. The explanation of this behavior is a kind of ‘warm-up’ effect: the Duet Enterprise services installed in the SAP Gateway system are not yet compiled / generated until first invocation, resulting in the CX_SY_REF_IS_INITIAL error on the Gateway system, and NullPointer exception in the invoking SharePoint UI-context. After a few invocation attempts, the service components are generated and available at runtime in the SAP system, and the invocation from there on gets successful the SAP data result.

Last week upon technical validation of the acceptance environment this symptom again manifested it as I expected. However, to my surprise the warm-up effect was not effectuated. All service invocations remained in erroneous state. Via ABAP debugging, I traced the error location. The problem originated at internal SAP Gateway method 'GET_META_MODELS' to instantiate the metadata model of the invoked service. I compared the runtime behavior on the faulty system versus an environment that has successfully be ‘warmed-up’. The difference was that on the latter system, the method to create the metadata model was not even reached. Well, not anymore… It was not needed to create the metadata model, because it could be retrieved from the Gateway Metadata cache.

This appeared the problem cause why the warm-up effect was not applied: at SAP Gateway deployment, one had forgot to activate the Gateway Metadata cache in this environment. After correcting that, the myHR services do start returning data. Well, that is... from each 3rd invocation on...

GWPAM - SAP data direct in Microsoft Office client-applications

Flagship of the Duet Enterprise / Gateway product team is Duet Enterprise for Microsoft SharePoint and SAP. Customers are very satisfied with the functionality and capabilities provided by this integration product, and the demonstrated product stability. A frequently asked question is to provide this level of exposing SAP data + processes also for use in Microsoft applications beyond SharePoint. The product team has responded to this market demand. Last week at SAP TechEd 2013 in Las Vegas, SAP NetWeaver Gateway Productivity Accelerator for Microsoft has been launched, shortly referred to as GWPAM.

As participant in the Duet Enterprise Customer Engagement Initiative (CEI) program, I was involved from the early development stage of GWPAM (under the internal codename BoxX). On request of the Duet Enterprise product team I performed so-called Takt-Testing, and reported my technical and functional thoughts + findings. Good to see that aspects of my feedback - predominantly influenced by my own technical background as an .Net architect/developer - have actually made it within the final product.

Like it’s big brother, GWPAM is in essence an end-product for the IT organization. It is an integration framework that internal IT departments and SAP + Microsoft partners (the ecosystem) can utilize, to build their own scenarios in which SAP / Microsoft integration is an important architectural element. GWPAM provides a Microsoft Visual Studio AddIn that .Net developers can use to directly in their familiair integrated development environment, lookup SAP Gateway OData services. And generate proxies to the Gateway OData services with standard .Net code.
The first foreseen scenarios are Microsoft Office Add-In’s, to expose and integrate the SAP business data in the everyday used Microsoft Office clients. For example, SAP CRM customer data in the form of Outlook contacts, invoice approval requests as Outlook tasks, functional data management of SAP masterdata through Excel, BW report data rendered in PowerPoint, and submit SAP CATS timetracking directly from your Outlook Calendar ...

Source: Introducing... SAP NetWeaver Gateway Productivity Accelerator for Microsoft [Holger Bruchelt]

Like Duet Enterprise for SharePoint, GWPAM provides support for the typical and recurring plumping aspects of SAP/Microsoft integration: Connectivity, Single Sign-on, End-2-End monitoring, .Net development tooling, integration with SAP Solution Manager. GWPAM offers a complete SAP / Microsoft integration package.
As with Duet Enterprise, the two suppliers have their collective strength and market presence behind this new product offering. This is also a major distinction compared to the various proprietary connectors of smaller parties.

As SAP / Microsoft interoperability expert, I am enthousiast about the addition of GWPAM to the SAP / Microsoft integration spectrum. GWPAM enables to build a new category of functional scenarios for end-customers. Now also for organizations that do not have SharePoint in their application landscape, but do have Microsoft Office installed on the desktops. And want to utilize that familiar employee environment for user intuitive operation of SAP data and business processes.

Related information:

• Introducing... SAP NetWeaver Gateway Productivity Accelerator for Microsoft
• SAP GWPAM Demo: Appointment Project Template - YouTube
• SAP GWPAM Demo: Add a Custom Value Help - YouTube

Applying SharePoint FAST for unlocking SAP data

An important new mantra is search-driven applications. In fact, "search" is the new way of navigating through your information. In many organizations an important part of the business data is stored in SAP business suites. A frequently asked need is to navigate through the business data stored in SAP, via a user-friendly and intuitive application context. For many organizations (78% according to Microsoft numbers), SharePoint is the basis for the integrated employee environment. Starting with SharePoint 2010, FAST Enterprise Search Platform (FAST ESP) is part of the SharePoint platform. All analyst firms assess FAST ESP as a leader in their scorecards for Enterprise Search technology. For organizations that have SAP and Microsoft SharePoint administrations in their infrastructure, the FAST search engine provides opportunities that one should not miss.

SharePoint Search

Search is one of the supporting pillars in SharePoint. And an extremely important one, for realizing the SharePoint proposition of an information hub plus collaboration workplace. It is essential that information you put into SharePoint, is easy to be found again. By yourself of course, but especially by your colleagues. However, from the context of 'central information hub', more is needed. You must also find and review via the SharePoint workplace the data that is administrated outside SharePoint. Examples are the business data stored in Lines-of-Business systems [SAP, Oracle, Microsoft Dynamics], but also data stored on network shares.

With the purchase of FAST ESP, Microsoft's search power of the SharePoint platform sharply increased. All analyst firms consider FAST, along with competitors Autonomy and Google Search Appliance as 'best in class' for enterprise search technology. For example, Gartner positioned FAST as leader in the Magic Quadrant for Enterprise Search, just above Autonomy. In SharePoint 2010 context FAST is introduced as a standalone extension to the Enterprise Edition, parallel to SharePoint Enterprise Search. In SharePoint 2013, Microsoft has simplified the architecture. FAST and Enterprise Search are merged, and FAST is integrated into the standard Enterprise edition and license.

SharePoint FAST Search architecture

The logical SharePoint FAST search architecture provides two main responsibilities:

1. Build the search index administration: in bulk, automated index all data and information which you want to search later. Depending on environmental context, the data sources include SharePoint itself, administrative systems (SAP, Oracle, custom), file shares, ...
2. Execute Search Queries against the accumulated index-administration, and expose the search result to the user.

In the indexation step, SharePoint FAST must thus retrieve the data from each of the linked systems. FAST Search supports this via the connector framework. There are standard connectors for (web)service invocation and for database queries. And it is supported to custom-build a .NET connector for other ways of unlocking external system, and then ‘plug-in’ this connector in the search indexation pipeline. Examples of such are connecting to SAP via RFC, or ‘quick-and-dirty’ integration access into an own internal build system.

In this context of search (or better: find) in SAP data, SharePoint FAST supports the indexation process via Business Connectivity Services for connecting to the SAP business system from SharePoint environment and retrieve the business data. What still needs to be arranged is the runtime interoperability with the SAP landscape, authentication, authorization and monitoring. An option is to build these typical plumping aspects in a custom .NET connector. But this not an easy matter. And more significant, it is something that nowadays end-user organizations do no longer aim to do themselves, due the involved development and maintenance costs.

An alternative is to apply Duet Enterprise for the plumbing aspects listed. Combined with SharePoint FAST, Duet Enterprise plays a role in 2 manners:
(1) First upon content indexing, for the connectivity to the SAP system to retrieve the data. The SAP data is then available within the SharePoint environment (stored in the FAST index files). Search query execution next happens outside of (a link into) SAP.
(2) Optional you'll go from the SharePoint application back to SAP if the use case requires that more detail will be exposed per SAP entity selected from the search result. An example is a situation where it is absolutely necessary to show the actual status. As with a product in warehouse, how many orders have been placed?

Security trimmed: Applying the SAP permissions on the data

Duet Enterprise retrieves data under the SAP account of the individual SharePoint user. This ensures that also from the SharePoint application you can only view those SAP data entities whereto you have the rights according the SAP authorization model. The retrieval of detail data is thus only allowed if you are in the SAP system itself allowed to see that data.
Due the FAST architecture, matters are different with search query execution. I mentioned that the SAP data is then already brought into the SharePoint context, there is no runtime link necessary into SAP system to execute the query. Consequence is that the Duet Enterprise is in this context not by default applied.

In many cases this is fine (for instance in the customer example described below), in other cases it is absolutely mandatory to respect also on moment of query execution the specific SAP permissions. The FAST search architecture provides support for this by enabling you to augment the indexed SAP data with the SAP autorisations as metadata.

To do this, you extend the scope of the FAST indexing process with retrieval of SAP permissions per data entity. This meta information is used for compiling ACL lists per data entity. FAST query execution processes this ACL meta-information, and checks each item in the search result whether it allowed to expose to this SharePoint [SAP] user.

This approach of assembling the ACL information is a static timestamp of the SAP authorizations at the time of executing the FAST indexing process. In case the SAP authorizations are dynamic, this is not sufficient.

For such situation it is required that at the time of FAST query execution, it can dynamically retrieve the SAP authorizations that then apply. The FAST framework offers an option to achieve this. It does require custom code, but this is next plugged in the standard FAST processing pipeline.

SharePoint FAST combined with Duet Enterprise so provides standard support and multiple options for implementing SAP security trimming. And in the typical cases the standard support is sufficient.

Applied in customer situation

The above is not only theory, we actually applied it in real practice. The context was that of opening up of SAP Enterprise Learning functionality to operation by the employees from their familiar SharePoint-based intranet. One of the use cases is that the employee searches in the course catalog for a suitable training. This is a striking example of search-driven application. You want a classified list of available courses, through refinement zoom to relevant training, and per applied classification and refinement see how much trainings are available. And of course you also always want the ability to freely search in the complete texts of the courses.

In the solution direction we make the SAP data via Duet Enterprise available for FAST indexation. Duet Enterprise here takes care of the connectivity, Single Sign-On, and the feed into SharePoint BCS. From there FAST takes over. Indexation of the exposed SAP data is done via the standard FAST index pipeline, searching and displaying the search results found via standard FAST query execution and display functionalities.

In this application context, specific user authorization per SAP course elements does not appy. Every employee is allowed to find and review all training data. As result we could suffice with the standard application of FAST and Duet Enterprise, without the need for additional customization.

Conclusion

Microsoft SharePoint Enterprise Search and FAST both are a very powerful tool to make the SAP business data (and other Line of Business administrations) accessible. The rich feature set of FAST ESP thereby makes it possible to offer your employees an intuitive search-driven user experience to the SAP data.

Duet Enterprise 2.0 Authentication flow

One of the changes in Duet Enterprise 2.0 is the Single Sign-On handling. Duet Enterprise 1.0 applies SAML to seamlessly authenticate and authorize the SharePoint authenticated user within SAP NetWeaver Gateway. For Duet Enterprise 2.0 this is no longer possible, result from the internal usage of OData for SAP - SharePoint data exchanges. The Duet Enterprise 2.0 SSO approach is X.509 based. On the Microsoft site you can find information that outlines the essential details of the authentication flow:

Duet Enterprise 2.0 Online / Extends reach of SAP data into Microsoft Cloud

SharePoint Online

Most significant in SharePoint 2013 is that it is ‘built for the cloud [1] [2]’. The motiviation for Microsoft to focus on cloud-enablement is market-demand: a considerable subset of its SharePoint customer-base struggles with hosting and operating SharePoint themselves on premise. And because of this struggle, a lot of SharePoint-using organizations are reluctant to upgrade their SharePoint deployment. Via the Cloud / on demand offering, Microsoft aims to reach 2 goals: relieve organizations from the burden of SharePoint operations, and enable those same organizations to stay on par with the feature progress in the SharePoint and Office platform. Microsoft plans an upgrade release cycle of every 90 days for SharePoint 2013 Online.

Duet Enterprise 2.0 Online

The SharePoint 2013 cloud offering also has positive implication for Duet Enterprise deployment options. In addition to the existing on premise installation option, Duet Enterprise 2.0 offers a cloud / on demand alternative. Extend the reach of your SAP data into the Microsoft cloud. In such infra architecture, the SAP landscape is on premise, but Microsoft SharePoint 2013 is serviced in the cloud.

The Duet Enterprise hybrid infrastructure setup imposes following requirements on the local and remote + shared environments:

• Each customer organization must [still] deploy the Duet Enterprise 2.0 SAP Add-On in own on premise SAP landscape; installed on the SAP NetWeaver Gateway 2.0 system
• SharePoint 2013 Online must include the Duet Enterprise 2.0 SharePoint Add-On
• The connectivity between on premise SAP Add-On and cloud hosted SharePoint Add-On requires a ‘light’ local SharePoint 2013 node. This node functions as intermediate layer ('broker') from internal SAP Gateway to SharePoint online and vice versa.
• Mutual trust-relationship between the on premise SAP Add-On and the on premise or cloud hosted SharePoint Add-On is based on X.509 certificates; no longer on SAML. The X.509 certificates are provisioned by the Duet Enterprise SSO Generator on the ('light') SharePoint 2013 node on premise.

Availability

Duet Enterprise 2.0 is already General Available for on premise deployments, but not yet for the cloud. It necessary follows upon the public availability of SharePoint 2013 Online, with the inclusion of Duet Enterprise 2.0 SharePoint Add-On. Microsoft has not yet announced the release date, but realistic expectation is ‘somewhere’ Q1/Q2 2013. Then also more details will be communicated on how to configure and operate Duet Enterprise 2.0 in a potential multi-tenancy SharePoint Online context.

OData Channel (ODC) development now preferred for new Duet Enterprise scenarios

As of Duet Enterprise FP1 SP3, the Duet Enterprise product team now clearly positions OData Channel as the recommended approach for new developments of custom Duet Enterprise scenarios:

Feature Pack 1 for Duet Enterprise 1.0 Development Introduction

Generic Channel remains supported, also within Duet Enterprise 2.0; but will not be enhanced with new features and capabilities. Message: OData is the way to go. Main reason is that OData has much more potential for consumption outside the SAP Business Suite boundaries, and therefore the Duet Enterprise team is focusing its resources on supporting that more prospective channel.

Consequence for Duet Enterprise custom development: learn new approach + tools

Although one thus still has the option to utilize the Generic Channel development approach, the signal send by Duet Enterprise team is clear. OData development has the momentum and the future. Consequence for custom Duet Enterprise development is that on SAP side the programmer must transition to develop Gateway OData Services instead of GSDO types. The concepts and steps of the 2 are somewhat comparable, but the tools and execution + coding are different. For GSDO the ABAP programmer uses SE80 and /IWFND/GWO_GEN; OData development is performed mainly within the new Service Builder (SEGW). Within both approaches you have the options to generate the data provider class through mapping to RFC and BOR, or to manual handcraft it. Once the data provider class is ready (OData DPC or Generic Channel GSDO), you expose it as SharePoint 2010 callable entity. For OData DPC, the BDC Browser tool is extended with the capability to convert OData Services into Gateway SOAP web service. At runtime, Gateway applies for this the Duet Enterprise FP1 SP3 OData-SOAP Bridge. The existence of the OData-SOAP Bridge is transparent for the SAP Duet Enterprise developer. One simple utilizes the BDC Browser as before for GSDO type, to now expose the OData Service for consumption within SharePoint 2010.

Noticeable is that for SharePoint 2010 development side, nothing changes. You still import the BDC model generated via BDC Browser containing one or more External Content Types that refer to Gateway SOAP webservice(s). And next utilize the out-of-the-box Business Data UI-controls, and/or the BCS API to develop a custom UI.

See also:

Duet Enterprise 2.0 General Available: improvements on service development and deployment options

After successful completion of Duet Enterprise 2.0 Ramp-Up and with SharePoint 2013 reaching RTM state, SAP and Microsoft have made the new version of Duet Enterprise General Available. Customers can download the software from SAP Market Place (Installations and Upgrades \ Duet Enterprise \ Duet Enterprise 2.0). Duet Enterprise 2.0 documentation can also be found on SAP Market Place, and also on MSDN.

Most significant of the new Duet Enterprise version is that it connects with the latest capabilities of the underlying platforms, respectively SAP Gateway 2.0 and Microsoft SharePoint 2013. In both these platforms a move is made to support cloud and lightweight usage scenarios - typically but not limited to mobility use cases.

Duet Enterprise 2.0 in overview

Key topics:

• Mobility
• Duet Enterprise now fully supports OData protocol; which is THE protocol for mobility because is it light-weight and self-describing, and supported by all mobile platforms
• Cloud
• Connect to SharePoint Online
• Only support for Odata channel; not for SOAP / Generic channel
• Requires a 'light' on premisse SharePoint installation, as intermediate layer ('broker') from internal SAP Gateway to SharePoint online. Includes SSO Generator, via reverse proxy connection with SharePoint Online
• Remark: Duet Enterprise cloud stuff is not yet GA; this will follow in a few month

Facilitate custom Duet Enterprise development

• Design Time improved
• 2 design time approaches supported
• Inside-Out: via the SAP NetWeaver Gateway generators
• Outside-In: start from SharePoint UI --> visually sketch / specify a screen in Visual Studio; export the .EDMX file and import into SAP Gateway (Service Builder) for constructing the Gateway data models
• Both approaches are restricted to flat SAP data entities
• Complex data structures still requires hand-crafting on SAP Gateway side
• Via OData service providers; the handcrafting work is far simpler as with Duet Enterprise 1.0 (FP1).
• Improve the development story:
• Reference architecture guidance
• Setting up local development landscape --> facilitate individual SharePoint developers to quickly start with Duet Enterprise development

Consumption of OData services

• Consequence: drop SAML, as OData does not support SAML. Use oAuth instead
• Generic Channel will still be supported; OData is extra channel possibility

Aligned with SharePoint 2013

• Duet Enterprise 2.0 REQUIRES SharePoint 2013
  • SharePoint 2013 BCS supports Odata consumption; earlier version not
  • SharePoint Online.

Duet Enterprise 2.0 deliberaty remains a SAP / Microsoft interoperability + development framework

In the philosophy of SAP and Microsoft, also Duet Enterprise 2.0 does not include end-user functionality, e.g. order-to-cash. Duet Enterprise remains a customization + development framework for SAP / Microsoft interoperability, that enables end-organizations and partners to compose concrete scenarios and build upon. Primary reason for the Duet Enterprise product team to not include end-user functionality is that partners and end-organizations know best what diverse scenarios are requested, and are therefore better suited to construct the optimal solution. Also, even SAP plus Microsoft simple do not have the development resources to realize and support a total set of end-user capabilities for which there is market demand.

Growing Duet Enterprise eco-system

It is left to the market / Duet Enterprise eco-system to deliver standard / common-of-the-shelf end-user functionalities. Well-known example is the Cordis Suite; for self-service HCM and purchase requisition solutions. More examples can be found on Duet Enterprise Unite Partner Connection. And more partners are known to be developing on additional scenarios.

With Duet Enterprise 2.0 General Availability, it is expected that more companies will find it benefial to develop end-user solutions. This is due the combination of Duet Enterprise 2.0 development improvements, out-of-the-box plus customization capabilities of SharePoint 2013 and Office 365, and deployment extended to cloud and mobile platforms resulting in a larger market. Partners will develop solutions for horizontal and vertical markets; for local to international markets, and from small/partial solutions to complete end-2-end scenarios.

Notification + data flow in Duet Enterprise Workflow mobile App

The Duet Enterprise Workflow capability can be used to expose SAP workflow tasks to SharePoint context. Once in SharePoint, it can be further built upon. You can extend the associated SharePoint workflow with additional workflow steps and business processing. Another option is to use the SharePoint Duet Enterprise Workflow site as integration layer to expose and handle the SAP tasks in a mobile application.

System architecture

In such system architecture, following systems/layers are present:

1. SAP backend / workflow engine: the host environment of the SAP workflows
2. SAP Gateway: general purpose SAP integration layer; here used to publish SAP workflows from backend to workflow notification subscribers
3. SharePoint Duet Enterprise Add-On: functions here as workflow notification subscriber; host of Duet Enterprise workflow instances, and task repository for the mobile application
4. Mobile view: retrieve plus display tasks, and propagate task approval decision back to the SharePoint task repository

Note that in this system architecture, the (mobile) view and SAP landscape operate disconnected; SharePoint is the glue between the backend and frontend parts.

Notification and data flow for exposing SAP task to non-SAP view

Sequence steps:

1. The SAP ERP workflow execution arives on a user decision task, that is configured for notification to Gateway (tx SPRO, Gateway workflow customization);
2. SAP Document Publisher composes a XML Payload document with task data: workitem identification plus business data. The addition of the workflow specific business data is optional in Gateway Workflow notification. If applicable, you can achieve this by extanding the workflow customization with the plug-in of a custom OutboundHandler;
3. SAP Document Publisher calls the SharePoint OBAWorkflowServer.asmx::CreateTask operation, with inputparameters the task(type) identifier, workitem identifier and XML PayLoad document;
4. SharePoint OBAWorkflowServer.asmx::CreateTask method:
   1. Queries the TaskLocator List in the Duet Enterprise Workflow rootsite, and determines on basis of tasktype and user language the Duet Enterprise Workflow subsite to which the received SAP task must be routed;
   2. Instantiates a new SharePoint workflow instance in the identified Duet Enterprise Workflow subsite, and associates this instance with the SAP workitem identifier;
   3. And uploads the received XML PayLoad document into the Workflow Business Data DocLibrary in the Workflow subsite;
5. The SharePoint Duet Enterprise Workflow contains by default only 1 step that represents the decision-making for the SAP task. It is possible to extend the SharePoint Workflow with additional steps and functionality. This will operate isolated from the SAP Workflow handling. The default approval task in the SharePoint Duet Enterprise workflow is the only connectionpoint between the SharePoint workflow and the SAP decision workflow task. Upon the Duet Enterprise Workflow arriving at the approval task, it appears in the SharePoint tasklist.
6. User opens the tasklist in the mobile application;
7. The tasklist App call SharePoint List.svc service (DataAccess Layer) to retrieve the open tasks for the logged-on user
8. SharePoint List.svc queries the SharePoint workflow tasklist, to determine and retrieve the open task items for logged-on user
9. SharePoint List.svc returns the task items in Odata format
10. The Duet Enterprise Workflow Approval Task and the XML PayLoad document are in the SharePoint Workflow administration logical associated to each other. If additional business extended properties are notified for the SAP workitem via OutboundHandler, the associated Business Data document must be opened from the SharePoint document library and its contents parsed to retrieve the additional properties.

Control and data flow for handling task decision from non-SAP mobile App to SAP workflow context

Sequence steps:

1. User makes a decision in the task management App;
2. Task management App calls SharePoint Workflow.asmx webservice to propagate the task approval decision to SharePoint task item;
3. SharePoint workflows.asmx updates the task item with approval decision;
4. The update on SharePoint taskitem triggers the TaskItemEventReceiver;
5. Triggered TaskItemEventReceiver::ItemUpdating method checks whether the SharePoint task item is completed; that is a approval decision is made. If completed, TaskItemEventReceiver calls [through SharePoint Business Connectivity Services] the Update method of External ContentType  Duet Enterprise Workflow Task;
6. SharePoint BCS calls for Duet Enterprise Workflow Task Update the wfUpdate method of SAP workflow consumer webservice; with input parameters workitem identifier, decision value and comments;
7. SAP workflow consumer webservice propagates task approval decision to the associated SAP workflow instance, and executes apply decision on the SAP workitem;
8. SAP workflow completes this workitem step, and proceeds to the next step in SAP workflow. If it was the last step in workflow, the SAP workflow is now completed;
9. TaskItemEventReceiver updates the status of SharePoint taskitem to completed. In default (generated) Duet Enterprise Workflow is this the only step/task, and the SharePoint workflow is completed. The SharePoint taskitem is removed from the users open tasks list.

Deploy custom Duet Enterprise scenario from Dev into SAP landscape

In the case that an organization develops custom business scenarios on top of Duet Enterprise foundation, an essential step is to implement it through to the organizations SAP production environment. Duet Enterprise relies herefore on the standard CTS+ transport mechanism.

Design time - prepare the custom business scenario

It is best practice to include all of the Duet Enterprise services for one scenario within an own isolated and recognizable namespace in the Gateway system. Each scenario will then be recognizable, and not per accident mingled with deliverables of another Duet Enterprise scenario. To achieve this, you create per business scenario an own Software Component in SLD, and use that to create in ESR an own Software Component Version and namespace.

Steps:

1. Open SLD, and create a new Software Component
2. Open ESR, and create a new Software Component Version via import of the newly created Software Component
3. Create a namespace in the newly created Software Component Version
4. Copy the Basic DataTypes from SAP IW TNG to the new Software Component Version and namespace
5. Copy the Fault Message from SAP IW TNG to the new Software Component Version and namespace
6. Activate the resulting changelist(s) in ESR
7. Logon to Gateway system, transaction SE80; and create a Package
8. Logon to Gateway system, transaction SProxy; and create + activate proxies for the Basic DataTypes, ExchangeFaultData and ExchangeLogData in the newly created Package.

With this, the preparation phase is completed. You now have an own business scenario container for all services that will be developed for the scenario.

That is, within the Dev environment. The same container must be created on the counterpart Gateway system into Test, QA and Production environments to implement the custom scenario in these environments. To facilitate this, all of the design time preparation steps performed on the Gateway system are recorded in a CTS+ Transport Workbench request: Create Package (step 7), and Create + Activate Proxies (step 8).

Note that it is not needed to also implement or repeat the SLD and ESR steps into the Test, QA and Production landscapes. For Duet Enterprise design time, the SLD and ESR only have a role in the design time phase; in the development environment. Once the service proxy objects for ESR content are generated (basic / standard ones, and the custom developed), there is no dependency on ESR content. The transport therefore only consists of the service definition(s) and the proxy objects that are part of them.

Design time - create the service definitions for the custom business scenario

Per service this consists of the 2 familiar actions: create and map a Gateway DataModel, and next generate the service definition + BDC Model for this Gateway DataModel (GSDO Type).

Steps:

1. Logon to Gateway system, transaction SE80; and create a new Gateway DataModel; make sure not to select OData channel (Duet Enterprise 1.0 and SharePoint 2010 do not support OData consumption; this will change with Duet Enterprise 2.0 and SharePoint 2013). The system enforces you to include the creation of the Gateway DataModel in a Workbench request. The advice is to not include it in the same Workbench request from the preparation step, but instead use an own Workbench request for the design time deliverables. As result you have the implementation of the Package including the generic basic DataTypes + FaultMessage separated from the implementation of the specific scenario entities. It is typical in a custom realization project that you incremently build new versions of an own service definition, and if a previous version already transported you must be able to replace that while preserving the Package.
2. Map 1 of more operations in the Gateway DataModel; e.g. Query and Read
3. Generate the Gateway DataModel; the system enforces you to include the generation and creation of SAP entities within a Customizing request.
4. Open transaction BDC Browser; if this is the first service definition for the custom business scenario, create a new Business Scenario. Otherwise you may opt to include the 2^nd, 3^rd and so on service definition all within the same Business scenario. I advice this as best practice, you then end up with a single Duet Enterprise Business Scenario that contains the entire set of Service Definitions developed for it. In case you create a new Business Scenario, it is again enforced by the system to include it within a customizing request. You can select the same as earlier created for the Gateway Model generation(s).
5. Select the Business Scenario, and Create BDC Model for GSDO Type; select the Gateway Model earlier created. Also here the system enforces to include within a customizing request.
6. [Optional] Open transaction SProxy, select the generated service interface and test its correct working for the diverse methods in the interface (e.g. Find, Read, Create, )
7. [Optional] Open SOAMANAGER on Gateway system, browse to Business Scenario Configuration. Select your Business Scenario, and export + save into a local XML file.

Implementaton time - transport the custom business scenario

When the custom realization has progressed or is even completed in the Gateway system of the SAP Development environment, next is to propagate it within the SAP landscape.

Steps:

1. Logon to Gateway system of SAP Dev environment, transaction SE03 or SE10; and release all the Workbench and Customizing requests that are related to the custom Duet Enterprise Business Scenario
2. Logon to Gateway system of the transport target environment (SAP Test or QA environment), and open Transport Management System (TMS). Import the released Workbench and Customizing requests from the Development environment.
3. Activate each of the requests. Make sure to do this in the correct logical order: first package, next the business scenario including service definitions.
4. Logon to Gateway system of the transport target environment, transaction BDC Browser. Validate that the Business Scenario(s) is/are present. Also validate that for each of the imported service definitions, there is no end point yet.
5. Release and activate all of the imported service definitions in SOAMANAGER. Browse to Business Scenario Configuration, and import the Business_Scenario XML file that is exported in the Development environment. If the Business Scenario already exists here from a previous import, first delete it. Or otherwise you can also explicitly associate new service definitions within a/the Business Scenario, and assign a profile. After activation of the service definitions, validate in transaction BDC Browser that for each of the service definitions now an end point is available.
6. [Optional] Open transaction SProxy, select the imported service interface and test its correct working for the diverse methods in the interface (e.g. Find, Read, Create, )
7. Transaction BDC Browser, select the imported Business Scenario and click on Export Business Scenario. Hand over the exported SharePoint BDC Model(s) to import within the SharePoint BCS Service Application of the implementation environment.
8. [Optional] In SharePoint webapplication, create an ExternalList for one of the imported External Content Types. Validate that you can retrieve SAP data via Duet Enterprise and SAP Gateway using one of the transported / imported Gateway Service Definitions.

Duet Enterprise troubleshooting pointers

Generic

• Duet Enterprise 1.0 Troubleshooting Guide
• Troubleshooting example
• Holger Bruchelt's Duet Enterprise blogpage

SAP side

• BDC Browser - Troubleshooting pointers
• BDC Browser Configuration - Software Component SAP IW TNG missing
• Workflow tasks not delivered?
  • txn SLG1 on backend
  • txn SRT_UTIL on Gateway
  • txn SBGRFCMON: retry queue contains wf-items failed to deliver to SharePoint
    • use 'Unit Analysis \ Run Unit with Trace' to execute again/debug
• ABAP runtime errors / dumps: txn ST22
• ABAP system log: txn SM21
• Duet Enterprise 1 - Troubleshooting Report Delivery
• GW 2.0 SP5: Which Tools Can Be Used for Troubleshooting
• For Payload tracing:
  • txn SRT_UTIL
  • txn SMICM (ICF trace tools)
• ABAP Function Modules and services: External Debugging, Set External Breakpoints for ABAP Debugging
• Troubleshooting Gateway Development
• SAP system HTTP Logging
• Using Starter Services without EhP4

SharePoint side

• ULS Logviewer
• SharePoint Developer Dashboard
• Troubleshoot Business Connectivity Services using Performance Monitor
• SCOM (System Center Operations Manager
• Overview of monitoring and troubleshooting