Utilize Azure Function to resolve lack of CORS aware within SharePoint Online active authentication flow

In earlier post I informed about facing CORS issue when interoperate from external JavaScript client via OAuth passive authentication with SharePoint Online REST API. And on how-to easily resolve this via Azure Function Proxy. That approach works for OAuth based passive authentication, with the OAuth AccessToken exchanged via HTTP header. However, in case the external client wants to apply active authentication, the situation complicates severely. The explanation is that in this authentication flow, the SharePoint Online authentication token is exchanged as HttpOnly cookie:

[Courtesy by Wictor Wilén: How to do active authentication to Office 365 and SharePoint Online]

The SharePoint Developer Support Team Blog published an article that outlines which set of requests one needs to implement for SharePoint Online active authentication. In case of non-browser clients, this is sufficient to enable SharePoint Online interoperability from an external client. For instance, I've applied this OAuth active authentication flow within an Excel VBA client context, a colleague of mine used it from Curl script, and yet another from a Java application. In a non-browser client context, the received OAuth Active Authentication HTTP response can successfully be parsed to extract the SPOIDCRL cookie.

Modern browsers utilize Cross-Origin protection themselves to mitigate risks: HttpOnly cookie cannot be read by clientside code, and the browser only include cookie for outgoing requests pointing to same domain as from which the HttpOnly cookie is earlier received within current browser session.

Modern secure browser have inherent protection that prevents access to secure HttpOnly cookies: on the wire [Fiddler] the SPOIDCRL cookie is included in the response, yet the browser [here Chrome, via Developer Tools] does not allow access it.

Implication is that the custom CORS handling needs to be further extended to have the browser include the required SharePoint Online authentication cookie cross-domain in SharePoint Online REST API calls. Although same approach via Azure Function Proxy can successfully be utilized to cross-domain request the SharePoint Online active authentication end-point, the browser accepts the returned cross-origin response but your clientside code is not enabled to extract the SharePoint Online authentication token included as HttpOnly cookie (SPOIDCRL). And even if that would be possible, the second blocker is that the SharePoint Authentication Token must be included as cookie for successful authentication again SharePoint Online, but browsers only include the cookie for requests going to the same domain. Meaning that all SharePoint interoperability requests coming from the external JavaScript client must be proxied to the same domain as of the proxied Active Authentication endpoint; <your tenant>/_vti_bin/IDCRL.svc

On the webclient side the needed changes are minimal: include ‘withCredentials:true’ in issued XMLHttpRequest requests. But on the receiving and processing Azure Function (Proxy) side, more must be changed:

 •   Function-Apps have out-of-the-box platform support for CORS; as utilized for the CORS handling in case of OAuth passive authentication flow. However, I experienced this is limited to only return the CORS headers Allowed-Domain + Allowed-Method. Current, Function-Apps platform CORS does not include support for cross-domain authentication handling via ‘withCredentials’. I tried alternatives to set the missing 'Access-Control-Allow-Credentials' header explicitly self in the responseOverrides of the Azure Function Proxy; and learned that in case CORS is configured on the platform level, any Cross-Origin headers in the responseOverrides object are just silently ignored; not included in the resultant http response of the Azure Function Proxy call. Then I tried to instead nullify the Function-App Platform CORS setting, as described in Azure Functions Access-Control-Allow-Credentials with CORS. This works to receive the SharePoint Online authentication token cross-domain.

 •   But on next usage to allow the browser to send the cookie, all your SharePoint Online REST calls must go to the same domain as from which the cookie was received. This can be achieved by also proxy these calls via the Azure Function Proxy. This on itself is very simple to configure in the Azure Function Proxy via generic pattern matching in a new proxy:

However then on invoking that proxy cross-domain from the client / browser, it is refused by Azure Function Proxy with Http 405 / Not Allowed ➔ because I removed all allowed domains at the platform CORS configuration of the Azure Function, while the browser includes in the request the CORS 'Origin' header that now no longer matches on the Azure Function-App level (a typical case of 'chicken-egg' situation).

 •   I then tried with '*' as allowed domain in the platform CORS configuration. But this again results that the explicit additional CORS response headers via responseOverrides object are ignored. And in addition '*' is not allowed on browser level in conjunction with 'withCredentials': Failed to load https://msdnwvstrienspocors.azurewebsites.net/IDCRL.svc: Response to preflight request doesn't pass access control check: The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'. Origin 'https://wvstrien.sharepoint.com' is therefore not allowed access. The credentials mode of requests initiated by the XMLHttpRequest is controlled by the withCredentials attribute.

 •   As implication of the above enumerated various experiences, I conclude that in their current implementation, Azure Function Proxy is unfit to resolve CORS handling in context with SharePoint Online active authentication flow. Therefore I decided for alternative approach to explicit self build the CORS-aware proxy behavior in custom Azure Functions. I minimal need to implement 2 functions; the CORS aware/allowed access to default MSO endpoint https://login.microsoftonline.com/rst2.srf and https://#ADFSHOST#/adfs/services/trust/2005/usernamemixed (username/password ADFS endpoint) can still be arranged via an Azure Function Proxy, similar as earlier configured for OAuth passive authentication endpoint. But for CORS aware access to combination of IDCRL.svc and any authenticated request to your SharePoint Online tenant the Azure Function Proxy approach thus falls short. For each of these 2, a custom proxy implementation is required.

 •   I observed that also in case of explicit CORS handling coded in Azure Function, still any Function-App platform CORS configuration prevales above that. Therefore it is required to completely disable platform CORS on the Function-App. As the CORS-aware Azure Function Proxies for the other / first 2 requests in the active authentication flow do depend on the platform CORS configuration, I decided to split within 2 seperate Function-Apps with own domain. I could also have resorted to custom proxy build for the first 2 requests and include in the same App container, but whereever possible I prefer a 'configuration' and out-of-the-box approach above (maintaining) custom code. So I stick with utilization of Azure Function Proxy where possible.

Complete CORS-enabled setup for all of the 4 requests involved in SharePoint Online active authentication: the first 2 can be CORS-enabled via Azure Function Proxy, the last 2 must be CORS-enabled via custom Azure Function.

Configuration of the Azure Function Proxies to CORS-enable "<your custom STS>/ adfs/services/trust/2005/usernamemixed" and "https://login.microsoftonline.com/RST2.srf"

Impression of the custom Azure Function: CORS-enable "<your SPO tenant>/_vti_bin/idcrl.svc" is simple; proxy-ing SharePoint REST API calls is more complex and challenging. Note: the custom proxy for active authentication requests in your SharePoint tenant is also fit for non-REST requests, e.g. to browse your SharePoint Online sites

Cross-Domain / CORS prepared JavaScript browser client, interoperating with SharePoint Online REST API

---

---

This is what happens in the webclient (browser) / SharePoint Online interoperability with adjustment for CORS:

First request automatic issued by secure-aware browser: OPTIONS to start with CORS Preflight

On server / client confirmed preflight, get cross-domain the SPO authentication cookie

Utilze the retrieved SPO authentication cookie within browser-issued SPO REST calls

Authenticate from Curl into SharePoint Online with Modern Authentication

Code-snippet for interoperability from Curl context - for example, could be from a Linux or MacOS workstation / server -, to Office 365 SharePoint Online; with service-based authentication by applying Active / Modern Authentication protocol handling:

#General variables
ProxyAccount="sa-curlAccount"
ProxyPassword="******************"
ProxyProtocol="http"
ProxyServer="xxx.xxx.xxx.xxx"
ProxyPort="8080"
SharePointCurlAccount="sa-curlAccount"
SharePointOnlineTenant="<URL of SharePoint Online tenant>"
UploadFile="<file to upload>"
UploadLocation="<URL of SharePoint Document Library>"

#Fixed variables
OUTPUT=${HOME}/Interop/output
TMP=${HOME}/Interop/tmp/spo

#the following steps are required to upload data from Curl context to SharePoint Online:
#
#1. Retrieve an authentication cookie to Office 365 through invocation of webservices
#1.a. (Optional) Step 0: determine the URL of the custom Security Token Service (STS) to next
#     request a SAML:assertion for account identified by credentials
#1.b. Step 1: request SAML:assertion from the identified custom STS for account identified by
#     credentials
#1.c. Step 2: use the SAML:assertion to request binary security token from Office 365
#1.d. Step 3: use the binary security token to retrieve the authentication cookie
#2. Step 4: Use that Office 365 authentication cookie in subsequent webservice requests to
#   SharePoint Online REST API
 
#1.a. (Optional) Step 0: determine the URL of the custom Security Token Service (STS) to next
#     request a SAML:assertion for account identified by credentials (outside datacenter, with proxy)
curl -U ${ProxyAccount}:${ProxyPassword} -k -x ${ProxyProtocol}://${ProxyServer}:${ProxyPort} -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "login=${SharePointCurlAccount}&xml=1" https://login.microsoftonline.com/GetUserRealm.srf -w "\n" > ${TMP}/O365_response_step_0

#Extract requested STSAuthURL from response step 1
STSURL=`sed -n 's:.*<STSAuthURL>\(.*\)</STSAuthURL>.*:\1:p' ${TMP}/O365_response_step_0`

#Create input for step 1

File: O365_request_step_1-1

<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope
    xmlns:s="http://www.w3.org/2003/05/soap-envelope"
    xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
    xmlns:saml="urn:oasis:names:tc:SAML:1.0:assertion"
    xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy"
    xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
    xmlns:wsa="http://www.w3.org/2005/08/addressing"
    xmlns:wssc="http://schemas.xmlsoap.org/ws/2005/02/sc"
    xmlns:wst="http://schemas.xmlsoap.org/ws/2005/02/trust">
    <s:Header>
        <wsa:Action s:mustUnderstand="1">http://schemas.xmlsoap.org/ws/2005/02/trust/RST/Issue</wsa:Action>
        <wsa:To s:mustUnderstand="1">https://sts.<tenant>.com/adfs/services/trust/2005/usernamemixed</wsa:To>
        <wsa:MessageID>b07da3ec-9824-46a5-a102-2329e0c5f63f</wsa:MessageID>
        <ps:AuthInfo
            xmlns:ps="http://schemas.microsoft.com/Passport/SoapServices/PPCRL" Id="PPAuthInfo">
            <ps:HostingApp>Managed IDCRL</ps:HostingApp>
            <ps:BinaryVersion>6</ps:BinaryVersion>
            <ps:UIVersion>1</ps:UIVersion>
            <ps:Cookies></ps:Cookies>
            <ps:RequestParams>AQAAAAIAAABsYwQAAAAxMDMz</ps:RequestParams>
        </ps:AuthInfo>
        <wsse:Security>
            <wsse:UsernameToken wsu:Id="user">
                <wsse:Username>sa-curlAccount@<tenant>.com</wsse:Username>
                <wsse:Password>*************</wsse:Password>
            </wsse:UsernameToken>
            <wsu:Timestamp Id="Timestamp">

File: O365_request_step_1-2

            </wsu:Timestamp>
        </wsse:Security>
    </s:Header>
    <s:Body>
        <wst:RequestSecurityToken Id="RST0">
            <wst:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</wst:RequestType>
            <wsp:AppliesTo>
                <wsa:EndpointReference>
                    <wsa:Address>urn:federation:MicrosoftOnline</wsa:Address>
                </wsa:EndpointReference>
            </wsp:AppliesTo>
            <wst:KeyType>http://schemas.xmlsoap.org/ws/2005/05/identity/NoProofKey</wst:KeyType>
        </wst:RequestSecurityToken>
    </s:Body>
</s:Envelope>

cat ${TMP}/O365_request_step_1-1 > ${TMP}/O365_request_step_1
echo "<wsu:Created>`date -u +'%Y-%m-%dT%H:%M:%SZ'`</wsu:Created>" >> ${TMP}/O365_request_step_1
echo "<wsu:Expires>`date -u +'%Y-%m-%dT%H:%M:%SZ' --date='-15 minutes ago'`</wsu:Expires>" >> ${TMP}/O365_request_step_1
cat ${TMP}/O365_request_step_1-2 >> ${TMP}/O365_request_step_1

#1.b. Step 1: request SAML:assertion from the identified custom STS for account identified by
#     credentials (internal datacenter, without webproxy to outside)
curl -X POST -H "Content-Type: application/soap+xml; charset=utf-8" -d "@${TMP}/O365_request_step_1" ${STSURL} -w "\n" > ${TMP}/O365_response_step_1

#Extract requested SAML:assertion from response step 1
sed 's/^.*\(<saml:Assertion.*saml:Assertion>\).*$/\1/' ${TMP}/O365_response_step_1 > ${TMP}/O365_response_step_1.tmp

#Create input for step 2

File: O365_request_step_2-1

<?xml version="1.0" encoding="UTF-8"?>
<S:Envelope
    xmlns:S="http://www.w3.org/2003/05/soap-envelope"
    xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
    xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy"
    xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
    xmlns:wsa="http://www.w3.org/2005/08/addressing"
    xmlns:wst="http://schemas.xmlsoap.org/ws/2005/02/trust">
    <S:Header>
        <wsa:Action S:mustUnderstand="1">http://schemas.xmlsoap.org/ws/2005/02/trust/RST/Issue</wsa:Action>
        <wsa:To S:mustUnderstand="1">https://login.microsoftonline.com/rst2.srf</wsa:To>
        <ps:AuthInfo
            xmlns:ps="http://schemas.microsoft.com/LiveID/SoapServices/v1" Id="PPAuthInfo">
            <ps:BinaryVersion>5</ps:BinaryVersion>
            <ps:HostingApp>Managed IDCRL</ps:HostingApp>
        </ps:AuthInfo>
        <wsse:Security>

File: O365_request_step_2-2

        </wsse:Security>
    </S:Header>
    <S:Body>
        <wst:RequestSecurityToken xmlns:wst="http://schemas.xmlsoap.org/ws/2005/02/trust" Id="RST0">
            <wst:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</wst:RequestType>
            <wsp:AppliesTo>
                <wsa:EndpointReference>
                    <wsa:Address>sharepoint.com</wsa:Address>
                </wsa:EndpointReference>
            </wsp:AppliesTo>
            <wsp:PolicyReference URI="MBI"></wsp:PolicyReference>
        </wst:RequestSecurityToken>
    </S:Body>
</S:Envelope>

cat ${TMP}/O365_request_step_2-1 > ${TMP}/O365_request_step_2
cat ${TMP}/O365_response_step_1.tmp >> ${TMP}/O365_request_step_2
cat ${TMP}/O365_request_step_2-2 >> ${TMP}/O365_request_step_2

rm ${TMP}/O365_response_step_1.tmp

#1.c. Step 2: use the SAML:assertion to request binary security token from Office 365
#     (outside datacenter, with proxy)
curl -U ${ProxyAccount}:${ProxyPassword} -k -x ${ProxyProtocol}://${ProxyServer}:${ProxyPort} -X POST -H "Content-Type: application/soap+xml; charset=utf-8" -d "@${TMP}/O365_request_step_2" https://login.microsoftonline.com/RST2.srf -w "\n" > ${TMP}/O365_response_step_2

#Extract requested binary security token from response step 2
sed 's/^.*\(<wsse:BinarySecurityToken.*wsse:BinarySecurityToken>\).*$/\1/' ${TMP}/O365_response_step_2 > ${TMP}/O365_response_step_2.tmp

#Create input for step 3

cat ${TMP}/O365_response_step_2.tmp | cut -d'>' -f2 | cut -d'<' -f1 > ${TMP}/O365_request_step_3
BinarySecurityToken=`cat ${TMP}/O365_request_step_3`

rm ${TMP}/O365_response_step_2.tmp

#1.d. Step 3: use the binary security token to retrieve the authentication cookie (outside
#     datacenter, need to pass webproxy)
curl -v -U ${ProxyAccount}:${ProxyPassword} -k -x ${ProxyProtocol}://${ProxyServer}:${ProxyPort} -X GET -H "Authorization: BPOSIDCRL ${BinarySecurityToken}" -H "X-IDCRL_ACCEPTED: t" -H "User-Agent:" ${SharePointOnlineTenant}/_vti_bin/idcrl.svc/ > ${TMP}/O365_response_step_3 2>&1

#Remove DOS ^M from response step 3
cat ${TMP}/O365_response_step_3 | sed 's/^M//' > ${TMP}/O365_response_step_3.tmp

#Extract requested authentication cookie from response step 3 and create input for step 4
echo "Set-Cookie: SPOIDCRL=`cat ${TMP}/O365_response_step_3.tmp | grep Set-Cookie | awk -F'SPOIDCRL=' '{print $2}'`" > ${TMP}/O365_request_step_4

rm ${TMP}/O365_response_step_3.tmp

#2. Step 4: Use that Office 365 authentication cookie in subsequent webservice requests to
#   SharePoint Online REST API (outside datacenter, with proxy)
curl -U ${ProxyAccount}:${ProxyPassword} -k -x ${ProxyProtocol}://${ProxyServer}:${ProxyPort} -b ${TMP}/O365_request_step_4 -T "{${OUTPUT}/${UploadFile}}" ${UploadLocation}

exit 0

Alternative for the upload handling; interoperation via SharePoint API / webservice:

curl -U ${ProxyAccount}:${ProxyPassword} -k -x ${ProxyProtocol}://${ProxyServer}:${ProxyPort} -X POST -H "Accept: application/json;odata=verbose" -d "" ${SharePointOnlineTenant}/_api/contextinfo > ${TMP}/O365_response_step_4_tmp

FormDigest=`sed -n 's:.*FormDigestvalue:\(.*\),.*:\1:p' ${TMP}/O365_response_step_4_tmp`
rm ${TMP}/O365_response_step_4.tmp

curl -U ${ProxyAccount}:${ProxyPassword} -k -x ${ProxyProtocol}://${ProxyServer}:${ProxyPort} -X POST -H "X-RequestDigest: @${FormDigest}; X-HTTP-Method: PUT” --data-binary  "{${OUTPUT}/${UploadFile}}"  ${SharePointOnlineTenant}/teams/siteX/_api/web/GetFileByServerRelativeUrl('Shared%20Documents/SubFolder/${UploadFile}')/Files/$value

How-to resolve peculiarity with .aspx file upload from automated client context

The capabilities (powers) of SharePoint as underlying business applications platform can be utilized in multiple ways. Example of a pragmatic one is to utilize SharePoint as authorized web-distribution platform for content created elsewhere. The added value it brings here are that the origin of the content itself does not need to be (made) accessible for the readers, no need to (web) serve content, the permission handling of SharePoint can be utilized to only make the content available for authorized persons.

This simple application usage is for instance applied to continuously publish and distribute system monitoring dashboard report on infra level from Linux servers to the monitoring people. They do not / are not allowed access to the Linux servers in the datacenter, but are granted access to SharePoint as application platform. This worked perfectly, until we recently migrated the hosting site from SharePoint on-prem to SharePoint Online.

The problem symptom is that the uploaded .aspx file on selecting it in the SharePoint Online UI, does not open in browser, but instead starts the ‘Download / Save As’ behavior. Which clearly obstructs the SharePoint role as host of the published infra dashboard. Other .aspx files in the same library that were migrated from the source site on-prem, all do open in the browser. That rules out document library settings. So it must be directly tied to the upload of the file. The particular upload is via Curl – which gave us some challenges to authenticate against SharePoint Online, but I will post on that separately -, but once uploaded nothing can be identified what clarifies why this file behaves different from the other .aspx files in the library. Inspected the document item properties, even up to detailed level via SharePoint Designer: all the same. The only noticeable difference is when trying to resolve in SharePoint Designer via file item properties the url to document: for the troublesome document this returns in ‘file not found’.

Strange, as the file is clearly present; and as such accessible both in the browser via the SharePoint listview UI, as when opening the library in Windows Explorer via ‘Open with Explorer’. Heck, even with sync via OneDrive, the file is included in the synced library content.

So this really kept us puzzled. Until business user self-remembered an action we did on restoring the upload via Curl: as good SharePoint citizen, I reduced the permission level of the automated client account from ‘Full Control’ to ‘Contributor’. This turned out to be the key to explaining and next resolving the issue. On SharePoint level, also uploaded .aspx files are treated as (content) page. And for completed upload + administration, the account uploading an .aspx file must have ”Add and Customize Pages - Add, change, or delete HTML pages or Web Part Pages, and edit the Web site using a Microsoft SharePoint Foundation-compatible editor”. And that permission is missing from “Contributor” permission level. It does have "Add items to library", and therefore the upload itself succeeds from the automated client context. But the next processing on SharePoint (Online) side after the file upload to convert it into a browsable page context is not allowed when only 'Contributor'. The needed permission is included in ‘Full Control’, but that gives away too much control to the automated client account. Applying ‘Least Privilege’security principle, I therefore configured a new Permission Level “Upload ASPX page”, included the needed permission, and assigned this permission level to the automated client account.

Enable CORS not possible on SharePoint authenticated application without HttpModule

On 1st research, enabling CORS on SharePoint (on-premisse) web applications seems to be easy: just configure it in the web.config.

Source: enable Cross-Origin Resource Sharing - CORS on IIS7:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
 <system.webServer>
   <httpProtocol>
     <customHeaders>
       <add name="Access-Control-Allow-Origin" value="*" />
     </customHeaders>
   </httpProtocol>
 </system.webServer>
</configuration>

(also given as answer on stackoverflow: Enable CORS in SharePoint 2013)

However, on validating the effect of the config change, it turns out that the web.config based approach is insufficient in case of authenticated SharePoint application. You encounter 2 issues with only the configuration based approach:

1. In the web.config you can only specify either the '*' or explicit client domain for the Access-Control-Allow-Origin header. To interoperate with an authenticated application, you need to pass authorization headers and set Access-Control-Allow-Credentials to true. The issue is that according to the CORS specification (see also explanation on MDN, HTTP Access Control), if Access-Control-Allow-Credentials is set to true, Access-Control-Allow-Origin cannot contain *, to disallow just any client making requests with credentials attached. Also on CORS specification level, it is documented that in case of specific domain list, it actually means 1 single specific domain (https://www.w3.org/TR/cors/#access-control-allow-origin-response-header). This limits enable CORS to a single client application.
2. First call in the CORS protocol, is HTTP OPTIONS to establish whether CORS is allowed between client host domain and server host domain. A web.config enabled SharePoint application responds successful on the OPTIONS call with CORS-allow headers. However, SharePoint / IIS returns for authenticated webapplication the OPTIONS response with status code 401; and as result the preflight handling is stopped in browsers that respect the CORS specification.

Conclusion:
Enable CORS for a SharePoint / IIS authenticated webapplication, cannot be done via configuration only. Resolving both issues strict on SharePoint level is possible, but requires a custom HttpModule. If such server-side deployment is not allowed, another option is to resolve it on infra level, via a Reverse Proxy setup.

Background resources:

• Mozilla Developer Network - HTTP Access Control
• Fetch - CORS protocol
• Enable CORS in SharePoint 2013
• SharePoint 2013 - Integration Challenges - #4 Same Origin Policy & Authentication - CORS
• How to make GET CORS request with authorization header
• Cross-Origin Resource Sharing with Credentials

Conceptual approach for exposing external SAP systems into SharePoint channel

The last 6 years I have focussed on utilizing SharePoint as additional channel to expose SAP data and operate SAP functionalities. As I am now about to start another challenge in the broad SharePoint arena, I decided to write down my advice on how-to achieve such SAP/SharePoint integration. What approach to take, architectural advice, and what steps. On intent I do not elaborate on technology products, as these by nature are only temporarily. Instead I focus my outline on concepts.

Rough outline of approach

1. Establish the reference solution architecture for integrating SAP business process handling into SharePoint front-end / UI
2. Establish which of the business processes to expose via SharePoint as additional channel, versus which to keep solely through SAP UI channel
3. Establish the IST situation: Business Process definitions and supportive landscape, Enterprise IT strategy + roadmap, IT landscape.
4. Establish the SOLL situation.
5. Determine the IST-SOLL gap
6. Map the solution architecture + IST-SOLL analysis onto concrete technology: Microsoft and SAP.
7. Architect, design, develop, test and implement specific scenario(s).

Architecture guidelines

• Apply a layered architecture
• Loosely-connected front-end / back-office system(s) landscape through the usage of interfaces (contracts)
• Interoperability standards based: webservices, security standards, monitoring
• Abstract the LOBsystem business process handling as a contract-specification
• Extend the decouped operation in alternative front-end with the concept of Operational Data Store to enable temporary administration outside the external LOB system, and allow complete preparation of administrative action before submitting into the external Lines of Business system.

Main steps for a specific scenario

Per specific scenario / use case to expose external LOB system into SharePoint

• Identify the data and functionalities of the external LOB system that you want to expose via SharePoint as (additional) channel for user operation.
• Identity in the platform of the external LOB system context, the building blocks (functional and technical) that can be used to operate the identified data and functions. The external LOB system, and its internal working, is a (magical) blackbox for the unknowingly. You need the aid of a knowledgable business analyst to help with identification of the proper functional building blocks, and how to utilize and invoke them.
• Service-enable the identified building blocks. Don’t be fooled by the concept of SAP Enterprise Services. In practice, these are only usable in a (SAP) laboratory scope, not in the concrete situation of an end-organization. In reality you need to provide an own service layer to expose the identified building blocks. SAP has acknowledged this, and provides multiple supporting foundations and building blocks. In my opinion, current the most significant is SAP Gateway to expose SAP data and processes as REST services. For consumption in Microsoft clients, SAP delivers Duet Enterprise and Gateway for Microsoft that extend on the basic interoperability capabilities of SAP Gateway. In addition to SAP, frameworks of other suppliers are also available on the market to service-enable SAP. A disadvantage of them is that it may complicate your IT landscape. Trend of latest years in end-organizations is to consolidate on SAP + Microsoft IT policy

Related information:

1. Considerations for SAP / Microsoft interoperability approach and technology
2. User-centric Development Process for Duet Enterprise projects

Programmatically set decision of Duet Enterprise task

In our award-winning VIEW (Virtual Integrated Enterprise Workplace) concept we include a generic inbox. In the generic inbox tasks are aggregated from multiple sources: SAP, SharePoint, Oracle, proprietary systems, and so on. The user sees in his/her inbox all outstanding tasks from the diverse task backends. And it is also supported to direct handle the tasks from the generic inbox.

To make the generic inbox real also for Duet Enterprise tasks, it is required to programmatically approve them via the inbox instead of via the standard Duet Enterprise task forms. A good start how to achieve this is article ‘how to create Duet SharePoint webparts’ of Ravi Sharma, in which he uses programmatic Duet Enterprise task approval as an example.

The approach Ravi takes in his setup is to directly update the SAP task by invoke via SharePoint BCS Application Programming Interface (API), explict self the wfUpdate method on the DuetEnterprise WorkflowTask External ContentType. And if the SAP task update is successful, also afterwards alter the SharePoint task that functions as reference or placeholder in SharePoint context to that SAP task.

Although this approach works, it has multiple architectural disadvantages:

• It bypasses the standard Duet Enterprise workflow capability to update the SAP task via the SharePoint task.
• It effectively duplicates some of the code and working of the internals of Duet Enterprise workflow update. As architect, I very much dislike any code duplication.
• Per conceptual task update, 2 tasks must be explicit updated: first the SAP task, next the SharePoint task. This makes the task handling more complex.
• There is a clear difference in the approval of SharePoint tasks that are created via Duet Enterprise workflow capability, versus regular SharePoint tasks (from any SharePoint workflow, e.g. document approval)

I favor the approach to handle all SharePoint tasks, including the ones that are instantiated as result of Duet Enterprise workflow publication, the same. And be ignorant of what happens in the SharePoint context and possible beyond as result of the task decision update. To achieve this, my SharePoint task update code is a variant on the ‘updateSharePointTask’ method of Ravi. And in this approach, I have no need for the ‘updateSAPTask’ method, as this responsibility is handled by the standard Duet Enterprise workflow capability.

There are 2 caveats when programmatically update Duet Enterprise tasks:

1. Generic for SharePoint tasks: extendedproperties field ‘ows_taskstatus’ must have been set for the task update to be accepted. However, you cannot set this direct self, but set it indirect by this code snippet:
2. Specific for Duet Enterprise tasks: In ItemUpdating eventreceiver, multiple checks are done to verify that it is indeed the intention to propagate to SAP. One of these checks is that a string-compare is done on both the set status text as status code. If this string-concattenation is not present in the Duet Enterprise task PossibleOutcomes (you specify this when you in SharePoint ‘Configure a new SAP workflow task type’), then the SharePoint task update succeeds, but the update is silently not propagated onto the connected SAP side. To make sure the programatically set status text is in sync with the configured PossibleOutcomes, I derive the status text based on that same configured PossibleOutcomes:

The complete code example to programmatically set the decision on SharePoint task, which via Duet Enterprise workflow capability then propagate the decision made to the connected SAP task:

Duet Enterprise 2.0 - installation troubleshooting

Initial, the installation of the first version (1.0) of Duet Enterprise was cumbersome and labour-intensive, and therefore also error prone. With the Duet Enterprise 1.0 installation wizard at SAP side, the installation already was substantially improved. And this has progressed on to the installation of the new version, Duet Enterprise 2.0. However, as the landscape is inherent complex (minimal one, but typically more SAP backends, a SAP Gateway system, and SharePoint 2013 farm), you still may face issues.

In this blog I describe some of the issues that I encountered during Duet Enterprise 2.0 installations, and share my resolutions.

Issue 1: ‘Add-on IW_DUETE Release 100 can only be installed in client 000’

Import of the Duet Enterprise 2.0 SAP Add-On via transaction SAINT gives the following error:

Resolution: Use client 000 to import the add-on via transaction SAINT.

Issue 2: ‘The remote certificate is invalid according to the validation procedure’.

This error can occur in multiple SharePoint-SAP scenarios: runtime invocation of Duet Enterprise service from SharePoint 2013 side, import of a Duet Enterprise 2.0 BDC Model: Application definition import failed. The following error occurred: The remote certificate is invalid according to the validation procedure.

Resolution: Typical the above error message is caused by mismatch of the SSL certificate used on SAP Gateway side to encrypt the traffic, versus the SAP SSL certificate that has been imported in SharePoint 2013 ‘Manage trust’. To repair, import the SSL certificate again. You can either ask again the SAP BASIS administrator to export from SAP Gateway STRUST. But a more convenient, and in my experiences also one with better results (was already so for Duet Enterprise 1.0) is to open on SharePoint web frontend the SAP service url, and then from SSL certificate warning export the SAP SSL certificate to file; and next add this to ‘Manage trust’.

Issue 3: ‘The root certificate that was just selected is invalid’

Importing in SharePoint 2013 ‘Manage Trust’ the SSL certificate of SAP NetWeaver Gateway results in error:

Resolution: import the SSL certificate via Internet Explorer (11) instead of Firefox.

Issue 4: ‘No service found for namespace /IWWRK/, name DUET_WORKFLOW_CORE

Manifest itself upon configuration of Duet Enterprise 2.0 Workflow solution.

Resolution: Add and activate the Duet Enterprise 2.0 Gateway REST service in MAINT_SERVICE.

Issue 5: DuetConfig: ‘The operation name is missing or invalid’

For instance occurs with the DuetConfig command to configure workflow, command with multiple operation parameters: DuetConfig.exe -importbdc -FeatureName Workflow -LsiUrl "https://tnvsrm.tnv.corp/sap/opu/odata/IWWRK/DUET_WORKFLOW_CORE;mo;c=SHAREPOINT_DE/" -BdcServiceApplication "Business Data Connectivity Service" -UserSubLsiUrl "https://tnvsrm.tnv.corp/sap/opu/odata/IWBEP/SUBSCRIPTIONMANAGEMENT; mo;v=2;c=SHAREPOINT_DE/"

Resolution: careful check the command line that the start character for each of the operation parameter names is indeed the minus (‘-‘) character. In case you copied the command line from the deployment guide, this may be ‘invisible’ the wrong character. To be sure, in the command line explicit (replace and) type the ‘-‘ character:

Issue 6: ‘Lobsystem (External System) returned authentication error'

For instance, occurs upon trying to import a Duet Enterprise 2.0 BDC Model into SharePoint 2013 Business Connectivity Services application:

Resolution 1: Check that the Duet Enterprise 2.0 generated ‘Duet Root Certificate Authority’ is added to SSL Server Standard in STRUST on the SAP Gateway system.

Resolution 2: Check the user mapping of the SharePoint account used to invoke the SAP Gateway service / import the Duet Enterprise 2.0 BDC Model. In particular check the extid that it contains the correct pattern (account name, comma + space, and then the domain name in small capitals):

Issue 7: Import of Duet Enterprise 1.0 BDC Model fails due missing ‘WSDL’ application definition.

Resolution: Add an “Duet Enterprise 1.0 WSDL” application definition in SharePoint 2013 Secure Store.

Issue 8: Invocation of Duet Enterprise 1.0 service fails due missing SSO

Browsing a SharePoint 2013 External List results in error ‘An unsecured or incorrectly secured fault was received from the other party’.

Inspect the SRT_UTIL ErrorLog:

Resolution: Besides the Duet Enterprise 2.0 SSO approach based on X.509 certificate, also enable Duet Enterprise 1.0 SAML2 approach. Use the Duet Enterprise 1.0 Deployment Guide for information how to enable SAML2 in the Duet Enterprise 2.0 (Gateway 2.0 + SharePoint 2013) landscape.

Epilogue

Compared to the excellent troubleshooting guide for Duet Enterprise 1.0, the above list of issues plus resolutions is much smaller. The Duet Enterprise product team of SAP plus Microsoft has clearly improved their delivery on this aspect.

Others that have been deploying Duet Enterprise 2.0 may have encountered other issues as I did so far. Still a general classification of all issues seen so far is that they are caused by manual error, indirect caused by the sometimes unclear, fragmented and even incorrect Duet Enterprise deployment guides.

Tip: bypass WebProxy for BCS service application

Setting up a fresh Duet Enterprise landscape, I was confronted with an issue trying to import BDC Models from the SAP Gateway system into SharePoint BCS:

Application definition import failed. The following error occurred: Error loading url: "http://....". This normally happens when url does not point to a valid discovery document, or XSD schema.

Using Fiddler I detected that the problem cause is a "(407) Proxy Authentication Required" issue: "The ISA server requires authorization to fulfill the request. Access to proxy filter is denied." Although I did setup a rule in Windows CredentialsManager for automatic authentication against the web proxy, this is not picked up in the context of BCS service application as an autonomous running process. As it turns out, by default .NET web applications and services will attempt to use a proxy, even if it doesn’t need one.

So how then to resolve from this situation? Multiple approaches are possible here:

1. Explicitly set the Proxy Credentials for the BCS application process.
   It is not possible to set the proxy credentials direct in the web.config of 14hive\webservices\bdc. Instead you must use a 2-step delegation approach: refer in the web.config to a custom Proxy module implementation, and build the custom Proxy to explicitly set the proxy credentials:

   namespace ByPassProxyAuthentication
   {
       public class ByPassProxy : IWebProxy
       {
           public ICredentials Credentials
           {
               get { 
                   return new NetworkCredential(
                       "username", "password", "domain"); }
               set { }
           }
       }
   }
   

   <system.net>
       <defaultProxy enabled="true" useDefaultCredentials="false">
           <module type="ByPassProxyAuthentication.ByPassProxy, 
                  ByPassProxyAuthentication"/>
       </defaultProxy>
   </system.net>
   

2. Disable usage of (default)proxy altogether for the BCS application process.
   This is a viable approach in case the consumed external systems are all within the internal company network infra.

   <system.net>  
     <defaultProxy  
       enabled="false"  
       useDefaultCredentials="false"/>  
     </system.net>

3. Disable usage of (default)proxy for specific addresses for the BCS application process.

   <system.net>
       <defaultProxy>
           <bypasslist>
               <add address="[a-z]+\.contoso\.com" />
               <add address="192\.168\..*" />
               <add address="Netbios name of server" />
           </bypasslist>
       </defaultProxy>
   </system.net>
   

   The first bypasses the proxy for all servers in the contoso.com domain; the second bypasses the proxy for all servers whose IP addresses begin with 192.168. The third bypass entry is for the ServerName
4. Disable usage of proxy for specific address on system level.
   This is in fact the most simple approach, just disable proxy usage for certain url's for all processes on system level. That is also the potential disadvantage, it can be that it is not allowed to disable proxy usage for all processes.
   You disable the proxy via IE \ Internet Options \ Connections \ LAN Settings \ Advanced \ Proxy Server \ Exception <Do not use proxy server for addresses beginning with>.

Informational sources used:

• Configuring Proxy settings in .NET config file
• Web Service access + internal traffic + 407 Proxy Authentication Required
• Proxy Configuration issues with UPA in SharePoint 2010 /2013

Explanation + resolution of BCS "Cannot find any matching endpoint configuration"

Explanation

SharePoint BCS operates with external systems through the connection information in BDC Models, administrated in its metadata store. BDC Models can 1. be imported via Central Admin UI, Business Connectivity Services application; 2. provisioned via features; 3. and manually be created via SharePoint Designer.

When BDC administrates a new BDC Model, the BDC administration process reads the WcfMexDocumentUrl value from the LobSystem node in the model; and uses this to runtime retrieve from the WcfMexDocument (= Service Metadata) all the service endpoints. The received service endpoins in the WcfMexDocument are administrated in the internal WcfConnectionManager class.

When the BDC service application is invoked by a BDC client to interoperate with the external system, BDC uses the WcfEndpointAddress value from the LobSystemInstance node in the model as the service endpoint url. But it does not directly invoke the (WCF) service on that endpoint address. It first uses this endpoint address to retrieve additional endpoint configuration data (e.g. what character is used as wildcard in the external system) from the WcfConnectionManager class. Information that was derived when administrating and processing the BDC Model, and stored as key-value pairs with the key equal to a service endpoint address included in the service metadata / WcfMexDocument.

Now, in case the WcfEndpointAdress value in the LobSystemInstance node is not present as one of the service endpoints in the WcfMexDocument; BDC is not able to locate the additional endpoint configuration; and all BDC client invocations of this External ContentType will fault with the error message “Cannot find any matching endpoint configuration”.

Resolution

The above sketched situation originates from a mismatch in the BDC Model versus the WcfMexDocument. The fix is to make sure that the WcfEndPointAddress in the BDC Model, does have a match with one of the service endpoint addresses of the WcfMexDocument. This either means to adjust the WcfEndPointAddress value in the BDC Model to a correct and present vaue, and reimport the model. Or to modify the WcfMexDocumentUrl to include the WcfEndPointAddress value as one of the service endpoints. Note that as BDC builds up its internal administration in WcfConnectionManager on processing a BDC Model, also in the latter case the BDC Model must be re-imported despite that itself has not changed. Through reimport the situation in WcfConnectionManager will be reset and corrected.

Tip: Resolve GWPAM issue with misnamed embedded Resources reference

It is a common approach applied by .NET developers to give multipart names to the individual projects in a Visual Studio solution. The name parts identify different aspects of the Visual Studio project, a typical name pattern is "<CompanyName>."<Application>."<Component>".

Example of this: "TNV.Purchase.OrderManagement".

I applied this naming approach also for a GWPAM project to build a Microsoft Outlook AddIn: TNV.TasksClustering.OutlookAddIn. However, after I installed the compiled AddIn, Outlook failed to load it. The error details were:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Resources.MissingManifestResourceException: Could not find any resources appropriate for the specified culture or the neutral culture. Make sure "TNV_TasksClustering_OutlookAddIn.Icons.resources" was correctly embedded or linked into assembly "TNV.TasksClustering.OutlookAddIn" at compile time’

I inspected the GWPAM generated code to analyze the problem cause. It appears that GWPAM replaces in the code that it generates, the dots ('.') in the Visual Studio project name into the underline ('_') character. Thus 'TNV.TasksClustering.OutlookAddIn' becomes 'TNV_TasksClustering_OutlookAddIn' in the generated C# code. But a side effect is that this results in a naming-inconsistency with embedded Resources: Visual Studio namely by default embeds .NET resources files in the compiled assembly with the unchanged project name, in this case thus still with the dots:

I have reported this issue to the Duet Enterprise / GWPAM product team. I expect it to be fixed in a forthcoming service pack. Until then, the pragmatic and simple workaround is to correct in the generated C# code the line that tries to load the embedded resource:

SharePoint 2010 BCS SAML Assertion too coarse-grained

Results in false positives notification at service providers of message replay

Security Assertion Markup Language (SAML) is an XML standard for exchanging authentication and authorization data between security domains. SAML is an XML-based protocol that uses security tokens containing assertions to pass information about a principal (usually an end user) between a SAML authority, that is an identity provider, and a SAML consumer, that is a service provider. SAML assertions contain statements that service providers use to make access control decisions. SAML enables web-based authentication and authorization scenarios including cross-domain single sign-on (SSO).

Source: http://en.wikipedia.org/wiki/SAML_1.1

SharePoint 2010 supports SAML 1.1 to authenticate as service consumer to external systems. Involved service applications are Secure Token Service as SharePoint internal identity provider, and Business Connectivity Services as service consumer to external systems. Required part of SAML support is that the identity provider must unique assert each identity token. This uniqueness is used by service provider to verify the validity of a SAML assertion: in case already ‘used’, the identity assertion should be interpretated as ‘stolen’ and potential misused in message replays.

Message replays are applied in 2 categories of security attacks:

• Service Availability: overflood the service provider with same requests over and over in a Denial of Service Attack.
• Data Integrity: request same data updates multiple times (e.g. bank transfer).

To protect against message replay attacks, service providers need to verify the idempotency of each received client request. Each unique message should be handled only once, also in the case it is send + received more than once to compensate for unreliable message delivery.

SAML assertion by SharePoint BCS

SharePoint BCS in the role of service consumer supports this by unique tagging each request sent with a SAML Assertion ID added in the message header. SharePoint BCS calculates this SAML SignatureValue based on 2 values to uniquely tag the message: message issuer and message instant id. The issuer value is fixed set to “SharePoint”, the instant value is dynamic set to timestamp at moment of sending:

<saml:Assertion xmlns:saml="urn:oasis:names:tc:SAML:1.0:assertion" AssertionID="_2519fc29-9577-403c-a66d-7926b55760a2" IssueInstant="2013-11-08T08:39:07.274Z" Issuer="SharePoint" MajorVersion="1" MinorVersion="1">

As it turns out this tagging is not guaranteed unique. In case more than 1 message is send by BCS in the same millisecond, the issue instant is equal for all those messages and therefore also the calculated SAML Assertion ID. And the receiving SAML consumer, aka the service provider will thus likely regard all these messages as equal, handle only the first, and refuse to handle the supposed duplicates.

Duet Enterprise: SAP Gateway as SAML consumer of SharePoint BCS

Duet Enterprise 1.0 applies SAML 1.1 to provide Single Sign-On from SharePoint 2010 into SAP NetWeaver Gateway 2.0. In the Duet Enterprise runtime client-service flow, SharePoint BCS is the service client, and SAP NetWeaver Gateway is the service provider. So Gateway must verify requests received from SharePoint BCS on idempotency, aka detect message replays. As result of the vulnerability in the BCS SAML assertion approach, Gateway may report false positives of message replay. This is visible in the NetWeaver WS (WebServices) errorlog via presence of error: CX_WS_ST_CACHE_ERROR / Error when writing the ST to DB.

I raised this issue to Duet Enterprise team for investigation. Their rightful response is that the problem cause is not within Duet Enterprise, but lies within SharePoint. So we contacted Microsoft. Microsoft Support acknowledges the problem cause, but sadly stated that “as it is not critical” no solution will be provided on short notice. Consequence is that you either have to accept the potential occurrence of false positives of message replays, or to ask SAP BASIS to disable the message replay detection mechanism at the receiving SAP side. This boils down to first answering the question whether it is likely that multiple requests are made from SharePoint BCS to SAP within same timestamp. A typical example of this is SharePoint search crawling, which issues a batch of requests into SAP. But it also may occur in a SharePoint webpart, when it uses a composition of SAP data services to render the webpart UI (e.g. customer + order headers).

See also: Explanation + resolution for CX_WS_ST_CACHE_ERROR, on the Duet Enterprise Space on SAP Community Network (SCN).

Duet Enterprise / GWPAM interview at SAP TechEd Amsterdam

Recording of the interview I gave at SAP Tech-Ed Amsterdam, talking about my experiences with SAP NetWeaver Gateway and Duet Enterprise, SAP Customer Engagement Initiative programs, and the recent released NetWeaver Gateway Productivity Accelerator for Microsoft (GWPAM).

Tip: activate Gateway Metadata cache to resolve NullPointer / CX_SY_REF_IS_INITIAL error

For a customer we are installing the myHR suite from Cordis Solutions to deliver ESS/MSS functionality within SharePoint application. The myHR suite is a SAP-certified Duet Enterprise product. As part of the project we first deploy the prerequisite supporting platform elements SAP NetWeaver Gateway 2.0 and Duet Enterprise for SharePoint and SAP. And next install the myHR suite. We do this across the customer landscapes of development/test, acceptance and production environments. Each environment installation ends with technical validation of the environment: Gateway, Duet Enterprise, Cordis myHR.

Per technical validation of the environments, I encounter the issue that the first 2 to 3 invocations for each of the myHR Duet Enterprise services returns the error CX_SY_REF_IS_INITIAL. After 3 service invocation calls, each myHR service successfully returns the requested SAP data. The explanation of this behavior is a kind of ‘warm-up’ effect: the Duet Enterprise services installed in the SAP Gateway system are not yet compiled / generated until first invocation, resulting in the CX_SY_REF_IS_INITIAL error on the Gateway system, and NullPointer exception in the invoking SharePoint UI-context. After a few invocation attempts, the service components are generated and available at runtime in the SAP system, and the invocation from there on gets successful the SAP data result.

Last week upon technical validation of the acceptance environment this symptom again manifested it as I expected. However, to my surprise the warm-up effect was not effectuated. All service invocations remained in erroneous state. Via ABAP debugging, I traced the error location. The problem originated at internal SAP Gateway method 'GET_META_MODELS' to instantiate the metadata model of the invoked service. I compared the runtime behavior on the faulty system versus an environment that has successfully be ‘warmed-up’. The difference was that on the latter system, the method to create the metadata model was not even reached. Well, not anymore… It was not needed to create the metadata model, because it could be retrieved from the Gateway Metadata cache.

This appeared the problem cause why the warm-up effect was not applied: at SAP Gateway deployment, one had forgot to activate the Gateway Metadata cache in this environment. After correcting that, the myHR services do start returning data. Well, that is... from each 3rd invocation on...

GWPAM - SAP data direct in Microsoft Office client-applications

Flagship of the Duet Enterprise / Gateway product team is Duet Enterprise for Microsoft SharePoint and SAP. Customers are very satisfied with the functionality and capabilities provided by this integration product, and the demonstrated product stability. A frequently asked question is to provide this level of exposing SAP data + processes also for use in Microsoft applications beyond SharePoint. The product team has responded to this market demand. Last week at SAP TechEd 2013 in Las Vegas, SAP NetWeaver Gateway Productivity Accelerator for Microsoft has been launched, shortly referred to as GWPAM.

As participant in the Duet Enterprise Customer Engagement Initiative (CEI) program, I was involved from the early development stage of GWPAM (under the internal codename BoxX). On request of the Duet Enterprise product team I performed so-called Takt-Testing, and reported my technical and functional thoughts + findings. Good to see that aspects of my feedback - predominantly influenced by my own technical background as an .Net architect/developer - have actually made it within the final product.

Like it’s big brother, GWPAM is in essence an end-product for the IT organization. It is an integration framework that internal IT departments and SAP + Microsoft partners (the ecosystem) can utilize, to build their own scenarios in which SAP / Microsoft integration is an important architectural element. GWPAM provides a Microsoft Visual Studio AddIn that .Net developers can use to directly in their familiair integrated development environment, lookup SAP Gateway OData services. And generate proxies to the Gateway OData services with standard .Net code.
The first foreseen scenarios are Microsoft Office Add-In’s, to expose and integrate the SAP business data in the everyday used Microsoft Office clients. For example, SAP CRM customer data in the form of Outlook contacts, invoice approval requests as Outlook tasks, functional data management of SAP masterdata through Excel, BW report data rendered in PowerPoint, and submit SAP CATS timetracking directly from your Outlook Calendar ...

Source: Introducing... SAP NetWeaver Gateway Productivity Accelerator for Microsoft [Holger Bruchelt]

Like Duet Enterprise for SharePoint, GWPAM provides support for the typical and recurring plumping aspects of SAP/Microsoft integration: Connectivity, Single Sign-on, End-2-End monitoring, .Net development tooling, integration with SAP Solution Manager. GWPAM offers a complete SAP / Microsoft integration package.
As with Duet Enterprise, the two suppliers have their collective strength and market presence behind this new product offering. This is also a major distinction compared to the various proprietary connectors of smaller parties.

As SAP / Microsoft interoperability expert, I am enthousiast about the addition of GWPAM to the SAP / Microsoft integration spectrum. GWPAM enables to build a new category of functional scenarios for end-customers. Now also for organizations that do not have SharePoint in their application landscape, but do have Microsoft Office installed on the desktops. And want to utilize that familiar employee environment for user intuitive operation of SAP data and business processes.

Related information:

• Introducing... SAP NetWeaver Gateway Productivity Accelerator for Microsoft
• SAP GWPAM Demo: Appointment Project Template - YouTube
• SAP GWPAM Demo: Add a Custom Value Help - YouTube

Duet Enterprise 2.0 General Available: improvements on service development and deployment options

After successful completion of Duet Enterprise 2.0 Ramp-Up and with SharePoint 2013 reaching RTM state, SAP and Microsoft have made the new version of Duet Enterprise General Available. Customers can download the software from SAP Market Place (Installations and Upgrades \ Duet Enterprise \ Duet Enterprise 2.0). Duet Enterprise 2.0 documentation can also be found on SAP Market Place, and also on MSDN.

Most significant of the new Duet Enterprise version is that it connects with the latest capabilities of the underlying platforms, respectively SAP Gateway 2.0 and Microsoft SharePoint 2013. In both these platforms a move is made to support cloud and lightweight usage scenarios - typically but not limited to mobility use cases.

Duet Enterprise 2.0 in overview

Key topics:

• Mobility
• Duet Enterprise now fully supports OData protocol; which is THE protocol for mobility because is it light-weight and self-describing, and supported by all mobile platforms
• Cloud
• Connect to SharePoint Online
• Only support for Odata channel; not for SOAP / Generic channel
• Requires a 'light' on premisse SharePoint installation, as intermediate layer ('broker') from internal SAP Gateway to SharePoint online. Includes SSO Generator, via reverse proxy connection with SharePoint Online
• Remark: Duet Enterprise cloud stuff is not yet GA; this will follow in a few month

Facilitate custom Duet Enterprise development

• Design Time improved
• 2 design time approaches supported
• Inside-Out: via the SAP NetWeaver Gateway generators
• Outside-In: start from SharePoint UI --> visually sketch / specify a screen in Visual Studio; export the .EDMX file and import into SAP Gateway (Service Builder) for constructing the Gateway data models
• Both approaches are restricted to flat SAP data entities
• Complex data structures still requires hand-crafting on SAP Gateway side
• Via OData service providers; the handcrafting work is far simpler as with Duet Enterprise 1.0 (FP1).
• Improve the development story:
• Reference architecture guidance
• Setting up local development landscape --> facilitate individual SharePoint developers to quickly start with Duet Enterprise development

Consumption of OData services

• Consequence: drop SAML, as OData does not support SAML. Use oAuth instead
• Generic Channel will still be supported; OData is extra channel possibility

Aligned with SharePoint 2013

• Duet Enterprise 2.0 REQUIRES SharePoint 2013
  • SharePoint 2013 BCS supports Odata consumption; earlier version not
  • SharePoint Online.

Duet Enterprise 2.0 deliberaty remains a SAP / Microsoft interoperability + development framework

In the philosophy of SAP and Microsoft, also Duet Enterprise 2.0 does not include end-user functionality, e.g. order-to-cash. Duet Enterprise remains a customization + development framework for SAP / Microsoft interoperability, that enables end-organizations and partners to compose concrete scenarios and build upon. Primary reason for the Duet Enterprise product team to not include end-user functionality is that partners and end-organizations know best what diverse scenarios are requested, and are therefore better suited to construct the optimal solution. Also, even SAP plus Microsoft simple do not have the development resources to realize and support a total set of end-user capabilities for which there is market demand.

Growing Duet Enterprise eco-system

It is left to the market / Duet Enterprise eco-system to deliver standard / common-of-the-shelf end-user functionalities. Well-known example is the Cordis Suite; for self-service HCM and purchase requisition solutions. More examples can be found on Duet Enterprise Unite Partner Connection. And more partners are known to be developing on additional scenarios.

With Duet Enterprise 2.0 General Availability, it is expected that more companies will find it benefial to develop end-user solutions. This is due the combination of Duet Enterprise 2.0 development improvements, out-of-the-box plus customization capabilities of SharePoint 2013 and Office 365, and deployment extended to cloud and mobile platforms resulting in a larger market. Partners will develop solutions for horizontal and vertical markets; for local to international markets, and from small/partial solutions to complete end-2-end scenarios.

Notification + data flow in Duet Enterprise Workflow mobile App

The Duet Enterprise Workflow capability can be used to expose SAP workflow tasks to SharePoint context. Once in SharePoint, it can be further built upon. You can extend the associated SharePoint workflow with additional workflow steps and business processing. Another option is to use the SharePoint Duet Enterprise Workflow site as integration layer to expose and handle the SAP tasks in a mobile application.

System architecture

In such system architecture, following systems/layers are present:

1. SAP backend / workflow engine: the host environment of the SAP workflows
2. SAP Gateway: general purpose SAP integration layer; here used to publish SAP workflows from backend to workflow notification subscribers
3. SharePoint Duet Enterprise Add-On: functions here as workflow notification subscriber; host of Duet Enterprise workflow instances, and task repository for the mobile application
4. Mobile view: retrieve plus display tasks, and propagate task approval decision back to the SharePoint task repository

Note that in this system architecture, the (mobile) view and SAP landscape operate disconnected; SharePoint is the glue between the backend and frontend parts.

Notification and data flow for exposing SAP task to non-SAP view

Sequence steps:

1. The SAP ERP workflow execution arives on a user decision task, that is configured for notification to Gateway (tx SPRO, Gateway workflow customization);
2. SAP Document Publisher composes a XML Payload document with task data: workitem identification plus business data. The addition of the workflow specific business data is optional in Gateway Workflow notification. If applicable, you can achieve this by extanding the workflow customization with the plug-in of a custom OutboundHandler;
3. SAP Document Publisher calls the SharePoint OBAWorkflowServer.asmx::CreateTask operation, with inputparameters the task(type) identifier, workitem identifier and XML PayLoad document;
4. SharePoint OBAWorkflowServer.asmx::CreateTask method:
   1. Queries the TaskLocator List in the Duet Enterprise Workflow rootsite, and determines on basis of tasktype and user language the Duet Enterprise Workflow subsite to which the received SAP task must be routed;
   2. Instantiates a new SharePoint workflow instance in the identified Duet Enterprise Workflow subsite, and associates this instance with the SAP workitem identifier;
   3. And uploads the received XML PayLoad document into the Workflow Business Data DocLibrary in the Workflow subsite;
5. The SharePoint Duet Enterprise Workflow contains by default only 1 step that represents the decision-making for the SAP task. It is possible to extend the SharePoint Workflow with additional steps and functionality. This will operate isolated from the SAP Workflow handling. The default approval task in the SharePoint Duet Enterprise workflow is the only connectionpoint between the SharePoint workflow and the SAP decision workflow task. Upon the Duet Enterprise Workflow arriving at the approval task, it appears in the SharePoint tasklist.
6. User opens the tasklist in the mobile application;
7. The tasklist App call SharePoint List.svc service (DataAccess Layer) to retrieve the open tasks for the logged-on user
8. SharePoint List.svc queries the SharePoint workflow tasklist, to determine and retrieve the open task items for logged-on user
9. SharePoint List.svc returns the task items in Odata format
10. The Duet Enterprise Workflow Approval Task and the XML PayLoad document are in the SharePoint Workflow administration logical associated to each other. If additional business extended properties are notified for the SAP workitem via OutboundHandler, the associated Business Data document must be opened from the SharePoint document library and its contents parsed to retrieve the additional properties.

Control and data flow for handling task decision from non-SAP mobile App to SAP workflow context

Sequence steps:

1. User makes a decision in the task management App;
2. Task management App calls SharePoint Workflow.asmx webservice to propagate the task approval decision to SharePoint task item;
3. SharePoint workflows.asmx updates the task item with approval decision;
4. The update on SharePoint taskitem triggers the TaskItemEventReceiver;
5. Triggered TaskItemEventReceiver::ItemUpdating method checks whether the SharePoint task item is completed; that is a approval decision is made. If completed, TaskItemEventReceiver calls [through SharePoint Business Connectivity Services] the Update method of External ContentType  Duet Enterprise Workflow Task;
6. SharePoint BCS calls for Duet Enterprise Workflow Task Update the wfUpdate method of SAP workflow consumer webservice; with input parameters workitem identifier, decision value and comments;
7. SAP workflow consumer webservice propagates task approval decision to the associated SAP workflow instance, and executes apply decision on the SAP workitem;
8. SAP workflow completes this workitem step, and proceeds to the next step in SAP workflow. If it was the last step in workflow, the SAP workflow is now completed;
9. TaskItemEventReceiver updates the status of SharePoint taskitem to completed. In default (generated) Duet Enterprise Workflow is this the only step/task, and the SharePoint workflow is completed. The SharePoint taskitem is removed from the users open tasks list.

User-centric Development Process for Duet Enterprise projects

In essence, a ‘Duet Enterprise custom scenario’ development project is not different from other application development projects. The customer – either the real end-user, or product management for package software -, has an idea about what to achieve and functional + non-functional (quality) requirements to underline the idea. Often this idea is still vague, and the requirement set far from complete and on aspects also contradictionary. The waterfall approach to yet start with application development (specify, build, test) typically ends up with wrong or failed end-results: the application does not live up to the real customer needs and expectations, and the budget is largely overrun as consequence of responding on the requirements changes. The agile movement came up to improve on this bad practice. Central is the structural involvement of the user (representative) during the full course of the development project. Continous, the user is in the lead about the specifics and the behaviour of the application functionality to be delivered. In Scrum, this responsibility is formalized within the product owner role.

Application mockup

A proven approach to enable the product owner to validate, correct, and ultimately confirm the application functionality is by make it experiencable, ‘touchable’. Provide the product owner with a realistic impression of the application behavour + feature(s) that the project will build: a clickable prototype / mockup.

Based on multiple positive experiences (also within non-Duet Enterprise project, e.g. BI Dashboards), we strongly believe in the added value and power of application mockup. We regard the role of UX / UID expert crucial for customer involvement, and thus for delivering the correct application functionality.

Meeting of 2 worlds

Yet, a Duet Enterprise project also is very much different from ‘normal’ projects. The cause lies with the meeting of 2 very different environments: IT technology, platform concepts, and human nature. The tradional SAP environment is transaction oriented, with focus on structural business process executions. The Microsoft environment, and in particular SharePoint, is far more loose. Ad-hoc processes are the norm, the individual user decides how to perform a specific (business) action. When these 2 worlds + thinking come together in a Duet Enterprise project, the first outcomes are often dramatic. SAP versus SharePoint architects, business analysts and developers do not understand each other (concepts). Both ‘parties’ disagree on how and where (that is, on SAP or on SharePoint side) custom development should be done. The personal preference on how to achieve the SAP+SharePoint integration is influenced by one’s own technology frames and familiair concepts. This has the risk that the user is forgotten: the user interests must be central, deliver an optimal user experience.

Align through Blueprint Duet Enterprise scenario’s

To break through this, we utilize in our Duet Enterprise development process a blueprint step with the focus exclusive on the Duet Enterprise specific scenarios. This blueprint is derived and agreed by a mixed group of SAP and SharePoint representatives. The goal of this step, and its endresult, is to come to a mutual understanding of the solution direction for each of the Duet Enterprise scenario’s. Self-employed constraints are to stick as close to the standards of both the SAP business package and SharePoint platform. Where feasible, we do allow custom development. Here we apply Duet Enterprise integration patterns that we have identified and defined: Consumer-oriented, Provider-oriented, and XML-Funneling pattern.

Of course, the discussion and preferences about where to make custom adjustments still manifest within the Duet Enterprise blueprinting step. However, as it now has the full focus of all involved, mutual agreement is earlier and more effective reached. This is the so-called ‘workshop effect’. Also, in this focussed approach, comparable application situations are easier to recognize. And where so identified, the earlier agreed solution approach can be reused. Starting point for the blueprint derivation is the User Interface Design as validated and confirmed by the product owner.

In the blueprint we derive and define per identified Duet Enterprise scenario the following pieces:

1. Global descripion of the scenario: screenshot of the mockup screen and textual explanation. The purpose is to facilitate on high level a common understanding of the functionality in the scenario. There is no intend to compete with or replace use cases.
2. Derivation, explanation and justification of solution direction. At minimal explanation of the proposed solution direction. But whenever applicable, also the rationale of why an alternative solution direction is dropped.
3. SAP / SharePoint interface
4. SAP Gateway datamodel(s)
5. SAP building blocks: standard RFC’s, BAPIs; identification of needed custom SAP building blocks
6. SharePoint building blocks: standard and custom webparts, SharePoint Service Applications, …

Blueprint validation by Design Authority / Authorities

Justified mantra nowadays within SAP and Microsoft IT departments is to avoid unnecessary or otherwise avoidable custom development. Instead stick close to the standard delivered functionalities of both landscapes.

It is also evident that fully excluding custom development is not feasible when you are developing company custom scenario’s.

We facilitate the consistency and continuity in the Duet Enterprise solution directions by confirming the blueprint to Duet Enterprise Rules, Conventions and Guidelines. The Design Authorities of both SAP and Microsoft IT departments validate the blueprint against these agreements. Derivation can be allowed, but requires the formal approvement of the respective Design Authority (SAP, Microsoft, and in some cases both).

SAP + SharePoint realizations based on the blueprint

With the blueprint accepted and delivered, from here on SAP and SharePoint teams in the project can start with their own development activities for the Duet Enterprise scenario’s. Since representatives of the teams were involved in deriving the blueprint, the concrete software design + development can be done relatively independent of the other party. Of course the data contract details of each SAP / Duet Enterprise interface are input for the SharePoint consumption. Regular sharing the SAP design with the SharePoint developers can achieve this.

Duet Enterprise Development Process

	Step	Responsible	Consulted	Approach	Input
1	Business analysis	Business Analyst	Business representatives	Interviews Workshops	Business vision
2	Mock-Up	UID / UX expert	Business representatives Business Analyst	Sketching Prototyping	Style guide
3	Blueprint	Duet Enterprise Solution Architect	SAP Business Analyst SharePoint lead	Workshop	UID Duet Enterprise Rules, Conventions, Guidelines
4	FoTo’s	SAP Business Analyst	SAP developer SharePoint developer Duet Enterprise Solution Architect		Blueprint Duet Enterprise Rules, Conventions, Guidelines
5	Build	SAP developer			Blueprint FoTo’s SAP Build – Rules, Conventions, Guidelines
		SharePoint developer	Duet Enterprise Solution Architect		UID Blueprint FoTo’s SharePoint Build – Rules, Conventions, Guidelines
6	Integrate	SAP developer + SharePoint developer			Operations – Rules, Conventions, Guidelines,