GWPAM Rapid Deployment Service on SAP-Microsoft Unite site

Earlier I reported about our involvement in the new SAP product for direct availability of SAP data into Microsoft Office clients: SAP NetWeaver Gateway Productivity Accelerator for Microsoft, short GWPAM.

Based on the knowledge and experience that we have gained early on in applying GWPAM, we have worked out a Rapid Deployment Service for GWPAM. This RDS solution is recognized by the SAP Gateway product team, and published on the SAP-Microsoft Unite Partner Connection site.

In the Rapid Deployment Service, we combine our proven approach for successful implementation of SAP-Microsoft integration scenarios with GWPAM product knowledge. Central concepts in our approach are user-centric focus, UI-design via (preferable) clickable prototypes, integration architecture, and technical expertise with the Microsoft and SAP application platforms. Check out in case you're interested in easily bring SAP data into Microsoft based front-ends.

SAML2 Protocol requires SSL-enabled Duet Enterprise 1.0 endpoints

An organization that is deploying Duet Enterprise in the landscape, default has no SSL enabled in their internal network. They questioned to neither activate SSL on the Duet Enterprise SAP Add-On / SAP NetWeaver Gateway 2.0 system.

However, this request cannot be granted. Reason is the usage of SAML2 for Duet Enterprise 1.0 Single Sign-On between SharePoint 2010 and SAP NetWeaver Gateway system.

Source 1: Duet Enterprise Security Considerations

The user accounts that are used to access SharePoint Server 2010 and Microsoft Office 2010 suites clients cannot be used to access the information in SAP directly. The Duet Enterprise security architecture solves this issue by configuring the Microsoft Business Connectivity Services Windows Communications Foundation (WCF) connector that is included in SharePoint Server 2010. This WCF connector interacts with the Security Token Service in SharePoint Server 2010 and with SAP NetWeaver in the SAP system. The goal of this implementation is to map user identities in SharePoint Server 2010 to user accounts in the SAP system so that a user who logs on to the SharePoint Server 2010 Web site can have access to the external data that is stored in the SAP system without having to log on again in the SAP system.

Duet Enterprise 1.0 applies SAML2 with Symmetric Key for Endorsing Signature.

Source 2: STS Scenario with Symmetric Key for Endorsing Signature

With this scenario, the STS and the WS consumer negotiate a symmetric key. This is used for an endorsing signature for messages between the WS consumer and the WS provider. The WS consumer uses this endorsing signature to prove that it is in possession of the key that the STS signed.

In short (a.o. source 2 contains a complete outline of the SAML2 Protocol steps): in the SAML2 Protocol with Symmetric Key,

• the webservice consumer (in Duet Enterprise scenario: SharePoint BCS) authenticates the logged-on SharePoint user at the Identity Provider (in Duet Enterprise scenario: SharePoint STS),
• the Identity Provider grants a SAML security token for the SharePoint account, generates a short-lived key and signs this with the public key of the X.509 certificate of the webservice provider (in Duet Enterprise scenario: SAP NetWeaver Gateway),
• STS returns the SAML token + signed STS key as SAML assertion to SharePoint BCS as webservice consumer.
• SharePoint BCS adds the signed token as SAML assertion (Holder-of-Key, HoK assertion) to the header of the service request and sends the request to SAP NetWeaver Gateway as webservice provider.
• Gateway uses the private key of its own SSL certificate to decipher the token, and validate it as genuine coming from trusted asserting party.
• If so, the SAML:NameIdentifier in the service request is relied on, and applied via SAP NetWeaver User Mapping to automatically log on the SharePoint account as its mapped SAP named user.

The SAML2 protocol effectively prevents other message recipients, which do not posses the private key, are able to decipher and misuse the security token.

Both the SharePoint 2010 and SAP Gateway participants in the Duet Enterprise SSO handling, uphold to the SAML2 Protocol. This implies that SharePoint expects requires that Gateway as webservice provider exposes https-enabled service endpoints. In case not, on each runtime Duet Enterprise service request, SharePoint STS will throw exception from method System.ServiceModel.ClientCredentialsSecurityTokenManager.CreateServerX509TokenProvider, like: "System.InvalidOperationException: The service certificate is not provided for target 'http://<hostname>/sap/bc/srt/pm/.....' ", as it is not enabled to setup a valid SAML2 Protocol handling with the invoked Gateway relying party.

For this customer organization, as the SSL is only applied in the internal network to uphold the SAML2 Protocol handling between SAP Gateway system and SharePoint 2010 farm, there is no need for a (pricy + period-bound) signed SSL certificate from a Certificate Authority. A self-signed certificate suffices. Note that the Duet Enterprise Configuration Wizard creates and configures a self-signed SSL certificate in case the Gateway system is not yet SSL-enabled.

Tip: bypass WebProxy for BCS service application

Setting up a fresh Duet Enterprise landscape, I was confronted with an issue trying to import BDC Models from the SAP Gateway system into SharePoint BCS:

Application definition import failed. The following error occurred: Error loading url: "http://....". This normally happens when url does not point to a valid discovery document, or XSD schema.

Using Fiddler I detected that the problem cause is a "(407) Proxy Authentication Required" issue: "The ISA server requires authorization to fulfill the request. Access to proxy filter is denied." Although I did setup a rule in Windows CredentialsManager for automatic authentication against the web proxy, this is not picked up in the context of BCS service application as an autonomous running process. As it turns out, by default .NET web applications and services will attempt to use a proxy, even if it doesn’t need one.

So how then to resolve from this situation? Multiple approaches are possible here:

1. Explicitly set the Proxy Credentials for the BCS application process.
   It is not possible to set the proxy credentials direct in the web.config of 14hive\webservices\bdc. Instead you must use a 2-step delegation approach: refer in the web.config to a custom Proxy module implementation, and build the custom Proxy to explicitly set the proxy credentials:

   namespace ByPassProxyAuthentication
   {
       public class ByPassProxy : IWebProxy
       {
           public ICredentials Credentials
           {
               get { 
                   return new NetworkCredential(
                       "username", "password", "domain"); }
               set { }
           }
       }
   }
   

   <system.net>
       <defaultProxy enabled="true" useDefaultCredentials="false">
           <module type="ByPassProxyAuthentication.ByPassProxy, 
                  ByPassProxyAuthentication"/>
       </defaultProxy>
   </system.net>
   

2. Disable usage of (default)proxy altogether for the BCS application process.
   This is a viable approach in case the consumed external systems are all within the internal company network infra.

   <system.net>  
     <defaultProxy  
       enabled="false"  
       useDefaultCredentials="false"/>  
     </system.net>

3. Disable usage of (default)proxy for specific addresses for the BCS application process.

   <system.net>
       <defaultProxy>
           <bypasslist>
               <add address="[a-z]+\.contoso\.com" />
               <add address="192\.168\..*" />
               <add address="Netbios name of server" />
           </bypasslist>
       </defaultProxy>
   </system.net>
   

   The first bypasses the proxy for all servers in the contoso.com domain; the second bypasses the proxy for all servers whose IP addresses begin with 192.168. The third bypass entry is for the ServerName
4. Disable usage of proxy for specific address on system level.
   This is in fact the most simple approach, just disable proxy usage for certain url's for all processes on system level. That is also the potential disadvantage, it can be that it is not allowed to disable proxy usage for all processes.
   You disable the proxy via IE \ Internet Options \ Connections \ LAN Settings \ Advanced \ Proxy Server \ Exception <Do not use proxy server for addresses beginning with>.

Informational sources used:

• Configuring Proxy settings in .NET config file
• Web Service access + internal traffic + 407 Proxy Authentication Required
• Proxy Configuration issues with UPA in SharePoint 2010 /2013

Function of SharePointResourceUrl property in BDC model

Lately we had an issue with SharePoint BCS usage. In my analysis I noticed that the value of the ‘SharePointResourceUrl’ in some models was not pointing to our SharePoint farm. But as the role of ‘SharePointResourceUrl’ parameter in BDC Models is badly documented, I was not certain this was really the problem cause. Also an Internet search on ‘SharePointResourceUrl’ only returns a few (2) hits, but none useful to explain its purpose.

This week I via-via received an answer from Microsoft Support on the functionality of this BDC model parameter. And although we already found the cause of our initial problem (it was SSL + webdispatcher related, between the external system and SharePoint farm), I still consider it worthwhile to log that described functionality.

Response from Microsoft support:

The SharePointResourceUrl is used to determine where the STS (Security Token Service) is located, i.e.:

SharePointResourceUrl + "/_vti_bin/sts/spsecuritytokenservice.svc?wsdl"

I've only encountered problems with SharePointResourceUrl on client solutions with SharePoint BCS when users sync an External List offline to Outlook or SharePoint Workspace. In these situations SharePointResourceUrl is required in order for the client to locate the STS service.

Manage Trust entire SSL certificate structure to avoid SharePoint-farm internal SSL trust issues

SharePoint BCS has the central role to consume data from external systems. One of the supported consumption approaches is via SOAP webservices. In case the external systems is SSL protected, SharePoint BCS must trust the SSL certificate of the external system. This is achieved by importing the SSL certificate in Central Admin, Manage Trust.

Recently we faced a situation that despite we set SharePoint to trust the SSL certificate, still SSL issues were reported (ULS, EventLog): An operation failed because the following certificate has validation errors....

Mind you, although the SSL issues are logged as critical, SharePoint BCS is tolerant and still sets up the connection to the external system for data exchange. But of course it is an undesired situation, certain for a production environment, that system logs (ULS, event logs) are piling up with critical errors; even when the platform is tolerant for them.

Upon investigating the logged SSL error, I noticed something strange. It was not the SSL certificate of the consumed external system that was qualified as non-trusted. Instead it appeared to be the SSL certificate that the SharePoint farm uses internally for the service communication between the SharePoint webapplication process and SharePoint BCS service application.

With this insight that the problem was internal in the SharePoint farm, the cause was good to locate. In the SharePoint farm only the certificate on lowest level was imported into SharePoint Manage Farm. Thanks to this post, SharePoint Operations learned that actual the entire certificate structure/hierarchy upto the certificate root level must be added to Manage Trust. With that fixed, the critical although tolerated errors are no longer polluting the logs on production system.

Explanation + resolution of BCS "Cannot find any matching endpoint configuration"

Explanation

SharePoint BCS operates with external systems through the connection information in BDC Models, administrated in its metadata store. BDC Models can 1. be imported via Central Admin UI, Business Connectivity Services application; 2. provisioned via features; 3. and manually be created via SharePoint Designer.

When BDC administrates a new BDC Model, the BDC administration process reads the WcfMexDocumentUrl value from the LobSystem node in the model; and uses this to runtime retrieve from the WcfMexDocument (= Service Metadata) all the service endpoints. The received service endpoins in the WcfMexDocument are administrated in the internal WcfConnectionManager class.

When the BDC service application is invoked by a BDC client to interoperate with the external system, BDC uses the WcfEndpointAddress value from the LobSystemInstance node in the model as the service endpoint url. But it does not directly invoke the (WCF) service on that endpoint address. It first uses this endpoint address to retrieve additional endpoint configuration data (e.g. what character is used as wildcard in the external system) from the WcfConnectionManager class. Information that was derived when administrating and processing the BDC Model, and stored as key-value pairs with the key equal to a service endpoint address included in the service metadata / WcfMexDocument.

Now, in case the WcfEndpointAdress value in the LobSystemInstance node is not present as one of the service endpoints in the WcfMexDocument; BDC is not able to locate the additional endpoint configuration; and all BDC client invocations of this External ContentType will fault with the error message “Cannot find any matching endpoint configuration”.

Resolution

The above sketched situation originates from a mismatch in the BDC Model versus the WcfMexDocument. The fix is to make sure that the WcfEndPointAddress in the BDC Model, does have a match with one of the service endpoint addresses of the WcfMexDocument. This either means to adjust the WcfEndPointAddress value in the BDC Model to a correct and present vaue, and reimport the model. Or to modify the WcfMexDocumentUrl to include the WcfEndPointAddress value as one of the service endpoints. Note that as BDC builds up its internal administration in WcfConnectionManager on processing a BDC Model, also in the latter case the BDC Model must be re-imported despite that itself has not changed. Through reimport the situation in WcfConnectionManager will be reset and corrected.

Augment GWPAM AddIn project to consume JSON dataformat

GWPAM consumption of efficient JSON format only possible when Gateway service supports OData V3

Standard the GWPAM generated AddIns consume the Gateway REST OData service via the AtomPub dataformat, and not via the more data-efficient JSON format. The reason is that the used .Net WCF Data Services Client library is not capable of consuming JSON dataformat. Because I prefer to consume REST services via the mobile-friendly JSON dataformat, I set out to augment a default generated GWPAM Outlook AddIn project to consume via OData JSON. As the service consumption is within .Net code, this is mostly a Microsoft.Net tail. However, as it turns out that to be even able to consume JSON in the Microsoft WCF client library, an important prerequisite is imposed on the consumed REST service; it also extends to there.

JSON consumption via WCF Data Services Client Library

Initially, WCF Data Services Client library only supported consumption of REST services via AtomPub dataformat. As of release 5.1 it is also possible to consume JSON, but with the limitation that the JSON format must be OData V3. Prerequisite for the consumed service is thus that it must be able to provide its data in OData JSON V3 dataformat. For the older OData versions (V1, V2), it is not [yet] possible to consume JSON in WCF Data Services Client library.

Steps to augment to GWPAM project to consume JSON dataformat

First prepare your project to be able to handle the JSON consumption:

1. Install the latest WCF Data Services Client Library, at present this is 5.6.0. Installation is done via NuGet Package Manager, and must be applied to each GWPAM project in which you want to consume via JSON dataformat. The effect per project is that reference ‘Microsoft.Data.Services.Client’, version 5.6.0 is added; and that the (via GWPAM project template) already existing references ‘Microsoft.Data.EDM’, ‘Microsoft.Data.OData’, ‘System.Spatial’ are also upgraded to version 5.6.0.
2. If present, remove the reference ‘System.Data.Services.Client’ from the GWPAM project(s). Note: Visual Studio 2010 default installs with that older WCF Data Services Client library.
3. Install WCF Data Services 5.3.0 RTM Tools Installer in Visual Studio; also via NuGet.

Next setup the consuming code to consume the REST service via OData. That is, if the service supports it

4. Validate that the consumed service is able to return data in the required JSON dataformat, OData V3. Query for this the $metadata of the service, and validate that it contains “m:MaxDataServiceVersion='3.0'”
5. Generate an EDMModel for the service, via ‘Add Service Reference’ (iso ‘Add SAP Service Reference’). Open the generated service proxy, and change the accessibility of ‘GeneratedEdmModel’ to public
6. Edit the earlier generated SAP data services client code to consume the service via JSON dataformat
1. In the constructor, set to initialize for ‘System.Data.Services.Common.DataServiceProtocolVersion.V3’
2. Set the Format to link to the generated EDM Model: this.Format.LoadServiceModel = GeneratedEdmModel.GetInstance;
3. Set the Format to useJson: serviceContext.Format.UseJson()
4. Copy the implementation + usage of methods ‘ResolveTypeFromName’ and ‘ResolveNameFromType’ from the generated service proxy in step 5. Modify the code to correspond to the full name of the SAP service proxy.

5. Query the consumed REST service for JSON format; either via querystring param ‘$format=JSON’, or via Http Request Header ‘accept
   = application/json’

That's it

And that’s it. Now the GWPAM based Office AddIn is able to consume the REST Service via the more efficient JSON dataformat. Note that for the consumption itself it does not even has to a Gateway REST Service; it is also possible to consume REST services from other webservice platforms. But the GWPAM project templates and code generated are focussed on standard SAP NetWeaver Gateway services + data models; WFService as one evident example.