OData service support in Duet Enterprise - 2.0 and 1.0FP1 SP3

SAP NetWeaver Gateway 2.0 has within the SAP architecture the role of transparent integration layer to the SAP Business Suite:

Duet Enterprise is a consumer of the SAP Gateway layer, Sybase Unwired Platform is another consumer for Mobile devices. The access through Gateway is based on current interoperability standards: W3* (SOAP / WSDL) Web Services and OData services (REST protocol).
In the current Duet Enterprise 1.0 FP1 product OData support iswas missing. The direct explanation is that SharePoint 2010 BCS does not support OData consumption. For Duet Enterprise interoperability it is was therefore necessary to unlock SAP data via the SAP Gateway Generic Channel. Sybase Unwired Platform since version 2.1 does have OData consumption support.
From the market (partners, including The Next View, and customers) is urged to include OData support in Duet Enterprise. Main limitation is, however, SharePoint 2010 BCS.

In SharePoint 2013, BCS supports OData consumption. And Duet Enterprise 2.0 can therefore also support the interoperability via Gateway OData Channel. Integration via Generic Channel continues to be supported, in terms of design-time and runtime.

SAP Gateway includes generator tools for Generic and OData channel. Both are subject to the restriction that SAP hierarchical data structure is not supported (no mapping capability). In case of hierarchical data it is possible to 'handcraft' the Gateway service in ABAP code. The programming model for custom OData providers is thereby a lot simpler compared to that for Generic Channel services.
XML Funneling architectural pattern, via which we still have the opportunity to utilize the benefits of the Gateway generator tools.

SAP and Microsoft have recognized that (potential) Duet Enterprise 1.0 FP1 customers, now want to connect to the future OData option of Duet Enterprise 2.0. For example, to reuse the same Gateway service for Mobile channel and SharePoint / Duet Enterprise. To make this possible, with Duet Enterprise 1.0 SP03 (released today, September 14) an OData-SOAP Bridge is included. In short this Bridge maps OData-based output of Gateway OData services to SOAP services, which then integrate into SharePoint 2010 BCS. The OData-SOAP Bridge is restricted to 'flat' SAP data structure, data-retrieval of complex SAP structure to SharePoint is thus not possible.

In Duet Enterprise 2.0 the OData-SOAP Bridge is not included; SharePoint 2013 BCS itself can directly consume OData.

Summarized:

• Duet Enterprise 2.0 requires SharePoint 2013
• In Duet Enterprise 2.0 the OData + Generic channels are supported; SharePoint 2013 BCS can consume OData services
• For Duet Enterprise 1.0 a OData-SOAP bridge is available via SP03 to bridge Gateway OData services for consumption as SOAP web service in SharePoint 2010 BCS
  • Application of the Bridge is restricted to flattened SAP structure; in reality SAP data often has a hierarchical / complex structure (e.g. SAP LSO)
• SAP advice on the interoperability endpoint decision: OData Channel or Generic Channel?

User-centric Development Process for Duet Enterprise projects

In essence, a ‘Duet Enterprise custom scenario’ development project is not different from other application development projects. The customer – either the real end-user, or product management for package software -, has an idea about what to achieve and functional + non-functional (quality) requirements to underline the idea. Often this idea is still vague, and the requirement set far from complete and on aspects also contradictionary. The waterfall approach to yet start with application development (specify, build, test) typically ends up with wrong or failed end-results: the application does not live up to the real customer needs and expectations, and the budget is largely overrun as consequence of responding on the requirements changes. The agile movement came up to improve on this bad practice. Central is the structural involvement of the user (representative) during the full course of the development project. Continous, the user is in the lead about the specifics and the behaviour of the application functionality to be delivered. In Scrum, this responsibility is formalized within the product owner role.

Application mockup

A proven approach to enable the product owner to validate, correct, and ultimately confirm the application functionality is by make it experiencable, ‘touchable’. Provide the product owner with a realistic impression of the application behavour + feature(s) that the project will build: a clickable prototype / mockup.

Based on multiple positive experiences (also within non-Duet Enterprise project, e.g. BI Dashboards), we strongly believe in the added value and power of application mockup. We regard the role of UX / UID expert crucial for customer involvement, and thus for delivering the correct application functionality.

Meeting of 2 worlds

Yet, a Duet Enterprise project also is very much different from ‘normal’ projects. The cause lies with the meeting of 2 very different environments: IT technology, platform concepts, and human nature. The tradional SAP environment is transaction oriented, with focus on structural business process executions. The Microsoft environment, and in particular SharePoint, is far more loose. Ad-hoc processes are the norm, the individual user decides how to perform a specific (business) action. When these 2 worlds + thinking come together in a Duet Enterprise project, the first outcomes are often dramatic. SAP versus SharePoint architects, business analysts and developers do not understand each other (concepts). Both ‘parties’ disagree on how and where (that is, on SAP or on SharePoint side) custom development should be done. The personal preference on how to achieve the SAP+SharePoint integration is influenced by one’s own technology frames and familiair concepts. This has the risk that the user is forgotten: the user interests must be central, deliver an optimal user experience.

Align through Blueprint Duet Enterprise scenario’s

To break through this, we utilize in our Duet Enterprise development process a blueprint step with the focus exclusive on the Duet Enterprise specific scenarios. This blueprint is derived and agreed by a mixed group of SAP and SharePoint representatives. The goal of this step, and its endresult, is to come to a mutual understanding of the solution direction for each of the Duet Enterprise scenario’s. Self-employed constraints are to stick as close to the standards of both the SAP business package and SharePoint platform. Where feasible, we do allow custom development. Here we apply Duet Enterprise integration patterns that we have identified and defined: Consumer-oriented, Provider-oriented, and XML-Funneling pattern.

Of course, the discussion and preferences about where to make custom adjustments still manifest within the Duet Enterprise blueprinting step. However, as it now has the full focus of all involved, mutual agreement is earlier and more effective reached. This is the so-called ‘workshop effect’. Also, in this focussed approach, comparable application situations are easier to recognize. And where so identified, the earlier agreed solution approach can be reused. Starting point for the blueprint derivation is the User Interface Design as validated and confirmed by the product owner.

In the blueprint we derive and define per identified Duet Enterprise scenario the following pieces:

1. Global descripion of the scenario: screenshot of the mockup screen and textual explanation. The purpose is to facilitate on high level a common understanding of the functionality in the scenario. There is no intend to compete with or replace use cases.
2. Derivation, explanation and justification of solution direction. At minimal explanation of the proposed solution direction. But whenever applicable, also the rationale of why an alternative solution direction is dropped.
3. SAP / SharePoint interface
4. SAP Gateway datamodel(s)
5. SAP building blocks: standard RFC’s, BAPIs; identification of needed custom SAP building blocks
6. SharePoint building blocks: standard and custom webparts, SharePoint Service Applications, …

Blueprint validation by Design Authority / Authorities

Justified mantra nowadays within SAP and Microsoft IT departments is to avoid unnecessary or otherwise avoidable custom development. Instead stick close to the standard delivered functionalities of both landscapes.

It is also evident that fully excluding custom development is not feasible when you are developing company custom scenario’s.

We facilitate the consistency and continuity in the Duet Enterprise solution directions by confirming the blueprint to Duet Enterprise Rules, Conventions and Guidelines. The Design Authorities of both SAP and Microsoft IT departments validate the blueprint against these agreements. Derivation can be allowed, but requires the formal approvement of the respective Design Authority (SAP, Microsoft, and in some cases both).

SAP + SharePoint realizations based on the blueprint

With the blueprint accepted and delivered, from here on SAP and SharePoint teams in the project can start with their own development activities for the Duet Enterprise scenario’s. Since representatives of the teams were involved in deriving the blueprint, the concrete software design + development can be done relatively independent of the other party. Of course the data contract details of each SAP / Duet Enterprise interface are input for the SharePoint consumption. Regular sharing the SAP design with the SharePoint developers can achieve this.

Duet Enterprise Development Process

	Step	Responsible	Consulted	Approach	Input
1	Business analysis	Business Analyst	Business representatives	Interviews Workshops	Business vision
2	Mock-Up	UID / UX expert	Business representatives Business Analyst	Sketching Prototyping	Style guide
3	Blueprint	Duet Enterprise Solution Architect	SAP Business Analyst SharePoint lead	Workshop	UID Duet Enterprise Rules, Conventions, Guidelines
4	FoTo’s	SAP Business Analyst	SAP developer SharePoint developer Duet Enterprise Solution Architect		Blueprint Duet Enterprise Rules, Conventions, Guidelines
5	Build	SAP developer			Blueprint FoTo’s SAP Build – Rules, Conventions, Guidelines
		SharePoint developer	Duet Enterprise Solution Architect		UID Blueprint FoTo’s SharePoint Build – Rules, Conventions, Guidelines
6	Integrate	SAP developer + SharePoint developer			Operations – Rules, Conventions, Guidelines,

Deploy Duet Enterprise in production infra landscape

Duet Enterprise PoC in sandbox environments

Organizations that evaluate Duet Enterprise for the added value it brings typically do so via a proof of concept. Mostly, these PoC’s are executed in an isolated sandbox, a laboratory environment. Naturally also in the sandbox all the minimal prerequisites must be present in order to rightfully test the effect and operation of the SAP-SharePoint interoperability. Thus (an element of) SAP Business Suite, a server hosting SAP NetWeaver Gateway 2.0 with the Duet Enterprise SAP Add-On installed, and a server hosting SharePoint 2010 with the Duet Enterprise SharePoint Add-On installed. But for cost reasons, the sandbox infra is typical less extensive as in the regular production environment. Thus no load-balancer, SharePoint 2010 farm in the single-server setup, no proxy server, no firewall, et cetera. Infrastructure elements that are regurlarly present in nowadays mature infra architectures.

Note: to facilitate organizations that quickly want to inspect and evaluate Duet Enterprise, without spending effort in installing it within own landscape; the Duet Enterprise team has prepared a fully installed Microsoft Hyper-V image: SAP NetWeaver Duet Enterprise Trial System.

Deploy Duet Enterprise in production landscape

At a client organization we also started with a Duet Enterprise PoC, in a sandbox environment. On basis of the succesful outcome, Enterprise Architecture has mandaded the application of Duet Enterprise as the organization standard interoperability approach when it concerns SAP-SharePoint business scenarios. Next logical, and a required step, is to make Duet Enterprise available for generic utilization in the regular IT landscape. Via the Duet Enterprise wizard a lot of the Duet Enterprise deployment effort on SAP side is automated, largely transparent to the specifics of the SAP landscape. For SharePoint side, there is no wizard. But the installation effort there is far less as on the SAP side.

Duet Enterprise deploy differences sandbox versus production landscape

During deployment of Duet Enterprise in the production landscape, we experienced a number of effects resulting from differences in the production versus sandbox landscapes.

SAP side

• Starting from NetWeaver 7.31, the consumer proxies for Duet Enterprise workflow are configured via transaction SOAMANAGER. The proxies configuration in LPCONFIG, although still present and possible, no longer has effect.
• For design-time, the ESR Builder must be associated with the Software Component Version and namespace in the Gateway system where custom development will reside. It is not feasible in production landscape to create Gateway DataModel and business scenarios as Local Objects.

SharePoint side

• At the external boundary from end-user to SharePoint, the infrastructure contains a Load Balancer with SSL-offloading to the internal network. Duet Enterprise however requires SSL enabled on the SharePoint 2010 server that is in direct connection with the SAP NetWeaver Gateway server, hosting the Duet Enterprise SAP Add-on. And the SSL certificate of the SharePoint server must be imported within SAML2 transaction of SAP Gateway system.
• For shared resource usage, and for security, the SharePoint infra is partitioned over 2 separate SharePoint farms: a SharePoint Service Applications farm with also a Central Admin webapplication; and a consuming SharePoint farm that hosts all the end-user SharePoint webapplications. Also SharePoint BCS Service Application is thus on the Service Applications farm. As BCS is the SharePoint consuming side in Duet Enterprise architecture, it seems logical to exchange the Security Token Service (STS) certificate of this farm to Gateway; to configure the SAML based Duet Enterprise Single Sign-On behaviour (see Execution flow of Duet Enterprise Single Sign-On). However, when we configured the Duet Enterprise landscape with the STS certificate of the Service Applications farm, the Single Sign-On derivation at Duet Enterprise connection time failed. In SRT_UTIL the infamous error message ”SSFW_KRN_VERIFY failed with: verification failed, see decrypted digest” was logged. Further analysis of the logged information in SRT_UTIL displayed that the REMOTE_ADDRESS in the runtime SAML connection is that of the consuming farm. After consultation with SAP Support we tried to use the STS certificate of this farm instead. And indeed this resolved the invalid certificate signature issue. The explanation for this behaviour is that the end-user that starts the Duet Enterprise request flow is authenticated within the consuming farm; thus also the secure token identifying the user must be taken from this farm.

Filter on current datetime in BCS::ReadList upon Search Crawling

SharePoint Business Connectivity Services plus SharePoint Enterprise Search and/or SharePoint FAST is a powerful combination. With them you can unlock selective data from external repositories and bring into SharePoint context to realize search-driven applications. The external data is retrieved at Search Crawl time, and administrated in the search index file.

The SharePoint Indexing Engine first invokes the Finder method of the external content type(s) that is tagged as ‘RootFinder’ in the BDC Metadata Model. And second per returned entity the SpecificFinder to retrieve more details. If all required entity details are already returned by the Finder method, the latter can be skipped for better crawl performance and throughput.

Via the Finder method thus the set of relevant entities of external content type is retrieved. The decision what is relevant is made from perspective of the external repository upon the moment of invoking the Finder/ReadList method. A typical filter here is that of current datetime: only return active external entities.

Current datetime is a dynamic value to be determined at Finder invocation time. A consequence is that this filter value cannot be a priori included in the BDC model. This leaves 3 options: derive and specify the dynamic value from either the context of the BCS::Finder consumer, from within the BCS::Finder provider as integration layer, or from the data access operation (web service, ADO.NET or custom connector) within the external repository.

BCS::Finder consumer context

In content crawling context the SharePoint Indexing Engine is the consumer. It is not possible the augment its the standard behaviour of invoking the Finder method.

BCS::Finder provider context

This option requires that the Finder execution invoke the associated access point of the external repository with DateTime.Now as filter parameter. This option is not supported for BCS no-code solutions, but requires the utilization of a custom BCS .Net Assembly Connector. If already a custom .Net Connector is used for retrieving data from the external repository, it is probably a minor change to extend it with this dynamic derived filter value. But if not, introducing the usage of a dedicated BCS .Net connector merely to filter on the current DateTime is somewhat of an overweighted solution approach.

DataAccess operation within external repository context

So that leaves as only option to apply the current datetime filter value within the interoperability point of the external repository. For example in Duet Enterprise scenarios to unlock SAP data to SharePoint: The invoked Query operation of the Gateway webservice must then self apply the system datetime. SAP NetWeaver supports this via the concept of default value for input parameters.
Sadly it is not possible to specify this at the level of the Gateway Data Model, so it must be specified in the invoked ABAP Function Module (RFC).

Note: avoid specifying application-specific default value in the RFC. Systemtime satisfies this guideline, but a concrete default value for DateTime input parameter typically does not.

See also MSDN forum thread on this subject: Invoke BCS::ReadList operation with 'today' as default value for datetime input parameter.

Influence on Duet Enterprise product development + roadmap

The Next View engages in select international comitee with SAP on development and roadmap of Duet Enterprise

The Next View is regarded in the Netherlands as the Duet Enterprise expert. SAP itself also recognizes this. We are therefore SAP Partner on SAP / Microsoft interoperability. SAP Netherlands refers customers that seek for information and guidance on this subject to us, and actively engages us in Duet Enterprise consultations. The Duet Enterprise product team in Walldorf and Bangalore [very much] appreciates the feedback we give based on our experiences with the product. As such we are also asked to participate in the Duet Enterprise Customer Engagement Initiative (CEI). This is a select international group of partners and customers that take the lead with implementations of Duet Enterprise. Participating organizations come from Germany, Australia, England, Denmark, and The Next View as sole and exclusive Dutch partner.

The CEI concept is to provide input for desired developments in the Duet Enterprise product roadmap. Also the Duet Enterprise product team demonstrates (potential) new product features, and inputs new ideas to discuss with and be assessed by the CEI participants. As the disclosed information concerns potential future product features, all CEI participants prior have signed a Non-Disclosure Agreement (NDA).

The recent Duet Enterprise 2.0 Preview announcement proves that via the CEI concept we (The Next View) indeed exert influence on the development of the product. Our main criticism on Duet Enterprise 1.0 is the missing of OData interoperability support. The underlying SAP NetWeaver Gateway integration layer includes OData provider support since version 2.0. The bottleneck here is that SharePoint 2010 Business Connectivity Services does not include OData consumption support. But now using OData sources with BCS is added in the upcoming SharePoint 2013, the next version of Duet Enterprise can utilize this to consume SAP data and processes.

Duet Enterprise is real, effort is in it’s correct application

In the 1st week of May, The Next View participated in the second edition of the Duet Enterprise Velocity event. We already joined the first edition, and thus are familiar with the Velocity concept. As consequence, we could start immediate at the begin of the Velocity week on the realization of our customer’s scenario: Coöperative Learning.

Duet Enterprise is a given; real effort is in it's application AND understanding the business process + SAP business package

Noteworthy finding is that Duet Enterprise knowledge + experience is a necessity to successfully and productively apply this SAP-SharePoint interoperability middleware. However, this alone is not sufficient. Also required is that your customer has, or can reach, a clear vision of the application context they aim to achieve. In a normal project (with aim to deliver business value) this is of course the essential starting point. In our development approach we facilitate this by usage of mock-ups: discuss and derive in workshop(s) together with customer representatives the application functionality and behavior. The visualization format can be as simple and basic as sketches on paper, onto fully operational prototypes the customer can click through. SharePoint GUI + SharePoint Designer are very helpful for achieving the latter.

With the customer vision visualized in mock-up, the next step is to map this User Interface Design (UID) onto the combined SAP and Microsoft landscape. Here it is essential that you understand, and be aware of the behavior of the SAP business package building blocks.

An example to illustrate this

Part of Coöperative Learning scenario is to deliver Training Collaboration Workspace. Each workspace displays information about the training itself, about the participants, qualifications. SAP Enterprise Learning contains standard BAPI Function Modules to retrieve this information. It appears as simple as to identify the proper Function Module, and expose this via Duet Enterprise to SharePoint. But in reality the required insight in the SAP business package goes further. You must also be aware of the proper way of Function Module invocation, depending on the application context. For our Training Workspace, we apply the standard BAPI LSO_TRAINING_GET_DETAIL_C to retrieve training title, description and date. On day X this worked correct. Next morning, I received functional error ‘Course already passed’ from SAP LSO. It appeared as if the standard BAPI FM was not usable to retrieve data on elapsed training data, making it unusable for historical workspace context. Strange enough, a co-worker could still retrieve the training data. Our differences? He was enlisted as participant in this training, in contrary to me. And the standard Function Module responds on this current user context by only allow course participants to view the training data. That is, if you invoke the Function Module in it’s default modus ‘User_Role = Learner’.

When you overrule this for anonymous modus, you can successfully retrieve the training data also for elapsed dates, irrespective of the participation status of current user. A (proper) Duet Enterprise manner to do this is in the Gateway Model mapping.

Blueprint Duet Enterprise scenario's

The Next View recognizes the derivation and mapping from business vision via UID to the combination of SharePoint capabilities and SAP business package functionality + building blocks as essential step to deliver the correct application. We therefore incorporate this structural in our applied Duet Enterprise development process. As soon as we receive the initial business approvement on the UID translation of the business vision, next step in The Next View process is to deliver a Blueprint for the Duet Enterprise scenario’s that can be identified from the UID. Involved roles in delivering the Duet Enterprise Blueprint are a) Duet Enterprise Solution Architect as owner and facilator, b) SAP Business Analist with functional and high-level technical knowledge of the SAP Business Package, and c) SharePoint (Lead) Developer with knowledge of the SharePoint front-end capabilities. The purpose of the Duet Enterprise Blueprint is to derive and describe the high-level Solution Design; identifying the SAP and the SharePoint side. In the Solution Design, also fundamental design decisions on the SAP-SharePoint boundary are made and argumented. And when applicable, for alternative approaches explained why decided not to apply that.

In the process of derivng the Blueprint, it may also become clear that the requested UID cannot, or only with substantial development effort (= costs), by achieved by the SAP + SharePoint combination. Whenever such a consequence is identified, this is discussed with the UID expert and business. Is it feasible to alter the UID to accompany the standard SAP + SharePoint functionality? And if not from either functional or UID perspective, is the business willing to pay the extra development and also maintenance costs? Thus, the Blueprint derivation has an iterative effect and relation towards the UID and the application functionality.

Microsoft Trends 2012

At The Next View we regularly have 'TheNextKnowledge' (pizza)sessions to update each other on subjects that are or will become of interest for us. The subjects can be of pure technical nature (SAP Gateway, SharePoint 2010, Duet Enterprise), architectural (e.g. Gateway vs PI), functional (BI Dashboards).
Our latest session addressed innovation and trends as we (!) recognize them at our major technology suppliers. As 'true trend watchers', our statements are completely on personal notice, and on intention also sometimes 'bold'.

Update: On 18 June, Microsoft announced the Microsoft Surface Tablet. This is a new product, and different from the 'old' Microsoft Surface (Multi)Touch Table product referred in this trends overview.