Development Approach with Duet Enterprise

This blog is, with some minor changes, earlier published on SAP Community Network Blogs

TopForce participated in the 2nd half of 2010 within the Duet Enterprise Rapid Deployment Program. Together with our RDP partner, we followed a well-chosen approach to evaluate this SAP – SharePoint integration product on it’s capabilities and potential.

In the Ramp-Up / RDP I was involved in plotting the Duet Enterprise prerequisites in the company IT infrastructure, selecting the PoC scenario, architecting the integration approach, and configuration + realization of the Duet Enterprise services for the PoC scenario. In this article I highlight the main points of our journey.

Product evaluation via a custom Proof-of-Concept

Both partners in the RDP regarded it as essential to evaluate Duet Enterprise as interoperability product via a custom Proof-of-Concept. Only such we are confronted wih actual experiences and issues in the application and potential of the Duet Enterprise capabilities. We do not suffice with merely a rather simple out-of-the-box scenario provided by Duet Enterprise.

Selection criteria for the PoC scenario

For us the main reason to consider Duet Enterprise is as integration foundation for company-specific SAP / Microsoft interoperability. To best evaluate the product capabilities on this aspect, we defined the selection criteria for the PoC scenario. We weighted the possible scenarios on the following aspects:

1. A real application context, with current and/or future purpose
2. Give input to SAP / Microsoft integration design guidelines

Validation approach

The PoC is intentionally approached as a regular development project. Thus functional specification, architectural and technical design, realization and testing, with ultimate ending in product implementation. In these Agile times, the phases are not necessarily in linear order; especially the integration architecture and realization have had some iterations as results of lessons learned.

Functional specification

In this phase, the application functionality is clearly defined, and agreed plus functional validated with the customer + end-users. Noticeable is that in this phase special attention is / should be given to the UX-factor: the User Experience of the application. This is because the most heard compliant against SAP is typically the lesser user friendliness. And also because nowadays information workers expect no less than a pleasant and familiar workspace, in which one is facilitated and helped to do your daily and also occasional work activities.

Integration and Software Architecture Design

Essential in this phase is close cooperation between the SAP and Microsoft solution architects. Both typically come from different backgrounds (‘different worlds’, almost like Mars versus Venus), yet it is required that they have a sufficient common understanding to architect a proper and future-proof design.

In the architecture itself, a best practice is to apply a layered service integration architecture. Each layer has its own responsibilities. In a nutshell, SharePoint sits at the front-end / presentation layer, SAP at the back-end layer; and Duet Enterprise is the glue as integration layer.

The integration layer is derived from the [use cases of the] functional specification. One of the design principles applied is that the SAP backend is and remains ultimate responsible for correctness of the data. This also implies that business rules are enforced and remain the responsibility of the SAP backend.

Duet Enterprise also imposes some constraints on the service interfaces. At conceptual level this means that Duet Enterprise [currently] only supports data oriented interfaces, operating via CRUD behavior. The explanation is that Duet Enterprise utilizes Business Connectivity Services to exchange data from SAP to and from SharePoint. BCS is a strict data oriented capability, it is not usable for [SAP] process oriented control.

Development Process

Once the service interfaces are derived and defined, the development team can go ahead with implementing the SAP-SharePoint integration via Duet Enterprise. The outline of this development process is as follows:

1. Configure / define the derived interfaces as SAP Enterprise Services in the ESR, using the SAP ES Builder. Hereby you must augment the interfaces as defined with 2 additional parameters required for proper Duet Enterprise handling, namely:
   • Correlation ID; this is used for the end-2-end monitoring of entire runtime flow through the SAP and Microsoft landscapes
   • Business Object Instance Key; this is used as generic identifier in all SCL framework components at runtime and at designtime. It consists of 3 components, Data Value Identier, SCL Business Object Name and the System Alias of the Backend. The key is to be generated in the SCL processing of each SCL service response, in the following structure: <data_value_identifier>_<Business Object Name>_<System Alias Of Backend>. The usage of this key is enable the SCL in subsequent handling to transparently identify the correct backend in a SAP landscape with possible multiple system instances. Notice the implication that if your SAP landscape consists of only a single backend, there is no concrete runtime usage of this field; all your SCL requests will be routed to the single backend. In that case you can simplify the key structure.
2. Export the defined SAP Enterprise Services in WSDL format
3. Hand over the WSDL file to both your SAP developer(s) as SharePoint developer(s). From here, both can in parallel configure and/or build their own side.

SAP NetWeaver 7.02 / Service Consumption Layer

1. In the SCL system, using ABAP Workbench / transaction SPROXY, create service proxy. This generates the skeleton of the provider class that includes the methods specified in the service interface. Later on you will implement the operations which involves some ABAP programming. The explanation of this break-up is that in the construction of the operations you need the GenIL model; and that you do not have yet generated in this stadium.
2. In the SCL system, in transaction se38 create the GenIL model based on the request and response structure of the service proxy. This generates a generic SCL Business Object Model. The reason that it is named ‘generic’ is because it is an intermediate mapping layer between the consumer (SharePoint BCS) specific data structures and the backend (e.g. SAP ECC 6.0) specific part. This enables that the consumer can transparently communicate with different backend systems/versions. That can be simultaneous in case of a distributed SAP landscape, consisting of a multitude of variant backends (e.g. ECC 6.0 for North America, while ECC 4.26 for the Dutch company division). But it is also beneficial when you upgrade your single system SAP environment – the GenIL model shields the client side from such backend environment change. And this also holds in the other direction – via the same GenIL model other clients as SharePoint can in theory also consume the exposed SAP data and functionality. An example is Alloy for IBM WebSphere.
   The GenIL model is actually to be more regarded as a generic Gateway derivative than strict Duet Enterprise specific
3. In the SCL via transaction SIMGH, create per service operation a Backend Operation Proxy that invokes the SAP Backend capability: either a SAP Enterprise Service, BAPI WebService or RFC / Function Module.
4. In the SCL, via transaction se80 create per service operation a Mapper object that implements 2 methods with predefined signatures for inbound and outbond mapping. This involves ABAP code to do the mapping from the GenIL model to the backend specific data format, and vice versa.
5. In the SCL / ABAP Workbench, return to the generated Service Proxy and fill in each of its generated operations. This also involves ABAP code to do the mapping from the received input to the GenIL model, and from the GenIL model to Duet Enterprise service output.
6. In the SCL via transaction SOAMANAGER, create the endpoint for the service proxy.

SharePoint 2010

1. Start SharePoint Designer to generate External ContentType(s) on basis of the imported WSDL from the SAP Enterprise Services.
2. If applicable, generate SharePoint External List as User Interface metaphore. Whether applicable is determined by a) the data structure; External List requires a flat structure (thus no hierarchical / parent-child relationships); b) the User Experience desired by the end-user.
3. if the External List is not applicable, realize in Visual Studio a custom build SharePoint webpart. Hereby you can use the whole richness of available ASP.Net and SharePoint webcontrols – datagrids, calendar, richtextbox, dropdownlist, … Also, SharePoint 2010 allows InfoPath forms for custom UI.
4. Configure the BCS model to connect for this External ContentType to the endpoint in the SCL of the Duet Enterprise specific service.

Conclusion

Introduction and application of Duet Enterprise is not a free ride. It takes time to get familiar with its concepts, and the development approach + tooling add-ons in the ABAP Workbench and SharePoint in the form of BCS Model Generator. It also requires preparation time at front in defining and architecting an application design. But that’s no different from any other software development project. Given that, Duet Enterprise is a noteworthy addition to the SAP / Microsoft interoperability toolbox; and I recommend to consider it for applicability in your specific SAP/MS integration scenario.

CQWP, CSS and jQuery applied to rearrange groups of overview links

A customer requirement for their SharePoint intranet is a reusable 'Overview-of-Handy-Links' capability, with the following specification:

1. Contributors must have a means to manage the overview links. The functional mananagment must be easy, without the need to maintain HTML.
2. The links must be grouped
3. The display-order of the overview groups must be controllable
4. The display-order of the links in a group must be controllable
5. The display space must be optimable used, white-spacing must be minimized
6. The overview backgroundcolor, links color and links hover color must be settable
7. The number of columns must be settable
8. Minimize the amount of custom code, favour the usage of standard SharePoint capabilities

As consequence of the first requirement, application of the ContentEditorWebPart is not an option. That would mean that the contributors are directly confronted with the html-specification of the overview. Utilization of the ContentQueryWebPart appears more promising. The rough solution design is then to 1) maintain the groupheaders in 1 SharePoint SPList; 2) maintain the overview links in another SharePoint SPList, with a Lookup to the GroupHeaders SPList; 3) Connect a ContentQueryWebPart to the OverviewLinks SPList; 4) Group on the GroupHeader Lookup field/column; 5) Codify in the CQWP Main, Group and Item xslt-files the templates to deliver the appropriate html.

The requirement to optimal use the display space can for a large extend be achieved via CSS float-functionality: automatically 'glue' the individual Group DIV's next to each other. In the horizontal direction there is then no lost white-spacing. However, in the vertical direction that cannot be achieved with CSS-float alone. Float flows in horizontal direction, not vertical.

But there is jQuery to the rescue. The basic idea is to first let the CSS-floating capability align the set of DIV's in as much as possible optimal usage of the display space. And afterwards, perform some minor rearrangement to also reduce the lost of white-spacing in vertical direction.

Via CQWP	After jQuery

Reusability of the OverviewLinks-functionality is achieved by applying the Feature capability. The Feature provisions the required SharePoint artifacts: the SharePoint SPLists - including the Lookup relation between them, a preconfigured CQWP, the XSLt-files.

Solution partitioning and dependencies with mixed scopes causes farm-deployment trouble

We probably all agree that it is wise to partition your SharePoint application deliverables and deployment into multiple packages. Typical examples are a Base assembly, deployed via a base wsp, providing a general foundation layer with diverse technical and functional building blocks. And then 1 or more functional packages, that provision the SharePoint artifacts and code for a specific application functionality; in which the functional building blocks may use and rely on parts provided via the Base assembly.

However, sometimes this architectural sane decision to partition your application deployment, can result in unforeseen effect. Let me demonstrate via a real-life example, that I encountered today:

• Application deployment deliverable
  1. includes a Base assembly, deployed via solution package Base.wsp. In this wsp package included the Base code, some basic Features, JobDefinitions, generic webparts. In the Base code / assembly there are FeatureReceivers, WebParts. As result of this, SharePoint solution generation (WSPBuilder or VS2010) puts SafeControl settings per contained WebPart in the Solution manifest; which again results in it that the scope of the Solution package is WebApplication.
  2. and also contains a pure functional package; with some Features, SiteTemplates. This functional package does not contain an assembly itself, but instead relies on the assembly of the Base.wsp for reused FeatureReceivers. As result this solution package does not contain any artifacts with are scoped at WebApplication level, and SharePoint solution framework requires to deploy it global in the farm. (nb: the further described situation can even occur in situation were the package does contain managed code, as long as the assembly does not contain SharePoint coded artifacts which transition the scope of the Solution package to WebApplication; e.g. no WebParts, Resources).
• The order of Solution package deployment is to first install the Base.wsp, and then the Functional.wsp. Then you would expect that the deployment dependency of Functional.wsp on Base.wsp is ascertained. However, in this scenario I discover in the SharePoint Solution store an error with the deployment of Functional.wsp. Error details:
  <server 1> : The solution was successfully deployed.
  <server 2> : The solution was successfully deployed.
  <server 3> : Feature '<GUID>' could not be installed because the loading of event receiver assembly "Base, Version=1.0.0.0, Culture=neutral, PublicKeyToken=13a12821e9f69237" failed: System.IO.FileNotFoundException: Could not load file or assembly 'Base, Version=1.0.0.0, Culture=neutral, PublicKeyToken=13a12821e9f69237' or one of its dependencies. The system cannot find the file specified.

At first, I thought this to be a timing issue. The Solution Framework uses timer jobs to deploy solutions throughout the farm, and it looked as if the Functional.wsp was tried to be already deployed on <server 3> while the deployment job for Base.wsp was not yet executed for <server 3>. However, also upon re-deployment of Functional.wsp - while making sure at forehand that the deployment of Base.wsp was now fully completed in the farm - , it resulted in the same error on <server 3>. So it must instead be directly related to the installation on <server 3> apparently the Base.dll was not installed in the GAC on that server; while the Features within Functional.wsp were to be installed on <server 3>. Why is this?

Well, the explanation is twofold. It relates to the SharePoint Farm infrastructure, as well as the scopes of the SharePoint packages. <server 1> and <server 2> have the SharePoint webapplication role in our load-balanced farm; and each contain the SharePoint webapplications. <server 3> however is dedicated as Search Index server; and does not host any SharePoint web applications. The SharePoint solution framework deploys WebApplication scoped packages only to the servers on which SharePoint web applications are hosted (Base.wsp: <server 1> + <server 2>). But it deploys Global scoped packages to ALL the servers in the farmer (Functional.wsp: <server 1>, <server 2> AND <server 3>).

Given this insight, a proper SharePoint Solution Deployment approach is to deploy the Global scoped Package with Force-flag, thus suppressing the error signalment. This worked sufficient in our case. However, it leaves you with the somewhat unwanted situation that via the Force-flag you suppress all error notifications; thus may be too brute. A better approach is to circumvent that Solution Framework deploys your Application package also to the servers on which the web application is not hosted. This can thus be achieved by making sure that each of your Application package as WebApplication scope. Either because it inherently contains such entities (e.g. WebParts), or artificially by including a dummy artifact directed at WebApplication level (e.g. an empty virtual bin file).