Thoughts on SharePoint Development

Impossible to use custom implementation of public BCS interface IEntityInstanceProvider

2012-02-04T21:33:00.025+01:00

SharePoint Business Connectivity Services contains out-of-the-box webparts to visualize external business data. All the standard BCS business data renderers utilize the pattern that data selection is delegated to a data provider control. The 2 controls are associated via consumer/provider connectionpoints. BusinessDataDetailsWebPart is one of them. Its standard usage is for BCS profile pages. However, usage of this webpart on itself is not necessarily limited to that context alone. The webpart is selectable in the webpart gallery, and can be added to arbitrair SharePoint webpart page. The challenge is how to connect it to an external BCS entity. The trivial approach is to do it similar as on profile pages; feed it via the provider BusinessDataItemBuilder. The working of BusinessDataItemBuilder depends on an ‘ID’ querystring parameter. However, when that parameter is included in the url of a page with the 2 BCS webparts on it, SharePoint faults with an error message:

System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.SharePoint.Publishing.Navigation.PortalSiteMapDataSource.OnInit(EventArgs e)

This occurs because PortalSiteMapDataSource also reacts on the presence of ‘ID’ querystring parameter, but assigns another and in this context incorrect meaning to the value. When the ‘ID’ querystring parameter is renamed, the error is prevented. However, then the standard BusinessDataItemBuilder no longer reacts on it.

A proper solution seemed to build a custom provider that relies on another named querystring parameter to retrieve the BCS entity item. The BCS architecture has the promise to support this: it heavily decouples via interfaces from the specific BCS implementation classes. But this appears to be a false promise. Microsoft already points you to that via the warning accompanying the public (!!) interface IEntityInstanceProvider

This class and its members are reserved for internal use and are not intended to be used in your code.

Still, since the interface is public; and also the connectionpoints are public; it sounded safe to expect it must work. But when I added both my own IEntityInstanceProvider implemenation class and a BusinessDataDetailsWebPart on a page; it was not possible to connect them; not in the GUI, and also not explicit in custom code. Via Reflector I reverse engineered the cause, to ultimately end up with the following:

Microsoft.SharePoint.WebPartPages.WebPartPageUserException was unhandled
Message=The connectionpoint BDWP Item on g_1e1e7d15_652d_44cd_bc7d_9026382ff33b is disabled.
Source=Microsoft.SharePoint
StackTrace:
at Microsoft.SharePoint.WebPartPages.SPWebPartManager.CanSPConnectWebPartsCore(WebPart provider, ProviderConnectionPoint providerConnectionPoint, WebPart consumer, ConsumerConnectionPoint consumerConnectionPoint, WebPartTransformer transformer, Boolean throwOnError)

and digging deeper:

if (!providerConnectionPoint.GetEnabled(connectionAgentByInterfaceName))
{
    if (throwOnError)
    {
        throw new WebPartPageUserException(WebPartPageResource.GetString("SPWebPartConnection_DisabledConnectionPoint", new object[] { providerConnectionPoint.ID, provider.ID }));
    }
    return false;
}

public override bool GetEnabled(Control control)
{
    bool flag = false;
    BusinessDataWebPart part = control as BusinessDataWebPart;
    if (part != null)
    {
        try
        {
            return part.CanConnect(true);
        }
        catch
        {
            return false;
        }
    }
    if (control is BusinessDataItemBuilder)
    {
        flag = true;
    }
    return flag;
}

Thus: although at the external front decoupled via interface IEntityInstanceProvider, internally BusinessDataDetailsWebPart demands it's provider to be either a BusinessDataWebPart or BusinessDataItemBuilder webpart!! So much for interface decoupling.
Confine to this internal constraint by inheriting either of these classes is also not possible: BusinessDataItemBuilder is sealed, and BusinessDataWebPart is public abstract, but with 4 internal abstract methods.

BusinessDataDetailsWebPart also allows to explicit connect it in code with a BCS entity. Less elegant as via the connectionpoint, but workable. The standard BusinessDataDetailsWebPart can then still be used to render the external data item; no need to custom build an own one (or ‘copy’).

BusinessDataDetailsWebPart busDetailsWp = wp as BusinessDataDetailsWebPart;
if (busDetailsWp.BdcEntity != null && busDetailsWp.BdcEntity.Equals(entity))
{
string entityInstanceId = ((IEntityInstanceProvider)this).GetEntityInstanceId();
busDetailsWp.ItemID = entityInstanceId;
}

Reasoning for Duet Enterprise in addition to SAP NetWeaver Gateway

2012-01-16T18:34:00.008+01:00

With the release of Gateway 2.0 as addition to the SAP NetWeaver stack, the question is justifiable whether this on itself is sufficient for SAP / Microsoft interoperability. Stated otherwise: ‘Do you need Duet Enterprise?’ The answer on that depends on your requirements, IT strategy, plans and situation. On level of desired application(s) functionalities as well as the Enterprise Architecture strategy, policy and plans in your company.

This blog is earlier published on SAP Community Network Blogs

Is Duet Enterprise required for SAP / Microsoft interoperability?

An answer to that question is: No, Duet Enterprise is not per se required, but it might be the wise decision. To make that decision, multiple aspects must be considered.

The extras Duet Enterprise brings to the table (compared to Gateway alone)
Your SAP / Microsoft roadmap and plans; multiple applications and a future-robust strategy, or just now for a single project?
Enterprise Architecture guidelines
Functional requirements
Limitations and implications of Duet Enterprise

Duet Enterprise extras to SAP Gateway for SAP / Microsoft interoperability

The extras fall into different categories.

Functional

Standard SharePoint rendering of the imported SAP data through SharePoint Business Data webparts
Standard CRUD+Q UI-process pattern on SAP data; for bidirectional data handling
Support / hook into SharePoint Enterprise Search
Collaboration workspace around SAP data entity: Customer, Product, Quote
Integration of SAP workflow into SharePoint context
Same for reporting: schedule + review SAP BW reports from SharePoint front-end

Connectivity

Single Sign-On from SharePoint into SAP
Propagation of SAP user authorizations into SharePoint roles + permissions
Propagation of SAP data authorizations to SharePoint Search context, via construction of ACL’s on the External Data entities
Transparant SharePoint (consumer) – SAP (provider) runtime connectivity via SharePoint BCS

Operations and lifecycle management

End-2-end monitoring; starting from SharePoint, via SCL, upto SAP backend
Audit trail per use case

Development

Design tool for generation of BCS Model on basis of Gateway Model (itself generated via Gateway generation tools)
Integration with SharePoint; Gateway is strict to provide standards-based access to SAP data, Duet Enterprise is also a first-class
SharePoint-aware user.

Roadmap + ecosystem

SAP and Microsoft have a roadmap outlined for Duet Enterprise
Base functionality provided by Duet Enterprise
product is augmented via the partner ecosystem (via Unite program)

Enterprise Architecture

The decision for Duet Enterprise implementation is not made in isolement. It must be considered and fit in the greater area of enterprise architecture. It is evident that Duet Enterprise only makes sense if your IT
landscape contains both SAP and Microsoft. But that alone is not sufficient to justify the purchase and implementation of the product Duet Enterprise. Does EA mandate usage of standards: interoperability protocols, products? And for
common-of-the-shelf products, prefer strategic suppliers Microsoft and SAP above third-party suppliers? These are just 2 examples of questions on which EA should make a thoughtful decision and statement, resulting in EA guidelines.

Functional requirements

The often required rationale for Duet Enterprise is actually a negativisme of the SAP (G)UI. So it is not so much that the decision is made for Microsoft / SharePoint, but more that it is decided against the 'detested' SAP user interface. But if the only motivation is to get rid of the SAP user interface and replace it by another, there are other possibilities. Build a custom user interface in SAP WebDynpro for instance.

Duet Enterprise starts to shine when you combine the powers of the SharePoint platform with the functionality of the SAP processing. Not merely rebuild the SAP user interface, but design and architect an improved user experience by integrating it within context of the information worker.

Limitations and implications of Duet Enterprise

Duet Enterprise is not a silver bullet for all SAP / Microsoft interoperability occassions. It has limitations, intended or as consequence of architectural desicisions. Also, implementation of Duet Enterprise has its costs.

Limitations of Duet Enterprise

No support of REST protocol
Limited to data-oriented, CRUD+Q service signatures
Design-time tooling support only for inside-out approach; not for outside-in
Full design-time tooling support only for flattened SAP data-structures
Standard SharePoint rendering only for flattened data structures. This is actually a limitation of the standard SharePoint BCS UI controls. Rendering of complex data structures can be done via custom build SharePoint UI-controls (example: Working with complex SAP business entities in Duet Enterprise).

Costs of Duet Enterprise

The costs fall apart in different categories:

Recurring license costs
One-time deployment and installation effort
Training of IT operations, architects and developers (for all: SAP and Microsoft)
Optional (but recommended for any except a simple SAP landscape): dedicated SAP Gateway application server
Duet Enterprise design time requires presence of SAP Enterprise Service Repository (ESR)

Conclusion

The decision for the implementation of Duet Enterprise must be made in perspective of the business requirements and middle-term enterprise architecture. If SAP / SharePoint policy is a fundamental part of the [future] business application and IT landscape, Duet Enterprise might very well be the sensible decision. Better as realizing + maintaining an own SAP / Microsoft interop layer (no matter how smart your IT people are, they will on longer term never win it from the sheer combination of SAP + Microsoft product team). And also better as purchasing + implementing third party products. That will result in more shattering in your IT landscape, while you must assure yourself of the continuity of the product and its supplier. Again here the combination SAP + Microsoft gives on the longer run more trust as that of smaller although dedicated suppliers.

Dynamic browser redirect with preserving UrlReferrer

2012-01-13T21:14:00.005+01:00

The standard way for a dynamic redirect executed from the browser context is via 'window.location = "<runtime derived url>";'. A side effect of this approach is that you have lost the UrlReferrer information after navigation to the second page. The browser namely treats this approach the same as if the user has explicitly entered the url in the location bar. There is no referrer since referrers are about one page directly referring to another (eg. via a link), not the browser just navigating to another page. If you need the UrlReferrer setting - a typical example is for (re)directing the browser to the first page after action performed on the second page -, a simple javascript trick can be used.

With this code setup, the dynamic derived redirect is added to the context of first page. As result, the UrlReferrer state is repaired and available for inquiry in the (webclient + webserver) context of second page.

User Experience improvement of OOTB SharePoint rendering of SAP data imported via Duet Enterprise

2012-01-09T04:52:00.012+01:00

Duet Enterprise makes it simpler to bring SAP data into the realm of SharePoint based front-end. At the SharePoint side the SAP data integration is enabled via Business Connectivity Services. The SharePoint platform provides multiple standard controls that hook into BCS Service Application to render external SAP data. XsltListViewWebPart to render a view on ExternalList, BusinessDataDetailsWebPart to display details of a single external SAP BCS entity. These standard controls have a generic ui-process handling. In case the SAP process handling is different, the result may in fact be a lesser user experience as that ‘detested’ SAP UI.

However, within limitations it is possible to extend and adapt the standard SharePoint ui-process handling, while still using and relying on the standard controls.

This blog is earlier published on SAP Community Network Blogs

The standard ExternalList UI-process handling is to display all selected fields in a gridview; and display details of a selected external entity in a popup window.

In case of a relational or hierarchical data model (typical SAP structures), it is more natural to navigate to another portal page with a composite view: display some details of the selected external entity, accompanied with table based overview(s) of related items. This modified UI-process can be achieved with standard SharePoint controls by a number of measures:

Use javascript/jQuery to overload the standard SharePoint client-side handling that is generated for (External)List. This can be done by utilizing the standard ContentEditorWebPart for injecting the jQuery in the SharePoint page, it does not require custom server control development + deployment.
Modify the Xslt-rendering of the standard WebParts to beatify the rendering. Examples of this are applying conditional rendering, other display-formats of data structures, html-layout changes. Also this can be done at SharePoint configuration level, from context of the SharePoint GUI. It does not require custom server controls. Note: Xslt programming is an expertise on its own. Although you can deploy it via SharePoint GUI, it is not something that you can expect the content owner or functional administrator to do.
Use CSS for consistent style of the rendered data displays. Again this can be done via SharePoint GUI; no deployment required to the SharePoint farm.
Readible names for SAP datafields by defining readible names in the BCS resource file per SAP external contenttype. This is something to be done by the SharePoint administrator that has access to BCS Service Application UI.
Build a custom server control to propagate value(s) of the selected SAP external entity as filter(s) to XsltListViewPart that exposes view on ExternalList of related SAP data entity. This does require some custom SharePoint development and a farm deployment. Afterwards this generic server control can be reused in multiple SharePoint pages.
Build a custom server control to provide to standard BusinessDataWebPart the entity identifier of the SAP entity that was selected in XsltListViewWebPart on the referrer page. This server control implements the BCS IEntityInstanceProvider interface. This does require some custom SharePoint development and a farm deployment. Afterwards this generic server control can be reused in multiple SharePoint pages.

The effect of these measures with still usage of OOTB SharePoint controls to display SAP data imported via Duet Enterprise:

Start: browse to the SharePoint page that initiates retrieval of SAP category data (via SAP NetWeaver Gateway Model Query method) + display within XsltListViewWebpart

User clicks on one of the categories, as result the browser navigates to a second page. On that page some details of the selected category are displayed (via BusinessDataDetailsWebPart, issuing via BCS the Gateway Model Read method), and also data of 2 additional SAP entities (via Gateway Model Query method,with one or more input parameters, the value of these set as BCS filter in ReadList method.

User clicks on one the related SAP entity instance, browser navigates to a third page. On that page some details of the selected instance are displayed (via BusinessDataDetailsWebPart, issuing via BCS the Gateway Model Read method), and again also data of additional SAP entity (via Gateway Model Query method).

Conclusion

Via such SharePoint customizations the UI-process handling for external data can be largely improved. In case the User Interaction Design demands more and deeper adjustments, somewhere the line will be crossed what is sensible possible via SharePoint customization. Then it is more sensible to build custom webparts for exposing the SAP data retrieved via Duet Enterprise. For the SAP side this is transparent, as long as it can deliver the required SAP data structures to compose the SharePoint UI-handling.

Synchronisation issue with SharePoint FBA claims-based

2011-12-10T20:30:00.016+01:00

In a SharePoint 2010 extranet I apply a custom membership provider for Forms-Based Authentication. The provider works like a charm, external users are authenticated when logging on with valid credentials, and denied access otherwise.

However, after functioning smoothly for a while, we suddenly encountered the error below when trying to logon via FBA:

[FaultException`1: The context has expired and can no longer be used. (Exception from HRESULT: 0x80090317)]
Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.ReadResponse(Message response) +1161205
Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst, RequestSecurityTokenResponse& rstr) +73
Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst) +36
Microsoft.SharePoint.SPSecurityContext.SecurityTokenForContext(Uri context, Boolean bearerToken, SecurityToken onBehalfOf, SecurityToken actAs, SecurityToken delegateTo) +26060225
Microsoft.SharePoint.SPSecurityContext.SecurityTokenForFormsAuthentication(Uri context, String membershipProviderName, String roleProviderName, String username, String password) +26063596
Microsoft.SharePoint.IdentityModel.Pages.FormsSignInPage.GetSecurityToken(Login formsSignInControl) +188
Microsoft.SharePoint.IdentityModel.Pages.FormsSignInPage.AuthenticateEventHandler(Object sender, AuthenticateEventArgs formAuthenticateEvent) +123
System.Web.UI.WebControls.Login.AttemptLogin() +152

Logon via Windows authentication had no problem, only the FBA route. Since there had been no software or configuration changes for the custom provider, the cause must be found to be at [SharePoint Farm] infra level. In the Application Eventlog I noticed the following Error log: An exception occurred when trying to issue security token: The context has expired and can no longer be used. (Exception from HRESULT: 0x80090317).

This steered me within the direction of the generic Security Token handling in the farm, instead of the context of the extranet webapplication self. As first attempt I decided to restart the SecureToken Service Application. And voila, this was already sufficient: problem resolved. That is, for a while... The problem namely structural reappears after a couple of days of minor or no activity in the SharePoint farm. It looks to me as something of a clock timer synchronization issue within the SharePoint farm, that can for a period be prevented by timely 'refreshing' the SecureToken application pool.

Exposing business information of SAP workflow via Duet Enterprise

2011-12-09T22:27:00.005+01:00

Often a SAP workflow includes specific business data. This data is needed at the decision moments within the workflow to make an informed decision. And thus also needed within the SharePoint context if the workflow decision step is exposed there via Duet Enterprise workflow handling.

As the default Duet Enterprise workflow handling is a generic set-up, it is evident that it cannot provide direct support for all imaginable and variant occurences of specific business data. Instead there are hooks in the workflow pipeline in which you can plug-in custom extensions for exchanging workflow specific/contextual data from SAP backend to SharePoint frontend. You need to build a custom workflow outbound handler to transfer the contextual SAP business data from SAP workflow to SharePoint via Duet Enterprise workflow feature. Herein you have 2 options.

Expose business data in the format of static HTML or XML content

In this approach the additional workflow information is added to the SAP workitem-details (SOSP_TT_WF_RUNTIME_INFO). The Duet Enterprise system dataflow is as follows:

SAP ERP	Outbound handler	Extend or replace the workitem details that are propagated via Duet Enterprise workflow capability, with pre-formatted business content (XML or HTML)
SCL	No action
SharePoint	Taskform	IF HTML: simple display IF XML: Xslt-transformation, preceeding display

Expose business data in the format of XML payload

In this approach the additional workflow information is passed via the XPROP key-value structure delivered with Duet Enterprise workflow handling at the SAP side. The Duet Enterprise system dataflow is as follows:

SAP ERP	Outbound handler	Populate the Duet Enterprise key-value XPROP structure with the required business properties (name => value; eg ‘CustomerId’ => ‘02345’) The business property values are retrieved from the SAP workflow container.
SCL	Configuration	Register EACH individual XPROP business property at the associated workflow template
SharePoint	Config	Register EACH individual XPROP business property as external/extended business property at the task definition in the Duet Enterprise workflow subsite
SharePoint	Workflow	Extend the ‘Approval task screen’ to render the additional business data properties [extend standard taskform via SharePoint Designer or via InfoPath]

Decide between the 2 approaches

Both options are viable to expose specific SAP business data to Microsoft SharePoint and Outlook context. Which is most appropriate depends on the characteristics of both the data, and what will be done with it within the Microsoft front-end. If it will only be displayed, it may be best to use the approach to expose it as single set of XML or HTML content; and then directly or via Xslt render within the UI. If the data is also needed within the front-end for additional processing, it is properly better to expose the data properties individual.

Workflow in Duet Enterprise: concept + implications

2011-11-22T21:17:00.013+01:00

One of the Duet Enterprise core capabilities is publishing SAP workflows into the realm of SharePoint sites. In this posting I outline the conceptual working, and what implications result from the Duet Enterprise workflow architecture. The information below is not entirely new, but builds upon blog material of Xiaosheng (Edward) Lu [SAP AG] and Kiki Shuxteau [Microsoft Corp].

This blog is earlier published on SAP Community Network Blogs

Prerequisites for exposing a SAP workflow via Duet Enterprise

SAP ERP workflow; it does not matter whether a standard workflow, or a company-custom developed one. SAP BPM workflows are currently not supported for Duet Enterpise exposition.
The SAP workflow must contain one or more discrete user interaction tasks.
Duet Enterprise out-of-the-box supports User Decision interaction step. Interaction task with an Activitity Dialog are also supported, but require some manual work. Effectively you need to replace both the SAP GUI screenhandling plus propagate the dialog resultdata as outcome for the SAP workflow Activity Dialog task.

source: Microsoft Technet

Conceptual working

The following steps are performed in the exposition of a SAP ERP workflow interaction task to SharePoint context:

In the SAP ERP workflow a status change occurs that next requires user interaction to act upon, via a SAP Workflow Interaction Task (Decision Step or Activity Dialog). When the SAP Workflow reaches the Interation Task, a Duet Enterprise specific outbound handler is invoked from within the runtime context of SAP workflow execution. This outbound handler uploads an XML payload document, containing the workflow instance details, to the associated [by configuration] SharePoint workflow subsite. In the Duet Enterprise architecture, this XML payload is send from SAP ERP Workflow context via the SAP Document Publisher, which resides on the Duet Enterprise SAP Add-on. The SAP Document Publisher service on its turn invokes the SharePoint OBAWorkflowService, which is deployed in the SharePoint farm as part of Duet Enterprise installation.
On the receiving SharePoint document library [in the workflow subsite] is a Duet Enterprise specific SharePoint workflow configured. This workflow is triggered on addition of new content in the document library. The workflow contains 1 user task to act from the context of SharePoint user interface upon the specific surfaced SAP interaction task.
The decision made by the SharePoint enduser in the SharePoint workflow task, is propagated to the SAP ERP workflow by means of a taskflow BDC entity. Duet Enterprise specific SAP webservices are herefore invoked via SharePoint BCS. These SAP webservices next invoke a Duet Enterprise specific inbound handler to receive and process the exchanged decision details, and propagate it as outcome of the user interaction task in the SAP ERP workflow.

source: Microsoft Technet

Implications

Duet Enterprise provides default Function Modules for both inbound and outbound handlers. By design, these are limited to generic handling of SAP workflow. In case the SAP ERP workflow decision task includes context information that is workflow specific, it is required to build a custom outbound handler. Also in case the result datacontainer of the decision task contains context data besides or instead of a simple decision outcome, it is required to build a custom inbound handler to process that received data from SharePoint task into the SAP workflow.
- In concrete reality, SAP workflows more often than not involve acting upon workflow specific data. In all such situations it is thus required to overload the default function module with a custom implementation, for surfacing that specific data outside the boundaries of the SAP workflow.
The default Duet Enterprise outbound handler S_OSP_WF_PAT_DEFAULT_CH_OB handles the publishing of XML payload to SharePoint. Even in case of a custom outbound handler, it must therefore still be utilized in the Duet Enterprise workflow publication. More specifically, the S_OSP_WF_PAT_DEFAULT_CH_OB outbound handler must always be the last invoked outbound handler in the SAP workflow outbound handlers pipeline. The order in which outbound handlers are invoked is configured within SAP ERP, not in the SCL (aka Gateway).
The default Duet Enterprise inbound handler S_OSP_WF_PAT_DEFAULT_CH_IB handles the receiving of Duet Enterprise taskflow entity from SharePoint BCS. It must therefore always be present as the first invoked inbound handler in the SAP workflow inbound handlers pipeline.

Alternative ways to programmatic read contents of External List with filtered view

2011-11-19T16:35:00.021+01:00

In order to export the displayed contents of a BCS External List to Excel (see previous post), I first have to programmatically retrieve the contents in custom code. The External List has a filter applied to it's Finder method, set via the default SPView:

My first thought was therefore that below code should retrieve the same filtered external data as when rendering the BCS External List on a SharePoint page:

However, GetItems returns an empty collection. Upon debugging the called GetNotifications webservice method, I discovered that the filter-param always has value ‘null’. To me unclear why, since it has been set in the DefaultView.

Via internet search I ended up with an approach to retrieve the contents directly by invoking the BCS API:

However, although this approach works; it left me rather dissatisfied. Conceptually I want to export the contents of the same (External) List that I already have provisioned on a SharePoint page; so why should I have to dive under the BCS hood to get the same contents? Also, this code is very much aware / coupled to BCS API, while the 'Export SPList to Excel' functionality on itself is general. Thus, with some ample time available, I decided to analyze the way in which the standard XsltViewWebPart is issuing the external data retrieval – using JetBrains DotPeek reflector tool (as .NET Reflector is no longer free of licensee charge). It appears that this is done slightly different as my original attempt:

Not only is the code LOC of this far less, but also this code is general; applicable for both regular SharePoint lists as BCS External Lists.

Excel 2010 Protected View hinders browser-opening of downloaded .xlsx file

2011-11-18T08:46:00.017+01:00

An user requirement in one of our SharePoint 2010 projects is to export at any moment the displayed contents of an External List (with content originating from SAP ERP, retrieved via SharePoint BCS connecting to BAPI based web services) to an offline file. The functional rationale is version-administration for history and auditing purposes. The SharePoint platform supports this out-of-the-box for regular Lists, by the Export into Excel functionality. However, not so for BCS External Lists. But you can realize it yourself via some custom code. First retrieve the External List contents, and next construct a .xlsx file via Open XML SDK. The .xlsx file is generated server-side in memory, and send to the browser as HttpResponse content. The end-user can next either open the file, or save it somewhere at client-side:

Strange thing I noticed was that when saving the file, that saved file can next be opened successfully. But when instead choose to directly open the file, Excel 2010 displays the error message “The file is corrupt and cannot be opened”.

This must be a client-side issue; the server-side is not aware of the context in which the client-side handles the received HttpResponse (Note: via Fiddler I even analyzed that the HttpResponse contents were identical).

The resolution is hinted at in the File Download window, by the trust-warning about internet downloaded files. The default Excel 2010 TrustSettings are to distrust all downloaded content from non-trusted locations. To validate this I unchecked in Excel 2010 the default settings (via File \ Options \ TrustCenter \ TrustCenter Settings \ Protected View):

This helps, Excel 2010 now direct opens the downloaded file.

Inconvenient configuration of Forms-Based Authentication in SharePoint 2010

2011-11-10T16:16:00.043+01:00

For an extranet we aim to utilize Forms-Based Authentication to authenticate external users. In SharePoint 2010 this means that you have to apply the Claims-Based authentication model. And next set up the SharePoint webapplication configuration for the Membership provider that will be used for FBA. This is where things can be a little confusing. In fact you have to configure FBA membership on 3 different locations, and it is essential that all 3 are in sync:

1. In Central Admin

Configure Web Application \ Authentication Providers; here you specify the name of the Membership- and RoleProviders used in FBA context within the web application

2. Security Token context

In the SharePoint 2010 service applications architecture, membership handling is delegated to the SecurityToken service application. This is a major difference wrt SharePoint 2007 architecture, in which the individual webapplication process themselves handle the membership handling. Direct consequence for configuration is that you need to specify in the web.config of the SecurityToken service application all the membershipproviders and roleproviders that are used in the SharePoint farm.

The SecurityToken service application directory is located at: 14hive\WebServices\SecurityToken

3. Webapplication context

Finally, despite that membership handling is within SharePoint 2010 done via the independent SecurityToken service application; you may still need to add the membershipprovider to the web.config of the individual webapplication. This is required in case you want to be able to use the standard PeoplePicker for selecting credentials via the FBA membershipprovider. Besides adding the ‘membershipprovider’ node, you then also have to set the peoplepicker directing to that membership provider.

What if the 3 locations are not in sync? I evaluated the different inconsistent configurations:

1. Name entered in Central Admin does not match with name in SecureToken’s web.config

In this case, SharePoint Identity handling cannot get an handle to the configured MembershipProvider. When someone tries to log in via Forms-Based Authentication, SharePoint Identity handling will report the displayed exception.

Note, to have the 'Cannot get Membership Provider' exception details displayed you already need to make a change to the default settings within SecurityToken web.config. Namely set attribute includeExceptionDetailInFaults of ServiceDebug to true. Without this, only a general error is displayed: The server was unable to process the request due to an internal error

2. Missing MembershipProvider in Central Admin

Well, you cannot actually forget to fill them in if you selected ‘Enable Forms Based Authentication’. But you could per accident forget to select that option.

The result of this is more a functional error: the forms-based authentication is now missing as option to logon to the web application.

3. Missing or different named MembershipProvider in web application’s web.config

In this case you can still logon to the webapplication: either via Windows Authentication or via FBA. See above, membership is namely handled by SecurityToken service application, and not by the webapplication self. This can be a bit confusing at first. The result of the configuration error is noticed within the application self, when you try to search credentials from the FBA-Membership provider via the PeoplePicker. Strange enough the PeoplePicker is aware of the configured FBA-membership, but apparently cannot include it in it’s search space.

4. Mismatch PeoplePicker (incorrect) versus named MembershipProvider in webapplication’s web.config

In this case strangely enough, PeoplePicker is able to use the Membership provider, that is if it is consistently named wrt SecurityToken web.config (otherwise issue 1).

At first I could not detect any PeoplePicker malfunction as result of this configuration mismatch. I could still find all credentials in the membershipprovider, also via wildcarding.

Only when I on purpose sabotaged the wildcarding, I could see the effect.

So it appears that for wildcard search the ‘%’ is already somewhere set as default for all Membership providers; and that you only have to override this in case your Membership provider uses a different wildcard-pattern.

Personally, I find it better to always be explicit; and thus also explicitly specify ‘%’ as wildcharacter for each Membership provider that is valid for your application. But it is optional, and others might differ with me on this…

Programatically open SPSite using Windows Credentials

2011-11-04T21:52:00.008+01:00

In a Proof of Concept I employ a SPListMembershipProvider for forms-based access into (sub)sites. For the PoC, a SPList in the rootweb is utilized as User Administration. In SharePoint 2010 architecture, Claims-Based authentication is handled by SecureToken service application. In the local development image, the STS service application may run under the same Application Pool account as the SharePoint webapplication. But this is not recommended, thus not to be expected within a real farm setup. As result, it is not possible to directly access from within runtime STS context the list in the external SPSite.

SPSecurity.RunWithElevatedPrivileges cannot help here; that only be used within the same application pool context. Instead the proper way is to open the SPSite in the external SPWebApplication via the credentials of a SPUser in that site; e.g. that of a service account. Problem is that the SharePoint API does not directly provide a way to open a SPSite with Windows Credentials. You can open a site under the credentials of a SharePoint user, but you need the SPUserToken of the user for this. And guess what, you can only determine that token when within the context of the site. Talking about a chicken-egg situation.

However I came up with a manner to get out of this loop. It consists of a 2-steps approach: first programmatically impersonate under the credentials of the service account, open the site, determine the SPUserToken of the site’s SystemAccount, and undo the impersonation; second apply the SPUserToken to (re)open the site under the authorization of the site’s SystemAccount. Since Windows Impersonation is a resource intensive operation, cache the SPUserToken in memory so that the impersonation is only initially required within the process lifetime.

internal static SPUserToken SystemTokenOfSite(Guid siteId)
{
    string account, pw, domain;
    RetrieveCredentialsFromSecureStore(<AppId>, out domain, out account, out pw);

    ImpersonationHelper _impersonator = new ImpersonationHelper(account, domain, pw);
    try {
        _impersonator.Impersonate();

        SPSite initialAccessIntoSite = new SPSite(siteId);
        return initialAccessIntoSite.SystemAccount.UserToken;
    } finally {
        _impersonator.Undo();
    }
}

...
if (sysToken == null)
{
    sysToken = SecurityUtils.SystemTokenOfSite(memberSitesToId[websiteIdent]);
}

SPSite site = new SPSite(memberSitesToId[websiteIdent], sysToken);

Data architecture considerations and guidelines for retrieving SAP data via Duet Enterprise into SharePoint

2011-10-24T21:27:00.011+02:00

In Duet Enterprise FP1 it is relatively easy to generate a Gateway Model, as long as your SAP data sources live up to the required constraints. The challenge and intellectual work transfers to the data architecture, for deciding on and defining the proper data service interfaces given the application context.

This blog is earlier published on SAP Community Network Blogs

Real-time retrieval versus Search indexing

Bringing SAP data into SharePoint based front-ends can take different forms. You can unlock the SAP data real-time via ExternalList, to display and even edit the SAP data via the familiar SharePoint List UI metaphor. If the row-based List format doesn’t fit, you can develop a custom SharePoint webpart. You can also access the SAP data via the extensive SharePoint Search architecture and capabilities. With the improved Enterprise Search in SharePoint 2010 [or FAST if also purchased by customer organization], more and more new applications will be architected as search-driven. In context of SAP data think of searching from SharePoint context for a certain Supplier administrated in PLM, a Customer in CRM, et cetera.

Duet Enterprise can facilitate all of the above scenarios, in combination with the strengths of the SAP and SharePoint landscapes. However, the different nature of these scenarios may well result into different approaches for the data integration architecture. In the case of real-time retrieval of SAP data to render into SharePoint, it is advisable to limit the amount of data retrieved per SharePoint - Gateway - SAP backend interoperability flow. Don’t put unnecessary SAP data on the line that is not being used in the front-end: only include the fields of the SAP data that will be displayed in the UI and are relevant in this application context for the end-user. Search however has a different context. It is feasible to search on any of the field data of the SAP entity. This requires that all that SAP field data must be indexed upon SharePoint Search crawling time. So here it is advisable to not limit the amount of SAP data retrieved, but instead retrieve as much of the SAP data that expresses some functional value.

What if the same SAP data is to be unlocked both via SharePoint ExternalList, as via SharePoint Enterprise Search? The data integration architectures of these are thus contra dictionary: reduce the retrieved SAP data versus give me all. Well, nothing prohibits you from constructing multiple data integration pipelines, tuned for the different scenarios. For Search, it’s Gateway Model returns in the Query method all fields that contain functional value. And for the real-time ExternalList, that Gateway Model returns in the Query method only those fields that will be rendered as list columns.

In Duet Enterprise 1.0, constructing the Gateway Model takes considerable time. This is an obstacle for constructing multiple Gateway Models with different data representation / signature, as it can easily double your Gateway mapping effort and time. Luckily in Duet Enterprise FP1 it is relatively easy to generate a Gateway Model, as long as your SAP landscape data sources live up to the required constraints.

Complex SAP data source structure

If the SAP data is conceptually a flat structure, it is sufficient to have a single Gateway Model to retrieve the data into SharePoint context. However, we all know that SAP (or, business) data is typically of a more complex structure: hierarchical with multiple child entities. When you want to retrieve such a structure into SharePoint, you basically have 2 options.

The first is to flatten the structure. This approach suffers from some disadvantages. In case of multiple occurrences of the same child type (e.g. Ordered Item); how many of them should you include in the flattened representation? All, or an arbitrary limited number? Also, for the end-user a flattened structure can be conceptually wrong and counter-intuitive: Ordered Item information is of different functional level as the Purchase Order information.

The second option is to maintain the hierarchical SAP structure within the Gateway Model architecture. For this you need to generate a Gateway Model for the parent SAP data entity, as for each type of its child data entities. At SharePoint client-side you associate the 2 resulting External ContentTypes, to establish the parent-child relation. SharePoint BCS respects the association between the External ContentTypes when it operates on the data. In case of SharePoint search, the child-associations are crawled in the context of the parent data, and either the parent or child SAP data entity will appear in search results. The BCS Profile page of a parent data entity renders also the data of its child entities.

In case of real-time retrieval, you can apply the BCS Business Data webparts: add to the SharePoint page both a Business Data WebPart for the parent entity and a Business Data Related List WebPart for the child data entities. The latter will display the child SAP data entities of the parent SAP data entity that is selected in the Business Data List web part.

Mind you, the user experience of this setup is not always intuitive. In such case, it can prove better to build an own custom presentation. An example of this is outlined in Working with complex SAP business entities in Duet Enterprise.

Unlocking SAP data via Duet Enterprise Feature Pack 1 in more agile approach

2011-10-16T11:00:00.007+02:00

This blog is earlier published on SAP Community Network Blogs

A major bottleneck in applying Duet Enterprise 1.0 is the time it takes the SAP + Microsoft development team to unlock SAP data via Gateway to SharePoint. You have to define the service interface in ES Builder, create a proxy in transaction SPROXY, and next via transaction SE38 realize the GenIL model. The latter requires a lot of handcrafting ABAP code to do the mapping from Gateway runtime context and data representation to SAP backend, and vice versa. Code that follows a pattern, so a good candidate for code generation. This was a major pain point experienced by us when initially applying Duet Enterprise 1.0 within the Ramp-Up in 2010, and with emphasis reported back to the Duet Enterprise product team.

In the coming Feature Pack 1 version, the Duet Enterprise product team has evidently got the message. FP1 comes with multiple generator tools to ease and speed-up the realization of the internal Gateway Model (new name for GenIL model) for your application scenarios. Handcrafting is largely eliminated and replaced by full-automatic generation of first the mapping code to unlock the SAP data via NetWeaver Gateway 2.0, and next the required SAP Gateway service proxy, plus the SharePoint BDC model. The latter can be handed over to SharePoint side to import the External ContentType definition into Business Connectivity Services. Something that took us with version 1.0 several days to set things up at both SAP as SharePoint side, now is done in matters of minutes. Very appreciated side-effect of this is that it enables agile development: if the initial Model does not fit the requested application context, just change the mapping in the tooling and regenerate. With handcrafting approach you would loose a lot of elapse time here rearranging and testing your own mapping code.

Of course not all now suddenly comes for free. Before you start the Gateway Model generation, you still first must think about the data integration architecture to achieve your application scenario. An action that involves and requires consensus of both the SharePoint and SAP backend architects plus developers. The usage of the FP1 / Gateway 2.0 generation tools itself put some constraints on the SAP backend data sources; BOR, RFC or Dynpro Screens. Basically it comes down to it that the backend data entities must provide at minimum both a ‘Query’ and ‘ReadItem’ operation, and also ‘Create/Update/Delete’ operations for update scenario via SharePoint.

What if the available SAP backend entities do not self satisfy the requested integration pattern and/or the generation constraints? Even then, it is far easier and better manageable to realize a custom wrapper RFC within the ERP level that does satisfy the pattern + constraints, than apply the manual Gateway Model code crafting. Spoken from own experiences…

Read content of uploaded file within ItemAdding method

2011-10-07T22:29:00.010+02:00

In an application it is required to validate the file content before allowing the upload into a document library. SharePoint 2010 enables this via the ItemAdding synchronous SPItemEventReceiver method. Problem is however that you cannot access the uploaded file through the SPItemEventProperties.ListItem object. The item is at runtime of ItemAdding not yet created in and thus not available via the list. Via blogpost Getting file content from the ItemAdding method of SPItemEventReceiver I found a code-snippet how to access the file. Last remaining issue was that I received empty result upon reading from the filestream. Debugging I discovered that the Stream position was set to the end of file. Which is logical since the file has been read in order to save it to the content database. By resetting the position I can read the file contents, validate it, and cancel the upload in case of invalid content

public override void ItemAdding(SPItemEventProperties properties)
{
  string searchForFileName = Path.GetFileName(properties.BeforeUrl);
  HttpFileCollection collection = _context.Request.Files;
  for (int i = 0; i < collection.Count; i++)
  {
    HttpPostedFile postedFile = collection[i];
    if (searchForFileName.Equals(
      Path.GetFileName(postedFile.FileName), StringComparison.OrdinalIgnoreCase))
    {
      Stream fileStream = postedFile.InputStream;
      fileStream.Position = 0;
      byte[] fileContents = new byte[postedFile.ContentLength];
      fileStream.Read(fileContents, 0, postedFile.ContentLength);
      ...

VNSG event 'SAP en Microsoft - het beste van 2 werelden'

2011-09-29T08:44:00.007+02:00

De VNSG focusgroep User Experience organiseert op 13 oktober een bijeenkomst met als onderwerp SAP - Microsoft integratie/samenwerking. Bij veel bedrijven zijn zowel Microsoft als SAP belangrijke leveranciers van (bedrijfskritische) software en wordt er continue gezocht naar mogelijkheden om het beste uit beide werelden te halen. Waar SAP vooral bekend staat om haar solide bedrijfsproces ondersteuning, is Microsoft vooral sterk in de User Experience.

De aard van het evenement is om SAP en Microsoft solution architecten van eindorganisaties met elkaar in gesprek te brengen, en te leren van elkaar. De sessies zijn hierop geselecteerd. Klantorganisaties die bereid zijn gevonden te vertellen over hun ervaringen, best practices en huidige status op SAP / Microsoft integratie vlak.

De volgende klantorganisaties geven een presentatie:

Achmea - Remco Jorna: Enterprise Architectuur aanpak
Tata Steel - Hans Brouwer en Tamas Szirtes: An Enterprise Portal Journey...
Eneco - Kees Voeten: HR futureproof bij Eneco
Ziut - Frank Voortman: Duet Enterprise voor SAP/Microsoft integratielaag in aannemersportaal

Volledige agenda.

Event details

Datum: 13 oktober 2011
Locatie: Microsoft, Schiphol-Rijk
Eventduur: middag
Doelgroep: Solution architecten van eindklanten (profit en non-profit)

Organisatie:
Marcel Rabe, secretaris VNSG Focusgroep User Experience
Pim de Wit, kernteam lid
William van Strien, kernteam lid
Gijs Leurs, kernteam lid

Aanmelden voor dit event is ook open voor niet-VNSG leden: aanmeldingsformulier.

On-the-fly content-targetting via custom AudienceManager

2011-09-03T23:16:00.009+02:00

In a SharePoint project, one of the requirements is to enable the end-user to set preferences, which are then immediately be applied in the portal. One of the settable preferences is the (non)interest in subjects/functions: if interested, webparts will be visible; if non-interest specified, the webpart will be invisible. Well, this kinda sounds like SharePoint audiences. As this is out-of-the-box SharePoint functionality, it is applicable for both our own custom webparts as well as standard and third-party webparts. The only problem is the sub-requirement to have the preferences immediately applied in the portal. Standard SharePoint audiencing requires to compile the Audience before it takes effect, which either occurs on scheduled basis or on request. But never immediately.

The solution for this is to utilize a custom PreferencesAudienceManager, with functionality to on-the-fly derive Audience membership on basis of the selected settings.

System architecture

PreferencesAudienceManager code-extract

public class PreferencesAudienceManager : IRuntimeFilter2
{
  ...
  public bool CheckRuntimeRender(string IsIncludedFilter)
  {
    bool render = false;
    if (SPContext.Current != null && SPContext.Current.Web.CurrentUser != null)
    {
      // Priority to OOTB SharePoint Audiencing.
      render = AudienceManager.IsCurrentUserInAudienceOf(IsIncludedFilter, true);
      if (!render)
      {
        string[] globalAudienceIDs = null;
        string[] dlDistinguishedNames = null;
        string[] sharePointGroupNames = null;
        int num = AudienceManager.GetAudienceIDsFromText(IsIncludedFilter, out globalAudienceIDs, out dlDistinguishedNames, out sharePointGroupNames);
        if (num > 0 && globalAudienceIDs != null && globalAudienceIDs.Length > 0)
        {
          string userIdent = PersonalizationUtils.GetCurrentUserIdentifier;
          UserPreferences userPreferences = PreferencesRepository.ReadUserPreferences(userIdent);

          SPServiceContext context = SPServiceContext.GetContext(SPContext.Current.Site);
          AudienceManager am = new AudienceManager(context);
          foreach (string audienceId in globalAudienceIDs)
          {
            Guid g = new Guid(audienceId);
            Audience a = am.Audiences[g];

            string preferenceAudienceName = a.AudienceName;
            UserPreferences preferenceAudience = (UserPreferences)Enum.Parse(typeof(UserPreferences), preferenceAudienceName );
          if ((preferenceAudience & userPreferences) == preferenceAudience)
          {
            render = true;
            break;
          }
        }
      }
    }
  }
  return render;
}

There is on caveat you need to be aware of; the ability to even use Audiences functionality, requires that the User Profile Service Application is installed in the farm. Microsoft namely placed the Audience functionality within the context of User Profile; even though you can thus apply Audiencing without dependency of User Profile values.

ListTemplate cached in Sandboxed process

2011-08-23T20:01:00.006+02:00

Sandboxed solution are a great addition to SharePoint 2010 capabilities whenever it is required to deploy own customizations. However, be aware of the restrictions, and some oddities. Today I encountered such an unexpected behaviour. ListTemplates are one of the SharePoint artefacts that allow themselves to be deployed via a Sandboxed solution. In the initial version of a custom ListTemplate, some errors and omissions were made with respect to schema.xml and some forms. I corrected these, and deployed (= uploaded + activated) the Sandboxed solution. To discover next that my changes were not present; not within the Feature-provisioned ListInstance based on the ListTemplate, nor within a manually created FormsLibrary instance based on the custom ListTemplate. It took an AppPool recycle to have the latest deployed ListTemplate definition become effective in the sitecollection. Tip: always recycle the AppPool before re-deploying a Sandboxed solution.

HowTo omit the standard xml-namespaces in WCF MQ requests

2011-08-17T21:50:00.015+02:00

Earlier I reported that we encountered an issue with communication via IBM MQ WCF Channel over a clustered IBM WebSphere MQ-queue. The problem was already known within IBM and the fixed software ready on the shelves, and naturally we were very willing to be a first beta-tester. With this IBM fix, the EndPointNotFound problem was indeed solved. Sadly, we next ran into another issue. This time it manifested itself in our own service implementation at the receiving side. Upon receiving a MQ-request, the service responded with a technical error. Due the holiday season it took some elapse time before the right people were available and able to investigate the cause. The problem analyse led to the suspicion that the presence of xml-namespaces in the webrequest was unexpected, and therefore the service refused the request:

Both namespaces are default included when serializing a System.ServiceModel.Channels.Message instance from inside WCF client processing. You can however prevent this by implementing an subclass of System.ServiceModel.Channel.Body.Writer, and override the OnWriteBodyContents method. Next inject this method within the WCF Channel:

    System.ServiceModel.Channels.Message message =
        Message.CreateMessage(
            MessageVersion.None,
            string.Empty,
            new XmlBodyWriter<TRrequest>(request));

    var responseMessage = webmqclient.Request(message);

...

public class XmlBodyWriter<TBody> : BodyWriter
{
    private readonly TBody request;

    public XmlBodyWriter(TBody request) : base(true)
    {
        this.request = request;
    }

    protected override void OnWriteBodyContents(XmlDictionaryWriter writer)
    {
        WriteBodyContents(SerializeRequest(), writer);
    }

    private static void WriteBodyContents(StringBuilder output, XmlWriter writer)
    {
        using (var reader = new StringReader(output.ToString()))
        {
           using (XmlReader xmlReader = XmlReader.Create(reader))
            {
                writer.WriteNode(xmlReader, true);
            }
        }
     }

    private StringBuilder SerializeRequest()
    {
        var output = new StringBuilder();

        using (var xmlWriter = XmlWriter.Create(output,
            new XmlWriterSettings { OmitXmlDeclaration = true, Encoding = Encoding.UTF8 }))
        {

            var serializer = new XmlSerializer(typeof(TBody));

            // Service does not expect xml-namespacing; including the standard
            // xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            // xmlns:xsd="http://www.w3.org/2001/XMLSchema"
            // Omit these from the serialized request
            XmlSerializerNamespaces ns = new XmlSerializerNamespaces();
            ns.Add("", "");
            serializer.Serialize(xmlWriter, request, ns);
        }

        return output;
    }
}

With this modification to the serialization processing of the WCF MQ request, the service at the receiving WebSphere MQ side now replies with a correct functional response.

Deleted SPWeb in RecycleBin can obstruct deactivation of Sandboxed Solution

2011-08-15T20:27:00.010+02:00

The other day, upon deactivating a Sandboxed Solution in our test farm, SharePoint aborted on it with the message:

Cannot access web-scoped feature {GUID} because it has references to a site with id {GUID}.
System.ArgumentException: Value does not fall within the expected range.
at Microsoft.SharePoint.SPWebCollection.get_Item(Guid id)
at Microsoft.SharePoint.SPFeatureEnumeratorBase.GetCachedWeb(SPSite site, Guid webId, Guid featureId)

We earlier encountered this problem in the development farm. Thorough investigation by SharePoint operations together with development led to the conclusion that the SharePoint content database had reached a corrupt state due deletion of a SPWeb. On that SPWeb a Feature provisioned via the Sandboxed Solution had been activated. And now after deletion of that SPWeb, apparently a lock internal in the SharePoint content database was present obstructing disablement of that Feature. And since it is not recommended - and certainly unsupported - to manually alter a SharePoint content database, there seemed no other realistic approach to get out of this erroneous situation as by recreating and reprovisioning the entire sitecollection.

For the development instance, this was an acceptable pragmatic solution. However, in our test environment a lot of content is already created by end-users. Just throwing away the SPSite and replacing it by a brand new, is not viable. The only remaining solution seemed to afterwards restore the backed-up content into the new SPSite. Not undoable, but it will take time [to setup and execute the site content restore; and next to validate the correctness and completeness of it]. And moreover, it will result in a loss of trust by our end-users and customer on the robustness of SharePoint 2010 as application platform. If it happens once [actually twice], what guarantee is there it will not happen again?

Luckily, then I had a smart thought while discussing the problem symptons with a co-developer. If the problem appeared to be caused by the deletion of a SPWeb, would it then help to restore this SPWeb instance from the RecycleBin? Worth a try. And guess what: it did!! After the earlier deleted SPWeb had been restored from the recyclebin, the earlier activated Sandboxed Solution could next successfully be deactivated.

Misleading SecureStore message with InfoPath Forms Services

2011-08-12T22:26:00.010+02:00

In current project we publish an InfoPath 2010 formtemplate to SharePoint 2010 sitecollection. In the formtemplate, multiple controls are populated with data retrieved from a single SharePoint list via application of owssvr.dll and filtering views on that list. The SharePoint WebApplication is set up as Claims-Based. A consequence is that from within InfoPath Forms Services context it is not possible to authenticate via owssvr.dll webservice to the SharePoint list. The proper solution for this is to retrieve the data via Universal Data Connections, and let each UDCX authenticate itself to SharePoint. Either by explicit authentication; that is including the credentials in all UDCX files. But preferably by using the SecureStore; so that the credentials are maintained in a single location and not readable included in the UDCX files.

So far for the theory. In real practice, we encountered a problem with this setup: our formtemplate failed to retrieve the filtered data. In the ULS following message was logged per owssvr.dll/XmlQuery data connection upon opening the formtemplate in browser:

InfoPath Forms Services Maintenance 82lm Information Delegation was attempted for Secure Store application APPL_InfoPathService. (User: 0#.w|domain\useraccount, IP: , Request: http://appl.dev.hosting.corp/Pages/orderform.aspx)

This seems to indicate as if the individual logged-in user has no permission to access/use the SecureStore ApplicationID. Inquiry with Operations refuted that suspicion: the SSS ApplicationID was configured for 'All Authenticated Users'. The actual cause appeared that the service account configured in the SSS ApplicationID no longer had permission/read access to the SharePoint site collection. Instead of above message, I would have preferred a direct 401 or AccessDenied cause-indication in the ULS...

Publication on 'Succesful disclosure of SAP data and processes in SharePoint'

2011-08-03T01:00:00.002+02:00

In the May-edition of the Dutch Software Development Network (SDN) magazine I published an article co-authored with Marcel Kempers on the why, and approach for how succesful disclose SAP data + processing within a SharePoint based solution. As of this month, the publication is also online available. Mind you; it is in Dutch...

Personalization approach / settings administrated outside content database

2011-07-26T10:09:00.001+02:00

Personalization is an important functional aspect in my current SharePoint 2010 project. Personalization on itself is a phrase that can have multiple meanings and appearances. Examples:

Customize a page for your own preferences; via personal settings of webpart properties
Content targeting; on basis of the visitors profile

SharePoint enables page personalization via the Provider model, inherited from ASP.net. Implication of the standard SharePoint personalization approach is that each authenticated user gets an own copy per personalized page in the content database. The personalized settings are thus administrated in the content database, similar as in case of shared webpart properties. A disadvantage of this approach is that this makes it very difficult to (web content) manage the page. A content manager can update the (template) page, but none of the content changes are automatically propagated to the individual personalization page copies in the content database.

In our application scenario, we have the additional functional requirement that it must be viable to extract management information reports of the personalized settings: what type of user (profiles) typically select personalized setting X, choose to close webpart Y etcetera. Although not impossible, it is rather impractical to extract this management information when the personalized data is stored in the content database. The schema of the SharePoint content database must be treated as internal black box, and cannot be relied on. The schema also does not support ad-hoc querying for answering varying management info requests.

We also have User Experience related requirements: the users must be able to personalize, without being aware or be directly confronted with SharePoint. It should be intuitive and natural, without explicit [noticable] personalization-setting modus. Examples of requested personalizations are tuning the webpart behavior via settings, and dynamically reposition the webparts in the page.

These combined end-user and management requirements effectively rank out the utilization of standard SharePoint personalization. So what’s a viable alternative? The one we came up consists of the following elements:

[Standard] shared webpart properties for enabling the content managers to set the initial and default values for personalizable settings. The shared value is regularly stored in the SharePoint content database, and applied for each visitor which has not [yet] explicitly personalized the shared setting.
Storage of personalization settings in an own SQL Server database
The iGoogle-like reposition behavior via jQuery and webpart zones tagged as droppable zones
An abstract base PersonalizationWebPart; that handles both the server-side aspects of personalization (retrieving + applying personalization settings, as well as saving them), as the client-side (webpart movement); and that implements virtual methods for concrete subclasses to hook into.
Custom AudienceProvider to derive on-the-fly whether visitor is within a certain audience by checking setting stored in the own SQL Server database [thus no need for User Profiles, nor compiling of Audiences]

Extra and a major advantage of this personalization approach is that there no longer originates a copy of the page being personalized. If the content manager updates the page, by adding or removing a webpart, modifying content on a publishing page; these changes are automatic and immediate effective for all the visitors, whether one has personalized the page or not.

Make AudienceFilter webpart setting visible without User Profile Service Application started

2011-07-13T23:29:00.011+02:00

In current project we aim to apply SharePoint's audiencing mechanisme for content targetting. As audience filters we'll use SharePoint group membership, we will not derive the audience from User Profile properties. In our current state of SharePoint farm, the User Profile service application is even not available .

One of the advantages of SharePoint audiencing is that it is out-of-the-box available for every webpart; standard SharePoint and custom webparts. However, when we intended to apply an audience filter to a standard ContentEditorWebPart in our test-environment, we were confronted with a missing Audiencing setting in the webpart toolpart. Search on the web points to prerequiste of the User Profile service application being activated. However, this is neither possible in our farm infrastructure planning, nor needed for the manner in which our application will use audiencing, that is on basis of SPGroup memberships.

Via reverse engineering SharePoint code [using .NET Reflector] I found out what directly determines the visiblity of the AudienceFilter webpart property. The responsible AdvancedToolpart class does a check on the web.config for config property "SharePoint/RuntimeFilter": if the property is not present, or the referred assembly is not valid, none of the webparts in this SharePoint webapplication will display the AudienceFilter webpart property in their toolpart. I validated this by outcommenting in web.config the property; for instance the toolpart of a ContentEditorWebPart then misses the AudienceFilter setting. After reinstating the web.config property, and reopening the toolpart; the webpart property is visible again. And thus available for usage for filters on basis of SharePoint groups, or a custom AudienceProvider.

IBM WebSphere MQ WCF Channel has problem with clustered Queues

2011-07-08T18:12:00.011+02:00

The Enterprise Architecture roadmap of customer aims at a SAP-unless policy, and SharePoint for all web-applications/front-ends. Currently there are still multiple applications in their IT landscape which do not obey do this future direction. For some of them, e.g. J2EE based service application, the enterprise architecture integration guidelines prescribes that the client-services communication occurs via IBM WebSphere MQ (WMQ). From the .NET client perspective, WCF based communication is nowadays preferred and proven technology, and must be applied for any new web-applications that invoke other applications.

For .NET client context there are in reality 2 viable WCF Channel options for putting requests as messages on an IBM WebSphere Queue (WCF WMQ). One is part of Microsoft Host Integration Server, the other one is provided by IBM.

For reasons explained else, the.NET and Integration architects decided to apply the IBM WebSphere MQ Channel for WCF. It is successful applied in several .NET applications to put messages on WMQ. In my current application project we also need to invoke services of a ‘legacy’ system, callable over WMQ. However, when this application puts a message on the assigned Queue via Request-Reply, we ran into the following exception:

System.ServiceModel.EndpointNotFoundException was unhandled by user code
Message=WCFCH0309E: An error occurred while attempting to open the channel for endpoint 'jms:/queue?destination=APPL.REQ.QUEUE@XXXX01&connectionFactory=connectQueueManager(XXXX01)clientChannel(WMQ.Channel.TST)clientConnection(dev-xx.xxx.com(7201))&initialContextFactory=com.ibm.mq.jms.Nojndi&persistence=1&replyDestination=APPL.RPL.QUEUE' The operation could not be completed. The endpoint may be down, unavailable, or unreachable, review the linked exception for further details.
Source=mscorlib
StackTrace:
Server stack trace:
at IBM.XMS.WCF.XmsChannelHelper.ThrowCommsException(OperationType op, Exception innerException, String endpointURI)
at IBM.XMS.WCF.XmsChannelHelper.CheckExceptionAndTimeout(OperationType op, Exception e, String timeout, String endpointURI)
at IBM.XMS.WCF.XmsRequestChannel.OnEndOpen(IAsyncResult result)
at IBM.XMS.WCF.XmsRequestChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Request(Message message, TimeSpan timeout)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.ServiceModel.Channels.IRequestChannel.Request(Message message)
at Application.<CustomCode>.Client.<InvocateAMethod>(MethodRequest_1_0 request)
at System.Web.UI.WebControls.Button.OnClick(EventArgs e)
at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)
at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument)
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
InnerException: IBM.XMS.IllegalStateException
Message=Failed to open MQ queue APPL.REQ.QUEUE. XMS attempted to perform an MQOPEN, but WebSphere MQ reported an error. Use the linked exception to determine the cause of this error. Check that the specified queue and queue manager are defined correctly.
Source=IBM.XMS.Client.WMQ
ErrorCode=XMSWMQ2008

It took me several hours and diverse try-outs together with an MQ developer at the receiving WebSphere MQ side, to ultimately find out the following. The IBM WCF transport channel for WMQ (WebSphere MQ 7.0.1 installation) has a problem with clustered queues, and errors when attempting to open the queue. In case of putting messages to a non-clustered queue – the WMQ infrastructure of the other .NET applications -, the channel opens successfully and messages request + replies are delivered. The problem is reproduced outside the context of our SharePoint application, via a simple testprogram; thus appears to be structural. We are currently addressing IBM to analyze it, and come up with a structural solution. For now, the pragmatic solution appears to communicate via non-clustered Queue.

Using ServiceLocator in Sandbox requires pre-generated xml-serialization

2011-07-05T21:20:00.025+02:00

The SharePoint Guidance [SPG] ServiceLocator provides you with a ready-to-use implementation of a Dependency Injection container. However, be aware of the following restriction when applying the SPG ServiceLocator in a constrained Sandbox context: it is not possible to compile on-the-fly generated xml-serialization classes, which are required for administrating the service interface and implementation within the servicelocator typemappings.

The execution of code statement:

typeMappings.RegisterTypeMapping<IServiceContract, ServiceImplementation>();

resulted in the following exception inside ServiceLocator code:

Microsoft.Practices.SharePoint.Common.Configuration.
ConfigurationException: Error on serializing configuration data ---> Microsoft.SharePoint.UserCode.SPUserCodeSolutionProxiedException: Error on serializing configuration data ---> Microsoft.SharePoint.UserCode.SPUserCodeSolutionProxiedException: Cannot execute a program. The command being executed was "C:\Windows\Microsoft.NET\Framework64\v2.0.50727\csc.exe" /noconfig /fullpaths @"C:\Users\SPadministrator\AppData\Local\Temp\OICE_925B8D58-F6D1-4BB3-9E01-FCBB9D816D0B.0\wkdrltlt.cmdline".

Explanation of this behavior:

SPG ServiceLocator applies XML serialization to administer the typemappings for the service interface and service implementation class. .NET XML serialization approach supports JIT compilation of the required xml-serialization classes. The benefit is that you do not need to generate and deploy these xml-serialization classes, the XmlSerializer framework generates them on spot. However, to be able to do this assembly generation; the .NET compiler is needed. And when within the Sandboxed constraints, you are not allowed to use filesystem resources.
By doing a little ‘digging’ in the SPG runtime pipeline, I discovered that not even my own custom classes caused the runtime serialization error; it actually could be derived back to the following serialization:

new System.Xml.Serialization.XmlSerializer(typeof(
Microsoft.Practices.SharePoint.Common.ServiceLocation.ServiceLocationConfigData))

The solution is to make sure that the required xml-serialization classes are already available: either in the GAC for generic use, or also deployed in the sandbox. I opted in this particular case to deploy Microsoft.Practices.SharePoint.Common.XmlSerialization.dll in the Sandbox.

Working with complex SAP business entities in Duet Enterprise

2011-06-19T11:44:00.039+02:00

Duet Enterprise natively only supports flattened SAP data entities. Typically however business data exhibits a complex structure. Via custom programming against the SharePoint BCS ObjectModel it is possible to integrate complex SAP business entities via Duet Enterprise in SharePoint.

This blog is earlier published on SAP Community Network Blogs

A sales talk of Duet Enterprise is that you can integrate with minimal effort SAP data and functionality in SharePoint UI. This however is only true if your SAP data complies to the following constraints: 1) the SAP data entities must at least be made accessible via a Query and ReadItem operation; 2) in case of full update context, also the Create, Update and Delete operations must be available in the SAP data entity; 3) the SAP data entity must exhibit a flat data structure.

Constraints 1 and 2 are imposed by the usage of SharePoint Business Connectivity Services (BCS) for working with SAP external data from within SharePoint context. As result of the central BCS role in the Duet Enterprise architecture, these constraints are inevitable. Constraint 3 is required to visualize the SAP data via SharePoint External List in the well-known UI List format. Aspect of the list UI metaphor is that it resembles a data table, with row-based data.

Problem however is that typically, enterprise data is non-flattened. Enterprise data often has an hierarchical structure: parent with multiple child data entities. E.g. an Order containing multiple Orderlines; an Expense form with multiple individual ExpenseDetails. As result of constraint 3, it follows that such complex data cannot be visualized in SharePoint UI via the out-of-the-box External List. Does this mean that SAP complex business entities thus cannot be integrated within SharePoint UI context via Duet Enterprise? The answer to that is negative. The applied Duet Enterprise architecture itself, with SharePoint BCS and the SAP Service Consumption Layer (SCL) in the middle, has neither restrictions nor problems with interoperating on and exchanging hierarchical SAP data. It is merely the External List concept that cannot visualize complex data (note: this limitation is generic, thus not only for SAP data structures, but also non-SAP business data structures; as Oracle, Microsoft Dynamics, SQL-based, WCF data objects, …). You can still apply Duet Enterprise as SAP / SharePoint integration foundation in case of complex SAP data structures, but it requires you to build a custom UI in SharePoint instead of the out-of-the-box External List concept. Nothing withholds you from constructing a custom SharePoint UI to display the hierarchical data, e.g. in a master-slave UI concept.

Well, not entirely true…; a noticeable problem with building a custom UI to interoperate Duet Enterprise, is that the custom UI must then program against the BCS ObjectModel API. And the BCS OM currently only provides a [very] weakly typed programming model, instead of the strong-types we are used to in a regular .NET context (e.g. WCF data objects plus WCF interfaces). The explanation for this is that BCS is a generic concept, intended to interoperate against arbitrary external data repositories and various and a priori unknown data structures. In its current stadium, the mapping at .NET client side to the concrete external data structures is enabled via a general purpose ‘BCS operation language’. The problem with that BCS language is that there is no compile-time checking nor support to prevent you from making typo or structure mistakes.

An example interoperating complex SAP data into SharePoint

Via an example I will demonstrate in overview what it takes to interoperate complex SAP data via Duet Enterprise into a SharePoint front-end.

Example

Employee Self-Service process for expense handling. The process consists of multiple steps: it starts when an employee submits an expense form for receiving a refund; next the employee’s manager will review the form, and either approve, deny, or return it for further explanation. In case of approval, the finance department will refund the made expenses. In case of return, the employee can augment the expense form with additional remarks, and then resubmit. Or the employee can decide to withdraw the returned expense form. In case of denied expense, the process directly stops. Each process execution ends with archival of the submitted form and other documentation.

Current, this process is already implemented within the SAP environment. Employees can enter a form via a WebDynpro form; which kicks of a SAP workflow process. The manager receives the review task in this workflow within the SAP Universal Work List. Payment is done via the HR business package.

Although thus the expense process is already implemented in the IT landscape, management feels the need to improve on it. Employees are complaining about the WebDynpro UI, which looks and behaves different as the SharePoint based intranet in which they are performing their primary work activities. The managers on the other hand are perfectly satisfied to perform the review task within the SAP GUI, since this is for them also for most other tasks the familiar work environment. The financial handling is mostly automatic performed in the SAP backend, near to the other financial processes; so neither any reason to change on that. The management therefore decides to bring the employee’s involvement in the process into the context of the SharePoint based intranet, while leaving the rest of the process execution untouched (phase 1).

Step 1: derive the requirements and the functional + process requirements

Since the process is already implemented; this step can largely reuse on what is already available qua system and process specifications.

Step 2: derive the integration and software architecture

The project goal is enabling employees to operate their steps in the expense handling process directly within the SharePoint intranet, with the same familiar UI look & feel as other parts in that intranet. Implication of that is to interchange the WebDynpro front-end for a SharePoint front-end. Via well-derived integration points this new front-end must hook into the existing expense process.

Step 3: define the interoperability interface of the SAP backend

With the integration and software architecture in place, including the conceptual specifications of the integration interfaces SharePoint front-end <– SAP process; the next step is to realize the process integration at SAP side. Following the Duet Enterprise Development Steps, this first means to map the conceptual interfaces onto SAP SCL Interfaces. That development activity is done in SAP Enterprise Service Builder (ES Builder). There, you specify the data types, message types, and method operations.

Screenshots of the specification at SAP side of a Query operation signature, with a complex hierarchical SAP data entity

Step 4: Implement the SCL Interfaces

Actually, this is more than one single step. It involves Mapping, connecting to relevant parts in the SAP backend, routing, potentially composition of multiple SAP data entities, et al. Although all very interesting, this has no direct relation or influence on the way complex SAP data can be handled in the SharePoint front-end. Therefore these steps are not discussed here. See the Duet Enterprise Development Guide for a proper explanation of the steps.

Step 5: Generate the interoperability interface for the SharePoint front-end

A result of step 3 is the WSDL specification of the SAP SCL interface. With this WSDL, and a runtime SAP Proxy implementation, it is possible to generate an External Content Type. The tool used for this is SharePoint Designer 2010.

Screenshots of the generation at SharePoint side of an External Content Type to SAP backend, with a complex hierarchical SAP data entity exchanged

Step 6: Program to the interoperability interface for the SharePoint front-end

In case the SAP data entity had a flat representation, we would be nearly finished. Just generate an External List in SharePoint Designer or via the SharePoint UI, and connect to the generated External Content Type. However, the expense data entity at SAP side exhibits a complex structure: form with multiple detail lines. This cannot be reasonable visualized in the table-oriented format of External List (remember that the start of this project is to improve on the user experience for the employee; so it must feel natural and intuitive). If the External List is not feasible, then it is required to build a custom UI in SharePoint context. The custom UI must interoperate against the BCS ObjectModel API, to query, retrieve, create and update the SAP expense data entities. In its current state, the BCS API can only be operated via a weakly-typed program model. The draw side is that programming at that low-level is both cumbersome as error prone. However, if done correctly and secure, it does the job. And enables you to interoperate complex SAP data entities via Duet Enterprise in a custom SharePoint / .Net context.

Example code of interoperating BCS to create a new expense entity within SAP backend

BdcService service = SPFarm.Local.Services.GetValue(String.Empty);
IMetadataCatalog catalog = service.GetDatabaseBackedMetadataCatalog(SPServiceContext.Current);
IEntity entity = catalog.GetEntity("TopForce.com", "Expense");
IView createView = entity.GetCreatorView("Create");
IFieldValueDictionary methodFields = createView.GetDefaultValues();
methodFields["EmployeeNumber"] = "1";
methodFields["Status"] = "open";
methodFields["EmployeeComment"] = "my comment";
methodFields["Date"] = DateTime.Now;

object tickets = methodFields.CreateCollectionInstance("Tickets", 2);
methodFields["Tickets"] = tickets;
methodFields["Tickets[0]"] = methodFields.CreateInstance("Tickets[0]");
methodFields["Tickets[0].Date"] = DateTime.Now;
methodFields["Tickets[0].Amount"] = "35.45";
methodFields["Tickets[0].Description"] = "Description 1";
methodFields["Tickets[1]"] = methodFields.CreateInstance("Tickets[1]");
methodFields["Tickets[1].Date"] = DateTime.Now;
...
Identity id = entity.Create(fieldValueDictionary, LobSysteminstance);

Step 7: Build an UX custom SharePoint front-end

With the integration layer SharePoint/BCS - Duet Enterprise - SAP ready, it is now rather standard ASP.NET programming to build the custom UI. In the custom UI you can utilize the full toolbox of ASP.NET webcontrols, jQuery, and even Silverlight. Another viable option in SharePoint 2010 is InfoPath based forms.

Some custom UI forms for the expense handling: overview, details and submit new expense

Final note

Thus although it requires more work – at the SharePoint side; it is possible to apply Duet Enterprise for SAP / Microsoft interoperability even when dealing with complex SAP data entities. The extra work is restricted to the SharePoint front-end part; the other parts of the Duet Enterprise pipeline are not influenced. Duet Enterprise on itself does not limit the usage of complex SAP data entities; this restriction is merely within the current out-of-the-box capabilities of SharePoint BCS and External List.

SPWebConfigModification playing tricks in farm

2011-06-11T17:04:00.023+02:00

At customer premisses we apply a.o. the following deployment guidelines:

In case of Farm-based deployment, then deploy to virtual bin (WebApplication), unless...
Apply SharePoint support for all required web.config modifications; instead of manual action (which is error prone in the farm).

Both guidelines are valid from SharePoint operations AND development perspective, to ensure a consistent and controlled SharePoint farm situation.

The SharePoint platform provides multiple deployment functionalities to comply with the 2nd guideline. Via the SharePoint solution manifest, you can specify modifications that typically have effect on the runtime operation of an assembly: SafeControls, CodeAccessSecurity. And via the SPWebConfigModification class you can modify in a controlled manner the web.config for other changes; e.g. for custom application settings, navigation providers. As from SharePoint 2007, I'm a big fan of applying SPWebConfigModification - it gives you as developer full control to have the needed web.config modifications executed upon application deployment and/or provision time. And to make it even better: the same set of changes are applied to all the individual web.config files of the SharePoint webapplication in the total farm; accross zones and accross servers. Even when at a later time another server is added to the farm, the same set of web.config modifications are applied (rather, repeated) also on that new server.

However, the SPWebConfigModification functionality does have it's peculiarities. Lately we encountered one I was not yet aware of, and which caused us both some headaches as well as cost some elapsed time.
The situation is as follows: SharePoint assembly that uses EntLib 5.0 for accessing an external SQL database. Comform the guidelines, this custom assembly is deployed to the virtual bin; and it is therefore required to set custom CAS-policies for this assembly. One of the required permission is the SqlClientPermission. That permissions is not present in the default WSS_Minimal trustlevel, but it is within WSS_Medium. To have the total set of custom CAS-policies based on the Medium trustlevel, a 2-steps approach is applied:

First change the trustlevel in the web.config from WSS_Minimal to WSS_Medium; a task performed via SPWebConfigModification class
Next, deploy the SharePoint solution with in its manifest the CodeAccessSecurity element for the assembly. The resulting custom CAS-policy file is now based on the Medium trust level, thus inheriting a.o. the SqlClientPermission SecurityClass setting. The SharePoint solution framework takes care of the required modifications in the web.config: link to the generated custom policy file, and set the trustlevel to WSS_Custom.

The setup worked perfectly, both local as in the shared test-farm.

That is, initially. From time to time our application appeared broken. Root cause analysis exhibited that in those situations our web.config was modified; but without any deployment activity on our own webapplication (???) However, another SharePoint webapplication in the same farm had been redeployed. Each time that application was redeployed, our web.config was modified; and the trustlevel reset from WSS_Custom to WSS_Medium.

It appears that this is standard behaviour of the SPWebConfigModification class: each time that it is requested to apply administrated SPWebConfigModifications entries in the context of a single SPWebApplication, it effectively re-applies the administrated SPWebConfigModifications of ALL the SharePoint web-applications in the farm. At minimal, the result of this is that all the web.config files in the farm are touched; and have their timestamp updated. But in our case, the administrated SPWebConfigModification for setting the trustlevel to WSS_Medium was reapplied; which broke our application!

So I learned 2 things here:

SPWebConfigModification is not a 'decent' SharePoint citizen; I regard it as ultimately incorrect that the intented application of SPWebConfigModification administrated entries on 1 SharePoint web-application, also effects all other SharePoint web-applications web.configs.
That as result of this behaviour the considered 2-steps approach for setting the correct medium level of a.o. SQLClientPermission cannot be maintained; due the inherent risk that the administrated SPWebConfigModification can be re-applied at any moment; out of the control and knowledge of the administrators of our webapplication. So I modified this to set the CAS-policies in a single step; directly steered via the Solution manifest file.

Thus instead of the below CodeAccessSecurity specification, which is relative and relies on the presence of a.o. SqlClientPermission SecurityClass in the base trust-level (WSS_Medium):

<PermissionSet class="NamedPermissionSet" version="1">
<IPermission class="SecurityPermission" version="1" Flags="Execution" />
<IPermission class="AspNetHostingPermission" version="1" Level="Medium" />
<IPermission class="SqlClientPermissionversion" version="1" Unrestricted="true" />
<IPermission class="SqlClientPermission" version="1" Unrestricted="true" />
<IPermission class="Microsoft.SharePoint.Security.SharePointPermission, Microsoft.SharePoint.Security, Version=14.0.0.0, Culture=neutral,
PublicKeyToken=71e9bce111e9429c" version="1" ObjectModel="True" />
<IPermission class="Microsoft.Office.SecureStoreService.Server.Security.SecureStorePermission, Microsoft.Office.SecureStoreService.Server.Security,
Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" version="1" Unrestricted="true" />
<IPermission class="System.Security.Permissions.ReflectionPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" version="1"
Unrestricted="true" />
<IPermission class="System.Diagnostics.EventLogPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" version="1"
Unrestricted="true" />
<IPermission class="WebPartPermission" version="1" Connections="True" />
</PermissionSet>

specify all the required permissions explicit:

<PermissionSet class="NamedPermissionSet" version="1">
<IPermission class="Microsoft.SharePoint.Security.SharePointPermission, Microsoft.SharePoint.Security, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" version="1" ObjectModel="true" Unrestricted="true" />
<IPermission class="SecurityPermission" version="1" Flags="Execution" Unrestricted="true" />
<IPermission class="System.Web.AspNetHostingPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" version="1" Level="Medium"/>
<IPermission class="System.Data.SqlClient.SqlClientPermission, System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" version="1" Unrestricted="true"/>
<IPermission class="Microsoft.Office.SecureStoreService.Server.Security.SecureStorePermission, Microsoft.Office.SecureStoreService.Server.Security, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" version="1" Unrestricted="true" />
<IPermission class="System.Security.Permissions.ReflectionPermission, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" version="1" Unrestricted="true" />
<IPermission class="System.Diagnostics.EventLogPermission, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" version="1" Unrestricted="true" />
<IPermission class="Microsoft.SharePoint.Security.WebPartPermission, Microsoft.SharePoint.Security, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" version="1" Connections="True"/>
</PermissionSet>

Utilizing Maps services for runtime determining nearest relations

2011-06-03T22:24:00.023+02:00

In a former SharePoint project – an external facing public website -, a required functionality was to display the nearest X insurance advisors given the submitted postal code of user. We weighed several alternatives to realize this:

Re-use the distance-calculation algorithm used by the predecessor of the website (note: the essence of the project was a technical migration from a non-strategic proprietary WCM platform to the enterprise architecture target platform: SharePoint)
Utilize service of postal company: a data repository of all postal code tuple combinations, with the distance between them administrated.
Utilize Geo-coordinates to runtime calculate the distance between 2 locations, via the Haversine formula.

The first would have the disadvantage to still depend on the non-strategic platform and supplier. Also it had a rather complex programming API, without documentation and support.

The second would require us to host an external database besides the SharePoint content database in which to administrate the millions of postal code / distance records. This database would have to be updated each year with the management update from the postal company. Also, security constraints would not allow to directly access from the SharePoint WFE in the DMZ, the external database hosted in the internal network. The access should be regulated from DMZ to inner network via a secure webservice interface; which of course implies extra work and additional hosting costs.

So, the third alternative appeared the best fitted. Issue here is how to translate the postal code (submitted by the website user) into geo-coordinates (required for the Haversine formula). Luckily, both GoogleMaps and BingMaps provide services for this. Rough outline of the applied approach:

Preparation

Store in a SharePoint List the relations (less than 5000) data, including their geo-coordinates

Runtime

Invoke Maps service to derive the geo-coordinates (Latitude + Longitude) given the postal code submitted by user
Calculate the distance to each applicable relation via the Haversine formula
Ascending sort on calculated distance
Display the X nearest advisors

An hosting constraint for step 2 is that it is not allowed to directly invoke from the SharePoint farm, external webservices. To circumvent that restriction we decided to call the Maps service in the security context of the browser, via jQuery/JSON. The returned geo-coordinated are next via a postback with arguments transmitted to the server side; where then the distance calculations are done.

Architecture sketch:

Automatically publishing multiple files fails with error '0x81020015'

2011-05-13T20:00:00.031+02:00

In my previous post I wrote about the well-known artefact of sandbox solutions that provisioned files are by default still checked-out. This can be fixed via a FeatureReceiver that checks-in all files and approves the ones that require moderation, so that all provisioned SharePoint artefacts are direct available for the end user.

Initially this worked like a charm, and the problem seemed resolved. However when our project progressed and the number of provisioned files increased, we structurally encountered the following error upon feature activation:

Microsoft.SharePoint.SPException occurred
  Message=The file _catalogs/masterpage/ApplicationX/ApplXPageLayoutXX.aspx
    is modified by domain\WvStrien on May 11 2011 17:06:12 +0200.
  Source=Microsoft.SharePoint
  ErrorCode=-2130575305
  NativeErrorMessage=FAILED hr detected (hr = 0x81020015)

In the root cause analysis of the problem I discovered that the problem only manifests itself when more than 1 file requires approval. This also explained why initially we did not encounter the problem: we started out with only a single PageLayout. With this insight I did an internet search for '0x81020015'. Although no direct related pointers to our situation, this led me in the direction of the problem cause. Apparently it is a race condition within SharePoint internally when first checking in a file and immediately approving it in the same execution context. The internal SharePoint administration is typically not ready yet to handle the second state change on the file.

If this is the cause, then the solution is to break up the execution context: first do the checkin(), and in another execution context the approve() invocation. Normally you would do this by delegating it to a background thread. However, this is not possible in a Sandbox context: the SharePoint ObjectModel is only supported on the main thread of the SPUserWorker process. The best next and pragmatic approach is then to break the execution flow by introducing a delay between the invocation of the 2 SPFile methods:

private static void PublishFile(SPFile checkedOutFile, bool approvalNeeded)
{
  checkedOutFile.CheckIn(String.Empty, SPCheckinType.MajorCheckIn);
  if (approvalNeeded)
  {
    int nrOfWaits = 0;
    while (!checkedOutFile.Exists && nrOfWaits++ < 5)Thread.Sleep(1000);
    if (checkedOutFile.Exists)
     checkedOutFile.Web.GetFile(checkedOutFile.UniqueId).Approve(COMMENT);
  }
}

Error “Unable to load assembly group” upon redeploying a sandbox solution with a feature receiver

2011-04-27T23:49:00.015+02:00

IT operations set up a fresh new SharePoint webapplication for my current project on the development farm, and provisioned it with a root site collection based on the publishing template. Next I took over the deployment control by following the sandbox route; in this preliminary phase merely to provision the first version of the branding: master page, CSS files, images and page layouts. A known artifact of sandbox solutions is that the provisioned files are by default still checked-out in the content database by the person who activated the solution, and thus not visible for anyone else. [It is a mysterie to me why Microsoft implemented it such; and moreover why they didn’t delivered us a setting to configure this provisioning behavior to automatic checkin and publish/activate the provisioned SharePoint artifacts.] Instead of manually via the browser or SharePoint Designer having to lookup in the site collection all the provisioned SharePoint artifacts to check them in and publish them, I more favor the approach to have this done automatically in the context of the feature activation. This can be easily accomplished via a feature receiver (Waldek Mastykarz has a good start/example for this in his blogpost Automatically publishing files provisioned with Sandboxed Solutions). So I added a feature receiver to the provisioning feature in the sandboxed solution, and in the FeatureActivated event let all the provisioned SharePoint artifacts be located in the content database and automatically checked in + published if applicable. Tested it locally, works like a charm. Deactivated and deleted the previous uploaded version of the sandboxed solution, and uploaded the new version with the feature receiver inside. However, upon activating this sandbox solution version, I consistently got the following error:

Error occurred in deployment step 'Add Solution': Unable to load assembly group. The user assembly group provider threw an exception while trying to provide user assemblies for the specified assembly group.

And in the ULS logging the following detail info:

Unable to load assembly group. The user assembly group provider threw an exception
Assembly Group Id: GroupId = "7C00665459714EEC9BDA8727AD711EC7-FES7hxwCdYldvtgExF/L+b259Rh3T1thWDeijNWTrSU="" - Inner Exception: System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
at Microsoft.SharePoint.SPListItemCollection.get_Item(Int32 iIndex)
at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.
GetSolutionInfoFromGallery(Guid siteId, Guid solutionId, String solutionHash, String& fileName, String& hash, Byte[]& fileBytes)
at Microsoft.SharePoint.UserCode.SPUserCodeLightweightSolutionAssemblyGroupProvider.
GetAssembliesInGroup(Guid siteId, String assemblyGroupId)
at Microsoft.SharePoint.UserCode.SPUserCodeAssemblyCacheManager.
EnsureUserCodeAssemblyGroupIsCached(Guid siteId, SPUserCodeAssemblyGroupId userCodeAssemblyGroupId)

An internet search did not give many nor useful hits; and neither could the SharePoint development gurus in my neighborhood help me. So it left me rather puzzled at first how to analyze and solve this. A good clue however was that a sandboxed solution from another SharePoint (web)application with also a feature receiver inside, had the same problem trying to activate it in our site collection, while it successfully activated in the own site collection. So it must be something at SharePoint infra level, not caused by code development. With this insight I went back to my operations companion, and provided him with a sandboxed solution to monitor what events occurred at system level upon trying to activate the solution. This quickly let to a conclusive result: somehow the permission state of the User Code service account on the C:\ProgramData\Microsoft\SharePoint\UCCache subfolder got compromized, which disallowed it to re-copy the new version of the sandboxed solution with code inside to the UCCache subfolder. The problem was not even resolved by restarting the User Code service; instead it was needed to manually reset the User Code service account, so that upon the next invocation the ACL on the UCCache subfolder was again built up.

Our operations guy was very thoughful to make a note of this behavior in the internal SharePoint operations Knowledge Base, and document it in a work instruction.

Duet Enterprise lessons learned for custom application scenario

2011-03-28T22:16:00.006+02:00

The Duet Enterprise proposition is multi-faceted. It is a product with ready-to-use functionalities. It is a composition environment for constructing your own solution by composing delivered functional and technical building blocks, e.g. SharePoint Site templates for SAP business objects, SAP reports scheduling and publication into SharePoint, expose SAP workflow decision points as SharePoint tasks. And it is an Integration Foundation for building your own custom application scenario.

In case of that last context, a Duet Enterprise project is a software development project. However, it is not just like any other development project. The application of the Duet Enterprise Integration Foundation in custom SAP / SharePoint application scenarios imposes specific prerequisites to be successful. From experiences with applying Duet Enterprise in custom application scenario, we have learned several lessons.

Starting with Duet Enterprise

Before you are ready to apply Duet Enterprise in a custom application scenario, the project team needs to invest time to get familiar with the product. Get acquainted with the Duet Enterprise concepts, architecture, capabilities, configuration, development approach, constraints and its restrictions. It proofs a steep learning curve to become familiarized with Duet Enterprise and the diverse aspects. This learning experience is made more difficult by the lack yet of concise study material and available reference cases in the open to learn from.

Also you need to install Duet Enterprise in your combined SAP and Microsoft IT landscape. It’s common that this task is performed by the IT operations in your company. Be aware that installation and configuration of Duet Enterprise is more than a simple ‘next-next-OK’ click exercise. Especially on the SAP side, the installation of the Duet Enterprise SAP Add-on consists of configuration steps in diverse parts of the SAP landscape (e.g. SE80, IMG, SOA Manager, …). The total elapse time for installing and configuring Duet Enterprise can easily extend to 5 or more workdays. For the major part the SAP and SharePoint installation and configuration can be done isolated and in parallel. However, there are also multiple moments where it is essential to work together, and exchange system information. From SAP imported into the SharePoint farm, and later on SharePoint security settings required at the SAP NetWeaver / Service Consumption Layer (SCL) side.

Starting within a custom SAP / Microsoft application scenario

This is no different from any SAP / MS integration scenario. Don’t start with the tool, start with the business question! Make sure you understand which problem(s) you are trying to solve. Invest proper time for performing the business and information analysis. Agree with the customer what kind of user experience is asked for. With these demand aspects sufficiently clear, continue with deriving the integration and the software architecture design. In this process both SAP and Microsoft solution architects should be involved. They bring in the insights, knowledge of best and bad practices from the experiences on their own turf. When sketching the integration architecture, continually evaluate whether and where Duet Enterprise can add value, given the Foundation capabilities and the imposed constraints. Don’t make the application of Duet Enterprise your goal; but determine for the application context if and where Duet Enterprise adds value.

Functional system architecture

Input here are the business process, the requested functionalities, the end-user audience, and the workplace characteristics. Decompose the system functionality in use cases. Often not all of the system use cases have the same set of actors. The functional and process architect(s) must carefully decide for which of the use cases there is a business case to expose outside of the SAP boundaries, and for which not. Example of the last is an approval task by a manager or the HR department. They are SAP power users, spending a large part of the day within the SAP GUI. The GUI has no secrets for them, it’s their familiar workplace. The larger workforce in the company is instead a non-casual user of the SAP environment. They are not used to the SAP GUI, and loose valuable time finding out how to operate within the GUI. The financial business case here is reducing the amount of required SAP licenses, and reducing the time spend by this larger set of employees due the unfamiliarity with the SAP UI. A soft business case is that the employee satisfaction is improved by integrating the SAP related work within their common and familiar workplace.

Starting with Development approach

Again, this is no different from other enterprise application integration scenarios. A proven best practice is to start with establishing and defining the contracts via which the application counterparts will communicate. Initially these contracts are defined in abstract format, focusing on the identification and separation of system responsibilities. Microsoft architects recognize this as applying the ‘Contract-First’ approach; SAP architects are more familiar with the term ‘Outside-In’. Do a design time validation on paper or whiteboard that all of the identified functional use cases can be realized by the set of identified interfaces. Augment and tune the interfaces until this design check leaves no more gaps.

Next, model the identified abstract service interfaces in the Duet Enterprise SAP Add-on. The tool for this is the ES Builder, a Java-based modeling environment delivered either as part of SAP PI, or of SAP NetWeaver Composite Environment (CE). In this development step the technology-agnostic service interfaces must be translated into specific SAP service-technology format: data structures, message types, interface operations. Here you also augment the abstract service interfaces with the specific Duet Enterprise requisites: Correlation ID and Business Object Instance Key . These 2 specific parameters are required to enable the Duet Enterprise system monitoring and routing capabilities. Also add here SOAP standards-based fault handling to each service operation. The ES Builder directly supports this via the SAP ESA SOAPFaultException datatype. The SharePoint consumer of the Duet Enterprise service interface can handle the received SOAP Faults in a standard manner through WCF, transparent to any SAP specifics in the fault notification. When all the identified service interfaces are modeled in ES Builder, generate in the ESB the WSDL service description. This W3* standards-based and platform independent service description is utilized and needed on both the SAP service provider and the SharePoint service consumer sides for building the integrated application.

Building the SAP / SharePoint integrated application

With the WSDL service description available, the SAP and SharePoint development teams can each go ahead with the realization of their respective application responsibilities.

At the SAP side the identified service interfaces must be implemented by concrete service realizations. In a Duet Enterprise landscape the term Service Provider refers to the SCL system, and SAP Backend refers to the system that hosts the Business Functionality. SAP Duet Enterprise service implementations are thus positioned in the SCL. The SCL services connect to the SAP Backend to consume the business functionalities and data. Since this SCL-SAP Backend connection is strictly within the SAP environment, it is possible to rely on SAP proprietary connection technology. In fact, between the SCL and the SAP backend it does not add anything additional to connect via web services (BAPI or Enterprise Services). Using web services to connect to the Backend brings some additional performance overhead so RFC-based connection is faster. The development work inside the SCL further consists of mapping the external faced service signatures to the internal SAP processing and data structures. This involves the technical mapping from the W3* standards-based integration handling to the SAP internal invocation model and data structures. And potentially at business process level the composition of multiple SAP actions in the scope of a single Duet Enterprise service operation.

The required work at the SharePoint Duet Enterprise consumption side is largely dependent on the operation and data signature of the provided SCL services. At SharePoint side, Duet Enterprise builds on Business Connectivity Services. Implication of this is that the SCL services must exhibit a CRUD+Q operation signature. If not, consumption via BCS is not even possible. If BCS-based consumption of the external SCL services is possible, another aspect is the data structure. SharePoint 2010 out-of-the-box delivers the concept of the External List UI to interoperate with external data. A practical limitation of the External List UI metaphor is that it requires a flat data structure. If not flat, it is not possible to display the external data in a row-based manner. Non-flattened / hierarchical data can still be consumed via BCS in SharePoint, however not via the External List concept. The project will have to build a custom UI to interoperate with hierarchical SAP data structures. The custom UI must connect to the BCS Object Model to interoperate with the external SCL services. Current drawback is that the BCS API only supports a weakly-typed programming model. You have to program per exchanged hierarchical SAP data structure an ORM-based mapping from strong-typed data representation at the SharePoint front-end to the BCS weakly-typed representation.

Debugging Duet Enterprise connection problems

Here is where the application of Duet Enterprise clearly adds additional value. If you obey to the Duet Enterprise constraints; that is if you have per service operation included the Correlation ID parameter; the Duet Enterprise landscape supports the problem analysis via an integral audit trail through the combined SharePoint + SAP systems chain. You look up the Correlation ID value in the SharePoint ULS logs for where the BCS-based connection starts, and follow it through the SCL layer via the SAP system monitoring. Problems can be of infrastructure nature, e.g. SSL certificate at SAP and SharePoint not in sync; SharePoint BCS-SCL service connection level as result of a wrongly chosen SAP proxy endpoint; SAP authentication and/or authorization due incomplete Duet Enterprise user mapping. The Correlation ID based audit trails proofs a big help in tracing where in the integrated SharePoint / SCL / SAP Backend landscape the actual problem cause manifests itself.

Duet Enterprise development work plus applicability

2011-03-19T22:12:00.008+01:00

At the global level, following development activity is involved in building a custom Duet Enterprise scenario.

SAP side

Plain and simple: Provide SCL-ready services, that are SharePoint BCS consumable.
There is no other option here, this work is required to deliver SAP data + processing services that are enabled for SCL-publishing. At minimum this involves ABAP custom development for mapping the W3*-standards based SCL data representation onto the native SAP data representation, and vice versa; and invoking from within the SCL runtime context the inner SAP processing at either RFC, BAPI web services or SAP Enterprise Services level.

SharePoint 2010 side

Here the amount of development work depends, namely on the signature of the SCL-published services. Ideally the services provide a CRUD+Q signature (required) + a flattened data structure (optional). If both aspects are present in the SCL-services, these can be directly consumed and displayed in SharePoint External List. Then there is no development work required at SharePoint side, it is all configuration done in SharePoint Designer. You get the maximum added Duet Enterprise value in case of flattened data: SAP/SharePoint connection, authorization, system monitoring; and out-of-the-box presentation of the SAP data.

In case of non-flattened, complex/hierarchical data structures, the External List UI metaphor is not viable. Duet Enterprise can still have its value, on the connection, authorization + system monitor aspects (plumping). What you loose here is the out-of-the-box UI presentation. You will have to build a custom UI instead, and program directly against the BCS Object Model. Thus more work, but Duet Enterprise still has standards-based value and role for SAP/SharePoint interoperability.

Connection requirements / BCS implication for Duet Enterprise

The Duet Enterprise SharePoint Add-on builds and relies upon BCS. Consequence is that BCS prerequisites are imposed on the SAP / SharePoint service connection, provided by the SCL-publishing and BCS-consumption. The SCL provided SAP-services must expose a CRUD+Q interface, with at minimal a ‘ReadList’ (= Query) and ‘ReadItem’ (= Read) type of operation. If the SAP processing behavior cannot be mapped on the CRUD+Q signature, you cannot reasonable apply Duet Enterprise. Point. The SAP .NET Connector 3.0 may be a alternative for such connections.

Schematic overview of the Duet Enterprise applicability

Positioning of .NET Connector 3.0 versus Duet Enterprise

2011-02-10T23:32:00.014+01:00

In a timeframe of a couple of weeks, SAP independently released 2 products for .NET interoperability: first [December 2010] the next version of the .NET Connector – NCo 3.0; followed a few weeks [Februari 2011] later by the public launch of Duet Enterprise – a combined effort of SAP and Microsoft. These near overlapping release moments may raise questions and uncertainty on the positioning of the 2 .NET interoperability products. Are they competing? Are the successive release moments a symptom of independent products groups within SAP, and will market acceptance determine which one will ‘win’? And what about the SAP ES Explorer? Earlier, SAP spokesman declared the .NET connector outdated in favor of the ES Explorer.

In my thoughts and analysis, the story is differently. NCo3.0 and ES Explorer on one side, and Duet Enterprise all have a distinct positioning. The 3 products are neither competing nor exclusive for .NET interoperability. Each serves a specific and dedicated purpose.

NCo3.0 and ES Explorer

Both NCo3.0 and ES Explorer are in essence .NET Interoperability technologies. Neither of them is positioned as a product. You can not purchase or license it, but instead can download them from SAP Marketplace if you have a valid S-User ID that acknowledges you as a SAP developer. The release of .NET Connector 3.0 passed rather silently, with only an announcement on SDN. It has not received attention from any of the influencing IT business magazines, nor IT Research Analysts. There was a Ramp-Up with selected beta-testers, but with minimal noise ahead and attention during.

NCO 3.0 is intended as a general purpose technology tool for low level integration plumping. Basically it enables bi-directional interoperability between .NET custom code and SAP RFCs and BAPI Function Modules. Nothing more, nothing less.
The role of SAP ES Explorer is in essence the same. In the earlier SAP statement, Rima Rudnik-Sirich characterized it as 'It succeeds SAP .NET Connector 2.0 for .NET'. The difference is within the interoperability manner. SAP ES Explorer works at the level of SAP Enterprise Services, W3*-compliant. You apply it as Visual Studio Add-In, to search through all the SAP Enterprise Services available in your landscape; standards ones from SAP, from third parties deployed in your landscape, and your customer-build Enterprise Services. Runtime invocation of the selected Enterprise Services occurs via a generated WCF service proxy.

Duet Enterprise

Duet Enterprise is a whole other story. Both SAP and Microsoft position it as a product that will directly provide business value. The Duet Enterprise release was done at large: a Virtual Launch Event, press releases and articles within the important IT business magazines, analysis reports by Forrester and CITO Research. Application of Duet Enterprise requires a license, the product can be purchased via SAP and Microsoft. (Note: at moment of writing there is no information disclosed on pricing and licensing model). Before reaching General Availability, Duet Enterprise has been evaluated in a combined Ramp-Up / Rapid Deployment Program hosted by SAP and Microsoft together. Participating in the RDP was given substantial noise ahead, and intensive attention during the program course self. (Note: TopForce participated together with a large Dutch insurance company in this RDP).

The role of Duet Enterprise is twofold. It comes out of the box with direct usable functionalities, and capabilities to compose customer-specific solutions. Its second face is what distinguished Duet Enterprise from the previous Duet versions: integration Foundation to build integration + interoperability that is specific for your situation. Also, SAP and Microsoft believe and want ISVs (the ecosystem) to deliver add-ons on the Duet Enterprise products, e.g. for specific vertical markets. To stimulate this, SAP and Microsoft also launched a partner program: Unite Partner Connection Program.

The Duet Enterprise Foundation is also interoperability plumping, but of higher level as NCo 3.0 and ES Explorer. SAP / .NET interoperability occurs via standard W3* services; there is out-of-the-box support for SSO, Authorization, landscape monitoring. All aspects that earlier you typically had to handcraft yourself, and thus also maintain.

Duet Enterprise is bounded to usage from within a SharePoint 2010 context, and via SharePoint 2010 as intermediary layer in Office 2010 clients. Duet Enterprise has no role in other .NET contexts, e.g. SharePoint 2007 or 2003, Silverlight, WinForms or WPF apps, WCF handling, BizTalk.

Implications

The added value of Duet Enterprise [product, foundation] wrt NCo 3.0 and SAP Explorer [basic interoperability technologies] is that it raises the level of SAP - .NET interoperability. In addition to the bi-directional SAP - .NET runtime communication, it also provides interoperability concepts as SSO, Authorization, System Monitoring. And on top, it comes with direct usable functionalities, and capabilities to compose solutions [building blocks].

The limitation is that it only works from a SharePoint 2010 context (and via SharePoint in Office 2010 clients). If you need .NET interoperability from different .NET context, you have to resort to another approach. SAP provides for this both the NCo 3.0 and SAP ES Explorer; which one is usable is dependent on the level of SAP back-end consumption. If SAP Enterprise Services are available; apply SAP ES Explorer; for RFCs and FMs you can use NCo 3.0.

Note: there are also several technologies and products delivered outside SAP to enable .NET interoperability. For instance Microsoft provides the BizTalk WCF LOB Adapter SDK, BizTalk itself; third parties provide product like Sitrion, Ometa, ERP Connect. For the scope of this article they are however not considered.

Development Approach with Duet Enterprise

2011-01-19T19:42:00.005+01:00

This blog is, with some minor changes, earlier published on SAP Community Network Blogs

TopForce participated in the 2nd half of 2010 within the Duet Enterprise Rapid Deployment Program. Together with our RDP partner, we followed a well-chosen approach to evaluate this SAP – SharePoint integration product on it’s capabilities and potential.

In the Ramp-Up / RDP I was involved in plotting the Duet Enterprise prerequisites in the company IT infrastructure, selecting the PoC scenario, architecting the integration approach, and configuration + realization of the Duet Enterprise services for the PoC scenario. In this article I highlight the main points of our journey.

Product evaluation via a custom Proof-of-Concept

Both partners in the RDP regarded it as essential to evaluate Duet Enterprise as interoperability product via a custom Proof-of-Concept. Only such we are confronted wih actual experiences and issues in the application and potential of the Duet Enterprise capabilities. We do not suffice with merely a rather simple out-of-the-box scenario provided by Duet Enterprise.

Selection criteria for the PoC scenario

For us the main reason to consider Duet Enterprise is as integration foundation for company-specific SAP / Microsoft interoperability. To best evaluate the product capabilities on this aspect, we defined the selection criteria for the PoC scenario. We weighted the possible scenarios on the following aspects:

A real application context, with current and/or future purpose
Give input to SAP / Microsoft integration design guidelines

Validation approach

The PoC is intentionally approached as a regular development project. Thus functional specification, architectural and technical design, realization and testing, with ultimate ending in product implementation. In these Agile times, the phases are not necessarily in linear order; especially the integration architecture and realization have had some iterations as results of lessons learned.

Functional specification

In this phase, the application functionality is clearly defined, and agreed plus functional validated with the customer + end-users. Noticeable is that in this phase special attention is / should be given to the UX-factor: the User Experience of the application. This is because the most heard compliant against SAP is typically the lesser user friendliness. And also because nowadays information workers expect no less than a pleasant and familiar workspace, in which one is facilitated and helped to do your daily and also occasional work activities.

Integration and Software Architecture Design

Essential in this phase is close cooperation between the SAP and Microsoft solution architects. Both typically come from different backgrounds (‘different worlds’, almost like Mars versus Venus), yet it is required that they have a sufficient common understanding to architect a proper and future-proof design.

In the architecture itself, a best practice is to apply a layered service integration architecture. Each layer has its own responsibilities. In a nutshell, SharePoint sits at the front-end / presentation layer, SAP at the back-end layer; and Duet Enterprise is the glue as integration layer.

The integration layer is derived from the [use cases of the] functional specification. One of the design principles applied is that the SAP backend is and remains ultimate responsible for correctness of the data. This also implies that business rules are enforced and remain the responsibility of the SAP backend.

Duet Enterprise also imposes some constraints on the service interfaces. At conceptual level this means that Duet Enterprise [currently] only supports data oriented interfaces, operating via CRUD behavior. The explanation is that Duet Enterprise utilizes Business Connectivity Services to exchange data from SAP to and from SharePoint. BCS is a strict data oriented capability, it is not usable for [SAP] process oriented control.

Development Process

Once the service interfaces are derived and defined, the development team can go ahead with implementing the SAP-SharePoint integration via Duet Enterprise. The outline of this development process is as follows:

Configure / define the derived interfaces as SAP Enterprise Services in the ESR, using the SAP ES Builder. Hereby you must augment the interfaces as defined with 2 additional parameters required for proper Duet Enterprise handling, namely:
- Correlation ID; this is used for the end-2-end monitoring of entire runtime flow through the SAP and Microsoft landscapes
- Business Object Instance Key; this is used as generic identifier in all SCL framework components at runtime and at designtime. It consists of 3 components, Data Value Identier, SCL Business Object Name and the System Alias of the Backend. The key is to be generated in the SCL processing of each SCL service response, in the following structure: <data_value_identifier>_<Business Object Name>_<System Alias Of Backend>. The usage of this key is enable the SCL in subsequent handling to transparently identify the correct backend in a SAP landscape with possible multiple system instances. Notice the implication that if your SAP landscape consists of only a single backend, there is no concrete runtime usage of this field; all your SCL requests will be routed to the single backend. In that case you can simplify the key structure.
Export the defined SAP Enterprise Services in WSDL format
Hand over the WSDL file to both your SAP developer(s) as SharePoint developer(s). From here, both can in parallel configure and/or build their own side.

SAP NetWeaver 7.02 / Service Consumption Layer

In the SCL system, using ABAP Workbench / transaction SPROXY, create service proxy. This generates the skeleton of the provider class that includes the methods specified in the service interface. Later on you will implement the operations which involves some ABAP programming. The explanation of this break-up is that in the construction of the operations you need the GenIL model; and that you do not have yet generated in this stadium.
In the SCL system, in transaction se38 create the GenIL model based on the request and response structure of the service proxy. This generates a generic SCL Business Object Model. The reason that it is named ‘generic’ is because it is an intermediate mapping layer between the consumer (SharePoint BCS) specific data structures and the backend (e.g. SAP ECC 6.0) specific part. This enables that the consumer can transparently communicate with different backend systems/versions. That can be simultaneous in case of a distributed SAP landscape, consisting of a multitude of variant backends (e.g. ECC 6.0 for North America, while ECC 4.26 for the Dutch company division). But it is also beneficial when you upgrade your single system SAP environment – the GenIL model shields the client side from such backend environment change. And this also holds in the other direction – via the same GenIL model other clients as SharePoint can in theory also consume the exposed SAP data and functionality. An example is Alloy for IBM WebSphere.
The GenIL model is actually to be more regarded as a generic Gateway derivative than strict Duet Enterprise specific
In the SCL via transaction SIMGH, create per service operation a Backend Operation Proxy that invokes the SAP Backend capability: either a SAP Enterprise Service, BAPI WebService or RFC / Function Module.
In the SCL, via transaction se80 create per service operation a Mapper object that implements 2 methods with predefined signatures for inbound and outbond mapping. This involves ABAP code to do the mapping from the GenIL model to the backend specific data format, and vice versa.
In the SCL / ABAP Workbench, return to the generated Service Proxy and fill in each of its generated operations. This also involves ABAP code to do the mapping from the received input to the GenIL model, and from the GenIL model to Duet Enterprise service output.
In the SCL via transaction SOAMANAGER, create the endpoint for the service proxy.

SharePoint 2010

Start SharePoint Designer to generate External ContentType(s) on basis of the imported WSDL from the SAP Enterprise Services.
If applicable, generate SharePoint External List as User Interface metaphore. Whether applicable is determined by a) the data structure; External List requires a flat structure (thus no hierarchical / parent-child relationships); b) the User Experience desired by the end-user.
if the External List is not applicable, realize in Visual Studio a custom build SharePoint webpart. Hereby you can use the whole richness of available ASP.Net and SharePoint webcontrols – datagrids, calendar, richtextbox, dropdownlist, … Also, SharePoint 2010 allows InfoPath forms for custom UI.
Configure the BCS model to connect for this External ContentType to the endpoint in the SCL of the Duet Enterprise specific service.

Conclusion

Introduction and application of Duet Enterprise is not a free ride. It takes time to get familiar with its concepts, and the development approach + tooling add-ons in the ABAP Workbench and SharePoint in the form of BCS Model Generator. It also requires preparation time at front in defining and architecting an application design. But that’s no different from any other software development project. Given that, Duet Enterprise is a noteworthy addition to the SAP / Microsoft interoperability toolbox; and I recommend to consider it for applicability in your specific SAP/MS integration scenario.

CQWP, CSS and jQuery applied to rearrange groups of overview links

2011-01-07T20:05:00.018+01:00

A customer requirement for their SharePoint intranet is a reusable 'Overview-of-Handy-Links' capability, with the following specification:

Contributors must have a means to manage the overview links. The functional mananagment must be easy, without the need to maintain HTML.
The links must be grouped
The display-order of the overview groups must be controllable
The display-order of the links in a group must be controllable
The display space must be optimable used, white-spacing must be minimized
The overview backgroundcolor, links color and links hover color must be settable
The number of columns must be settable
Minimize the amount of custom code, favour the usage of standard SharePoint capabilities

As consequence of the first requirement, application of the ContentEditorWebPart is not an option. That would mean that the contributors are directly confronted with the html-specification of the overview. Utilization of the ContentQueryWebPart appears more promising. The rough solution design is then to 1) maintain the groupheaders in 1 SharePoint SPList; 2) maintain the overview links in another SharePoint SPList, with a Lookup to the GroupHeaders SPList; 3) Connect a ContentQueryWebPart to the OverviewLinks SPList; 4) Group on the GroupHeader Lookup field/column; 5) Codify in the CQWP Main, Group and Item xslt-files the templates to deliver the appropriate html.

The requirement to optimal use the display space can for a large extend be achieved via CSS float-functionality: automatically 'glue' the individual Group DIV's next to each other. In the horizontal direction there is then no lost white-spacing. However, in the vertical direction that cannot be achieved with CSS-float alone. Float flows in horizontal direction, not vertical.

But there is jQuery to the rescue. The basic idea is to first let the CSS-floating capability align the set of DIV's in as much as possible optimal usage of the display space. And afterwards, perform some minor rearrangement to also reduce the lost of white-spacing in vertical direction.

Via CQWP	After jQuery

Reusability of the OverviewLinks-functionality is achieved by applying the Feature capability. The Feature provisions the required SharePoint artifacts: the SharePoint SPLists - including the Lookup relation between them, a preconfigured CQWP, the XSLt-files.

Solution partitioning and dependencies with mixed scopes causes farm-deployment trouble

2011-01-04T20:51:00.008+01:00

We probably all agree that it is wise to partition your SharePoint application deliverables and deployment into multiple packages. Typical examples are a Base assembly, deployed via a base wsp, providing a general foundation layer with diverse technical and functional building blocks. And then 1 or more functional packages, that provision the SharePoint artifacts and code for a specific application functionality; in which the functional building blocks may use and rely on parts provided via the Base assembly.

However, sometimes this architectural sane decision to partition your application deployment, can result in unforeseen effect. Let me demonstrate via a real-life example, that I encountered today:

Application deployment deliverable
1. includes a Base assembly, deployed via solution package Base.wsp. In this wsp package included the Base code, some basic Features, JobDefinitions, generic webparts. In the Base code / assembly there are FeatureReceivers, WebParts. As result of this, SharePoint solution generation (WSPBuilder or VS2010) puts SafeControl settings per contained WebPart in the Solution manifest; which again results in it that the scope of the Solution package is WebApplication.
2. and also contains a pure functional package; with some Features, SiteTemplates. This functional package does not contain an assembly itself, but instead relies on the assembly of the Base.wsp for reused FeatureReceivers. As result this solution package does not contain any artifacts with are scoped at WebApplication level, and SharePoint solution framework requires to deploy it global in the farm. (nb: the further described situation can even occur in situation were the package does contain managed code, as long as the assembly does not contain SharePoint coded artifacts which transition the scope of the Solution package to WebApplication; e.g. no WebParts, Resources).
The order of Solution package deployment is to first install the Base.wsp, and then the Functional.wsp. Then you would expect that the deployment dependency of Functional.wsp on Base.wsp is ascertained. However, in this scenario I discover in the SharePoint Solution store an error with the deployment of Functional.wsp. Error details:
<server 1> : The solution was successfully deployed.
<server 2> : The solution was successfully deployed.
<server 3> : Feature '<GUID>' could not be installed because the loading of event receiver assembly "Base, Version=1.0.0.0, Culture=neutral, PublicKeyToken=13a12821e9f69237" failed: System.IO.FileNotFoundException: Could not load file or assembly 'Base, Version=1.0.0.0, Culture=neutral, PublicKeyToken=13a12821e9f69237' or one of its dependencies. The system cannot find the file specified.

At first, I thought this to be a timing issue. The Solution Framework uses timer jobs to deploy solutions throughout the farm, and it looked as if the Functional.wsp was tried to be already deployed on <server 3> while the deployment job for Base.wsp was not yet executed for <server 3>. However, also upon re-deployment of Functional.wsp - while making sure at forehand that the deployment of Base.wsp was now fully completed in the farm - , it resulted in the same error on <server 3>. So it must instead be directly related to the installation on <server 3> apparently the Base.dll was not installed in the GAC on that server; while the Features within Functional.wsp were to be installed on <server 3>. Why is this?

Well, the explanation is twofold. It relates to the SharePoint Farm infrastructure, as well as the scopes of the SharePoint packages. <server 1> and <server 2> have the SharePoint webapplication role in our load-balanced farm; and each contain the SharePoint webapplications. <server 3> however is dedicated as Search Index server; and does not host any SharePoint web applications. The SharePoint solution framework deploys WebApplication scoped packages only to the servers on which SharePoint web applications are hosted (Base.wsp: <server 1> + <server 2>). But it deploys Global scoped packages to ALL the servers in the farmer (Functional.wsp: <server 1>, <server 2> AND <server 3>).

Given this insight, a proper SharePoint Solution Deployment approach is to deploy the Global scoped Package with Force-flag, thus suppressing the error signalment. This worked sufficient in our case. However, it leaves you with the somewhat unwanted situation that via the Force-flag you suppress all error notifications; thus may be too brute. A better approach is to circumvent that Solution Framework deploys your Application package also to the servers on which the web application is not hosted. This can thus be achieved by making sure that each of your Application package as WebApplication scope. Either because it inherently contains such entities (e.g. WebParts), or artificially by including a dummy artifact directed at WebApplication level (e.g. an empty virtual bin file).

Emphasis on Office 2010 hindering Duet Enterprise implementations?

2010-12-30T17:22:00.003+01:00

In a blog of Venture Research I read the following rumor: "…about the partnership with Microsoft in Duet Enterprise, which my sources say is not advancing as fast as desired because many organizations are not anxious to upgrade to the latest release of Microsoft Office, which is necessary to derive the true value of Duet. This caution by organizations to update their underlying platform has good reasons in terms of cost and resource constraints, and SAP is not able to do anything about that."

If true, I find this a misconception and incorrect positioning of the Duet Enterprise potential. Surely, it also provides a Microsoft Office 2010 clients connection. But the true strength and potential of Duet Enterprise is that of a standards-based SAP / Microsoft interoperability foundation. It comes out-of-the-box with multiple integration plumping capabilities which you otherwise need(ed) to implement yourself. And being a commercial product, Duet Enterprise is backed-up by both SAP AG and Microsoft Corp as strong and future-proof IT suppliers.

That said, the implementation of Duet Enterprise puts its demands on the software server infrastructure in your company: SharePoint 2010 Enterprise Edition at the Microsoft stack, and NetWeaver 7.02 at the SAP stack. These are minimal necessities, without the availability of the both of them Duet Enterprise is not an option. But Microsoft Office 2010 at the client side is merely a bonus, not a prerequisite.

SAP Connector for Microsoft .NET 3.0 released

2010-12-23T19:06:00.011+01:00

Yesterday SAP made the renewed version 3.0 of the SAP Connector for Microsoft .NET (NCo) officially available. This version is the long awaited successor of NCo 2.0 - which still had a design-time and runtime dependency on .NET Framework 1.x and Visual Studio 2003.

With the NCo, it is possible to directly interoperate from a .NET consumer context with SAP RFC's. It is thus a different approach as with Duet Enterprise - in which the SAP/SharePoint interoperability is achieved via standards-based webservices. The Duet Enterprise services are provided by the SCL, which transparently encapsulates and shields the specifics of the SAP backend towards the SharePoint consuming side.

In the earlier versions the NCo used the SAP librfc32.dll to achieve the SAP/.NET binary interoperability. In NCo 3.0 the RFC protocol is fully re-implemented in the NCo itself. Main advantage is that there is no longer the dependency on the librfc32.dll being available on each .NET consuming system. Also this should result in better performance as there is no more marshalling required in .NET client runtime context from the managed .NET code to the unmanaged librfc32 invocation.

In the previous NCo versions the programming model was to generate at design-time .NET-based proxies in Visual Studio 2003 per SAP RFC that you intend to call. This involved a proxy method for the ABAP Function Module, and an individual proxy class for each structure or data table used in the RFC signature. In case of a change in the SAP backend, the implication is that it is required to regenarate the proxies, and rebuild plus deploy a new assembly for the re-generated code. In NCo 3.0, the RFC call pipeline is dynamically constructed. Whenever there is a change in the enclosed SAP back-end, the NCo on-the-fly adapts the internal RFC invocation. NCo 3.0 transparently shields the .NET consumer from modifications and upgrades in the SAP backend. However, this approach also has its downside. You are in effect developing against a weakly-typed integration API, much resembling the .NET reflection pattern. There is no compile-time validation nor protection for incorrect RFC invocations.

The NCo 3.0 can be downloaded from SAP Service Marketplace, including accompanying documentation.

Participating in Duet Enterprise Ramp-Up / Rapid Deployment Program

2010-11-30T14:34:00.000+01:00

This blog is earlier published on SAP Community Network Blogs

In May this year, TopForce applied as IT partner together with an end-user organization for participating in the Duet Enterprise Ramp-Up. Rationale for both organizations to participate in this Rapid Deployment Program is to gain in an early stadium insights and practical experience on the potential and added value of Duet Enterprise.

Our mutual application was accepted by the RDP Program in June. As one of the first activities, both organizations each sent 2 to 3 employees for Duet Enterprise training. Actually this was a requisite for RDP participation. The 5-days training addressed both the infra/operations aspects of Duet Enterprise in the SAP and Microsoft landscapes [3 days], and the development approach in the SAP and Microsoft development suites [2 days].

After this first introduction to the Duet Enterprise product, we received access to the software bits and accompanying documentation. The last mostly in draft versions, understandable as the product is still in Ramp-Up.

Reasons for participating

The mission of TopForce is to deliver the concept of High Performance Workplace for our customers. In essence this means that we aim for ease of use in the daily workspace of the employees, thereby abstracting the nifty details of the total backbone IT landscape. As our background comes from SAP consultancy, at a lot of our customers SAP is [part of] the business back-end. In our HPW vision, one of the means to achieve a pleasant employee workplace is operating the SAP back-end via a SharePoint based front-end (Unlock the value of SAP business processes within a SharePoint based HPW). We defined a conceptual integration architecture for this SAP-backend / MS-frontend interoperability. The architecture is validated via multiple interoperability technologies and products; e.g. WCF to BAPI WebServices, WCF LOB Adapter, Sitrion. Drawback in all was that we still had to develop a lot of the interoperability plumbing ourselves, most noticeable the support for Single Sign-On. Duet Enterprise now promises to be a good (best?) additional alternative for enabling SAP / MS interoperability.

Our partner shares this same goal, but also has a clear additional target. In the recent history several projects have emerged in which SAP / Microsoft integration might be subject for business and IT architectural decision making, and/or in which SAP / Microsoft integration is concrete realized in some aspects. As the role of both SAP as Microsoft server landscapes is gaining importance at this end-user organization, they have the need for consistent design guidance on when and how to achieve SAP / Microsoft interoperability. Duet Enterprise is participated in this context as an important integration technology to consider.

Project team

Our RDP project team consists of the following roles / persons:

From the end-user organization

Project lead
SAP solution architect
Microsoft solution architect
SAP business analyst
SAP infra/operations
SharePoint infra/operations
SharePoint developer
SAP NetWeaver solution architect / consultant

From TopForce as IT partner

SAP / SharePoint interoperability consultant (my participation in this Ramp-Up)
ABAP developer [only temporarily required]

From the combined SAP / Microsoft RDP support team

SAP NetWeaver consultant [from SAP AG]
SharePoint consultant [from Microsoft Services]
Whenever applicable, supplemented with SAP and Microsoft consultants with varying expertise’s; dependent on nature of discussions and encountered issues

Validation approach and activities

The potential of Duet Enterprise is twofold. On one side it delivers directly usable out-of-the-box functionalities, customizable in some degrees to your situation. Second to that, and this is where it strongly differentiates from the original DUET proposition, it provides a SAP / SharePoint interoperability foundation. Although the first is certainly interesting, our main combined focus for now is evaluating Duet Enterprise for the added value on interoperability plumping aspects. This validation is done by means of a real-life custom-developed application. The following activities are executed by the RDP team:

Attend the Duet Enterprise training to get acquainted with the infra and development aspects
Install and configure Duet Enterprise in the SAP NetWeaver 7.02 and SharePoint 2010 landscape
Derive and define the Software Architecture Design for the selected real-life application
Discuss the Software Architecture Design with RDP consultants, from SAP AG and Microsoft Corp.
Model, compose and [only] where required custom-develop the interoperability + integration between SAP backend environment, and SharePoint 2010 based front-end
Derive and define the Design Guide for SAP / Microsoft integration decision making at business and IT architecture level

Where do we stand / Results so far

In this near-end phase of the Ramp-Up program, approaching the General Availability of Duet Enterprise, it is not viable to go into product details. This will be addressed later, after the successful conclusion of our Ramp-Up participation.

Results and findings that I can share now are:

Getting acquainted with the design and development approach for Duet Enterprise application has an initial steep learning curve. Mind you, this will be less after General Availability when the accompanying documentation will be more complete and mature.
Installation and configuration of Duet Enterprise takes its time. This is mainly due the complexity of any typical SAP landscape anno 2010 (aka ECC, NetWeaver, Solution Manager, SLD, ESS and MSS, …), and not so much to direct back to Duet Enterprise specifically. On SharePoint 2010 side the work is much less, although also here it depends on the characteristics (complexities) of the SharePoint farm.
This RDP project is an evidence for the added value of a mixed SAP / Microsoft development team (Build Your Interoperability Team as 1 of the Steps HowTo begin with SAP / MS interoperability)
It pays out if the SAP and Microsoft solution architects are on a general level familiar with the ‘opposite’ platform stack. E.g. what’s the positioning of SAP PI versus Microsoft BizTalk; what is the concept and support of SAP Enterprise Services;
It pays out to have at least one team member aboard that has knowledge and practical experience on the SAP / MS interoperability area; knowledge of both technology stacks, interoperability technologies and approaches.
And final remark: SAP / MS interoperability is and remains interesting stuff; from business and IT architectural point of views…

Duet Enterprise is approaching the General Availability date. If you are considering its application for SAP / Microsoft interoperability, I like to share ideas and actions.

2 approaches to architecting SAP / SharePoint interoperability

2010-11-04T21:25:00.013+01:00

This blog is earlier published on SAP Community Network Blogs

In today’s market the interest is growing to integrate the structural business processing of SAP within the familiar workplace environment of the Information Worker. These enterprise workplaces are in more and more companies provided by means of SharePoint 2007/2010. To achieve the SAP / SharePoint integration on structural and future-proof manner, requires investigation at forehand to come up with a solid interoperability architecture.

Main steps to define a solid SAP / Microsoft.NET interoperability architecture

Derive and define guiding principles; originating as first from the business perspective, and next from IT. Typically the latter are more of constrictive nature; e.g. required to apply a service architecture, required to conform to W3*-standards, ...
Analyze the current state of the IT landscapes (‘IST’) within the company: SAP and Microsoft environments and server products.
Define and describe the interoperability architecture
- Conceptual level; on purpose technology and product agnostic, to make it more timeless and future-proof
- Concrete level; with a choice for interoperability technologies and products that are nowadays available
Validate the interoperability architecture by means of either a Proof-of-Concept, or via a small launching interoperability project.
Adjust the defined interoperability architecture on the lessons learned.

Guiding architecture principles

Layered architecture, with separation of concerns. A typical layer architecture is:

Presentation
Integration
Application
Data

Service architecture
Loosely coupled
SAP business backend is and remains responsible for the correctness of business process
Responsibility of business data consistency within the SAP backend layer

Approaches to derive the interoperability architecture for a concrete case

When in context of a concrete application, there are basically 2 approaches you can apply to derive the layered interoperability architecture:

Inside-Out
Outside-In (nb: in Microsoft terminology, this approach is also referred to with the phrase ‘Contract-First’)

Inside-Out

As the name already suggests, the starting point here is your current SAP landscape state. Start with identifying the available functional building blocks in the SAP environment – existing BAPI Function Modules, RFC’s, SAP workflow business objects, and already available SAP webservices. And expose these to outside world, to have the related SAP data and functionality consumed by a non-SAP front-end.

Advantages

Biggest advantage of this approach is that you have a faster time2market. You can base your SAP / Microsoft.NET interoperability on already existing, and thus also tested, SAP building blocks. The only thing that needs to be done is to put a (web)service interface on them, and then you can integrate with the Microsoft based presentation layer.

Disadvantages

This can be summarized with the phrase ‘Garbage In – Garbage Out’. If you base your interoperability architecture on the current state of your SAP environment, there is a large likelihood that SAP-proprietary concepts will be visible on the integration surface level. In general, an architecture that originates via this approach will be less pure and transparent. And thus less future-proof.

Outside-In

The essence of the Outside-In approach is to first agree on and define the conceptual contract [interface] between the service provider side [SAP], and the service consumer side [SharePoint]. The idea is to start from the requested application functionalities. Derive and define at a conceptual level the services you require from the SAP backend to deliver the application and system functionalities. Describe the service interfaces in W3*-standards based notation and data structures. From here on, map onto required SAP building blocks: existing if available, new ones otherwise. At SharePoint / presentation side, you can build the consumption layer for the defined service interfaces via wsdl.

Thus start at ‘outside’ with the conceptual, externally visible service interfaces; and continue then for both provider and consumer / SAP and Microsoft sides to the inside with their respective technologies.

Advantages

Because you start in this approach with a green field, the resulting interoperability architecture typical has a cleaner interface, which inherently conforms to interoperability standards. And because you start from the required business services, it also has a better chance on being conceptual correct, and future-proof.

Disadvantages

Biggest disadvantage is that this approach requires more investment and time at front in deriving and describing the service interface layer. Also it requires to get both SAP and Microsoft departments representatives on par, to have a common understanding of the applied service concepts. And in case of existing SAP building blocks, a transformation can be required to map the standards-based service interface to the SAP-specifics Function Modules, RFC’s, workflow business objects.

SAP project "Gateway" and Duet Enterprise

2010-10-28T22:47:00.010+02:00

Project "Gateway" was prominent on the agenda of this years SAP TechEd. In an interview with SearchSAP.com, SAP’s Chief Technology Officer Vishal Sikka got into more detail on the role and proposition of Gateway, and the relation it has with the Duet Enterprise product.

Some remarkable statements, and messages derived from this interview:

Gateway is the bridge from the existing system, e.g. [SAP ERP] 4.6c application with all customizations, to the new world – SharePoint, Facebook, Twitter; without requiring your SAP customers to upgrade to a new version which is inherently service enabled.
Gateway is a mechanism to enable an existing SAP system that is dark to the outside world, and to put windows on it.
SAP Enterprise Services are usually large, and they are designed for process consumption, not for UIs.
The Gateway, think of it as a protocol adapter, that you can attach to a legacy SAP system and have it speak to the world outside.
Duet Enterprise is a product we have built with Microsoft that has the Gateway inside it that has the abilities to connect the world of SharePoint.

Duet Enterprise Overview presentation at SAP TechEd 2010

2010-10-24T13:05:00.024+02:00

Virtual SAP TechEd 2010 presents the recorded 1-hour session CD109 - Duet Enterprise Overview: Consume and Extend SAP Applications Through Microsoft SharePoint and Office. The presenters were from SAP AG, Microsoft Corp and CapGemini. In this session the business goals of Duet Enterprise are outlined, at high-level the advantages it brings for companies doing or considering SAP / Microsoft interoperability, discussion of the architecture and runtime flow, and the development toolset + process. Accompanied by (life and pre-recorded) demo's.

Some of the take-aways of this presentation:

Duet [Enterprise] Service Consumption Layer is the first version of SAP's project "Gateway", embedded in Duet. The aim of project "Gateway" is provide simplified access to SAP:
SCL, internally referred to as 'Gateway Layer'

Framework built as an NetWeaver ABAP add-on, enabling simplified access to SAP software from any device or environment using standard market protocols

Primarily, Duet Enterprise is working as an Add-On on both sides; NetWeaver stack and SharePoint stack
[SAP service-enabling] Business entities from SAP available as Web Services; [SAP consumption] Discover SAP information as External Content Types in BCS, and connect to SharePoint/Office.
The ready to use Duet Enterprise capabilities, are also building blocks that you can use and even build upon in your custom solutions
SharePoint Duet Enterprise front-end development distincts 3 Solution Types:
1. Simple - by end-user; within the SharePoint GUI
2. Intermediate - by Power User; via SharePoint designer
3. Advanced - by a .NET developer; via Visual Studio

Study on the effort it takes to develop SAP/SharePoint integration learns that without Duet Enterprise it costs 6 times
Create solutions very fast to address business (needs) very quickly. Not create very big applications, but small ones and do it fast
SAP services are fairly complicated, with nested and complex structures. SAP WSDL's kinda have their own definition, slightly away from the WS*-standards
Duet Enterprise can either map SharePoint users to SAP users through the Duet Enterprise User Mapping component, or connect to a LDAP that this mapping.

Steps HowTo begin with SAP / MS interoperability

2010-10-02T20:07:00.011+02:00

Important aspect of applying SAP-Microsoft interoperability within a company IT landscape, is how to start. Almost always the Microsoft and SAP departments form their own community in the company, with only limited contacts and co-operation between them, and no clear understanding of each others platforms, technologies and capabilities. See also earlier posts of Kristian Kalsing [1] and of myself [2] addressing this issue.

Raymond Smith [Microsoft Corporation] published an interesting article on this: Building an SAP/Microsoft Interoperability Team. In his article he distinguishes and details on the following aspects:

Aspects of Building an SAP/Microsoft Interoperability Team

Build a SAP Interoperability Lab
Determine a Proof of Concept Business Process
Build Your Interoperability Team
Cross Train Developers
SAP Interoperability Training
Determine Architecture
- Security
- SAP Architecture
- SharePoint/.NET Architecture
POC Development

I recognize [most of] these steps from own experiences with SAP / MS interoperability implementations. For the details, I strongly advice to read Raymond's article.

Debugging Error upon creating MySite

2010-09-25T16:52:00.007+02:00

In a SharePoint intranet, feature stapling is applied to the standard MySite site definition to customize and extend the newly created employee’s MySite’s. After making some requested changes to the custom MySite masterpage, I deployed the SharePoint solution to the development farm. When validating that upon first visit an employee’s MySite is still successful and correct created, I got a runtime error. The display error message gave no details on the problem cause, and in the 12hive log no related information was found. Due a lack of rights to open the EventLog on the remote development server, I could not [myself] check whether in there useful logging was available.

So, what to do? Due the nature of my changes I was pretty sure that the malfunction could not be related back to them. But pretty sure is not good enough, when the ultimate deployment target is the company production intranet. I needed to exactly pinpoint the problem cause, and proof that it was not related to my deployment.

The SharePoint standard MySite creation handling hides the details of the thrown and catched exceptions. To detect what exception was thrown, I therefore created my own DebugMySite.aspx, and filled it with the basics of the standard MySite creation handling – with the noticeable difference to expose the exception details within the browser. Via this debugging approach I learned that the internal error message reported the following:

My Site creation failure for user '<domain>\<username>' for site url 'https://SharePointServer/personal/<username>'. The exception was:
Microsoft.Office.Server.UserProfiles.PersonalSiteCreateException: There has been an error creating the personal site ---> System.ArgumentException: Value does not fall within the expected range.
at Microsoft.SharePoint.Library.SPRequestInternalClass.SscCreateSite(Guid gApplicationId, String bstrUrl, String bstrServerRelativeUrl, Int32 lZone, Guid gSiteId, Guid gDatabaseId, String bstrDatabaseServer, String bstrDatabaseName, String bstrDatabaseUsername, String bstrDatabasePassword, String bstrTitle, String bstrDescription, UInt32 nLCID, String bstrWebTemplate, String bstrOwnerLogin, String bstrOwnerUserKey, String bstrOwnerName, String bstrOwnerEmail, String bstrSecondaryContactLogin, String bstrSecondaryContactUserKey, String bstrSecondaryContactName, String bstrSecondaryContactEmail, Boolean bADAccountMode)
at Microsoft.SharePoint.Library.SPRequest.SscCreateSite(Guid gApplicationId, String bstrUrl, String bstrServerRelativeUrl, Int32 lZone, Guid gSiteId, Guid gDatabaseId, String bstrDatabaseServer, String bstrDatabaseName, String bstrDatabaseUsername, String bstrDatabasePassword, String bstrTitle, String bstrDescription, UInt32 nLCID, String bstrWebTemplate, String bstrOwnerLogin, String bstrOwnerUserKey, String bstrOwnerName, String bstrOwnerEmail, String bstrSecondaryContactLogin, String bstrSecondaryContactUserKey, String bstrSecondaryContactName, String bstrSecondaryContactEmail, Boolean bADAccountMode)
at Microsoft.SharePoint.Administration.SPSiteCollection.Add(SPContentDatabase database, String siteUrl, String title, String description, UInt32 nLCID, String webTemplate, String ownerLogin, String ownerName, String ownerEmail, String secondaryContactLogin, String secondaryContactName, String secondaryContactEmail, String quotaTemplate, String sscRootWebUrl, Boolean useHostHeaderAsSiteName)
at Microsoft.SharePoint.SPSite.SelfServiceCreateSite(String siteUrl, String title, String description, UInt32 nLCID, String webTemplate, String ownerLogin, String ownerName, String ownerEmail, String contactLogin, String contactName, String contactEmail, String quotaTemplate)

With these error details, I could do a focussed search on the internet. And found this related forum-entry Error creating MySite: There has been an error creating the personal site, which had the answer: "add the account configured as the Application Pool Identity for the web site to the "Farm Administrators" group through the Central Administration pages". This was the situation earlier for the intranet webapplication on our development farm, but somehow this got broken. With this knowledge I went to our IT pro, and asked them to restate the AppPool identity as a farm admin, followed by an IISReset. After that, the MySite creation problem was solved.

Duet Enterprise - Technical Overview presentation

2010-09-04T21:59:00.008+02:00

Upon searching on 'Duet Enterprise', I came accross this posted presentation with technical overview. Next to the expected high-level on the Duet Enterprise proposition, this powerpoint also contains some more in-depth explanatory visualizations of the technical and system flow in capabilities like authorization and single sign-on, SAP role based authorization, workflow, monitoring.

Note, the presentation is [declared] prelimary; in awaiting of RTM.

More [functional] robustness issues with OOTB InfoPath Forms Services

2010-08-26T09:11:00.030+02:00

Earlier I blogged about some robustness issues I encountered with the application of InfoPath Forms Services in an external facing SharePoint site. The basis of the framework developed in that project is re-used in context of the company's intranet. Hereby I ran into some new robustness issues. I'll describe them here, and also the outline of the applied solutions.

Placing XmlFormView webpart on a SharePoint page conflicts with inline server codeblocks in MasterPage
Immediately upon adding the out-of-the-box Microsoft.Office.InfoPath.Server.Controls.XmlFormView webpart on a publishing page in the company's intranet, SharePoint throwed an exception:
Source: System.Web
Message: The Controls collection cannot be modified because the control contains code blocks (i.e. ).
Stacktrace:
at System.Web.UI.ControlCollection.Add(Control child)
at Microsoft.Office.InfoPath.Server.SolutionLifetime.ScriptIncludes.AddCssLinkToHeader(HtmlHead head, String href)
at Microsoft.Office.InfoPath.Server.Controls.XmlFormView.TryToAddCssLinksToHeader()
at Microsoft.Office.InfoPath.Server.Controls.XmlFormView.OnDataBindHelper()
at Microsoft.Office.InfoPath.Server.Controls.XmlFormView.OnPreRender(EventArgs e)
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.WebControls.WebParts.WebPart.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint

First analysis learned that the problem was related to the applied SharePoint MasterPage: it manifested itself with 2 of our custom MasterPages, but not on other custom nor the standard (Publishing) Masterpages. Inspection of the particular custom MasterPages disclosed the following server-side code:
<% Response.Write("<script>var Collaboration_SiteRelativeUrl='" + SPContext.Current.Site.Url + "';</script>"); %>

The inclusion of this inline server codeblocks resulted in the conflict with the server-side operation of the XmlFormView webpart. The solution was to replace the inline server codeblocks-tags '<%' and '%>' approach by a HtmlHead (which is a webcontrol) databinding approach:

<script>var Collaboration_SiteRelativeUrl="<%# SPContext.Current.Site.Url %>";</script>

And in the MasterPage codebehind:

protected override void OnLoad (EventArgs e)
{
base.OnLoad(e);
Page.Header.DataBind();
}

IE 6 gives a runtime Javascript error in case of RichTextBox control on the form
While demonstrating the potential of the self-service WebForms functionality to internal stakeholders, I was asked to modify an earlier published InfoPath form with the placement of a RichTextBox control. After re-publishing the InfoPath form to the SharePoint target site, the form still correctly displayed in the content page; including the added RichTextBox control. However, when positioning the mouse in the RichTextBox control, the IE6 browser (Red: still the company's standard) reported a runtime Javascript error. Also the RichTextBox toolbox was not displayed, leaving the control pretty useless wrt a plain TextBox control.

With these problem symptons, I kinda got a 'Deja-Vu' feeling. It looked very much like the problem earlier encountered with displaying InfoPath forms in IE 6. And indeed: via Javascript debugging I detected that the client-side runtime error manifested itself in the same OOTB InfoPath Inc/Core.js method as before: ErrorVisualization.ComputeAbsoluteTop(). Was then the problem caused by the usage of relative CSS-positioning applied in the custom branding, this time it proved completely InfoPath related. I could easily reproduce the same problem within a standard SharePoint WebPart page, with a standard MasterPage. The explanation of the IE 6 Javascript problem is in the HTML that InfoPath Forms Services renders for a RichTextBox control: an embedded iframe element within the DOM document. Due to the fault behaviour of the IE 6 offsetParent method implementation in combination with the non-robust/error-prone implementation of the OOTB InfoPath method, this results in the runtime Javascript error.The solution was to runtime at browser/client-side expand that OOTB InfoPath method to make it robust for the IE 6 faulty CSS offsetParent implementation.

InfoPath re-publishing adds duplicate site columns to the ContentType

Upon re-publishing an InfoPath formtemplate during the demonstration, I noticed another issue. It looked as if all promoted fields in the ContentType were now duplicate present. A quick internet search learned that this is a known InfoPath feature/behaviour, just something I myself was not aware of upto now. And neither something I was particular happy with. I think it is pretty ridiculous to add a complete other set of fields to the InfoPath ContentType each time you re-publish the associated formtemplate. You have some control-options in the InfoPath publishing wizard to prevent the creation of another set of promoted fields, but this is manual-dependent and thus error-prone; let alone it is also very cumbersone and clumsy for the functional managers (the first-class users of the self-service WebForms functionality in the company portal). Therefore I sought for a manner to afterwards correct the modified ContentType, by removing all duplicate FieldLinks. Ideally did would be done fully automatic in the scope of the InfoPath publishing, and thus transparent and invisible to the WebForms functional managers. Sadly, SharePoint does not have an EventReceiver signalling the creation or update of a ContentType. The solution is therefore to deliver a custom SharePoint application page via which the WebForms functional manager can afterwards fix the InfoPath modified ContentType. He/she selects in the dropdown the ContentType that [potentially] needs a cleanup. The custom application page then inspects
the FieldLinks in the ContentType for duplicate occurences, and removes each one found. The changes (deletions) in the SiteCollection ContentType are propagated to all childs. Also each duplicate detected FieldLink is removed from the Fields collection of each SPDocumentLibrary that is associated with the selected InfoPath ContentType (code snippet).

Handling save conflict within SPList EventReceiver

2010-08-12T07:11:00.013+02:00

Background

Publishing an InfoPath form to a SharePoint site creates a new contenttype in the site. Via the InfoPath publishing wizard you have some control over the structure of the created contenttype. But in essence the control is very limited when compared with explicitly provision a contenttype yourself [see also previous post Administrate data from InfoPath form in a self-provisioned ContentType for further explanation why I prefer this above the automatic created information architecture entities]. However, it is not always possible or viable to explicit provision the InfoPath utilized IA entities. The big selling point of InfoPath is that it enables self-service (web)form-definition by functional management. A key cornerstone in this self-services proces is the InfoPath publishing functionality to allow functional managers to themselves distribute the InfoPath forms to a SharePoint site.
One of the aspects missing with the InfoPath publishing managed contenttypes is including the InfoPath promoted fields in the display form. Displaying all InfoPath form fields is in particular handy when quickly browsing the data submitted to a SharePoint forms library.

Solution approach

To fix the functional shortcoming requires to afterwards make the promoted fields displayed. This is not something you want to burden the functional managers with. The aim is therefore to execute it automatically in the context of the target SharePoint site. The ideal moment would be when the InfoPath managed contenttype is created or updated. SharePoint however does not provide an eventreceiver for these events. The next best moment is when the contenttype is associated with the forms library to which the InfoPath form data is submitted. Here you neither have a direct event notifying the contentype association. But you can receive indirect notification via SPListEventReceiver::FieldAdded. The idea is then to check each added field whether it is an InfoPath promoted field, and if so to alter its definition so that the field will appear in the standard SharePoint display form.

Issues encountered

So, easily said and done. Associate the custom ListTemplate with a SPListEventReceiver, and fill in the FieldAdded method. In my local development image it worked like a charm: after you associate via the SharePoint GUI a document library with an InfoPath contenttype, all promoted fields of that contenttype appear in the document library display form. But after deploying to the central SharePoint development farm, I ran into the first issue. Upon associating a library with a contenttype, SharePoint faulted with a weird message: The specified program requires a newer version of Windows.

Despite this misleading error message, it was immediate clear that the issue was caused by the custom FieldAdded eventhandler:

Apparently, SharePoint does not allow to update a list field in the runtime context of that field being added to a list.

Solution outline

Solution is to isolate the execution of the field-update from the field-addition runtime context. Instead of direct updating the field, schedule this work via a worker thread. One extra aspect to take into account is that the scheduled worker threads can still run into a parallel update conflict:

Save Conflict

Your changes conflict with those made concurrently by another user. If you want your changes to be applied, click Back in your Web browser, refresh the page, and resubmit your changes.

  at Microsoft.SharePoint.Library.SPRequest.UpdateField(
        String bstrUrl,String bstrListName, String bstrXML)
  at Microsoft.SharePoint.SPField.UpdateCore(Boolean bToggleSealed)
  at Microsoft.SharePoint.SPField.Update()
  at WebformulierFieldsHandler.Worker.b__1()

The likelihood of these save conflicts increases with the number of promoted fields in the contenttype, and can already manifest itself when that number is 4 to 5. It can be made functional robust by implementing a retry mechanism in the worker thread.

What about the competition - new roadmap for SAP NetWeaver Portal

2010-08-04T22:52:00.012+02:00

As a SharePoint adept, I don't really consider the SAP Enterprise Portal as competition. Let's face it, the typical UI is horrible, and it lacks many of the capabilities the SharePoint platform brings. But moreover, the impression [within the market] is that SAP has stopped with actual new major developments in this product (Web2.0, content management, social media). SAP is putting it's intellectual resources and budget mainly on the business capabilities, and less on fancy user interaction. Fair enough, as it leaves room for integration and interoperability via foreign UIs, as SharePoint.

Contradicting the above, it's knownable that at SAPPHIRE some new additions for SAP NetWeaver Portal were announced. Most notable the concept of 'Enterprise Workspaces'. Read and see more of this in SAP NetWeaver Portal – SAPPHIRE NOW Summary.

It aims to provide SAP users with an iGoogle- or myYahoo-type experience within SAP, allowing them to build their own pages with structured and unstructured assets -- reports, RSS feeds and so on. It applies a self-service approach within defined roles and security. [SAP NetWeaver Portal roadmap includes 'Workspaces,' easier third-party integration]

Well, the UI impressions and the self-service concepts are rather impressive and promising. I do hold on to my opinion that SharePoint is a better overall portal platform. But it looks as if SAP could be making a new step forward in its own portal product.

Standards-Based Interoperability between SAP NetWeaver 7.0 and Microsoft .NET 3.5/4.0

2010-07-29T23:04:00.007+02:00

Microsoft opened a new interoperability area within MSDN on Web Service Interoperability between the major web services vendors. Amongst these naturally also SAP. The site contains an extensive (75 pages) Demonstration Scenario of the Collaboration Technology Support Center addressing information about how to set up standards-based Web services communication between Microsoft .NET Framework applications and SAP Application Server ABAP-based consumers and providers.

Portal Embedding of SAP UWL into SharePoint

2010-07-28T17:11:00.002+02:00

There are multiple alternatives for utilizing SharePoint as presentation layer to SAP functionality. The most basic is portal launch, next comes portal embedding. Portal embedding has some advantages (mainly being rapidly to achieve: you only visually integrate the 2 [SAP, Microsoft] environments, without any direct systems/applications integration and interoperability), but remains a rather poor man's approach to portal integration. The typical SAP UI / Look and Feel is very different from the SharePoint UI (standard or company-customized), and the SAP Portal pages exhibit themselves browser navigation in addition to the SharePoint navigation.

Andre Fischer a.o. from SAP recently published the whitepaper Integration of SAP Universal Worklist into Microsoft Office SharePoint. This paper contains an example of the portal embedding approach, while also addressing the double navigation issue. In particular it describes an approach to integrating the SAP Universal Worklist into SharePoint context. Basically it comes down at making sure at the SAP Portal side that the portal pages are rendered headerless, without the portal navigation. The elegance of the sketched approach is that it is limited to SAP Portal configuration and customization work, there is no custom development required at either SAP nor SharePoint side.

SharePoint 2010 CMIS Connector

2010-07-22T21:20:00.011+02:00

To enable content management interoperability between different CMS repositories, the OASIS organization defined the Content Management Interoperability Standard specification. Although a.o. Microsoft played a major role in the OASIS working group defining this proposed standard, up to now it did not provide a concrete and product-ready implementation of the CMIS specification for SharePoint 2007. There is only a reference/example implementation available on MSDN, with code download.

Apparently Microsoft decided to wait with active support of the CMIS specification for the next SharePoint version. For SharePoint Server 2010 Microsoft now packages a CMIS connector as part of the Administration Toolkit v1.0. The CMIS Connector for SharePoint provides a CMIS (Content Management Interoperability Services) interface over the top of SharePoint as well as a CMIS consumer Web Part that can be used to display content from other CMIS enabled repositories. [source: Microsoft SharePoint Teamblog] Or stated otherwise: via this CMIS Connector, SharePoint Server 2010 can both function as CMIS consumer of content stored in external repositories, and as CMIS producer of content stored within SharePoint lists and libraries.

Mind you...

the CMIS Connector is not supported for SharePoint Foundation 2010
and there is still only limited support of the CMIS standard by other CMS vendors; the real value of CMIS requires [at least] 2 to tango...

Beware: timerjob processes invoked via Central Admin deploy solution do not reload FeatureReceiver assembly

2010-07-12T07:00:00.000+02:00

Recent we ran into an exception when deploying an external facing website to the production SharePoint farm. The problem originated in the FeatureInstalled() method of a custom FeatureReceiver class. We did not encounter this particular deployment problem in the integration nor QA environment. Since on a production environment it is not viable to debug, we had to resort to inserting logging statements in the code, and then redeploy the assembly. But strangely, no logging was written leaving us still blank with respect to the exact code location and cause of the deployment problem.

After several deployment attempts together with operations, we came up with the explanation why the logging was not done. The timerjob process that executes the FeatureReceiver FeatureInstalled() method still has the old code loaded!! To force that the assembly is reloaded from either GAC or virtual bin after deploySolution and before execution of FeatureInstalled, you need to recycle the timerjobs processes. The right time for this is after the retractSolution of the previous version of the deployed SharePoint solution package.

NB: we didn't notice this 'faulty' deploy/timerjobs behaviour before because it's not very common to actually utilize the FeatureInstalled() method. In this particular case it was necessary to timely correct something before the SharePoint feature framework performs the OOTB feature activation work.

Duet® Enterprise on stage at SAPPHIRE 2010

2010-07-09T22:55:00.009+02:00

Duet® Enterprise is now clearly gaining momentum in the marketspace. It was also on the agenda of the SAPPHIRE 2010 Conference. Watch the session presented by Pascal Gilbert (Microsoft). In this presentation Pascal discusses at a conceptual level the business proposition of Duet® Enterprise, enlists in what aspects it is an improvement of the previous Duet versions, and gives a real life example of how Duet Enterprise can be put in sensible action applying strictly the OOTB provided capabilities.

Another short demo of what can be achieved with Duet® Enterprise for business collaboration in a mixed SAP (business processing platform) and Microsoft (Information Worker environment) landscape is given by Howard Beader:

SAP announces a new release of the SAP .NET Connector

2010-06-30T17:25:00.005+02:00

In 2003 SAP released the SAP .NET Connector (NCo) to enable communication between the .NET platform and SAP systems. The first NCo versions (1.0x, 2.0) were aimed at .NET 1.x (with support for Visual Studio .NET and later on VS 2003), and it remained at that level. This resulted in the overall conclusion that the NCo was considered 'dead', or at least strongly outdated (Future Support of NCo). This idea was re-enfored in May 2009 by 1) the advice of Juergen Daiberl to use webservices or WCF Lob Adapter in favor of NCo, and 2) a statement of SAP's Rima Rudnik-Sirich phrasing that 'ES Explorer is a successor of NCo'.

However, the truth (or weakness) of SAP Enterprise Services is that currently only a limited set is available, supporting a restricted subset of the SAP business processes functionality. There are alternatives to custom develop SAP/MS intergration whenever it is not standard provided: WCF on SAP webservices, WCF BizTalk Adapter on SAP BAPIs and RFC. And forthcoming is Duet Enterprise, with a positioning for interoperability and collaboration between SharePoint 2010 and Office 2010 clients towards SAP processes and data.

And now SAP apparently decided to give new life to the NCo proposition. They have announced .NET Connector 3.0, to be released in December 2010. At first, this appears strange when you combine this announcement with the upcoming release of Duet Enterprise. However, Duet Enterprise is thus strictly scoped for interoperability between SharePoint plus Office 2010 and SAP environments. A renewed NCo will provide an additional alternative to communicate from other .NET application formats (e.g. WPF, WF, Services) to the SAP systems.

Test, QA and Production SharePoint infra should be comparable

2010-06-22T22:34:00.006+02:00

In multiple SharePoint projects I have been confronted with delivered functionality succesfully validated in the test environment, does not operate the same in the QA environment. Since the application software is identical in the environments, the cause of the malfunctions must originate from differences at infrastructure level. Especially when it occurs in a later project stadium, this can be a very unpleasant surprise. Of course it's not realistic to expect nor demand that the Test, QA and Production environments are 100% comparable. However, the differences should be limited to hardware resources level; number of front-end servers, sizing of database, cpu, ... The infra architecture and configuration should be [logically] the same; load-balancing, firewall settings, proxy, Central Admin settings (e.g. Search, SSO, Policy for Web Application).

Extra complication to achieve such an ideal Test/QA/Production setup, is that typically the Test environment is much more volatile. This is the first environment to try-out your stuff, be it application functionality, infrastructure settings, or both. And in case of failure, not always the deployed trial stuff is rolled back. This is an extra argument to use virtualized Test environments per SharePoint project. When a new project start, an own Test environment is provisioned, with the current production state as its baseline. The project deploys all the project deliverables to this own Test environment. There is no overlap or conflicts with the work from other projects. And after project end (successful completion, or earlier stopped), the project Test environment can be deactivated and retired.

Installing CU 2010.0 corrupts local SharePoint environment

2010-06-20T14:45:00.003+02:00

As at many, at my current customer deployment we're running our local SharePoint development environments in a virtualized image (currently Virtual PC; before it was VMWare Workstation). Recently we developers were informed by the SDE team on the availability of Cumulative Update 2010.0. Main reason to install that CU within your own local development environment is to come in sync again with the SharePoint installation state at the hosting platforms (development, QA and production).

Last week I installed the update. To discover next that my local SharePoint environment was broken! None of the local SharePoint sites would start up, including Central Admin. Running the Configuration Wizard also halted with an error. The eventlog contained the following error: The schema version (3.1.8.0) of the database WSS_Content_Project on <Machinename> is not consistent with the expected database schema version (3.1.11.0) on <Machinename>. Connections to this database from this server have been blocked to avoid data loss. Upgrade the web front end or the content database to ensure that these versions match.

Via this message I found the cause + solution in the following post: The schema version (3.0.149.0) of the database sps_content_db on MOSS. And indeed, after manually modifying for each SharePoint content database the version to 3.1.11.0 (in my case), all local SharePoint sites came back to live again.

Duet® Enterprise on stage at seminar 'SAP and Microsoft BI'

2010-06-15T22:38:00.008+02:00

Recently I attended the Microsoft seminar 'SAP and Microsoft BI'. The focus of this seminar, and the audience, was evidently on Business Intelligence in a heterogenous SAP + MS IT landscape. But within this scope, also the forthcoming DUET Enterprise receives it's role. Juergen Grebe of the SAP / MS Alliance team had a good session on the interoperability architecture and the provided capabilities of DUET Enterprise.

White paper discussing the promise of Duet® Enterprise

2010-05-23T22:46:00.015+02:00

CITO Research has published The Business Value of Duet Enterprise, a research white paper sponsored by Microsoft. Such sponsorship always contains the risk of the result being more of an advertorial instead of an independent and thorough analysis + evaluation. This paper at locations certainly qualifies as unconditional product promotion, but it also presents several useful insights and details on the mission and capabilities of Duet Enterprise.

The paper discussions the mission and values of Duet® Enterprise from different viewpoints and angles:

[as it should be] Starts with the business vision; the why
the architecture and interoperability approach; the how at high level
examples of what can be achieved; use cases
the components of the Duet Enterprise foundation; it's building blocks for composing applications that surface SAP line-of-business content and processes in user interface formats tailored to the needs of the nowadays Information Workers
the personality of composites that are best realized via either Duet Enterprise, Business Connectivity Services, or rather BizTalk; choosing the right interop tool for the task/scenario at hand

Some of the most noticable points to take from the paper are:

The statement that although Duet® Enterprise relies on SAP NetWeaver 7.02; it can handle scenario's where a company's SAP environment consists of many generations of SAP applications. From SAP R3 4.6C to SAP ERP 6.0 and beyond.
Duet® Enterprise is SAP / MS interoperability foundation; for rapid development and construction of custom applications
Duet® Enterprise enables the rapid development approach by providing the needed interop plumping
Duet® Enterprise builds on existing investments in SAP [business applications + architecture] and Microsoft [Information Worker] software.
Duet® Enterprise fits in with existing modern IT management, development processes and environment/technologies.
Duet® Enterprise is one way for realizing the vision of Office Business Applications when it concerns connecting with SAP line-of-business.

Missing in the paper is a thorough consideration of the strengths versus weaknesses; of the product and of the proposition itself. With respect to that, Duet® Enterprise could be seen as a threat to some of the SAP business. Although it's promise is to extend the reach of SAP business capabilities beyond its traditional usage and departments, it does so via Microsoft instead of SAP software. It will be interesting to watch hows this develops, and what it will mean for the business of both SAP as Microsoft. Ideally, both will prospher from it. Only then, the combined effort will truly deliver, and have a future.

Claims-based SSO with SAML 2.0

2010-05-17T22:07:00.007+02:00

On SAP Community Network the presentation Next Generation SSO for SAP Applications with SAML 2.0 is published containing an overview about authentication, single sign-on (SSO), and identity federation for SAP centric applications. Combined with the recent release of Microsoft's Claims-Based identity platform (Windows Identity Foundation, formerly known as Geneva), and IBM enabling Identity Federation via its WebSphere Application Server (Identity federation using SAML and WebSphere software); 3 of the largest software companies are taking significant steps on this requisite for the interoperability puzzle. Of the remaining other large software companies, Oracle has an handle to the OpenSSO product via the SUN acquisition. But doubt rises about the willingness and commitment of Oracle to OpenSSO - a first sign of this is demonstrated by some broken links to OpenSSO information. From the Google camp I yet miss notion of their plans and products on the Claims-Based Identity subject. I expect that it will either originate from the Open Source community, or donate to it. Who knows, perhaps Google (allthough IBM is a more likely candidate) would be the ideal new founder for OpenSSO...

Unable to recover from a corrupted Publishing infrastructure

2010-05-11T23:23:00.009+02:00

For a functional extension to the intranet (enterprise portal) of a client organization, we needed to provision a content page to an existing site. A precondition is thus that the Publishing infrastructure is activated within the context of that SharePoint site: at Site level (PublishingSite feature) and Web (PublishingWeb feature). Upon activating the latter we encountered the following error: Provisioning did not succeed. Details: Failed to create the 'Pages' library. OriginalException: The feature failed to activate because a list at 'Pages' already exists in this site. Delete or rename the list and try activating the feature again.

Strangly enough: via the SharePoint GUI the Pages library was not visible. Could be that it was set to hidden. But trying to directly navigate via its url merely resulted in a NotFound. Upon opening the site collection in SharePoint Designer, it showed there was a left-over of the Pages library: merely an empty folder, no Forms nor other content. But this presence was in the way of activating the Publishing feature. So we deleted this folder via SPD. And then tried to activate PublishingWeb feature again. This time it progressed a bit further, to stop with the message: The page you selected contains a list that does not exist. It may have been deleted by another user. Inspecting again: the Pages library was correctly available now, including associations with PublishingPage contenttypes. Missing this time was the Style Library documentlibrary. That is, when inspecting via SharePoint GUI. Again looking at filesystem level via SharePoint designer: same situation, Style Library folder present, but empty. Deleted this folder, and retried to activate the PublishingWeb feature. Sadly we kept on running into the error of missing List. Even after manually adding the Style Library; forcefully deactivate and re-activate the PublishingWeb feature. We did not manage to recover from the initial corrupted situation wrt Pages list. Search results on the Web for information on both the error messages gave some hits to problem descriptions, but sadly not to resolutions.

Due to a pressing time schedule, we therefore had to resort to a pragmatic alternative. Deleting the corrupted site collection, and re-creating it. This was a viable approach due to the current nature of the site collection: containing no content yet. But it leaves me with somewhat of a frustrated / disappointed mind. I would rather have seen us able to recover from the incorrect situation, while preserving the site collection.

A Technical Overview of Duet Enterprise for Microsoft SharePoint and SAP

2010-04-21T20:09:00.004+02:00

On the Duet Enterprise Team Blog an article is published discussing on high-level the technical architecture of Duet Enterprise, the positioning within a mixed SAP and Microsoft landscape, and the required products versions within these landscapes: SAP NetWeaver 7.02 together with SharePoint Server 2010 Enterprise Edition.

A perspective from within SAP AG on Duet Enterprise

2010-04-06T20:08:00.015+02:00

Up until now, most news on and attention given to the new Duet Enterprise proposition seemed to come from Microsoft only; leaving SAP AG rather silent on this subject. I am therefore pleased with a first concrete signal from SAP commenting and committing onto Duet Enterprise. Somya Kapoor from the Duet Enterprise Solution Management Team at SAP recently shared some thoughts from within SAP on the Duet Enterprise proposition.

Some noticable statements in this short interview:

"Duet Enterprise is primarily focussed on integrating SAP and SharePoint - as opposed with Duet which focussed on integrating SAP and Microsoft Office"
"We [SAP] are aiming at customers who have SharePoint deployments, or experiences with it;
"We [SAP] want to bring the collaboration expertise of SharePoint into the SAP business processes in the back-end.

A compact overview of SSO Technologies supported by SAP NetWeaver for surfacing from .NET based clients

2010-04-01T21:00:00.002+02:00

Earlier I blogged some about the area of Single Sign-On in a mixed SAP NetWeaver – Microsoft .NET world. Yesterday Andre Fischer from SAP AG published an up-to-date compact overview on this topic. Recommended reading for those interested in SAP / MS interoperability.

Efficient batch deletion from SharePoint List without filling up the site RecycleBin

2010-03-31T19:27:00.009+02:00

For a data management function I need a performant way to first delete all items from a SharePoint list, before re-filling it with up-to-date content. Looping through the list and delete each item one-by-one is not acceptable. It would instantiate an internal SPRequest object for each single listitem deletion, and take a longer time to complete. The answer to do it time-efficient is to use the SPWeb method ProcessBatchData(). You feed this method with a batch string of multiple delete commands to perform in a single SharePoint request.

However, usage of ProcessBatchData has the disadvantage that all deleted items are put in the SharePoint SPSite Recycle Bin. Kinda ridiculous for a programmatic batch-based deletion. And because of the larger number of deleted items (the reason for doing the deletion batch-based...), it makes the Recycle Bin pretty much unworkable thus useless for UI initiated restore of manual deleted content items.

SharePoint does not provide an elegant way to prevent this standard behaviour. Elsewhere on the web it is suggested to temporarily disable the Recycle Bin at SPWebApplication level. This however suffers from 2 drawbacks:

The current user needs to have the SPFarm administrator role. This is unlikely for a functional management role, and unacceptable from a security viewpoint.
Disabling the Recycle Bin at webapplication level has a nasty side-effect. It namely clears all Recycle Bins in the webapplication. Not only that of your current context site and sitecollection, but all of them in the webapplication. This effectively destroys the Recycle Bin backup-purpose for manual deleted items; and it is thus not isolated to your scope but affects the entire webapplication. This is functional unacceptable.

So if it's not possible to prevent the batch deleted items from appearing in the Recycle Bin, it is then required to delete them there afterwards. This could be done via a call to SPContext.Current.Web.RecycleBin.DeleteAll(). But this clears the entire Recycle Bin, still potential removing too much. What we need is an approach to delete exactly those items from the Recycle Bin that were put there as result of the list batch-deletion. The Recycle Bin has a different API interface as regular SharePoint lists. Usage of ProcessBatchData to delete the same relevant items from the Recycle Bin is not possible. But it is undesirable to first clear the SharePoint list via a batch-deletion, and next be forced to loop-based remove the same items from the Recycle Bin. Luckily the SPRecycleBinItemCollection class exposes its own method to issue a batch deletion: SPRecycleBinItemCollection.Delete(GUID[]). It has a different method signature, which requires you to first determine the GUID per SPRecycleBinItem. Also you must take care of keeping the chunck of deletion below a thresshold. SharePoint namely locks the database tables when executing the Recycle Bin batch deletion, which hangs the SharePoint server upon deleting a larger set. The Recycle Bin cleanup/reset can be done asynchronous in the background, doing it chunk based.

The code for efficient batch deletion all items of a list, and next clear the same deleted items from the Recycle Bin is as follows:

1. Control for time-efficient purge all items from SharePoint List

2. Cleanup the Recycle Bin in the background

Beware: usage of ImageButton can cause a 401 NotAuthorized in external facing site

2010-03-29T23:21:00.016+02:00

In one of our custom webparts we are using the ASP.NET ImageButton class. Instead of setting the image-source via the ImageUrl property, we specify the image via CSS. All works fine. That is, until we "got anonymous". Suddenly when browsing to a page with our WebPart on it, the (acceptance test) end-user was confronted with the IE loginbox. Via Fiddler (how I love this http debugging tool) I quickly detected the indirect cause: IE issued an HTTP GET for the URL of the Pages document library. In an authenticated mode this will pass unnoticed. But for an anonymous user the particular HTTP GET will result in a 401 NotAuthorized on trying to directly access the Pages folder URL.

So the next question was what caused the browser to issue this request for the Pages URL? Well, this was the direct result of our on intention lack of setting the ImageUrl property, in favor of flexible CSS specification. The resulting HTML is then something like:

IE (6) browsers interpretate the empty image-src attribute as referring to the relative root of the webpage. In our case with the WebPart placed on a publishing page, this translates to a request for the Pages folder URL. FireFox and Chrome do not exhibit this behavior, they simple ignore the empty source attribute as it should. The solution for preventing IE to request the NotAuthorized Pages URL is to set a non-empty value for the ImageUrl property. It’s not even required to point to an existent image file. In that case the HTTP request will silently result in a 404 NotFound HTTP response. And since the image is set via CSS, the visual effect in the UI remains unnoticed for the end-user. I dislike polluting the HTTP transfer between browser and server with requests which are known before to fail. Therefore I point to ImageUrl to the current and available image-file. Via CSS it is then still possible to later flexible modify the displayed image.

FormsService SSP in farm holds on to previous version of Feature re-deployed InfoPath Form

2010-02-23T19:33:00.027+01:00

Earlier I blogged about the ALM manner to develop and deploy InfoPath forms. One major benefit of this approach is that the InfoPath forms are considered as first-class source artifacts in the development environment and project, just as ‘regular’ source code (C#, XML, .js, css). An even bigger benefit is upon deployment: no need for manual actions to publish the form templates to your deployed and provisioned SharePoint application, it’s all done automatic in the context of a feature activation.

SharePoint out-of-the-box provides the XsnFeatureReceiver class to provide the InfoPath Form Templates publishing support. However, the functionality of this standard class is not enough for real-life deployment scenarios. I ran into a few missing parts before, which I handled by extending and augmenting the XsnFeatureReceiver class. This week I was confronted with another issue. After a re-deployment with updated InfoPath forms, the earlier deployed forms were still active. That is, when opening them in the browser. When opening in InfoPath client, the new version was applied. Thus the deployment to SharePoint server was performed, but somehow for InfoPath Forms Services context the earlier deployed version per InfoPath form remained active. I did not experience this effect in my local single-server local development image. But when doing a re-deployment to the distributed test farm-environment, the issue manifests itself.

I inspected the managed forms templates administration of the FormsService SSP, and discovered that still the initial deployed versions were administered. And after uninstalling the InfoPathInfrastructure feature, the InfoPath form templates in FormsService were not cleared. This obstructs the activation of the new versions of the forms upon re-deployment. As solution I extended my InfoPathInfrastructure feature to also make sure the InfoPath forms that it installed, are removed from the FormsService SSP upon uninstalling the feature.

With this fix, I’m able to deploy and activate new versions of the InfoPath forms to the distributed target SharePoint farm environment (test, staging, production) for usage via InfoPath Forms Services. And still conform ALM principles.

SharePoint [2010] as development, composition + application platform

2010-01-29T20:15:00.012+01:00

I consider SharePoint having 2 distinct faces:

It's in itself an application, with lot's of end-user oriented functionality. This is true for the foundation WSS, and significant more for the licensed version MOSS 2007
It's a rich development and runtime application platform, to host your own custom developed applications.

Being primarily development-oriented, it will be clear that personally I like SharePoint best for the second role.

I'm not alone on this. More and more custom company-internal applications are based on and within the SharePoint platform. SharePoint comes out-of-the-box with a well-filled toolbox of both technical as functional building blocks. Developing custom SharePoint applications is therefore a combination of composing and customizing OOTB SharePoint building blocks, and augmenting it with custom developed functionality in which the SharePoint platform does not standard delivers. This implies a significant mindset switch for [old-school] application developers. Instead of the default decision to custom build all application parts, the SharePoint best-practice approach is to first consider the OOTB building blocks. Can you deliver the custom-requested functionality on sufficient level via the application of the standard SharePoint blocks? If so, choose that path. This composition-based approach will gain you in the initial application development stage. But even more it will be cost-benificial within the application maintenance period. It's better to have Microsoft fix and improve on standard building blocks, than have to do it yourself on your own proprietary ones. Of course, this principle can especially be applied on the more technical functionalities. And this gives you more time and relative budget to focus on the differential functionality for your company.

David Chappell has written a conceptual white paper highlighting the application platform aspects of the forthcoming SharePoint 2010. You can download it on the Microsoft site, SharePoint 2010: Developer Platform White Paper.

AllUsersWebPart and SPWebApplication.ApplyWebConfigModifications don't match

2010-01-26T07:00:00.002+01:00

In a Feature, I apply the AllUsersWebPart construct to automatically add default webparts to all publishing pages based on the PageLayout File.
In a Feature, I utilize the SPWebConfigModification class to add web.config modifications

When done in separate Features, both constructs operate successfully. However, when included within the same Feature activation, the invocation of ApplyWebConfigModifications method results in a SecurityException:

Access Denied
at Microsoft.SharePoint.Administration.SPPersistedObject.Update()
at Microsoft.SharePoint.Administration.SPWebApplication.ApplyWebConfigModifications()
at Microsoft.SharePoint.Administration.SPWebService.ApplyWebConfigModifications()

It looks as if internally SharePoint somehow puts a lock on the Administration persistent object. I tried to verify my suspicion via Reflector, but not surprisingly this internal code is obfuscated. The runtime error only manifests itself when the Feature is (re)activated via the GUI. The Feature activation via stsadm reports no problem, and successfully performs the deployment work.

Still, I want to hold on to the idea of one single self-contained Feature to deploy all the required parts. And I want to be able to turn the Feature on and off interactively via the SharePoint GUI (so that I can also operate the feature remote, without access to the deployment server). A resolution is to apply another approach to provision the default webparts on the pagelayouts. Besides the usage of AllUsersWebPart, it is also possible to directly include the webpart specifications in the PageLayout file itself:

With this construction, the invocation of SPWebApplication.ApplyWebConfigModifications method is done successfully. And as a bonus, also another problem is prevented; that of magically duplicating the default webparts provisioned via AllUsersWebPart upon Feature re-activation.

A final word of caution. With this approach I at first encountered another problem. When editing on a created publishing page the properties of the default WebPart provisioned via the PageLayout file, SharePoint displayed an error warning in the edit toolpane, "a web part you attempted to change is either invalid or has been removed by another user". Although the net effect of the WebPart settings is simple done, this is not very trustwordy towards the Web Content Manager. Hard to explain that they can just ignore this warning. Before deciding to then have to abandon the approach, I re-examed the default WebPart specification in the PageLayout file to see if anything there could be causing SharePoint to suspect a concurrent update. And yes there was, 'thanx' to the automatic editing behaviour of Visual Studio. When you add a control within an .aspx file, Visual Studio automatically add a default 'ID' property to the control. For the default WebPart however this is not needed, the property is actually set when creating a publishing page off this PageLayout. But the mere presence of it in the default part results in SharePoint detecting a concurrent update. So remember to don't include the 'ID' property within WebPart specifications included within PageLayout files.

Tip: psconfig saved my local SharePoint environment

2010-01-20T07:15:00.003+01:00

Once in a while, your local SharePoint installation may become corrupted. It happened to me today. I experienced diverse problems and symptons. Most noticable the malfunctioning of SharePoint search: crawling no longer worked, connect to the SharePoint Search service gave a "not able to access remote service". And when I inspected the eventlog, I detected several Office Server Search errors:

A seemingly total other problem was upon site collection creation via a custom site definition. This site definition applies a custom SPWebProvisioningProvider. Here I encountered a problem in loading the assembly of that provider, due to apparently missing referenced (system) assemblies.

When I next activated custom FeatureReceiver code that programmatically manages the Search content sources, SharePoint threw a Microsoft.Office.Server.Search.Administration.DeletedConcurrencyException with message Object e26da097-50ff-4386-b3f6-2b2b30f73b96 not found. This was the start to fixing my environment... I found via searching on DeletedConcurrencyException a useful advice to run the SharePoint admin tool psconfig:

psconfig -cmd upgrade -inplace b2b -wait

And indeed, after doing that magically the above described problems have vanished. That saved my day, I already feared that I had to re-install my local SharePoint environment.

PublishingWeb.IsPublishingWeb malfunctions on custom site definition

2010-01-19T07:00:00.002+01:00

I typically apply a Feature-based approach for the provisioning of a SharePoint site into a full-functioning application . I prefer this above putting everything within a custom site definition. I'm in good company with my resistance against (overly complex) Onet.xml's [Andrew Connel, You don't need to create site definitions, Joel Oleson, Do you Really Need to Create Custom Site Definitions?]. I actually reuse the same site provisioning engine functionality within multiple SharePoint projects and applications. The reuse is made possible by specifying the provision actions via an XML feed file. Via this, the engine can be instructed to provision structure (Lists, Libraries), content (publishing pages, ListItems), and configuration (web.config via SPWebConfigModification, and PublishingWeb + SPWeb properties).

With respect to the latter, I discovered that the OOTB PublishingWeb.IsPublishingWeb sometimes plays tricks on you. Let me explain. In the provision engine I've implemented functionality to set diverse navigation settings: IncludeSubSites, NavigationShowSiblings, InheritGlobalNavigation, and so on. These properties can only be set on publishing webs, and are in the SharePoint Object API available via a PublishingWeb instance. In the engine code I first do a check whether the SPWeb on which the Feature is activated, is actually a PublishingWeb. And only then via PublishingWeb.GetPublishingWeb access a valid wrapper reference, and set the navigation properties.

This all worked fine in situations in which I applied the provision engine afterwards on an already created Publishing site collection. In my current project I do it slightly different. I've implemented a minimal site definition, and within the context of it activate per web in the created site topology the Publishing feature, and next execute the provision engine. In this setup, the navigation settings appeared to be ignored by the provision engine. When I debugged the provision execution, I discovered that PublishingWeb.IsPublishingWeb returned false for all webs in the site topology. Very strange, since I made sure to first activate the Publishing feature...

I ran into the forum-thread IsPublishingWeb() - what does it REALLY check for? posting a question on this issue. And followed up on the advice given there: implement an own SharePointHelper.IsPublishingWeb method to determine whether the SPWeb instance is valid for being wrapped into a PublishingWeb instance.

BTW: My own experiences learn that it is not due the asynchronous site creation behaviour. Also when later via site settings (de)activating the provision engine feature, PublishingWeb.IsPublishingWeb remains returning false, despite that the Publishing feature is activated on the (sub)web. It looks more as if PublishingWeb.IsPublishingWeb is somehow dependent on something that is set behind the curtains via OOTB Publishing site definitions.

Aggregate the values from an InfoPath repeating section for data administration

2010-01-15T07:00:00.009+01:00

A common requirement for setting up (web)forms is to enter multiple rows of similar information. For example, entering the name(s) of your child(ren) for an holiday booking. The InfoPath Repeating Section enables an intuitive usage model for this. Initial the form displays only 1 section for data entrance. Via a link or button the user can on-the-fly expand the form with more occurrences of that same section.

Question is how best to administer the entered data from the repeating section in a SharePoint library. Is this a Master/Slave relationship? Or should it be considered simple as embedded in the form data self? The former requires a relation data model employed within the SharePoint information architecture. Although possible, this is not the typical SharePoint model which is namely to administrate the entity data in a single List or Library. However, for the second alternative it is still required to potentially administrate multiple values per field in the repeating section.

SharePoint supports this via the concept of AggregateFunction at Field level. To collect the data from a repeating section, specify aggregate="merge" for the field within the InfoPath form template

When explicitly self-provision the SiteColumns and ContentType (Administrate data from InfoPath form in a self-provisioned ContentType), you ran into a problem here. The FieldSpecification CAML currently does not allow the Aggregate attribute as in:

<FieldRef
  ID="{5e046132-9284-4322-b1c7-9a742d60270e}"
  Name="FirstNameChild"
  ReadOnly="TRUE"
  Node="/my:myFields/my:group/my:Children/my:FirstNameChild" />

SharePoint will fail the activation of your Feature with the error message: The 'Aggregation' attribute is not allowed.
The remedie is to skip the 'Aggregate' attribute from the CAML specification, and afterwards add it via an EventReceiver to the provisioned SPField in the RootWeb:

  SPField aggregateField = site.RootWeb.Fields[new Guid("{5e046132-9284-4322-b1c7-9a742d60270e}")];
  aggregateField.AggregationFunction = "merge";
  aggregateField.Update();

Administrate data from InfoPath form in a self-provisioned ContentType

2010-01-13T07:00:00.002+01:00

When registrating via XSNFeatureReceiver an InfoPath Form Template in a SharePoint sitecollection, automatically a ContentType and the associated SiteColumns are provisioned. This ContentType enables promotion of the data entered via the form to a SharePoint library. However, I prefer to apply explicit self-provisioned ContentType and SiteColumns for the data administration. The rationale is to have full control over the created information architecture entities. In the automatic created ContentType, the ID is arbitrary and thus differerent accross (re)deployments and environments. This hampers the content deployment process between environments. And at rolling out an upgrade of the InfoPath form template in the same site collection, a new version of the ContentType is provisioned with potential another ID. The automatic provisioned SiteColumns do not expose this particular problem. Their IDs are namely contained in the InfoPath form template, and therefore remain identical accross (re)deployments. But the SiteColumns have their own issues. Most noticable is a SiteColumn with type (Multi)Lookup. This cannot be correctly provisioned via a CAML FieldSpecification only, but requires a fix afterwards (Creating Lookup columns as a feature). Another problem that both the automatic provisioned SiteColumns and ContentType suffer of, is when there are changes in the definiton while the entity is already in use. To propagate the changes at the sitecollection-defined entities to their usage at List/Library level requires some afterwards fix via custom code.

Because of these issues I prefer to explicit self-provision both the SiteColumns and the ContentType for administrating the forms data. You then control the ID of the ContentType, you can fix the Lookup-relationships for SiteColumns, and you can propagate changes within the ContentType to all List/Libraries which already use the type. It costs some extra development time, but it surely pays off upon repeated full-automic deployments, and accross environments.

Server-side generation of InfoPath email

2010-01-11T07:00:00.001+01:00

In a SharePoint based application, we utilize InfoPath Forms Server to serve InfoPath forms. The reason to apply InfoPath are the well-known: to enable business itself to design and maintain the forms, within a rich designer tool [the InfoPath designer]. To make that viable, a design principle is to keep the forms as slim and simple as possible. Meaning, no (complex) business logic embedded in the forms, no code behind, no web services invocation; just restrict to form input and validation. Another advantage of this restricted InfoPath usage, is that business can itself directly deploy the forms. These type of forms do not require FullTrust, nor is it needed to approve and activate them by IT operations in central admin.

A requirement is to email the forms; to a functional mailbox, and conditionally to the end-user him/herself. The latter is flagged by an option on the form. A management requirement is that the mailbox per form type must be configurable, without need to modify the forms. The only allowed management actions on the form level are for altering data collection and form display/layout.

InfoPath provides standard functionality to automatically email the form upon submit. However, because the destination functional mailbox may not be hardcoded in the form (the management requirement); and because of the conditional email copy to end-user would result in a more complex submit option; I rejected that solution approach. Instead I opt for a setup in which the form email handling is done via server side code. The high level design:

associate an SPItemEventReceiver for the ItemAdded event of new instances of the Form ContentType. The SPItemEventReceiver can either be associated at the ContentType definition, or per FormLibrary
The SPItemEventReceiver handles the emailing of each new added form. One to the configurable functional mailbox. This setting can thus be server side retrieved and applied, and it can be managed without need to modify the form for this. And if the end-user selected so in the form, then also send a copy to the end-user mailbox. This conditional logic can easily be developed within the SPItemEventReceiver handling

To set up the email-version of a submitted form, it is needed to retrieve the form data, and transform that into an email body-message. Design decision here is to apply the same view translation as embedded within the source form template. Rationale for that decision is that the email layout always will correspond with the displayed form layout. In that context, it is desired to retrieve the view stylesheet directly from the source form template self. Then the email layout will remain to automatically follow the form layout, irrespective of changes made in the form layout by business. With this, the steps to setup the email body are:

read in the xmfile associated with the added form item
retrieve the mso-infoPathSolution processing instruction, and parse the name of the source form template
download the source form template, and extract the view stylesheet from the cabinet container
cache the relation <name source form template, view> for later added forms the view can directly be found without need to get it from the source form template
apply the view stylesheet to transform the data of the xmlfile into html

The result is an email message in the same layout as the form rendering within InfoPath Forms Server. Warning: typically the resulting message is too large to be correctly handled by SPUtily.SendMail, so best to apply SmptClient instead for sending the email.

Tip: submit form via InfoPath client when InfoPath Forms Server submit reports a problem

2010-01-07T07:00:00.001+01:00

Context:

Upon submit of a form via InfoPath Forms Server, encounter the InfoPath error "An error occurred while the form was being submitted". That's it, no more error details displayed, nor to be found within the logging.

Problem analysis

So, how to detect what is the concrete problem cause? Tip: try to submit the same form directly via the InfoPath designer client. If the problem also occurs within this host context, you can here view more error details. In my situation, the error details displayed "A value in the form may be used to specify the file name. If you know the value in the form that specifies the file name, revise it and try again. Otherwise, contact the author of the form template". Translation, I forgot to specify a filename pattern within the submit options to make it unique.

Automatic publish InfoPath forms via Feature-based ALM deployment

2010-01-05T07:00:00.009+01:00

Fixed the Feature-based deployment of InfoPath forms to SharePoint site while treating them as first-class source artifacts in VS + TFS environment

I’m a supporter of applying Application Lifecycle Management (ALM) principles within application development projects. This also goes for SharePoint-based custom applications. Basically this means:

designing and developing the SharePoint artifacts within a team-based source repository environment (aka, Team Foundation Server);
applying version control to the individual application/building blocks;
daily build;
and full automatic and repeatable deployments

With InfoPath development, this gives some challenges. First of all, the InfoPath forms are setup and maintained via its own designer-tool, namely the InfoPath Designer client; and not directly from within the scope of Visual Studio. However, the most significant issue is the manner of deployment: the InfoPath client contains functionality to publish the InfoPath form to the destination environment, with SharePoint site as one of the available options. Given my ALM aspirations, I want to copy this InfoPath client deployment-experience to within the regular SharePoint deployments, via the Solution framework and SharePoint Features. In essence, I want to achieve the following:

Automatically provision the required Information Architecture artifacts:
- SiteColumns
- Masterdata for LookupFields
- ContentTypes
- Forms Libraries, associated with the ContentTypes, for administration of the filled in InfoPath forms
Automatically provision and activate the required InfoPath infrastructure

InfoPath forms
DataConnections

The first category, the Information Architecture artifacts, is well-known SharePoint provisioning. It can be done via a Site Definition, Features, or a combination. I favor the Feature-based approach, because it is modular, and allows me to turn it off and on.

The second category gave me more challenges. On internet search, I came across some posts that mentioned the SharePoint OOTB XSNFeatureReceiver that aids in here. However, it quickly appeared to me that it only takes care of one part, namely registrating the InfoPath form templates within the target SharePoint site. To successfully achieve this, XSNFeatureReceiver has constraints on the deployable InfoPath forms. And when even a single one of them does not comply to these constraints, XSNFeatureReceiver rather silently fails to correctly install the forms. Another missing part is for the DataConnections. The upload is not the issue here, being standard SharePoint Module functionality. But rather adjusting them to the specific target environment. In my ALM principles, I want to hold on to 1 single instance per DataConnection file, and not be forced to maintain versions for each of them per target environment (development, test, staging, production).

Luckily, Microsoft has not made XSNReceiverFeature sealed. So I decided to overload it to augment the standard deployment functionality with the above described aspects. Adjusting the DataConnections appeared rather simple, with thanks to this blog (Building InfoPath Form Services Solutions using Visual Studio 2008), including a link to demo provision code. I took this as basis to realize a generic approach, feeded by an XML configuration file delivered within the feature scope

The correct deployment of the InfoPath forms appeared to be more of a challenge. Eventually, I found out that the direct cause of the failed deployment was that I earlier on published the forms during my development and testing of the forms. As soon as you publish an InfoPath form, the InfoPath client administrates this within the form itself. And this appeared to result in the malfunctioning XSNFeatureReceiver based InfoPath forms registration. This can be mitiligated by correcting the InfoPath forms. An alternative is to modify the original InfoPath form artifacts. However, this would then be a corrective action to be repeated every time you’d published an InfoPath form just for an InfoPath designer (programmer) test. I therefore went for the alternative to do the correction as part of the Feature installation procedure. This way during the development phase the team can just go one and (re)publish the forms, without worrying about the effect on deployment later on.

Another thing to take into account when doing Feature-based deployment of InfoPath forms, is that the Forms must be contained at the Feature root-directory. I like to structure Feature contents within subfolders for different deployable types. ContentTypes, SiteColumns, DataConnections, and also InfoPath forms, each within their own subfolders within the feature. However, in case of InfoPath forms deployed via XSNFeatureReceiver, this is not working. Without a concrete error message, the registrating will then just not be performed. So keep your Forms at the Feature root-level.

Deployment steps, and thus functionality of InfoPathInfrastructure feature

Upon Feature installation event:

Fix the forms files, for successful automatic publishing via XSNFeatureReceiver
- Extract the forms; this on itself presented a challenge, since .NET does not provide standard cabinet (.cab) handling
- Inspect the manifest.xsf file on the presence of publishUrl and trustLevel; remove any of them present
- Repack the .xsn container / file

Within the Feature Activation:

upload forms files to FormsServerTemplate library
Registrate the forms
upload data connections to DataConnections library
provision site columns
provision master data lists
fix the lookup references
provision content types; including reference to the uploaded forms
provision contents/data libraries
assign content types to the contents libraries
fix retrieve data connections --> associate with the master data list in the deploy environment (iso your local / development source)
fix submit data connection --> associate with the content data list in the deploy environment (iso your local / development source)

InfoPath embedded via XMLFormView conflicts in IE with CSS relative positioning

2010-01-02T11:26:00.006+01:00

You can apply the out-of-the-box XMLFormView webpart to embed an InfoPath form within a SharePoint page. This way, the form displays as a visual integrated part within the SharePoint application, maintaining for the user the full application context (branding, navigation menu's). However, upon doing this (after applying the required infrastructure initialization steps, see this whitepaper), I continuously encounter a runtime Javascript eror when browsing the page in Internet Explorer. The error occurs in INC\Core.js in the ErrorVisualization_ComputeAbsoluteLeft function, used to position an error asterix within the form. Within this method, the DOM-tree is traversed up to the HTML-root. At least that is the intention, but somehow this goes wrong with IE; and the OOTB script is not made robust. This article gives more background details.

So, the problem cause is known. What about the resolution? I don't intend to sacrifice the correct CSS handling, just because IE doesn't handle it well. I see multiple options:

Making the HTML robust by wrapping the relative positioned DIV within an absolute positioned Body-container
Runtime overloading the OOTB ErrorVisualization_ComputeAbsoluteLeft function to fix it for IE behaviour
Use the PageViewer webpart iso XMLFormView to embed the InfoPath form
Inspect the Core.js code, and find a proper, and more localized solution

Your guess is right, I applied the latter. It appears that although the runtime errors occurs within the ErrorVisualization_ComputeAbsoluteLeft function, the solution can be found within the invoked helper ErrorVisualization_IsPositionCausingElement function. That function returns true if the inspected style object has its overflow property set, and in that case the loop within ErrorVisualization_ComputeAbsoluteLeft stops. The solution is therefore to assign an overflow setting to a proper container element of the XmlFormView. Since CSS styles are applied afterwards, it's not possible to assign this property via CSS. Options are to either place the style inline, or to add it via code - before the ErrorVisualization_ComputeAbsoluteLeft will be invoked. I applied the last. Rationale for this is that I only want to apply the IE-hack / resolution when it is needed. That is, when an XmlFormView is rendered within either IE 6 or 7. Inline style does not allow me to that, javascript does:

<UPDATE>The above workaround is dependent on the installation of SharePoint SP2 in your environment. Prior SP2, the ErrorVisualization_ComputeAbsoluteLeft function does not check via the ErrorVisualization_IsPositionCausingElement function; but just tries to traverse the offsetParent hierarchy until reaching the document.body object. Remedie then to avoid the runtime error is option 1: set the position of the document.body container to static.</UPDATE>

Peculiarities for handling space-character in field names for query and search: ManagedProperty name vs Field / ColumnName

2009-12-29T18:30:00.023+01:00

For better human readability, and thus understanding by the end users, it is often desired to use spaces in the DisplayName of SiteColumns and ContentTypes FieldRefs: "The columnname". But beware, this gives some peculiarities with the different approaches to query and display the SharePoint content...

Enterprise Search

Upon first crawling of the content source, this ends up in a CrawledProperty with the name "The_x0020_ColumnName". This can be mapped to a ManagedProperty. The name of this property may not include any spaces. Here it could be "TheColumnName". With Enterprise Search, one can now query and select on the property 'TheColumnName'. For instance, query via the AdvancedSearchBox webpart, and display the results via the SearchCoreResults webpart. Rather handy to skip in the XML-based query (SQL-CAML), and the rendering of the query result (XSLT), the special handling of the space character.

CAML-based site content query

When query and displaying site data via the ContentQueryWebPart, one must select and display on the actual field / column name. That is, in the CommonViewFields specification, the actual column name must be used. Due to the space in the name, CAML space-mapping must be applied; resulting in 'The_x0020_columnname'. Example:

<property name="CommonViewFields" type="string">
Title,String;The_x0020_columnname,Choice;Editor,User
</property>

This is all pretty well-known SharePoint stuff.

Something that is less known, is that yet another CAML translation must be applied to successfully display the CQWP queryresults via XSLT rendering. The CAML-space character '_x0020_' must itself be translated into '_x005F_x0020_'. So for instance:

Thus...

To summarize: when you put a space in the displayname of a field, this may wind up in 4 different 'names' within the query/display pipeline:

The actual displayname itself: "The columnname"
The managed property name for within enterprise search: "TheColumnName"
The query fieldspecification within CQWP: "The_x0020_columnname"
The query result rendering used by CQWP: "The_x005F_x0020_columnname"

Tip: difference in metadata only does not qualify document items as different

2009-12-28T10:14:00.012+01:00

In my last post I described the approach to set-up your managed properties collection as part of initializing the enterprise search experience. To test the outcome of this initialization, I next uploaded a couple of documents, and arbitrary filled in some of the metadata fields. Actually, I uploaded the same dummy / test document multiple times, each time renaming it. To my surprise, enterprise search next continuously returned only 1 of the uploaded documents. The cause of this is that the search crawling detected that the renamed documents are actually the same / duplicates. And on default, enterprise search via the SearchCoreResults webpart does not include duplicates. The tip is therefore in order to properly test the enterprise search in your application, make sure to upload different documents (that is, with difference in the document content) within the crawled content source.