Service Application Proxies runtime reported with locale-dependent TypeName

The SharePoint architecture enables shared usage of service applications across farms. A typical setup is a Shared Services farm that hosts the service applications, and multiple consumer / front-end farms that host the webapplications. In the consumer farm(s), each individual webapplication is associated with the service applications it requires. For instance, webapplication A is associated with Secure Store and Business Connectivity Services, and webapplication B is associated with Secure Store and Secure Token service applications.

In a distributed SharePoint architecture you cannot programmatically access the service applications in the local farm. Instead you must access via the service application proxy that is associated with the webapplication. The retrieval model of this is weakly-typed, you retrieve the desired service application proxy by string-comparision (!) on the proxy TypeName. This weakly-typed usage model is errorprone; you can easily make a typo error that goes unnoticed at compile time. But you will be immediately aware upon the first runtime test of the code.

However, this weakly-typed model incorporates another strange behavior: the reported TypeName is locale dependent! In my local SharePoint image, I tested against an EN-US sitecollection to retrieve the BCS service proxy, filtering on TypeName:

BdcServiceApplicationProxy proxy =
    webApplication.ServiceApplicationProxyGroup.Proxies.SingleOrDefault(
       p => p.TypeName == 
            "Business Data Connectivity Service Application Proxy" 
    ) as BdcServiceApplicationProxy;
 

With above code, I successfully retrieve the BCS service application proxy.

But the same code running against a webapplication in the integration-test farm does not select the BCS service application proxy. In Central Admin I verified that the webapplication is associated with the BCS service application. So what is the problem here? The cause rather surprised me: the sitecollection in the integration-test environment is provisioned via a Dutch-locale site definition. And as unexpected side-effect the TypeName of the associated service application proxies are now all reported in their Dutch localization name:

Fix for the above code is to make it locale-independent. For BCS this is possible by filtering on TypeName pattern ‘Business Connectivity Service’;

BdcServiceApplicationProxy proxy =
     webApplication.ServiceApplicationProxyGroup.Proxies.SingleOrDefault(
          p => p.TypeName.Contains("Business Data Connectivity")
     ) as BdcServiceApplicationProxy;
 

For other service application proxies it might be required to compare the TypeName against the established Resources value:

private static string _BCSApplicationProxyTypeName = null;

// Derived from Microsoft.SharePoint.CoreResource
static string BCSApplicationProxyTypeName
{
   get
   {
      if (String.IsNullOrEmpty(_BCSApplicationProxyTypeName))
      {
        Assembly _aIntl = Assembly.Load("Microsoft.SharePoint.intl, 
          Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c");
        ResourceManager _BusinessDataRM = 
          new ResourceManager("Microsoft.BusinessData.strings", _aIntl);
        _BCSApplicationProxyTypeName = _BusinessDataRM.GetString(       
          "ApplicationRegistry_BdcServiceApplicationProxy_TypeName");
      }

      return _BCSApplicationProxyTypeName;
   }
}


BdcServiceApplicationProxy proxy =
     webApplication.ServiceApplicationProxyGroup.Proxies.SingleOrDefault(
          p => p.TypeName == BCSUtility.BCSApplicationProxyTypeName
     ) as BdcServiceApplicationProxy;
 

Handling save conflict within SPList EventReceiver

Background

Publishing an InfoPath form to a SharePoint site creates a new contenttype in the site. Via the InfoPath publishing wizard you have some control over the structure of the created contenttype. But in essence the control is very limited when compared with explicitly provision a contenttype yourself [see also previous post Administrate data from InfoPath form in a self-provisioned ContentType for further explanation why I prefer this above the automatic created information architecture entities]. However, it is not always possible or viable to explicit provision the InfoPath utilized IA entities. The big selling point of InfoPath is that it enables self-service (web)form-definition by functional management. A key cornerstone in this self-services proces is the InfoPath publishing functionality to allow functional managers to themselves distribute the InfoPath forms to a SharePoint site.
One of the aspects missing with the InfoPath publishing managed contenttypes is including the InfoPath promoted fields in the display form. Displaying all InfoPath form fields is in particular handy when quickly browsing the data submitted to a SharePoint forms library.

Solution approach

To fix the functional shortcoming requires to afterwards make the promoted fields displayed. This is not something you want to burden the functional managers with. The aim is therefore to execute it automatically in the context of the target SharePoint site. The ideal moment would be when the InfoPath managed contenttype is created or updated. SharePoint however does not provide an eventreceiver for these events. The next best moment is when the contenttype is associated with the forms library to which the InfoPath form data is submitted. Here you neither have a direct event notifying the contentype association. But you can receive indirect notification via SPListEventReceiver::FieldAdded. The idea is then to check each added field whether it is an InfoPath promoted field, and if so to alter its definition so that the field will appear in the standard SharePoint display form.

Issues encountered

So, easily said and done. Associate the custom ListTemplate with a SPListEventReceiver, and fill in the FieldAdded method. In my local development image it worked like a charm: after you associate via the SharePoint GUI a document library with an InfoPath contenttype, all promoted fields of that contenttype appear in the document library display form. But after deploying to the central SharePoint development farm, I ran into the first issue. Upon associating a library with a contenttype, SharePoint faulted with a weird message: The specified program requires a newer version of Windows.

Despite this misleading error message, it was immediate clear that the issue was caused by the custom FieldAdded eventhandler:

Apparently, SharePoint does not allow to update a list field in the runtime context of that field being added to a list.

Solution outline

Solution is to isolate the execution of the field-update from the field-addition runtime context. Instead of direct updating the field, schedule this work via a worker thread. One extra aspect to take into account is that the scheduled worker threads can still run into a parallel update conflict:

Save Conflict

Your changes conflict with those made concurrently by another user. If you want your changes to be applied, click Back in your Web browser, refresh the page, and resubmit your changes.

  at Microsoft.SharePoint.Library.SPRequest.UpdateField(
        String bstrUrl,String bstrListName, String bstrXML)
  at Microsoft.SharePoint.SPField.UpdateCore(Boolean bToggleSealed)
  at Microsoft.SharePoint.SPField.Update()
  at WebformulierFieldsHandler.Worker.b__1()
 

The likelihood of these save conflicts increases with the number of promoted fields in the contenttype, and can already manifest itself when that number is 4 to 5. It can be made functional robust by implementing a retry mechanism in the worker thread.

Efficient batch deletion from SharePoint List without filling up the site RecycleBin

For a data management function I need a performant way to first delete all items from a SharePoint list, before re-filling it with up-to-date content. Looping through the list and delete each item one-by-one is not acceptable. It would instantiate an internal SPRequest object for each single listitem deletion, and take a longer time to complete. The answer to do it time-efficient is to use the SPWeb method ProcessBatchData(). You feed this method with a batch string of multiple delete commands to perform in a single SharePoint request.

However, usage of ProcessBatchData has the disadvantage that all deleted items are put in the SharePoint SPSite Recycle Bin. Kinda ridiculous for a programmatic batch-based deletion. And because of the larger number of deleted items (the reason for doing the deletion batch-based...), it makes the Recycle Bin pretty much unworkable thus useless for UI initiated restore of manual deleted content items.

SharePoint does not provide an elegant way to prevent this standard behaviour. Elsewhere on the web it is suggested to temporarily disable the Recycle Bin at SPWebApplication level. This however suffers from 2 drawbacks:

1. The current user needs to have the SPFarm administrator role. This is unlikely for a functional management role, and unacceptable from a security viewpoint.
2. Disabling the Recycle Bin at webapplication level has a nasty side-effect. It namely clears all Recycle Bins in the webapplication. Not only that of your current context site and sitecollection, but all of them in the webapplication. This effectively destroys the Recycle Bin backup-purpose for manual deleted items; and it is thus not isolated to your scope but affects the entire webapplication. This is functional unacceptable.

So if it's not possible to prevent the batch deleted items from appearing in the Recycle Bin, it is then required to delete them there afterwards. This could be done via a call to SPContext.Current.Web.RecycleBin.DeleteAll(). But this clears the entire Recycle Bin, still potential removing too much. What we need is an approach to delete exactly those items from the Recycle Bin that were put there as result of the list batch-deletion. The Recycle Bin has a different API interface as regular SharePoint lists. Usage of ProcessBatchData to delete the same relevant items from the Recycle Bin is not possible. But it is undesirable to first clear the SharePoint list via a batch-deletion, and next be forced to loop-based remove the same items from the Recycle Bin. Luckily the SPRecycleBinItemCollection class exposes its own method to issue a batch deletion: SPRecycleBinItemCollection.Delete(GUID[]). It has a different method signature, which requires you to first determine the GUID per SPRecycleBinItem. Also you must take care of keeping the chunck of deletion below a thresshold. SharePoint namely locks the database tables when executing the Recycle Bin batch deletion, which hangs the SharePoint server upon deleting a larger set. The Recycle Bin cleanup/reset can be done asynchronous in the background, doing it chunk based.

The code for efficient batch deletion all items of a list, and next clear the same deleted items from the Recycle Bin is as follows:

1. Control for time-efficient purge all items from SharePoint List

2. Cleanup the Recycle Bin in the background

PublishingWeb.IsPublishingWeb malfunctions on custom site definition

I typically apply a Feature-based approach for the provisioning of a SharePoint site into a full-functioning application . I prefer this above putting everything within a custom site definition. I'm in good company with my resistance against (overly complex) Onet.xml's [Andrew Connel, You don't need to create site definitions, Joel Oleson, Do you Really Need to Create Custom Site Definitions?]. I actually reuse the same site provisioning engine functionality within multiple SharePoint projects and applications. The reuse is made possible by specifying the provision actions via an XML feed file. Via this, the engine can be instructed to provision structure (Lists, Libraries), content (publishing pages, ListItems), and configuration (web.config via SPWebConfigModification, and PublishingWeb + SPWeb properties).

With respect to the latter, I discovered that the OOTB PublishingWeb.IsPublishingWeb sometimes plays tricks on you. Let me explain. In the provision engine I've implemented functionality to set diverse navigation settings: IncludeSubSites, NavigationShowSiblings, InheritGlobalNavigation, and so on. These properties can only be set on publishing webs, and are in the SharePoint Object API available via a PublishingWeb instance. In the engine code I first do a check whether the SPWeb on which the Feature is activated, is actually a PublishingWeb. And only then via PublishingWeb.GetPublishingWeb access a valid wrapper reference, and set the navigation properties.

This all worked fine in situations in which I applied the provision engine afterwards on an already created Publishing site collection. In my current project I do it slightly different. I've implemented a minimal site definition, and within the context of it activate per web in the created site topology the Publishing feature, and next execute the provision engine. In this setup, the navigation settings appeared to be ignored by the provision engine. When I debugged the provision execution, I discovered that PublishingWeb.IsPublishingWeb returned false for all webs in the site topology. Very strange, since I made sure to first activate the Publishing feature...

I ran into the forum-thread IsPublishingWeb() - what does it REALLY check for? posting a question on this issue. And followed up on the advice given there: implement an own SharePointHelper.IsPublishingWeb method to determine whether the SPWeb instance is valid for being wrapped into a PublishingWeb instance.

BTW: My own experiences learn that it is not due the asynchronous site creation behaviour. Also when later via site settings (de)activating the provision engine feature, PublishingWeb.IsPublishingWeb remains returning false, despite that the Publishing feature is activated on the (sub)web. It looks more as if PublishingWeb.IsPublishingWeb is somehow dependent on something that is set behind the curtains via OOTB Publishing site definitions.

Workaround shortcoming of SPWebConfigModification behavior wrt explicit positioning childNode

I structurally apply the SPWebConfigModification class upon all my SharePoint deployments to propagate the needed web.config modifications to the target environment (be it my local VM or in test/staging/production environments, typically in a farm). Benefits of using this functionality are:

• All web.config modifications are done automatically. No need to have someone (yourself or in production, IT operations) do this manually by editing the web.config file(s);
• Addition of the modifications becomes part of your ALM process; they can be tested and validated before going to production, and repeatedly fully automatic executed;
• The same web.config modifications are propagated throughout the entire farm; and even when later on a new WFE or a new extension is added;
• Added modifications can also be neatly removed upon uninstalling your SharePoint application.

Although I’m a big fan of this functionality, I’m also aware of some of its shortcomings. One of this is that upon adding a new childNode, it will always be appended as the last childNode in its parent webconfig node. Sometimes this is not good enough, and you need to be able to have it inserted at a specific position. In my case it was needed for enabling an ExceptionHandler HttpModule. To have this correctly operating, it must be the first active httpModule in the pipeline. Typically however the httpModules section in the web.config already has multiple httpModule nodes added:

Initial HttpModules section for a new provisioned publishing portal

The standard behavior of SPWebConfigModification places my ExceptionHandler behind them, effectively making it non-function. I could resort to manually correct my / all the web.configs; yeah right, you gotta be kidding…

Instead I decided to devote some time to investigate how to make a full-automatic correction for this. Via Reflector I inspected the runtime operation of the SharePoint ApplyWebConfigModifications method. Then I did an attempt to ‘break in’ on this functionality by a combination of first removing the already present httpModule childnodes, next add my exceptionHandler so it be the first, and then add the original httpModule nodes again. However, this was unsuccessful; and the outcome unpredictable. I suspect that an explanation for this is that my application is not the ‘owner’ of the (re)moved nodes. But that is hard to determine, since that requires a deep insight in the inner internals of this SharePoint behavior. After this I came up with a solution to correct the sequence in the httpModules afterwards the invocation of SharePoint’s ApplyWebConfigModifications. And this works satisfactory!! I must admit it has one weakness in that this afterwards correction will not be applied when later the farm is extended with new servers or web app zones. But honestly, seeing the SPWebConfigModification code I do not have a clue to make that happen. Another issue to watch out for is that SharePoint administrates in its config database all commanded web.config modifications, and reapplies them all upon each ApplyWebConfigModifications. I solved that by administrating in SharePoint’s PropertyBag the name of the ExceptionHandler, and then after each invocation of ApplyWebConfigModifications invoke the correct positioning of this HttpModule.

Code extracts:

1. Detect whether an ExceptionHandler HttpModule is provisioned

2. Set required ordering of childNode in each web.config

And ultimately, this gives me the desired system/debug-enabling functionality in the deployed application:

Oops, apparently I've forgotten to enable an implementation for the IExpensesHandling interface...