SharePoint [2010] as development, composition + application platform

I consider SharePoint having 2 distinct faces:

1. It's in itself an application, with lot's of end-user oriented functionality. This is true for the foundation WSS, and significant more for the licensed version MOSS 2007
2. It's a rich development and runtime application platform, to host your own custom developed applications.

Being primarily development-oriented, it will be clear that personally I like SharePoint best for the second role.

I'm not alone on this. More and more custom company-internal applications are based on and within the SharePoint platform. SharePoint comes out-of-the-box with a well-filled toolbox of both technical as functional building blocks. Developing custom SharePoint applications is therefore a combination of composing and customizing OOTB SharePoint building blocks, and augmenting it with custom developed functionality in which the SharePoint platform does not standard delivers. This implies a significant mindset switch for [old-school] application developers. Instead of the default decision to custom build all application parts, the SharePoint best-practice approach is to first consider the OOTB building blocks. Can you deliver the custom-requested functionality on sufficient level via the application of the standard SharePoint blocks? If so, choose that path. This composition-based approach will gain you in the initial application development stage. But even more it will be cost-benificial within the application maintenance period. It's better to have Microsoft fix and improve on standard building blocks, than have to do it yourself on your own proprietary ones. Of course, this principle can especially be applied on the more technical functionalities. And this gives you more time and relative budget to focus on the differential functionality for your company.

David Chappell has written a conceptual white paper highlighting the application platform aspects of the forthcoming SharePoint 2010. You can download it on the Microsoft site, SharePoint 2010: Developer Platform White Paper.

AllUsersWebPart and SPWebApplication.ApplyWebConfigModifications don't match

• In a Feature, I apply the AllUsersWebPart construct to automatically add default webparts to all publishing pages based on the PageLayout File.
• In a Feature, I utilize the SPWebConfigModification class to add web.config modifications

When done in separate Features, both constructs operate successfully. However, when included within the same Feature activation, the invocation of ApplyWebConfigModifications method results in a SecurityException:

Access Denied
at Microsoft.SharePoint.Administration.SPPersistedObject.Update()
at Microsoft.SharePoint.Administration.SPWebApplication.ApplyWebConfigModifications()
at Microsoft.SharePoint.Administration.SPWebService.ApplyWebConfigModifications()

It looks as if internally SharePoint somehow puts a lock on the Administration persistent object. I tried to verify my suspicion via Reflector, but not surprisingly this internal code is obfuscated. The runtime error only manifests itself when the Feature is (re)activated via the GUI. The Feature activation via stsadm reports no problem, and successfully performs the deployment work.

Still, I want to hold on to the idea of one single self-contained Feature to deploy all the required parts. And I want to be able to turn the Feature on and off interactively via the SharePoint GUI (so that I can also operate the feature remote, without access to the deployment server). A resolution is to apply another approach to provision the default webparts on the pagelayouts. Besides the usage of AllUsersWebPart, it is also possible to directly include the webpart specifications in the PageLayout file itself:

With this construction, the invocation of SPWebApplication.ApplyWebConfigModifications method is done successfully. And as a bonus, also another problem is prevented; that of magically duplicating the default webparts provisioned via AllUsersWebPart upon Feature re-activation.

A final word of caution. With this approach I at first encountered another problem. When editing on a created publishing page the properties of the default WebPart provisioned via the PageLayout file, SharePoint displayed an error warning in the edit toolpane, "a web part you attempted to change is either invalid or has been removed by another user". Although the net effect of the WebPart settings is simple done, this is not very trustwordy towards the Web Content Manager. Hard to explain that they can just ignore this warning. Before deciding to then have to abandon the approach, I re-examed the default WebPart specification in the PageLayout file to see if anything there could be causing SharePoint to suspect a concurrent update. And yes there was, 'thanx' to the automatic editing behaviour of Visual Studio. When you add a control within an .aspx file, Visual Studio automatically add a default 'ID' property to the control. For the default WebPart however this is not needed, the property is actually set when creating a publishing page off this PageLayout. But the mere presence of it in the default part results in SharePoint detecting a concurrent update. So remember to don't include the 'ID' property within WebPart specifications included within PageLayout files.

Tip: psconfig saved my local SharePoint environment

Once in a while, your local SharePoint installation may become corrupted. It happened to me today. I experienced diverse problems and symptons. Most noticable the malfunctioning of SharePoint search: crawling no longer worked, connect to the SharePoint Search service gave a "not able to access remote service". And when I inspected the eventlog, I detected several Office Server Search errors:

A seemingly total other problem was upon site collection creation via a custom site definition. This site definition applies a custom SPWebProvisioningProvider. Here I encountered a problem in loading the assembly of that provider, due to apparently missing referenced (system) assemblies.

When I next activated custom FeatureReceiver code that programmatically manages the Search content sources, SharePoint threw a Microsoft.Office.Server.Search.Administration.DeletedConcurrencyException with message Object e26da097-50ff-4386-b3f6-2b2b30f73b96 not found. This was the start to fixing my environment... I found via searching on DeletedConcurrencyException a useful advice to run the SharePoint admin tool psconfig:

psconfig -cmd upgrade -inplace b2b -wait

And indeed, after doing that magically the above described problems have vanished. That saved my day, I already feared that I had to re-install my local SharePoint environment.

PublishingWeb.IsPublishingWeb malfunctions on custom site definition

I typically apply a Feature-based approach for the provisioning of a SharePoint site into a full-functioning application . I prefer this above putting everything within a custom site definition. I'm in good company with my resistance against (overly complex) Onet.xml's [Andrew Connel, You don't need to create site definitions, Joel Oleson, Do you Really Need to Create Custom Site Definitions?]. I actually reuse the same site provisioning engine functionality within multiple SharePoint projects and applications. The reuse is made possible by specifying the provision actions via an XML feed file. Via this, the engine can be instructed to provision structure (Lists, Libraries), content (publishing pages, ListItems), and configuration (web.config via SPWebConfigModification, and PublishingWeb + SPWeb properties).

With respect to the latter, I discovered that the OOTB PublishingWeb.IsPublishingWeb sometimes plays tricks on you. Let me explain. In the provision engine I've implemented functionality to set diverse navigation settings: IncludeSubSites, NavigationShowSiblings, InheritGlobalNavigation, and so on. These properties can only be set on publishing webs, and are in the SharePoint Object API available via a PublishingWeb instance. In the engine code I first do a check whether the SPWeb on which the Feature is activated, is actually a PublishingWeb. And only then via PublishingWeb.GetPublishingWeb access a valid wrapper reference, and set the navigation properties.

This all worked fine in situations in which I applied the provision engine afterwards on an already created Publishing site collection. In my current project I do it slightly different. I've implemented a minimal site definition, and within the context of it activate per web in the created site topology the Publishing feature, and next execute the provision engine. In this setup, the navigation settings appeared to be ignored by the provision engine. When I debugged the provision execution, I discovered that PublishingWeb.IsPublishingWeb returned false for all webs in the site topology. Very strange, since I made sure to first activate the Publishing feature...

I ran into the forum-thread IsPublishingWeb() - what does it REALLY check for? posting a question on this issue. And followed up on the advice given there: implement an own SharePointHelper.IsPublishingWeb method to determine whether the SPWeb instance is valid for being wrapped into a PublishingWeb instance.

BTW: My own experiences learn that it is not due the asynchronous site creation behaviour. Also when later via site settings (de)activating the provision engine feature, PublishingWeb.IsPublishingWeb remains returning false, despite that the Publishing feature is activated on the (sub)web. It looks more as if PublishingWeb.IsPublishingWeb is somehow dependent on something that is set behind the curtains via OOTB Publishing site definitions.

Aggregate the values from an InfoPath repeating section for data administration

A common requirement for setting up (web)forms is to enter multiple rows of similar information. For example, entering the name(s) of your child(ren) for an holiday booking. The InfoPath Repeating Section enables an intuitive usage model for this. Initial the form displays only 1 section for data entrance. Via a link or button the user can on-the-fly expand the form with more occurrences of that same section.

Question is how best to administer the entered data from the repeating section in a SharePoint library. Is this a Master/Slave relationship? Or should it be considered simple as embedded in the form data self? The former requires a relation data model employed within the SharePoint information architecture. Although possible, this is not the typical SharePoint model which is namely to administrate the entity data in a single List or Library. However, for the second alternative it is still required to potentially administrate multiple values per field in the repeating section.

SharePoint supports this via the concept of AggregateFunction at Field level. To collect the data from a repeating section, specify aggregate="merge" for the field within the InfoPath form template

When explicitly self-provision the SiteColumns and ContentType (Administrate data from InfoPath form in a self-provisioned ContentType), you ran into a problem here. The FieldSpecification CAML currently does not allow the Aggregate attribute as in:

<FieldRef
  ID="{5e046132-9284-4322-b1c7-9a742d60270e}"
  Name="FirstNameChild"
  ReadOnly="TRUE"
  Node="/my:myFields/my:group/my:Children/my:FirstNameChild" />

SharePoint will fail the activation of your Feature with the error message: The 'Aggregation' attribute is not allowed.
The remedie is to skip the 'Aggregate' attribute from the CAML specification, and afterwards add it via an EventReceiver to the provisioned SPField in the RootWeb:

  SPField aggregateField = site.RootWeb.Fields[new Guid("{5e046132-9284-4322-b1c7-9a742d60270e}")];
  aggregateField.AggregationFunction = "merge";
  aggregateField.Update();

Administrate data from InfoPath form in a self-provisioned ContentType

When registrating via XSNFeatureReceiver an InfoPath Form Template in a SharePoint sitecollection, automatically a ContentType and the associated SiteColumns are provisioned. This ContentType enables promotion of the data entered via the form to a SharePoint library. However, I prefer to apply explicit self-provisioned ContentType and SiteColumns for the data administration. The rationale is to have full control over the created information architecture entities. In the automatic created ContentType, the ID is arbitrary and thus differerent accross (re)deployments and environments. This hampers the content deployment process between environments. And at rolling out an upgrade of the InfoPath form template in the same site collection, a new version of the ContentType is provisioned with potential another ID. The automatic provisioned SiteColumns do not expose this particular problem. Their IDs are namely contained in the InfoPath form template, and therefore remain identical accross (re)deployments. But the SiteColumns have their own issues. Most noticable is a SiteColumn with type (Multi)Lookup. This cannot be correctly provisioned via a CAML FieldSpecification only, but requires a fix afterwards (Creating Lookup columns as a feature). Another problem that both the automatic provisioned SiteColumns and ContentType suffer of, is when there are changes in the definiton while the entity is already in use. To propagate the changes at the sitecollection-defined entities to their usage at List/Library level requires some afterwards fix via custom code.

Because of these issues I prefer to explicit self-provision both the SiteColumns and the ContentType for administrating the forms data. You then control the ID of the ContentType, you can fix the Lookup-relationships for SiteColumns, and you can propagate changes within the ContentType to all List/Libraries which already use the type. It costs some extra development time, but it surely pays off upon repeated full-automic deployments, and accross environments.

Server-side generation of InfoPath email

In a SharePoint based application, we utilize InfoPath Forms Server to serve InfoPath forms. The reason to apply InfoPath are the well-known: to enable business itself to design and maintain the forms, within a rich designer tool [the InfoPath designer]. To make that viable, a design principle is to keep the forms as slim and simple as possible. Meaning, no (complex) business logic embedded in the forms, no code behind, no web services invocation; just restrict to form input and validation. Another advantage of this restricted InfoPath usage, is that business can itself directly deploy the forms. These type of forms do not require FullTrust, nor is it needed to approve and activate them by IT operations in central admin.

A requirement is to email the forms; to a functional mailbox, and conditionally to the end-user him/herself. The latter is flagged by an option on the form. A management requirement is that the mailbox per form type must be configurable, without need to modify the forms. The only allowed management actions on the form level are for altering data collection and form display/layout.

InfoPath provides standard functionality to automatically email the form upon submit. However, because the destination functional mailbox may not be hardcoded in the form (the management requirement); and because of the conditional email copy to end-user would result in a more complex submit option; I rejected that solution approach. Instead I opt for a setup in which the form email handling is done via server side code. The high level design:

• associate an SPItemEventReceiver for the ItemAdded event of new instances of the Form ContentType. The SPItemEventReceiver can either be associated at the ContentType definition, or per FormLibrary
• The SPItemEventReceiver handles the emailing of each new added form. One to the configurable functional mailbox. This setting can thus be server side retrieved and applied, and it can be managed without need to modify the form for this. And if the end-user selected so in the form, then also send a copy to the end-user mailbox. This conditional logic can easily be developed within the SPItemEventReceiver handling

To set up the email-version of a submitted form, it is needed to retrieve the form data, and transform that into an email body-message. Design decision here is to apply the same view translation as embedded within the source form template. Rationale for that decision is that the email layout always will correspond with the displayed form layout. In that context, it is desired to retrieve the view stylesheet directly from the source form template self. Then the email layout will remain to automatically follow the form layout, irrespective of changes made in the form layout by business. With this, the steps to setup the email body are:

1. read in the xmfile associated with the added form item
2. retrieve the mso-infoPathSolution processing instruction, and parse the name of the source form template
3. download the source form template, and extract the view stylesheet from the cabinet container
4. cache the relation <name source form template, view> for later added forms the view can directly be found without need to get it from the source form template
5. apply the view stylesheet to transform the data of the xmlfile into html

The result is an email message in the same layout as the form rendering within InfoPath Forms Server. Warning: typically the resulting message is too large to be correctly handled by SPUtily.SendMail, so best to apply SmptClient instead for sending the email.

Tip: submit form via InfoPath client when InfoPath Forms Server submit reports a problem

Context:

Upon submit of a form via InfoPath Forms Server, encounter the InfoPath error "An error occurred while the form was being submitted". That's it, no more error details displayed, nor to be found within the logging.

Problem analysis

So, how to detect what is the concrete problem cause? Tip: try to submit the same form directly via the InfoPath designer client. If the problem also occurs within this host context, you can here view more error details. In my situation, the error details displayed "A value in the form may be used to specify the file name. If you know the value in the form that specifies the file name, revise it and try again. Otherwise, contact the author of the form template". Translation, I forgot to specify a filename pattern within the submit options to make it unique.

Automatic publish InfoPath forms via Feature-based ALM deployment

Fixed the Feature-based deployment of InfoPath forms to SharePoint site while treating them as first-class source artifacts in VS + TFS environment

I’m a supporter of applying Application Lifecycle Management (ALM) principles within application development projects. This also goes for SharePoint-based custom applications. Basically this means:

1. designing and developing the SharePoint artifacts within a team-based source repository environment (aka, Team Foundation Server);
2. applying version control to the individual application/building blocks;
3. daily build;
4. and full automatic and repeatable deployments

With InfoPath development, this gives some challenges. First of all, the InfoPath forms are setup and maintained via its own designer-tool, namely the InfoPath Designer client; and not directly from within the scope of Visual Studio. However, the most significant issue is the manner of deployment: the InfoPath client contains functionality to publish the InfoPath form to the destination environment, with SharePoint site as one of the available options. Given my ALM aspirations, I want to copy this InfoPath client deployment-experience to within the regular SharePoint deployments, via the Solution framework and SharePoint Features. In essence, I want to achieve the following:

1. Automatically provision the required Information Architecture artifacts:
   • SiteColumns
   • Masterdata for LookupFields
   • ContentTypes
   • Forms Libraries, associated with the ContentTypes, for administration of the filled in InfoPath forms
2. Automatically provision and activate the required InfoPath infrastructure
• InfoPath forms
• DataConnections

The first category, the Information Architecture artifacts, is well-known SharePoint provisioning. It can be done via a Site Definition, Features, or a combination. I favor the Feature-based approach, because it is modular, and allows me to turn it off and on.

The second category gave me more challenges. On internet search, I came across some posts that mentioned the SharePoint OOTB XSNFeatureReceiver that aids in here. However, it quickly appeared to me that it only takes care of one part, namely registrating the InfoPath form templates within the target SharePoint site. To successfully achieve this, XSNFeatureReceiver has constraints on the deployable InfoPath forms. And when even a single one of them does not comply to these constraints, XSNFeatureReceiver rather silently fails to correctly install the forms. Another missing part is for the DataConnections. The upload is not the issue here, being standard SharePoint Module functionality. But rather adjusting them to the specific target environment. In my ALM principles, I want to hold on to 1 single instance per DataConnection file, and not be forced to maintain versions for each of them per target environment (development, test, staging, production).

Luckily, Microsoft has not made XSNReceiverFeature sealed. So I decided to overload it to augment the standard deployment functionality with the above described aspects. Adjusting the DataConnections appeared rather simple, with thanks to this blog (Building InfoPath Form Services Solutions using Visual Studio 2008), including a link to demo provision code. I took this as basis to realize a generic approach, feeded by an XML configuration file delivered within the feature scope

The correct deployment of the InfoPath forms appeared to be more of a challenge. Eventually, I found out that the direct cause of the failed deployment was that I earlier on published the forms during my development and testing of the forms. As soon as you publish an InfoPath form, the InfoPath client administrates this within the form itself. And this appeared to result in the malfunctioning XSNFeatureReceiver based InfoPath forms registration. This can be mitiligated by correcting the InfoPath forms. An alternative is to modify the original InfoPath form artifacts. However, this would then be a corrective action to be repeated every time you’d published an InfoPath form just for an InfoPath designer (programmer) test. I therefore went for the alternative to do the correction as part of the Feature installation procedure. This way during the development phase the team can just go one and (re)publish the forms, without worrying about the effect on deployment later on.

Another thing to take into account when doing Feature-based deployment of InfoPath forms, is that the Forms must be contained at the Feature root-directory. I like to structure Feature contents within subfolders for different deployable types. ContentTypes, SiteColumns, DataConnections, and also InfoPath forms, each within their own subfolders within the feature. However, in case of InfoPath forms deployed via XSNFeatureReceiver, this is not working. Without a concrete error message, the registrating will then just not be performed. So keep your Forms at the Feature root-level.

Deployment steps, and thus functionality of InfoPathInfrastructure feature

Upon Feature installation event:

1. Fix the forms files, for successful automatic publishing via XSNFeatureReceiver
   • Extract the forms; this on itself presented a challenge, since .NET does not provide standard cabinet (.cab) handling
   • Inspect the manifest.xsf file on the presence of publishUrl and trustLevel; remove any of them present
   • Repack the .xsn container / file

Within the Feature Activation:

1. upload forms files to FormsServerTemplate library
2. Registrate the forms
3. upload data connections to DataConnections library
4. provision site columns
5. provision master data lists
6. fix the lookup references
7. provision content types; including reference to the uploaded forms
8. provision contents/data libraries
9. assign content types to the contents libraries
10. fix retrieve data connections --> associate with the master data list in the deploy environment (iso your local / development source)
11. fix submit data connection --> associate with the content data list in the deploy environment (iso your local / development source)

InfoPath embedded via XMLFormView conflicts in IE with CSS relative positioning

You can apply the out-of-the-box XMLFormView webpart to embed an InfoPath form within a SharePoint page. This way, the form displays as a visual integrated part within the SharePoint application, maintaining for the user the full application context (branding, navigation menu's). However, upon doing this (after applying the required infrastructure initialization steps, see this whitepaper), I continuously encounter a runtime Javascript eror when browsing the page in Internet Explorer. The error occurs in INC\Core.js in the ErrorVisualization_ComputeAbsoluteLeft function, used to position an error asterix within the form. Within this method, the DOM-tree is traversed up to the HTML-root. At least that is the intention, but somehow this goes wrong with IE; and the OOTB script is not made robust. This article gives more background details.

So, the problem cause is known. What about the resolution? I don't intend to sacrifice the correct CSS handling, just because IE doesn't handle it well. I see multiple options:

1. Making the HTML robust by wrapping the relative positioned DIV within an absolute positioned Body-container
2. Runtime overloading the OOTB ErrorVisualization_ComputeAbsoluteLeft function to fix it for IE behaviour
3. Use the PageViewer webpart iso XMLFormView to embed the InfoPath form
4. Inspect the Core.js code, and find a proper, and more localized solution

Your guess is right, I applied the latter. It appears that although the runtime errors occurs within the ErrorVisualization_ComputeAbsoluteLeft function, the solution can be found within the invoked helper ErrorVisualization_IsPositionCausingElement function. That function returns true if the inspected style object has its overflow property set, and in that case the loop within ErrorVisualization_ComputeAbsoluteLeft stops. The solution is therefore to assign an overflow setting to a proper container element of the XmlFormView. Since CSS styles are applied afterwards, it's not possible to assign this property via CSS. Options are to either place the style inline, or to add it via code - before the ErrorVisualization_ComputeAbsoluteLeft will be invoked. I applied the last. Rationale for this is that I only want to apply the IE-hack / resolution when it is needed. That is, when an XmlFormView is rendered within either IE 6 or 7. Inline style does not allow me to that, javascript does:

<script type="text/javascript">
function fixInfoPathFormView()
{
  /*@cc_on
  @if (@_jscript_version <= 5.7)
  var objFormView = document.all["__XmlFormView"];
  if (objFormView != null)
  {
    var objFormViewContainer = document.all["wrap-center"];
    if (objFormViewContainer.style != null && objFormViewContainer.style.overflow == "") objFormViewContainer.style.overflow = "auto";
  }
  /*@end
  @*/
}
</script>

<UPDATE>The above workaround is dependent on the installation of SharePoint SP2 in your environment. Prior SP2, the ErrorVisualization_ComputeAbsoluteLeft function does not check via the ErrorVisualization_IsPositionCausingElement function; but just tries to traverse the offsetParent hierarchy until reaching the document.body object. Remedie then to avoid the runtime error is option 1: set the position of the document.body container to static.</UPDATE>