Showing posts with label Add-In. Show all posts
Showing posts with label Add-In. Show all posts

Saturday, February 3, 2018

(Risky) Approach to invoke SharePoint Online API on whatever site from external automation client

For an external automation client to access SharePoint Online via the API, same holds as for you as human visitor: it first needs to authenticate itself against the accessed SharePoint Online tenant, and next it must have the needed authorization in the specific accessed SharePoint content entity (site collection, site or list). This can be achieved by registrating an Add-In, and assign it the needed permissions. Then that Add-In can be used as authentication + authorization endpoint for the external client. Post Access SharePoint Online using Postman includes a good reference of the steps that you should automate in the external client. Drawback of the described approach is that it is on scope of individual site collection or even site. The Add-In registration is not utilizable as generic gateway to all sites in the SharePoint Online tenat.
To extend the scope to full tenant, the Add-In Only permissions must be requested on the 'tenant' level'. This can only be done by a tenant administrator; and must be done via the tenant-admin site (How to provide add-in app only tenant administrative permissions in SharePoint Online). But be very careful and considerated before going this approach: the implication is that whoever knows the client id and secret of the registrated Add-In, is now enabled to access via an external client data from whatever site in your tenant. Making site-specific permission management pretty useless / even a laughter. But your information security will certainly not consider it a good + acceptable joke...
With Add-In Only permission on 'tenant' level; external client that knows the Add-In's client id + secret can request an accesstoken; and then use that same single token to access whatever site in the SharePoint tenant:

Saturday, January 28, 2017

Pragmatic approach to push update of Add-In when via SharePoint UI fails

In previous blogpost, I recalled on a malforming COTS Add-In. Once the supplier delivered a fix, our SharePoint IT Operations team executed a change to update the Add-In version in the SharePoint environment. First upload the new version to the SharePoint AppCatalog, and next update the installed Add-In in the using SharePoint site(s). Both steps can be done via the SharePoint GUI, the second via the 'Get It' capability. The operations teams verified the deployment approach first in the acceptance environment, and validated that the problem with Add-In was resolved. All well. However, during the change execution in the productive environment, a problem was raised on the 'Get It' step to update the Add-In installation in the using site: Sorry apps are turned off. if you know who runs the server tell them to enable apps. I first told them to verify that both App Management Service and Subscription Settings Services were (still) enabled on the SharePoint webapplication. On the positive answer, I then concluded the problem to be likely as a glitch in the SharePoint site. The Add-In itself namely was unlikely to have a deployment problem, as we earlier successful updated the deployed installation in the acceptance farm. Thinking out-of-the-box, I next advised the IT Operations team to retry the Add-In update through PowerShell iso via the SharePoint GUI. Either it would work, or otherwise a bigger chance of getting meaningful error information. In our case, the former was true: via PowerShell the Add-In update was successful pushed in the using SharePoint site.
PowerShell snippet:
$web = Get-SPWeb ‘https://<site-url>’ $appPackage = Import-SPAppPackage -Path ‘<Add-In deployment package>.app’ -Site $web.Site -Source ‘CorporateCatalog’ -Confirm:$false $curAppInstall = Get-SPAppInstance -Web $web | where Title -eq ‘<Add-In title>’ Update-SPAppInstance -Identity $curAppInstall -App $appPackage -ErrorAction SilentlyContinue -ErrorVariable err if($err) { $err[0].Exception }

Friday, January 27, 2017

Pragmatic approach to display maintenance message on malfunctioning Add-In

One of the promises of the SharePoint Add-In model is that one can purchase packaged / of-the-shelf Add-Ins to install and use in your SharePoint sites. This promise is picked up by the SharePoint community - although in lesser amount as what Microsoft was expecting. One of the selling charms of using pre-packaged software Add-Ins is that your organization is not self responsible for maintenance, and this is delegated to the supplier. However, this loss of technical ownership also implies that you are dependent on the external supplier to fix their Add-In in case of issue, you cannot call in your own troops to resolve the problem. When such a faulty Add-In is prominent present in your SharePoint site, this is rather confronting: the end-users are visually notified and remembered of the malfunctioning behavior. In case the Add-In is added as a shared AppPart, you can mitigate this effect by temporary hide the AppPart from the page. But in case the Add-In is potentially added by individual users to a personalizable page, this is undoable: you don't want to traverse all the personalized instances of the page, check whether the AppPart is present in that instance, and if so hide or delete the AppPart and save + checkin the modified page instance. And then later when a fix of the Add-In is delivered + tested, redo the same to make it visible again.
So what to do then, to avoid the embarishing user experience with the faulty Add-In? As often in these 'modern Apps' days, it is javascript to the rescue. Ultimately on browser level, every Add-In is included in it's own isolated iframe on the containing SharePoint page. Through javascript the iframe containing the faulty Add-In can runtime be located in the DOM, and if found on-the-fly have it's src modified to refer to a maintenance page instead. The javascript snippet can be inserted via a ScriptEditor added in the shared part of the SharePoint page, and easily removed again once the Add-In behavior is recovered.
Code snippet:
function displayMaintenanceMessage() { var addInIframe = document.querySelector('iframe[src*="<identifying part in Add-In start url>"]'); if (addInIframe != null) { addInIframe.src = '/StyleLibrary/snippets/MaintenanceMode.aspx'; } else { setTimeout(displayMaintenanceMessage, 1000); } } displayMaintenanceMessage();

Friday, December 23, 2016

Shortcircuit launch sequence of provider-hosted Add-Ins

The launch of provider-hosted Add-Ins consists of 2 recurring steps:
  1. First, go to the SharePoint hostweb, and request AppRedirect.aspx
  2. The returned AppRedirect.aspx response contains a FORM structure, with as Action parameter the start-url of Add-In; via javascript embedded in the AppRedirect.aspx response the browser POST submits to request the Add-In start page.
See Source: Chris Johnson, Launching SharePoint Apps, launch sequence, debugging and AppRedirect.aspx
An essential role and value of AppRedirect.aspx is to pass the authenticated identity of current user as context token from SharePoint WFE to the remote web. However, this only applies for OAuth based, low-trust authentication via Azure Access Control Service (ACS). In case of Server-2-Server, High-Trust authentication, SharePoint host(web) environment does not pass a context token at all. And thus there is no concrete added value nor necessity for the AppRedirect call before the launching of the provider-hosted Add-In.
The most important FORM parameter is SPAppToken. SPAppToken is specific to low-trust configurations. If you are using a high-trust configuration the SPAppToken token is missing. Managing Identity and Context in Low-Trust Hybrid SharePoint 2013 Apps

The recurring invocation of AppRedirect.aspx per Add-In launch has some drawbacks:
  1. it involves an extra http request/response handling; in particular with longer network-distance, this costs noticable latency
  2. AppRedirect.aspx itself can take processing and elapse time. In IIS logs I observed that although not always, multiple times the server-side execution of this requests takes > 0.5 seconds to 1 second or more
  3. with higher load, I've earlier (Load testing SharePoint Add-in (former App) Model) noticed that the AppRedirect.aspx handling eventually becomes the bottleneck within the SharePoint WFEs
In the Add-In launch sequence, there is optimization possible as the AppRedirect.aspx step is avoidable in an High-Trust authentication situation. The added-value is in S2S authentication: zero. All the requested information to launch the Provider-Hosted Add-In is already included by SharePoint in the response of the host page that contains the Add-In(s): the 'src' value of provider-hosted Add-In iFrame element(s) exists of the pattern '<host>/_layouts/15/AppRedirect.aspx' and 'redirect_uri=<encoded uri of the Add-In startpage>'. Thus it is possible to shortcircuit the Add-In launch sequence by aborting the AppRedirect.aspx request, and replace it to instead immediate request as iFrame src the url-decoded 'redirect_uri' value.
Options to shortcircuit AppRedirect are:
  • Server-side: via an HTTP-module, to intercept the HTTP response before it reaches the browser, and for every iFrame element replace the value of the src parameter from AppRedirect.aspx into it's redirect_uri value. Disadvantages of this approach are that it is non-discrimating: all responses returned will be parsed, thus also all those that do not even contain an Add-In structure; it requires custom server-side deployment, something that is nowadays not always allowed as companies aim for cloud-ready / future-proof deployments
  • Client-side: on-the-fly change in browser / javascript-engine context, for every iFrame element replace the value of the src parameter from AppRedirect.aspx into it's redirect_uri querystring value. Disadvantage of this approach is that it is late, the browser typically already has requested the AppRedirect.aspx call. It still benefits to on-the-fly change the frame-src, and as such abort the waiting on AppRedirect.aspx response: in particular with longer network-distance, e.g. in case of a global company, it pays out to not wait for AppRedirect.aspx response to return, but immediate request the launch of the provider-hosted Add-In. And also it shortens in the scenario in which the processing time of AppRedirect.aspx would otherwise be [a] noticable bottleneck.
Snippet of the client-side approach:
<lSharePointWebControls:ScriptBlock runat="server"> function ShortCircuitLauchProviderHostedAppsInWPZone(webpartZone) { var frmRedirectSnippet = '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"' + '"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">' + '<html dir="ltr" lang="en-US">' + '<head><meta http-equiv="Expires" content="0" /><meta http-equiv="X-UA-Compatible" content="IE=8"/><title>Working on it...</title>' + '<link rel="shortcut icon" href="/_layouts/15/images/favicon.ico?rev=23" type="image/vnd.microsoft.icon" />' + '</head>' + '<body>' + '<div style="text-align:center;margin-top:40px;">' + '<img src=\'data:image/gif;base64,R0lGODlh...'>' + '<span style="color:#0072c6;font-size:36px;font-family:\'Segoe UI Light\', \'Segoe UI\', Tahoma, Helvetica, Arial, sans-serif;">' + 'Working on it...' + '</span>' + '</div>' + '</body>' + '</html>'; var iframesProviderHostedApps = webpartZone.querySelectorAll('iframe[src*="appredirect.aspx"]'); var i; for (i = 0; i < iframesProviderHostedApps.length; i++) { var src= iframesProviderHostedApps[i].getAttribute("src"); var redirect_uri = src.substring(src.indexOf("redirect_uri=") + "redirect_uri=".length); var addInUrl = decodeURIComponent(redirect_uri); iframesProviderHostedApps[i].contentWindow.document.write(frmRedirectSnippet); iframesProviderHostedApps[i].setAttribute('src', addInUrl); } } ShortCircuitLauchProviderHostedAppsInWPZone(document.getElementById("app_1")); </SharePointWebControls:ScriptBlock>

The client-script approach enables yet another performance and resource usage optimization: delayed loading per Add-In until and only if it is in the visible part of the browser window. The user experiences the effect one is familair with on mobile devices: swipe the screen, and the application's content is on-the-fly retrieved + screen estate filled (Twitter, Facebook, ...).
Slightly modified snippet for lazy loading Provider-Hosted Add-Ins:
<lSharePointWebControls:ScriptBlock runat="server"> function isWithinVisibleViewport(element) { var $w = jQuery(window); var windowWidth = $w.width(), windowHeight = $w.height(), viewTop = $w.scrollTop(), viewBottom = viewTop + windowHeight, viewLeft = $w.scrollLeft(), viewRight = viewLeft + windowWidth, offset = element.offset(), _top = offset.top, _bottom = _top + element.height(), _left = offset.left, _right = _left + element.width(), compareTop = _bottom, compareBottom = _top, compareLeft = _right, compareRight = _left; return ((compareBottom <= viewBottom) && (compareTop >= viewTop)) && ((compareRight <= viewRight) && (compareLeft >= viewLeft)); } function ShortCircuitLauchProviderHostedApps() { var nonVisibleInViewport = false; var frmRedirectSnippet = '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"' + ..... '</html>'; var iframesProviderHostedApps = document.querySelectorAll('iframe[src*="appredirect.aspx"]'); var i; for (i = 0; i < iframesProviderHostedApps.length; i++) { var delayedFrame = iframesProviderHostedApps[i]; var src = jQuery(delayedFrame).attr('src'); if (src != '') { var redirect_uri = src.substring(src.indexOf("redirect_uri=") + "redirect_uri=".length); var addInUrl = decodeURIComponent(redirect_uri); } else { addInUrl = jQuery(delayedFrame).attr('DelayedSrc'); } if (isWithinVisibleViewport(jQuery(delayedFrame))) { jQuery(delayedFrame).contents().find('html').html(frmRedirectSnippet); jQuery(delayedFrame).setAttribute('src', addInUrl); } else { jQuery(delayedFrame).setAttribute('src', ''); jQuery(delayedFrame).setAttribute('DelayedSrc', addInUrl); nonVisibleInViewport = true;' } } jQuery('#s4-workspace').off('scroll', ShortCircuitLauchProviderHostedApps); if (nonVisibleInViewport) { jQuery('#s4-workspace').on('scroll', ShortCircuitLauchProviderHostedApps); } } </SharePointWebControls:ScriptBlock>

Thursday, July 14, 2016

Future-proof handling of custom filetypes in SharePoint document libraries

Build Office 365 FileHandler to return external filetypes with application specific MIME-type(s)

Business in our company works with programs that store data in non-Microsoft and non-standards file-formats, e.g. Mathlab files, MindManager, Spotfire reports, ... Their aim is to store these file-formats also on SharePoint. Two SharePoint issues are an obstacle here.

SharePoint blocks file-extensions for upload

SharePoint blocks a large number of file-extensions for document library upload: Types of files that cannot be added to a list or library. However, in SharePoint 2016 this list is severely reduced to only few .Net programming files left, and in SharePoint Online the file-extension blocking is even completely gone. We’re not yet on SharePoint 2016 nor Online, but our roadmap is heading towards this. Therefore it is viable to relax the blocking to the minimal list of SharePoint 2016.

No recognition of application-specific MIME-type

SharePoint storage is agnostic for file-contents and extension. It can store whatever binary content in the content database as file. However, upon retrieving + opening a file from SharePoint document library, SharePoint has no recollection what to do with the unknown filetypes, and does a best guess. This results that e.g. a Tibco Spotfire report file with extension .dxp, is returned by SharePoint as a .zip file. And next cannot seamless be opened in the Spotfire Player. Same for Mathlab files, MindManager / MindMab, … In SharePoint on-prem you can overload this behavior via configuration of custom MIME-types. However, this is a WebApplication plus IIS setting, and thus clearly not available for usage in SharePoint Online tenant context. This withholds us now from utilizing the on-prem approach, as otherwise we would introduce business disruption upon going to the cloud.
Yet, recently I came across the concept of Office 365 FileHandlers and this promised to be a cloud-ready manner to achieve the same effect. I conducted a proof-of-concept with positive outcome: via a custom build Office 365 FileHandler Add-In that connects to the SharePoint Online site via the Office 365 Graph API, is deployed in Azure and registered in Azure AD, it is possible to open and return non-standard filetypes with custom MIME-types. The FileHandler Add-In concept allows to register multiple handlers, e.g. per specific filetype; and also to have a single FileHandler Add-In that can handle multiple filetypes. For my PoC, I needed to verify that custom MIME-types can be returned in the HTTP-response, and I wanted to validate this for multiple file-types. I therefore restrained to one, and determine in the FileHandler on basis if the extension-part of the filetype, the MIME-type to include in the response. Only disadvantage of this is that the FileHandler administration only allows 1 file-icon association, thus all file-types for which the FileHandler is registrated are displayed in Office 365 document libraries with the same icon.
Office 365 FileHandler applied to SharePoint Online site:
Click to select/download Spotfire file with .dxp extension, result:
and open on client in Spotfire Player application:
Code snippet FileHandler MVC open action to return file-content with application-specific content-type:

ToDo, figure out WOPI protocol against SharePoint Online

SharePoint Online acts as a Web Application Open Platform Interface (WOPI) host, and interacts via the WOPI protocol. A result is that in Office 365 FileHandler activation-parameters the file is identified by WOPI 'FileId' parameter. In my 'generic' FileHandler I want to return the application-specific MIME-type per configured file-extension (remember: I have the single FileHandler Add-In associated with multiple file-extensions), and therefore need the filename with the extension. The WOPI protocol supports this via 'CheckFileInfo' action. So far, I did not manage to get this working, still run into errors on illegal REST requests wrt WOPI protocol. To be continued...

Wednesday, June 29, 2016

SharePoint Hosted Add-In or Plain Old JavaScript / HTML5 / CSS?

When Microsoft released the SharePoint App-model (meanwhile renamed to Add-In model), it almost instantaneously became the preferred and dominant development approach for SharePoint customization. Now several years later, and the hype is gone, the question arises whether the Add-In model is the right answer for all SharePoint customization scenarios.
I dare to say no. And feel myself strengthened in that opinion by Microsoft’s latest and greatest SharePoint development approach: the SharePoint Framework. Yes, the Add-In model brings a lot of value, but at the expense of additional development, maintenance and runtime (performance) costs. In case you can bring the same functional value via a strict JavaScript + HTML5 + CSS setup, you can avoid these additional costs. Simple deploy the functionality via resource files, upload to a document library, and include them in the runtime SharePoint page context via a ScriptEditor. Much simpler to develop and deploy than a SharePoint hosted Add-In. And not hindered by iFrame boundaries.
Does this mean I see no value at all for the SharePoint-hosted Add-In model? No, there are definitely scenarios. A clear one is that of a software vendor: with an Add-In the vendor has a self-contained package that buyers can install in their SharePoint site. Another example is for a piece of functionality that is reused throughout the site structure, and that needs per usage its own storage in SharePoint. The Add-In model can provision this per Add-In installation in either the hostweb or the appweb.

Wednesday, June 1, 2016

Performance repercussions of the Add-In model

The Add-In model is great to deliver self-contained functionalities, isolated from (harming) the SharePoint farm health - runtime and wrt physical installation. But be aware that there is a price: performance. In general, the overall performance will be negatively impacted due the Add-In model. Let me clarify:
  1. Conceptual every Add-In is an own and independent (mini) webapplication. Requesting a SharePoint page that hosts <X> Add-Ins, effectively means that the browser is visiting 1+<X> webapplications to load and render the page: first the SharePoint hostweb for the SharePoint hostpage, and next the individual AppWebs for the contained Add-Ins (SharePoint-hosted or Provider-Hosted).
  2. Each Provider-Hosted Add-In (re)enters in an own Add-In launch cycle, starting with App-authentication via the _layouts/AppRedirect.aspx page.
    Our renewed SharePoint based intranet heavily applies the Add-In model, with personalized homepage containing between 2-10 Add-Ins (mixture of SharePoint-hosted and Provider-hosted). In preparation of the Go-Live date, I validated performance and scalability through Visual Studio loadtesting. Due the multiplication of Add-In instances on the homepage there are multiple AppRedirect.aspx calls per homepage visit. On increasing the load the SharePoint server execution of that request became the scalability bottleneck, resulting in CPU to reach 100%. Investigation by Microsoft Premier Support confirmed that the high number (higher as what Microsoft architects had foreseen) of AppRedirect.aspx requests caused the peak in CPU:
    • The high CPU is caused by triggering of Garbage Collector
    • Garbage Collector is triggered due large managed heap, with int64 array datastructures
    • The int64 array datastructures are allocated by App Authentication, to encrypt and decrypt SharePoint internal certificates

    Note: As this (CPU reaching 100%) was for load well beyond both the expected / typical as the peak load, it was not needed to qualify as a No-Go for our Go-Live.
  3. Each Add-In is included in the page-html via iframe element. Due iframe boundaries, there is no runtime sharing possible of client-side objects and resources: every iframe must load its own required objects in own runtime browser context (can be from browser cache for static resources, if that is enabled at client and server-side).
  4. Be careful with building (or buying) too much self-contained Add-Ins, that all store the resources used by the Add-In in own AppWeb. Typical examples is sp.js. If that is administrated per AppWeb, then this heavy library will be retrieved multiple times: typical from the hostweb for standard SharePoint handling, and next from AppWeb(s) due Add-In usage. Due different urls, the browser does not recognize them as same resource, that is already retrieved. On each 'first' visit scenario this will result that the browser will retrieve the same resource multiple times, despite that it is already retrieved before and cached in browser.
  5. Resources that are provisioned as Add-In rootcontent are not included in the blobcache. The effect of this is twosome: [1] SharePoint does not cache the content in blobcache, and on every request must get it again from SharePoint content database; and [2] retrieved resources do not include the ‘max-age’ cache-control header setting (see Max-age cache-control setting for SharePoint content), and therefore the browser must every time ask the webserver whether client-cached resource is still valid or changed on the webserver side. Which in case of static resources (e.g., jQuery library) typically will not have changed, always responding with HTTP 304, and therefore wasting requests and network bandwidth + time.
    Via Yammer Office 365 Network I’ve inquired how-to reduce the number of 304 responses for an Add-In, but apparent this answer is not known as no answer is given - also not by any of the connected Microsoft Office 365 architects and engineers.
  6. Due own appdomain's, no sharing of http-connections across hostweb and the Add-Ins, and as result also no sharing of http-authentication context (see SharePoint App-Model + NTLM results in more 401’s).
  7. Client-side altering of the page-DOM results in a reload of all Add-Ins. This is standard iFrame behavior, implemented as such in all current browsers (E.g. iFrame reloading when moving it in DOM).
All of the above enumerated performance-degradation aspects are inevitable when utilization of Add-In model. Our end-users on average are confronted with one or more 'Working on it...' spinners when they open up their homepage filled with Add-Ins. We managed to mitigate for a large part by combination of some asynchronous behaviour, lazy loading, and enforce the Add-Ins to retrieve resources from shared location (hostweb, but can also be a CDN - internal or SharePoint static).
Once we're enabled to utilize the new SharePoint Framework, we will evaluate which of the Add-Ins can be migrated to that new model and thus break out of the iFrame boundaries + constraints. At minimal the SharePoint-hosted Add-Ins are good candidates for migration to SharePoint Framework based solutions.

Sunday, May 8, 2016

Critical thoughts on [yet another] new SharePoint development approach

This Wednesday May 4th, Microsoft announced GA of SharePoint Server 2016. This event is accompanied by blogposts from multiple MVP's. Waldek Mastykarz posted an interesting one "When to use the new SharePoint Framework". However, this posts holds a rather distracting message for us: Solutions built using other development models introduced in the past, such as farm solutions or SharePoint add-ins, cannot be used on modern team sites. Rather then repeating my concerns and thoughts here, I refer to the post of Waldek for full context, and the comments-thread between Waldek and myself.

Tuesday, January 19, 2016

Approach for inability to rollback an AddIn update

Our SharePoint governance prescribes to include in the runbook of any change to our SharePoint environment, the steps to roll back the change in case of unexpected issues. However, as we experienced, technical rollback to previous state is not possible for AddIn (former: SharePoint Apps) update. The AddIn framework does not support to remove/downgrade the AddIn update, and return to the previous version of installed AddIn as active version. The only option is to re-deploy the version previous to the update; but for this to succeed that previous version must be changed to receive a version number later than that of the failed AddIn update. This approach is only possible for company-build AddIns, but not feasible for purchased AddIns. And from purist perspective it is undesired, as it is not a removal but a compensating rollback to return to previous state of working AddIn. While technical still a change is executed in the environment: the updated AddIn remains deployed, and the previous AddIn version is installed twice, last time with increased version number to force the AddIn update to be executed.

Mitigation approach

To cope with the unavailability of a pure rollback for AddIn update, we instead apply a procedural approach to minimize the likelihood of failed AddIn update. In an isolated subsite, that is non-accessible for regular end-users, we prepare the same installation status of the AddIn we intend to upgrade. In the change runbook, we start with upgrading the AddIn in the isolated subsite, and validate the functionality there for the production environment. Only in case the validation succeeds, we continue with deployment of the AddIn upgrade in the production hostweb(s) to which end-users have access. And in the undesired situation that the validation fails, we stop the deployment, and only need to cleanup by removal of the isolated validation subsite.

Monday, December 21, 2015

Our approach to deliver aligned business solutions

In my role as Collaboration Solution Architect I align with our business stakeholders to deliver company internal collaboration solutions. In these solution alignments, the technology is of secondary nature. Its impact is both as enabler as well constraining. Our technology options are in line with the generic enterprise IT architecture: SAP and Microsoft unless. So we apply SAP platform for some dedicated collaboration scenarios, and Microsoft for almost all the rest. As most SharePoint customers came to realize, we also impose governance in the collaboration solution setups we are allowed to deliver to our business. The overarching principle is to deliver only future-proof solutions. This translates to apply standard / out-of-the-box platform capabilities where feasible, and restrain from building custom solutions just for the sake of building custom solutions. And in case custom solution is needed to deliver on the requested functionality, then comply with the Microsoft guideline to stay away from farm-based solutions and instead rely on the Add-In model.
So, how does this work in practice? Let me clarify by a true example. Few months ago I was invited for an alignment meeting with our internal finance department. The title of the meeting was ‘BI portal’. In the meeting, the business stakeholders told about their functional intent. And extended on all of their requirements. Being aware of the technology strategy, they upfront realized that the target platform would be SharePoint. And they already themselves build an image on how the solution should look like.
Next step was to map their functional vision, and detailed requirements to feasibility. Initial focus here is to discuss and challenge on the functional vision. Not that I’m the subject expert, they are, but still it makes sense to ‘walk through’ the functional vision to evaluate it on true added business value. Next is to map the vision, and the detailed requirements, to the technology. How will the generic setup be, what can be delivered out-of-the-box via SharePoint features, what can be delivered through customization, what would require custom solutions, and what is not allowed due our governance constraints? Helpful in both alignments – the functional and the technical – is to use examples of solutions delivered for others, to trigger and inspire. These example solutions can be your own’s, but also of course from anywhere. The SharePoint community is very generous in sharing knowledge and experiences.
As follow-up of the alignment meeting, I sketched a potential solution direction on high level. I deliberately use PowerPoint as format for this, as that by its nature limits you for over-extensive writings. Outline of the solution architecture is: a) Sketch of the context, b) Main requirements, c) UI impressions (mockups), d) Global Design utilizing SharePoint platform capabilities, e) the information architecture. a) and b) serve to verify whether my understanding of the request is valid, and c) is to agree on the user experience.
After fine-tuning on business aspects with the stakeholders, next step is to get technical consent: d) and e). At minimal, communicate the setup of the solution direction with technical peers and our SharePoint operations. In our company we have formalized this via ‘Templace Control Teams’ per technology platform.
If and after both functional and technical alignment, next is to ‘build’ it in agile manner. Deliver a first version, not feature-complete yet, but it must already have functional meaning and value. Demonstrate in a ‘sprint-demo’ to business, discuss on behaviour and new insights, and deliver these in next ‘sprint’.
If the solution setup is restricted to SharePoint standard only, and potential customization as 'SharePoint content' (html, CSS, javascript), it is possible to build up the application direct in the production collaboration space. Although I then typically choose to first build it in my own 'development/playground' site, and after business agreement deploy it to the target location by repeating the 'content-based' provisioning.

Wednesday, December 9, 2015

Lessons learned with Add-In update execution

In our intranet we have Add-In instances installed throughout the site hierarchy. The content owners of subsites are enabled to utilize any of the provided set to their own will. In a release, IT takes care of automatic update of all the installed instances of Add-In(s) included in the release. This is done in a 2-fold approach:
  • Add the updated Add-In(s) to the SharePoint Add-In catalog;
  • Then first manual update ('Get It') the installed Add-In on the rootweb, for immediate result visible to end-users;
  • Next, via powershell script, update all the Add-In instances in the site hierarchy: traverse through the hierarchy, and on each hostweb that has the Add-In installed, execute 'Update-SPAppInstance'. Under the SharePoint hood, this delegates the Add-In update(s) to execution via a timerjob.
We've learned 3 important lessons with this Add-In update approach:
  1. The completion of the Add-In update throughout the site hierarchy is very time-consuming. In our intranet we've experienced elapse time of over 10 hours.
  2. Until the full completion of the Add-In update, it is not possible to manual update the same Add-In. Situation in which you want / need to do that - and we encountered one - is when in the release it is observed that after the Add-In update, the updated Add-In has an issue. As Add-In rollback is then not longer possible (once a Add-In update is executed), the remedy is to deploy a fix (on the update/fix).
  3. Most surprising: until the full completion of update of Add-In 'X', also the manual update of any other Add-In 'Y' is blocked for completion.
Lessons we took from these observations is that in intranet releases, we first complete all manual Add-In updates. This includes the potential additional installation of required 'fix' on 'updated Add-In'. And only once all manual Add-In updates are completed, and we have ascertained the effect of each is as expected; then execute the long-running Add-In update(s) via powershell maintenance script throughout the entire site hierarchy.

Friday, November 20, 2015

Chrome the better performant browser for App/AddIn-model based applications

In the SharePoint App/AddIn model, each AddIn operates in it's own isolated runtime security context. The separation is achieved by app-individual DNS domains. Beyond security isolation, the individual DNS domains also has performance ramifications. Good and bad (see SharePoint App-Model + NTLM results in more 401’s). A positive effect on performance is that the individual DNS app-domains, enables the utilization of more parallel http connections by browsers. Modern browsers support parallel http connections, but limit this on host level (maximum http connections per host). In SharePoint App/AddIn context, the browser maximizes the number of http connections multiplied for the separate DNS domains.
The Chrome browser has an additional performance advantage. As only browser, it predicts the DNS destinations that will be requested in the handling of a request - DNS Prefetching & TCP Preconnect:
A unique and important optimization in Chrome is the ability to remember the set of domains used by all subresources referenced by a page. Upon the next visit to the page, Chrome can preemptively perform DNS resolution and even establish a TCP connection to these domains.
This prediction is based on previous visit(s). Upon subsequent visit, Chrome initializes http connections per dnshost, and up to the maximum of 6 per dnshost. For a SharePoint page with one or more AddIns on it, Chrome immediate sets up 1 to maximum 6 http connections for the SharePoint hostweb domain, and per AddIn app-domain also up to maximum of 6 http connections. Upon receiving and processing the SharePoint page, for the dependent urls for the contained AddIn(s), Chrome can immediate send the request(s) over the already instantiated http connections. IE, Firefox, Safari, Edge (?) all apply a more 'just-in-time' approach. Delay with opening (new) http connection only when and until needed.

Thursday, November 12, 2015

Tip: how-to pragmatic and quickly mitigate from malfunctioning Add-In in production

Earlier this year we brought our new SharePoint 2013 based intranet ('digital workplace') into production. Crucial functional element of the workplace-concept is that employees can customize their own homepage with 'Apps' that are relevant for them given their own role and personal interests. The majority of these functional Apps are technical provided as SharePoint Add-Ins - SharePoint-Hosted plus Provider-Hosted. In the project to deliver the new intranet, we put a lot of effort validating the correct operation of the diverse workplace Apps, in particular focus on their performance. But despite that, soon after 'Go-Live' we experienced a performance problem with one, which effectively 'killed' our intranet experience. As such is destructive for user-acceptance, we needed an approach to quickly, on-spot, mitigate the non-performance App; so that next with less pressure the rootcause could be investigated.
Since the "App" (aka, as Add-In instance) is installed on the personalized pages of the homepage, it was not really an option to remove and deinstall it: it would require automated modification of all the personalized pages to remove the App, that is if the particular employee has the App added to his/her own page. Such automoted modification of what in functional essence is under personal control of end-users, was and is unsellable. We needed an alternative approach, in which transparent to the end-users temporarily the App-execution is overruled.
Here it is important to understand how SharePoint deals with SharePoint Add-Ins. In technical sense, in the SharePoint page response an Add-In is nothing more than an 'iframe++'. The frame-src refers to the 'AppRedirect.aspx' launchpage, including information about the Add-In to launch (see Launching SharePoint Apps, launch sequence, AppRedirect.aspx for a good explanation of this). The browser first receives and renders the html-response for the SharePoint page, and next per included Add-In, resolves the resulting iframe.
<iframe src="https://.../_layouts/15/appredirect.aspx?redirect_uri=....&client_id=..." id="..." height="250px" frameborder="0" width="720px"></iframe>
The mitigation approach for non-functioning Add-In is now to on-the-fly in the browser-context, replace the 'AddRedirect.aspx' launchpage with an url to alternative 'application'. This can be as simple as an .htm content page. The direct effect is that users are abstracted from issues in the actual Add-In, and instead for now receive alternative html-snippet to replace the malfunctioning App. This approach does not require automated modification per each of the personalized pages, it is sufficient to include a ScriptEditor in shared webpartzone. It is an approach that can be executed from the SharePoint GUI, without code changes. And thus can be applied for SharePoint on-premisse as in Office 365 / SharePoint Online.
<script type="text/javascript"> var iframes = document.getElementsByTagName('iframe'); var i; for (i = 0; i < iframes .length; i++) { var src= iframes[i].src; if (src.indexOf('<AppRederict.aspx url with reference to mailfunctioning Add-In>') != -1) { iframes[i].src = 'https://.../Style%20Library/ Mitigation/CodeSnippetAsAlternativeForMalfunctioningAddIn.aspx' } } </script>

Monday, August 17, 2015

Project-level Best Practices for delivering a performant Add-in based application

Indeed, all of the below are open doors!!
But yet in practice often either overlooked, or on purpose but due wrong reasoning (e.g. save time now in the project) bypassed in development project(s) to deliver a SharePoint Add-in based application:
  1. Performance and Capacity Management must be applied as an integral subject during the application development project
  2. In the requirements, agree with the application owner on the performance aspects: page download times considered acceptable, applicaton parallel usage
  3. Include performance Best Practices in the development guidelines. And make sure that all involved developers know of the guidelines, and that they apply them in their individual developed Add-Ins
  4. Do not only develop + test SharePoint Add-ins ('Apps') in isolement; also conduct integration test with multiple Add-ins on a page as the user is likely to use them, and monitor the page payload
  5. Thorough intake of any external Add-ins before before purchase; intake on architecture, functionality, capacity management and maintenance aspects
  6. Structural monitor and proof the application performance during the project, to detect in early stage when something is introduced with a severe performance decrease impact. Load testing is a good means to implement this in the project, for performance quality assurance

Some architectural and technical tips

  • Cache where appropriate (but be aware that caching costs resources itself)
  • Reduce the number of network roundtrips from client to application server - batch requests
  • Retrieve resources that are used in multiple Add-ins from a shared location - e.g. root site of the HostWeb, or a CDN (external or internal)
  • Reduce the impact of (NTLM) Authentication by retrieving non-authorized resources from anonymous accessible location
  • Utilize the sprites-concept for images
  • And the same for custom javascripts; for maintenance it is good to separate responsibilities in different libraries, but for performance it is better to collapse in a single resource file
  • Apply minimalization in resource files: javascript and CSS resource files
  • Apply lazy loading where appropriate (e.g. avoid processing and retrieval impact for Add-in functionality that is initial non-visible, and/or only rare used; in favor of delayed execution if and only if the user intends to use the Add-in)

Thursday, August 6, 2015

Load testing SharePoint Add-in (former App) Model

Validate healthy application performance behaviour

Essential for any Enterprise Application is that it can performant and scalable handle the varying usage load by the users. Nothing as embarrasing as a new application that soon after Go-Live, breaks by the enthiousastic usage of the users. To prevent such, you must build trust in the scalability of the application, and establish before Go-Live that the application – application software + system infra - can handle the expected load. Introduce load testing.
This also holds for modern SharePoint application that is composed with the Add-in model, former SharePoint App-model. But loadtesting of AddIn model does bring some extra peculiarities to loadtesting. I enumerated below the ones I encountered.
And note: our loadtesting proofed both valuable and successful: initial the loadtest revealed some performance and scalability problems. We then made some essential changes in the application code (in particular in the applied custom Add-ins / Apps), until we achieved our usage load target goal. And at the crucial moment of Go-Live, the application did not give a blinch, and perfectly handled the usage load of > 14.000 users.

Application Performance health factors

2 health factors monitored:
  1. Responsiveness of the application for the user, measured as Page Download Time
  2. Scalability of the SharePoint infra, measured as CPU, Memory and I/O utilization on the servers

Application Performance validation approach

  1. Identify target goals for application utilization
  2. "Green zone"
  3. Proof the health factors at the target-utilization goals via load testing, to simulate the real usage
  4. Identify the ‘breaking’ point via increased load/stress testing
  5. "Red zone" - performance issues monitored
  6. Determine the rootcause of the issue; this can be non-optimal code, insufficient infra parts (CPU, memory, network throughput, database IOPS)
  7. Fix the issue(s)
  8. Repeat the validation, at step 2

Loadtest execution

loadtest preparation

  1. Identify the usage/application scenarios you will use to build trust. You should select scenarios for which you expect these will be used during typical usage. An heavy transaction that in the normal operation will only be rare executed, will have a neglectable effect on the application load.
  2. Establish the target load. This is the application load for average usage. For web applications, this is typically stated in ‘Page Visits per Second’. Note that this is different from Requests per Second / RPS. In nowadays modern apps, a single page visit encompasses multiple http requests: for the page itself, dependent resources as javascript and css, and javascript calls to execute service calls for data retrieval and application functions.
    The determination/specification of the concrete target value is a challenge on itself. One easily is tempted to overstress the target value - we have 'X' users, so the parallel application usage will be 'X * Y'... However, in reality those 'X' users do not continuously all hit the application: they log on at different times, stay on pages, use other applications, go to the coffee machine, ... In our setup we identified the target value twofold:
    1. Fact: as we were introducing a renewed intranet, we could reuse the application usage statistics of the current intranet;
    2. Prediction: determine the target value via Microsoft (Bill Baer) Capacity Management Formula, an unofficial best practice recommendation
    And in our situation, the 2 values determined via the different paths delivered about the same target value, which confirmed us that we determined a realistic value.
  3. Establish the heavy load: this is abnormal but still foreseenable application usage, in special circumstances.
  4. Determine how-to build trust: manual load testing, custom test software, or utilize a load test tool – e.g. HP LoadRunner, Visual Studio LoadTest.
  5. Get sufficient test accounts to simulate different users. This is also required to prevent cache effect during load test execution. E.g. continuous retrieving user profile values of the same user.
  6. Prepare the test context for the test accounts. E.g. if the application makes use of SharePoint user profile, then the user profile must be provisioned for the test accounts to ensure reasonable load behavior.

Specialities with setting up testscripts for Add-ins / Apps

  1. The load test scenario must join in App authentication flow. In essence, this means that SPAppToken value must be set as FORM POST parameter in submit request to appredirect.aspx. The value is runtime determined in the App launcher, and returned in the initial AppRedirect.aspx response.
    In the Visual Studio webtest recording, the reference is made to this hidden field in the response.
    We encountered that the SPAppToken value is not successful runtime retrieved. This can in some circumstances be corrected by monitoring the traffic via Fiddler, and set SPAppToken to a fixed value that you get from the Fiddler trace.
  2. FormDigest value returned in JSON response from contextinfo call instead of hidden FORM parameter in response body.
    Resolution is augment the Visual Studio loadtest: Add a Text Extraction Rule to extract the value from the /_api/contextinfo JSON response.
  3. Default, Visual Studio LoadTest execution does not mimic browser-cache, resulting that each dependent resource is requested over and over. You can change/fix this by configuring the loadtest script to ‘parseDependentRequests = false’.
  4. Visual Studio LoadTest does not include the execution of javascript in the browser. If required, the activity of the javascript code must be simulated in the test scripts.
  5. With multiple provider-hosted Apps in the load test scenario, the Visual Studio loadtest scenario can make error in the runtime construction of the load test recording and assign a wrong {app_?} value. In such case, you must manually add a '<ContextParameter Name="AppId_1" Value="<APP domain value>" />', and correct the relevant Requests in the script to send request to the correct app-domain:
  6. Visual Studio LoadTest recording misses to set header variable ‘Origin’, which hinders CORS protocol handling.
  7. You can easily overwhelm the usage load by setting the ‘concurrent user’ configuration value. The use of this configuration parameter is misleading: it does not really simulate actual users. It merely sets the threads in the loadtest execution from which to continuously execute the webtest(s) in the loadtest scenario. Per thread, after finishing the webtest, the execution halts for the thinktime value; and then repeats. If you set the thinktime to zero – which is what Microsoft advices on Technet, "Don't use think times…" -, the effect is that continuously requests are fired against your application. The load on the application is then much higher as the value configured in ‘concurrent users’.
  8. Visual Studio loadagent itself can become the limiting factor. If you want to simulate a larger concurrent usage, this results in equal large set of threads in the Visual Studio execution, all of which busy to execute and monitor a webtest instance. The cpu on the load agent grows to 100%, and the load does not linear increase with the number of ‘concurrent users’ aka threads.

Load test monitoring

  • CPU, memory and disk IO per server: WFE, SharePoint backend, AppHost
  • State of the IIS queue on WFE and AppHost
  • Page download times
  • Slowest pages

Interpretation of load test output

  1. The (average) Page Response Time is the summation of the download time for that request, AND augmented with including the download times of all dependent requests beneath that main request.
  2. The RPS / Requests per Seconds output is not fit to determine whether the application + infrastructure can handle the foreseen application usage. The application usage translates in Page Visits per Second, in which each page visit typically encompasses multiple (http) requests: the .aspx request, requests for javascript and css resources. In the App execution model, each App launch on the page is in effect an own page visit. As result, the RPS factor is of little use. You must measure the ‘Page Visits per Second’ factor. Pragmatic way to monitor this is to set the thinktime for webtest on 1 minute; so that each minute the webtest is executed. The ‘Page Visits per Second’ factor then equals the Visual Studio reported 'Test per Second'.