(scroll)height of children influences scrollheight of page

This is a follow-up on earlier post ‘Freeze pane on SharePoint list’

Business users very much appreciate the inserted functionality to freeze the listview header in visible top of screen, while one scrolls through the content. However, they reported a problem in case of smaller display size: it was not possible to scroll completely to the bottom, and also the page navigation controls cannot be reached.

The cause is that in my initial code I decided to unconditional hide the page scrollbar. I decided for this as otherwise the page scrollbar would remain in state to scroll through a large list, despite my changes to reduce the height of that list and give it a scrollbar for itself.

The simple way out to resolve the reported issue would be to just not hide the page scrollbar. But from User Experience point of view I dislike that, the result with 2 scrollbars and the page scrollbar over full initial height, is confusing for end users. I instead opted for only displaying the scrollbar when needed (screen smaller as minimal height for listview), and in such case reduce the scrollheight of page to only scroll to the bottom of the height-reduced listview. Thus not for the full initial height.

At first (html) code attempts this was not that simple to achieve, whatever reduction I made still the page scrollheight remained at the initial large value. Via debugger I identified the cause. The scrollheight of the listview remained at the large value, despite that I reduced the visible height. And the scrollheight of page as parent is directly influenced by the sum of scrollheights of the child elements in page. The scrollheight value is readonly, thus not possible to also reduce this to match the reduced height of the listview Div element.

Post ‘Making elements not affect page height (thus scrolling)’ put me on track to break out of this: It appears that scrollheight of relative positioned children has impact; but not that of absolute positioned children. So what I needed to change in my initial FreezePane code was to break the relative parent-child positioning relation for the listview with respect to page. This is accomplished by following changes:

    // Set visible height, with minimum of 300
    var visibleTableHeight = Math.max(availableHeight, 300);

//    $table.wrap("<DIV class='FreezedLV' style='OVERFLOW: auto; HEIGHT: 500px;'></DIV>");
    $table.wrap("<DIV class='FreezedLV' style='position:absolute; left:0; top: 0; width:100%; OVERFLOW: auto; HEIGHT: " + visibleTableHeight + "px;'></DIV>");
 
//    $(".FreezedLV").wrap("<DIV class='FreezedLVContainer'></DIV>");
    $(".FreezedLV").wrap("<DIV class='FreezedLVContainer' style='position:relative; margin:0px; height:" + (visibleTableHeight + freezedTRHeight) + "px;'></DIV>");

The end-result: now truly happy users, some of which forced to view the page on small(er) laptop screensizes.

The complete 'FreezePane' method:

function FreezePane() {
    var $table = $(".ms-listviewtable").first().data("summary", "list name");

    // Determine the available height for table to render in visible screen.
    var origWorkspaceHeight = $("#s4-workspace").height();
    var origTableHeight = $table.height();
    var spaceInWorkspaceWithoutTable = origWorkspaceHeight - origTableHeight;

    var windowHeight = window.innerHeight;
    if (windowHeight == undefined) {
        windowHeight = document.documentElement.clientHeight;
    }
    var topPos = $("#s4-workspace").offset().top;
    
    var freezedTRHeight = 25;
    var availableHeight = windowHeight - topPos - spaceInWorkspaceWithoutTable - freezedTRHeight;

    // Set visible height, with minimum of 300
    var visibleTableHeight = Math.max(availableHeight, 300);

    // WRAP TABLE IN SCROLL PANE
    $("#s4-workspace").css( { 'overflow-y': 'auto' } );

    $table.wrap("<DIV class='FreezedLV' style='position:absolute; left:0; top: 20; width:100%; OVERFLOW: auto; HEIGHT: " + visibleTableHeight + "px;'></DIV>");

    // FROZEN HEADER ROW
    $(".FreezedLV").wrap("<DIV class='FreezedLVContainer' style='position:relative; margin:0px; height:" + (visibleTableHeight + freezedTRHeight) + "px;'></DIV>");
    $("<table id='FreezedTR' class='ms-listviewtable' cellPadding='1' cellSpacing='0'></table>").insertBefore(".FreezedLV");
    $("#FreezedTR").width($table.width() + "px");

    var $origHeader = $("TR.ms-viewheadertr:first", $table);
    var $firstRowTable = $("TR.ms-itmhover:first", $table);
    var $freezeHeader = $origHeader.clone();
    // Propagate computed width of columns of origheader to freezeheader
    $origHeader.children("th").each(function() {
        var width = $(this).width();
        var ownerIndex = $(this).index();
        $($freezeHeader.children("th")[ownerIndex]).width(width + "px");
    });

    $("#FreezedTR").append($freezeHeader);
    $("#FreezedTR").append($firstRowTable.clone());
    $("#FreezedTR").wrap("<DIV style='OVERFLOW: hidden; HEIGHT: " + freezedTRHeight  + "px;'></DIV>");
    // Visualize "hide" the orig header, make sure it still is rendered as otherwise the alignment of 'body' rows is altered
    $origHeader.children("th").each(function() {
        $(this).css( { 'height' : '0px' , 'max-height' : '0px', 'min-height' : '0px' , 'padding-top' : '0px', 'padding-bottom' : '0px' } );
        $(this).find("div").css( { 'height' : '0px' , 'max-height' : '0px', 'min-height' : '0px', 'margin-top' : '0px', 'margin-bottom' : '0px' } );
        $(this).find("input").css( { 'height' : '0px' , 'max-height' : '0px', 'min-height' : '0px' } );
    });
    $origHeader.css( { 'max-height' : '1px', 'visibility' : 'hidden' } );

    // Delegate eventhandlers from the copied+freezed header to the actual header of the listview.
    var $inputFH = $freezeHeader.find("input[title='Select or deselect all items']");
    $inputFH[0].onclick = null;
    $inputFH[0].onfocus = null;
    $table.onmouseover = null;
    $inputFH.click(function() {
       $(this).closest(".FreezedLVContainer").find(".FreezedLV .ms-viewheadertr").find("input[title='Select or deselect all items']").trigger('click');
    });
    $inputFH.focus(function() {
       $(this).closest(".FreezedLVContainer").find(".FreezedLV .ms-viewheadertr").find("input[title='Select or deselect all items']").trigger('focus');
    });
    $("#FreezedTR").mouseover(function() {
       $(this).closest(".FreezedLVContainer").find(".FreezedLV .ms-listviewtable").trigger('mouseover');
    });
}

Future-proof handling of custom filetypes in SharePoint document libraries

Build Office 365 FileHandler to return external filetypes with application specific MIME-type(s)

Business in our company works with programs that store data in non-Microsoft and non-standards file-formats, e.g. Mathlab files, MindManager, Spotfire reports, ... Their aim is to store these file-formats also on SharePoint. Two SharePoint issues are an obstacle here.

SharePoint blocks file-extensions for upload

SharePoint blocks a large number of file-extensions for document library upload: Types of files that cannot be added to a list or library. However, in SharePoint 2016 this list is severely reduced to only few .Net programming files left, and in SharePoint Online the file-extension blocking is even completely gone. We’re not yet on SharePoint 2016 nor Online, but our roadmap is heading towards this. Therefore it is viable to relax the blocking to the minimal list of SharePoint 2016.

No recognition of application-specific MIME-type

SharePoint storage is agnostic for file-contents and extension. It can store whatever binary content in the content database as file. However, upon retrieving + opening a file from SharePoint document library, SharePoint has no recollection what to do with the unknown filetypes, and does a best guess. This results that e.g. a Tibco Spotfire report file with extension .dxp, is returned by SharePoint as a .zip file. And next cannot seamless be opened in the Spotfire Player. Same for Mathlab files, MindManager / MindMab, … In SharePoint on-prem you can overload this behavior via configuration of custom MIME-types. However, this is a WebApplication plus IIS setting, and thus clearly not available for usage in SharePoint Online tenant context. This withholds us now from utilizing the on-prem approach, as otherwise we would introduce business disruption upon going to the cloud.

Yet, recently I came across the concept of Office 365 FileHandlers and this promised to be a cloud-ready manner to achieve the same effect. I conducted a proof-of-concept with positive outcome: via a custom build Office 365 FileHandler Add-In that connects to the SharePoint Online site via the Office 365 Graph API, is deployed in Azure and registered in Azure AD, it is possible to open and return non-standard filetypes with custom MIME-types. The FileHandler Add-In concept allows to register multiple handlers, e.g. per specific filetype; and also to have a single FileHandler Add-In that can handle multiple filetypes. For my PoC, I needed to verify that custom MIME-types can be returned in the HTTP-response, and I wanted to validate this for multiple file-types. I therefore restrained to one, and determine in the FileHandler on basis if the extension-part of the filetype, the MIME-type to include in the response. Only disadvantage of this is that the FileHandler administration only allows 1 file-icon association, thus all file-types for which the FileHandler is registrated are displayed in Office 365 document libraries with the same icon.

Office 365 FileHandler applied to SharePoint Online site:

Click to select/download Spotfire file with .dxp extension, result:

and open on client in Spotfire Player application:

Code snippet FileHandler MVC open action to return file-content with application-specific content-type:

ToDo, figure out WOPI protocol against SharePoint Online

SharePoint Online acts as a Web Application Open Platform Interface (WOPI) host, and interacts via the WOPI protocol. A result is that in Office 365 FileHandler activation-parameters the file is identified by WOPI 'FileId' parameter. In my 'generic' FileHandler I want to return the application-specific MIME-type per configured file-extension (remember: I have the single FileHandler Add-In associated with multiple file-extensions), and therefore need the filename with the extension. The WOPI protocol supports this via 'CheckFileInfo' action. So far, I did not manage to get this working, still run into errors on illegal REST requests wrt WOPI protocol. To be continued...

SharePoint Hosted Add-In or Plain Old JavaScript / HTML5 / CSS?

When Microsoft released the SharePoint App-model (meanwhile renamed to Add-In model), it almost instantaneously became the preferred and dominant development approach for SharePoint customization. Now several years later, and the hype is gone, the question arises whether the Add-In model is the right answer for all SharePoint customization scenarios.

I dare to say no. And feel myself strengthened in that opinion by Microsoft’s latest and greatest SharePoint development approach: the SharePoint Framework. Yes, the Add-In model brings a lot of value, but at the expense of additional development, maintenance and runtime (performance) costs. In case you can bring the same functional value via a strict JavaScript + HTML5 + CSS setup, you can avoid these additional costs. Simple deploy the functionality via resource files, upload to a document library, and include them in the runtime SharePoint page context via a ScriptEditor. Much simpler to develop and deploy than a SharePoint hosted Add-In. And not hindered by iFrame boundaries.

Does this mean I see no value at all for the SharePoint-hosted Add-In model? No, there are definitely scenarios. A clear one is that of a software vendor: with an Add-In the vendor has a self-contained package that buyers can install in their SharePoint site. Another example is for a piece of functionality that is reused throughout the site structure, and that needs per usage its own storage in SharePoint. The Add-In model can provision this per Add-In installation in either the hostweb or the appweb.

Preserve code blocks ('xd:preserve') not supported in InfoPath Forms Services

See Readonly display of MultiLookup value in InfoPath Forms Services: my initial approach was to handle this in the InfoPath template itself, via Custom XSLT:

That approach works when testing in InfoPath Designer - preview.

However, upon publishing the template to SharePoint my custom XSLT is lost. Microsoft in MSDN article Using Custom XSLT in InfoPath Form Templates states that you can protect your XSLT customization for overwriting via 'xd:preserve'. Indeed with that inserted, upon saving the template the customizations are preserved:

However, I ran into another issue that Microsoft does not make clear in the referred MSDN article: forms with preserve code blocks in it cannot be published to SharePoint.

Hidden in other MSDN article Creating InfoPath Form Templates That Work With InfoPath Forms Services it states that 'XSLT extensibility (xd:preserve blocks)' are amongst the 'Features with No Direct Parallel on the InfoPath Forms Services'.

Note that you can [still] apply Custom XSLT in InfoPath Forms Services context, the limitation is that code preservation is not supported. So on every save in InfoPath Designer, the custom XSLT code blocks get overwritten, and you would have to insert them again. That effectively makes this InfoPath customization approach [at least for me] unmanageable and unworkable for sustainable utilization in InfoPath Forms Services context. Therefore my design decision to implement the required behavior 'outside' InfoPath Forms Services, via a clientscript approach.

Readonly display of MultiLookup value in InfoPath Forms Services

Business question:

Have an InfoPath form that on one tab allows submitter to select zero to more impacted IT applications, and on another tab (for other business role) display this selection readonly.

InfoPath / IT answer:

• Have multiple Views in the InfoPath form / template;
• On the 'input' View, include a Multiple Selection List (MultiSelectList) control, and bind to the MultiLookup List Column (or ContentType Field);
• On the 'display' View, visualize the current value of MultiLookup via a Repeating Table control. Via InfoPath configuration (control properties) remove the option for user to include new entries in the table: This part takes care of displaying [only] the selected value(s);
• What then remains is to make the control readonly, to prevent that the MultiLookup value can be changed on this tab. InfoPath does not itself support to disable the Repeating Table control. However, you can achieve this for InfoPath Forms Services context via (the power of) javascript: include on 'editifs.aspx' page script with a method that sets the html control to disabled. A complexity here is that IFS loads the form asynchronously after the surrounding page is loaded in browser, and that no event is triggered to notify the form is loaded. To handle that, I've programmed a polling approach that recurring checks whether the resulting HTML table is now present in DOM, and then disable it:

  
  
  var InfoPathCheck;
  var maxNrChecks = 10;
  
  $(document).ready(function(){
      InfoPathCheck = setInterval(function() {  //Wait for InfoPath to finish loading
          HasInfoPathLoaded();
      }, 200);
  });
  
  function HasInfoPathLoaded() {
      var selectAppl = $("#ctl00_m_g_d1ea83c5_3306_41e4_8419_01cfcf73921e_FormControl0_V1_I1_R5");
      if (selectAppl.length > 0 || --maxNrChecks === 0) {
          clearInterval(InfoPathCheck);
          if (selectAppl.length > 0) {
              $(selectAppl).prop('disabled', 'disabled');
          }
      }
  }
  
  

SharePoint mobile app + SharePoint Framework support?

The latest development approach for SharePoint customizations is via the new SharePoint Framework. MVP Waldek Mastykarz has multiple informative posts on what it is and how to utilize. In Everything you need to know about the SharePoint Framework he makes a remarkable statement: "When building solutions on the SharePoint Framework, if you follow the guidelines provided by Microsoft, not only will your solution look great on mobile devices but it will also be rendered in the native SharePoint mobile app which will be released shortly".

Remarkable, as I'm confused by how a native App could be running clientscript. So I challenged Waldek and Bill Baer via twitter to expand on that:

Well, turns out the statement is for now a bit too early. Microsoft is working on bringing this capability in the new SharePoint mobile app, and Vesa Juvonen acknowledges that Microsoft should tackle my question in their message.

UPDATE (June 13) Jeff Teper, interviewed at SharePoint Saturday Paris, answered on my question, being "We host html in JavaScript" [within the native code of the mobile app]. Although this is merely the high-over answer, and I want to understand more of the lower level details; this is sufficient for now. Details will be shared later by Microsoft and informed MVPs.

Performance repercussions of the Add-In model

The Add-In model is great to deliver self-contained functionalities, isolated from (harming) the SharePoint farm health - runtime and wrt physical installation. But be aware that there is a price: performance. In general, the overall performance will be negatively impacted due the Add-In model. Let me clarify:

1. Conceptual every Add-In is an own and independent (mini) webapplication. Requesting a SharePoint page that hosts <X> Add-Ins, effectively means that the browser is visiting 1+<X> webapplications to load and render the page: first the SharePoint hostweb for the SharePoint hostpage, and next the individual AppWebs for the contained Add-Ins (SharePoint-hosted or Provider-Hosted).
2. Each Provider-Hosted Add-In (re)enters in an own Add-In launch cycle, starting with App-authentication via the _layouts/AppRedirect.aspx page.
   Our renewed SharePoint based intranet heavily applies the Add-In model, with personalized homepage containing between 2-10 Add-Ins (mixture of SharePoint-hosted and Provider-hosted). In preparation of the Go-Live date, I validated performance and scalability through Visual Studio loadtesting. Due the multiplication of Add-In instances on the homepage there are multiple AppRedirect.aspx calls per homepage visit. On increasing the load the SharePoint server execution of that request became the scalability bottleneck, resulting in CPU to reach 100%. Investigation by Microsoft Premier Support confirmed that the high number (higher as what Microsoft architects had foreseen) of AppRedirect.aspx requests caused the peak in CPU:

   • The high CPU is caused by triggering of Garbage Collector
   • Garbage Collector is triggered due large managed heap, with int64 array datastructures
   • The int64 array datastructures are allocated by App Authentication, to encrypt and decrypt SharePoint internal certificates

   
   Note: As this (CPU reaching 100%) was for load well beyond both the expected / typical as the peak load, it was not needed to qualify as a No-Go for our Go-Live.
3. Each Add-In is included in the page-html via iframe element. Due iframe boundaries, there is no runtime sharing possible of client-side objects and resources: every iframe must load its own required objects in own runtime browser context (can be from browser cache for static resources, if that is enabled at client and server-side).
4. Be careful with building (or buying) too much self-contained Add-Ins, that all store the resources used by the Add-In in own AppWeb. Typical examples is sp.js. If that is administrated per AppWeb, then this heavy library will be retrieved multiple times: typical from the hostweb for standard SharePoint handling, and next from AppWeb(s) due Add-In usage. Due different urls, the browser does not recognize them as same resource, that is already retrieved. On each 'first' visit scenario this will result that the browser will retrieve the same resource multiple times, despite that it is already retrieved before and cached in browser.
5. Resources that are provisioned as Add-In rootcontent are not included in the blobcache. The effect of this is twosome: [1] SharePoint does not cache the content in blobcache, and on every request must get it again from SharePoint content database; and [2] retrieved resources do not include the ‘max-age’ cache-control header setting (see Max-age cache-control setting for SharePoint content), and therefore the browser must every time ask the webserver whether client-cached resource is still valid or changed on the webserver side. Which in case of static resources (e.g., jQuery library) typically will not have changed, always responding with HTTP 304, and therefore wasting requests and network bandwidth + time.
   Via Yammer Office 365 Network I’ve inquired how-to reduce the number of 304 responses for an Add-In, but apparent this answer is not known as no answer is given - also not by any of the connected Microsoft Office 365 architects and engineers.
6. Due own appdomain's, no sharing of http-connections across hostweb and the Add-Ins, and as result also no sharing of http-authentication context (see SharePoint App-Model + NTLM results in more 401’s).
7. Client-side altering of the page-DOM results in a reload of all Add-Ins. This is standard iFrame behavior, implemented as such in all current browsers (E.g. iFrame reloading when moving it in DOM).

All of the above enumerated performance-degradation aspects are inevitable when utilization of Add-In model. Our end-users on average are confronted with one or more 'Working on it...' spinners when they open up their homepage filled with Add-Ins. We managed to mitigate for a large part by combination of some asynchronous behaviour, lazy loading, and enforce the Add-Ins to retrieve resources from shared location (hostweb, but can also be a CDN - internal or SharePoint static).

Once we're enabled to utilize the new SharePoint Framework, we will evaluate which of the Add-Ins can be migrated to that new model and thus break out of the iFrame boundaries + constraints. At minimal the SharePoint-hosted Add-Ins are good candidates for migration to SharePoint Framework based solutions.