SharePoint Output Cache

What is this SharePoint Output Cache and why should I use it?

In ASP.NET there is an output cache that manages how pages content is severed.  It allows IIS to cache static elements, such as images and pages so that subsequent requests do not need to go looking for these items.  Similar to how your browser cache keeps you from downloading the same images over and over again.  The big advantage with SharePoint is that when this is enabled it caches fully rendered versions of pages. 

When SharePoint loads a page it is a really big process.  It needs to get the Master Page from the file system or the database.  It needs to get the Page Layout from the file system or again the database.  It needs to get all CSS, all the images and all the JavaScript.  From here it can start rendering the page, but there is more.  For every security trimmed or audience trimmed control it needs to make a call back to the database to determine if it should render it or not.  BTW:  You read correctly it doesn’t make one giant call to figure out what to do with each security trimmed object, it calls them one at a time.  This can take a bit of time when you think about all the buttons in the ribbon that may or may not appear, on top of the web parts the page requires.

As you can see that is a lot of calls just to render one page.  With output caching enabled this fully rendered page will be cached and every subsequent call will not have to go through all the above process.  It will simply make one call and get one page back.  As you can image this makes a huge difference on throughput performance.

To demonstrate just how much of an improvement this makes, I load tested the a customized SharePoint 2013 site.  This site isn't a very heavy site, but does have a lot of JavaScript for responsive design on top of all the other SharePoint scripts that are required.  I ran the same set of tests, with same number of users against the same site and you see the results in the below chart:

Counter	Before	After
Average Page Load Time (s)	23.4	1.4
%Processor Time App Server	24.8	16.8
%Processor Time WFE Server	81.1	41.5

As you can see it made quite an improvement in Page Load Time and took a significant amount of stress off both servers.

Now you are probably wondering why this wouldn't be enabled by default, since it provides such significant performance gains.  Well there is a bit of a catch to this.  It is caching a version of the page, this means there is more memory being used on the server to store the page.  On a publishing site this is a bigger piece as it will cache two version of page (published and draft).  Also, as you've guessed this could lead to some inconsistencies as well.  The cached page is removed when the page is updated, but there is a bit of a delay and each WFE has its own timer on when it updates the page.  In a farm with load balanced multiple WFE servers it is possible that the first request could go to a WFE that has updated the page in cache and a second request goes to a WFE that has not.  The time to update is by default 60 seconds, so this is a small window, but still possible.

To address some of the issues is the ability to create caching rules.  You can target rules to a Site Collection, Site and Page Layouts.  You can also create caching rules based on a user’s access rights, so that all readers see cached versions, where contributors do not.  This rules can also be extended programmatically through the VaryByCustom handler.

Overall, in my opinion, Output Caching should always be leverage for low write, high read sites.  Also considering that you can have rules set for different page layouts, you could set the caching time higher for the a low write page, like maybe the home page of an intranet, and set the time lower on high write pages within the site.  With careful planning this feature can really help scale out the farm to handle more users while saving server resources.

For more information on output caching and how to enable it, see this helpful MS link: http://msdn.microsoft.com/en-us/library/aa661294(v=office.14).aspx

SharePoint Project Planning

Congratulate yourself for actually accepting planning is required with SharePoint.  I know Microsoft likes to tell us you can just turn it on and use it, but we all know that’s just not the case.  Too many places are stuck with the pilot that never ends.  Starting with pilot or POC is great approach, but for it to work you need to circle back and actually plan a proper deployment that includes DR, Availability and Performance.  The Pilot that never dies had none of these consideration and is like a ticking time bomb, who will be the first user that picks up the phone to complain the site is unusable?  Will it be the CEO?  More importantly will you be able to fix it?

To begin with make sure you have the correct people for the job.  If you are going to train up internal resources to do the job, ensure the training happens early, ideally before any solution planning.  This is because the team planning the SharePoint solution will need to lead the end users to the solution.  It's hard for the end user to ask for what they want, when they don't know what is available.  Ask any SharePoint professional and there are many tools, tips and quirks to using SharePoint that only someone with experience would think of.  If no one knows how SharePoint works, it becomes the blind leading the blind.  Which is why, even if you are planning to train up your internal resources, you should ensure you have an experienced SharePoint professional on the project.  As we all know, there are something's that only experience can teach us.

Once you have armed your team with this knowledge, the end users are going to ask for things that are known limitations of SharePoint.  While it is always a great strategy to use OOTB functionality, don't be too rigid on this.  Rather than just saying no, or agreeing to a complex custom solution, be prepared with work arounds or alternatives that may speak more to the spirit of the request.  After all, if the site is usable to the end user, who is going to use it.  But if it's overly customized who is going to support it?

Now that you have enough information to design a solution that will meet the end users' needs don’t' forget to consider the following in your design: High Availability, Disaster Recovery, Automated deployment to the multiple environments (UAT, Staging, Production), Security, Internal vs. External access, mobile (becoming bigger by the day), performance and of course monitoring. 

Be agile in development, especially with people that don't have much SharePoint exposure or experience.  Again, when people don't know SharePoint, they can't know what to ask for.  By giving the users prototypes or sneak previews into the solution you can identify gaps early and have a much better chance of delivering a solution the end user will actually use.

Test, test and test; be sure testers also have SharePoint training.  Otherwise how will they know what to test and how to test OOTB features?  During this testing be sure to test in an environment that simulates production.  If you're going to have a highly available environment with multiple Web Front End Servers, or you're using SSL, be sure to test these conditions.  I've seen many issues arise from these that are not reproducible in a single WFE, non SSL setup.  In addition ensure you test your nonfunctional features like Performance, High Availability, Disaster Recovery and monitoring.  After all if you don't test them, you don’t' have them.

Now that you have it in Production, the job isn't over.  We've simply moved to the next step in the SharePoint lifecycle: Build, monitor, review and improve.  After all the end users job is always evolving, your site will need to keep up.

Machine Translation Service

Overview

The new machine translation service in SharePoint 2013 is an attempt to finally make the variations feature useful.

The feature is intended to translate published content (documents, list items, entire sites) into other languages.  It is an extension of variations, that either synchronously or asynchronously sends the content to a translation service in the cloud.  By default it is configured to use a Microsoft cloud hosted translation service, but can be configured to use other third party services.  The key to this is that it uses a cloud hosted service to take the load of your internal infrastructure.

Besides the cloud based translation, this service application offers a few other improvements to the variations feature:

• Variations now supports up to 209 variation labels for on prem deployments and 50 variation labels in the cloud
•  No longer required to unpublish/publish documents to get changes to sync in related resources.  List/Libraries now sync independently
•  Localized sites now use XML Localization Interchange File Format (XLIFF).  This is the standard and makes it easier to use third party translation services to translate your app

Architecture

The architecture of this service is very similar to that of the Word Automation service; they both have similar components, for example: Timer Jobs and Document queues.  If you are familiar with the Word Automation service object model you should have very little problems using the machine translation object model.  Here is a great technet overview.

Asynchronous translation requests are handled through the document queue and a timer job.  By default the timer job is set to run every 15 minutes, but this configurable.

Like all other service applications, it can be configured through either PowerShell or the SharePoint Central Admin…but seriously, who uses Central Admin?

Prerequists

The following pieces are required to use the Machine Translation Service:

•  SharePoint 2013 Standard or Enterprise
• App Management Service is started
• Server – to – Server app authentication is configured
• User Profile Service Application Proxy in default group
• User Profile Service provision and configured
• Internet connection

API

The machine translation service provides the following APIs:

• Server Object Model
• Client Object Model
• REST Web Services

Through all three methods you are able to translate a single file or all items in a list, library or folder either synchronously or asynchronously.  You are also able to translate a file stream (must be synchronous), for on the fly translation of uploaded content.

Caution

This new service application is a very powerful new feature for SharePoint, but like anything else comes with its own set of risks.  The two main ones that jump out at me are: Security and Reliability.  Both are fairly obvious.  On the security front, you are now sending your content into the cloud.  This puts your content at risk of being seen by people both inside and outside your organization that may not have permissions to the content.  Generally this feature will be used for public sites which require multiple language support.  This mitigates the risk somewhat as the content was always intended for public consumption.  Of course this leads into the second risk, reliability.  Remember this is a machine doing the translation, not a human.  It is basically taking each work in the source document and turning it into the equivalent word in the destination language.  This may not end up conveying the message you are looking for.  Anyone who has traveled to non-English speaking countries have seen their fair share of engrish.  You probably want someone to proof reading the translations to ensure your website doesn’t become popular for the wrong reasons.

SharePoint 2013 Service Applications

What’s the same?

Architecture

Over all this is the same architecture as SharePoint 2010:

• Proxy Group (Groups of Service Applications consumed by a Web Application)
• Service Application Proxy (Proxy between the Group and the Application)
• Service Application (Search, User Profile, etc)
• Service Application Instance (may be multiple instances of certain service applications)
• Database (if needed)

This architecture is intended for multitenant (think cloud hosted) farms.  It allows you to have a central set of service applications and share them, as required, to all the site collections within the farm and even to site collections in different SharePoint farms.  The inter-farm sharing is a very interesting concept for the larger enterprise clients, along with companies hosting SharePoint.  When dealing with a large group of users these service applications can become very resource intensive.  One way to plan for this is to have a farm that is dedicated to the services.  This allows you to have smaller farm(s) that are only required the resources needed to render the SharePoint sites, but still have a common set of search results or user profiles for example.

Management

You have the same two choices: either through the Central Admin GUI or through PowerShell.  Although if you really consider yourself a SharePoint administrator, then your only option is through PowerShell.  All jokes aside, avoid setting up Service Applications through the GUI.  The SharePoint GUI makes lots of bad decisions when creating your service applications, for example: using GUIDs when creating the databases, using the server name when creating web applications for the service applications.

What’s Changed

Federation

This is what allows you to share service applications across farms both locally and remotely.  Although this hasn't really changed in SharePoint 2013, the service applications that can shared across farms has.  Here is a list of the service applications that can shared across farms:

• BCS 
• Managed Metadata 
• Search 
• Secure Store 
• Machine Translation Services 
• User Profile

In addition the remote farm no longer requires permissions to the parent database.

New Service Applications

Here is a list of the new Service Applications for SharePoint 2013

• Access Services: Create, deploy and manage collaborative web-based Access applications. This can also be used when developing SharePoint Apps 
• App Management Service: For the Market Place 
• Machine Translation Services: Cloud based translation service for documents, pages and sites. Has been built to be extendable, has the ability to use third party translators 
• Work Management Services: Puts all your outstanding tasks in My Tasks. Has two way sync with Project Server, Exchange and plugins for other systems (MS is very vague on what that means). Even has the ability to remind you of tasks on your mobile device. You’ll never be safe again

Improved Service Applications

These service applications have gotten even better in SharePoint 2013

• Managed Metadata: Improved Managed Metadata navigation 
• Search Service: FAST and Web Analytics have been rolled into search 
• Subscription Settings Service: Now handles app management 
• User Profile Service: Added back in a 2007 style sync (strictly read only and faster), ability to import additional properties from BCS

Removed Service Applications

These service applications did not make the cut in SharePoint 2013

  

• Web Analytics: Rolled into Search Service 
• Office Web Apps: Now its own product, but available for externally facing (internet) SharePoint sites

Alternate Access Mappings and SPFieldURLValue fields

Be careful if you're trying to programmatically pull back the value of a Hyperlink field when your site is using Alternate Access Mappings (AAM).
If the value of the Hyperlink field is pointing to an object within the current SharePoint site, the SharePoint front end will automatically update the URL for the AAM site you are currently using.  BUT if you are pulling the value programmatically it will always default to the URL of the default AAM site.


UPDATE:
Better Way to Fix:
I discovered an even better to fix this problem and it all has to do with how you open your SPSite Object.
For example if you do
SPSite site = new SPSite(SPContenxt.Current.Site.ID)
It will open the site in the default zone, which will then always return the URL of the default zone.
But if you do the following:
SPSite site = new SPSite(SPContext.Current.Site.ID, SPContext.Current.Site.Zone)
It will open the site in the zone the user is currently in and return the correct URL.

To fix this, you must turn the URL into a relative URL and then it will go to the AAM of the site you are currently logged into.  One way to achieve this is as follows:

SPFieldUrlValue linkfield = new SPFieldUrlValue(listItem["URL"].ToString());

string returnValue = linkfield.Url;

            try
            {
                using (SPSite site = new SPSite(returnValue))
                {
                    return returnValue.Replace(site.Url, string.Empty);
                }
            }
            catch
            {
                //swallow exception from Site Not Found
                return returnValue;
            }

SharePoint 2013: Initial Thoughts

Some initial thoughts after skimming through all the material Microsoft posted here: http://www.microsoft.com/en-us/download/details.aspx?id=30361

Over all this release looks to be more about improvements and refinements.  It does not contain any major overhauls like the Service Application layer of SharePoint 2010.  That being said there are some exciting new features.

Note:

SharePoint 2010 == SharePoint 14 == 14 Mode

SharePoint 2013 == SharePoint 15 == 15 Mode

Architecture:

This version of SharePoint is built on .Net 4.0.  I suspect it is using the new .NET Web API for their REST web services.

Looks like it would be possible to have a Service Farm in the cloud being consumed by an on-prem SharePoint farm.  In SharePoint 15 they have removed the database permission required in cross farm sharing of service applications.

Single Web Application can now consume both local and remote services.

Service Applications:

The service application architecture is the largely the same as SharePoint 2010.

Web Analytics has been rolled into Search.

Office Web Applications is no longer a Service Application.  Instead it now requires its own set of VM(s).

New: App Management Service – To support SharePoint Apps from either the Market Place or Corporate Catalog

New: Translation Service:  Cloud based translation service for documents, pages and sites.

User Profile Application now includes the User Profile Replication Engine.  In 2010 this was a separate download required to replicate User Profiles and social to another farm.  This was very handy for Disaster Recovery.

Request Management:

SharePoint 15 provides the ability to route requests based on the following information:

·         Url

·         UrlReferrer

·         UserAgent

·         Host

·         IP

·         HttpMethod

·         SoapAction

·         CustomHeader

This could prove to be one of the most useful features of the new version.  This allows you to reject potentially malicious requests.  It could also allow you to route request to certain servers based on the data requested or the person who sent the request.  In a large organization this could allow you specify sets of servers to handle requests for certain departments.

Custom Code:

SharePoint 2013 will still handle Farm Solutions (same as 2007 & 2010) and Sandboxed solutions.  In addition it has added SP Apps.  This, apparently, will be the preferred method of deploying solutions to SharePoint 15.

SP Apps:  Not really sure how solutions are going to map to an app yet, but here some details on how an app will function:

·         One App == One Web == One URL

·         Only deploy Web Scoped Features (SPWeb)

·         App features are not available outside of the App

·         End users cannot manipulate the app web with SharePoint Designer or the browser

·         App Package will contain: Pages, Scripts, Styles and Manifest

·         App can be hosted on: A Developer Server, Azure, Any SharePoint server

·         A SharePoint farm can use Apps from the Market Place or from their private Corporate Catalog

·         Separate license for each app (The App Management Service will manage all these licenses)

·         Looks like web config must have Trust Level set to Full for apps to work

·         Can be built in the new version of Visual Studio

Upgrade:

Looks like there have been some more improvements to the upgrade process from SharePoint 2010.  Most notably is there will be a SharePoint 2010 mode, like Visual Upgrade but more.  SharePoint 15 will have all the SharePoint 14 features, when in SharePoint 2010 mode the site (and all customizations) will still be running all the SharePoint 2010 code and features and all customizations will still live in the 14 hive, not the new 15 hive.

Preferred method for upgrade is the database attach method (same as 2010).  The following databases will support this method:

·         Content Databases

·         Project Database

·         Search Admin

·         Profile

·         Social

·         Managed Metadata

·         Secure Store

PowerShell Commands: Test-SPContentDatabase (same as 2010) and the new Repair-SPCotnentDatabase to fix the issue discovered in the Test-SPContentDatabase command.

A Site Collection Health check that will run before the upgrade to ensure everything is ok before upgrade.

Ability to create side by side copies, one in 14 mode and one in 15 mode that can be run at the same time.

You can also create new site collections in 14 mode.  This could come in handy during a migration.

Social:  Haven’t read too much about this, but the images look very similar to the NewsGator sites we’ve setup…

That’s it for now