SharePoint Output Cache

What is this SharePoint Output Cache and why should I use it?

In ASP.NET there is an output cache that manages how pages content is severed.  It allows IIS to cache static elements, such as images and pages so that subsequent requests do not need to go looking for these items.  Similar to how your browser cache keeps you from downloading the same images over and over again.  The big advantage with SharePoint is that when this is enabled it caches fully rendered versions of pages. 

When SharePoint loads a page it is a really big process.  It needs to get the Master Page from the file system or the database.  It needs to get the Page Layout from the file system or again the database.  It needs to get all CSS, all the images and all the JavaScript.  From here it can start rendering the page, but there is more.  For every security trimmed or audience trimmed control it needs to make a call back to the database to determine if it should render it or not.  BTW:  You read correctly it doesn’t make one giant call to figure out what to do with each security trimmed object, it calls them one at a time.  This can take a bit of time when you think about all the buttons in the ribbon that may or may not appear, on top of the web parts the page requires.

As you can see that is a lot of calls just to render one page.  With output caching enabled this fully rendered page will be cached and every subsequent call will not have to go through all the above process.  It will simply make one call and get one page back.  As you can image this makes a huge difference on throughput performance.

To demonstrate just how much of an improvement this makes, I load tested the a customized SharePoint 2013 site.  This site isn't a very heavy site, but does have a lot of JavaScript for responsive design on top of all the other SharePoint scripts that are required.  I ran the same set of tests, with same number of users against the same site and you see the results in the below chart:

Counter	Before	After
Average Page Load Time (s)	23.4	1.4
%Processor Time App Server	24.8	16.8
%Processor Time WFE Server	81.1	41.5

As you can see it made quite an improvement in Page Load Time and took a significant amount of stress off both servers.

Now you are probably wondering why this wouldn't be enabled by default, since it provides such significant performance gains.  Well there is a bit of a catch to this.  It is caching a version of the page, this means there is more memory being used on the server to store the page.  On a publishing site this is a bigger piece as it will cache two version of page (published and draft).  Also, as you've guessed this could lead to some inconsistencies as well.  The cached page is removed when the page is updated, but there is a bit of a delay and each WFE has its own timer on when it updates the page.  In a farm with load balanced multiple WFE servers it is possible that the first request could go to a WFE that has updated the page in cache and a second request goes to a WFE that has not.  The time to update is by default 60 seconds, so this is a small window, but still possible.

To address some of the issues is the ability to create caching rules.  You can target rules to a Site Collection, Site and Page Layouts.  You can also create caching rules based on a user’s access rights, so that all readers see cached versions, where contributors do not.  This rules can also be extended programmatically through the VaryByCustom handler.

Overall, in my opinion, Output Caching should always be leverage for low write, high read sites.  Also considering that you can have rules set for different page layouts, you could set the caching time higher for the a low write page, like maybe the home page of an intranet, and set the time lower on high write pages within the site.  With careful planning this feature can really help scale out the farm to handle more users while saving server resources.

For more information on output caching and how to enable it, see this helpful MS link: http://msdn.microsoft.com/en-us/library/aa661294(v=office.14).aspx

Tweet from SharePoint

Sounds easy enough, but it turned out to be a little more challenging that I had originally anticipated.  Twitter recently updated their API and now requires authentication on every request, using OAuth.  It’s actually fairly simple once you know what it’s expecting, so I thought I’d share.

If you are using SharePoint 2013 there is a good library on CodePlex.  It’s called Tweetinvia and it’s a nice C# library that takes care of everything for you.  You can download it here and then proceed to the security sections below.

If you are still using SharePoint 2010 this library will not work for you, as it was compiled in .NET 4.0 and uses a lot of features that are only available in .NET 4 and higher.  Unfortunately I was in this boat, so I had to build my own .NET 3.5 library to do this. I created two projects: Twitter and TwitterTest.  The first one contains all the code required to post an update to Twitter.  The second is console application to test it with.  This project is not as robust as the Tweetinvia project.  I only required the ability to post, so that is all I built.  

Ok, so how did I do this and what was required.

The basics:

The Twitter API is basically a set of REST web services.  To make an update you must send a POST request to this address: https://api.twitter.com/1.1/statuses/update.json?status=[tweet]&trim_user=False.  Replace the [tweet] with your message and you are almost there.  Not bad so far.

As I mentioned above the Twitter API now requires authentication, which is provided in the HTTP authentication header.  Twitter is looking for 4 keys, the values are unique per Twitter application and are provided by Twitter (see below for more details):

•  Consumer key
• Consumer secret
• Access token
• Access token secret

So all you need to do is send a tweet to the API, using the URL above, and ensure you have the proper OAuth Authentication token from the above keys.

To do all this I used the Tweetinvia library as a map.  I ripped our required methods from the OAuthToken, OAuthWebRequest projects.  Ripped out the required Utilities, Enums and Interfaces and created my own Tweet class that wired them all together as required.  Fortunately for the most part the methods I needed did not have any dependencies on .NET 4.0 and the little bit that did was easy enough to rewrite in .NET 3.5.  

At a high level the OAuthToken class is responsible for OAuthToken and leveraging the OAuthWebRequest methods to send a proper request.  The OAuthWebRequest generates the web request with the proper authentication header.

Security, Getting the Key values from Twitter:

To do this you need to go https://dev.twitter.com, login as your twitter account (BTW: at imason you can use imasontest, Alice…).  Go to My applications.  To get there in the top right hand corner there is your profile picture (or the beautiful egg if you haven’t updated it), click on the drop down arrow and select my applications.  Create a new Application, fill out the fields, it doesn't seem to matter what the website or callback URLs are (disclaimer: it didn't for my project and I don’t actually know what they are used for…so it may for you).  Once created, on the settings tab there is a section called Application Type: make sure it’s set to Read and Write (assuming you want to POST) and click the update this Twitter applications settings.  Now on the OAuth Tool tab you will see the 4 keys and their values.  If you use these and try to POST to this account, it should work.

Security, the SharePoint Twist

So at this point, the console application will work…but when you try to call this from SharePoint it fails…of course it does, there is always a “helpful” SharePoint twist.  The missing piece of the puzzle is configuring SharePoint to trust the Twitter API URL.  To do this you need to get the Root CA Certificate that Twitter uses.  You can grab this directly from Verisign or you get it from Twitter, you are looking for the Class 3 Public Primary Certification Authority – G2.cer file.

Once you have this file you can configure SharePoint to trust it.  Open Central Admin.  Go to the Security Section.  Click on Manage Trust.  Click New.  Fill in a name, I used “Twitter”.  In Root Certificate for the trust relationship, click browse and locate the Class 3 Public Primary Certification Authority – G2.cer file.

With that the communication between SharePoint and Twitter should now work.  

SharePoint Client Object Model

I have been doing a lot of work with the SharePoint Client Object Model (COM) and wanted to share some of my experiences.  The COM is going to be more important as we start building more mobile applications and think about building SharePoint apps.

First off, I did not start with the SharePoint COM; I thought I would be able to build my site using the SharePoint REST web services.  One of my requirements though was to be able to use Anonymous access and that just doesn’t seem to be possible with the REST web services in 2010.  Although, if anybody knows how to do this, I would love to know.  So I ended up with the COM as it supports both authenticated and anonymous users (assuming of course that your Site Collection has been properly setup to allow anonymous users).  Overall the COM and REST web services do pretty much the same thing, although the COM does have this neat feature of being able to put together multiple actions into a single request.  All and all I may have been forced into the correct method after all.

The basic usage of the COM is very similar to the Object Model.  For example here is the code to get all the list items from a list:

using (ClientContext sharePoint = new ClientContext(_strSPURL))

{

                Web web = sharePoint.Web;

                ListCollection spLists = web.Lists;

                List spList = spLists.GetByTitle("Pages");

                CamlQuery camlQuery = new CamlQuery();

                camlQuery.ViewXml = @"<View>

                                    <ViewFields>

                                        <FieldRef Name='ID'/>

                                        <FieldRef Name='AudienceTaxHTField0' />

                                        <FieldRef Name='ServiceTaxHTField0' />

                                        <FieldRef Name='Title' />

                                        <FieldRef Name='ContentTypeId' />

                                    </ViewFields>

                                    <Query>

                                        <OrderBy>

                                            <FieldRef Name='Modified' Ascending='False'/>

                                        </OrderBy>

                                    </Query>

                                </View>";

                Microsoft.SharePoint.Client.ListItemCollection spListItems = spList.GetItems(camlQuery);

                sharePoint.Load(spListItems, items => items.Include(

                                                        item => item["ID"],

                                                        item => item["AudienceTaxHTField0"],

                                                        item => item["ServiceTaxHTField0"],

                                                        item => item["Title"],

                                                        item => item["ContentTypeId"]));

                sharePoint.ExecuteQuery();

                return spListItems;

}

As you can see, it is pretty much the same as you do now.  The ClientContext is like opening a SPSite object.  After that we open a SPWeb object, get the list, run a CAML query and return the items.  The other thing you may notice is that I explicitly tell SharePoint which fields to return.  This ensures SharePoint only returns the data I’m interested, the smallest package possible; as a side note you should also be doing this when using the Object Model…I mean have you seen all the crap SharePoint returns if you don’t do this?

Some gotcha’s

You can only use this against a Site Collection.  You have no access to anything higher.

The biggest gotchas I ran into where with the Taxonomy fields.  In the 2010 COM there are no Taxonomy Field objects (they are in the 2013 COM).  This means when trying to get data from Taxonomy fields you will have to reference the hidden Taxonomy field that is associated to the field you are interested.  You’ll noticed in the above example I’m referencing AudienceTaxHTField0 and not Audience which is the field you would see when using the site.  The other piece is the value coming back from this field will be the raw form, meaning that you will need to massage it a bit for it to be useful, unless of course you love GUIDs.

The other thing to keep in mind is performance.  Since your application will be making web service calls to SharePoint, using the COM, each request will have a minimum round trip time.  When coding try to limit the amount of round trips as much as possible and if possible think of using asynchronous methods so that the page doesn’t have to wait for all the data before it starts rendering.  Remember, it will be quicker to get a bunch of data and massage it in your app/site rather then make several little calls…to a point as if you ask for too much data this will also have a negative impact on the round trip time…you’ll need to find a balance.

References:

SharePoint 2010 Client Object Model: http://msdn.microsoft.com/en-ca/library/ee537247(v=office.14).aspx

Using the SharePoint 2010 Managed Client Object Model: http://blogs.msdn.com/b/ericwhite/archive/2009/11/20/using-the-sharepoint-2010-managed-client-object-model.aspx

SharePoint Project Planning

Congratulate yourself for actually accepting planning is required with SharePoint.  I know Microsoft likes to tell us you can just turn it on and use it, but we all know that’s just not the case.  Too many places are stuck with the pilot that never ends.  Starting with pilot or POC is great approach, but for it to work you need to circle back and actually plan a proper deployment that includes DR, Availability and Performance.  The Pilot that never dies had none of these consideration and is like a ticking time bomb, who will be the first user that picks up the phone to complain the site is unusable?  Will it be the CEO?  More importantly will you be able to fix it?

To begin with make sure you have the correct people for the job.  If you are going to train up internal resources to do the job, ensure the training happens early, ideally before any solution planning.  This is because the team planning the SharePoint solution will need to lead the end users to the solution.  It's hard for the end user to ask for what they want, when they don't know what is available.  Ask any SharePoint professional and there are many tools, tips and quirks to using SharePoint that only someone with experience would think of.  If no one knows how SharePoint works, it becomes the blind leading the blind.  Which is why, even if you are planning to train up your internal resources, you should ensure you have an experienced SharePoint professional on the project.  As we all know, there are something's that only experience can teach us.

Once you have armed your team with this knowledge, the end users are going to ask for things that are known limitations of SharePoint.  While it is always a great strategy to use OOTB functionality, don't be too rigid on this.  Rather than just saying no, or agreeing to a complex custom solution, be prepared with work arounds or alternatives that may speak more to the spirit of the request.  After all, if the site is usable to the end user, who is going to use it.  But if it's overly customized who is going to support it?

Now that you have enough information to design a solution that will meet the end users' needs don’t' forget to consider the following in your design: High Availability, Disaster Recovery, Automated deployment to the multiple environments (UAT, Staging, Production), Security, Internal vs. External access, mobile (becoming bigger by the day), performance and of course monitoring. 

Be agile in development, especially with people that don't have much SharePoint exposure or experience.  Again, when people don't know SharePoint, they can't know what to ask for.  By giving the users prototypes or sneak previews into the solution you can identify gaps early and have a much better chance of delivering a solution the end user will actually use.

Test, test and test; be sure testers also have SharePoint training.  Otherwise how will they know what to test and how to test OOTB features?  During this testing be sure to test in an environment that simulates production.  If you're going to have a highly available environment with multiple Web Front End Servers, or you're using SSL, be sure to test these conditions.  I've seen many issues arise from these that are not reproducible in a single WFE, non SSL setup.  In addition ensure you test your nonfunctional features like Performance, High Availability, Disaster Recovery and monitoring.  After all if you don't test them, you don’t' have them.

Now that you have it in Production, the job isn't over.  We've simply moved to the next step in the SharePoint lifecycle: Build, monitor, review and improve.  After all the end users job is always evolving, your site will need to keep up.

Alternate Access Mappings and SPFieldURLValue fields

Be careful if you're trying to programmatically pull back the value of a Hyperlink field when your site is using Alternate Access Mappings (AAM).
If the value of the Hyperlink field is pointing to an object within the current SharePoint site, the SharePoint front end will automatically update the URL for the AAM site you are currently using.  BUT if you are pulling the value programmatically it will always default to the URL of the default AAM site.


UPDATE:
Better Way to Fix:
I discovered an even better to fix this problem and it all has to do with how you open your SPSite Object.
For example if you do
SPSite site = new SPSite(SPContenxt.Current.Site.ID)
It will open the site in the default zone, which will then always return the URL of the default zone.
But if you do the following:
SPSite site = new SPSite(SPContext.Current.Site.ID, SPContext.Current.Site.Zone)
It will open the site in the zone the user is currently in and return the correct URL.

To fix this, you must turn the URL into a relative URL and then it will go to the AAM of the site you are currently logged into.  One way to achieve this is as follows:

SPFieldUrlValue linkfield = new SPFieldUrlValue(listItem["URL"].ToString());

string returnValue = linkfield.Url;

            try
            {
                using (SPSite site = new SPSite(returnValue))
                {
                    return returnValue.Replace(site.Url, string.Empty);
                }
            }
            catch
            {
                //swallow exception from Site Not Found
                return returnValue;
            }

How to Properly use RunWithElevatedPrivileges

RunWithElevatgedPrivileges will run the code block with Full Control rights even if the current user does not have full control.  This piece can be very useful, but must be used carefully to avoid creating a security hole.

The interesting part of this method is that for any objects to be run with this elevated context, they must be created within the RunWithElevatedPrivileges block.  This may seem obvious but lets take a look at a couple examples:

  

SPWeb web = SPContext.Current.Web;

SPSecurity.RunWithElevatedPrivileges(delegate()  

       {  

         PublishingWeb pWeb = PublishingWeb.GetPublishingWeb(web);

         ...

       })  

This method will result in an unauthorized error, unless the logged in user has enough permissions to open a Publishing Web object.  But we said the object must be created within the Elevelvated Permission block, so it looks like an easy fix to the above code:

SPSecurity.RunWithElevatedPrivileges(delegate()  

       {
        SPWeb web = SPContext.Current.Web;

         PublishingWeb pWeb = PublishingWeb.GetPublishingWeb(web);

         ...

       })

This method will also result in an unauthorized error.  Wait you say, the SPWeb object is created within the Elevated block, so it should work.  The issue is that the SPContext.Current.Web method returns an SPWeb object under the context of the current user, not the Elevated user, regardless of where the method is run...so how do you get an SPWeb object to run under the elevated permission?  You need to open the web using the GUID constructor, like this:


Guid siteID = SPContext.Current.Site.ID;
Guid currentWebID = SPContext.Current.Web.ID;

SPSecurity.RunWithElevatedPrivileges(delegate()
            {
             using (SPSite site = new SPSite(siteID))
                {
                    SPWeb currentWeb = site.OpenWeb(currentWebID);
                    PublishingWeb pWeb = PublishingWeb.GetPublishingWeb(web);
                    ...
                }
            })

The difference being the first and second examples are returning an already created SPWeb object, that was created under the current users context.  The last example is returning newly created SPSite and SPWeb objects, since they are within the elevated code block they will be created under the elevated context.  Now everything should work as expected, regardless of who is logged into the site.

             

SharePoint 2010 Install Error

During a recent install of SharePoint 2010 I got this error message:

An exception of type System.Security.Cryptography.CryptographicException was thrown…

There was a little more to it, but not really anything of help. 

I googled the error message and found many blogs that suggested giving Network Service full control over the 14 hive (SharePoint root).  This not only seemed like over kill but also a security hole, there is a good reason why the Network Service account does not have full control over the 14 hive.  

Not liking the popular fix, I dove in a little more and found a much easier and secure solution.

When you run the SharePoint 2010 Products and Technology wizard, be sure to right click and select “Run As Administrator”

…that’s it. 

SharePoint 2010 Console App returns ErrorWebParts

On a recent upgrade project I needed to update the properties of a Content Query Web Part. Sounds easy enough, just need to whip a quick console app and away we go.

Coding went very quickly, but when I ran the app I couldn't get any Content Query Web Parts, they were all returned as"ErrorWebParts".

I consulted my good friend google and found this blog post [1]. This guy was having the same issue as me and discovered that this web part references a SPContext object, unfortunately from a console app there is no web-context.

After reading this, I modified my code to set the HTTPContext.Current and like magic, I was able to edit the properties of all the CQWP.
[1] http://sharepoint-insight.blogspot.com/2008/10/sharepoint-all-webparts-appear-as.html

SharePoint Content Query Web Part not returning all results

On a recent SharePoint 2010 migration project we noticed that when we created new items (pages in the pages library), the Content Query Web Parts did not immediately display the results.

This seemed very odd at first, as I had never noticed a delay before with these web parts, so my first thought was a migration issue. But why would a migration only cause the web part to work slower.

So with the help of my good friend Google, I was able to find a blog post [1] that offered the following:CQWP ignores the following items:

1. Items are check-out
2. Items are not published and not-approved

But I was hitting publish on the page, so my items would not have fallen into either of these categories...unless it was just a delay in SharePoint marking the item as published.


Following the advice in the same blog post, I wrote a console app that targeted all the CQWP and changed the "UseCache" property to false.


Just like magic the items I created are now showing up immediately in their respective CQWPs.


Keep in mind this may not be the best solution for a site with a high load or CQWPs that are running large queries. You'll need to weigh the benefits of performance against immediate results. In most cases users will understand if there is a small delay as long as they have an understanding of the situation.


[1] http://msmvps.com/blogs/laflour/archive/2008/12/24/why-content-query-web-part-cqwp-doesn-t-return-all-results.aspx

Migrating SharePoint MySites from 2007 to 2010

The MySites, for me anyways, are like the forgotten child of SharePoint. When migrating I put a lot of thought into how to move their existing site collections and solutions. What do they currently have setup in the SSP and how are we going to recreate them in the Service Application architecture of 2010. But I never put a lot of thought into how are we going to move the MySites?

On the surface this doesn't sound like a big deal. MySites are site collections like any other, simply built off a different template. Ideally they are hosted on their own Web Application and have their own separate content database(s); so what's the big deal?

This view isn't that far off, but there are a couple gotcha's that you need to consider before making the move:

1. No Visual Upgrade option for MySites (or at least I couldn't find one). This means you will be forced to the 2010 Master Page.
2. All Profile Properties are stored in the SSP database, not the MySite Database. Any custom properties, or any customized values will not be migrated over with just the MySites Content Database.
3. Audiences are stored in the SSP. Web Parts referencing audiences will be referencing the GUID of the Audience, not the title. This makes it very difficult to just recreate your Audiences on the new environment.

The solution to the last two issues is to migrate over the SSP database as the profile database of the UserProfile Service Application in the SharePoint 2010 environment.

To do this you can use this PowerShell cmdlet to create the new user profile Service:
$ProfileGUID = New-SPProfileServiceApplication –Name [User Profile Name] –ApplicationPool [App Pool Name] –ProfileDBName [SSP Database] [1]

The above command will create a User Profile Service Application and upgrade the SSP Database. You can review the upgrade by looking at the Upgrade Status page. Next you'll need to run this command:
New-SPProfileApplicationProxy –Name [User Profile Proxy] –ServiceApplication $ProfileGUID [2]

Next up, the multi value User Properties do not come through right away. This, again, is due the new Service Application structure. The multi values were stored in the SSP in MOSS, but are now stored in the Managed Metadata Service in SharePoint 2010. Fortunately there is a handy PowerShell cmdlet that help map the Properties to the Managed Metadata Service:
Move-SPProfileManagedMetadataProperty -Identity [Profile Property] -ProfileApplicationProxy [GUID of User Profile Application Proxy] -AvailableForTagging -TermSetName [Term Set Name] [3]
I ran the above command against these properties:
SPS-Interests
SPS-MemeberOf
SPS-PastProjects
SPS-Responsibility
SPS-School
SPS-Skills

Unfortunately, this command no longer works after you install the July CU or later [4]. If possible run this command after SP1, but before the CU.


Finally, I hope, there is the matter of the picture dimensions. In SharePoint 2010 they have changed things and now use three different sized pictures for the different areas of SharePoint. [5] When a user upload a picture SharePoint automatically creates these 3 different sized pictures. But what do we do about the pictures that already exist due to our migration? Once again it's powershell to the rescue. You'll need to run this cmdlet:
Update-SPProfilePhotoStore -MySiteHostLocation [URL to MySite Host] [6]



Now you should be able to open up this newly created User Profile Service Application. You should see that all your audiences, custom user profile properties and all the user profiles are all there.
To finish off this off, you will need to run a couple configuration steps:

1. Start the User Profile Synchronization Service
2. Re-Map any custom user profile properties to their AD property
3. Configure Synchronization Connection and any filters
4. Setup MySites
5. Start Full Profile Synchronization

Now your upgraded MySites should be ready to use.


[1] http://technet.microsoft.com/en-us/library/ff608036.aspx
[2] http://technet.microsoft.com/en-us/library/ff607737.aspx
[3] http://technet.microsoft.com/en-us/library/ff608037.aspx
[4] http://blogs.technet.com/b/rycampbe/archive/2011/11/22/june-2011-cu-move-spprofilemanagedmetadataproperty-bug-and-the-workaround.aspx
[5] http://blogs.msdn.com/b/spsocial/archive/2011/01/07/photo-management-in-sharepoint-2010.aspx
[6] http://technet.microsoft.com/en-us/library/ff607547.aspx