The Problem

At Bamboo SPX Services, we support a large custom-developed SharePoint 2013 solution for a client at the Department of Justice. This solution began development about three years ago; the method of development was by way of a custom WSP solution. This application uses SharePoint to automatically process and store all email correspondence received from the US Courts PACER system.

Currently, we process, download, and store all pertinent data and documents related to each notice received. On average, this system is storing 15k Electronic Court Filing records and 9k downloaded documents per day through this automated solution. I would love to discuss the methods we have used to scale this project in SharePoint but that would require its own blog post (coming soon). The problem we have with this current solution is that the future of SharePoint does not include the now, “untrendy”, WSP full-trust-farm solutions. We are all very aware of the application migration tales that occur with every version change in SharePoint. This is a key reason that many SharePoint environments never upgrade to the latest versions. Compatibility issues and effort required to rewrite custom apps into “the way we do it now” development method.

Today, we consult and support many of our government clients with their migrations to SharePoint online (Office 365), where it is not possible to install WSP-type solutions. So how do we prepare this large application to be cloud-ready and adaptable to future practices?

The Applications Features

The CaseView PACER Application is comprised of 105 site collections, custom timer jobs, and an email-enabled centrally located library to accept all of the mail that comes in each day from the US Courts PACER system. The SharePoint timer job runs on each server in the farm, thus allowing us to simultaneously process messages in a multithreaded fashion. Each instance of the timer job is responsible for processing one email at a time and when complete, will go and find another from the mail enable library. After successful parsing and downloading of documents for each email, we store the associated files in their respective site collection.

The performance has been solid and able to keep up with the firehose of mail coming from the US Courts PACER system but has a few bottlenecks, as well as it not being ready to move into the cloud-ready future.

The Way Forward

After reviewing the functionality the application currently provides and comparing that with the direction and methodology currently prescribed in the SharePoint development community, we came up with our solution. A standalone Windows Service Application to do all of the processing, followed by REST functions that do all of the communicating with SharePoint. I.e. sending the results (email and documents) back to SharePoint. This way we can scale our processing power. Limited only by the virtual machine specs in which we install the service application. Converting to a Windows service application meant we had to refactor a lot of the code for this solution to stop relying on timer jobs to initiate processing and instead use C# code and multi-threading. Now through a simple configuration file change, we can adjust many features of our application such as how many threads will process simultaneously without any re-compilation of our code to implement. We simply restart the windows service and now we can run X amount of threads at a time to process mail. Much of the SharePoint .Net API that existed in our full-trust solution had to be removed and rewritten using standard C# patterns and practices so that this application could run standalone and with no dependencies to SharePoint.

Refactoring away from the hardcoded dependencies with SharePoint affords our customers the ability to host this application in any virtualized environment and to utilize any SharePoint environment (Cloud, On-Premise, Hybrid) that they may choose to migrate to in the near future. When the next version of SharePoint releases, this solution will be ready to migrate/integrate, and will not require a large-scale development effort. So long as the SharePoint REST API exists, which the web as a whole has adopted, then this application can continue to add features and store data for future versions to come.