Thursday, October 30, 2008

Learning Surf 4 - Modifying Share documentLibrary with Create Content button


This and the next blog entries will walk through the process of adding a new button to the documentLibrary section of Share which will allow the creation of new content using an in-line WYSIWYG editor.

This blog will add a 'Create' button to the toolbar of the Document Library module and open a popup when clicked showing a message 'feature not implemented yet'

Step 1 - mapping the terrain

As I have done in previous blog entries, I will draw a picture of the dependent files (xml, freemarker and javascript) that renders the relevant portion of the document library, and indicate what needs to be modified.



As shown before, the key files are:
  • 'site-data/pages/documentLibrary.xml' page references 'documentLibrary.xml' template-instance
  • 'site-data/template-instances/documentLibrary.xml' contains reference to 'org/alfresco/documentLibrary' template type
  • 'templates/org/alfresco/documentLibrary.ftl' contains the structure of the page with regions containing components including the 'toolbar' region
  • 'site-data/components/template.toolbar.documentLibrary.xml' defines the component to fit in region 'toolbar' of template 'documentLibrary' refrencing a webscript ui component registered to the url '/components/documentlibrary/toolbar'
  • site-webscripts/org/alfresco/components/documentlibrary/toolbar.get.desc.xml' describes the ui webscript registered to the '/components/library/toolbar' url, it includes freemarker renderers 'toolbar.get.head.ftl' and 'toolbar.get.html.ftl'.
  • 'toolbar.get.head.ftl' references toolbar.js and toolbar.css found in 'web/components/documentlibrary/' source folder.
Step 2 - Add the button to the template

The 'toolbar' component contains the space for the 'create' button I wish to add. Within the 'toolbar.get.html.ftl' I have added a couple of lines for the new button:

 <div class="new-folder hideable DocListTree"><button id="${args.htmlid}-newFolder-button" name="newFolder">${msg("button.new-folder")}</button></div>

<!-- Added by EEW 20081030 -->
<div class="separator hideable DocListTree">&nbsp;</div>
<div class="file-create hideable DocListTree"><button id="${args.htmlid}-fileCreate-button" name="fileCreate">${msg("button.create")}</button></div>
<!-- End Added -->

<div class="separator hideable DocListTree">&nbsp;</div>
<div class="file-upload hideable DocListTree"><button id="${args.htmlid}-fileUpload-button" name="fileUpload">${msg("button.upload")}</button></div>


Step 3 - Add Messages for the button

This code also changes the 'toolbar.get.properties' file adding the key/value 'button.create=Create',


Step 4 - Add CSS Entries and Images for the button

 and the toolbar.css used to render the button:


/* added by EEW 20081030 */

.toolbar .file-create button
{
background: transparent url(images/create-16.png) no-repeat 12px 4px;
padding-left: 32px;
}
.toolbar .file-create .yui-button-disabled button
{
background-image: url(images/create-disabled-16.png);
}

/* done added */




As implied by the css above, I have also created two new images to appear with the button: create-16.png and create-disabled-16.png. For testing purposes, I simply copied edit-blog-16.png.

Step 5 - Add the Button as a YUI Button

In 'toolbar.js', I have added a line to instantiate the YUI button named in the toolbar template in the 'onReady' YUI event listener

   
// added by EEW 20081030

// File Create button: user needs "create" access
this.widgets.fileCreate = Alfresco.util.createYUIButton(this, "fileCreate-button", this.onFileCreate,
{
disabled: true,
value: "create"
});
// done added

// File Upload button: user needs "create" access
this.widgets.fileUpload = Alfresco.util.createYUIButton(this, "fileUpload-button", this.onFileUpload,
{
disabled: true,
value: "create"
});



In this code, we see that the YUI button ties the name 'fileCreate-button' (which is the id in the button tag defined in 'toolbar.get.html.ftl'), and the event name 'onFileCreate', which will fire when the button is clicked. 

Step 6 - Add the Event Code

The event is also defined in toolbar.js:

      /**
* File Create click handler
*
* @method onFileCreate
* @param e {object} DomEvent
* @param p_obj {object} Object passed back from addListener method
*/
onFileCreate: function DLTB_onFileCreate(e, p_obj)
{
/*
var url = YAHOO.lang.substitute(Alfresco.constants.URL_CONTEXT + "page/site/{site}/blog-postedit?container={container}",
{
site: this.options.siteId,
container: this.options.containerId
});
window.location = url;
Event.preventDefault(e);
*/
var notimpTitle = this._msg("title.notimp");
var notimpMsg = this._msg("message.notimp", "Create Content");

Alfresco.util.PopupManager.displayPrompt(
{
title: notimpTitle,
text: Alfresco.util.decodeHTML(notimpMsg),
noEscape: true,
modal: true,
buttons: [
{
text: this._msg("button.cancel"),
handler: function DLTB_onActionDelete_cancel()
{
this.destroy();
}
}]
});

},





This event handler will open a popup message for now showing that this feature is not yet implemented.

Step 7 - Add The Messages for the Popup Message Box

At this point I am only defining a placeholder for this event. In order to support the above popup, I have included the message keys to 'toolbar.get.properties':

## Added by EEW 20081031

button.create=Create
title.notimp=Not Implemented
message.notimp = Feature {0} is not implemented yet.

## Done Added



Step 8 - Build, Deploy and Test

To test, I use ant task 'incremental-slingshot-tomcat' to build and deploy my changes to tomat. As a remider, the ant task requires environment variables TOMCAT_HOME and APP_TOMCAT_HOME to be defined to the installed tomcat.

And you should see this:





Wednesday, October 29, 2008

Learning Surf 3 - More analysis of Slingshot (Share) - looking at Blog List

Please see my previous blog entry for more detail on general concepts of Slingshot's configuration as a Surf platform application.

In the next blogs, I will attempt enhancing Share to add a 'create' button along side of the 'upload' button in the 'document Library' section which will open an in-place WYSIWYG editor to create html based content. In preparation for this, I will look at similar code implemented to post a blog entry.

In Share (Slingshot), creating or selecting a site from the dashboard 'sites' dashlet opens the site dashboard. The top navigation provides buttons to select 'Blog' to view and enter blog entries. Selecting 'blog' creates the URL:

http://localhost:8080/share/page/site/mc/blog-postlist

One of the difficulties in getting your head around working with Surf based web applications is understanding how everything is connected together. I will attempt to show how the bloglist page is rendered.




This diagram shows my map of the dependencies of the various xml, javascript and freemarker files that assemble the bloglist page.

Decomposing the url, we see the pattern /page/* which maps to the pageRendererServlet. Using the 'config/alfresco/web-framework-config-application.xml' which defines url patterns to find the /site/{site}/blog-postlist from the object model defined in xml files, found according to the search path for the types of components it is looking for (see previous blog).

'site-data/pages/blog-postlist.xml' defines a page model object for the page. It refers to a template instance blog-postlist.xml.

'site-data/template-instances/blog-postlist.xml' defines a template-instance model object which refers to the template-type found using org/alfresco/blog-postlist

'templates/org/alfresco/blog-postlist.ftl' is found as the search for template types assumes a freemarker template at the path defined if not explicitly defined see surf developer documentation. This template works in conjunction with component definitions to associate components with regions defined in the source template.

'site-data/components/template.postlist.blog-postlist.xml' provides component mapping to region 'postlist' in source id 'blog-postlist', mapping to a component found by url '/components/blog/postlist'

'site-webscripts/org/alfresco/components/blog/postlist.get.desc.xml' ui webscript is registered to the url pattern defined for the template and uses 'postlist.get.head.ftl' and 'postlist.get.html.ftl' as renderers.

'site-webscripts/org/alfresco/components/blog/postlist.get.head.ftl' links to the 'postlist.js' javascript.

'source/web/components/blog/postlist.js' among other things defines a yui button to create a blog post (createPost-button), which when pressed will call the onCreatePost action handler.

'site-webscripts/org/alfresco/components/blog/postlist.get.html.ftl' renders the Blog post list page, including the 'createPost-button' yui button. Clicking on this button calls the onCreatePost action handler defined in postlist.js' which will in tern compose the url

http://localhost:8080/share/page/site/mc/blog-postedit?container=blog

This page renders a WYSIWYG editor configured to allow creation of blog entries.

Monday, October 27, 2008

Learning Surf 2 - Examining slingshot configuration

Examining slingshot configuration

You can see my previous blog entry for a background in getting started with surf.

The Surf platform is a web script based framework for developing web 2.0 based applications that can use the alfresco content repository. It relies on the technologies of xml, freemarker and javascript. It heavily leverages alfresco's freemarker templates and the javascript api, and the alfresco's web script technologies that define ui components or RESTful services.

Surf based applications can be developed without an Alfresco content repository access, but ones that use the content repository are developed and deployed as two war files, one for the web application built on surf (i.e. share.war), and one for the content repository it accesses (alfresco.war). These war files can be deployed within the same application container (i.e. tomcat) or different application containers on the same machine or different machines. The web application communicates with the content repository over RESTful services (essentially over http).

The share application is a surf framework application. The source tree stores share in a project called 'slingshot'. Slighshot depends on other projects in the source tree, including:
  • 3rd Party
  • Core
  • Web Framework
  • Webscript Framework
Configurations of a surf application are contained in the following types of configurations:
  • war configuration files (web.xml)
  • alfresco configuration files (i.e. web-framework-config.xml and LOTS more)
  • spring bean configurations (i.e. slingshot-application-context.xml)
  • resource bundles (i.e. slingshot.properties)
This blog will look at the first three to understand how a surf application is configured.

To trace the configuration, we start with the war. Whithin the Slingshot application is a source folder which includes web/WEB-INF that contains the web.xml for the share.war.

looking at web.xml

A context-param section is included to define the contextConfigurationLocation for spring to find all of the spring beans that are needed to configure the surf application. These include:
  • classpath:alfresco/webscript-framework-application-context.xml
  • classpath:alfresco/web-framework-application-context.xml
  • classpath:alfresco/web-framework-model-context.xml
  • classpath:alfresco/slingshot-application-context.xml
This ties slingshot to depend on the Webscript Framework and the Web Framework projects.

A listener is defined with a springframework web context loader listener wihich loads the spring context form the class paths defined in the 'contextConfigurationLocation' context parameter.

Several servlets are defined

url-patternservlet nameservlet class
/logoutlogoutServletorg.alfresco.web.site.servlet.LogoutServlet
/login/*loginServletorg.alfresco.web.site.servlet.LoginServlet
/service/*apiServletorg.alfresco.web.scripts.servlet.WebScriptServlet(container=webframework.webscripts.container)
/proxy/*proxyServletorg.alfresco.web.scripts.servlet.EndPointProxyServlet
/page/*, p/*pageRendererServletorg.alfresco.web.site.servlet.DispatcherServlet
/s/*uriTemplateServletorg.alfresco.web.uri.UriTemplateServlet
/control/*frameworkControlServletorg.alfresco.web.site.servlet.FrameworkControlServlet

Finally, weblome file list includes:
  • index.jsp
  • index.html
looking at the web-framework-config alfresco configuration file

The configuration file defines
  • page-mapper
  • link-builder
  • file-system
  • format default
  • format print
  • format wap
  • persisters
  • model-type (object model) definitions
each model-type defines id, name, class, search path and default store for the following model types:
  • chrome
  • component
  • component-type
  • configuration
  • content-association
  • page
  • page-type
  • page-association
  • template-instance
  • template-type
  • theme
These model-type definitions define where to look for marshaled configurations for model objects of each type, whether to look on the remote store, local path or in the class path.

looking at spring bean application context files

Using spring explorer in Eclipse (Ganymede) allowed me to view spring bean configuration dependencies.

The slingshot-application-context.xml references beans defined in web-framework-application-context.xml and web-framework-model-context.xml.

The web framework beans reference beans defined in webscript-framework-application-context.xml.

slingshot-application-context.xml contains the following beans:

The 'webscript.resource' bean list resource bundles that contain messages to use. The other beans (i.e.'webframework.searchpath.chrome') tell surf where to look (i.e. search paths) for definitions of the different model components that surf needs to run. For instance, to know how 'chrome' is configured.


web-framework-application-context.xml contains the following beans:



Tracing through rendering the dashboard
To demonstrate the configuration, lets look at the beginning, rendering the dashboard. The share.war defines the welcome page 'index.jsp'. This jsp redirects to '/page/site-index'. As defined in web.xml, /page/* url pattern runs the 'pageRendererServlet'. This servlet looks for a model object named 'site-index'. The name 'site-index' is defined as a page model object found in the default remote store (as defined in web-framework-config.xml for the page model-type) in './site-data/pages'.
'site-index.xml' defines a page model object named 'site-index' which defines:
  • title = welcome
  • descrition = Landing page for users - will create user site dashboard
  • template-instances = site-index
  • authentication = user
The site-index page model object references a template instance named 'site-index' found also in the default store ( ./site-data/template-instance):
The template-instance refers to a template-type 'site-index' found in (site-data/template-types)
'site-index.xml' is a template-type model object and defines:
  • title = site index landing page template type
  • description = Site index landing page JSP Template Type
  • renderer = /site-index.jsp
  • renderer-type = jsp
Looking at '/site-index.jsp' as a jsp template renderer, we see scriptlet logic that follows:
If not already created the user dashboard, create the user's dashboard using the definition in presets (found in ./site-data/presets/presets.xml) for 'user-dashboard'.
Render the user dashboard.
In 'user-dashboard' preset, the 'components' section creates components and assigns to regions according to the template insance (dashboard-3-columns) defined in the 'pages' section.

The 'dashboard-3-columns.xml' template-instance definition defines

template-type = org/alfresco/dashboard.ftl.

'dashboard.ftl' imports alfresco-template.ftl and alfresco-layout.ftl and creates regions named 'component--' where the preset components are added.

The preset creates and places the following components into the dashboard template:
define title from (site-webscripts/org/alfresco/components/title/user-dashboard-title*)
define list of page components (dashlets) positioned at {column#}-{row#):
  • user-welcome at 1-1
  • user-calendar at 1-2
  • rssfeed at 2-1
  • alfresco-network at 2-2
  • my-activities at 2-3
  • cmisfeed at 3-4
  • my-profile at 3-1
  • my-sites at 3-2
  • my-tasks at 3-3
Each dashlet is defined by a ui web-script found in site-webscripts/orgb/alfrsco/components/dashlets/. See a description of alfresco's web-scripts technology for details on ui web scripts.

Monday, October 20, 2008

Learning Surf 1 - building from source

My goal was to build the latest Alfresco from source, and begin to develop and extend it using Eclipse. I followed the instructions on: Alfresco's developer wiki . I followed the guidance of the Eclipse section to import all alfresco projects.

Unfortunately, between the first time I did this (early September) and the latest time (October 15), there was significant changes to the codebase that affected the Eclipse projects I imported earlier. I went through the pain of refreshing projects and rebuilding everything until I finally got everything to pass. It was probably better that i deleted these older projects and started with fresh imports. This is a large code base (over 500 meg compressed) so I warn you that this will take some time. Once the code base is built from source, I can follow along refreshing as needed.

Note, since I am developing on a Windows laptop (dont ask why), I used tortoise svn. This worked fairly well for me. I also got the subversion plug in for Eclipse. These two seem to work well with each other well for now.

My first step was to build the alfresco war. I had already created an alfresco database in MySQL as the wiki suggested. I also had tomcat 5.5 installed and Ant so the build was fairly painless.


ant build-tomcat



This created the alfresco.war and deployed it to TOMCAT_HOME. Make sure to set up the environment variables as described. TOMCAT_HOME will be where the alfresco.war will be deployed. APP_TOMCAT_HOME will be where share.war and alfwf.war will be deployed. These can be different or the same. Alfresco 3.0 is now truly service oriented and webapps can be developed (i.e. share.war) and deployed independently of the repository (alfresco.war). For development purposes, I set these to the same tomcat instance.

One problem my team ran into was that share.war refused to authenticate admin/admin. We later determined this was due to the fact that we were running tomcat on a different port than 8080 and there were several configuration files that needed to be changed to reflect this:


$ALF_HOME/tomcat/webapps/alfresco/wsdl/administration-service.wsdl
$ALF_HOME/tomcat/webapps/alfresco/wsdl/dictionary-service.wsdl
$ALF_HOME/tomcat/webapps/alfresco/wsdl/content-service.wsdl
$ALF_HOME/tomcat/webapps/alfresco/wsdl/authoring-service.wsdl
$ALF_HOME/tomcat/webapps/alfresco/wsdl/access-control-service.wsdl
$ALF_HOME/tomcat/webapps/alfresco/wsdl/action-service.wsdl
$ALF_HOME/tomcat/webapps/alfresco/wsdl/authentication-service.wsdl
$ALF_HOME/tomcat/webapps/alfresco/wsdl/repository-service.wsdl
$ALF_HOME/tomcat/webapps/alfresco/wsdl/classification-service.wsdl
$ALF_HOME/tomcat/webapps/share/WEB-INF/urlrewrite.xml
$ALF_HOME/tomcat/webapps/share/WEB-INF/classes/alfresco/pagerenderer-config.xml
$ALF_HOME/tomcat/webapps/share/WEB-INF/classes/alfresco/webscript-framework-config-test.xml
$ALF_HOME/tomcat/webapps/share/WEB-INF/classes/alfresco/webscript-framework-config.xml
$ALF_HOME/tomcat/webapps/share/WEB-INF/classes/alfresco/webscripts/org/alfresco/indexall.get.mediawiki.ftl


After building and deploying alfresco.war and share.war, and testing, I switched to the instructions for building and deploying the surf platform. In these instructions, the alfwf.war is built and deployed (alfwf.war) to APP_TOMCAT_HOME. This is a minimal implementation of the webframework that only brings up a test page:


http://localhost:8080/alfwf/page?p=welcome


which runs some tests and dumps some configurations.

Alfresco Surf applications are created leveraging the following technologies and we should be familiar with all of them:
At this point, the fun begins - creating templates and components to the new surf platform.

Sunday, October 12, 2008

Alfresco Community Conference in DC

Last week, I attended the Alfresco Community Conference in DC where, among other things, Alfresco's Surf platform was discussed. Alfresco is moving in a bold new technical direction: towards a Web 2.0 framework developed from the ground up, completely overtaking its current Enterprise web framework based on JSF. This is bold because most vendors incrementaly develop over thier legacy frameworks. Alfresco has chosen to radically rewrite theirs.

One of Alfresco's key competitive advantages is that it had the clarity of heindsight. Alfresco could look back at the experience of the past: including Documentum, where it's CTO John Newton was co-founder, and former engineers of Vingette, Interwoven and others. This experience has allowed them to develop a better architecture from the ground up, solving the pain of the past, in a culture steeped in risk taking and innovation. Further, its committment to the community and to open principles has allowed it to continue to lead in innovation, rather than falling behind.

Sometimes openness and innovation leads to chaos. Several potential Alfresco users have hedged thier commitment to alfresco to see what Alfresco 3.0 is all about. Its radical rearchitecture has risked alienating customer customization of its previous JSF based frameworks. It is unclear what existing customers will do with this new technology, but Alfresco has committed to continue to support its legacy JSF applications for a time.

For those of us who are just starting to develop on Alfresco, the future is wide open. My company has decided to participate as Beta customers to Alfresco Enterprise 3.0. This leads us to figuring out how we can capitalize on this new platform architecture, and how to avoid the inevitable barbs of its newness. Among other blog tracks, I will update my experiences with Alfresco Surf platform with how-to's and findings. I hope this proves useful to some.

From a Technical perspective, Alfresco has rearchitected its front end web client and services relying on Web 2.0 components from Yahoo User Interface framework. The web client talks to services via RESTful JSON enabled services typically created using Alfresco's webscript framework. Surf introduces a new component model with page and component dispatching frameworks. Web client things like Templates, Pages and Components are defined declaratively leveraging lightweight XML, like the webscript framework for declaring RESTful services on alfresco. Like services, client component behavior is developed in JavaScript and benefit from a client side alfresco object model to interact with predefined components and/or alfresco remote services.

As I learn this new framework, I will blog on each step I take. I hope this will be helpful to others also learning Surf.

Monday, September 22, 2008

Enterprise 2.0

I recently attended a workshop sponsored by the Chicago chapter of AIIM about Enterprise 2.0. The audience was a typical spread of IT professionals interested in document management, Image scanning and management, records management and of course consultants looking to make contacts. My team, we numbered 4, were first time attendees and were interested in getting ideas relating to our web content management applications.

Two presentations were given: Jim Vaselopulos (VP and Partner of PSC Group) focused on the human perspective in collaboration, a key aspect of Enterprise 2.0, and Andrew MacMillan (VP of Product Mgmt at Oracle ECM (formally Stellent)) was focused on its technology aspects. Both presentations were compelling: Jim's presentation brought the point home that Enterprise 2.0 technologies were focused on enabling collaboration among content stakeholders, Andrew's presentation talked about the evolving nature of technology through web 1.0 to web 1.5 and web 2.0 like technologies. After the presentations, they were joined by others in a panel discussion.

The first question on the tips of participants was 'what was Enterprise 2.0?' This was answered variously as Web 2.0 technologies applied to the enterprise, and enablement of more ad-hoc collaboration to manage content (the 'adhocracy' as Jim put it). Web 2.0 brings to the table technologies like blogs, wiki's, messaging and tagging ('tagonomy' and 'folksonomy' were entered into the lexicon). Other capabilities of Web 2.0 like 'mashups' were also mentioned. These Web 2.0 technologies are user directed evolve in an ad-hoc manner. This differs from older technology enablers that are more directed, such as workflows and taxonomies that are defined by business analysts and direct collaboration.

My take... So what is E20? E20 is Web 2.0 for the Enterprise. This soundbite will help explain it to curious management. To technologists, it is using Web 2.0 technologies like wiki's, blogs and mashups evolving in the Internet to web based business applications behind the firewall. For ECM, E20 is the addition of pure web based services and collaboration tools to enable user directed, ad-hoc collaboration and use.

The next obvious question of the group was 'what does E20 mean to me?' The group varied in their sophistication and technical backgrounds, but a rift became obvious. This rift was pointed out very succinctly by Jim Vaselopulos in his talk. As he defined it, there are 'digital immigrants' and 'digital natives'. Jim looked at the generation gap between those who were born with digital technologies and those who were introduced to it much later. Their sensibilities were different. The 'digital native's' acceptance (and demand) for truly self controlled digital collaboration far exceeded the 'digital immigrant's' desire and tolerance for the same. From a generation's perspective, baby boomer types look at technology as an addition to traditional communication and formality, necessarily organized in hierarchies. From the perspective of the 'millenniums' (born 1980 - present), technology enables 'flat' and immediate collaboration and informality.

So, from the above distinctions, users may adapt to E20 differently. A pure 'adhocracy' where users have complete freedom to structure information and collaboration may be perceived as a threat to formality of policy and hierarchical organizations. A wiki for instance is controlled by no one. Everyone is a contributor and a consumer. There is no formal structure, there is no formal control. It is subject to the whims of the users. To digital natives (typified by the 20 somethings) this isn't a problem, this is essential. To digital immigrants (the gray hairs), this is an out of control mess. How can you manage important business information this way and expect the information and its structure to just 'evolve?' Playing off of the generation gap isn't fair or accurate, but it does represent the problem in some way.

The room tended towards the 'gray hairs.' Most in the room came from IT where information management policies were more formal, and information organization existed in well structured taxonomies. For instance, introducing a wiki to manage important business information was looked at suspiciously. 'How do we control it? How do we organize it?' they would ask. Our contingent tended toward the other perspective. To a digital native, this is the very point of a wiki: 'no one controls or manages it and everyone controls and manages it.' To the digital immigrant, this can only lead to a mess. To the digital native, control and formal organization limit usefulness and frustrate collaboration. To the digital immigrant, this ad-hoc nature would limit usefulness and frustrate collaboration. Clearly, different users have different perspectives.

In my company, we are wiki happy. Everything goes on the wiki. There are no controls. There is no formal organization. Its organization evolves. The integrity of the information is the responsibility of all. Occasionally, some refactoring has taken place. There is an element of security. Groups of wiki pages are organized into departments. Users within a department are free to contribute. Others may be restricted to viewing. Personal pages remain in control of the author but anyone can read. Inappropriate content would cause some to complain to our tools team. Although this has rarely happened. Organizational structure as evolved. It is still hard to find what you are looking for. Index pages are created occasionally to organize information into loose taxonomies but not as a formal exercise--someone gets tired of hunting for things and creates a page that organizes it.

For our company, this has worked for most people. But we have also recognized that others are not that comfortable. They have tasked to our internal tools team to reconstitute a more formal intranet. This intranet will be controlled and organized formally. Information will be authored by our tech writers at the direction of the management hierarchy. The very thought of this had caused a furor amongst some in our workforce. Some were vehemently apposed to this 'throwback' to the old. Some who were baffled and dismayed by the by the wild west of the wiki were relieved and fought vigorously for its return. It was an interesting example of the digital generation divide pointed out so deftly by Jim Vaselopulos. We should use this as an example that there are more viewpoints to serve. Perhaps the lessons learned are that Web 2.0 is not for everyone. Perhaps E20 adds elements of control and formality that businesses need. Perhaps we should take away from this effort that sometimes the wild west needs a sheriff.

Tuesday, September 16, 2008

Enchancing ECM-SOA with Enterprise Content Bus

Other capabilities we are looking for in our SOA approach is to leverage an Event Driven Architecture (EDA) focusing on Document Oriented Messages (actually Content Oriented Messages) and the ability to choreograph many content oriented services. This is the job of a message broker, but more specifically an Enterprise Service Bus (ESB).

We wish to build configurations oriented to choreographing processes for:
  • Acquisition - acquire content from many sources and many modes (push, pull, scheduled)
  • Processing - transform, aggregate, index, repurpose, replicate content
  • Delivery - syndicate or publish in multiple formats over multiple channels
  • Integration - integrate with other information sources to enrich content
We believe to fully allow us to integrate with the rest of the enterprise, we will need the flexibility that an ESB brings to a SOA. It may be helpful to begin by trying to explain what an ESB is (at least from my perspective). To truly leverage a multitude of services distributed throughout the enterprise, we need to choreograph multiple services together in a process flow. These services may need to be accessed differently using REST, SOAP, JMS or FTP. An ESB centralizes integration and choreography of services and provides many capabilities to talk over multiple channels, route and process messages in many different ways, all mostly through configuration rather than code. Also an ESB can be considered as a container for services oriented toward integration such as routing and transformation. An ESB brings shared capabilities like security, transformations, routing, security, transactionality and high-availability so these cross cutting concerns can be applied uniformly and and not have to be custom developed. An ESB is a specialist in integration and choreography. It allows integration logic to move out of each client, service or service container and reside in a Service Oriented Integration container.

An example may help. A process flow in this case can be a pathway for content to travel from acquisition, to management, processing, to delivery. Each one of these stages in the content's life-cycle might require integrating with many distributed services. Suppose I am automating the process of acquiring word documents to index and post on an intranet.
  • For acquisition, i may want to poll an external ftp drive to see if new content has arrived
  • For processing, I may want to enrich this content with computed metadata and store it in the content repository for retension
  • I also I may want to transform this content into a different representation (i.e. word to PDF)
  • For delivery I may want to deliver this pdf to a website
  • Further, I want to update an index page to this document
One approach is to build this as a customization of a content manager. Most content managers offer a way to ftp content to their content repository. But to 'poll' an ftp site would require custom coding. Once the content is in the repository, a workflow or rules can be triggered that integrates automated actions and human tasks. The first step would be to create a PDF copy. Once a PDF copy is created, a job can be scheduled to index the PDFs as a web page index, and then publish the index and latest PDF files to the site.

Another approach would be to leverage an ESB. Leveraging an ESB may improve agility of developing these processes. For instance servicemix has pre-developed and configurable ways to 'poll' and 'send' messages via an ftp channel. One a file is found on the ftp via polling, Further using servicemix a route can be configured by using various Enterprise Integration Pattern implementations. In this case, a 'pipeline' can be configured to choreograph services:
  1. Store acquired word doc via content service
  2. Transform word to pdf via transformation service
  3. Store pdf via content service
  4. Generate index invoking a template service
Finally, the pdf and the index page can be stored to the site's file system via the file sender (another configuration).

Further benefits exist leveraging an ESB. Each step in this process is made via a SEDA architecture using durable queues. That means that if any of the choreographed services break, take longer than expected or otherwise behave unexpectedly, the process continues unabated. The ESB route can be configured to handle error cases such as a service returning an error result. The processing routes can be transactional, and roll back all changes if one step changes. The ESB can be clustered to support High Availability, and can be configured to route based on the type of processing. Further, the services choreographed can be accessed over numerous channels: REST (http), ftp, file, jms, jabber,SOAP (http). And modifying the process in many cases requires reconfiguring, not recoding. And the process configurations are not distributed in many services or clients, but centralized on the ESB.

While there are many ways to choreograph services, an ESB approach may improve agility by leveraging a set of configurable components specialized in integration, orchestration and choreography and that can speak many languages to different distributed systems. The purpose of the ESB is not to take over services from the content manager or other systems but to leverage them. Moving choreography of services to a specialist like an ESB removes the need to create a lot of custom scripting in a content manager which may not be as good at these tasks.

But existing ESB implementations are focused on choreographing messages, not content. Existing ESBs don't have configurable components around processing of content, and may not do well passing around large content. An ESB needs to be customized to manage content and provide configurable components to enhance content processing. Thus, we are constructing an ECB (Enterprise Content Bus) that builds these content centric capabilities on top of an Enterprise Service Bus.

Although not mentioned in the example, choreography using enterprise integration patterns provides a lot of flexibility in combining many services, but the addition of Business Process Management allows these services to be orchestrated according to configurable business processes, and is a great addition to the ESB. Simple processes from acquisition to management to deployment can be implemented via piplelines and wiretaps and content switches. But processing content often requires a business process that integrates invocation of services, integration of systems and human tasks, and provides the visibility into the content processing pipeline. (See following posts on our approach to integrating BPM to our ECB.)


The details to follow in the next post...