Getting started with setting up an Alfresco cluster
In this article I will try and get you started on clustering the Alfresco application. I will setup a basic Alfresco cluster with 2 nodes and for this I will use 2 Windows laptops and an Ubuntu workstation that I have at home. One of the laptops is a Windows 7 box and the other one is a Windows Vista box. The Ubuntu box will run the shared MySQL database for content metadata and provide the shared file system for content files.
This type of cluster will take you a bit on the way to high availability and a more scalable Alfresco solution. I have not included a hardware load balancer in this example but it usually sits in front of the Alfresco server nodes and is then configured to use sticky sessions. Each incoming request will be handled by either node 1 or node 2, so a lot of the processing can take place in parallel. And if one node goes down the system will still work with the other node. However, at the end of this blog I will also show you how the Apache HTTP Server can be used as a software load balancer in front of the Alfresco servers.
The database and file system is still a single point of failure so this is not a disaster and recovery (DR) solution in any way. Also, all servers are in one geographical location making it vulnerable to fire, flooding, power cuts etc.
The following picture illustrates:
The index files need to be located on the local disk as the index store is transaction-aware but the file system is non-transactional. Avoid also putting the index files on a SAN or any other remote disk for performance reasons.
Verifying the Alfresco file system share and the Alfresco database from the Windows laptops
After setting up Samba and MySQL on the Ubuntu box (not covered in this blog) it’s good to verify that they can be accessed from the Windows laptops that will be running Alfresco. I have created a new share in Samba called “alfresco” where I will keep the shared alf_data directory.
Verify that you can see it from the Windows boxes as follows:
C:\Users\mbergljung>net view \\192.168.0.7
Shared resources at \\192.168.0.7
deathstar server (Samba, Ubuntu)
Share name Type Used as Comment
alfresco Disk Alfresco files
PDF Print PDF
The command completed successfully.
Here my Ubuntu box has IP 192.168.0.7. I do not use a DNS so IP addresses are the way to go then. The alfresco share has been setup like this in the Samba configuration file:
[alfresco] comment = Alfresco files read only = no guest ok = no path = /var/alfresco
Now make sure that the SMB ports are accessible and there is no firewall blocking them on the Ubuntu box:
C:\Users\mbergljung>telnet 192.168.0.7 139
This should give you a blank screen if you have direct access to this port. Try also 445, which is a TCP port too. If you cannot access these ports then check the firewall on the Ubuntu box. There are also two UDP ports 137 and 138 that need to be accessible.
We can now test the Alfresco share with the username and password we intend to use and map a local drive on the Windows boxes to the content store as follows:
C:\Users\mbergljung>net use R: \\192.168.0.7\alfresco /USER:martin
The command completed successfully.
This will map the Samba share called “alfresco” as drive R: on the Windows laptop. Make sure you got permission to write to this share by creating a folder and then deleting it from the Windows laptop.
Now, make sure to disconnect this drive, it should not be mounted at the same time as the Alfresco server is running as this can cause problems.
net use R: /delete
Make sure to also add a Windows user with the same username and password that was used to connect to the Samba Alfresco share:
C:\Users\mbergljung>net user martin /ADD
The command completed successfully.
C:\Users\mbergljung>net localgroup Administrators martin /add
The command completed successfully.
This user will be used to start the Alfresco Tomcat service. If we do not start the Alfresco server with a user that can access the Samba share we will get errors such as:
Caused by: org.alfresco.service.cmr.repository.ContentIOException: 09250000 Failed to create store root: \\192.168.0.7\alfresco\alf_data\contentstore at org.alfresco.repo.content.filestore.FileContentStore.(FileContentStore.java:99)
The database is setup with the following SQL code:
create database alfresco_cluster;
grant all on alfresco_cluster.* to 'alfresco'@'localhost' identified by 'alfresco' with grant option;
grant all on alfresco_cluster.* to 'alfresco'@'localhost.localdomain' identified by 'alfresco' with grant option;
grant all on alfresco_cluster.* to 'alfresco'@'%' identified by 'alfresco' with grant option;
Note the last line; it enables the alfresco user to connect from any host, not just localhost. If you skip this line it will not be possible to connect to MySQL from the Windows boxes and the following error will be displayed:
ERROR 1045 (28000): Access denied for user 'alfresco'@'192.168.0.2' (using password: YES)
Test that you can connect to the database from the Windows laptops:
C:\Users\mbergljung>mysql -h 192.168.0.7 -u alfresco -p
Enter password: ********
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 13989
Server version: 5.0.51a-3ubuntu5.5 (Ubuntu)
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL v2 license
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement
We are now ready to install Alfresco on the two laptops.
Setting up Alfresco on the Windows Laptops
This section will first take you through how to setup laptop 1, which will be the one that creates the necessary folders and files under the alfresco share/alf_data folder. It will also create the Alfresco database. The second part of this section will take you through how to setup laptop 2 when the database and alf_data already exists.
Setting up Alfresco on Laptop 1
When setting up Alfresco 3.4.4 on laptop 1 choose the Advanced installation option. Then uncheck PostgreSQL, SharePoint, Records Management, Web Quick Start, Web Project Management, and Quickr Connector Support:
Then select a folder for the installation. In the next dialog the MySQL connection parameters should be specified as follows:
The complete JDBC URL is as follows:
The rest of the steps in the installation process is pretty self explanatory and I am skipping them here.
When you are done with the installation the MySQL JDBC driver (e.g. mysql-connector-java-5.1.13-bin.jar) needs to be copied into the alfresco\tomcat\lib directory of the installation.
Now we need to open up the alfresco\tomcat\shared\classes\alfresco-global.properties file and configure the location of the content store, start by changing the following line:
So it points to the Samba alfresco share on the Ubuntu box (the alf_data directory does not need to exist, it will be created when Alfresco starts up for the first time from one of the Windows boxes).
Make sure to specify the dir.root value as a UNC path. Do not map a drive locally and then use for example R:\alf_data as dir.root. Also, make sure that you do not have a local mapping such as R: pointing to the alfresco share, not even when you are using a UNC path.
Then add the following properties that make the Lucene indexes to be stored locally on the box where Alfresco is running:
dir.indexes=C:/Alfresco3.4.4/alf_data/lucene-indexes dir.indexes.backup=C:/Alfresco3.4.4/alf_data/backup-lucene-indexes dir.indexes.lock=C:/Alfresco3.4.4/alf_data/locks
There is one more property we need to set as we are not using the PostgreSQL database and that is the Hibernate dialect. Set it as follows:
And finally make sure the windows Alfresco Tomcat service is running as the correct user:
This is pretty much it as far as non-cluster configuration goes. We can now start Alfresco on laptop 1 and the alf_data directory and the alfresco_cluster database should be automatically created for us on the Ubuntu box.
Start Alfresco on laptop 1:
C:\Users\mbergljung>net start alfrescoTomcatnum20
The alfrescoTomcatnum20 service is starting.
The alfrescoTomcatnum20 service was started successfully.
Make sure Alfresco Share can be accessed properly by hitting http://localhost:8080/share and login admin/admin.
When the system starts always watch the log files in the alfresco/tomcat/logs directory with a tool like for example BareTail.
Note. If there are any problems during the start-up - like the system goes into read-only mode because of licensing problems, or the share cannot be accessed etc. Then always delete the local indexes, drop the database and create it again, and delete the alf_data directory on the alfresco share. This way you always start from scratch when you make changes and you don’t build on errors generated by previous configuration mistakes.
Setting up Alfresco on Laptop 2
Setting up the second laptop with Alfresco 3.4.4 is done in exactly the same way as we setup laptop 1, with one important difference. Because the index is not going to be up to date with the database and content store (that laptop 1 created) we need to tell Alfresco on laptop 2 to create the index from scratch the first time we start the system.
So follow the installation procedure for laptop 1 but before you start Alfresco add the following property to the alfresco\tomcat\shared\classes\alfresco-global.properties file:
This will initiate a complete re-build of the index at startup to get it in sync with database and content store.
Now start Alfresco on laptop 2. After it started successfully do not add any content or create folders etc. We have not yet turned on clustering so there is no index tracking from laptop1. Any changes you make via laptop 2 will not yet be reflected in laptop1’s Alfresco index.
Configuring the Alfresco servers on the laptops to participate in a Cluster
Now it’s time to setup the cluster configuration and have both the Alfresco servers to participate in a cluster. Alfresco requires the participating servers to be able to discover each other on the network in order to set up cluster communications. We will enable this by configuring the JGroups tool (http://www.jgroups.org).
Before Alfresco version 3.1, this discovery process was done using a UDP multicast message provided by EHCache (http://ehcache.org). Servers in the cluster picked up the message and used the information to set up inter-server communication for inter-cache communication.
This solution was limiting some installations by not providing a more flexible cluster discovery process, which is why JGroups was integrated into the repository. JGroups is a toolkit for multicast communication between servers. It allows inter-server communication using a highly configurable transport stack, which includes UDP and TCP protocols. Additionally, JGroups manages the underlying communication channels and cluster entry and exit.
Note. Cache communication using JGroups is an Enterprise specific feature not available on Alfresco Community.
Checking accessibility between the laptops running Alfresco
The built in EHCache system in Alfresco will be communicating over the multicast protocol as per the following configuration in the alfresco/tomcat/shared/classes/alfresco/extension/ehcache-custom.xml.sample.cluster file:
<cacheManagerPeerProviderFactory class="org.alfresco.repo.cache.AlfrescoCacheManagerPeerProviderFactory" properties="heartbeatInterval=5000, peerDiscovery=automatic, multicastGroupAddress=220.127.116.11, multicastGroupPort=4446"/>
So both laptops have to let through traffic on multicast address 18.104.22.168 and port 4446. There will be multicast heartbeats sent every 5 seconds.
The JGroups toolkit will be communicating over TCP and port 7800 in the configuration we are going to setup in a bit. So make sure that port is accessible between the machines.
Ping the other machine from each machine and see that this works. If that does not work then you need to open up the Windows firewall so both machines can access each other.
Making sure the clocks are synchronized on the laptops running Alfresco
The server times need to be within a specified time difference. This will be configured in Alfresco, but it is recommended that the value is less than 10sec. We need ours to be less than 5 seconds. The easiest way to do this is via an NTP server. There are directions on the web on how to configure this, so we won’t go into the details here. My Windows laptops are both synchronizing their clocks with an Internet time server so they are synced to within a couple of seconds.
Configuring Alfresco to participate in a cluster
There is a couple of things that we need to configure to get the cluster up and running:
- The first thing is the properties for the JGroups protocol so it knows how to talk to the other Alfresco instance
- The second thing is the index tracking configuration so the local index is always up-to-date
- And finally, the distributed Level 2 EHCache
Configure the distributed Level 2 cache
The Level 2 (L2) cache provides out-of-transaction caching of Java objects inside the Alfresco system. Alfresco provides support for EHCache. Using EHCache does not restrict the Alfresco system to any particular application server, so it is completely portable.
To enable the Level 2 cache to be replicated lookup the alfresco/tomcat/shared/classes/alfresco/extension/ehcache-custom.xml.sample.cluster file and rename it to ehcache-custom.xml.
This will turn on the multicast heartbeat messages between the nodes in the cluster.
Configure JGroups and Index tracking
To configure JGroups and index tracking update the alfresco\tomcat\shared\classes\alfresco-global.properties file as follows for both Alfresco installations:
JGroups specific configuration:
alfresco.cluster.name=home-test-cluster alfresco.tcp.initial_hosts=192.168.0.2,192.168.0.5 alfresco.jgroups.defaultProtocol=TCP
Index tracking specific configuration:
index.recovery.mode=AUTO index.tracking.cronExpression=0/5 * * * * ? index.tracking.reindexLagMs=5000
Note. Make sure to remove the
property configuration in the node that had it.
The properties have the following meaning:
|Once the cluster name is specified, JGroups uses that name to uniquely distinguish inter-server communication. This allows machines to use the same protocol stacks, but to ignore broadcasts from different clusters.||The JGroups cluster, including index tracking, will only initialize if this property is defined!|
|A list of hosts and start ports that must be pinged. This can be all potential members of the cluster, including the current server and servers that might not be available. The port listed in square brackets is the port to start pinging.||If the port is changed from 7800, which is the default TCP port to use, then the
property has to be set too.
|The protocol stack to use||Default is UDP|
|Set to AUTO to ensure indexes are refreshed properly on start-up||Default is VALIDATE so make sure to set this property to AUTO|
|Cron expression that defines when the index tracking should be run||Note. If you want 5 seconds interval then this does not need to be set as it is “0/5 * * * * ?” by default. Note. The index tracking code will not activate unless the cluster name has been set!|
|Set the allowed time difference between the two machines. Ours is 5 seconds|
Configure debug Logging
It is a good idea to turn on debug logging the first time you start the system up with the new cluster configuration. Open up tomcat/webapps/alfresco/WEB-INF/classes/log4j.properties and add the following line:
Testing the Alfresco Cluster
It’s now time to test the cluster configuration that we have setup. If you have not done so restart both of the Alfresco servers.
First thing you can check after both servers have started is that you can login to http://localhost:8080/alfresco. If that works then we can move on to check that the cluster works.
Checking that the cluster is working
In the Guest Home space do the following to test cache clustering (M1 is cluster node 1 and M2 cluster node 2):
- M1: Login as admin
- M1: Browse to the Guest Home space, upload a PDF document and view the document properties
- M2: Login as admin
- M2: Browse to the Guest Home space, locate the tutorial PDF document and view the document properties
- M1: Modify the PDF's author and title field, setting them to 'abcdef'
- M2: Refresh the document properties view of the PDF document
- M2: The title and author fields must have changed to include 'abcdef'
To test index clustering search for a title 'abcdef' on both M1 and M2, using the Advanced Search dialog.
Load balancing between the nodes in the cluster
We now have a working Alfresco cluster but no one would really run a production solution with this setup, we need to add one more component to our minimal cluster for it to be more useful in a production scenario.
Most of the times we would put an Apache HTTP Server in front of the Alfresco Tomcat Server when setting up a production solution. This is for better scalability and security.
Some of the benefits from this are:
- Tomcat does not have to serve static files. Apache HTTP server does that a lot better and faster
- Port 80 can be used by default
- Using a rewrite rule we can add /alfresco automatically, no need to specify http://:8080/share, just http:// is enough
- A lot more secure to offer your end users a web server interface and keep the application server interface including the management console separate. Only open port 80 (and 443) in the firewall
- It will be easy to implement SSL on Apache HTTP Server
Another advantage is that you can use the web server to load balance incoming requests between the Alfresco application servers as in the following picture:
Setting up Load balancing with Apache HTTP Server
The web layer will consist of an Apache HTTP server that connects to Alfresco on Apache Tomcat via the mod_proxy_ajp (binary protocol) module and the integrated software load balancer (mod_proxy_balancer) module. This solution, in conjunction with Apache Tomcat, supports full system fail-over including failing over session data to an active Apache Tomcat instance.
In my demo setup I will add Apache Tomcat to the Windows 7 laptop and let it know about Alfresco on the Windows Vista laptop (normally you should of course have Apache HTTP Server running on its own machine in the DMZ):
An incoming HTTP request will be routed via Apache HTTP Server to either Alfresco on node 1 or node 2. If it is a request for static content like JS, CSS, images etc then it will be served directly from Apache HTTP Server.
Here are steps to set this up:
- Install Apache HTTP Server, can be downloaded from http://httpd.apache.org/download.cgi
- Enable load balancing via the AJP protocol, open up the httpd.conf file located in the C:\Program Files (x86)\Apache Software Foundation\Apache2.2\conf directory and enable the following modules:
LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_ajp_module modules/mod_proxy_ajp.so LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
- Next configure the members of the cluster to load balance between, and use sticky sessions so all requests in a HTTP session goes to the same node in the cluster (The name of the session cookie used by Apache Tomcat is JSESSIONID (upper case)). Add the following at the end of the httpd.conf file:
BalancerMember ajp://192.168.0.2:8009 min=10 max=100 route=node1 loadfactor=1 BalancerMember ajp://192.168.0.10:8009 min=20 max=200 route=node2 loadfactor=2 ProxySet stickysession=JSESSIONID
- Then map the remote Alfresco servers into the space of the local Apache server:
ProxyPass /alfresco balancer://home-test-cluster/alfresco ProxyPass /share balancer://home-test-cluster/share
We also need to make configuration changes for each Alfresco Tomcat installation on cluster node 1 and node 2. Apache Tomcat adds the name of the Tomcat instance to the end of its session id cookie (i.e. JSESSIONID), separated with a dot (.) from the session id. Thus if the Apache web server finds a dot in the value of the stickyness cookie, it only uses the part behind the dot to search for the route.
In order to let Tomcat know about its instance name, we need to set the attribute jvmRoute inside the Tomcat configuration file conf/server.xml to the value of the route of the BalancerMember that connects to the respective Tomcat.
To set this up do the following:
- Open up the server.xml file located in the C:\Alfresco\tomcat\conf directory on cluster node 1 and configure/add the jvmRoute attribute:
<Engine name="Catalina" defaultHost="localhost" jvmRoute="node1">
By default the AJP connector will want to redirect to HTTPS connector on 8443 but as we have not set that up yet change it to 8080 as follows:
<Connector port.="8009" protocol="AJP/1.3" redirectPort="8080" />
- Then open up the server.xml file located in the C:\Alfresco\tomcat\conf directory on cluster node 2 and configure/add the jvmRoute attribute:
<Engine name="Catalina" defaultHost="localhost" jvmRoute="node2">
<Connector port="8009" protocol="AJP/1.3" redirectPort="8080" />
If you now access Alfresco with the http://localhost/alfresco URL you should be redirected to one of the Alfresco Tomcat servers.
If you want information about how to setup Tomcat to accept HTTPS connection, have a look in this blog that talk about how to do just that.