Installing GlassFish V2 on FreeBSD 8

---

3.1. Prerequired Ports

You need to following ports to build GlassFish

1. java/jdk15
2. devel/maven
3. devel/apache-ant
4. security/nss
5. devel/nspr

Note:
You must use jdk 1.5 to build GlassFish V2 but it run fine with jdk 1.6.
The java/diablo-jdk15 can not be used because the missing sunpkcs11.jar

---

3.2. Fetch Source Code

The source code branch that I use to build my GlassFish V2 is SGES211_FCS_BRANCH. If you want to try another branch, you can see list of all GlassFish V2 branches at GlassFish bootstrap

% cvs -d:pserver:your_java_net_account@cvs.dev.java.net:/cvs login 
#May got a warning, just ignore it
% export JAVA_HOME=/usr/local/jdk1.5.0 #Assume bash
% mkdir workspace
% cd workspace
#For slow network add -z9
% cvs -z9 -d:pserver:your_java_net_account@cvs.dev.java.net:/cvs checkout -r SGES211_FCS_BRANCH glassfish/bootstrap
% vi glassfish/bootstrap/project.properties
   glassfish.os.name=Darwin 
   glassfish.cvs.username=your_java_net_account
--Change all of 
   download.sun.com  to  dlc.sun.com.edgesuite.net because the old download site is now defunc.


#For slow network add the following ant:cvs atributes
% vi glassfish/bootstrap/maven.xml
--Search for <goal name="checkout">
   <ant:cvs 
	compression="true"
        compressionlevel="9"

% cd glassfish/bootstrap
% maven checkout

---

3.3. Patch to run in FreeBSD jail

GlassFish is a very noisy program, it open a lot of TCP/IP ports on all IPs of your machine. It is impossible to fix GlassFish to run on a paricular IP. You can set http-listener or admin-listener to a particular IP but that all you can do.

For FreeBSD , we can use jail to limit the glassfish instance to bind to only the IP address of the jail. Unfortunely, GlassFish can not run smoothly in the Jail. GlassFish assume that all administration command must come from localhost (127.0.0.1) but in the jail, when you try to connect to any TCP/IP port in within the jail, your local IP is the jail IP not 127.0.0.1.

I need to patch GlassFish source code to allow admistration command from any host not limit to 127.0.0.1. This may post a security risk compare to the default GlassFish setting but we can use IPFW to limit access to only http-listener for our GlassFish in the jail.

% vi appserv-core/src/java/com/sun/enterprise/admin/server/core/channel/AdminChannel.java
--Change
static String getAccessLevel() {
        return LOCAL_ONLY_ACCESS;
    }

--To
 static String getAccessLevel() {
        return ALLOW_ALL_ACCESS;
    }

You may also need a rc.d script to start GlassFish when the jail start. This is the simple one that I use

#!/bin/sh
#
# $FreeBSD: src/etc/rc.d/jail,v 1.43.2.1 2009/08/03 08:13:06 kensmith Exp $
#

# PROVIDE: glassfish
# REQUIRE: LOGIN cleanvar sshd
# KEYWORD: shutdown

# WARNING: This script deals with untrusted data (the data and
# processes inside the jails) and care must be taken when changing the
# code related to this!  If you have any doubt whether a change is
# correct and have security impact, please get the patch reviewed by
# the FreeBSD Security Team prior to commit.

. /etc/rc.subr

name="glassfish"

rcvar=`set_rcvar`
start_cmd="glassfish_start"
stop_cmd="glassfish_stop"
glassfish_user=glassfish
glassfish_admin=/home/${glassfish_user}/glassfish/bin/asadmin
glassfish_passwordfile=/home/${glassfish_user}/glassfish/config/password.txt

glassfish_start()
{
        echo 'Starting glassfishi domains:'
        for _glassfish_domain in ${glassfish_domains}
        do
                /usr/bin/su -l ${glassfish_user} -c "${glassfish_admin} start-domain ${_glassfish_domain}" &
        done
        echo '.'

        echo 'Start glassfish nodeagents:'
        for _glassfish_nodeagent in ${glassfish_nodeagents}
        do
                /usr/bin/su -l ${glassfish_user} -c "${glassfish_admin} start-node-agent --user admin --passwordfile ${glassf
ish_passwordfile} ${_glassfish_nodeagent}" &
        done
        echo '.'
}

glassfish_stop()
{
	echo 'Stopping glassfish:'

	echo 'Stop glassfish nodeagents:'
	for _glassfish_nodeagent in ${glassfish_nodeagents}
	do
		/usr/bin/su -l ${glassfish_user} -c "${glassfish_admin} stop-node-agent ${_glassfish_nodeagent}"
	done
	echo '.'

	echo 'Stop glassfish domains:'
	for _glassfish_domain in ${glassfish_domains}
	do
		/usr/bin/su -l ${glassfish_user} -c "${glassfish_admin} stop-domain ${_glassfish_domain}"
	done
	echo '.'
}

glassfish_status()
{
	/usr/bin/su -l ${glassfish_user} -c "${glassfish_admin} list-domains"
	/usr/bin/su -l ${glassfish_user} -c "${glassfish_admin} list-node-agents"
}

load_rc_config $name
cmd="$1"

case "${cmd}" in
status | onestatus)
        glassfish_status
        ;;
*)
        run_rc_command "${cmd}"
        ;;
esac

---

3.4. Start the build

With correct prerequired ports and GlassFish code branch, building GlassFish is very easy

#You must use bootstrap-all because there is no-prebuild binary for bootstrap on download site 
% maven bootstrap-all
#Must do build and build-jarinstaller together , seperate them into 2 steps don't work -- at least for me
% maven build build-jarinstaller 

Now you will got the file glassfish-installer.jar in the publish folder of your workspace

---

4. Create a Jail

Create a normal FreeBSD jail. No special requirement is need. Assume that you know how to build FreeBSD and already rebuild your FreeBSD from source code.

% su
% cd /usr/src
#If your jail is in ZFS, add NO_FSCHG=yes
% make installworld DESTDIR=/home/vhost/dysnomia NO_FSCHG=yes
% mergemaster -iFU -D /home/vhost/dysnomia
% cp {path_to_glassfish_rc.d} /usr/local/etc/rc.d
% mkdir -p /home/vhost/dysnomia/root/pkgs
% cd /home/vhost/dysnomia/root/pkgs
% pkg_create -R -b jdk-1.5.0.16p9_6,1.tbz
% chroot /home/vhost/dysnomia
% cd /root/pkgs
% pkg_add  jdk-1.5.0.16p9_6,1.tbz
% pw user add glassfish
% mkdir /home/glassfish
% chown glassfish:glassfish /home/glassfish
% passwd glassfish
#Set DNS, use your value not copy mine
% vi /etc/resolv.conf
  search net0.intranet
  nameserver 10.0.0.1   
#Set Host, use your own IP
% vi /etc/hosts
   10.0.0.53 dysnomia dysnomia.net0.intranet
#Enable sshd in jail
% vi /etc/rc.conf
   hostname="dysnomia.net0.intranet"
   sendmail_enable="NO"
   sshd_enable="YES"
   syslogd_flags="-ss"
   
   glassfish_enable="YES"
   glassfish_domains="domain1"
   glassfish_nodeagents="agent1"   
% exit
% vi /etc/rc.conf
   jail_enable="YES"
   jail_set_hostname_allow="NO"
   jail_sysvipc_allow="YES"
   jail_list="dysnomia"  
% /etc/rc.d/jail start dysnomia

---

5. Install GlassFish

The default config in GlassFish jar can not run unmodified in FreeBSD 8,AMD64. You can change the default value in the build step or change it after install GlassFish in the jail.

% Assume that your jail is running.
% scp {path_to_glassfish-installer.jar} glassfish@dysnomia.net0.intranet:~
% ssh glassfish@dysnomia.net0.intranet
% set JAVA_HOME=/usr/local/jdk1.5.0
% export JAVA_HOME
# java -version should return jdk1.5
% java -Xmx256m -jar glassfish-installer.jar
% cd glassfish
% chmod -R +x lib/ant/bin
% lib/ant/bin/ant -f setup-cluster.xml
#For AMD64 minimum stack size is 256k
% vi imq/bin/imqbrokerd 
   --Change  -Xss192k 
   --To      -Xss256k
#Create password file for unattended startup
% vi config/password.txt
   --Add your password
    AS_ADMIN_PASSWORD=adminadmin
% chmod og-r config/password.txt
#Create domain -- not require because the default GlassFish installation will create a domain1 for you
#bin/asadmin create-domain --user admin --adminport 4848  --passwordfile /home/glassfish/glassfish/config/password.txt  --savemasterpassword=true domainXXXX
#Start domain
% bin/asadmin start-domain domain1

Now you can go to dysnomia.net0.intranet:4848 to create Cluster/Instances. If you want to use GlassFish with the application that use a lot of memory (i.e hibernate library), you may need to increase Max stack size to 512m (-Xmx512m) and MaxPermSize from the default value to 256m or larger (-XX:MaxPermSize=192m) in Configurations/server-config/JVM Settings/JVM Options

---

6. Setup Clustser, Load Balance and Rolling Update

6.1. Cluster Topology

I use the following topology for test cluster

                  [ front-end demeter.net0.intranet port 80 ] 
                    |
                    +--DAS (dysnomia.net0.intranet)            
                    |
                    +--agent-dysnomia (dysnomia.net0.intranet)
                    |    |     
                    |    +--- dysnomia-1 (instance)
                    |    |
                    |    +--- dysnomia-2 (instance)
                    |
                    +--agent-haumea (haumea.net0.intranet]
                         |
                         +--- haumea-1 (instance)
                         |   
                         +--- haumea-2 (instance)

6.2. Fix incorrect das.properties

There are some problems if you want to set remote node agent in GlassFish V2. I put the sumary of fix and problems below but you can read full detail at this page.

You need to modify domain.xml on your DAS to allow remote node agent to connect to the DAS, otherwise everytime you start node agent with -syncinstance=true (the default), the value of agent.das.host in das.properties will be reset to localhost.

In glassfish folder on DAS server (dysnomia.net0.intranet)

% vi domains/domain1/config/domain.xml
     -- Change <property name="client-hostname" value="localhost"/>
     -- to     <property name="client-hostname" value="dysnomia.net0.intranet"/>

6.3. Create and start node agents

Now it a time to create and start node agents. Assume that both dysnomia and haumea have the user glassfish

#Assume that we still in glassfish folder on dysnomia
% bin/asadamin create-node-agent --savemasterpassword=true --passwordfile /home/glassfish/glassfish/config/password.txt --host dysnomia.net0.intranet --port 4848  --user admin agent-dysnomia
% vi /etc/rc.conf
  -- Add
    glassfish_enable="YES"
    glassfish_domains="domain1"
    glassfish_nodeagents="agent-dysnomia"
% /usr/local/etc/rc.d/glassfish start
% ssh glassfish@haumea.net0.intranet
# Install glassfish on haumea as describe above
% cd glassfish
% bin/asadamin create-node-agent --savemasterpassword=true --passwordfile /home/glassfish/glassfish/config/password.txt --host dysnomia.net0.intranet --port 4848  --user admin agent-haumea
% su
% vi /etc/rc.conf
  -- Add
	 glassfish_enable="YES"
     #No DAS is need on this server
	 glassfish_nodeagents="agent-haumea"
% /usr/local/etc/rc.d/glassfish start

Now you can access DAS using web browser at http://dysnomia.net0.intranet:4848 and create a cluster and instances as indicated in Sun GlassFish Enterprise Server v2.1.1 High Availability Administration Guide

There are 4 server instances on 2 node agents.

6.4. Required Application

You need to following ports to add Load balancing to GlassFish Cluster

1. ports/www/apache22
2. ports/www/mod_jk
3. tomcat-ajp.jar from Apache Tomcat 5.5
4. commons-logging-1.1.1.jar from Apache Commons Logging
5. commons-modeler-2.0.1.jar from Apache Commons Modeler

Install the apache and mod_jk one from port and copy the last 3 jars to ~/glassfish/lib of each server.

6.5. Loadbalance with Apache + mod_jk

You can follow the document at http://blog.arungupta.me. The following is the summary.

First, you must define jvmRoute and enableJK in your cluser. This can be run using command line or Web Interface at DAS. For command line, see above link

Then, config AJP_INSTANCE_NAME and AJP_PORT for each instance.

The name is the instance name and port is HTTP_LISTENER_PORT of the instance.

After restart the server instance, Your instance will ready to accept load balance request from mod_jk. Unfortunely, it will not accept normal request any more. If you try to access the instance using web browser you will got empty response and the following error in the log:

[#|2010-11-10T06:57:37.549+0000|SEVERE|glassfish2.1|org.apache.jk.common.ChannelSocket|_ThreadID=20;_ThreadName=TP-Processor2;_RequestID=ab70fe89-022c-4e1a-af4b-c3e2aa0e2962;|Error, processing connection
java.lang.IndexOutOfBoundsException
        at java.io.BufferedInputStream.read(BufferedInputStream.java:306)
        at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:627)
        at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:584)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:692)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:682)
        at java.lang.Thread.run(Thread.java:595)

After install apache22 + mod_jk and config GlassFish cluser, You also need to config apache and load mod_jk.

% vi /usr/local/etc/apache22/httpd.conf
 --
LoadModule jk_module          libexec/apache22/mod_jk.so
 --
% vi /usr/local/etc/apache22/Includes/mod_jk.conf
 --

        JKWorkersFile /usr/local/etc/apache22/workers.properties
        JkLogFile  /var/log/jk.log
        JkShmFile  /var/log/jk-runtime-status
        JkLogLevel warn

        # Sample JkMounts.  Replace these with the paths you would
        # like to mount from your JSP server.
	JKMount /clusterjsp   demeter
        JkMount /clusterjsp/* demeter

        #JKMount /* demeter
        #JkMount /servlet/* demeter
        #JkMount /examples/* demeter

 --
% vi /usr/local/etc/apache22/worker.properties
 --
worker.list=demeter

worker.dysnomia-1.type=ajp13
worker.dysnomia-1.host=dysnomia.net0.intranet
worker.dysnomia-1.port=38080
worker.dysnomia-1.lbfactor=50

worker.dysnomia-2.type=ajp13
worker.dysnomia-2.host=dysnomia.net0.intranet
worker.dysnomia-2.port=38081
worker.dysnomia-2.lbfactor=50

worker.haumea-1.type=ajp13
worker.haumea-1.host=haumea.net0.intranet
worker.haumea-1.port=38080
worker.haumea-1.lbfactor=50

worker.haumea-2.type=ajp13
worker.haumea-2.host=haumea.net0.intranet
worker.haumea-2.port=38081
worker.haumea-2.lbfactor=50

worker.demeter.type=lb
worker.demeter.balance_workers=dysnomia-1,dysnomia-2,haumea-1,haumea-2
 --

Now you can use the url http://demeter.net0.intranet/clusterjsp to access the clusterjsp Enterprise Applications in the cluster.

6.6. Rolling Update

One of the problem on GlassFish application is slow start up time due to pre-fetch of commonly use data from the database server. The cluster can reduce this downtime by allow another instances in the cluster to service the request while any particular instance is starting up.

Anyways, the default behavior of GlassFish cluster when redeploy the application to the cluster is to automatically update each instances and restart them simulatinously. To prevent this default action you need to uncheck the Dynamic Reconfiguration: option in Configurations/cluster1-config/System Properties.

After uncheck this option, whenever you redeploy the application, you need to restart each instance manualy.

---