Access Keys:
Skip to content (Access Key - 0)

CBF - Community Build Framework

Community Build Framework

Author and Maintainer: Pedro Alves
Contributors: Daniel Einspanjer,Pedro Reininho
Latest Version: 3.7.0 - Pentaho 3.7 compatible
Latest Update: December 13th, 2010

New Version available
Released version 3.7.0 compatible with pentaho 3.7

Table of contents

Concepts

Overview

Community Build Framework (CBF) is an ant build.xml file script and alternate way to setup and deploy Pentaho based applications. Focused on a multi-project/ multi-environment scenario, here are the main characteristics:

  • Ability to switch from totally different projects on the fly
  • Supports multiple environments (eg: development, production, worker1, worker2...)
  • No changes to the original files that could get overwritten in a upgrade
  • Supports multiple platform versions
  • Version 2.0 support for jboss and tomcat (the latest only for platform >= 1.7)
  • Version 2.1/3.x supports tomcat only (as imposed by Pentaho's dev build)
  • Simplifies version upgrades
  • Debug-friendly
  • VCS-friendly
  • Support all kinds of different customization in different projects, from using a different databases to different security types.
  • Supports patches to the source code for fine-grain customization
  • Supports deploy to local/remote machines
  • OS independent

Description

Project pentaho already has a good dev_build.xml ant script, but it wasn't enough for my needs, since I needed to change the svn sources in order to configure a solution. So I built my own structure, and its like this:

Directory layout and code checkout

MyProjectDir
|-- build.xml
|-- pentaho
|   |-- bi-platform-api
|   |-- bi-platform-appserver
|   |-- bi-platform-assembly
|   |-- bi-platform-assembly-open
|   |-- bi-platform-build
|   |-- bi-platform-engine-core
|   |-- bi-platform-engine-security
|   |-- bi-platform-engine-services
|   |-- bi-platform-legacy
|   |-- bi-platform-plugin-actions
|   |-- bi-platform-plugin-services
|   |-- bi-platform-repository
|   |-- bi-platform-sample-data
|   |-- bi-platform-sample-solution
|   |-- bi-platform-scheduler
|   |-- bi-platform-test-foundation
|   |-- bi-platform-ui-foundation
|   |-- bi-platform-util
|   |-- bi-platform-web
|   |-- bi-platform-web-portlet
|   |-- bi-platform-web-servlet
|   |-- dummy-jre
|   |-- mantle
|   `-- test-solutions
|-- project-client
|   |-- patches
|   |   |-- pentaho
|   |   `-- target-preconfiguredinstall
|   `-- solution
|-- target-build
`-- target-dist

pentaho/* directories are cheched out from svn, and for me are read-only. Feel free to use any svn client you want; I usually work in linux, or on a windows machine with cygwin, so I use the original client. I checked out the sources with:

$ svn co svn://source.pentaho.org/svnroot/bi-platform-v2/branches/3.0

If you are under a proxy you can use http:// instead of svn:// . I'm currently using version 3.5 from trunk. For earlier releases please check earlier CBF versions

target-/* directories are the ones generated by the build script. target-build is the temporary build directory. Since we may need to patch the pentaho source, CBF will make an entire copy of pentaho/ to target-build and patch the later, leaving the original source intact.

Pentaho 3.x build system requires tomcat to be downloaded separately. This has been tested with Tomcat 5.5.  The location of the tomcat directory is not important, and can be placed almost any where.  The location of the tomcat will be set inside of the build.properties file.  For now don't worry about this, it will be mentioned later on in this articale.  The same build system also requires xmltask to be installed.

xmltask installation
 The xmltask.jar should be placed in /home/<username>/.ant/lib/ directory to be accessible by ant.

This leaves us with the "project-client" directory. As mentioned before, the idea is not write anything under pentaho/* directories, so all my changes go to this directory. Here is an example structure in more detail:

project-client/
|-- config
|   |-- build.properties
|   `-- build-pedro.properties
|-- patches
|   |-- target-build
|   |   `--
|   `-- target-dist
|       `-- server
|            |-- conf
|            |   |--
|            |   `-- jboss-service.xml
|            |-- webapps
|            |   |-- pentaho
|            |   |   `-- WEB-INF
|            |   |       `-- web.xml
|            `-- lib
|                `-- postgresql-8.2-505.jdbc3.jar
`-- solution
    |-- Portal.properties
    |-- Portal.url
    |-- Portal_pt.properties
    |-- admin
    .... etc

All this is created manually according to the specific project needs.

Note to CBF version 2 users
the old target-preconfiguredinstall is now target-dist, where you will find the server, the admin console and licences

The idea is very simple: All changes that would normally go to pentaho/* are placed under "patches" directory (project-client/patches/). The CBF ant script will pick up the files in the project-client/patches/ directory, scan for tokens and replace the tokens with the variables defined inside the project-client/config/build.properties files, and copy the files to the top level directory of the entire project (In this example MyProjectDir). It's not recommended to patch anything under pentaho/*; sources changes are patched in to target-build/* and all other changes are made by patching the final directory, target-dist.

For the example scenario, those were the files I had to change. I had to change some jboss's config files just because I had to change a port that was unavailable; added my datasource (client-ds.xml) and changed the jboss-web.xml and web.xml to register it; changed all other ds's cause I'm using postgresql instead of hsql; added the jdbc lib; changed the security system from memory to jdbc.

Getting and setting web.xml
web.xml can be found at pentaho/bi-platform-appserver/webapps/pentaho/WEB-INF/web.xml and should be copied to the inside of your patches directory at project-client/patches/target-dist/server/webapps/pentaho/WEB-INF/web.xml

Inside of this file, tokens can be placed that will be set by the CBF ant script.  See the example snippet of web.xml bellow.

Example snippet of web.xml:

<context-param>
                <param-name>solution-path</param-name>
                <param-value>@solution.deploy.path@</param-value>
        </context-param>
        <context-param>
                <param-name>base-url</param-name>
                <param-value>@BASE_URL@</param-value>
        </context-param>

The tokens

@solution.deploy.path@ and @BASE_URL@

are defined in the project-client/config/build.properties or project-client/config/build-client.properties files and will be replaced by the CBF ant script and the new revised web.xml with the replaced tokens will be copied to the top level directory (In this example MyProjectDir).

This is scalable. Just change files there and they will be copied to the destination target. CBF does not allow (yet) to remove files from target to destination, but I also don't think that's a relevant issue.

Tip: Upgrading Jars
Sometimes there's the need to upgrade some Jar version. A very simple way to do so is to patch 2 files: myJar-oldversion.jar and myJar-newVersion.jar where the first one is an empty zip file that will overwrite the original one

The solution is under project-client/solution, and our build.properties points to this dir. For now, I just copied pentaho-solutions to here, and will start from there by adding new solutions and playing a bit.

The idea is to be able to check in the whole project-client directory into svn.
Tip: Maintaining system files
Tip: Unlike previous version, 3.0.1 doesn't require to manually copy admin and system directories, and can even copy samples

Aside: Heres a screenshot of all the files I changed to make pentaho 3.0 work for me:

Configuration files

The main config file is in config/build.properties; specific environment configurations can then be set in config/build-env.properties . Here's an example file:

build.properties
#####################################
## GENERIC PROPERTIES
######################################

javac.path = NOT SET

# Solution path. Required for compile time
# Set this to the absolute path to project-client/solution
solution.path = NOT SET


# Solution path required for runtime. Defaults to the above value but can be different if we're
# deploying to another place. Should be used in WEB-INF/web.xml in the solution-path parameter
solution.deploy.path = NOT SET

#accepted values: tomcat or jboss
server.name = tomcat

# Tomcat 5.5 path:
tomcat.path = NOT SET

# Copy the sample solutions to our project directory? true | false
copy.pentaho.samples = true


# Java options for the run command
java.opts = -Xmx512m -XX:MaxPermSize=512m -Xrunjdwp:transport=dt_socket,address=8765,server=y,suspend=n

#####################################
## PROJECT DIRECTORIES - The defaults usually work fine
######################################
pentaho.dir = pentaho/
pentaho.build.dir = target-build/

#####################################
## DEPLOY OPTIONS
######################################
deploy.mode = rsync
deploy.args = -avz --exclude '.svn/' --exclude '*.log' --exclude 'work/' --exclude  'temp_user/' --exclude 'temp/' --exclude 'tmp/'
deploy.dest.server = user@host:path
deploy.dest.solution = user@host:path

#####################################
## DYNAMIC TOKENS TO BE USED AS FILTERS
#####################################

DB_LOCATION = 127.0.0.1
DB_USER = pedro
DB_PASSWORD = bar
BASE_URL = http://127.0.0.1:8080/pentaho/

Tokens

To extend customization to a greater extent, I've implemented dynamic filter tokens for changeable attributes, such as database locations and passwords. Whenever CBF finds a text file with a string of the form  ( (DB_LOCATION)) that token will be replaced at compile time.

CBF execution and available targets

Building and executing

After all this environment is correctly set up, CBF is ready to be executed. ant executable needs to be on the path and the properties project and optionally env need to be passed. If no env property is passed only build.properties will be used.

Here are the available targets. The description speaks for itself:

CDF targets
$ ant -Dproject=client -Denv=pedro -p
Buildfile: build.xml
     [echo] --------------------------------------------------------------
     [echo] --------------------------------------------------------------
     [echo] ------        CBF - Community Build Framework          -------
     [echo] ------                 Version: 3.5.0                  -------
     [echo] ------ Author: Pedro Alves (pedro.alves@webdetails.pt) -------
     [echo] --------------------------------------------------------------
     [echo] --------------------------------------------------------------
     [echo]
     [echo]                          SETTINGS:
     [echo]
     [echo]    Project:         metrics
     [echo]    Environment:     build-pedro.properties
     [echo]    Config File:     project-client/config/build-pedro.properties
     [echo]    Server:          tomcat
     [echo] --------------------------------------------------------------
     [echo] --------------------------------------------------------------
Pentaho BI Platform build helper..
Main targets:

 all              Compiles and builds the entire project
 clean            Cleans the solution
 copy-finish      Copy target files
 copy-init        Copy project files
 deploy-all       Deploys both the solution and the server to the specified destinations
 deploy-server    Deploys the entire server to the specified destination
 deploy-solution  Deploys the solution to the specified destination
 deploy-war       Deploys the pentaho.war and pentaho-style.war to the specified destination
 dev-setup        Compiles and builds the entire project
 dist-clean       Deletes the target preconfigured install dir
 run              Runs the solution
Default target: all

To clean, compile and run pentaho all that's needed is, for example:

Simple execution
$ ant -Dproject=client -Denv=pedro dist-clean all run
The command will first load build.properties and then because of the -Denv=pedro, the command will load build-pedro.properties which must be in the <project root directory>/project-client/config/ directory. Loading build-pedro.properties will keep everything loaded from build.properties but replace tokens that are specified in build-pedro.properties

Just by passing different properties we can immediately build and run a totally different project, in a different platform version or different application server. With the right properties and tokens, we can instantly deploy a solution in a remote server.

Deploy

Despite the property deploy.mode, only the rsync method is currently supported. It's by far the most appropriate way to handle my kind of requirements, but a lot more modes can be implemented, like copy, scp or even ftp transfer.

Debug

With the appropriate JAVA_OPTS parameters, all that's needed to debug the platform is do a socket attach using your favorite IDE.

Ctools

Starting from version 3.10.1, there is an additional target named ctools that can automatically execute the ctools-installer.sh available here. You need to save the script in your CBF root (same folder where you have build.xml) and add the following properties in build.properties in order to have it working:

  • ctools.install = true
  • ctools.branch = dev (or stable, according to your preference)

Setting ctools.install to false or whatever value different from true will result in the target not to be executed, without build failure.
The ctools target is in the "depend" attribute of "all" target, so it suffices to use the command

ant -Dproject=client dist-clean all run

Download

Old versions

FAQ

I don't have any questions. Do you?

Changelog

Version Changes
3.10.1 Support for pentaho 3.10. It can install ctools script.
3.5.0 Suuport for pentaho 3.5
3.0.1 No longer required to manually handle system/admin folders. Project revision no longer set
3.0 Support for Pentaho 3.0
2.1 Support for Pentaho 2.x
2.0 Major changes; Introduced different configurations and deployments; dynamic tokens; deploys
1.0 Initial CBF

  1. Jul 30, 2008

    Christian Stamm says:

    For MySQL users: Due to a limitation of MySql which does not allow key columns ...

    For MySQL users:

    Due to a limitation of MySql which does not allow key columns to have more than 1000 Byte (see http://bugs.mysql.com/bug.php?id=4541) you have to apply patches to some Hibernate mapping files. These patches are already available in the sources you checked out from svn. So applying the patches is as easy as:

    cd project-client/
    mkdir -p patches/target-build/server/pentaho/src/
    cp -R ../pentaho/server/pentaho/res/hibernate/mysql5/org patches/target-build/server/pentaho/src/
    cp -R ../pentaho/server/pentaho/res/hibernate/mysql5/com patches/target-build/server/pentaho/src/

    Thanks to Kaapa for helping me out and pointig me at the solution!

  2. Oct 27, 2008

    Phillip Cole says:

    For the uninitiated: If working on CentOS use sun java and freshly downloaded ve...

    For the uninitiated:
    If working on CentOS use sun java and freshly downloaded versions of ant and tomcat. Don't try to use the versions supplied with CentOS.

  3. Nov 05, 2008

    Pius Onobhayedo says:

    I was able to build the solution following your steps only after modifying the f...

    I was able to build the solution following your steps only after modifying the file pentaho/bi-platform-build/dev_build.xml as follows:

     
    1. Changed bi-platform-assembly-open to bi-platform-assembly. The former does not exist and causes build error.

    Commented out:    <!-ant antfile="assembly.xml" dir="../bi-platform-assembly-open"->
    Replaced with:    <ant antfile="assembly.xml" dir="../bi-platform-assembly">

    2.  <!- Remove pentaho-bi-platform*.jar since we want to use classes instead. -->

    Commented out:
          <!--
        <delete>
          <fileset dir="$

    Unknown macro: {target.tomcat.dir}

    /webapps/pentaho/WEB-INF/lib">
            <include name="pentaho-bi-platform-*.jar" />
          </fileset>
        </delete>
          -->

      And:  

         <!--
        <antcall target="dev-update" />
        -->

  4. Nov 05, 2008

    Pedro Alves says:

    Hello Pius. The first one is a bug in pentaho build system. It's solved in trun...

    Hello Pius.

    The first one is a bug in pentaho build system. It's solved in trunk and should not be an issue when 2.0 is ga. What I did was just rename the directory.

    And I'm not sure I understood the second; Why do you specifically want the classes instead of the jar? There were problems in your comment, so i didn't understand the changes you did.

    I can run it w/o any changes apart from the directory rename

  5. Mar 24, 2009

    Sergio Souza says:

    Pedro,  CBF 3 works fine. I made changes to your code (CBF 3.0) and adde...

    Pedro,

     CBF 3 works fine.

    I made changes to your code (CBF 3.0) and added a feature to delete files in build target and target-dist directories.
     
    []
    Sergio Souza

  6. Mar 24, 2009

    Pedro Alves says:

    Hello Sergio, I'm glad it worked. What were the changes? And what are the diffe...

    Hello Sergio, I'm glad it worked.

    What were the changes? And what are the differences between your target and the dist-clean target?

    Thanks

  7. Mar 25, 2009

    TRIII says:

    Hi Pedro! Maybe I am missing something, where is the version 3 package? I can´t...

    Hi Pedro!

    Maybe I am missing something, where is the version 3 package? I can´t see it under the download section, only version 2.1.

    /Stefan

  8. Mar 25, 2009

    Pedro Alves says:

    I attached the file but forgot the link. thanks for mentioning it Pedro

    I attached the file but forgot the link. thanks for mentioning it

    Pedro

  9. Mar 26, 2009

    TRIII says:

    Hi again Pedro! Nice work with the build script, really appriciate the effort i...

    Hi again Pedro!

    Nice work with the build script, really appriciate the effort in this.

    Had some problem with the build script and specially the build.properties file. It turns out that I had to set the "project.revision" property in the "myproject/conf/build.properties" to the same value as the "project.revision" property in "myproject\pentaho\bi-platform-build\build.properties" to make it work.

    Is this intentional?

    Best regards
    Stefan

  10. Mar 26, 2009

    Pedro Alves says:

    That shouldn't even be there; It was a fix to a pentaho build problem that was s...

    That shouldn't even be there; It was a fix to a pentaho build problem that was solved in the meanwhile. I'll update the docs.

    In the meanwhile, there's also a problem in the system/admin dirs, it needs to be copied from the target-build directory and not from the original place

  11. Mar 30, 2009

    Jem Matzan says:

    I'd like to clean up the comments section here so that this page is easier to re...

    I'd like to clean up the comments section here so that this page is easier to read and interpret. Have all of the issues above been properly noted in the page content? If not, could someone volunteer to do that?

This documentation is maintained by the Pentaho community, and members are encouraged to create new pages in the appropriate spaces, or edit existing pages that need to be corrected or updated.

Please do not leave comments on Wiki pages asking for help. They will be deleted. Use the forums instead.

Adaptavist Theme Builder Powered by Atlassian Confluence