Hitachi Vantara Pentaho Community Wiki
Child pages
  • Loading Data into the MapR filesystem

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin
Wiki Markup
{scrollbar}

Excerpt

This guide shows you how How to use a PDI job to move a file into CLDBthe MapR filesystem.

Prerequisites

In order to follow along with this how-to guide you will need the following:

  • MapRPentaho
  • Data IntegrationKettle Spoon

Sample Files

The sample data files file needed for this guide is:

File Name

Content

weblogsweblog_rebuild.txt.zip

Unparsed, raw weblog data

Step-By-Step Instructions

Setup

Start MapR if it is not already running.

...

Create a Job to Put the Files into MapR

In this task you will load a file into the MapR 's CLDBfilesystem.

Tip
titleSpeed Tip

You can download the Kettle Job load_cldb.kjb if you don't want to do every step

  1. Start PDI on your desktop. Once it is running choose 'File' -> 'New' -> 'Job' from the menu system or click on the 'New file' icon on the toolbar and choose the 'Job' option.

  2. Add a Start Job Entry: You need to tell PDI where to start the job, so expand the 'General' section of the Design palette and drag a 'Start' job entry onto the job canvas.
    Image Removed Image Added

  3. Add a Copy Files Job Entry: You will copy files from your local disk to the MapR CLDBfilesystem, so expand the 'HadoopBig Data' section of the Design palette and drag a 'Hadoop Copy Files' job entry onto the job canvas. Your canvas should look like this:
    Image Removed Image Added

  4. Connect the Start and Copy Files Job Entries: Hover the mouse over the 'Start' node and a tooltip will appear. Image Removed Image Added Click on the output connector (the green arrow pointing to the right) and drag a connector arrow to the 'Hadoop Copy Files' node. Your canvas should look like this:
    Image Removed Image Added

  5. Edit the Copy Files Job Entry: Double-click on the 'Hadoop Copy Files' node to edit its properties. Enter this information:
    1. File/Folder source(s): The folder containing the sample files you want to add to the CLDBMapR filesystem.
    2. File/Folder destination(s): maprfs://<CLDB>:<PORT>/weblogs/raw
      When running PDI on the same machine as the MapR cluster use: maprfs:///weblogs/raw the CLDB MapR filesystem and port are not required.
      <CLDB> is the server name of the machine running the MapR CLDBfilesystem.
      <PORT> is the port the MapR CLDB filesystem is running on.
    3. Wildcard (RegExp): Enter ^.*\.txt
    4. Click the Add button to add the above entries to the list of files you wish to copy.
    5. Check the "Create destination folder" option to ensure that the weblogs folder is created in CLDB the MapR filesystem the first time this job is executed.
      When you are done your 'Copy Files' window should look like this (your file paths may be different):

      Click 'OK' to close the window.

  6. Save the Job: Choose 'File' -> 'Save as...' from the menu system. Save the transformation as 'load_cldb.kjb' into a folder of your choice.

  7. Run the Job: Choose 'Action' -> 'Run' from the menu system or click on the green run button on the job toolbar. An 'Execute a job' window will open. Click on the 'Launch' button. An 'Execution Results' panel will open at the bottom of the PDI window and it will show you the progress of the job as it runs. After a few seconds the job should finish successfully:
    If any errors occurred the job step that failed will be highlighted in red and you can use the 'Logging' tab to view error messages.

Check MapR

  1. If you have mounted your MapR CLDB filesystem onto your local machine you may verify the file loaded by navigating to the MapR directory.
    Code Block
    ls /mapr/my.cluster.com/weblogs/raw
    This should return:
    weblog_raw.txt
  2. If you have not mounted your MapR CLDB filesystem onto your local machine you may alternatively check MapR by:
    Code Block
    Hadoophadoop fs -ls /weblogs/raw
    This should return:
    -rwxrwxrwx 3 demo demo 77908174 2011-12-28 07:16 /weblogs/raw/weblog_raw.txt

    Summary

    In this guide you learned how to copy local files into MapR's CLDB filesystem using PDI's graphical design tool. You can use this tool to put files into the CLDB MapR filesystem from many different sources.
    Wiki Markup
    {scrollbar}