With JMS it is possible to process messages in two basic methods:
A message can be send to a specified JMS Queue and can be received by exactly one consumer. If there are no consumers available at the time the message is sent it will be kept until a consumer is available that can process the message. This is also know as a point-to-point model.
In JMS a Topic implements publish and subscribe semantics. When you publish a message it goes to all the subscribers who are interested - so zero to many subscribers will receive a copy of the message. Only subscribers who had an active subscription at the time the broker receives the message will get a copy of the message. This is also known as a publish/subscribe model.
In the following sample solution we use the JMS Consumer and JMS Producer steps that are delivered as part of the Pentaho Data Integration Enterprise Edition and use Apache ActiveMQ as a JMS test server.
This was tested with PDI EE Version 4.2 GA and ActiveMQ version 5.5.1
Please see also the Knowledge Base for reference documentation about the JMS Consumer and JMS Producer steps.
- Download ActiveMQ from the Apache site.
- Start the ActiveMQ server, e.g. by executing apache-activemq-5.5.1-bin\bin\activemq.bat
- After the server startup, you can connect with your web browser to the default port, e.g. http://localhost:8161/admin/ to see if the server is up and running correctly.
- Have a PDI EE version available. If you don't have an Enterprise Edition, yet, an evaluation version can be downloaded.
- Download the attached sample transformations.
Execute the transformation JMS Producer - QUEUE.ktr
This transformation creates 10 test messages and puts these into the message server queue test_queue.
You can see that the queue is created and the messages have arrived at the server, you can monitor it by http://localhost:8161/admin/queues.jsp
Now retrieve the messages by the transformation JMS Consumer - QUEUE.ktr
After the execution, you see that the messages have been dequeued and are no longer pending:
You can redo this multiple times and can also do a preview on the Dummy step or process this further, e.g. for the first five rows:
The ongoing for the topics is pretty much the same, but you need to start the consumer first! It is also possible to start multiple consumers as subscribers.
- Start the JMS Consumer - TOPIC.ktr
- Keep it running as a subscriber
- Start the JMS Producer - TOPIC.ktr
- You may monitor the topics in the Topics browser section or look at the file JMS_Topics_Received.xml (after you stopped the JMS Consumer - TOPIC transformation since the file is written with delay).
The transactional behavior is set to AUTO_ACKNOWLEDGE by default in the JMS steps: With this acknowledgment mode, the session automatically acknowledges a client's receipt of a message either when the session has successfully returned from a call to receive or when the message listener the session has called to process the message successfully returns.
This means, producers and consumers are decoupled from each other and these messages are loosely coupled and asynchronous.
- Step documentation in the Knowledge Base PDI User Guide: JMS Consumer / JMS Producer
- Basic JMS API Concepts
- Apache ActiveMQ
Note: There are issues with showing the real exceptions in versions 4.2.0 GA and 4.2.1 GA. In case of exceptions you may need to look at the Spoon console or within the Pentaho DI Server at the Tomcat console. Further information about known issues is shown in JIRA for the JMS Consumer and JMS Producer steps.