Added by Matt Casters, last edited by Jens Bleuel on Nov 19, 2007  (view change)

Labels:

Enter labels to add to this page:
Wait Image 
Looking for a label? Just start typing.

Splits up into a master and executes automatically usining CTRL-U in the latest Spoon.

Execution results (Master):

Step name Copy Nr Read Written Input Output Updated Errors Active Time Speed pr/in/out Sleeps
Reader : 60000) 0 0 10000 10000 0 0 0 Finished 22.5 445.0 - 0/23845
Assemble 0 20000 20000 0 0 0 0 Finished 24.0 834.9 - 24533/82638
Reader : 60001) 0 0 10000 10000 0 0 0 Finished 24.0 417.4 - 0/112479
Read 0 0 10000 10001 0 0 0 Finished 15.2 657.8 - 0/109743
Writer : 60002) 0 10000 0 0 0 0 0 Finished 15.8 633.6 - 119571/0
Writer : 60003) 0 10000 0 0 0 0 0 Finished 16.1 620.1 - 2366/0
Read 2 0 0 10000 10001 0 0 0 Finished 15.2 658.4 - 0/112458
Get zipcode-region 0 0 2774 0 0 0 0 Finished 7.4 376.9 - 0/7347
Writer : 60004) 0 2774 0 0 0 0 0 Finished 8.2 337.5 - 854/0
Writer : 60005) 0 2774 0 0 0 0 0 Finished 7.7 359.3 - 5282/0
Table output 0 20000 0 0 20000 0 0 Finished 23.9 836.0 - 38059/0

And Two slaves: 

One:

Step name Copy Nr Read Written Input Output Updated Errors Active Time Speed pr/in/out Sleeps
Look up Region using zipcode 0 12774 10000 0 0 0 0 Finished 18.4 692.8 - 5930/221122
Writer : 60001) 0 10000 0 0 0 0 0 Finished 20.5 488.9 - 68926/0
Reader : 60003) 0 0 10000 10000 0 0 0 Finished 15.8 634.3 - 0/80362
Calc 0 10000 10000 0 0 0 0 Finished 17.3 577.6 - 34376/107801
Value Mapper 0 10000 10000 0 0 0 0 Finished 18.4 544.2 - 2914/260912
Reader : 60005) 0 0 2774 2774 0 0 0 Finished 5.6 498.7 - 0/0

Two:

Step name Copy Nr Read Written Input Output Updated Errors Active Time Speed pr/in/out Sleeps
Look up Region using zipcode 0 12774 10000 0 0 0 0 Finished 12.6 1010.1 - 62589/152
Writer : 60000) 0 10000 0 0 0 0 0 Finished 13.9 721.1 - 111172/0
Reader : 60002) 0 0 10000 10000 0 0 0 Finished 12.6 792.7 - 0/4485
Calc 0 10000 10000 0 0 0 0 Finished 12.6 791.8 - 81480/29648
Value Mapper 0 10000 10000 0 0 0 0 Finished 12.6 792.1 - 79631/34977
Reader : 60004) 0 0 2774 2774 0 0 0 Finished 4.5 611.4 - 0/0

Comments  (Hide Comments)

Wow - looks sweet

Comment: Posted by Marc Batchelor at Dec 01, 2006 13:52

So what was the total time for each of the two configurations?

James

Comment: Posted by admin at Dec 01, 2006 13:56

From the data it looks like it took 24 seconds in total.  However, this is not a performance driven example, rather than a functionality test.

The test here was to see if the system could handle remote stream lookup driven by local data.  In the case where you have a remote server with lots of RAM, that might make sense.

This sample is obviously limited by the text file input and table output steps, NOT the stream lookup.

Comment: Posted by Matt Casters at Dec 29, 2006 14:16