public interface RecordRollup extends Transform
Represents an interface of a rollup transform which processes groups of data records. Each group of data records may share a data record referred to as a group "accumulator". It may be used to store intermediate results. The group "accumulator" is created an initialized when the first data record of a group is encountered and updated for each data record in this group (including the first and the last data record). When the last data record of a group is encountered, processing of this group is finished and the group "accumulator" is disposed. If the input data records are not sorted, each group is finished as soon as all input data records are read and processed.
The life cycle of a rollup transform is as follows:
init(Properties, DataRecordMetadata, DataRecordMetadata, DataRecordMetadata[])
method is called
to initialize the transform.initGroup(DataRecord, DataRecord)
method is called to initialize processing of the group
and to initialize the "accumulator" (if it exists).updateGroup(DataRecord, DataRecord)
method is called for the current data record and
the corresponding group "accumulator" (if it was requested).true
, the updateTransform(int, DataRecord, DataRecord, DataRecord[])
method is called repeatedly to generate output data record(s) until it returns RecordRollup.SKIP. If it returns
value < RecordRollup.SKIP, the Transform.getMessage()
method is called.finishGroup(DataRecord, DataRecord)
method is called to finish the group processing.true
, the transform(int, DataRecord, DataRecord, DataRecord[])
method is called repeatedly to generate output data record(s) until it returns RecordRollup.SKIP. If it
returns value < RecordRollup.SKIP, the Transform.getMessage()
method is called.Transform.finished()
method is called to notify that the rollup transform finished.Transform.reset()
method may optionally be called to reset the transformation and so that the previous
two steps may be executed again.| Modifier and Type | Field and Description |
|---|---|
static int |
ALL
the return value of the transform() method specifying that the record will be sent to all the output ports
|
static int |
SKIP
the return value of the transform() method specifying that the record will be skipped
|
| Modifier and Type | Method and Description |
|---|---|
boolean |
finishGroup(DataRecord inputRecord,
DataRecord groupAccumulator)
This method is called for the last data record in a group in order to finish the group processing.
|
boolean |
finishGroupOnError(java.lang.Exception exception,
DataRecord inputRecord,
DataRecord groupAccumulator)
This method is called for the last data record in a group in order to finish the group processing.
|
void |
init(java.util.Properties parameters,
DataRecordMetadata inputMetadata,
DataRecordMetadata accumulatorMetadata,
DataRecordMetadata[] outputMetadata)
Initializes the rollup transform.
|
void |
initGroup(DataRecord inputRecord,
DataRecord groupAccumulator)
This method is called for the first data record in a group.
|
void |
initGroupOnError(java.lang.Exception exception,
DataRecord inputRecord,
DataRecord groupAccumulator)
This method is called for the first data record in a group.
|
int |
transform(int counter,
DataRecord inputRecord,
DataRecord groupAccumulator,
DataRecord[] outputRecords)
This method is used to generate output data records based on the input data record and the contents of the group
"accumulator" (if it was requested).
|
int |
transformOnError(java.lang.Exception exception,
int counter,
DataRecord inputRecord,
DataRecord groupAccumulator,
DataRecord[] outputRecords)
This method is used to generate output data records based on the input data record and the contents of the group
"accumulator" (if it was requested).
|
boolean |
updateGroup(DataRecord inputRecord,
DataRecord groupAccumulator)
This method is called for each data record (including the first one as well as the last one) in a group
in order to update the group "accumulator".
|
boolean |
updateGroupOnError(java.lang.Exception exception,
DataRecord inputRecord,
DataRecord groupAccumulator)
This method is called for each data record (including the first one as well as the last one) in a group
in order to update the group "accumulator".
|
int |
updateTransform(int counter,
DataRecord inputRecord,
DataRecord groupAccumulator,
DataRecord[] outputRecords)
This method is used to generate output data records based on the input data record and the contents of the group
"accumulator" (if it was requested).
|
int |
updateTransformOnError(java.lang.Exception exception,
int counter,
DataRecord inputRecord,
DataRecord groupAccumulator,
DataRecord[] outputRecords)
This method is used to generate output data records based on the input data record and the contents of the group
"accumulator" (if it was requested).
|
finished, getGraph, getMessage, getNode, postExecute, preExecute, reset, setNode
static final int ALL
static final int SKIP
void init(java.util.Properties parameters, DataRecordMetadata inputMetadata, DataRecordMetadata accumulatorMetadata, DataRecordMetadata[] outputMetadata) throws ComponentNotReadyException
parameters
- custom component attributes defined for the component which calls this transformationinputMetadata
- metadata of input data recordsaccumulatorMetadata
- metadata of a group "accumulator" used to store intermediate results
or null
if no metadata were specifiedoutputMetadata
- array of metadata of output data recordsComponentNotReadyException
- if an error occurred during the initializationvoid initGroup(DataRecord inputRecord, DataRecord groupAccumulator) throws TransformException
inputRecord
- the first input data record in the groupgroupAccumulator
- the data record used as an "accumulator" for the group or null
if none
was requestedTransformException
- if any error occurred during initialization of the groupvoid initGroupOnError(java.lang.Exception exception, DataRecord inputRecord, DataRecord groupAccumulator) throws TransformException
initGroup(DataRecord, DataRecord)
throws an exception.exception
- an exception that caused initGroup(DataRecord, DataRecord)
to failinputRecord
- the first input data record in the groupgroupAccumulator
- the data record used as an "accumulator" for the group or null
if none
was requestedTransformException
- if any error occurred during initialization of the groupboolean updateGroup(DataRecord inputRecord, DataRecord groupAccumulator) throws TransformException
inputRecord
- the current input data recordgroupAccumulator
- the data record used as an "accumulator" for the group or null
if none
was requestedtrue
if the updateTransform(int, DataRecord, DataRecord, DataRecord[])
method
should be called to generate output data record(s) and send them to the output, false
otherwiseTransformException
- if any error occurred during update of the groupboolean updateGroupOnError(java.lang.Exception exception, DataRecord inputRecord, DataRecord groupAccumulator) throws TransformException
updateGroup(DataRecord, DataRecord)
throws an exception.exception
- an exception that caused updateGroup(DataRecord, DataRecord)
to failinputRecord
- the current input data recordgroupAccumulator
- the data record used as an "accumulator" for the group or null
if none
was requestedtrue
if the updateTransform(int, DataRecord, DataRecord, DataRecord[])
method
should be called to generate output data record(s) and send them to the output, false
otherwiseTransformException
- if any error occurred during update of the groupboolean finishGroup(DataRecord inputRecord, DataRecord groupAccumulator) throws TransformException
inputRecord
- any input data record from the current groupgroupAccumulator
- the data record used as an "accumulator" for the group or null
if none
was requestedtrue
if the transform(int, DataRecord, DataRecord, DataRecord[])
method should
be called to generate output data record(s) and send them to the output, false
otherwiseTransformException
- if any error occurred during final processing of the groupboolean finishGroupOnError(java.lang.Exception exception, DataRecord inputRecord, DataRecord groupAccumulator) throws TransformException
finishGroup(DataRecord, DataRecord)
throws an exception.exception
- an exception that caused finishGroup(DataRecord, DataRecord)
to failinputRecord
- any input data record from the current groupgroupAccumulator
- the data record used as an "accumulator" for the group or null
if none
was requestedtrue
if the transform(int, DataRecord, DataRecord, DataRecord[])
method should
be called to generate output data record(s) and send them to the output, false
otherwiseTransformException
- if any error occurred during final processing of the groupint updateTransform(int counter, DataRecord inputRecord, DataRecord groupAccumulator, DataRecord[] outputRecords) throws TransformException
updateGroup(DataRecord, DataRecord)
method returns true.counter
- the number of previous calls to this method for the current group updateinputRecord
- the current input data recordgroupAccumulator
- the data record used as an "accumulator" for the group or null
if none
was requestedoutputRecords
- output data records to be filled with dataTransformException
- if any error occurred during the transformationint updateTransformOnError(java.lang.Exception exception, int counter, DataRecord inputRecord, DataRecord groupAccumulator, DataRecord[] outputRecords) throws TransformException
updateGroup(DataRecord, DataRecord)
method returns true. Called only
if updateTransform(int, DataRecord, DataRecord, DataRecord[])
throws an exception.exception
- an exception that caused updateTransform(int, DataRecord, DataRecord, DataRecord[])
to failcounter
- the number of previous calls to this method for the current group updateinputRecord
- the current input data recordgroupAccumulator
- the data record used as an "accumulator" for the group or null
if none
was requestedoutputRecords
- output data records to be filled with dataTransformException
- if any error occurred during the transformationint transform(int counter, DataRecord inputRecord, DataRecord groupAccumulator, DataRecord[] outputRecords) throws TransformException
finishGroup(DataRecord, DataRecord)
method returns true.counter
- the number of previous calls to this method for the current groupinputRecord
- any input data record from the current groupgroupAccumulator
- the data record used as an "accumulator" for the group or null
if none
was requestedoutputRecords
- output data records to be filled with dataTransformException
- if any error occurred during the transformationint transformOnError(java.lang.Exception exception, int counter, DataRecord inputRecord, DataRecord groupAccumulator, DataRecord[] outputRecords) throws TransformException
finishGroup(DataRecord, DataRecord)
method returns true. Called only
if transform(int, DataRecord, DataRecord, DataRecord[])
throws an exception.exception
- an exception that caused transform(int, DataRecord, DataRecord, DataRecord[])
to failcounter
- the number of previous calls to this method for the current groupinputRecord
- any input data record from the current groupgroupAccumulator
- the data record used as an "accumulator" for the group or null
if none
was requestedoutputRecords
- output data records to be filled with dataTransformException
- if any error occurred during the transformationCopyright © 2002-2015 Javlin a.s.


Today's top highlights
Latest breaking news, pictures, videos, and special reports
Last week top highlights
Check last weeks top highlights, news, stories and headline news
Trending topics
View all categoriesSports update
Get the latest sports news and updates from football, tennis, formula one, cricket, NBA, and golf with live scores and stats
Sponsored news
Content by: BootstrapNever miss a story!
Get the freshest headlines and updates sent uninterrupted to your inbox.


fun checkValidity(): Unit
This method will detect graph cycles in the full composite key structure to protect against infinite loops when traversing the graph and key duplicates in the each layer. It also checks if the threshold and weight constraint requirements are met, while it tests for aggregated-weight integer overflow. In practice, this method should be always invoked on the root CompositeKey, as it inherently validates the child nodes (all the way till the leaves).


Woah there. Whatever you're trying to do is not allowed for this URL, my friend. You better shape up or we're gonna have a hell of a time around here.
If you feel offended or think that this shouldn't be an error, ">shoot us a love letter and let us know you saw this.


I've been getting a lot of e-mail asking things like: "Brent, your simple database is great, can it handle a file with over a million entries?" The answer to that question is simple, NO, don't do it! Flat-file text "databases" are nice for smaller projects, but what if you have a large database or want to maintain complex data structures? Perl is very flexible and can be easily connected to just about any SQL database.
In this article, I am not going to have any "live" examples. I'll be talking about Perl and SQL databases, where you can get a SQL database, where to get the drivers needed for Perl to talk to the database and show you a few code snippets to get you going.
This article, and subsequent ones, will focus on the Perl/CGI aspect of databases. Many of the techniques apply equally to non-CGI programs, but since my true "calling" is the web, the web is what I write about.
Although I like my "simple" database program and what it can do, once you get the basics of using Perl with SQL databases, you may find that it is easier to use the SQL databases for all projects!
On a performance note, since regular CGI programs start a new process each time the program is accessed, they are not necessarily ideal for high traffic sites. The good news is that there are programs out there, like mod_perl or VelociGen, that plug directly into the web server and speed everything up dramatically because they keep Perl running rather than start it each time it is needed.
There are so many different databases and system configurations, it would not be possible to teach you how to install a database. Most come with good documentation, make sure you follow along exactly with any installation instructions. I had to re-install my SQL server because I forgot a step during installation, don't let this happen to you!
SQL databases can be very gigantic, expensive entities or they can be smaller and more manageable. You don't have to spend thousands to get the performance you need. I've used Perl to connect to databases on both NT and Unix systems and it works great. Actually, once you have it set up and working, I think it is easier to work with the SQL databases than my "simple" one. Plus, creating new tables and queries is much easier if you know a little SQL.
Here are some links to SQL databases. By no means is this list complete, but should be a good starting point for you.
You need to acquire, install, and configure the database before you will be able to access it with Perl. I currently use MySQL on a Linux system and have had great success with it.
Once you have your database installed and working, you need to get the proper drivers/modules so that Perl will be able to talk to the database. Normally there are two things you'll need, the DBD DataBase Driver module and a DBI DataBase Interface for the DBD.
The most comprehensive place for getting these files is at the DBI Home Page. You will also find code samples and documentation. Almost everything I've learned thus far about Perl and SQL databases was obtained from these pages, use them well!
If you are on a Windows system, make sure that you have downloaded and installed the latest version of Perl from ActiveState. Then, you can use the Perl Package Manager (PPM) to load the modules needed for database connectivity. The PPM is an extremely valuable resource for Windows Perl users, it downloads and installs modules for you!
Ok, once you have the database installed and the DBI/DBD drivers loaded - you can read past this point. I am going to provide you with a few snippets of Perl code that I use all the time for calling SQL databases. Play with these snippets to test your installation, then use them to build more powerful applications.
(Snippet 1) ########################################## ## Create a connection to the database. ## ########################################## 1: sub Create_DB_Connection{ 2: use DBI; 3: $DSN = "DBI:mysql:mydatabase"; 4: $user = "dbuser"; 5: $pw = "dbpass"; 6: $dbh = DBI->connect($DSN,$user,$pw) 7: || die "Cannot connect: $DBI::errstr\n" unless $dbh; 8: return; 9: } # End of Create_DB_Connection subroutine. ##########################################
Line 1: Begins the subroutine. This subroutine creates a connection to a database.
Line 2: Tells Perl to use the DBI module. If things are not installed properly, you'll probably get an error at this point.
Line 3: Sets the variable $DSN to DBI:mysql:mydatabase
The "DSN" stands for Data Source Name.
Lines 4-5: Set variables for username and password. These are the username and password that will be used to access the database and are not necessarily the system username and password.
Note: Lines 3-5 can be set in your call to the subroutine. You could set the variables to the values passed in @_[0], @_[1], and @_[2] respectively. Remember that when you call a subroutine like so:
&Create_DB_Connection("DBI:mysql:mydatabase","dbuser","dbpass");
The values passed are stored in the array called @_.
Line 6: Calls the connect method in the DBI module and passes it the values of the variables we just created. The results of this are stored in the $dbh variable. This variable stands for DataBase Handle because it is how we access the database.
Line 7: This line is actually a continuation of line 6. It generates an error message telling you that it was unsuccessful if there was a problem connecting to the database. Many times the errors will be because the database name was misspelled or the user you are trying to connect as does not have sufficient rights.
Line 8: This line simply returns us back to the program.
Line 9: Ends the subroutine.
(Snippet 2) ########################################## ## Executes the SQL command and then ## ## returns to the calling area of the ## ## program. ## ########################################## 1: sub Do_SQL{ 2: eval{ 3: $sth = $dbh->prepare($SQL); 4: }; # End of eval 5: # Check for errors. 6: if($@){ 7: $dbh->disconnect; 8: print "Content-type: text/html\n\n"; 9: print "An ERROR occurred! $@\n<P>"; 10: exit; 11: } else { 12: $sth->execute; 13: } # End of if..else 14: return ($sth); 15: } # End of Do_SQL subroutine ##########################################
Line 1: This line begins our next snippet. This snippet is called Do_SQL. Do_SQL takes a SQL statement stored in the variable $SQL and tries to execute it.
Line 2: Begins an eval block. In Perl, whatever is in an eval block is treated like a separate Perl program. Any errors generated in an eval block are passed to the $@ variable. If there are no errors, $@ will be not contain any value.
Line 3: Calls the prepare function of our database. We call the database functions via the statement handle. Prepare basically checks the SQL to make sure there are no errors in it.
Line 4: Ends the eval block.
Line 5: A Perl comment.
Line 6: Checks to see if there is anything in<EMAIL_ADDRESS>If there is, then we had an error and need to do something about it.
Line 7: Disconnects us from the database. Whenever you are connected to the database, by creating a $dbh database handle, you should disconnect from the database before you exit the program. This helps prevent open, unused database connections and possible errors.
Line 8: Prints the standard HTML header. We need to print some text to the user and since we are programming a CGI application, we must tell the server what type of data we are sending.
Line 9: Prints an error message and also the contents of $@ so that the user can see what error was generated.
Line 10: Exits the program. We cannot go on because we are not connected to the database.
Line 11: Our else condition. If there was no error, we should end up in this block.
Line 12: We use the execute method of the statement handle to actually execute the SQL statement in the database.
Line 13: Ends the if..else block.
Line 14: Returns the $sth (statement handle) so that we can do useful things with the data.
Line 15: Ends the subroutine.
(Snippet 3) 1: sub filter{ 2: $_[0]=~s/\'/\\\'/g; 3: return $_[0]; 4: } # End of filter subroutine
This snippet may be small, but I also find it very useful. It takes any characters that may cause problems in your SQL statements and escapes the characters in question. This snippet would need to be modified for your database since some characters cause errors in some databases and not others.
To find out what characters cause errors, I normally run simple SQL statements through the database and use different characters each time to see which ones generate errors. If I find one that generates an error, I add it to this subroutine.
This subroutine is called like this:
$value_to_check = &filter($value_to_check);
Line 1: Begins the subroutine.
Line 2: Takes the value passed to the subroutine and changes any bad characters by placing a backslash in front of them. This escapes the characters so that the database can process them.
Line 3: Returns the value passed to the subroutine back to the calling variable.
Line 4: Ends the subroutine.
Well, those are the snippets I use most often. If you want to see a CGI program that actually does something, here is some code that I use. Here are the common subroutines that I use too.
I am not going to go over each line of the search code yet. We will be covering more of with SQL soon. If you want some great documentation on the DBI/DBD modules, go to the MySQL Perl DBI/DBD Manual. Although it is for MySQL, 95% if the information applies to other databases as well.
I hope I've provided you with enough snippets, code, and information to get you started down the read of Perl/SQL. We'll be covering more about this soon!
Perl on!