Incorporate Comments 1

Incorporate Comments 1

////
/**
* @@@ START COPYRIGHT @@@
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
*   http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*
* @@@ END COPYRIGHT @@@
*/
////

[#working with lobs]
= Working with LOBs

[#creating a sql table with lob columns]
== Creating a SQL Table with LOB Columns

When creating a SQL table with LOB columns, following relevant tables and files are created as well:

* One LOB MD table.
* Two dependent descriptor tables.
* HDFS data file (locates at /user/trafodion/lobs) for each column.

[#syntax]
== Syntax

```
CREATE TABLE table-name (column-spec[, column-spec]…)
```

```
column-spec is:
lob-column-spec     |
other-column-spec

lob-column-spec is:
column-name {lob-data-type}[column-constraint]

other-column-spec is:
column-name {data-type}[column-constraint]

lob-data-type is:
BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']

unit is:
empty |
K     |
M     |
G
```

[#semantics]
=== Semantics

* `_storage literal_`

+
Currently Trafodion only supports `'EXTERNAL'` here.

+
External LOB object that are not managed by Trafodion.

* `_empty_`

+
Number of bytes specified by the numeric literal.

* `_K_`

+
Numeric literal value * 1024.

* `_M_`

+
Numeric literal value * 1024 * 1024.

* `_G_`

+
Numeric literal value * 1024 * 1024 * 1024.

[#examples]
=== Examples

* This example creates a table tlob1 with 2 columns and primary key on the c1.

+

```
CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
```

* This example creates a table tlob2 with 3 columns and primary key on the c1.

+

```
CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
```

* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.

+

```
CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
```

* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.

+

```
CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
```

* This example creates a table tlob130ext with 4 columns and primary key on the c1.

+

```
CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
```

[#hdfs location of lob data]
=== HDFS Location of LOB Data

When a LOB table is created, the underlying LOB data needs to be stored in HDFS. It is in the /user/trafodion/lobs by default.

All columns of a table that are declared as LOB types will have all their data in one file derived from the table's Object UID and the LOB number of that column which gets assigned during creation.

The following is a LOB file with 2 columns you will see 2 files in HDFS:

/user/trafodion/lobs/LOBP_03683514167332904796_0001

/user/trafodion/lobs/LOBP_03683514167332904796_0002

As rows are added to this table, the LOB data for each row gets appended to the corresponding column’s LOB data file.

[#inserting into a sql table containing lob columns]
== Inserting into a SQL Table Containing LOB Columns

[#syntax]
=== Syntax

```
INSERT INTO table-name [(target-col-list)] insert-source
```

```
target-col-list is:
colname[, colname]...

insert-source is:
VALUES(column-expr[, column-expr]...)

column-expr is:
lob-query-expr           |
other-query-expr

lob-query-expr is:
NULL | ?                                                                     |
EMPTY_BLOB()                                                                 |
EMPTY_CLOB()                                                                 |
STRINGTOLOB('string literal expression')                                     |
FILETOLOB('lob source file name')                                            |
BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
EXTERNALTOLOB('external lob source file name')

lob source file name is:
hdfs:///{local hdfs file name}     |
{local linux file name}            |
{file:///linux file name}

external lob source file name is:
hdfs:///{local hdfs file name}
```
[#semantics]
=== Semantics

* `_other-query-expr_`
+
For the syntax and description of `_other-query-expr_`, see the `_query-expr_` in the <<select_statement,SELECT Statement>>.

* `_EMPTY_BLOB(), EMPTY_CLOB()_`
+
Returns an empty LOB handle.

* `_STRINGTOLOB_`
+
Converts a simple string literal into LOB format.

** `_string literal expression_`
+
is a series of characters enclosed in single quotes.

* `_FILETOLOB_`
+
Converts data from a local linux/hdfs file into LOB format.

* `_BUFFERTOLOB_`
+
Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB.

** `_lob source buffer address_`
+
The long value of the user buffer address in int64.

** `_lob length value_`
+
The length of the user specified lob buffer in int64.

[#considerations]
=== Considerations

The source for inserting into a LOB can be any of the following:

* A parameter.
+
An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.

* `EMPTY_BLOB()` or `EMPTY_CLOB()`

** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created.

*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.

*** The dummy LOB handle will get the same datatype as the underlying column.
+
For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB.

** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.

* An in-memory LOB which is simple string data.
+
To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.

* An on-platform file (linux/hdfs file) containing binary or text data.
+
To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.

* A user buffer of a specified length allocated in user space.
+
To insert a buffer, you need to provide the address and size of the buffer.

* An external LOB.
+
When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file.
+
For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data.

[#examples]
=== Examples

* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
+
```
INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
```

* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that value.
+
```
INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
```

* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
+
```
INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
```

* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file.
+
```
INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),
FILETOLOB('hdfs:///lobs/lob_input_a1.txt'),
EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
```

* This example uses a parameter.
+
```
PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
```

* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle.
+
```
INSERT INTO t130lob2 VALUES (1, empty_blob());
```

[#inserting into a sql table containing lob columns using select clause]
== Inserting into a SQL Table Containing LOB Columns Using Select Clause

[#syntax]
=== Syntax

```
INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
```

```
target-col-list-expr is:
target-colname[, target-colname]...

target-colname is:
lob-column-name                   |
other-column-name

source-col-list-expr is:
source-colname[, source-colname]...

source-colname is:
hive varchar column name          |
trafodion varchar column name     |
trafodion char column name        |
trafodion lob column name

source-table is:
hive table                        |
trafodion table
```

[#semantics]
=== semantics

* `_target-col-list-expr_`
+
names a single column or multiple columns enclosed in parentheses in the target table in which to insert values.

+
The data type of each target column must be compatible with the data type of its corresponding source value.

+
Within the list, each target column must have the same position as its associated source value.

** `_target-colname_`
+
is a SQL identifier that specifies a target column name.

* `_source-col-list-expr_`
+
names a single column or multiple columns enclosed in parentheses in the source table from which to get values.

** `_source-colname_`
+
is a SQL identifier that specifies a source column name.

[#considerations]
=== Considerations

When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type.

The source for the *select clause* can be any of the following:

* A source hive table column that is a hive varchar column
* A source table column that is a Trafodion varchar, char and LOB column
* A source table column that is also a LOB datatype.

[#examples]
=== Examples
* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
+
```
INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
```

* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
+
```
INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
```

* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
+
```
INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
```

* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
+
```
INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
```

[#updating a sql table containing lob columns]
== Updating a SQL Table Containing LOB Columns

The source for updating a LOB can be divided into 2 groups:

* Update using parameters/functions
* Update using LOB handle

[#updating using parameters/functions]
=== Updating Using Parameters/Functions

The following parameters/functions can be used to update.

* A parameter
* `EMPTY_BLOB` or `EMPTY_CLOB`
* An in-memory lob which is a simple string data
+
To insert this string, a literal needs to be provided
* An on-platform file (linux/hdfs file) containing text or binary data
* A user buffer of a specified length allocated in user space
* An external LOB file in HDFS

[#syntax]
==== Syntax

```
UPDATE table-name {set-clause-type1 | set-clause-type2}
```

```
set-clause-type1 is:
SET set-clause[, set-clause ]..

set-clause is:
lob_column-name = {lob_query-expr}

lob_query-expr is:
NULL | ?                                                                                |
EMPTY_BLOB()                                                                            |
EMPTY_CLOB()                                                                            |
STRINGTOLOB('string literal expression'[, APPEND])                                      |
FILETOLOB('lob source file name'[, APPEND])                                             |
BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
EXTERNALTOLOB('external lob source file name'[, APPEND])

lob source file name is:
hdfs:///{local hdfs file name}      |
{local linux file name}             |
{file:///linux file name}

external lob source file name is:
hdfs:///{local hdfs file name}
```

[#semantics]
==== Semantics

For more information, see Semantics in <<inserting into a sql table containing lob columns,Inserting into a SQL Table Containing LOB Columns>>.

[#examples]
==== Examples

* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
+
```
UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
```

* In the table tlob1, updates (overwrites) the value of c2 to anoush.jpg.
+
```
UPDATE tlob1 SET c2=filetolob('anoush.jpg');
```

* In the table tlob1 where c1 is 3, updates (appends) the value of c2 to lob_update.txt stored in hdfs:///lobs/.
+
```
UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
```

* In the table tlob1, updates (overwrites) the value of c2 to the buffer location at 1254674 with 4000-byte length.
+
```
UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
```

* In the table tlob130ext where c1 is 2, updates (overwrites) the value of c4 to lob_input_d1.txt stored in hdfs:///lobs/.
+
```
UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
```

* In the table t130lob2 where c1 is 1, updates (overwrites) value of the c2 to xyxyxyxyxyxyx.
+
```
PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
EXECUTE S USING 'xyxyxyxyxyxyx';
```

[# updating using lob handle]
=== Updating Using Lob Handle

A LOB handle is specified to the update (similar to extract).

[#syntax]
==== Syntax

```
UPDATE LOB (LOB 'lob handle', lob update expression)
```

```
lob update expression is:
LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |
EMPTY_BLOB()                                                                 |
EMPTY_CLOB()
```

For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].

[#considerations]
=== Considerations

* The keyword `APPEND` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.

* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.

* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data.

+
So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.

* `APPEND` cannot be used on function `EXTERNALTOLOB`.

* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion.
+
For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned.

* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column.
+
Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
+
That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified.

[#selecting column from a sql table containing lob columns]
== Selecting Column from a SQL Table Containing LOB Columns

Selecting a row from a table will give back the lob handle for the lob column.

NOTE: The entry in each lob column in the SQL table only contains the LOB handle.

Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.

[#syntax]
=== Syntax

```
SELECT lob_query-expr[, lob_query-expr] FROM table-name
```

```
lob_query_expr is:
lobtostring(lob column)      |
lob column
```

[#examples]
=== Examples

* This example selects c2 from table tlob1 and returns the lob handle of c2.
+
```
SELECT c2 FROM tlob1;
C2
----------------------------------------------------------------------------------------------------

LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE"

--- 1 row(s) selected.
```

[#extracting lob data from a sql table containing lob columns]
== Extracting LOB Data from a SQL Table Containing LOB Columns

Extract lob data can be done in two ways:

* Extract lob data into a file for a given lob handle
+
Extract from a LOB column straight into an on-platform linux or HDFS file in one shot.

+
Trafodion engine will take care of buffering the data and retrieve the lob data into the target file.

* Extract lob data into a user specified buffer
+
Extract from a LOB column into a target user buffer of a specified size.
+
Trafodion engine will retrieve exactly the amount of requested data.
+
The user/application will be responsible for redriving the extract until end of data.
+
The extracted buffer data can then be written to files or another location as the application chooses.

[#extracting lob data into a file for a given lob handle]
=== Extracting Lob Data into a File for a Given Lob Handle

[#syntax]
==== Syntax

```
EXTRACT LOBTOFILE (LOB 'lob handle as quoted string', 'filename URI format' [OPTION])
```

```
OPTION is:
[, TRUNCATE]
[, CREATE, TRUNCATE]
[, APPEND]
[, CREATE, APPEND]
```

[#semantics]
==== Semantics

* `TRUNCATE`
+
If the target file exists, Trafodion will truncate and write to it.
+
If the target file does not exist, an error will be raised.

* `CREATE, TRUNCATE`
+
If the target file exists, Trafodion will truncate and write to it.
+
If the target file does not exist, Trafodion will create a file and write to it.

* `APPEND`
+
If the target file exists, Trafodion will append to it.
+
If the target file does not exist, an error will be raised.

* `CREATE, APPEND`
+
If the target file exists, Trafodion will append to it.
+
If the target file does not exist, Trafodion will create a file and append to it.

[#considerations]
==== Considerations

If the target file exists, the `OPTION` must be specified, or else an error will be raised. This is the default behavior.

If the target file does not exist, you can create a target file by specifying the `OPTION`.

[#examples]
==== Examples

* This example extracts LOB to the tlob130_txt1.txt:

+
```
EXTRACT LOBTOFILE (LOB 'LOBH00000200010520117997292583625519884121437206093184618212317486018305654020"TRAFODION"."LOB130"', 'tlob130_txt1.txt');
Success. Targetfile:tlob130_txt1.txt  Length: 19
```

* This example extracts LOB to the tlob130_deep.jpg:
+
```
EXTRACT LOBTOFILE (LOB 'LOBH00000200010520117997292583681719884121437210516812518212317486062586654020"TRAFODION"."LOB130"',
Success. Targetfile:tlob130_deep.jpg  Length: 159018
```

[#extracting lob data into a user specified buffer]
=== Extracting Lob Data into a User Specified Buffer

Extract from a LOB column into a target user buffer of a specified size. The Trafodion engine will retrieve exactly the amount of requested data. The user/application will be responsible for redriving the extract until end of data. Then the extracted buffer data can be written to files or another location as the application chooses.

Extract LOB data into a user specified buffer like a cursor until EOD is returned. For this method, the user specifies an input buffer and specifies the input length in an in/out variable.

For each execution of the extract, Trafodion will return SUCCESS, ERROR or EOD.
For the cases of SUCCESS or EOD, a length will also be returned to the user, so the user knows exactly how much data was actually extracted and returned.

[#syntax]
==== Syntax

```
EXTRACT LOBTOBUFFER (LOB 'lob handle as quoted string', LOCATION lob output buffer address as long, SIZE input/output address of length container as long)
```

[#extracting lob length for a given lob handle]
=== Extracting Lob Length for a Given Lob Handle

Extract from a LOB column straight into an on-platform linux or HDFS file in one shot.

[#syntax]
==== Syntax

```
EXTRACT LOBLENGTH (LOB 'lob handle as quoted string'[, LOCATION address of length container for lob length])
```

[#semantics]
==== Semantics

* `_LOCATION address of length container for lob length_`

+
This is used by programs/applications that will use this syntax to retrieve the LOB length prior to extracting data. The address should be an address of a 64-bit container that will hold the LOB length.
+
If the length is omitted or 0, only the status message is returned that displays the length.

[#examples]
==== Examples

* This example extracts LOB length and returns 30.

+
```
EXTRACT LOBLENGTH (LOB 'LOBH0000000800030554121478481170502119554121478546064413218212330526373762019024"TRAFODION"."TESTEXTLOB"');
LOB Length: 30
--- SQL operation complete.
```

* This example extracts LOB length and returns 4.

+
```
EXTRACT LOBLENGTH (LOB 'LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE"');
LOB Length: 4
--- SQL operation complete.
```

[#considerations]
=== Considerations

* LOB Max Extract Data Length
+
CQD `LOB_OUTPUT_SIZE` (default 32000) controls the maximum data length that can be extracted.

* LOB Max Extract Chunk Length
+
CQD `LOB_MAX_CHUNK_MEM_SIZE` (512 MB expressed in bytes [536870912]) controls the maximum chunk of data that can be read from HDFS into memory and written to the target file location.

* LOB Max Size
+
CQD `LOB_MAX_SIZE` (default 10G expressed in M [10000M]).

* Extract Target Locations
+
The file to extract to can be a local linux file or a local HDFS file.

[#deleting column from a sql table containing lob columns]
== Deleting Column from a SQL Table Containing LOB columns

[#syntax]
=== Syntax

```
DELETE lob-column-name FROM table-name [WHERE CLAUSE]
```

[#considerations]
=== Considerations

When one or more rows containing LOB columns are deleted from LOB table, only the metadata information is dropped and the hdfs data remains as it is. The references to the lob data are removed from the lob descriptor file.

This mechanism has not been implemented yet as a separate utility but it is triggered as a part of insert, update and append operations. For more information, see <<garbage collection,Garbage Collection>>.

[#dropping a sql table containing lob columns ]
== Dropping a SQL Table Containing LOB Columns

Drop works like any other drop table. All dependent tables are deleted. All files in hdfs (data and descriptor) files are also deleted.

For more information, see <<drop_table_statement,DROP TABLE Statement>> in http://trafodion.incubator.apache.org/docs/sql_reference/index.html[Trafodion SQL Reference Manual].

[#garbage collection]
== Garbage Collection

When a lob datafile for a column has reached a certain limit, defined by a CQD `LOB_GC_LIMIT_SIZE`, then a compaction is triggered automatically. +
The default Garbage Collection (GC) Limit is 10GB and can be changed if needed.

The need for GC arises because when a delete operation or an update operation is performed, the old data black in the hdfs file will be left as unused. +
In the case of update, the old data will be left as unused and the new data will be written into a new section, so all these “holes” in the LOB data file are needlessly occupying space.

The LOB descriptor chunks file is looked at to see which ranges and offsets are actually used. The LOB datafile is temporarily saved. The compaction is done into a new tempfile. When the sections have all been copied into the tempfile, Trafodion will delete the existing lob data file and rename the tempfile.

Finally, the saved copy of the LOB datafile is dropped. The saved copy is there just in case you need to fall back to it in case of an error. Since this operation is triggered as part of an IUD operation, a definite slowdown will occur for that insert/update operation compared to subsequent inserts/updates.

Also, each lob column of a table can be compacted separately as needed. GC does not have to be done to all columns of the LOB table all at once.

NOTE: Currently the GC is done in the same transaction as the transaction being used for the insert or update operation. If any part of the GC fails, then the entire transaction is aborted.

When Trafodion has support for local transactions, Trafodion will do the GC in a separate transaction or in a separate process, so you can fail the GC with a warning and allow the insert to go through.

Setting the CQD `LOB_GC_LIMIT_SIZE` to 0 would prevent GC from occurring.

[#cleanup of a sql table containing lob columns]
== Cleanup of a SQL Table Containing LOB Columns

Cleanup works like cleanup of any other table. The command ensures all dependent SQL LOB tables and hdfs files are dropped ignoring errors if any.

For more information, see <<cleanup_statement,CLEANUP Statement>> in http://trafodion.incubator.apache.org/docs/sql_reference/index.html[Trafodion SQL Reference Manual].

[#showddl for lob]
== SHOWDDL for LOB

SHOWDDL for LOB with a special option will show all the dependent objects, names and details about the table.

[#syntax]
=== Syntax

```
SHOWDDL table-name, LOB DETAILS
```

[#examples]
=== Examples

* This example displays the details of the table t1ob1.

+

```
>>SHOWDDL tlob1, LOB DETAILS;
CREATE TABLE TRAFODION.SEABASE.TLOB1
  (
    C1                               INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , C2                               BLOB DEFAULT NULL NOT SERIALIZED
  , PRIMARY KEY (C1 ASC)
  )
;

LOB Metadata
============

CREATE TABLE TRAFODION.SEABASE.LOBMD_04239091936503896833
  (
    LOBNUM                           SMALLINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , STORAGETYPE                      SMALLINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , LOCATION                         VARCHAR(4096) CHARACTER SET ISO88591 COLLATE DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , PRIMARY KEY (LOBNUM ASC)
  )
;

************************************************
LobNum: 1

Data Storage
============

Location: /user/trafodion/lobs
DataFile: LOBP_04239091936503896833_0001

LOB Descriptor Handle
=====================

CREATE TABLE TRAFODION.SEABASE."LOBDescHandle_04239091936503896833_0001"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , NUMCHUNKS                        INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , LOBLEN                           LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  )
  STORE BY (DESCPARTNKEY ASC)
  SALT USING 8 PARTITIONS
;

LOB Descriptor Chunks
=====================

CREATE TABLE TRAFODION.SEABASE."LOBDescChunks_04239091936503896833_0001"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , DESCSYSKEY                       LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKNUM                         INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKLEN                         LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , INTPARAM                         LARGEINT DEFAULT NULL SERIALIZED
  , STRINGPARAM                      VARCHAR(400) CHARACTER SET ISO88591 COLLATE DEFAULT DEFAULT NULL SERIALIZED
  , PRIMARY KEY (DESCPARTNKEY ASC, DESCSYSKEY ASC, CHUNKNUM ASC)
  )
  SALT USING 8 PARTITIONS
;

--- SQL operation complete.
```

* This example displays the details of the table tlob130ext.

+

```
>>CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', primary key (c1));

--- SQL operation complete.

>>SHOWDDL tlob130ext, LOB DETAILS;

CREATE TABLE TRAFODION.SCH.TLOB130EXT
  (
    C1                               INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , C2                               BLOB DEFAULT NULL NOT SERIALIZED
  , C3                               CLOB DEFAULT NULL NOT SERIALIZED
  , C4                               BLOB DEFAULT NULL NOT SERIALIZED
  , PRIMARY KEY (C1 ASC)
  )
;

LOB Metadata
============

CREATE TABLE TRAFODION.SCH.LOBMD__04474425229029907479
  (
    LOBNUM                           SMALLINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , STORAGETYPE                      SMALLINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , LOCATION                         VARCHAR(4096) CHARACTER SET ISO88591 COLLATE DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , PRIMARY KEY (LOBNUM ASC)
  )
;

************************************************
LobNum: 1

Data Storage
============

Location: /user/trafodion/lobs
DataFile: LOBP_04474425229029907479_0001

LOB Descriptor Handle
=====================

CREATE TABLE TRAFODION.SCH."LOBDescHandle__04474425229029907479_0001"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , NUMCHUNKS                        INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , LOBLEN                           LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  )
  STORE BY (DESCPARTNKEY ASC)
  SALT USING 8 PARTITIONS
;

LOB Descriptor Chunks
=====================

CREATE TABLE TRAFODION.SCH."LOBDescChunks__04474425229029907479_0001"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , DESCSYSKEY                       LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKNUM                         INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKLEN                         LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , DATAOFFSET                       LARGEINT DEFAULT NULL SERIALIZED
  , STRINGPARAM                      VARCHAR(400) CHARACTER SET ISO88591 COLLATE DEFAULT DEFAULT NULL SERIALIZED
  , PRIMARY KEY (DESCPARTNKEY ASC, DESCSYSKEY ASC, CHUNKNUM ASC)
  )
  SALT USING 8 PARTITIONS
;

************************************************
LobNum: 2

Data Storage
============

Location: /user/trafodion/lobs
DataFile: LOBP_04474425229029907479_0002

LOB Descriptor Handle
=====================

CREATE TABLE TRAFODION.SCH."LOBDescHandle__04474425229029907479_0002"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , NUMCHUNKS                        INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , LOBLEN                           LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  )
  STORE BY (DESCPARTNKEY ASC)
  SALT USING 8 PARTITIONS
;

LOB Descriptor Chunks
=====================

CREATE TABLE TRAFODION.SCH."LOBDescChunks__04474425229029907479_0002"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , DESCSYSKEY                       LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKNUM                         INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKLEN                         LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , DATAOFFSET                       LARGEINT DEFAULT NULL SERIALIZED
  , STRINGPARAM                      VARCHAR(400) CHARACTER SET ISO88591 COLLATE DEFAULT DEFAULT NULL SERIALIZED
  , PRIMARY KEY (DESCPARTNKEY ASC, DESCSYSKEY ASC, CHUNKNUM ASC)
  )
  SALT USING 8 PARTITIONS
;

************************************************
LobNum: 3

Data Storage
============

<External HDFS location>
<External HDFS file>

LOB Descriptor Handle
=====================

CREATE TABLE TRAFODION.SCH."LOBDescHandle__04474425229029907479_0003"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , NUMCHUNKS                        INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , LOBLEN                           LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  )
  STORE BY (DESCPARTNKEY ASC)
  SALT USING 8 PARTITIONS
;

LOB Descriptor Chunks
=====================

CREATE TABLE TRAFODION.SCH."LOBDescChunks__04474425229029907479_0003"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , DESCSYSKEY                       LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKNUM                         INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKLEN                         LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , DATAOFFSET                       LARGEINT DEFAULT NULL SERIALIZED
  , STRINGPARAM                      VARCHAR(400) CHARACTER SET ISO88591 COLLATE DEFAULT DEFAULT NULL SERIALIZED
  , PRIMARY KEY (DESCPARTNKEY ASC, DESCSYSKEY ASC, CHUNKNUM ASC)
  )
  SALT USING 8 PARTITIONS
;

--- SQL operation complete.
```

[#get lob statistics for a lob table ]
== Get Lob Statistics for a LOB Table

There are two ways to get lob statistics for a lob table:

* Get Statement: the lob information is formatted for human readability.
* Select Statement: the lob information is formatted for machine readability.

[#get statment]
=== Get Statement

[#syntax]
==== Syntax

```
GET LOB STATS FOR TABLE table-name;
```

[#examples]
==== Examples

* This Get Statement displays statistics for the table tlob130gt2.

+

```
>>CREATE TABLE tlob130gt2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));

--- SQL operation complete.

>>GET LOB STATS FOR TABLE tlob130gt2;

Lob Information for table: "TRAFODION".LOB130.TLOB130GT2
=========================

ColumnName:  C2
Lob Location:  /user/trafodion/lobs
LOB Data File:  LOBP_07468755986685501835_0001
LOB EOD:  0
LOB Used Len:  0
ColumnName:  C3
Lob Location:  /user/trafodion/lobs
LOB Data File:  LOBP_07468755986685501835_0002
LOB EOD:  0
LOB UsedLen:  0
ColumnName:  C4
Lob Location:  External HDFS Location
LOB Data File:  External HDFS File
LOB EOD:  0
LOB Used Len:  0
--- SQL operation complete.
```

[#select statment]
=== Select Statement

[#syntax]
==== Syntax

```
SELECT * FROM TABLE(LOB STATS table-name);
```

[#examples]
==== Examples

* This Select Statement displays statistics for the table tlob130gt.

+

```
>>CREATE TABLE tlob130gt (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB, PRIMARY KEY (c1));

--- SQL operation complete.

>>SELECT left(trim(catalog_name) || '.' || trim(schema_name) || '.' || trim(object_name) , 20),
left(trim(column_name),5),
left(trim(lob_location),5),
left(trim(lob_data_file),20),
LOB_DATA_FILE_SIZE_EOD,
LOB_DATA_FILE_SIZE_USED
FROM TABLE(lob stats(tlob130gt));

(EXPR)      (EXPR)     (EXPR)     (EXPR)     LOB_DATA_FILE_SIZE_EOD    LOB_DATA_FILE_SIZE_USED
------      ------     ------     ------     ----------------------    ----------------------

TRAFODION.SCH.TLOB13   C2   /user/trafodion/lobs   LOBP_044744252290302   15   10
TRAFODION.SCH.TLOB13   C3   /user/trafodion/lobs   LOBP_044744252290302   15   10
TRAFODION.SCH.TLOB13   C4   /user/trafodion/lobs   LOBP_044744252290302   45   30

--- 3 row(s) selected.
```

* This Select Statement displays statistics for the table tlob130gt2.

+

```
>>CREATE TABLE tlob130gt2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));

--- SQL operation complete.

>>SELECT left(trim(catalog_name) || '.' || trim(schema_name) || '.' || trim(object_name), 20),
left(trim(column_name),5),
left(trim(lob_location),15),
left(trim(lob_data_file),20),
LOB_DATA_FILE_SIZE_EOD,
LOB_DATA_FILE_SIZE_USED
FROM TABLE(lob stats(tlob130gt2));

(EXPR)      (EXPR)     (EXPR)     (EXPR)     LOB_DATA_FILE_SIZE_EOD    LOB_DATA_FILE_SIZE_USED
------      ------     ------     ------     ----------------------    ----------------------

TRAFODION.SCH.TLOB13   C2   /user/trafodion/lobs   LOBP_044744252290300   0   0
TRAFODION.SCH.TLOB13   C3   /user/trafodion/lobs   LOBP_044744252290300   0   0
TRAFODION.SCH.TLOB13   C4   External HDFS Location External HDFS File     0   0

--- 3 row(s) selected.
```

////
/**
* @@@ START COPYRIGHT @@@
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
*   http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*
* @@@ END COPYRIGHT @@@
*/
////

[#working with lobs]
= Working with LOBs

[#creating a sql table with lob columns]
== Creating a SQL Table with LOB Columns

When creating a SQL table with LOB columns, following relevant tables and files are created as well:

* One LOB MD table.
* Two dependent descriptor tables.
* HDFS data file (locates at /user/trafodion/lobs) for each column.

[#syntax]
== Syntax

```
CREATE TABLE table-name (column-spec[, column-spec]…)
```

```
column-spec is:
lob-column-spec     |
other-column-spec

lob-column-spec is:
column-name {lob-data-type}[column-constraint]

other-column-spec is:
column-name {data-type}[column-constraint]

lob-data-type is:
BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']

unit is:
empty |
K     |
M     |
G
```

[#semantics]
=== Semantics

* `_storage literal_`

+
Currently Trafodion only supports `'EXTERNAL'` here.

+
External LOB object that are not managed by Trafodion.

* `_empty_`

+
Number of bytes specified by the numeric literal.

* `_K_`

+
Numeric literal value * 1024.

* `_M_`

+
Numeric literal value * 1024 * 1024.

* `_G_`

+
Numeric literal value * 1024 * 1024 * 1024.

[#examples]
=== Examples

* This example creates a table tlob1 with 2 columns and primary key on the c1.

+

```
CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
```

* This example creates a table tlob2 with 3 columns and primary key on the c1.

+

```
CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
```

* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.

+

```
CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
```

* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.

+

```
CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
```

* This example creates a table tlob130ext with 4 columns and primary key on the c1.

+

```
CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
```

[#hdfs location of lob data]
=== HDFS Location of LOB Data

When a LOB table is created, the underlying LOB data needs to be stored in HDFS. It is in the /user/trafodion/lobs by default.

All columns of a table that are declared as LOB types will have all their data in one file derived from the table's Object UID and the LOB number of that column which gets assigned during creation.

The following is a LOB file with 2 columns you will see 2 files in HDFS:

/user/trafodion/lobs/LOBP_03683514167332904796_0001

/user/trafodion/lobs/LOBP_03683514167332904796_0002

As rows are added to this table, the LOB data for each row gets appended to the corresponding column’s LOB data file.

[#inserting into a sql table containing lob columns]
== Inserting into a SQL Table Containing LOB Columns

[#syntax]
=== Syntax

```
INSERT INTO table-name [(target-col-list)] insert-source
```

```
target-col-list is:
colname[, colname]...

insert-source is:
VALUES(column-expr[, column-expr]...)

column-expr is:
lob-query-expr           |
other-query-expr

lob-query-expr is:
NULL | ?                                                                     |
EMPTY_BLOB()                                                                 |
EMPTY_CLOB()                                                                 |
STRINGTOLOB('string literal expression')                                     |
FILETOLOB('lob source file name')                                            |
BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
EXTERNALTOLOB('external lob source file name')

lob source file name is:
hdfs:///{local hdfs file name}     |
{local linux file name}            |
{file:///linux file name}

external lob source file name is:
hdfs:///{local hdfs file name}
```
[#semantics]
=== Semantics

* `_other-query-expr_`
+
For the syntax and description of `_other-query-expr_`, see the `_query-expr_` in the <<select_statement,SELECT Statement>>.

* `_EMPTY_BLOB(), EMPTY_CLOB()_`
+
Returns an empty LOB handle.

* `_STRINGTOLOB_`
+
Converts a simple string literal into LOB format.

** `_string literal expression_`
+
is a series of characters enclosed in single quotes.

* `_FILETOLOB_`
+
Converts data from a local linux/hdfs file into LOB format.

* `_BUFFERTOLOB_`
+
Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB.

** `_lob source buffer address_`
+
The long value of the user buffer address in int64.

** `_lob length value_`
+
The length of the user specified lob buffer in int64.

[#considerations]
=== Considerations

The source for inserting into a LOB can be any of the following:

* A parameter.
+
An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.

* `EMPTY_BLOB()` or `EMPTY_CLOB()`

** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created.

*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.

*** The dummy LOB handle will get the same datatype as the underlying column.
+
For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB.

** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.

* An in-memory LOB which is simple string data.
+
To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.

* An on-platform file (linux/hdfs file) containing binary or text data.
+
To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.

* A user buffer of a specified length allocated in user space.
+
To insert a buffer, you need to provide the address and size of the buffer.

* An external LOB.
+
When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file.
+
For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data.

[#examples]
=== Examples

* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
+
```
INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
```

* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that value.
+
```
INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
```

* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
+
```
INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
```

* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file.
+
```
INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),
FILETOLOB('hdfs:///lobs/lob_input_a1.txt'),
EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
```

* This example uses a parameter.
+
```
PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
```

* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle.
+
```
INSERT INTO t130lob2 VALUES (1, empty_blob());
```

[#inserting into a sql table containing lob columns using select clause]
== Inserting into a SQL Table Containing LOB Columns Using Select Clause

[#syntax]
=== Syntax

```
INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
```

```
target-col-list-expr is:
target-colname[, target-colname]...

target-colname is:
lob-column-name                   |
other-column-name

source-col-list-expr is:
source-colname[, source-colname]...

source-colname is:
hive varchar column name          |
trafodion varchar column name     |
trafodion char column name        |
trafodion lob column name

source-table is:
hive table                        |
trafodion table
```

[#semantics]
=== semantics

* `_target-col-list-expr_`
+
names a single column or multiple columns enclosed in parentheses in the target table in which to insert values.

+
The data type of each target column must be compatible with the data type of its corresponding source value.

+
Within the list, each target column must have the same position as its associated source value.

** `_target-colname_`
+
is a SQL identifier that specifies a target column name.

* `_source-col-list-expr_`
+
names a single column or multiple columns enclosed in parentheses in the source table from which to get values.

** `_source-colname_`
+
is a SQL identifier that specifies a source column name.

[#considerations]
=== Considerations

When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type.

The source for the *select clause* can be any of the following:

* A source hive table column that is a hive varchar column
* A source table column that is a Trafodion varchar, char and LOB column
* A source table column that is also a LOB datatype.

[#examples]
=== Examples
* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
+
```
INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
```

* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
+
```
INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
```

* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
+
```
INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
```

* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
+
```
INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
```

[#updating a sql table containing lob columns]
== Updating a SQL Table Containing LOB Columns

The source for updating a LOB can be divided into 2 groups:

* Update using parameters/functions
* Update using LOB handle

[#updating using parameters/functions]
=== Updating Using Parameters/Functions

The following parameters/functions can be used to update.

* A parameter
* `EMPTY_BLOB` or `EMPTY_CLOB`
* An in-memory lob which is a simple string data
+
To insert this string, a literal needs to be provided
* An on-platform file (linux/hdfs file) containing text or binary data
* A user buffer of a specified length allocated in user space
* An external LOB file in HDFS

[#syntax]
==== Syntax

```
UPDATE table-name {set-clause-type1 | set-clause-type2}
```

```
set-clause-type1 is:
SET set-clause[, set-clause ]..

set-clause is:
lob_column-name = {lob_query-expr}

lob_query-expr is:
NULL | ?                                                                                |
EMPTY_BLOB()                                                                            |
EMPTY_CLOB()                                                                            |
STRINGTOLOB('string literal expression'[, APPEND])                                      |
FILETOLOB('lob source file name'[, APPEND])                                             |
BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
EXTERNALTOLOB('external lob source file name'[, APPEND])

lob source file name is:
hdfs:///{local hdfs file name}      |
{local linux file name}             |
{file:///linux file name}

external lob source file name is:
hdfs:///{local hdfs file name}
```

[#semantics]
==== Semantics

For more information, see Semantics in <<inserting into a sql table containing lob columns,Inserting into a SQL Table Containing LOB Columns>>.

[#examples]
==== Examples

* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
+
```
UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
```

* In the table tlob1, updates (overwrites) the value of c2 to anoush.jpg.
+
```
UPDATE tlob1 SET c2=filetolob('anoush.jpg');
```

* In the table tlob1 where c1 is 3, updates (appends) the value of c2 to lob_update.txt stored in hdfs:///lobs/.
+
```
UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
```

* In the table tlob1, updates (overwrites) the value of c2 to the buffer location at 1254674 with 4000-byte length.
+
```
UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
```

* In the table tlob130ext where c1 is 2, updates (overwrites) the value of c4 to lob_input_d1.txt stored in hdfs:///lobs/.
+
```
UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
```

* In the table t130lob2 where c1 is 1, updates (overwrites) value of the c2 to xyxyxyxyxyxyx.
+
```
PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
EXECUTE S USING 'xyxyxyxyxyxyx';
```

[# updating using lob handle]
=== Updating Using Lob Handle

A LOB handle is specified to the update (similar to extract).

[#syntax]
==== Syntax

```
UPDATE LOB (LOB 'lob handle', lob update expression)
```

```
lob update expression is:
LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |
EMPTY_BLOB()                                                                 |
EMPTY_CLOB()
```

For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].

[#considerations]
=== Considerations

* The keyword `APPEND` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.

* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.

* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data.

+
So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.

* `APPEND` cannot be used on function `EXTERNALTOLOB`.

* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion.
+
For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned.

* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column.
+
Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
+
That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified.

[#selecting column from a sql table containing lob columns]
== Selecting Column from a SQL Table Containing LOB Columns

Selecting a row from a table will give back the lob handle for the lob column.

NOTE: The entry in each lob column in the SQL table only contains the LOB handle.

Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.

[#syntax]
=== Syntax

```
SELECT lob_query-expr[, lob_query-expr] FROM table-name
```

```
lob_query_expr is:
lobtostring(lob column)      |
lob column
```

[#examples]
=== Examples

* This example selects c2 from table tlob1 and returns the lob handle of c2.
+
```
SELECT c2 FROM tlob1;
C2
----------------------------------------------------------------------------------------------------

LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE"

--- 1 row(s) selected.
```

[#extracting lob data from a sql table containing lob columns]
== Extracting LOB Data from a SQL Table Containing LOB Columns

Extract lob data can be done in two ways:

* Extract lob data into a file for a given lob handle
+
Extract from a LOB column straight into an on-platform linux or HDFS file in one shot.

+
Trafodion engine will take care of buffering the data and retrieve the lob data into the target file.

* Extract lob data into a user specified buffer
+
Extract from a LOB column into a target user buffer of a specified size.
+
Trafodion engine will retrieve exactly the amount of requested data.
+
The user/application will be responsible for redriving the extract until end of data.
+
The extracted buffer data can then be written to files or another location as the application chooses.

[#extracting lob data into a file for a given lob handle]
=== Extracting Lob Data into a File for a Given Lob Handle

[#syntax]
==== Syntax

```
EXTRACT LOBTOFILE (LOB 'lob handle as quoted string', 'filename URI format' [OPTION])
```

```
OPTION is:
[, TRUNCATE]
[, CREATE, TRUNCATE]
[, APPEND]
[, CREATE, APPEND]
```

[#semantics]
==== Semantics

* `TRUNCATE`
+
If the target file exists, Trafodion will truncate and write to it.
+
If the target file does not exist, an error will be raised.

* `CREATE, TRUNCATE`
+
If the target file exists, Trafodion will truncate and write to it.
+
If the target file does not exist, Trafodion will create a file and write to it.

* `APPEND`
+
If the target file exists, Trafodion will append to it.
+
If the target file does not exist, an error will be raised.

* `CREATE, APPEND`
+
If the target file exists, Trafodion will append to it.
+
If the target file does not exist, Trafodion will create a file and append to it.

[#considerations]
==== Considerations

If the target file exists, the `OPTION` must be specified, or else an error will be raised. This is the default behavior.

If the target file does not exist, you can create a target file by specifying the `OPTION`.

[#examples]
==== Examples

* This example extracts LOB to the tlob130_txt1.txt:

+
```
EXTRACT LOBTOFILE (LOB 'LOBH00000200010520117997292583625519884121437206093184618212317486018305654020"TRAFODION"."LOB130"', 'tlob130_txt1.txt');
Success. Targetfile:tlob130_txt1.txt  Length: 19
```

* This example extracts LOB to the tlob130_deep.jpg:
+
```
EXTRACT LOBTOFILE (LOB 'LOBH00000200010520117997292583681719884121437210516812518212317486062586654020"TRAFODION"."LOB130"',
Success. Targetfile:tlob130_deep.jpg  Length: 159018
```

[#extracting lob data into a user specified buffer]
=== Extracting Lob Data into a User Specified Buffer

Extract from a LOB column into a target user buffer of a specified size. The Trafodion engine will retrieve exactly the amount of requested data. The user/application will be responsible for redriving the extract until end of data. Then the extracted buffer data can be written to files or another location as the application chooses.

Extract LOB data into a user specified buffer like a cursor until EOD is returned. For this method, the user specifies an input buffer and specifies the input length in an in/out variable.

For each execution of the extract, Trafodion will return SUCCESS, ERROR or EOD.
For the cases of SUCCESS or EOD, a length will also be returned to the user, so the user knows exactly how much data was actually extracted and returned.

[#syntax]
==== Syntax

```
EXTRACT LOBTOBUFFER (LOB 'lob handle as quoted string', LOCATION lob output buffer address as long, SIZE input/output address of length container as long)
```

[#extracting lob length for a given lob handle]
=== Extracting Lob Length for a Given Lob Handle

Extract from a LOB column straight into an on-platform linux or HDFS file in one shot.

[#syntax]
==== Syntax

```
EXTRACT LOBLENGTH (LOB 'lob handle as quoted string'[, LOCATION address of length container for lob length])
```

[#semantics]
==== Semantics

* `_LOCATION address of length container for lob length_`

+
This is used by programs/applications that will use this syntax to retrieve the LOB length prior to extracting data. The address should be an address of a 64-bit container that will hold the LOB length.
+
If the length is omitted or 0, only the status message is returned that displays the length.

[#examples]
==== Examples

* This example extracts LOB length and returns 30.

+
```
EXTRACT LOBLENGTH (LOB 'LOBH0000000800030554121478481170502119554121478546064413218212330526373762019024"TRAFODION"."TESTEXTLOB"');
LOB Length: 30
--- SQL operation complete.
```

* This example extracts LOB length and returns 4.

+
```
EXTRACT LOBLENGTH (LOB 'LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE"');
LOB Length: 4
--- SQL operation complete.
```

[#considerations]
=== Considerations

* LOB Max Extract Data Length
+
CQD `LOB_OUTPUT_SIZE` (default 32000) controls the maximum data length that can be extracted.

* LOB Max Extract Chunk Length
+
CQD `LOB_MAX_CHUNK_MEM_SIZE` (512 MB expressed in bytes [536870912]) controls the maximum chunk of data that can be read from HDFS into memory and written to the target file location.

* LOB Max Size
+
CQD `LOB_MAX_SIZE` (default 10G expressed in M [10000M]).

* Extract Target Locations
+
The file to extract to can be a local linux file or a local HDFS file.

[#deleting column from a sql table containing lob columns]
== Deleting Column from a SQL Table Containing LOB columns

[#syntax]
=== Syntax

```
DELETE lob-column-name FROM table-name [WHERE CLAUSE]
```

[#considerations]
=== Considerations

When one or more rows containing LOB columns are deleted from LOB table, only the metadata information is dropped and the hdfs data remains as it is. The references to the lob data are removed from the lob descriptor file.

This mechanism has not been implemented yet as a separate utility but it is triggered as a part of insert, update and append operations. For more information, see <<garbage collection,Garbage Collection>>.

[#dropping a sql table containing lob columns ]
== Dropping a SQL Table Containing LOB Columns

Drop works like any other drop table. All dependent tables are deleted. All files in hdfs (data and descriptor) files are also deleted.

For more information, see <<drop_table_statement,DROP TABLE Statement>> in http://trafodion.incubator.apache.org/docs/sql_reference/index.html[Trafodion SQL Reference Manual].

[#garbage collection]
== Garbage Collection

When a lob datafile for a column has reached a certain limit, defined by a CQD `LOB_GC_LIMIT_SIZE`, then a compaction is triggered automatically. +
The default GC Limit is 10GB and can be changed if needed.

The need for GC arises because when a delete operation or an update operation is performed, the old data black in the hdfs file will be left as unused. +
In the case of update, the old data will be left as unused and the new data will be written into a new section, so all these “holes” in the LOB data file are needlessly occupying space.

The LOB descriptor chunks file is looked at to see which ranges and offsets are actually used. The LOB datafile is temporarily saved. The compaction is done into a new tempfile. When the sections have all been copied into the tempfile, Trafodion will delete the existing lob data file and rename the tempfile.

Finally, the saved copy of the LOB datafile is dropped. The saved copy is there just in case you need to fall back to it in case of an error. Since this operation is triggered as part of an IUD operation, a definite slowdown will occur for that insert/update operation compared to subsequent inserts/updates.

Also, each lob column of a table can be compacted separately as needed. GC does not have to be done to all columns of the LOB table all at once.

NOTE: Currently the GC is done in the same transaction as the transaction being used for the insert or update operation. If any part of the GC fails, then the entire transaction is aborted.

When Trafodion has support for local transactions, Trafodion will do the GC in a separate transaction or in a separate process, so you can fail the GC with a warning and allow the insert to go through.

Setting the CQD `LOB_GC_LIMIT_SIZE` to 0 would prevent GC from occurring.

[#cleanup of a sql table containing lob columns]
== Cleanup of a SQL Table Containing LOB Columns

Cleanup works like cleanup of any other table. The command ensures all dependent SQL LOB tables and hdfs files are dropped ignoring errors if any.

For more information, see <<cleanup_statement,CLEANUP Statement>> in http://trafodion.incubator.apache.org/docs/sql_reference/index.html[Trafodion SQL Reference Manual].

[#showddl for lob]
== SHOWDDL for LOB

SHOWDDL for LOB with a special option will show all the dependent objects, names and details about the table.

[#syntax]
=== Syntax

```
SHOWDDL table-name, LOB DETAILS
```

[#examples]
=== Examples

* This example displays the details of the table t1ob1.

+

```
>>SHOWDDL tlob1, LOB DETAILS;
CREATE TABLE TRAFODION.SEABASE.TLOB1
  (
    C1                               INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , C2                               BLOB DEFAULT NULL NOT SERIALIZED
  , PRIMARY KEY (C1 ASC)
  )
;

LOB Metadata
============

CREATE TABLE TRAFODION.SEABASE.LOBMD_04239091936503896833
  (
    LOBNUM                           SMALLINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , STORAGETYPE                      SMALLINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , LOCATION                         VARCHAR(4096) CHARACTER SET ISO88591 COLLATE DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , PRIMARY KEY (LOBNUM ASC)
  )
;

************************************************
LobNum: 1

Data Storage
============

Location: /user/trafodion/lobs
DataFile: LOBP_04239091936503896833_0001

LOB Descriptor Handle
=====================

CREATE TABLE TRAFODION.SEABASE."LOBDescHandle_04239091936503896833_0001"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , NUMCHUNKS                        INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , LOBLEN                           LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  )
  STORE BY (DESCPARTNKEY ASC)
  SALT USING 8 PARTITIONS
;

LOB Descriptor Chunks
=====================

CREATE TABLE TRAFODION.SEABASE."LOBDescChunks_04239091936503896833_0001"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , DESCSYSKEY                       LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKNUM                         INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKLEN                         LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , INTPARAM                         LARGEINT DEFAULT NULL SERIALIZED
  , STRINGPARAM                      VARCHAR(400) CHARACTER SET ISO88591 COLLATE DEFAULT DEFAULT NULL SERIALIZED
  , PRIMARY KEY (DESCPARTNKEY ASC, DESCSYSKEY ASC, CHUNKNUM ASC)
  )
  SALT USING 8 PARTITIONS
;

--- SQL operation complete.
```

* This example displays the details of the table tlob130ext.

+

```
>>CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', primary key (c1));

--- SQL operation complete.

>>SHOWDDL tlob130ext, LOB DETAILS;

CREATE TABLE TRAFODION.SCH.TLOB130EXT
  (
    C1                               INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , C2                               BLOB DEFAULT NULL NOT SERIALIZED
  , C3                               CLOB DEFAULT NULL NOT SERIALIZED
  , C4                               BLOB DEFAULT NULL NOT SERIALIZED
  , PRIMARY KEY (C1 ASC)
  )
;

LOB Metadata
============

CREATE TABLE TRAFODION.SCH.LOBMD__04474425229029907479
  (
    LOBNUM                           SMALLINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , STORAGETYPE                      SMALLINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , LOCATION                         VARCHAR(4096) CHARACTER SET ISO88591 COLLATE DEFAULT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , PRIMARY KEY (LOBNUM ASC)
  )
;

************************************************
LobNum: 1

Data Storage
============

Location: /user/trafodion/lobs
DataFile: LOBP_04474425229029907479_0001

LOB Descriptor Handle
=====================

CREATE TABLE TRAFODION.SCH."LOBDescHandle__04474425229029907479_0001"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , NUMCHUNKS                        INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , LOBLEN                           LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  )
  STORE BY (DESCPARTNKEY ASC)
  SALT USING 8 PARTITIONS
;

LOB Descriptor Chunks
=====================

CREATE TABLE TRAFODION.SCH."LOBDescChunks__04474425229029907479_0001"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , DESCSYSKEY                       LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKNUM                         INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKLEN                         LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , DATAOFFSET                       LARGEINT DEFAULT NULL SERIALIZED
  , STRINGPARAM                      VARCHAR(400) CHARACTER SET ISO88591 COLLATE DEFAULT DEFAULT NULL SERIALIZED
  , PRIMARY KEY (DESCPARTNKEY ASC, DESCSYSKEY ASC, CHUNKNUM ASC)
  )
  SALT USING 8 PARTITIONS
;

************************************************
LobNum: 2

Data Storage
============

Location: /user/trafodion/lobs
DataFile: LOBP_04474425229029907479_0002

LOB Descriptor Handle
=====================

CREATE TABLE TRAFODION.SCH."LOBDescHandle__04474425229029907479_0002"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , NUMCHUNKS                        INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , LOBLEN                           LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  )
  STORE BY (DESCPARTNKEY ASC)
  SALT USING 8 PARTITIONS
;

LOB Descriptor Chunks
=====================

CREATE TABLE TRAFODION.SCH."LOBDescChunks__04474425229029907479_0002"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , DESCSYSKEY                       LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKNUM                         INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKLEN                         LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , DATAOFFSET                       LARGEINT DEFAULT NULL SERIALIZED
  , STRINGPARAM                      VARCHAR(400) CHARACTER SET ISO88591 COLLATE DEFAULT DEFAULT NULL SERIALIZED
  , PRIMARY KEY (DESCPARTNKEY ASC, DESCSYSKEY ASC, CHUNKNUM ASC)
  )
  SALT USING 8 PARTITIONS
;

************************************************
LobNum: 3

Data Storage
============

<External HDFS location>
<External HDFS file>

LOB Descriptor Handle
=====================

CREATE TABLE TRAFODION.SCH."LOBDescHandle__04474425229029907479_0003"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , NUMCHUNKS                        INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , LOBLEN                           LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  )
  STORE BY (DESCPARTNKEY ASC)
  SALT USING 8 PARTITIONS
;

LOB Descriptor Chunks
=====================

CREATE TABLE TRAFODION.SCH."LOBDescChunks__04474425229029907479_0003"
  (
    DESCPARTNKEY                     LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , DESCSYSKEY                       LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKNUM                         INT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , CHUNKLEN                         LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE SERIALIZED
  , DATAOFFSET                       LARGEINT DEFAULT NULL SERIALIZED
  , STRINGPARAM                      VARCHAR(400) CHARACTER SET ISO88591 COLLATE DEFAULT DEFAULT NULL SERIALIZED
  , PRIMARY KEY (DESCPARTNKEY ASC, DESCSYSKEY ASC, CHUNKNUM ASC)
  )
  SALT USING 8 PARTITIONS
;

--- SQL operation complete.
```

[#get statement for lob tables]
== Get Statement for LOB Tables

The GET Statement for lob tables gives column specific information on any lob columns in a table.

[#syntax]
=== Syntax

```
GET LOB STATS FOR TABLE table-name;
SELECT * FROM TABLE(LOB STATS table-name);
```

[#examples]
=== Examples

Suppose the following tables have been created:

```
>>CREATE TABLE tlob130gt (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB, PRIMARY KEY (c1));

--- SQL operation complete.

>>CREATE TABLE tlob130gt2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));

--- SQL operation complete.
```

* This Get Statement displays statistics for the table tlob130gt, the lob information is formatted for machine readability.

+

```
>>SELECT left(trim(catalog_name) || '.' || trim(schema_name) || '.' || trim(object_name) , 20), left(trim(column_name),5), left(trim(lob_location),5) , left(trim(lob_data_file),20),LOB_DATA_FILE_SIZE_EOD,LOB_DATA_FILE_SIZE_USED from table(lob stats(tlob130gt));

(EXPR)      (EXPR)     (EXPR)     (EXPR)     LOB_DATA_FILE_SIZE_EOD    LOB_DATA_FILE_SIZE_USED
------      ------     ------     ------     ----------------------    ----------------------

TRAFODION.SCH.TLOB13   C2   /user/trafodion/lobs   LOBP_044744252290302   15   10
TRAFODION.SCH.TLOB13   C3   /user/trafodion/lobs   LOBP_044744252290302   15   10
TRAFODION.SCH.TLOB13   C4   /user/trafodion/lobs   LOBP_044744252290302   45   30

--- 3 row(s) selected.
```

* This Get Statement displays statistics for the table tlob130gt2, the lob information is formatted for human readability.

+

```
>>GET LOB STATS FOR TABLE tlob130gt2;

Lob Information for table: "TRAFODION".LOB130.TLOB130GT2
=========================

ColumnName:  C2
Lob Location:  /user/trafodion/lobs
LOB Data File:  LOBP_07468755986685501835_0001
LOB EOD:  0
LOB Used Len:  0
ColumnName:  C3
Lob Location:  /user/trafodion/lobs
LOB Data File:  LOBP_07468755986685501835_0002
LOB EOD:  0
LOB UsedLen:  0
ColumnName:  C4
Lob Location:  External HDFS Location
LOB Data File:  External HDFS File
LOB EOD:  0
LOB Used Len:  0

--- SQL operation complete.
```

* This Get Statement displays statistics for the table tlob130gt2, the lob information is formatted for machine readability.

+

```
>>SELECT left(trim(catalog_name) || '.' || trim(schema_name) || '.' || trim(object_name), 20), left(trim(column_name),5), left(trim(lob_location),15), left(trim(lob_data_file),20),LOB_DATA_FILE_SIZE_EOD,LOB_DATA_FILE_SIZE_USED from table(lob stats(tlob130gt2));

(EXPR)      (EXPR)     (EXPR)     (EXPR)     LOB_DATA_FILE_SIZE_EOD    LOB_DATA_FILE_SIZE_USED
------      ------     ------     ------     ----------------------    ----------------------

TRAFODION.SCH.TLOB13   C2   /user/trafodion/lobs   LOBP_044744252290300   0   0
TRAFODION.SCH.TLOB13   C3   /user/trafodion/lobs   LOBP_044744252290300   0   0
TRAFODION.SCH.TLOB13   C4   External HDFS Location External HDFS File     0   0

--- 3 row(s) selected.
```


CORS text (#24) (#1219)

CORS text (#24) (#1219)

* first draft CORS text

* peer review

[id='environment-trial-proc']
= Deploying a trial environment

You can deploy a trial (evaluation) {PRODUCT} environment. It consists of {CENTRAL} for authoring or managing services and {KIE_SERVER} for test execution of services.

This environment does not include permanent storage. Assets that you create or modify in a trial environment are not saved.

This environment is intended for test and demonstration access. It supports cross-origin resource sharing (CORS). This means that {KIE_SERVER} endpoints can be accessed using a browser when other resources on the page are provided by other servers. {KIE_SERVER} endpoints are normally intended for REST calls, but browser access can be needed in some demonstration configurations.

The procedure is minimal. There are no required settings and all passwords are set to a single value (the default password is `RedHat`).

To deploy a single authoring environment, use the `{PRODUCT_INIT}{ENTERPRISE_VERSION_SHORT}-trial-ephemeral.yaml` template file. You can extract this file from the `{PRODUCT_FILE}-openshift-templates.zip` product deliverable file. You can download the file from the {PRODUCT_DOWNLOAD_LINK}[Software Downloads] page.

.Procedure

. Use one of the following methods to deploy the template:
* In the OpenShift Web UI, select *Add to Project -> Import YAML / JSON* and then select or paste the `{PRODUCT_INIT}{ENTERPRISE_VERSION_SHORT}-trial-ephemeral.yaml` file. In the *Add Template* window, ensure *Process the template* is selected and click *Continue*.
* To use the OpenShift command line console, prepare the following command line:
+
[subs="attributes,verbatim,macros"]
----
oc new-app -f <template-path>/{PRODUCT_INIT}{ENTERPRISE_VERSION_SHORT}-trial-ephemeral.yaml
----
+
In this command line, replace `<template-path>` with the path to the downloaded template file.
. Optionally, set any parameters as described in the template. However, a typical trial deployment does not require any parameters.
. Complete the creation of the environment, depending on the method that you are using:
* In the OpenShift Web UI, click *Create*.
* Complete and run the command line.

[id='environment-trial-proc']
= Deploying a trial environment

You can deploy a trial (evaluation) {PRODUCT} environment. It consists of {CENTRAL} for authoring or managing services and {KIE_SERVER} for test execution of services.

This environment does not include permanent storage. Assets that you create or modify in a trial environment are not saved.

The procedure is minimal. There are no required settings and all passwords are set to a single value (the default password is `RedHat`).

To deploy a single authoring environment, use the `{PRODUCT_INIT}{ENTERPRISE_VERSION_SHORT}-trial-ephemeral.yaml` template file. You can extract this file from the `{PRODUCT_FILE}-openshift-templates.zip` product deliverable file. You can download the file from the {PRODUCT_DOWNLOAD_LINK}[Software Downloads] page.

.Procedure

. Use one of the following methods to deploy the template:
* In the OpenShift Web UI, select *Add to Project -> Import YAML / JSON* and then select or paste the `{PRODUCT_INIT}{ENTERPRISE_VERSION_SHORT}-trial-ephemeral.yaml` file. In the *Add Template* window, ensure *Process the template* is selected and click *Continue*.
* To use the OpenShift command line console, prepare the following command line:
+
[subs="attributes,verbatim,macros"]
----
oc new-app -f <template-path>/{PRODUCT_INIT}{ENTERPRISE_VERSION_SHORT}-trial-ephemeral.yaml
----
+
In this command line, replace `<template-path>` with the path to the downloaded template file.
. Optionally, set any parameters as described in the template. However, a typical trial deployment does not require any parameters.
. Complete the creation of the environment, depending on the method that you are using:
* In the OpenShift Web UI, click *Create*.
* Complete and run the command line.


Update 2015-07-08-Preguntas-preguntas-preguntas.adoc

= Preguntas, preguntas, preguntas…
:hp-image: IMG_0606.jpg

We’re in our final days of living stateside.  Technically we’re homeless but the wonderful community of family and friends we have has allowed us to couch surf until we take off to PR.  Backyard BBQ’s and lazy summer days with family and friends only continue to confirm the fact that we are so fortunate to be able to have this opportunity to pick up and go.  We hope to have many more of these enjoyable days as our mainland connections come visit us on the island of enchantment.

We’ve been getting a lot of questions about the island, our intentions while there, and our feelings about going - considering how much the 2 of us have moved over the past 6 years this sort of move isn’t all that daunting to us, but I can see how it might be from a less nomadic person’s perspective.  Below I’ve attempted to answer some of the more common questions we’ve gotten over the last few weeks and some we’ve had to really think to answer ourselves.

 - *Do you need a passport to go?*  Only if you’re not a US Citizen or if you’re flying in from another country.  US Citizens or flights from within the US don’t have any customs requirements to visit Puerto Rico as it is a US Territory.

 - *Do you need to apply to live or work in Puerto Rico?*  Again, if you’re a US Citizen, nope!  Moving to PR is just like moving to any other state.  The difference for work is the income tax you pay. Because Puerto Rico doesn’t have official representation in the US Congress, Puerto Rican residents are not subject to federal income tax.  With that said, we’ll be consulting a CPA to make sure we file correctly next year since we’ll have lived half this year in the states and Alejandro will be working as an independent contractor.

 - *What is the closest mainland state to PR?*  Puerto Rico is about 1000 miles or a 2.5 hr flight from Miami.  The island is situated at the point where the Caribbean and Atlantic Ocean meet. The closest island nation is the Dominican Republic.

- *Can you get all the essentials and comforts of home?*  Like anywhere, Puerto Rico has metropolitan areas and rural areas.  With the island being only 100 miles long by 35 miles wide, all of these areas are within a day’s drive.  Our initial home, in Carolina, is within the San Juan metropolitan area and is very close to many shopping areas and stores including Bed Bath and Beyond, Walgreens, CVS, Dave’s Famous BBQ, Walmart, and Costco.  Since we are big proponents of supporting the local community we are very excited to try local varieties of fruits, vegetables, and meats.  I’ve already researched the farmers markets and fish mongers in the area.  If you’re interested here are some I plan on exploring initially -  http://www.puertoricodaytrips.com/rio-piedras-market/[Rio Piedras Farmer’s Market] and http://www.marisqueriaatlantica.com/marisqueria.php?id=2[Marqueseria Atlantica].  I’m sure once we’re on the ground we’ll find more places to explore…we’ll keep you up to date on what we find.  Where Alejandro’s family property is, in Rincon, is a little more rural.  Think beach/farm community with a small downtown and no traffic signals.  There we will have smaller grocers and stores but there is still a farmers market, an organic grocery store, a local hardware store and Mayaguez is a city of about 85,000 and about a 45 minute drive away (without traffic).

- *Is it safe?*  Like anywhere else, there are pockets of the metropolitan areas you should be vigilant in and be aware of your surroundings, but its not unlike being in any other metropolitan area.  Crime varies from city to city just like it does on the mainland and occurs with less frequency in the rural areas.

- *Do you have to know Spanish?*  Knowing Spanish is not a necessity to get around in Puerto Rico but learning it will help you tremendously if you plan on living there.  Many Puerto Ricans speak english to one degree or another.  I’m using Duolingo to improve upon my Spanish and luckily Alejandro is fluent.

 - *Are you anxious about going?*  While I’m anxious to get *TO* Puerto Rico and not be living out of my backpack I don’t have any anxiety about going. I’m excited for the new adventure that awaits us and I’m truly looking forward to soaking up the sun and seeing what path we will end up on. Excitement yes, anxiety no.

 - *What are you going to do?*  This is a question that really has no answer.  I plan on exploring, learning the culture, the environment, and the people.  I’ve been doing a lot of research on the various environmental programs happening throughout the island and hope to be able to get involved with some of these groups.  We will also be assessing the work that needs to be done on the property, meeting with lawyers, submitting necessary paperwork for the property and enjoying the occasional coconut.

 - *Can you find sushi?* Yes! Don’t worry, we will research and report back and will have recommendations for those that visit.

- *Where is all your stuff?*  We’ve managed to consolidate or sell most of our stuff.  Many of our friends have been the lucky recipients of kitchenware, plants, baking items, furniture, and even clothes.  The things we did keep (nicer kitchen items, mementos, records, a few pieces of furniture) have been packed up and are in my mom’s garage.  Depending on where we end up or how long we stay here will determine when we reunite with these possessions. For now, we’re taking 5 suitcases and 2 backpacks that contain mostly clothes and essentials.  Even though we move pretty often, we’d been in our apartment a year and half and its amazing how much you collect in a short period of time.  Even just the amount of change we found throughout the house was incredible!  Note - most banks won’t take loose change and some won’t even take rolled change…we had to settle for Coinstar taking their 10%…

.Our Puerto Rico Baggage
image:IMG_0607.jpg[Puerto Rico Baggage]

 - *How will you get around?*  This is mostly a question we get from our San Francisco friends.  We haven’t owned a car in years since we never needed one in the city.  We’ll be renting a car using our airline mile points for about a week and we have already searched online for some possible locations to buy a car when we get to PR.

 - *Will your phone work there?*  Yep!  Cell phone use in Puerto Rico is the same as if you’re in the mainland.  Our phone numbers will remain the same and we hope to continue getting phone calls and text messages from everyone!

- *What’s the weather like?*  Puerto Rico has some of the most consistent temperatures year round with highs staying in the upper 80’s and lows in the mid 70’s.  Hurricane season ranges from August through October with a major hurricane gracing the island around every 7 years.

 - *What clothes did you pack?* This was a pretty difficult question for me to answer while I was going through my clothes.  Being used to summers in San Francisco its was very difficult for me to part with my summer sweaters and jackets.  I gave away a lot, packed a few in storage, and brought a couple that I just couldn’t part with.  Taking a sweater everywhere I go is going to be hard to give up initially but I’m sure I’ll acclimate to island temperature very quickly. For the most part my suitcases are filled with shorts, t-shirts, tank tops, skirts, and dresses - pretty much the items I only got to wear on occasion in San Francisco.

- *What has been the hardest part of this move?*  This is a loaded question.  Leaving our friends and family that we see on a regular basis will definitely be the most difficult. In terms of practical changes we’ve had to make I could say deciding which clothes and household items to keep and what to give away; I could say all the trips to Goodwill when our things wouldn’t sell on Craigslist…but the most difficult moving task we’ve faced is changing our address!  Addresses in Puerto Rico are often very different than here in the states, with your address being a kilometer distance on a particular road in a barrio within a city.   Even putting in a request for a change of address with USPS proved challenging.  We’ve had to call pretty much every vendor we have bills with to change our address and even had to close a few accounts that won’t accept an address outside the 50 states.  Some changes will have to wait until we have a PO Box in Puerto Rico that we can use as an address since our kilometer number doesn’t work for some businesses.  For those of you wanting to send mail, we’ll update you with our PO Box information once we have it.

- *Will you miss San Francisco?*  Absolutely!! SF has been my home for many years and will always be one of my favorite cities.  I have wonderful friends and great memories of the bay area and will always consider it a part of me.  We might live there again, we might not, but we will always return for visits.

I fully expect questions from ourselves, from family & friends to never end during this adventure and indeed I welcome them.  If you are curious about anything related to Puerto Rico or our move in general don’t hesitate to ask us in the comments and we’ll try to answer as best we can.  Hopefully as we discover Puerto Rico we’ll be able to answer a lot more questions and we’ll definitely keep you all informed of what we learn!

= Preguntas, preguntas, preguntas…
:hp image:IMG_0606.jpg

We’re in our final days of living stateside.  Technically we’re homeless but the wonderful community of family and friends we have has allowed us to couch surf until we take off to PR.  Backyard BBQ’s and lazy summer days with family and friends only continue to confirm the fact that we are so fortunate to be able to have this opportunity to pick up and go.  We hope to have many more of these enjoyable days as our mainland connections come visit us on the island of enchantment.

We’ve been getting a lot of questions about the island, our intentions while there, and our feelings about going - considering how much the 2 of us have moved over the past 6 years this sort of move isn’t all that daunting to us, but I can see how it might be from a less nomadic person’s perspective.  Below I’ve attempted to answer some of the more common questions we’ve gotten over the last few weeks and some we’ve had to really think to answer ourselves.

 - *Do you need a passport to go?*  Only if you’re not a US Citizen or if you’re flying in from another country.  US Citizens or flights from within the US don’t have any customs requirements to visit Puerto Rico as it is a US Territory.

 - *Do you need to apply to live or work in Puerto Rico?*  Again, if you’re a US Citizen, nope!  Moving to PR is just like moving to any other state.  The difference for work is the income tax you pay. Because Puerto Rico doesn’t have official representation in the US Congress, Puerto Rican residents are not subject to federal income tax.  With that said, we’ll be consulting a CPA to make sure we file correctly next year since we’ll have lived half this year in the states and Alejandro will be working as an independent contractor.

 - *What is the closest mainland state to PR?*  Puerto Rico is about 1000 miles or a 2.5 hr flight from Miami.  The island is situated at the point where the Caribbean and Atlantic Ocean meet. The closest island nation is the Dominican Republic.

- *Can you get all the essentials and comforts of home?*  Like anywhere, Puerto Rico has metropolitan areas and rural areas.  With the island being only 100 miles long by 35 miles wide, all of these areas are within a day’s drive.  Our initial home, in Carolina, is within the San Juan metropolitan area and is very close to many shopping areas and stores including Bed Bath and Beyond, Walgreens, CVS, Dave’s Famous BBQ, Walmart, and Costco.  Since we are big proponents of supporting the local community we are very excited to try local varieties of fruits, vegetables, and meats.  I’ve already researched the farmers markets and fish mongers in the area.  If you’re interested here are some I plan on exploring initially -  http://www.puertoricodaytrips.com/rio-piedras-market/[Rio Piedras Farmer’s Market] and http://www.marisqueriaatlantica.com/marisqueria.php?id=2[Marqueseria Atlantica].  I’m sure once we’re on the ground we’ll find more places to explore…we’ll keep you up to date on what we find.  Where Alejandro’s family property is, in Rincon, is a little more rural.  Think beach/farm community with a small downtown and no traffic signals.  There we will have smaller grocers and stores but there is still a farmers market, an organic grocery store, a local hardware store and Mayaguez is a city of about 85,000 and about a 45 minute drive away (without traffic).

- *Is it safe?*  Like anywhere else, there are pockets of the metropolitan areas you should be vigilant in and be aware of your surroundings, but its not unlike being in any other metropolitan area.  Crime varies from city to city just like it does on the mainland and occurs with less frequency in the rural areas.

- *Do you have to know Spanish?*  Knowing Spanish is not a necessity to get around in Puerto Rico but learning it will help you tremendously if you plan on living there.  Many Puerto Ricans speak english to one degree or another.  I’m using Duolingo to improve upon my Spanish and luckily Alejandro is fluent.

 - *Are you anxious about going?*  While I’m anxious to get *TO* Puerto Rico and not be living out of my backpack I don’t have any anxiety about going. I’m excited for the new adventure that awaits us and I’m truly looking forward to soaking up the sun and seeing what path we will end up on. Excitement yes, anxiety no.

 - *What are you going to do?*  This is a question that really has no answer.  I plan on exploring, learning the culture, the environment, and the people.  I’ve been doing a lot of research on the various environmental programs happening throughout the island and hope to be able to get involved with some of these groups.  We will also be assessing the work that needs to be done on the property, meeting with lawyers, submitting necessary paperwork for the property and enjoying the occasional coconut.

 - *Can you find sushi?* Yes! Don’t worry, we will research and report back and will have recommendations for those that visit.

- *Where is all your stuff?*  We’ve managed to consolidate or sell most of our stuff.  Many of our friends have been the lucky recipients of kitchenware, plants, baking items, furniture, and even clothes.  The things we did keep (nicer kitchen items, mementos, records, a few pieces of furniture) have been packed up and are in my mom’s garage.  Depending on where we end up or how long we stay here will determine when we reunite with these possessions. For now, we’re taking 5 suitcases and 2 backpacks that contain mostly clothes and essentials.  Even though we move pretty often, we’d been in our apartment a year and half and its amazing how much you collect in a short period of time.  Even just the amount of change we found throughout the house was incredible!  Note - most banks won’t take loose change and some won’t even take rolled change…we had to settle for Coinstar taking their 10%…

.Our Puerto Rico Baggage
image:IMG_0607.jpg[Puerto Rico Baggage]

 - *How will you get around?*  This is mostly a question we get from our San Francisco friends.  We haven’t owned a car in years since we never needed one in the city.  We’ll be renting a car using our airline mile points for about a week and we have already searched online for some possible locations to buy a car when we get to PR.

 - *Will your phone work there?*  Yep!  Cell phone use in Puerto Rico is the same as if you’re in the mainland.  Our phone numbers will remain the same and we hope to continue getting phone calls and text messages from everyone!

- *What’s the weather like?*  Puerto Rico has some of the most consistent temperatures year round with highs staying in the upper 80’s and lows in the mid 70’s.  Hurricane season ranges from August through October with a major hurricane gracing the island around every 7 years.

 - *What clothes did you pack?* This was a pretty difficult question for me to answer while I was going through my clothes.  Being used to summers in San Francisco its was very difficult for me to part with my summer sweaters and jackets.  I gave away a lot, packed a few in storage, and brought a couple that I just couldn’t part with.  Taking a sweater everywhere I go is going to be hard to give up initially but I’m sure I’ll acclimate to island temperature very quickly. For the most part my suitcases are filled with shorts, t-shirts, tank tops, skirts, and dresses - pretty much the items I only got to wear on occasion in San Francisco.

- *What has been the hardest part of this move?*  This is a loaded question.  Leaving our friends and family that we see on a regular basis will definitely be the most difficult. In terms of practical changes we’ve had to make I could say deciding which clothes and household items to keep and what to give away; I could say all the trips to Goodwill when our things wouldn’t sell on Craigslist…but the most difficult moving task we’ve faced is changing our address!  Addresses in Puerto Rico are often very different than here in the states, with your address being a kilometer distance on a particular road in a barrio within a city.   Even putting in a request for a change of address with USPS proved challenging.  We’ve had to call pretty much every vendor we have bills with to change our address and even had to close a few accounts that won’t accept an address outside the 50 states.  Some changes will have to wait until we have a PO Box in Puerto Rico that we can use as an address since our kilometer number doesn’t work for some businesses.  For those of you wanting to send mail, we’ll update you with our PO Box information once we have it.

- *Will you miss San Francisco?*  Absolutely!! SF has been my home for many years and will always be one of my favorite cities.  I have wonderful friends and great memories of the bay area and will always consider it a part of me.  We might live there again, we might not, but we will always return for visits.

I fully expect questions from ourselves, from family & friends to never end during this adventure and indeed I welcome them.  If you are curious about anything related to Puerto Rico or our move in general don’t hesitate to ask us in the comments and we’ll try to answer as best we can.  Hopefully as we discover Puerto Rico we’ll be able to answer a lot more questions and we’ll definitely keep you all informed of what we learn!