| PostgreSQL 9.5.19 Documentation | |||
|---|---|---|---|
| Prev | Up | Chapter 54. Writing A Foreign Data Wrapper | Next |
The FDW handler function returns a palloc'd FdwRoutine struct containing pointers to the callback functions described below. The scan-related functions are required, the rest are optional.
The FdwRoutine struct type is declared in src/include/foreign/fdwapi.h, which see for additional details.
void GetForeignRelSize (PlannerInfo *root, RelOptInfo *baserel, Oid foreigntableid);
Obtain relation size estimates for a foreign table. This is called at the beginning of planning for a query that scans a foreign table. root is the planner's global information about the query; baserel is the planner's information about this table; and foreigntableid is the pg_class OID of the foreign table. (foreigntableid could be obtained from the planner data structures, but it's passed explicitly to save effort.)
This function should update baserel->rows to be the expected number of rows returned by the table scan, after accounting for the filtering done by the restriction quals. The initial value of baserel->rows is just a constant default estimate, which should be replaced if at all possible. The function may also choose to update baserel->width if it can compute a better estimate of the average result row width.
See Section 54.4 for additional information.
void GetForeignPaths (PlannerInfo *root, RelOptInfo *baserel, Oid foreigntableid);
Create possible access paths for a scan on a foreign table.
This is called during query planning.
The parameters are the same as for GetForeignRelSize
,
which has already been called.
This function must generate at least one access path
(ForeignPath node) for a scan on the foreign table and
must call add_path
to add each such path to
baserel->pathlist. It's recommended to use
create_foreignscan_path
to build the
ForeignPath nodes. The function can generate multiple
access paths, e.g., a path which has valid pathkeys to
represent a pre-sorted result. Each access path must contain cost
estimates, and can contain any FDW-private information that is needed to
identify the specific scan method intended.
See Section 54.4 for additional information.
ForeignScan * GetForeignPlan (PlannerInfo *root, RelOptInfo *baserel, Oid foreigntableid, ForeignPath *best_path, List *tlist, List *scan_clauses, Plan *outer_plan);
Create a ForeignScan plan node from the selected foreign
access path. This is called at the end of query planning.
The parameters are as for GetForeignRelSize
, plus
the selected ForeignPath (previously produced by
GetForeignPaths
or GetForeignJoinPaths
),
the target list to be emitted by the plan node,
the restriction clauses to be enforced by the plan node,
and the outer subplan of the ForeignScan,
which is used for rechecks performed by RecheckForeignScan
.
(If the path is for a join rather than a base
relation, foreigntableid is InvalidOid.)
This function must create and return a ForeignScan plan
node; it's recommended to use make_foreignscan
to build the
ForeignScan node.
See Section 54.4 for additional information.
void BeginForeignScan (ForeignScanState *node, int eflags);
Begin executing a foreign scan. This is called during executor startup.
It should perform any initialization needed before the scan can start,
but not start executing the actual scan (that should be done upon the
first call to IterateForeignScan
).
The ForeignScanState node has already been created, but
its fdw_state field is still NULL. Information about
the table to scan is accessible through the
ForeignScanState node (in particular, from the underlying
ForeignScan plan node, which contains any FDW-private
information provided by GetForeignPlan
).
eflags contains flag bits describing the executor's
operating mode for this plan node.
Note that when (eflags & EXEC_FLAG_EXPLAIN_ONLY) is
true, this function should not perform any externally-visible actions;
it should only do the minimum required to make the node state valid
for ExplainForeignScan
and EndForeignScan
.
TupleTableSlot * IterateForeignScan (ForeignScanState *node);
Fetch one row from the foreign source, returning it in a tuple table slot
(the node's ScanTupleSlot should be used for this
purpose). Return NULL if no more rows are available. The tuple table
slot infrastructure allows either a physical or virtual tuple to be
returned; in most cases the latter choice is preferable from a
performance standpoint. Note that this is called in a short-lived memory
context that will be reset between invocations. Create a memory context
in BeginForeignScan
if you need longer-lived storage, or use
the es_query_cxt of the node's EState.
The rows returned must match the fdw_scan_tlist target list if one was supplied, otherwise they must match the row type of the foreign table being scanned. If you choose to optimize away fetching columns that are not needed, you should insert nulls in those column positions, or else generate a fdw_scan_tlist list with those columns omitted.
Note that PostgreSQL's executor doesn't care whether the rows returned violate any constraints that were defined on the foreign table — but the planner does care, and may optimize queries incorrectly if there are rows visible in the foreign table that do not satisfy a declared constraint. If a constraint is violated when the user has declared that the constraint should hold true, it may be appropriate to raise an error (just as you would need to do in the case of a data type mismatch).
void ReScanForeignScan (ForeignScanState *node);
Restart the scan from the beginning. Note that any parameters the scan depends on may have changed value, so the new scan does not necessarily return exactly the same rows.
void EndForeignScan (ForeignScanState *node);
End the scan and release resources. It is normally not important to release palloc'd memory, but for example open files and connections to remote servers should be cleaned up.
If an FDW supports performing foreign joins remotely (rather than by fetching both tables' data and doing the join locally), it should provide this callback function:
void GetForeignJoinPaths (PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, JoinPathExtraData *extra);
Create possible access paths for a join of two (or more) foreign tables
that all belong to the same foreign server. This optional
function is called during query planning. As
with GetForeignPaths
, this function should
generate ForeignPath path(s) for the
supplied joinrel, and call add_path
to add these
paths to the set of paths considered for the join. But unlike
GetForeignPaths
, it is not necessary that this function
succeed in creating at least one path, since paths involving local
joining are always possible.
Note that this function will be invoked repeatedly for the same join relation, with different combinations of inner and outer relations; it is the responsibility of the FDW to minimize duplicated work.
If a ForeignPath path is chosen for the join, it will represent the entire join process; paths generated for the component tables and subsidiary joins will not be used. Subsequent processing of the join path proceeds much as it does for a path scanning a single foreign table. One difference is that the scanrelid of the resulting ForeignScan plan node should be set to zero, since there is no single relation that it represents; instead, the fs_relids field of the ForeignScan node represents the set of relations that were joined. (The latter field is set up automatically by the core planner code, and need not be filled by the FDW.) Another difference is that, because the column list for a remote join cannot be found from the system catalogs, the FDW must fill fdw_scan_tlist with an appropriate list of TargetEntry nodes, representing the set of columns it will supply at run time in the tuples it returns.
See Section 54.4 for additional information.
If an FDW supports writable foreign tables, it should provide some or all of the following callback functions depending on the needs and capabilities of the FDW:
void AddForeignUpdateTargets (Query *parsetree, RangeTblEntry *target_rte, Relation target_relation);
UPDATE and DELETE operations are performed against rows previously fetched by the table-scanning functions. The FDW may need extra information, such as a row ID or the values of primary-key columns, to ensure that it can identify the exact row to update or delete. To support that, this function can add extra hidden, or "junk", target columns to the list of columns that are to be retrieved from the foreign table during an UPDATE or DELETE.
To do that, add TargetEntry items to
parsetree->targetList, containing expressions for the
extra values to be fetched. Each such entry must be marked
resjunk = true, and must have a distinct
resname that will identify it at execution time.
Avoid using names matching ctidN,
wholerow, or
wholerowN, as the core system can
generate junk columns of these names.
If the extra expressions are more complex than simple Vars, they
must be run through eval_const_expressions
before adding them to the targetlist.
Although this function is called during planning, the information provided is a bit different from that available to other planning routines. parsetree is the parse tree for the UPDATE or DELETE command, while target_rte and target_relation describe the target foreign table.
If the AddForeignUpdateTargets
pointer is set to
NULL, no extra target expressions are added.
(This will make it impossible to implement DELETE
operations, though UPDATE may still be feasible if the FDW
relies on an unchanging primary key to identify rows.)
List * PlanForeignModify (PlannerInfo *root, ModifyTable *plan, Index resultRelation, int subplan_index);
Perform any additional planning actions needed for an insert, update, or
delete on a foreign table. This function generates the FDW-private
information that will be attached to the ModifyTable plan
node that performs the update action. This private information must
have the form of a List, and will be delivered to
BeginForeignModify
during the execution stage.
root is the planner's global information about the query. plan is the ModifyTable plan node, which is complete except for the fdwPrivLists field. resultRelation identifies the target foreign table by its range table index. subplan_index identifies which target of the ModifyTable plan node this is, counting from zero; use this if you want to index into plan->plans or other substructure of the plan node.
See Section 54.4 for additional information.
If the PlanForeignModify
pointer is set to
NULL, no additional plan-time actions are taken, and the
fdw_private list delivered to
BeginForeignModify
will be NIL.
void BeginForeignModify (ModifyTableState *mtstate, ResultRelInfo *rinfo, List *fdw_private, int subplan_index, int eflags);
Begin executing a foreign table modification operation. This routine is
called during executor startup. It should perform any initialization
needed prior to the actual table modifications. Subsequently,
ExecForeignInsert
, ExecForeignUpdate
or
ExecForeignDelete
will be called for each tuple to be
inserted, updated, or deleted.
mtstate is the overall state of the
ModifyTable plan node being executed; global data about
the plan and execution state is available via this structure.
rinfo is the ResultRelInfo struct describing
the target foreign table. (The ri_FdwState field of
ResultRelInfo is available for the FDW to store any
private state it needs for this operation.)
fdw_private contains the private data generated by
PlanForeignModify
, if any.
subplan_index identifies which target of
the ModifyTable plan node this is.
eflags contains flag bits describing the executor's
operating mode for this plan node.
Note that when (eflags & EXEC_FLAG_EXPLAIN_ONLY) is
true, this function should not perform any externally-visible actions;
it should only do the minimum required to make the node state valid
for ExplainForeignModify
and EndForeignModify
.
If the BeginForeignModify
pointer is set to
NULL, no action is taken during executor startup.
TupleTableSlot * ExecForeignInsert (EState *estate, ResultRelInfo *rinfo, TupleTableSlot *slot, TupleTableSlot *planSlot);
Insert one tuple into the foreign table. estate is global execution state for the query. rinfo is the ResultRelInfo struct describing the target foreign table. slot contains the tuple to be inserted; it will match the row-type definition of the foreign table. planSlot contains the tuple that was generated by the ModifyTable plan node's subplan; it differs from slot in possibly containing additional "junk" columns. (The planSlot is typically of little interest for INSERT cases, but is provided for completeness.)
The return value is either a slot containing the data that was actually inserted (this might differ from the data supplied, for example as a result of trigger actions), or NULL if no row was actually inserted (again, typically as a result of triggers). The passed-in slot can be re-used for this purpose.
The data in the returned slot is used only if the INSERT query has a RETURNING clause or the foreign table has an AFTER ROW trigger. Triggers require all columns, but the FDW could choose to optimize away returning some or all columns depending on the contents of the RETURNING clause. Regardless, some slot must be returned to indicate success, or the query's reported row count will be wrong.
If the ExecForeignInsert
pointer is set to
NULL, attempts to insert into the foreign table will fail
with an error message.
TupleTableSlot * ExecForeignUpdate (EState *estate, ResultRelInfo *rinfo, TupleTableSlot *slot, TupleTableSlot *planSlot);
Update one tuple in the foreign table.
estate is global execution state for the query.
rinfo is the ResultRelInfo struct describing
the target foreign table.
slot contains the new data for the tuple; it will match the
row-type definition of the foreign table.
planSlot contains the tuple that was generated by the
ModifyTable plan node's subplan; it differs from
slot in possibly containing additional "junk"
columns. In particular, any junk columns that were requested by
AddForeignUpdateTargets
will be available from this slot.
The return value is either a slot containing the row as it was actually updated (this might differ from the data supplied, for example as a result of trigger actions), or NULL if no row was actually updated (again, typically as a result of triggers). The passed-in slot can be re-used for this purpose.
The data in the returned slot is used only if the UPDATE query has a RETURNING clause or the foreign table has an AFTER ROW trigger. Triggers require all columns, but the FDW could choose to optimize away returning some or all columns depending on the contents of the RETURNING clause. Regardless, some slot must be returned to indicate success, or the query's reported row count will be wrong.
If the ExecForeignUpdate
pointer is set to
NULL, attempts to update the foreign table will fail
with an error message.
TupleTableSlot * ExecForeignDelete (EState *estate, ResultRelInfo *rinfo, TupleTableSlot *slot, TupleTableSlot *planSlot);
Delete one tuple from the foreign table.
estate is global execution state for the query.
rinfo is the ResultRelInfo struct describing
the target foreign table.
slot contains nothing useful upon call, but can be used to
hold the returned tuple.
planSlot contains the tuple that was generated by the
ModifyTable plan node's subplan; in particular, it will
carry any junk columns that were requested by
AddForeignUpdateTargets
. The junk column(s) must be used
to identify the tuple to be deleted.
The return value is either a slot containing the row that was deleted, or NULL if no row was deleted (typically as a result of triggers). The passed-in slot can be used to hold the tuple to be returned.
The data in the returned slot is used only if the DELETE query has a RETURNING clause or the foreign table has an AFTER ROW trigger. Triggers require all columns, but the FDW could choose to optimize away returning some or all columns depending on the contents of the RETURNING clause. Regardless, some slot must be returned to indicate success, or the query's reported row count will be wrong.
If the ExecForeignDelete
pointer is set to
NULL, attempts to delete from the foreign table will fail
with an error message.
void EndForeignModify (EState *estate, ResultRelInfo *rinfo);
End the table update and release resources. It is normally not important to release palloc'd memory, but for example open files and connections to remote servers should be cleaned up.
If the EndForeignModify
pointer is set to
NULL, no action is taken during executor shutdown.
int IsForeignRelUpdatable (Relation rel);
Report which update operations the specified foreign table supports. The return value should be a bit mask of rule event numbers indicating which operations are supported by the foreign table, using the CmdType enumeration; that is, (1 << CMD_UPDATE) = 4 for UPDATE, (1 << CMD_INSERT) = 8 for INSERT, and (1 << CMD_DELETE) = 16 for DELETE.
If the IsForeignRelUpdatable
pointer is set to
NULL, foreign tables are assumed to be insertable, updatable,
or deletable if the FDW provides ExecForeignInsert
,
ExecForeignUpdate
, or ExecForeignDelete
respectively. This function is only needed if the FDW supports some
tables that are updatable and some that are not. (Even then, it's
permissible to throw an error in the execution routine instead of
checking in this function. However, this function is used to determine
updatability for display in the information_schema views.)
If an FDW wishes to support late row locking (as described in Section 54.5), it must provide the following callback functions:
RowMarkType GetForeignRowMarkType (RangeTblEntry *rte, LockClauseStrength strength);
Report which row-marking option to use for a foreign table. rte is the RangeTblEntry node for the table and strength describes the lock strength requested by the relevant FOR UPDATE/SHARE clause, if any. The result must be a member of the RowMarkType enum type.
This function is called during query planning for each foreign table that appears in an UPDATE, DELETE, or SELECT FOR UPDATE/SHARE query and is not the target of UPDATE or DELETE.
If the GetForeignRowMarkType
pointer is set to
NULL, the ROW_MARK_COPY option is always used.
(This implies that RefetchForeignRow
will never be called,
so it need not be provided either.)
See Section 54.5 for more information.
HeapTuple RefetchForeignRow (EState *estate, ExecRowMark *erm, Datum rowid, bool *updated);
Re-fetch one tuple from the foreign table, after locking it if required. estate is global execution state for the query. erm is the ExecRowMark struct describing the target foreign table and the row lock type (if any) to acquire. rowid identifies the tuple to be fetched. updated is an output parameter.
This function should return a palloc'ed copy of the fetched tuple,
or NULL if the row lock couldn't be obtained. The row lock
type to acquire is defined by erm->markType, which is the
value previously returned by GetForeignRowMarkType
.
(ROW_MARK_REFERENCE means to just re-fetch the tuple without
acquiring any lock, and ROW_MARK_COPY will never be seen by
this routine.)
In addition, *updated should be set to true if what was fetched was an updated version of the tuple rather than the same version previously obtained. (If the FDW cannot be sure about this, always returning true is recommended.)
Note that by default, failure to acquire a row lock should result in raising an error; a NULL return is only appropriate if the SKIP LOCKED option is specified by erm->waitPolicy.
The rowid is the ctid value previously read for the row to be re-fetched. Although the rowid value is passed as a Datum, it can currently only be a tid. The function API is chosen in hopes that it may be possible to allow other data types for row IDs in future.
If the RefetchForeignRow
pointer is set to
NULL, attempts to re-fetch rows will fail
with an error message.
See Section 54.5 for more information.
bool RecheckForeignScan (ForeignScanState *node, TupleTableSlot *slot);
Recheck that a previously-returned tuple still matches the relevant scan and join qualifiers, and possibly provide a modified version of the tuple. For foreign data wrappers which do not perform join pushdown, it will typically be more convenient to set this to NULL and instead set fdw_recheck_quals appropriately. When outer joins are pushed down, however, it isn't sufficient to reapply the checks relevant to all the base tables to the result tuple, even if all needed attributes are present, because failure to match some qualifier might result in some attributes going to NULL, rather than in no tuple being returned. RecheckForeignScan can recheck qualifiers and return true if they are still satisfied and false otherwise, but it can also store a replacement tuple into the supplied slot.
To implement join pushdown, a foreign data wrapper will typically construct an alternative local join plan which is used only for rechecks; this will become the outer subplan of the ForeignScan. When a recheck is required, this subplan can be executed and the resulting tuple can be stored in the slot. This plan need not be efficient since no base table will return more than one row; for example, it may implement all joins as nested loops.
void ExplainForeignScan (ForeignScanState *node, ExplainState *es);
Print additional EXPLAIN output for a foreign table scan.
This function can call ExplainPropertyText
and
related functions to add fields to the EXPLAIN output.
The flag fields in es can be used to determine what to
print, and the state of the ForeignScanState node
can be inspected to provide run-time statistics in the EXPLAIN
ANALYZE case.
If the ExplainForeignScan
pointer is set to
NULL, no additional information is printed during
EXPLAIN.
void ExplainForeignModify (ModifyTableState *mtstate, ResultRelInfo *rinfo, List *fdw_private, int subplan_index, struct ExplainState *es);
Print additional EXPLAIN output for a foreign table update.
This function can call ExplainPropertyText
and
related functions to add fields to the EXPLAIN output.
The flag fields in es can be used to determine what to
print, and the state of the ModifyTableState node
can be inspected to provide run-time statistics in the EXPLAIN
ANALYZE case. The first four arguments are the same as for
BeginForeignModify
.
If the ExplainForeignModify
pointer is set to
NULL, no additional information is printed during
EXPLAIN.
bool AnalyzeForeignTable (Relation relation, AcquireSampleRowsFunc *func, BlockNumber *totalpages);
This function is called when ANALYZE is executed on a foreign table. If the FDW can collect statistics for this foreign table, it should return true, and provide a pointer to a function that will collect sample rows from the table in func, plus the estimated size of the table in pages in totalpages. Otherwise, return false.
If the FDW does not support collecting statistics for any tables, the
AnalyzeForeignTable
pointer can be set to NULL.
If provided, the sample collection function must have the signature
int AcquireSampleRowsFunc (Relation relation, int elevel, HeapTuple *rows, int targrows, double *totalrows, double *totaldeadrows);
A random sample of up to targrows rows should be collected from the table and stored into the caller-provided rows array. The actual number of rows collected must be returned. In addition, store estimates of the total numbers of live and dead rows in the table into the output parameters totalrows and totaldeadrows. (Set totaldeadrows to zero if the FDW does not have any concept of dead rows.)
List * ImportForeignSchema (ImportForeignSchemaStmt *stmt, Oid serverOid);
Obtain a list of foreign table creation commands. This function is called when executing IMPORT FOREIGN SCHEMA, and is passed the parse tree for that statement, as well as the OID of the foreign server to use. It should return a list of C strings, each of which must contain a CREATE FOREIGN TABLE command. These strings will be parsed and executed by the core server.
Within the ImportForeignSchemaStmt struct, remote_schema is the name of the remote schema from which tables are to be imported. list_type identifies how to filter table names: FDW_IMPORT_SCHEMA_ALL means that all tables in the remote schema should be imported (in this case table_list is empty), FDW_IMPORT_SCHEMA_LIMIT_TO means to include only tables listed in table_list, and FDW_IMPORT_SCHEMA_EXCEPT means to exclude the tables listed in table_list. options is a list of options used for the import process. The meanings of the options are up to the FDW. For example, an FDW could use an option to define whether the NOT NULL attributes of columns should be imported. These options need not have anything to do with those supported by the FDW as database object options.
The FDW may ignore the local_schema field of the ImportForeignSchemaStmt, because the core server will automatically insert that name into the parsed CREATE FOREIGN TABLE commands.
The FDW does not have to concern itself with implementing the filtering
specified by list_type and table_list,
either, as the core server will automatically skip any returned commands
for tables excluded according to those options. However, it's often
useful to avoid the work of creating commands for excluded tables in the
first place. The function IsImportableForeignTable()
may be
useful to test whether a given foreign-table name will pass the filter.
If the FDW does not support importing table definitions, the
ImportForeignSchema
pointer can be set to NULL.


This is the story of Sabina, an average woman from Kazakhstan (Sabina is a very popular Kazakh female name). Through her story we will get insight about the current mariatal situation in Kazakhstan. Why this is important? I believe, family is a building block of any society. Healthy families lead to healthy society. Unfortunately, the mariatal situation in Kazakhstan is getting worse every year. The purpose of this visualization is to inform people of Kazakhstan about this situation and let them know about the common reasons of the break-ups.
"Sabina was 18 when she said "Yes" to her loved one."
In this section, I will talk about the age statistics. I will tell the reader about the average age of women when they get married.
In this section, I will talk abou statistics such as: after how many years of the marriage an average women becomes a mom, and how many kids an average family has.
Here I will talk about:
-on average, after how many years Sabina has a chances to break-up?
-let the user change the location of Sabina, and show that different regions in Kazakshtan have different statistics
-correlate with the previous section, and try to visualize how many kids are affected
Here I will talk about the common problems and diffuculties Sabina had to go through and overcome in order to save the family.
I might include links to books and articles related to "Happy Family" life.
Files:[TBA]Similar projects:Divorce by OccupationDivorce Rates for Different Groups Effects of Divorce on Children |
Data sources:Government websites:Annual Socio-Economic Report by Ministry of Economy of KZNews articles:Article #1Article #2 Article #3 Article #4 Article #5 Article #6 Article #7 |
Code sources:Timeline template |


Namespace DouglasDwyer.PowerSerializer
Classes
ClassLimitedTypeResolver
Instances of FinalizerLimitedGuidTypeResolver allow for serialization of all the classes specified in the constructor, as well as all structs loaded at resolver construction. They utilize the MD5 hash of a type's name in order to resolve it across serialization boundaries.
FinalizerLimitedGuidTypeResolver
Instances of FinalizerLimitedGuidTypeResolver allow for serialization of any type loaded in the current System.AppDomain upon resolver construction, provided that the type doesn't have a finalizer. They utilize the MD5 hash of a type's name in order to resolve it across serialization boundaries.
FinalizerLimitedNameTypeResolver
Instances of FinalizerLimitedNameTypeResolver allow for serialization of any type loaded in the current System.AppDomain upon resolver construction, provided that the type doesn't have a finalizer. They utilize the a type's assembly-qualified name in order to resolve it across serialization boundaries.
FullGuidTypeResolver
Instances of FullGuidTypeResolver allow for serialization of any and all types loaded in the current System.AppDomain upon resolver construction. They utilize the MD5 hash of a type's name in order to resolve it across serialization boundaries.
GuidTypeResolver
Instances of GuidTypeResolver utilize the MD5 hash of a type's name in order to resolve it across serialization boundaries. They allow for serialization of any type specified in the constructor.
NameTypeResolver
Instances of NameTypeResolver utilize the a type's assembly-qualified name in order to resolve it across serialization boundaries. They allow for serialization of all types.
PowerDeserializationContext
Stores data about a PowerSerializer's deserialization operation, including a list of serialized types as well as the current object graph. This class, along with CreateDeserializationContext(), may be overriden to store additional information.
PowerSerializationContext
Stores data about a PowerSerializer's serialization operation, including a list of serialized types as well as the current object graph. This class, along with CreateSerializationContext(), may be overriden to store additional information.
PowerSerializer
Allows for customizable binary serialization and deserialization of complex types. Instances of PowerSerializer can serialize absolutely anything, provided that their type resolver allows it.
Interfaces
ITypeResolver
Types that implement ITypeResolver mediate which types can be serialized/deserialized by a PowerSerializer. Additionally, they each provide a way to identify types across serialization boundaries.


Repetitive tasks are very common in programming.
The code below shows a commonly used pattern for a for loop
##some object that I would like to iterate over
vec_iterate <- 1:10
##create the output vector
name_output <- vector(mode = "character", length = length(vec_iterate))
name_output
## [1] "" "" "" "" "" "" "" "" "" ""
👉👉Task Set 1👈👈
## # A tibble: 6 x 6
## country continent year lifeExp pop gdpPercap
## <fct> <fct> <int> <dbl> <int> <dbl>
## 1 Afghanistan Asia 1952 28.8 8425333 779.
## 2 Afghanistan Asia 1957 30.3 9240934 821.
## 3 Afghanistan Asia 1962 32.0 10267083 853.
## 4 Afghanistan Asia 1967 34.0 11537966 836.
## 5 Afghanistan Asia 1972 36.1 13079460 740.
## 6 Afghanistan Asia 1977 38.4 14880372 786.
I would like to take gapminder data set and create a histogram if it is a continous variable and bar chart for continent
and do nothing for country
.


[1]
overall though i would recommend this hotel and would come back in the future.
[2]
of course if you plan on this hotel and would come back in the future.


class: left, middle, inverse, title-slide # An Introduction to R ## ED 217 B Week 3 ### Daniel Katz ### 1/20/2021 --- class: middle class: middle ## Working Directories + The notion of the working directory isn't unique to R + Directories are just folders on your computer with a certain `path` + For instance, your `desktop` is a directory + A folder within `Documents` is a directory + A `working directory` is just where you tell `R` that you'll be working + when you load files, it knows to look to a certain folder + When you export files, `R` exports automatically to the working directory --- class: middle ## File paths (for PC) + A path to a Word doc on my desktop looks like: `C:/Users/katzd/Desktop/myfile.docx` + A path to a file in a folder called, `thesis` in the `Documents directory` looks like `C:/Users/katzd/Documents/thesis/myfile.docx` + But these are long! + Instead, we'll tell `R` where to read files from, where to export, etc, via `projects` --- class: middle ## File Paths: Getting the Wd... + To see the currect working directory, simply run the code `getwd()` + To set the working directory `setwd("filepath")` (the filepath has to be in quotes) + It's often useful to set the working directory at the top of your script **via menu options**: Session ➡ Set Working Directory ➡ Choose Directory (copy and paste outputted code to the top of your script) + Some people don't like this practice: [**Link**](https://whattheyforgot.org/safe-paths.html) + Run: `getwd()` --- class: middle ### For this class, what I might do(with various optional permutations).... --- class: middle ## Option 1 **New RStudio project for each lab**: **Step 1.** Create a folder called ED_217B (no spaces) **Step 2.** Create a ffolder called `labs` **Step 3.** Open up `RStudio` **Step 4.** Create a new project from within R studio, creating a new directory within `labs` called `lab2` path to hypothetical `data.csv` file: The path to hypothetical data in a data folder will be `data/data.csv` --- class: middle ## Option 2 **One RStudio project for all labs** Do the above but... **Step 1.** Create a folder called ED_217B (no spaces) **Step 2.** Open up `RStudio` **Step 3.** Create a new project from within RStudio, creating a new directory called labs. **Step 4.** Add a folder for each week's lab materials. You add one extra level to your R project. path to hypothetical `data.csv` file: `lab2/data/data.csv` --- # Setting up a working directory My working directory looks like this:  --- class: middle ## R and Functions + R is a (mostly) functional programming language + instead of `\(f(x)\)` we say things like, `mean(x)` + We can also have composite/multiple functions - `\(f(g(x))\)` - `print(mean(x))` + R packages are bundles of these functions written by the public + You can also write your own personal functions to use (we can go over this, later) --- class: middle # R and Packages + Think of `packages` as very specialized bundles of software. + They're collections of functions developed by the "community" + They're publically accessible (after passing various tests) via `CRAN` or `Bioconductor` + May be able to get some `development` versions on `github` -- + These packages have documentation if they're on `CRAN: The Comprehensive R Archive Network` (and even sometimes if they're not) -- + You don't need all the packages! -- + Packages add functions to base R -- + For instance, the `psych` package is built with tools commonly needed in the social sciences while the `TAM` package is specially built for IRT (with a Rasch emphasis) --- class: middle ## Packages, cont'd + When you `install.packages()`, R will search for a package on `CRAN` and download it to your computer (have to be connected to the internet) + To see where it's downloaded, run `.libpaths()` + Each time you start a new R session, you have to load a package via `library()` but you don't have to install each time --- ## Getting help + First and foremost, you can see installed packages in the bottom-right pane:  + Click on a package to see the documentation --- ## Other Help + You can also type `?packagename` such as `?dplyr` + To get help with a base R function, you can type ?function + To get help with a package function, you can type: `?packagename::function` such as `?dplyr::select` where select is a particular function in R. + Under the Help tab in R studio (next to `Tools` at the top), exists a few resources as well, including "cheat sheets" --- # Run the examples in documentation, usually at the bottom of the section to see how something works!  --- ### This is one of those necessary evils, so you know how R works - To select individual elements in a vector, we use `[]` and the numeric position that you would like to extract. ```r # vector reversing from 100 to 15 vec <- c(100:15) # select the first ten elements vec[1:10] ``` ``` ## [1] 100 99 98 97 96 95 94 93 92 91 ``` - This is also how one may pull out individual columns, rows, or specific rows and columns in a dataframe in R using base R - We can also get descriptives or compute across a vector, or specific vector elements - Working with datasets or matrices, we can subset rows, columns or both --- class: middle #### Rows: `df[1, ]` (gets just the first row) #### Columnn `df[,1]` or `df[1]` (gets just the first column) #### Both `df[1,1]` #### Can Select multiple `df[1:5, 1:6]` or `df[c(1,3, 8), c(1, 9, 2)]` --- class: middle ### Random, not too often needed information about `dataframes`: 1. Dataframes are a particular type, or class, of object in R. 2. The tidyverse will return something that's also called a `tibble`. 3. It's just like a dataframe except, better printing and... 4. When you subset a dataframe with just a row or column (but not both), it'll return a vector, not a df. 5. A `tibble` won't do this. --- ## Example ```r df <- data.frame(col1 = 1:10, col2 = 11:20) # just an integer class(df[,1]) ``` ``` ## [1] "integer" ``` ```r # add drop=FALSE (or drop = F) class(df[,1, drop=FALSE]) ``` ``` ## [1] "data.frame" ``` ```r df_tibble <- tibble(col1 = 1:10, col2 = 21:30) class(df_tibble[,1]) ``` ``` ## [1] "tbl_df" "tbl" "data.frame" ``` --- class: middle ### Arrays + An array is a general multidimensional data structure + mindblowingly, a matrix that essentially has an x, y, and a z axis (or more!) + One can make lists of matrices or dataframes, and lists can be [_**IN dataframes**_](https://jennybc.github.io/purrr-tutorial/ls13_list-columns.html) + Sometimes, you want to do the same thing to a number of dataframes [_**iteration**_](https://r4ds.had.co.nz/iteration.html). + In a list, you have a method for doing this very easily --- class: middle ## Getting data sets in to R with the `readr` package from the `tidyverse` + The first challenge: getting data into R + R needs a command to read in the data + R needs a path, where to find the data + We're going to use the [_**`readr`**_](https://readr.tidyverse.org/) package to handle most file types #### However, the `foreign` package and `haven` package can handle files from SAS, SPSS, STATA + You don't need SPSS, SAS, or STATA to read data into R + We'll start today with a standard `.csv` file. --- # Getting set up. 1. Download the R script `Intro_to_R2.R` 2. Place `Intro_to_R2.R` in your `scripts` subdirectory. 3. Download `eirt_data.csv` into your `data` or `raw_data` folder (however you have it set up) --- # What we'll be using today: 1. `readr` 2. `dplyr` 3. `ggplot` ---


Team 254 strives to enhance awareness of STEM education and FIRST robotics through demonstrations of the team’s robots to sponsors and the community at large. The team’s T-shirt cannon, Shockwave, has appeared at sporting events at Bellarmine College Preparatory, local universities like Stanford and Santa Clara, and many other events. Shockwave has been featured on Sports Illustrated. In addition, the team provides all of its sponsors with robot demonstrations of Shockwave and its other robots.
Team 254 strives not only for technical excellence but also maximum community impact. The team accomplishes this through demonstrations of the team’s robots, volunteering and outreach, and hosting workshops.
In recent years, the team contributed to or participated in over 100 community events, including:
Team 254 is also particularly proud that it finishes each year distributing gifts and demonstrating its robot at Working Partnership's annual Holiday Fair and Toys for Tots event.
Finally, Team 254 also hosts the largest privately organized Offseason robotics competition tournament in California.
Team 254 is very fortunate to possess certain resources not available to other teams. Because of this, our team is devoted to spreading our resources to other teams in the form of outreach. Team 254 has started and mentored many teams throughout the West Coast of the United States and Israel. As early as the first few years of the decade, members from Team 254 traveled to Hawaii and Alaska to spread the message of FIRST to the farthest corners of the country. Team 254 was also involved in the creation of the Western Regional Robotics Forum (WRRF), which provides services to many northern California teams and hosts the CalGames Off-Season Competition each year
Team 254 has started and mentored many teams throughout the West Coast of the United States and Israel. As early as the first few years of the decade, members from Team 254 traveled to Hawaii and Alaska to spread the message of FIRST to the farthest corners of the country. Team 254 was also involved in the creation of the Western Regional Robotics Forum (WRRF), which provides services to many northern California teams and hosts the CalGames Off-Season Competition each year.
For example, 254 began helping Team 359, the Hawaiian Kids, in 1999, and mentored them until their 2011 Championship Chairman’s Award win, which marked the first time in FIRST history that a Hall of Fame team mentored another into the Hall of Fame.
Team 971, inspired by 254’s commitment to excellence, began holding design reviews for other teams, inspired by the ones 254 gave them and other veteran and rookie teams in years past. They credit 254 for the inspiration and continue to collaborate with the team today.
In addition, in the spirit of Coopertition, the FIRST robotics commitment to cooperation and competition, team mentors and students lead electrical, programming, CAD, competition analysis, and organizational workshops for local teams. For example, the team teaches classes at the WRRF workshops and the team’s mentors continue to provide extensive engineering support to teams 968 and 1868.
Before the team ships off the robot to competition, Team 254 works with Team 1868 to organize a practice scrimmage at the NASA Ames Research Center so that other teams can better prepare for competition by fixing any problems and practicing their driving.
Team 254 also mentors many FRC teams, both locally and over a distance. Furthermore, Team 254 mentors many First Lego League (FLL) and VEX Robotics Competition (VRC) teams throughout the San Francisco Bay Area. The team has mentored numerous FLL teams including those at St. Martin’s and St. Christopher middle schools.
Besides work with individual teams, Team 254 has a 80% size practice field which we open up to other teams regularly. Besides an annual practice weekend and scrimmage on the last weekend of the build season, Team 254 is willing to work with any team so that they can come in when it fits their schedule. Team 254′s extensive mentoring throughout the years forms an integral part of its team history and we are proud to support many teams throughout the West Coast and throughout the world.


Writing a Unit Test for a Stage
- Regular Unit Test
- Unit Testing with our Stage Testing Framework
- Testing a Usual Stage
- Testing a Stage With Multiple Output Ports
- Testing a Producer Stage
- Testing a Sink Stage
Regular Unit Test
For a regular unit test of a stage, you need to create and execute a custom configuration which contains your stage under test (SUT), at least one producer to feed your SUT, and at least one sink to collect and test the output elements of your SUT. An example is shown below. It does not only include much boilerplate code, but also hides the actual SUT. The test does not name the SUT because it is encapsulated by the custom configuration.
@Test public void shouldExecutePipelineCorrectly() throws Exception { String[] inputElements = { "a", "b", "c" }; GlobalTaskQueueConfig<String> config = new GlobalTaskQueueConfig<>(inputElements); Execution<GlobalTaskQueueConfig<String>> execution = new Execution<>(config); execution.executeBlocking(); String[] expectedElements = { "a", "b", "c" }; assertThat(inputElements, is(equalTo(expectedElements))); }
Unit Testing with our Stage Testing Framework
Instead of building and executing a test configuration by yourself, our testing framework takes over this task. It automatically creates a configuration which contains one InitialElementProducer per input port and one CollectorSink per output port of your stage under test. Just start with the entry method teetime.framework.test.StageTester.test(..) and enjoy the fluent API. Examples are given below.
Testing a Usual Stage
import static org.hamcrest.core.Is.is; import static org.junit.Assert.assertThat; import static teetime.framework.test.StageTester.*; public class CounterTest { private Counter<Integer> counter; @Before public void initializeCounter() { counter = new Counter<Integer>(); } @Test public void counterValueShouldBeCorrect() { test(counter).and() .send(1, 2, -5, 10, 9).to(counter.getInputPort()).and() .start(); assertThat(counter.getOutputPort(), produces(1, 2, -5, 10, 9)); assertThat(counter.getNumElementsPassed(), is(5)); } }
Testing a Stage With Multiple Output Ports
@Before public void before() { filter = InstanceOfFilter<Object, Clazz>(Clazz.class); } @Test public void outputMatchedAndMismatchedElements() { final Clazz clazz = new Clazz(); final Integer number = 42; test(filter).and() .send(clazz, number, clazz).to(filter.getInputPort()) .start(); assertThat(filter.getMatchedOutputPort(), produces(clazz, clazz)); assertThat(filter.getMismatchedOutputPort(), produces(number)); }
Testing a Producer Stage
@Before public void before() { producer = new InitialElementProducer<Integer>(1, 2, 3); } @Test public void testProducer() throws Exception { test(producer).start(); assertThat(producer.getOutputPort(), produces(1, 2, 3)); }
Testing a Sink Stage
@Before public void before() { consumer = new CollectorSink<Integer>(); } @Test public void testSink() throws Exception { test(consumer).and() .send(1, 2, 3).to(consumer.getInputPort()).and() .start(); assertThat(consumer.getElements(), contains(1, 2, 3)); }
More example tests written with our stage testing framework can be found in the source code.


{% for form in forms %}
{% include "bootstrap.form.html" %}
{% endfor %}
Cryptographically verified:
{{ proof_response.verified }}
Available credential data:
{{ proof_response.parsed_proof | jsonify }}
Received proof:
{{ proof_response.proof | jsonify }}


Overview
Package
Class
Use
Tree
Deprecated
Index
Help
PREV LETTER
NEXT LETTER
FRAMES
NO FRAMES
All Classes
A
B
C
D
E
F
G
H
I
L
M
N
O
P
Q
R
S
T
U
V
W
X
Y
S
SaveWorkspaceMenuItem
- Class in
gui.menubar
scalarMultiply(Tuple, double)
- Static method in class util.
Tuple
SCREEN_DIMENSION
- Static variable in class gui.
WorldGraphicsPanel
serialize(String, T)
- Static method in class util.
Serializer
Serializer
- Class in
util
Serializer()
- Constructor for class util.
Serializer
setBackgroundImage(String)
- Method in class backend.
WorldModel
SetBackgroundMenuItem
- Class in
gui.menubar
setHeading(double)
- Method in class backend.
TurtleModel
setHeadingTo(double)
- Method in class backend.
WorldModel
Points turtle at point (x,y).
SetHeadingTurtleCommand
- Class in
commands.turtleCommands
SetHeadingTurtleCommand()
- Constructor for class commands.turtleCommands.
SetHeadingTurtleCommand
setImagePath(String)
- Method in class backend.
TurtleModel
SetPaletteGuiCommand
- Class in
commands.guiCommands
SetPaletteGuiCommand()
- Constructor for class commands.guiCommands.
SetPaletteGuiCommand
setPen(boolean)
- Method in class backend.
WorldModel
setPenColor(Color)
- Method in class backend.
WorldModel
SetPenColorGuiCommand
- Class in
commands.guiCommands
SetPenColorGuiCommand()
- Constructor for class commands.guiCommands.
SetPenColorGuiCommand
setPosition(Tuple)
- Method in class backend.
TurtleModel
SetShapeGuiCommand
- Class in
commands.guiCommands
SetShapeGuiCommand()
- Constructor for class commands.guiCommands.
SetShapeGuiCommand
setShowing(boolean)
- Method in class backend.
TurtleModel
setUp()
- Method in class slogo_tests.
BooleanCommandTests
setUp()
- Method in class slogo_tests.
InterpreterTest
setUp()
- Method in class slogo_tests.
MathCommandTests
setUp()
- Method in class slogo_tests.
ProgramTester
setUpBeforeClass()
- Static method in class slogo_tests.
InterpreterTest
setUserCommands(Object)
- Method in class backend.
WorldModel
setVariables(Object)
- Method in class backend.
WorldModel
ShapeQueryGuiCommand
- Class in
commands.guiCommands
ShapeQueryGuiCommand()
- Constructor for class commands.guiCommands.
ShapeQueryGuiCommand
ShowingQueryTurtleCommand
- Class in
commands.turtleCommands
ShowingQueryTurtleCommand()
- Constructor for class commands.turtleCommands.
ShowingQueryTurtleCommand
ShowTurtleTurtleCommand
- Class in
commands.turtleCommands
ShowTurtleTurtleCommand()
- Constructor for class commands.turtleCommands.
ShowTurtleTurtleCommand
SinMathCommand
- Class in
commands.mathCommands
SinMathCommand()
- Constructor for class commands.mathCommands.
SinMathCommand
slogo_tests
- package slogo_tests
SlogoDefaultConstraints
- Class in
gui
SlogoException
- Exception in
exceptions
SlogoException(String)
- Constructor for exception exceptions.
SlogoException
SlogoFrame
- Class in
gui
stateChanged(ChangeEvent)
- Method in class gui.
ColorChooser
StatsPanel
- Class in
gui
StringOps
- Class in
util
StringOps()
- Constructor for class util.
StringOps
subtract(Tuple, Tuple)
- Static method in class util.
Tuple
sum(Tuple, Tuple)
- Static method in class util.
Tuple
SumMathCommand
- Class in
commands.mathCommands
SumMathCommand()
- Constructor for class commands.mathCommands.
SumMathCommand
switchCurrentWorld(int)
- Method in class backend.
WorldsCollection
Overview
Package
Class
Use
Tree
Deprecated
Index
Help
PREV LETTER
NEXT LETTER
FRAMES
NO FRAMES
All Classes
A
B
C
D
E
F
G
H
I
L
M
N
O
P
Q
R
S
T
U
V
W
X
Y


Appendix B: FM Profiles
This section specifies the attributes and rules for the following list of profiles.
| Deductible and limit |
1 |
1 |
| Franchise deductible and limit |
2 |
3 |
| Deductible only |
3 |
12 |
| Maximum Deductible |
4 |
10 |
| Minimum Deductible |
5 |
11 |
| Deductible, limit and share |
6 |
2 |
| Deductible and limit as a proportion of loss |
10 |
5 |
| Limit with deductible as a proportion of limit |
11 |
9 |
| Limit only |
12 |
14 |
| Limit as a proportion of loss |
13 |
15 |
| Deductible as a proportion of loss |
14 |
16 |
An allocation rule can be assigned to any profile at any stage in the calculation, which determines whether calculated losses should be back-allocated to the contributing items, and if so how. This is specified via an allocrule_id in the fm_profile data.
The choices are as follows;
| Don't back-allocate losses |
0 |
| Back allocate losses to items in proportion to ground up loss |
1 |
| Back-allocate losses to items in proportion to previous level loss |
2 |
In the following notation;
- x.loss is the input loss to the calculation
- x.retained_loss is the input retained loss to the calculation (where required)
- loss is the calculated loss
- ded, lim, and share are alias variables for the profile fields as required
1. Deductible and limit
| policytc_id |
1 |
| ccy_id |
1 |
| deductible |
50000 |
| limit |
900000 |
Calculation logic
loss = x.loss - ded;
if (loss < 0) loss = 0;
if (loss > lim) loss = lim;
2. Franchise deductible and limit
| policytc_id |
1 |
| ccy_id |
1 |
| deductible |
100000 |
| limit |
1000000 |
Calculation logic
if (x.loss < ded) loss = 0;
else loss = x.loss;
if (loss > lim) loss = lim;
3. Deductible only
| policytc_id |
1 |
| ccy_id |
1 |
| deductible |
100000 |
Calculation logic
loss = x.loss - ded;
if (loss < 0) loss = 0;
4. Maximum deductible
If the accumulated deductibles from the previous level exceed the maximum deductible, the deductible is capped at the maximum deductible value
| policytc_id |
1 |
| ccy_id |
1 |
| deductible |
40000 |
Calculation logic
if (x.retained_loss > ded) {
loss = x.loss + x.retained_loss - ded;
if (loss < 0) loss = 0;
}
else {
loss = x.loss;
}
5. Minimum deductible
If the accumulated deductibles from the previous level are less than the minimum deductible, the deductible is increased to the total loss or the minimum deductible value, whichever is greater.
| policytc_id |
1 |
| ccy_id |
1 |
| deductible |
70000 |
Calculation logic
if (x.retained_loss < ded) {
loss = x.loss + x.retained_loss - ded;
if (loss < 0) loss = 0;
}
else {
loss = x.loss;
}
6. Deductible, limit and share
| policytc_id |
1 |
| ccy_id |
1 |
| deductible |
70000 |
| limit |
1000000 |
| share_prop_of_lim |
0.1 |
Calculation logic
if (x.loss > (ded + lim)) loss = lim;
else loss = x.loss - ded;
if (loss < 0) loss = 0;
loss = loss * share;
10. Deductible and limit as a proportion of loss
| policytc_id |
1 |
| deductible_prop_of_loss |
0.05 |
| limit_prop_of_loss |
0.3 |
Calculation logic
loss = x.loss * (lim - ded);
11. Limit with deductible as a proportion of limit
| policytc_id |
1 |
| ccy_id |
1 |
| deductible_prop_of_loss |
0.05 |
| limit |
100000 |
Calculation logic
loss = x.loss - (ded * lim);
if (loss < 0) loss = 0;
if (loss > lim) loss = lim;
12. Limit only
| policytc_id |
1 |
| ccy_id |
1 |
| limit |
100000 |
Calculation logic
loss = x.loss;
if (loss > lim) loss = lim;
13. Limit as a proportion of loss
| policytc_id |
1 |
| limit_prop_of_loss |
0.3 |
Calculation logic
loss = x.loss;
loss = loss * lim;
14. Deductible as a proportion of loss
| policytc_id |
1 |
| deductible_prop_of_loss |
0.05 |
Calculation logic
loss = x.loss;
loss = loss - (loss * ded);
if (loss < 0) loss = 0;
Return to top
Go to Appendix C Multi-peril model support
Back to Contents


# Copyright (c) 2021 - present / Neuralmagic, Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Code related to running a module through training and testing over a dataset.
Allows reporting of progress and override functions and hooks.
"""
import time
from abc import ABC, abstractmethod
from collections import OrderedDict
from contextlib import ExitStack
from typing import Any, Callable, Dict, List, Union
import torch
from torch import Tensor
from torch.nn import Module
from torch.optim.optimizer import Optimizer
from torch.utils.data import DataLoader
from torch.utils.hooks import RemovableHandle
from tqdm import auto
from sparseml.pytorch.utils.helpers import (
get_optim_learning_rate,
tensors_batch_size,
tensors_module_forward,
tensors_to_device,
)
from sparseml.pytorch.utils.logger import PyTorchLogger
from sparseml.pytorch.utils.loss import DEFAULT_LOSS_KEY, LossWrapper
try:
from torch.cuda.amp import GradScaler, autocast
amp_import_error = None
except Exception as amp_error:
autocast = None
GradScaler = None
amp_import_error = amp_error
__all__ = [
"def_model_backward",
"ModuleRunFuncs",
"ModuleRunHooks",
"ModuleRunResults",
"ModuleDeviceContext",
"ModuleTester",
"ModuleTrainer",
]
[docs]def def_model_backward(
losses: Dict[str, Tensor], model: Module, scaler: GradScaler = None
):
"""
Default function to perform a backwards pass for a model and the calculated losses
Calls backwards for the DEFAULT_LOSS_KEY in losses Dict
:param model: the model to run the backward for
:param losses: the losses dictionary containing named tensors,
DEFAULT_LOSS_KEY is expected to exist and backwards is called on that
:param scaler: GradScaler object for running in mixed precision with amp. If scaler
is not None will call scaler.scale on the loss object. Default is None
"""
# assume loss is at default loss key
loss = losses[DEFAULT_LOSS_KEY]
if scaler is not None:
loss = scaler.scale(loss)
loss.backward()
[docs]class ModuleRunHooks(object):
"""
Container for hooks that can be added to module runs like training and testing
for different stages of running a batch through a model.
| Lifecycle:
| - data batch size callback
| - data to device callback
| - batch start hook
| - data model forward callback
| - batch forward hook
| - loss calculation
| - batch loss hook
| - model backward callback
| - batch backward hook
| - optimizer / gradient update
| - batch end hook
"""
def __init__(self):
self._batch_start_hooks = OrderedDict()
self._batch_forward_hooks = OrderedDict()
self._batch_loss_hooks = OrderedDict()
self._batch_backward_hooks = OrderedDict()
self._batch_end_hooks = OrderedDict()
[docs] def register_batch_start_hook(
self, hook: Callable[[int, int, int, Any], None]
) -> RemovableHandle:
"""
Called at the start of a batch with the following info:
(counter, step_count, batch_size, data)
where counter is passed in to the run (ex: epoch),
step_count is the number of items run so far,
batch_size is the number of elements fed in the batch,
data is the data output from the loader
:param hook: the hook to add that is called into when reached in the
batch process
:return: a removable handle to remove the hook when desired
"""
handle = RemovableHandle(self._batch_start_hooks)
self._batch_start_hooks[handle.id] = hook
return handle
[docs] def register_batch_forward_hook(
self, hook: Callable[[int, int, int, Any, Any], None]
) -> RemovableHandle:
"""
Called after forward execution of a batch in the model with the following info:
(counter, step_count, batch_size, data, pred)
where counter is passed in to the run (ex: epoch),
step_count is the number of items run so far,
batch_size is the number of elements fed in the batch,
data is the data output from the loader,
pred is the result from the model after the forward
:param hook: the hook to add that is called into when reached in the
batch process
:return: a removable handle to remove the hook when desired
"""
handle = RemovableHandle(self._batch_forward_hooks)
self._batch_forward_hooks[handle.id] = hook
return handle
[docs] def register_batch_loss_hook(
self, hook: Callable[[int, int, int, Any, Any, Dict[str, Tensor]], None]
):
"""
Called after loss calculation of the batch with the following info:
(counter, step_count, batch_size, data, pred, losses)
where counter is passed in to the run (ex: epoch),
step_count is the number of items run so far,
batch_size is the number of elements fed in the batch,
data is the data output from the loader,
pred is the result from the model after the forward,
losses are the resulting loss dictionary
:param hook: the hook to add that is called into when reached in the
batch process
:return: a removable handle to remove the hook when desired
"""
handle = RemovableHandle(self._batch_loss_hooks)
self._batch_loss_hooks[handle.id] = hook
return handle
[docs] def register_batch_backward_hook(
self, hook: Callable[[int, int, int, Any, Any, Dict[str, Tensor]], None]
):
"""
Called after calling backward on the loss for the batch with the following info:
(counter, step_count, batch_size, data, pred, losses)
where counter is passed in to the run (ex: epoch),
step_count is the number of items run so far,
batch_size is the number of elements fed in the batch,
data is the data output from the loader,
pred is the result from the model after the forward,
losses are the resulting loss dictionary
:param hook: the hook to add that is called into when reached in the
batch process
:return: a removable handle to remove the hook when desired
"""
handle = RemovableHandle(self._batch_backward_hooks)
self._batch_backward_hooks[handle.id] = hook
return handle
[docs] def register_batch_end_hook(
self, hook: Callable[[int, int, int, Any, Any, Dict[str, Tensor]], None]
):
"""
Called after all calculations are done for the batch with the following info:
(counter, step_count, batch_size, data, pred, losses)
where counter is passed in to the run (ex: epoch),
step_count is the number of items run so far,
batch_size is the number of elements fed in the batch,
data is the data output from the loader,
pred is the result from the model after the forward,
losses are the resulting loss dictionary
:param hook: the hook to add that is called into when reached in the
batch process
:return: a removable handle to remove the hook when desired
"""
handle = RemovableHandle(self._batch_end_hooks)
self._batch_end_hooks[handle.id] = hook
return handle
[docs] def invoke_batch_start(
self, counter: int, step_count: int, batch_size: int, data: Any
):
for hook in self._batch_start_hooks.values():
hook(counter, step_count, batch_size, data)
[docs] def invoke_batch_forward(
self, counter: int, step_count: int, batch_size: int, data: Any, pred: Any
):
for hook in self._batch_forward_hooks.values():
hook(counter, step_count, batch_size, data, pred)
[docs] def invoke_batch_loss(
self,
counter: int,
step_count: int,
batch_size: int,
data: Any,
pred: Any,
losses: Dict[str, Tensor],
):
for hook in self._batch_loss_hooks.values():
hook(counter, step_count, batch_size, data, pred, losses)
[docs] def invoke_batch_backward(
self,
counter: int,
step_count: int,
batch_size: int,
data: Any,
pred: Any,
losses: Dict[str, Tensor],
):
for hook in self._batch_backward_hooks.values():
hook(counter, step_count, batch_size, data, pred, losses)
[docs] def invoke_batch_end(
self,
counter: int,
step_count: int,
batch_size: int,
data: Any,
pred: Any,
losses: Dict[str, Tensor],
):
for hook in self._batch_end_hooks.values():
hook(counter, step_count, batch_size, data, pred, losses)
[docs]class ModuleRunFuncs(object):
"""
Functions used as callables to calculate or perform necessary operations
for running a model through training or testing.
| Lifecycle:
| - data batch size callback
| - data to device callback
| - batch start hook
| - data model forward callback
| - batch forward hook
| - loss calculation
| - batch loss hook
| - model backward callback
| - batch backward hook
| - optimizer / gradient update
| - batch end hook
"""
def __init__(self):
self._batch_size = tensors_batch_size
self._to_device = tensors_to_device
self._model_forward = tensors_module_forward
self._model_backward = def_model_backward
@property
def batch_size(self) -> Callable[[Any], int]:
"""
:return used to calculate the batch size of a given grouping of tensors.
Expected to be called with the output from a data loader and
then return an int representing the batch size.
"""
return self._batch_size
@batch_size.setter
def batch_size(self, value: Callable[[Any], int]):
"""
Used to calculate the batch size of a given grouping of tensors.
Expected to be called with the output from a data loader
then return an int representing the batch size.
:param value: the callable used to calculate batch size
for a grouping of tensors
"""
self._batch_size = value
@property
def to_device(self) -> Callable[[Any, str], Any]:
"""
:return used to place a given grouping of tensors onto the proper device.
Expected to be called with the output from a data loader and the
desired device as a string then return the grouping on the proper device.
"""
return self._to_device
@to_device.setter
def to_device(self, value: Callable[[Any, str], Any]):
"""
Used to place a given grouping of tensors onto the proper device.
Expected to be called with the output from a data loader and the
desired device as a string then return the grouping on the proper device
:param value: the callable used to place a grouping of tensors onto
the proper device
"""
self._to_device = value
@property
def model_forward(self) -> Callable[[Any, Module], Any]:
"""
:return used to propagate a given grouping of tensors through a model and
return the result.
Expected to be called with the model and the output from a data loader
then return the result from the model forward pass.
"""
return self._model_forward
@model_forward.setter
def model_forward(self, value: Callable[[Any, Module], Any]):
"""
Used to propagate a given grouping of tensors through a model
and return the result.
Expected to be called with the model and the output from a data loader
then return the result from the model forward pass.
:param value: the callable used to run a grouping of tensors through a model
:return: the result of running the data through the model
"""
self._model_forward = value
@property
def model_backward(self) -> Callable[[Dict[str, Tensor], Module], None]:
"""
:return used to call backward for a given model and the calculated losses.
Expected to be called with the model and the output from the loss function
as a dict mapping of names to tensors returns nothing
"""
return self._model_backward
@model_backward.setter
def model_backward(self, value: Callable[[Dict[str, Tensor], Module], None]):
"""
Used to call backward for a given model and the calculated losses.
Expected to be called with the model and the output from the loss function
as a dict mapping of names to tensors returns nothing
:param value: the callable used to run a backwards pass for the given
loss functions
"""
self._model_backward = value
[docs] def copy(self, run_funcs):
"""
Copy the functions from the current instance into a new instance
:param run_funcs: the instance to copy the functions into
"""
run_funcs = run_funcs # type: ModuleRunFuncs
self._batch_size = run_funcs._batch_size
self._to_device = run_funcs._to_device
self._model_forward = run_funcs._model_forward
self._model_backward = run_funcs._model_backward
[docs]class ModuleRunResults(object):
"""
Class containing the results / losses from a model run for training or testing
Keeps all result values as a dictionary and Tensor containing all values
"""
def __init__(self):
self._results = {}
def __repr__(self):
results = [
"{}={}".format(key, self.result_mean(key).item()) for key in self._results
]
return "ModuleRunResults({})".format(", ".join(results))
@property
def results(self) -> Dict[str, List[Tensor]]:
"""
All of the stored results for the loss functions
:return: a dictionary containing a mapping of name (str) to a list of tensors
that were recorded for that loss
"""
return self._results
[docs] def result(self, key: str) -> List[Tensor]:
"""
The result of a single loss function
:param key: the name of the loss function to get the results for
:return: a list of tensors containing all of the results for that loss
"""
return self._results[key]
[docs] def result_list_tensor(self, key: str) -> Tensor:
"""
Get the results as a list tensor where all items have been stacked into
the first index of the tensor.
:param key: the name of the loss function to get the results for
:return: a tensor containing all of the tensors for that result
"""
res = self.result(key)
return torch.cat(res)
[docs] def result_mean(self, key: str) -> Tensor:
"""
The mean result of a single loss function
:param key: the name of the loss function to get the mean result for
:return: a single tensor containing the average of all the results for that loss
"""
res = self.result_list_tensor(key)
return torch.mean(res)
[docs] def result_std(self, key: str) -> Tensor:
"""
The standard deviation of the result for a single loss function
:param key: the name of the loss function to get the standard
deviation result for
:return: a single tensor containing the standard deviation of all
the results for that loss
"""
res = self.result_list_tensor(key)
return torch.std(res)
[docs] def append(self, losses: Dict[str, Tensor], batch_size: int):
"""
add new losses to the current stored results
:param losses: the losses to be added
:param batch_size: the batch size the losses were run for
"""
for key, val in losses.items():
if key not in self._results:
self._results[key] = []
result = val.detach_().cpu()
result = result.repeat(batch_size)
self._results[key].append(result)
[docs]class ModuleDeviceContext(object):
"""
Simple class to define device settings or context to be used when running a Module
:param use_mixed_precision: set True to execute model using mixed precision with
torch.cuda.amp. Default is False
:param world_size: the world size (total number of devices) used when running
the given module using DistributedDataParallel. Losses will be scaled by the
world size. Default is 1.
"""
def __init__(self, use_mixed_precision: bool = False, world_size: int = 1):
self._use_mixed_precision = use_mixed_precision
self._world_size = world_size
self._validate()
[docs] @staticmethod
def default_context():
"""
:return: A ModuleDeviceContext with default settings enabled
"""
return ModuleDeviceContext(use_mixed_precision=False, world_size=1)
@property
def use_mixed_precision(self) -> bool:
"""
:return: True if mixed precision with torch.cuda.amp should be used.
False otherwise
"""
return self._use_mixed_precision
@use_mixed_precision.setter
def use_mixed_precision(self, value: bool):
"""
:param value: True if mixed precision with torch.cuda.amp should be used.
False otherwise
"""
self._use_mixed_precision = value
self._validate()
@property
def world_size(self) -> int:
"""
:return: the world size (total number of devices) used when running
the given module using DistributedDataParallel. Losses will be scaled by the
world size
"""
return self._world_size
@world_size.setter
def world_size(self, value: int):
"""
:param value: the world size (total number of devices) used when running
the given module using DistributedDataParallel. Losses will be scaled by the
world size
"""
self._world_size = value
self._validate()
def _validate(self):
assert isinstance(
self.use_mixed_precision, bool
), "use_mixed_precision must be a boolean"
assert (
isinstance(self.world_size, int) and self.world_size > 0
), "world_size must be a positive int"
class ModuleRunner(ABC):
"""
Abstract class for running data through a module and recording the results
:param module: the model to run evaluation for
:param device: the default device to run evaluation on
(where data will be copied to)
:param loss: the loss functions callable used to calculate loss values after
executing a forward pass
:param loggers: list of loggers to log training results to
:param log_name: the key to store all log files under
:param log_steps: The number of steps (batches) to log at,
ex 100 will log every 100 batches
:param log_summary: True to log the final summary results after the run completes
:param device_context: ModuleDeviceContext with settings to enable mixed precision
using torch.cuda.amp or adjust losses when using DistributedDataParallel.
Default settings do not use mixed precision or account for DDP.
"""
def __init__(
self,
module: Module,
device: str,
loss: Union[LossWrapper, Callable[[Any, Any], Tensor]],
loggers: List[PyTorchLogger],
log_name: str,
log_steps: int,
log_summary: bool,
device_context: ModuleDeviceContext = ModuleDeviceContext.default_context(),
):
self._module = module
self._device = device
self._loss = (
loss
if isinstance(loss, LossWrapper)
else LossWrapper(loss, deconstruct_tensors=False)
)
self._loggers = loggers
self._log_name = log_name
self._log_steps = log_steps
self._log_summary = log_summary
self._device_context = device_context
self._run_funcs = ModuleRunFuncs()
self._run_hooks = ModuleRunHooks()
@property
def module(self) -> Module:
"""
:return: the model to run
"""
return self._module
@property
def device(self) -> str:
"""
:return: the default device to run on (where data will be copied to)
"""
return self._device
@property
def loss(self) -> LossWrapper:
"""
:return: the loss functions callable used to calculate loss values
after executing a forward pass
"""
return self._loss
@property
def run_funcs(self) -> ModuleRunFuncs:
"""
:return: functions used while running evaluation of the model as
callbacks to do certain stages
"""
return self._run_funcs
@property
def run_hooks(self) -> ModuleRunHooks:
"""
:return: hooks used while running evaluation of the model to
receive intermediate results
"""
return self._run_hooks
@property
def device_context(self) -> ModuleDeviceContext:
"""
:return: ModuleDeviceContext with settings for enabling mixed precision
using torch.cuda.amp or adjusting losses when using DistributedDataParallel.
"""
return self._device_context
def run(
self,
data_loader: DataLoader,
desc: str,
counter: int = -1,
show_progress: bool = True,
track_results: bool = True,
max_steps: int = -1,
) -> Union[None, ModuleRunResults]:
"""
Run evaluation over all the data in the given data loader
:param data_loader: the data loader used to gather batches to be
run through the model
:param desc: description used in the progress indicator
:param counter: counter passed to the hooks for external
state keeping (ex: epoch)
:param show_progress: True to show a progress bar, False otherwise
:param track_results: True to track and return the results of the evaluation,
False to return None
:param max_steps: maximum number of steps/batches to run through,
will stop after reaching this. if <= 0 then no restriction is placed
:return: the results of evaluation if track_results else None
"""
if self._log_summary and not track_results:
raise ValueError(
"runner must be run with track_results=True to log the final results"
)
self._runner_setup()
try:
counter_len = len(data_loader)
except Exception:
# can't track data loaders length
counter_len = 0
if max_steps > 0 and counter_len > 0:
progress_steps = min(max_steps, counter_len)
elif max_steps > 0:
progress_steps = max_steps
elif counter_len > 0:
progress_steps = counter_len
else:
progress_steps = None
data_iter = (
enumerate(data_loader)
if not show_progress
else auto.tqdm(enumerate(data_loader), desc=desc, total=progress_steps)
)
results = ModuleRunResults() if track_results else None
previous_steps = (counter if counter > -1 else 0) * counter_len
first_batch_size = None
epoch_timer = time.time()
for batch, data in data_iter:
step_timer = time.time()
batch_size = self._run_funcs.batch_size(data) # type: int
if first_batch_size is None:
first_batch_size = batch_size
should_log = (
self._loggers
and self._log_steps
and self._log_steps > 0
and batch % self._log_steps == 0
)
log_step = previous_steps + batch
batch_results = self._runner_batch(
counter, batch, batch_size, data, should_log, log_step
)
if should_log:
for loss, val in batch_results.items():
self._log_scalar(
"{}/{}".format(self._log_name, loss),
val.item(),
log_step,
)
self._log_scalar(
"{}/Epoch Counter".format(self._log_name),
counter,
log_step,
)
self._log_scalar(
"{}/Batch Size".format(self._log_name),
batch_size,
log_step,
)
step_time = time.time() - step_timer
self._log_scalar(
"{}/Seconds per step".format(self._log_name),
step_time,
log_step,
)
self._log_scalar(
"{}/Steps per second".format(self._log_name),
1.0 / step_time,
log_step,
)
if progress_steps:
remaining_steps = progress_steps - batch - 1
self._log_scalar(
"{}/Est remaining minutes".format(self._log_name),
(step_time * remaining_steps) / 60,
log_step,
)
if results is not None:
results.append(batch_results, batch_size)
if 0 < max_steps <= batch:
break
should_log = self._loggers and self._log_summary and results
log_step = counter # log under the counter step for the summaries
if should_log:
for loss in results.results.keys():
val = results.result_mean(loss)
self._log_scalar(
"{}/{} Summary".format(self._log_name, loss),
val.item(),
log_step,
)
self._log_scalar(
"{}/Batch Size Summary".format(self._log_name),
first_batch_size,
log_step,
)
self._log_scalar(
"{}/Minutes per epoch".format(self._log_name),
(time.time() - epoch_timer) / 60,
log_step,
)
self._runner_complete(results, should_log, log_step)
return results
def run_epoch(
self,
data_loader: DataLoader,
epoch: int,
show_progress: bool = True,
track_results: bool = True,
max_steps: int = -1,
):
"""
Convenience function for evaluation over all the data in the given data loader
for a specific epoch and making the progress visible.
:param data_loader: the data loader used to gather batches to be run
through the model
:param epoch: the current evaluation epoch number
:param show_progress: True to show a progress bar, False otherwise
:param track_results: True to track and return the results of the training,
False to return None
:param max_steps: maximum number of steps/batches to run through,
will stop after reaching this. if <= 0 then no restriction is placed
:return: the results of evaluation if track_results else None
"""
return self.run(
data_loader,
"{} epoch {}".format(self._log_name, epoch),
epoch,
show_progress,
track_results,
max_steps,
)
def _log_scalar(self, key: str, item: Any, step: int):
for logger in self._loggers:
logger.log_scalar(key, item, step)
@abstractmethod
def _runner_setup(self):
raise NotImplementedError()
@abstractmethod
def _runner_batch(
self,
counter: int,
batch: int,
batch_size: int,
data: Any,
should_log: bool,
log_step: int,
) -> Dict[str, Any]:
raise NotImplementedError()
@abstractmethod
def _runner_complete(
self, results: ModuleRunResults, should_log: bool, log_step: int
):
raise NotImplementedError()
[docs]class ModuleTrainer(ModuleRunner):
"""
Container for running a module through training over a given data loader
for specific settings.
| Lifecycle:
| - data batch size callback
| - data to device callback
| - batch start hook
| - data model forward callback
| - batch forward hook
| - loss calculation
| - batch loss hook
| - model backward callback
| - batch backward hook
| - optimizer / gradient update
| - batch end hook
:param module: the model to run training for
:param device: the default device to run training on (where data will be copied to)
:param loss: the loss functions callable used to calculate loss values
after executing a forward pass
:param optimizer: the optimizer used to apply gradient updates with
:param num_accumulated_batches: number of batches to accumulate before
updating the optimizer
:param optim_closure: a closure passed into the optimizer on step
:param loggers: list of loggers to log training results to
:param log_name: the key to store all log files under
:param log_steps: The number of steps (batches) to log at,
ex 100 will log every 100 batches
:param log_summary: True to log the final summary results after the run completes
:param device_context: ModuleDeviceContext with settings to enable mixed precision
using torch.cuda.amp or adjust losses when using DistributedDataParallel.
Default settings do not use mixed precision or account for DDP.
Will raise an exception if torch version does not support amp.
"""
def __init__(
self,
module: Module,
device: str,
loss: Union[LossWrapper, Callable[[Any, Any], Tensor]],
optimizer: Optimizer,
num_accumulated_batches: int = 1,
optim_closure: Union[None, Callable] = None,
loggers: List[PyTorchLogger] = None,
log_name: str = "Train",
log_steps: int = 100,
log_summary: bool = True,
device_context: ModuleDeviceContext = ModuleDeviceContext.default_context(),
):
super().__init__(
module,
device,
loss,
loggers,
log_name,
log_steps,
log_summary,
device_context,
)
self._optimizer = optimizer
self._num_accumulated_batches = num_accumulated_batches
self._optim_closure = optim_closure
self._accumulated = None
if self.device_context.use_mixed_precision:
if autocast is None or GradScaler is None:
raise type(amp_import_error)(
amp_import_error.msg
+ " autocast and GradScaler introduced in torch version 1.6.0."
)
if optim_closure is not None:
raise RuntimeError(
"Optimizer closures are not currently supported when training "
"using torch.cuda.amp.GradScaler."
)
self._scaler = GradScaler()
else:
self._scaler = None
@property
def optimizer(self) -> Optimizer:
"""
:return: the optimizer used to apply gradient updates with
"""
return self._optimizer
@property
def num_accumulated_batches(self) -> int:
"""
:return: number of batches to accumulate before updating the optimizer
"""
return self._num_accumulated_batches
@property
def optim_closure(self) -> Union[None, Callable]:
"""
:return: a closure passed into the optimizer on step
"""
return self._optim_closure
def _runner_setup(self):
self._module = self._module.train()
self._accumulated = 0
def _runner_batch(
self,
counter: int,
batch: int,
batch_size: int,
data: Any,
should_log: bool,
log_step: int,
):
# setup
self._accumulated += 1
data = self._run_funcs.to_device(data, self._device)
self._run_hooks.invoke_batch_start(counter, batch, batch_size, data)
# optimizer / gradients reset
if self._accumulated == self._num_accumulated_batches:
self._optimizer.zero_grad()
forward_context = (
autocast if self.device_context.use_mixed_precision else ExitStack
)
with forward_context():
# forward steps
pred = self._run_funcs.model_forward(data, self._module)
self._run_hooks.invoke_batch_forward(counter, batch, batch_size, data, pred)
# loss calculation
losses = self._loss(data, pred)
# scale loss by world size
losses[DEFAULT_LOSS_KEY] *= self.device_context.world_size
self._run_hooks.invoke_batch_loss(
counter, batch, batch_size, data, pred, losses
)
# backward steps
self._run_funcs.model_backward(losses, self._module, scaler=self._scaler)
self._run_hooks.invoke_batch_backward(
counter, batch, batch_size, data, pred, losses
)
# optimizer / gradients update
if self._accumulated == self._num_accumulated_batches:
if self.device_context.use_mixed_precision:
self._scaler.step(self._optimizer)
self._scaler.update()
else:
self._optimizer.step(closure=self._optim_closure)
self._accumulated = 0
self._run_hooks.invoke_batch_end(counter, batch, batch_size, data, pred, losses)
if should_log:
self._log_scalar(
"{}/Learning Rate".format(self._log_name),
get_optim_learning_rate(self._optimizer),
log_step,
)
return losses
def _runner_complete(
self, results: ModuleRunResults, should_log: bool, log_step: int
):
if should_log:
self._log_scalar(
"{}/Learning Rate Summary".format(self._log_name),
get_optim_learning_rate(self._optimizer),
log_step,
)
[docs]class ModuleTester(ModuleRunner):
"""
Container for running a module through evaluation over a given data loader
for specific settings.
| Lifecycle:
| - data batch size callback
| - data to device callback
| - batch start hook
| - data model forward callback
| - batch forward hook
| - loss calculation
| - batch loss hook
| - batch end hook
:param module: the model to run evaluation for
:param device: the default device to run evaluation on
(where data will be copied to)
:param loss: the loss functions callable used to calculate loss values after
executing a forward pass
:param loggers: list of loggers to log training results to
:param log_name: the key to store all log files under
:param log_steps: The number of steps (batches) to log at,
ex 100 will log every 100 batches
:param log_summary: True to log the final summary results after the run completes
:param device_context: ModuleDeviceContext with settings to enable mixed precision
using torch.cuda.amp or adjust losses when using DistributedDataParallel.
Default settings do not use mixed precision or account for DDP.
Will raise an exception if torch version does not support amp.
"""
def __init__(
self,
module: Module,
device: str,
loss: Union[LossWrapper, Callable[[Any, Any], Tensor]],
loggers: List[PyTorchLogger] = None,
log_name: str = "Test",
log_steps: int = 100,
log_summary: bool = True,
device_context: ModuleDeviceContext = ModuleDeviceContext.default_context(),
):
super().__init__(
module,
device,
loss,
loggers,
log_name,
log_steps,
log_summary,
device_context,
)
if self.device_context.use_mixed_precision:
if autocast is None or GradScaler is None:
raise type(amp_import_error)(
amp_import_error.msg
+ " autocast and GradScaler introduced in torch version 1.6.0."
)
def _runner_setup(self):
self._module = self._module.eval()
def _runner_batch(
self,
counter: int,
batch: int,
batch_size: int,
data: Any,
should_log: bool,
log_step: int,
):
with torch.no_grad():
# setup
data = self._run_funcs.to_device(data, self._device)
self._run_hooks.invoke_batch_start(counter, batch, batch_size, data)
forward_context = (
autocast if self.device_context.use_mixed_precision else ExitStack
)
with forward_context():
# forward steps
pred = self._run_funcs.model_forward(data, self._module)
self._run_hooks.invoke_batch_forward(
counter, batch, batch_size, data, pred
)
# loss steps
losses = self._loss(data, pred)
# scale loss by world size
losses[DEFAULT_LOSS_KEY] *= self.device_context.world_size
self._run_hooks.invoke_batch_loss(
counter, batch, batch_size, data, pred, losses
)
self._run_hooks.invoke_batch_end(
counter, batch, batch_size, data, pred, losses
)
return losses
def _runner_complete(
self, results: ModuleRunResults, should_log: bool, log_step: int
):
pass


---
title: "SPRING FLOWERING BULBS"
date: "2000-01-01 18:09:13 -06:00"
categories:
- adult
subtitle:
layout: "events"
event-image: "spring-flowering-bulbs.jpg"
event-dates: "THURSDAYS, SEPTEMBER 13 & 20"
event-location: "Red Butte Garden"
button-type: "register"
tickets-link: "https://continue.utah.edu/lifelong/class/llhg_519_spring_flowering_bulbs"
registration-info: " Phone Registration Available:801-587-5433
Lifelong Learning: LLHG 519"
instructor-info: "Michelle Cook"
member-cost: "$58"
public-cost: "$64"
event-notes: ""
tickets-button: ""
published: false
---
REGISTRATION REQUIRED
THURSDAYS
SEPTEMBER 13 & 20
6:30PM - 8:30PM
Plan and plant this fall for an early spring garden filled with color and diversity from a combination of spring blooming bulbs. Daffodils and tulips take center stage as we look at recommended varieties and discuss selecting various cultivars so you have continuous color from early to late spring. We’ll also discuss lesser known spring bloomers like Allium, Glory-of-the- Snow, Winter Aconite, Checkered Lily, and Snowdrops. Design, planting, maintenance, and how to force some of these little gems for winter flowers indoors will be discussed. Going dormant in summer, spring blooming bulbs are great landscape plants in Utah’s dry climate. Get excited for spring this fall and have flowers next year as soon as the snow melts. Co-sponsored with Lifelong Learning. Members pay $53 + sp fee, register for section 002.


In our Kerberos-Chat implementation, the first thing you must do is enter your username and password
During the pause, the frontend will receive a salt from the server based upon the username, then generate a hashed password based upon that
Next the frontend will take that generated data to form the first authentitcator and send it to the server
If the server approves, the user will be sent to the messaging page and the server will send back a TGT and the session key


This page allows copying an existing test run from one database to another.
Depending on the amount of the test data the whole operation may take a significant period of time.
The following table specifies the source and destination parameters
Database Host
Database Port
Database Name
Run Id
FROM:
TO:
Copy Run
Info console


Source,Target (1) Jeremy,Jennifer Valerian,Jeremy
last modified: 2017-02-06
Clément Levallois
Gephi handles the import of network files in a variety of formats:
To import any of these files, just choose the File → Open
menu
(looking for the complete, full featured csv import version? skip to here.)
The easiest is probably to just have a text file like this one:
Source,Target (1) Jeremy,Jennifer Valerian,Jeremy
<1>This line is mandatory. Don’t put a space between the comma and "Target"
Just write this in a text editor and save it as a file example.txt
To import this file in Gephi, go to File→Import Spreadsheet
. Then in the window:
In this window, make sure the box "Create missing nodes" is checked:
Let’s look again at the simple text file we used:
Source,Target Jeremy,Jennifer Valerian,Jeremy
There are 2 issues and several missing features, listed below:
Imagine our characters have first names and last names separated by a comma:
Dubois, Jeremy
Rodriguez, Valerian
This will look like:
Source,Target Dubois, Jeremy,Jeremy,Jennifer Rodriguez, Valerian,Dubois, Jeremy
Note that we have 3 commas per line, instead of one! In this mess, Gephi will not detect where the node labels start and end. The import will break.
The proper solution is to put node labels inside double quotes:
Source,Target "Dubois, Jeremy","Jeremy,Jennifer" "Rodriguez, Valerian","Dubois, Jeremy"
Be careful to use these straight " " double quotes, not the curvy ones: “ ” which French keyboards sometimes have.
Example: let’s imagine that one of our characters has a middle name: Jeremy "Danger" Dubois. Our text file will look like:
Source,Target "Dubois, Jeremy "Danger"","Jeremy,Jennifer" "Rodriguez, Valerian","Dubois, Jeremy "Danger""
These extra " " will make the Gephi import break:
The solution consists in adding an extra double quote in front of the double quotes. Our text file will look like:
Source,Target "Dubois, Jeremy ""Danger""","Jeremy,Jennifer" "Rodriguez, Valerian","Dubois, Jeremy ""Danger"""
This time, Gephi imports the network correctly:
To import attributes we will need to proceed differently.
We need 2 text files: one for the list of nodes, one for the list of relations (edges)
An example file with a list of nodes:
Id,Label,Date of Birth,Place of Birth,Years of experience,Rating (1) 3,"Dubois, Jeremy ""Danger""",17/09/1980,"Paris",8,9.27 1,"Jeremy,Jennifer",25/03/1978,"Tampa",8,4.34 45,"Rodriguez, Valerian",30/04/1985,"Berlin",5,6.66
Nodes must have at least an Id and a Label. Don’t put spaces after the commas
An example file with a list of edges:
Source,Target,Weight,Type,Where first met (1) 1,45,3,"undirected",London (2)
Edges must have at least a Source and Target. Other fields are optional.
"undirected", the alternative is "directed". Directed edges have arrow heads.
Let’s import the list of nodes first. File → Import Spreadsheet
in the next screen, we must be careful with a couple of things:
Then we can import the file with the list of relations. File → Import Spreadsheet
Nodes
header must be at least Id,Label
Edges
header must be at least Source,Target
To have an attribute for the "thickness" or "strength" of a relation, this attribute must be called "Weight".
Want arrows on your links? Add an attribute "Type", with value "Directed"
Don’t wan’t arrows? Add an attribute "Type", with value "Undirected"
Types of attributes: which to choose?
Textual attribute: String
. Nodes sharing the same textual value can be colored the same, or filtered together…
Numerical attribute: Integer
, Double
or Float
. Nodes can be resized according to their value, or colorized in a gradient. Filters can be applied based on the range of values.
Boolean attribute: just a true
or false
value. Useful for filtering out some nodes which are true or false on some parameter.
Other types of attributes: not needed for the moment.
Visit the Gephi group on Facebook to get help,
or visit the website for more tutorials