Section paragraph.
-
General Considerations
- Transition: Different types of systems have unique considerations when replacing a legacy application
- Risk Management: Risk management is an important part software/system lifecyles
- Value Proposition A strategic statement defining the purpose of the system.
- Stakeholders: Stakeholders are the interested parties to the system and it's lifecycle.
- Cold Fusion: Considerations specific to Cold Fusion.
-
Business Analysis
- Domain Analysis: Understanding the Domain
- Personas: Understanding the Actors, thier motivations, their purpose.
- Context: Understanding the environment
-
Requirements
- Functional Requirments: Functionality that implements the intended utility of the application.
- Non-functional Requirmeents: Often overlooked, non-functional requirements are mostly related to the non User Story related behavior of the application.
-
User Interface (UI) Design:
- Wireframes: Visual representations of the layout and structure of user interfaces.
- Mockups: Detailed visual designs of the user interfaces, showcasing the look and feel of the software.
- Prototypes: Interactive models or simulations of the software interfaces to demonstrate functionality.
- User flow diagrams: Diagrams illustrating the steps and paths a user takes within the software.
- Style guides: Guidelines for maintaining consistency in UI design, including fonts, colors, and element styles.
-
Database Design:
- Entity-Relationship Diagrams (ERDs): Visual representations of entities, relationships, and attributes in a database.
- Database schemas: Structure and organization of tables, fields, and relationships in the database.
- Data dictionaries: Documentation specifying data types, descriptions, and usage of database elements.
- Index and query optimization plans: Strategies to optimize database performance through indexing and query tuning.
-
System Architecture Design:
- High-level system architecture diagram: Overview of the software system's components and their interactions.
- Component diagrams: Visual representation of the software's modular components and their dependencies.
- Deployment diagrams: Illustration of how software components are deployed across hardware or cloud infrastructure.
- Infrastructure design: Planning the required hardware, servers, networking, and storage for the software system.
- Integration patterns: Strategies and patterns for integrating different software components or services.
-
Software Design:
- Class diagrams: Visualization of classes, their attributes, methods, and relationships in object-oriented programming.
- Sequence diagrams: Visual representation of interactions between objects or components in a specific scenario.
- State diagrams: Modeling the different states and transitions of an object or system during its lifecycle.
- Use case diagrams: Illustration of actors, use cases, and their relationships to describe system functionality.
- Activity diagrams: Flowcharts depicting the workflow or process of a system through various activities.
- Dataflow diagrams: Graphical representation of how data moves through a system, showing processes, inputs, and outputs.
- Context diagrams: Overview of the system's boundary, external entities, and data flows with the outside world.
- Scope diagrams: Visualization of the system's boundaries, interfaces, and components within the defined scope.
- User Stories: Brief, high-level descriptions of system features or functionality from a user's perspective.
- Use Cases diagrams: Descriptions of interactions between actors and the system to accomplish specific goals or tasks.
-
API Design:
- API specification documents (e.g., OpenAPI/Swagger): Documentation defining the structure and functionality of APIs.
- API contract definitions (e.g., JSON Schema): Standards or formats that describe the structure of API payloads.
- Endpoint definitions: Specification of API endpoints, including their URLs, methods, and parameters.
- Request/response examples: Sample payloads illustrating expected data formats for API requests and responses.
- API versioning strategy: Approach to versioning APIs to manage backward compatibility and changes.
-
Security Design:
- Security architecture diagrams: Visual representation of security controls and components within the software system.
- Threat models: Identification and assessment of potential security threats and vulnerabilities.
- Authentication and authorization mechanisms: Techniques and protocols for user authentication and access control.
- Encryption protocols and algorithms: Methods for encrypting data in transit or at rest to ensure confidentiality.
- Access control lists (ACLs): Lists specifying permissions and privileges for users or groups.
-
Testing Design:
- Test plans: Detailed strategies and approaches for testing the software system.
- Test cases: Specific scenarios or conditions to be tested to ensure software functionality and reliability.
- Test scripts: Step-by-step instructions for executing tests and validating expected outcomes.
- Test data sets: Predefined sets of data used to perform tests and verify the software's behavior.
- Test environments configuration: Preparation and setup of dedicated environments for testing activities.
-
Documentation:
- System documentation: Comprehensive documentation covering system architecture, components, and usage.
- Technical specifications: Detailed documentation describing the technical aspects of the software system.
- User manuals: Guides or instructions aimed at end-users to help them understand and use the software.
- API documentation (e.g., API reference, developer guides): Documentation specifically focused on APIs, including reference documentation and guidance for developers.
- Release notes: Documentation highlighting changes, bug fixes, and new features in software releases.
-
Performance and Scalability Design:
- Performance test plans: Strategies and methodologies for testing and measuring the performance of the software system.
- Load balancing strategies: Techniques for distributing workload across multiple servers or resources to improve performance.
- Caching mechanisms: Methods for storing frequently accessed data to enhance system performance.
- Horizontal and vertical scaling approaches: Strategies for scaling the software system either by adding more servers (horizontal) or enhancing existing resources (vertical).
- Resource utilization monitoring: Techniques to measure and monitor the consumption of system resources such as CPU, memory, and disk.
-
Error Handling and Logging Design:
- Error handling strategies: Approaches to manage and handle errors or exceptions that occur during software execution.
- Exception handling guidelines: Rules and practices for catching, handling, and reporting exceptions in the software.
- Logging formats and levels: Specification of log formats and the level of detail to be captured in log messages.
- Log aggregation and analysis mechanisms: Tools and techniques for collecting and analyzing logs to identify issues or patterns.
- Error reporting and notification systems: Methods for reporting errors or sending notifications to system administrators or users.
Please note that the inclusion of design artifacts may vary based on project requirements and the specific software development methodology used.


You are now a manager at CarFreeAtoZ. This management console allows you to...
View Feedback View User Activity View Alerts


I have some rogue input Nodes appearing on my local Emoncms, They all show as inactive. I have no idea why they have appeared. The existing Nodes are all OK.
How can I delete them?
Archived Forum |
|
how to remove rogue node IDsSubmitted by borpin123 on Tue, 09/06/2015 - 09:29I have some rogue input Nodes appearing on my local Emoncms, They all show as inactive. I have no idea why they have appeared. The existing Nodes are all OK. How can I delete them? » |
Re: how to remove rogue node IDs
You should try and locate the cause and/or prevent reoccurrence before completely removing them as they may well keep popping up again!
Are you running emonhub? if so does the log show anything odd? is the loglevel set to debug?
What access points do you have eg rfm2pi, serial or urls etc?
Which node ids are created ? do they have many inputs? do the inputs have values?
Is it possible you are getting crosstalk from another rfm "group" ? could you try another group setting?
Paul
Re: how to remove rogue node IDs
I get them frequently and some data is recognisable as belonging to other nodes which leads me to believe there are issues with RFM12B error checking I just go in occasionally and delete the errant readings
Re: how to remove rogue node IDs
I do as well have extra inactive nodes with values that don't make sense to me.
I am transmitting (EmonTx) one node only and the Raspi pushes it to emoncms.org
Base station is RFM12Pi
Re: how to remove rogue node IDs
Could this simply be bad data packets that are sufficiently good to be decoded (if you see what I mean)?
Could EmonCMS have a setting that allows you to specify which nodes are good and to ignore anything else?
I'm using one emonTX and one emonTH back to a raspi with the RFM board on the Pi.
@pb66 - how would you change group? Would each node need re-programming?
I think I am using emonhub - how do I set the debug level? Where would the log be? (I am using the old HDD image (upgraded).
Re: how to remove rogue node IDs
Rogue RF packets was my suspicion, I just wasn't sure if you were using RF,
Although here maybe ways to "limit" the inputs (are you using emoncms.org or a local install?) this will not prevent rogue inputs with live node ids contaminating your data, which is what I suspect your High Power reading thread is reporting.
The visability of the unused nodes created are an easy target for diagnosis as any rogue inputs using used nodeids will be hard to locate, BUT the hidden one's that effect your data are the most important to eliminate as they effect your data, as Br1an says you can just delete the others, they have no operational impact.
The group is set in each nodes sketch, so yes unfortunately each change of group required each node to have new firmware uploaded, a pain I know. setting the grooup id at the base is much easier (in emonhub.conf)
If you are using emonhub then you can edit loglevel = DEBUG in /etc/emonhub/emonhub.conf (emonhub -c on later SDcard images) the /var/log/emonhub/emonhub.log (emonhub -dl on later SDcard images) will record the movement of every packet it see's so you can use it to trace the root.
You may also find it useful to set the rfm2pi "quiet mode" off for a while to observe the packets that are not being passed on by the rfm2pi (set quiet = True in the [[RFM2Pi]] settings of emonhub.conf) BUT I think early rfm2pi firmwares (ie pre rfm2pi with rfm69) had that feature disabled so may need a firmware update to use.
Another "tool" that may help is to temporarily set the group id of the base to group 0, this is a promiscuous mode and will not filter received data by node id. (It may be best to do this using minicom or similar as it may let in stray data), in promiscuous mode the data frame is prefixed by the groupid so you may be able to select a quite group id rather than unloading sketches in a random trial and error fashion.
see Spurious RF sent to emoncms for a similar issue that a group change cured,
Paul
Re: how to remove rogue node IDs
Ok so first things first, the emonhub.log has the folowing error message;
Warning: fopen(/var/lib/phpfina/11.dat): failed to open stream: Permission denied in /var/www/emoncms/Modules/feed/engine/PHPFina.php on line 118
Permissions are;
drwxr-xr-x 2 www-data root 4096 Apr 3 22:45 phpfina
I remember having some issues around this when I upgraded to 8.4 and emonhub.
What should these be?
Re: how to remove rogue node IDs
Oh my, I deleted the emonhub.log files as they were huge and now I am not getting a log at all. I (Perhaps) foolishly thought the logger would recreate them :(
Re: how to remove rogue node IDs
Plot thickens!
I wonder if the emonhub is actually feeding emoncms. How can I tell? Recreated log file and changed group and owner. Rebooted and got this log
2015-06-15 21:00:31,431 INFO EmonHub Pre-Release Development Version (rc1.2)
2015-06-15 21:00:31,446 INFO Opening hub...
2015-06-15 21:00:31,448 INFO Logging level set to DEBUG
2015-06-15 21:00:31,451 INFO Creating EmonHubEmoncmsReporter 'emonCMS'
2015-06-15 21:00:31,466 INFO Set up reporter 'emonCMS' (buffer: memory | size: 1000)
2015-06-15 21:00:31,476 INFO Setting emonCMS url: http://localhost/emoncms
2015-06-15 21:00:31,479 INFO Setting emonCMS apikey: set
2015-06-15 21:00:31,482 INFO Creating EmonHubJeeInterfacer 'RFM2Pi'
2015-06-15 21:00:31,497 DEBUG Opening serial port: /dev/ttyAMA0 @ 38400 bits/s
2015-06-15 21:00:35,574 WARNING Device communication error - check settings
2015-06-15 21:00:35,577 INFO Setting RFM2Pi frequency: 433 (4b)
2015-06-15 21:00:36,580 INFO Setting RFM2Pi group: 210 (210g)
2015-06-15 21:00:37,584 INFO Setting RFM2Pi quiet: 1 (1q)
2015-06-15 21:00:38,596 INFO Setting RFM2Pi baseid: 15 (15i)
Note it is opening the port at 38400. Should this be 9600?
From the other thread I was pointed to it talks about modifying the rfm12 file. is that the firmware for the board or the pi file? If so where is it?
Cheers
Re: how to remove rogue node IDs
So I am getting no inputs now.
Re: how to remove rogue node IDs
You can define the serial baud in the [[RFM2Pi]] settings in emonhub.conf by adding com_baud = 9600 to the [[[init_settings]]]. The auto baud selection can occasionally be a bit quirky, it works most of the time but if you are having issues or just for peace of mind it can be best to define it if known.
The emonhub.log files get created at start up and if you are running read-only the logfiles are held in RAM so the folder gets recreated at each boot too. In future you can use emonhub -rl (rl = reset logs) if using the last pre-built image or just ' echo "" > /var/log/emonhub/emonhub.log '.
The stray "permission issue" in emonhub's log is purely recorded as it was emoncms's response to emonhub posting data, that is probably unrelated and will need further checks if it reoccurs.
Paul
Re: how to remove rogue node IDs
Brilliant. I have data again! Why on earth did it stop having run perfectly (well sort of) for ages?
Now, where is the emoncms.log? I do not seem to have one. if I create it, what shoudl the group/owner be?
Re: how to remove rogue node IDs
Hi Paul, The issue was that when I deleted the logs, emonhub did not seem able to recreate them. I am running this on a HDD (the old HDD image upgraded to 8.4) as I run other stuff off this as well.
I have a feeling it was because I deleted the log file but did not restart the daemon although I thought I had rebooted so the problem may have been I did not delete the directory.
What is the best version of emonhub to run. This reports as rc1.2
Re: how to remove rogue node IDs
rc1.2 is the current (non-emonpi) version
Paul
Re: how to remove rogue node IDs
Ok Thanks.


Villain get the money like curls
They just tryin' to get a nut like squirrels in his mad world
Land of milk and honey with the swirls
Where reckless naked girls get necklaces of pearls
Compliments of the town jeweler
Left back now-schooler tryin' to sound cooler
On the microphone known as the crown ruler
Never lied to ma when we said we found the moolah
Five-hundred somethin' dollars layin' right there in the street
Huh, now let's try and get somethin' to eat
Then he turned four and started flowin' to the poor
That's about when he first started going raw
Kept the 'dro in the drawer
A rhymin' klepto who couldn't go up in the store no more
His life is like a folklore legend
Why you so stiff, you need to smoke more, bredrin
Instead of trying to riff with the broke war veteran
Spliff made him swore he saw heaven he was seven
Yup, you know it, growin' up too fast
Showin' up to class with Moet in a flask
He ask the teacher if he leave will he pass
His girl is home alone he tryin' to get the▬▬▬▬
If you want to sip get a paper water fountain glass
How I'm 'posed to know where your mouth been last
Hands so fast he can out-spin the Flash
Known to smoke a whole mountain of hash to the ash
Boom-bash leave the room with the stash
Assume it's in a smash
Doom get the cash


Easier for Students
Keep students connected and organized, while promoting collaboration among students and teachers!
All Assignments, Notes, and Tests in One Place
By putting all class materials in one place, Students can easily prepare themselves for class and study the subject material at home.
Automatic Planner
Tests, Quizzes, and assignments show up automatically, so students do not have to worry about remembering everything that they need for class.
Collaborate with Classmates
Public discussion about class material enables students to quickly clear confusion on assignments or subject material.
Easy to Use
Designed like social networks that students know and love, ClassSpark is instantly familiar to students and requires no training whatsoever.
Clear Deadlines
Because each post has a date attached, students will never say that they did not know when an assignment is due. It's spelled out in plain text.


CRTS Development Tests
float widget test
Webtop test page
Scope tests
Spectrum Tests
back to top index


Output colorization¶
As of version 6.0.0, Miller supports output colorization. This includes colors per se, as well as bold, underline, and/or reverse. As of version 6.1.0, the default is bold+underline for keys -- this works well as a default regardless of your terminal's color settings -- with further customization options as described below:
Things having colors¶
- Keys in CSV header lines, JSON keys, etc
- Values in CSV data lines, JSON scalar values, etc
- "PASS" and "FAIL" in regression-test output
- Some online-help strings
Rules for colorization¶
-
By default, colorize output only if writing to stdout and stdout is a TTY.
- Example: color:
mlr --csv cat foo.csv
- Example: no color:
mlr --csv cat foo.csv > bar.csv
- Example: no color:
mlr --csv cat foo.csv | less
- Example: color:
-
NIDX output format (plain text, Unix-style, with implicitly positionally indexed fields) is not colored. This is because in other formats, keys are one color and values are another. For NIDX, all output is values.
Mechanisms for colorization¶
- Miller uses ANSI escape sequences only.
- Doesn't try to check to see whether the terminal is capable of 256-color ANSI vs 16-color ANSI. Note that if colors are in the range 0..15 then 16-color ANSI escapes are used, so this is in the user's control.
How you can control colorization¶
-
Suppression/unsuppression:
export MLR_NO_COLOR=true
means Miller won't color even when it normally would.export MLR_ALWAYS_COLOR=true
means Miller will color even when it normally would not. For example, you might want to use this when pipingmlr
output toless -r
.- Command-line flags
--no-color
or-M
,--always-color
or-C
. - On Windows, replace
export
withset
-
Color choices can be specified by using environment variables or command-line flags, with values 0..255:
export MLR_KEY_COLOR=208
export MLR_VALUE_COLOR=33
- On Windows, replace
export
withset
- Likewise for
MLR_PASS_COLOR
,MLR_FAIL_COLOR
,MLR_HELP_COLOR
,MLR_REPL_PS1_COLOR
, andMLR_REPL_PS2_COLOR
. - Command-line flags
--key-color 208
,--value-color 33
, etc., and likewise for--pass-color
,--fail-color
,--repl-ps1-color
,--repl-ps2-color
, and--help-color
. - This is particularly useful if your terminal's background color clashes with current settings.
If environment-variable settings and command-line flags are both provided, the latter take precedence.
How to see available color codes¶
Use mlr --list-color-names
to see available names, like orchid
. Colors can also be specified
using numbers in the range 0..255, like 170: please see mlr --list-color-codes
. You can also use
bold
, underline
, and/or reverse
. Additionally, combinations of all of these can be joined
with a -
, like red-bold
, bold-170
, bold-underline
, etc.


tensorbuilder.builders module
Tensor Builder
TensorBuilder is light wrapper over TensorFlow that enables you to easily create complex deep neural networks using the Builder Pattern through a functional fluent immutable API. The main goals of TensorBuilder are
- Be a light-weight wrapper around TensorFlow fully compatible with its functions.
- Let the user name and inspect the tensorflow Variables generated by TensorBuilder
- Enable the user to create complex branched topologies while maintaining a fluent API (see Builder.branch)
TensorBuilder has a small set of primitives that enable you to express complex networks while maintaining a consistent API. Its branching mechanism enables you to express through the structure of your code the structure of the network, even when you have complex sub-branching expansions and reductions, all this while keeping the same fluid API.
TensorBuilder takes inspiration from prettytensor but its internals are simpler, its API is smaller but equally powerfull, its branching mechanism is more expresive and doesn't break the fluent API, and its immutable nature helps avoid a lot of conceptual complexity.
Installation
Tensor Builder assumes you have a working tensorflow
installation. We don't include it in the requirements.txt
since the installation of tensorflow varies depending on your setup.
From source
git clone https://github.com/cgarciae/tensorbuilder.git
cd tensorbuilder
python setup.py install
From pip
Coming soon!
Getting Started
Create neural network with a [5, 10, 3] architecture with a softmax
output layer and a tanh
hidden layer through a Builder and then get back its tensor:
import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) h = ( x.builder() .connect_layer(10, fn=tf.nn.tanh) .connect_layer(3, fn=tf.nn.softmax) .tensor )
Branching
If you are sufficiently familiar with tensorflow or use prettytensor then you might appreciate the branching capabilities of Tensor Builder in this (overly complex) example
import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) keep_prob = tf.placeholder(tf.float32) h = ( x.builder() .connect_layer(10) .branch(lambda root: [ root .connect_layer(3, fn=tf.nn.relu) , root .connect_layer(9, fn=tf.nn.tanh) .branch(lambda root2: [ root2 .connect_layer(6, fn=tf.nn.sigmoid) , root2 .map(tf.nn.dropout, keep_prob) .connect_layer(8, tf.nn.softmax) ]) ]) .connect_layer(6, fn=tf.nn.sigmoid) .tensor )
Documentation
The main documentaion is in the tensorbuilder module. The documentation for the complete project is here.
Examples
Here are the examples for each method of the API. If you are understand all examples, then you've understood the complete API.
""" # Tensor Builder TensorBuilder is light wrapper over TensorFlow that enables you to easily create complex deep neural networks using the Builder Pattern through a functional [fluent](https://en.wikipedia.org/wiki/Fluent_interface) [immutable](https://en.wikipedia.org/wiki/Immutable_object) API. The main goals of TensorBuilder are * Be a light-weight wrapper around TensorFlow fully compatible with its functions. * Let the user name and inspect the tensorflow Variables generated by TensorBuilder * Enable the user to create complex branched topologies while maintaining a fluent API (see [Builder.branch](http://cgarciae.github.io/tensorbuilder/tensorbuilder.m.html#tensorbuilder.builders.Builder.branch)) TensorBuilder has a small set of primitives that enable you to express complex networks while maintaining a consistent API. Its branching mechanism enables you to express through the structure of your code the structure of the network, even when you have complex sub-branching expansions and reductions, all this while keeping the same fluid API. TensorBuilder takes inspiration from [prettytensor](https://github.com/google/prettytensor) but its internals are simpler, its API is smaller but equally powerfull, its branching mechanism is more expresive and doesn't break the fluent API, and its immutable nature helps avoid a lot of conceptual complexity. ## Installation Tensor Builder assumes you have a working `tensorflow` installation. We don't include it in the `requirements.txt` since the installation of tensorflow varies depending on your setup. #### From source 1. `git clone https://github.com/cgarciae/tensorbuilder.git` 2. `cd tensorbuilder` 3. `python setup.py install` #### From pip Coming soon! ## Getting Started Create neural network with a [5, 10, 3] architecture with a `softmax` output layer and a `tanh` hidden layer through a Builder and then get back its tensor: import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) h = ( x.builder() .connect_layer(10, fn=tf.nn.tanh) .connect_layer(3, fn=tf.nn.softmax) .tensor ) ## Branching If you are sufficiently familiar with tensorflow or use prettytensor then you might appreciate the branching capabilities of Tensor Builder in this (overly complex) example import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) keep_prob = tf.placeholder(tf.float32) h = ( x.builder() .connect_layer(10) .branch(lambda root: [ root .connect_layer(3, fn=tf.nn.relu) , root .connect_layer(9, fn=tf.nn.tanh) .branch(lambda root2: [ root2 .connect_layer(6, fn=tf.nn.sigmoid) , root2 .map(tf.nn.dropout, keep_prob) .connect_layer(8, tf.nn.softmax) ]) ]) .connect_layer(6, fn=tf.nn.sigmoid) .tensor ) ## Documentation The main documentaion is in the [tensorbuilder module](http://cgarciae.github.io/tensorbuilder/tensorbuilder.m.html). The documentation for the complete project is [here](http://cgarciae.github.io/tensorbuilder/). ## Examples Here are the examples for each method of the API. If you are understand all examples, then you've understood the complete API. """ __version__ = "0.0.1" import tensorflow as tf import numpy as np import functools from decorator import decorator # Decorators @decorator def _immutable(method, self, *args, **kwargs): """ Decorator. Passes a copy of the entity to the method so that the original object remains un touched. Used in methods to get a fluent immatable API. """ return method(self.copy(), *args, **kwargs) class Builder(object): """ The Builder class is a wrapper around a Tensor. Most of its method are immutable in the sense that they don't modify the caller object but rather always make a copy, they also tend to return a Builder so you you can keep fluently chaining methods. To create a builder from a method you have these options: 1. Use the `tensorbuilder.builders.builder` function tb.builder(tensor) 2. Use the monkey-patched method on the Tensor class tensor.builder() """ def __init__(self, tensor, variables={}): super(Builder, self).__init__() self.tensor = tensor """A `tensorflow` Tensor.""" self.variables = variables """A dictionary that accumulates the **tf.Variable** tensors generated during the building process, it has the form {tensor_name: String -> tensor: tf.Variable}. Since functions like `tensorbuilder.builders.Builder.connect_layer` hides you the complexity of creating the **bias** and **weights** of your network, `tensorbuilder.builders.Builder` stores them in this field. The methods of this class enables you to set the names of these variables, but take into account that the final name is actually the name of the tensor, with is set by `tensorflow`. Check their documentation to see how the name is defined.""" def copy(self): """Returns a copy of this Builder""" return Builder(self.tensor, self.variables.copy()) @_immutable def connect_weights(builder, size, name=None, weights_name=None): """ `@_immutable` Let **x** be `tensorbuilder.builders.Builder.tensor` of shape **[m, n]**, and let **w** be a **tf.Variable** of shape **[n, size]**. Then `builder.connect_weights(size)` computes `tf.matmul(x, w)`. The returned `tensorbuilder.builders.Builder` has **w** stored inside `tensorbuilder.builders.Builder.variables`. **Parameters** * `size`: an `int` representing the size of the layer (number of neurons) * `name`: the name of the tensor (default: "connect_weights") * `weights_name`: the name of the **w** tensor of type `tf.Variable`. **Return** * `tensorbuilder.builders.Builder` **Examples** The following builds `tf.matmul(x, w)` import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) z = x.builder().connect_weights(3, weights_name="weights") """ m = int(builder.tensor.get_shape()[1]) n = size w = tf.Variable(tf.random_uniform([m, n], -1.0, 1.0), name=weights_name) var_name = weights_name if weights_name else w.name builder.variables[var_name] = w builder.tensor = tf.matmul(builder.tensor, w, name=name) return builder @_immutable def connect_bias(builder, name=None, bias_name=None): """ `@_immutable` Let **x** be `tensorbuilder.builders.Builder.tensor` of shape **[m, n]**, and let **b** be a **tf.Variable** of shape **[n]**. Then `builder.connect_bias()` computes `tf.add(x, b)`. The returned `tensorbuilder.builders.Builder` has **b** stored inside `tensorbuilder.builders.Builder.variables`. **Parameters** * `name`: the name of the tensor (default: "connect_bias") * `bias_name`: the name of the `w` tensor of type `tf.Variable` (default: "b"). **Return** * `tensorbuilder.builders.Builder` **Examples** The following builds `tf.matmul(x, w) + b` import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) z = ( x.builder() .connect_weights(3, weights_name="weights") .connect_bias(bias_name="bias") ) Note, the previous is equivalent to using `tensorbuilder.builders.Builder.connect_layer` like this z = ( x.builder() .connect_layer(3, weights_name="weights", bias_name="bias") ) """ m = int(builder.tensor.get_shape()[1]) b = tf.Variable(tf.random_uniform([m], -1.0, 1.0), name=bias_name) var_name = bias_name if bias_name else b.name builder.variables[var_name] = b builder.tensor = tf.add(builder.tensor, b, name=name) return builder @_immutable def connect_layer(builder, size, fn=None, name=None, weights_name=None, bias=True, bias_name=None): """ `@_immutable` Let **x** be `tensorbuilder.builders.Builder.tensor` of shape **[m, n]**, let **w** be a **tf.Variable** of shape **[n, size]**, let **b** be a **tf.Variable** of shape **[n]**, and **fn** be a function from a tensor to a tensor. Then `builder.connect_layer(size, fn=fn)` computes `fn(tf.matmul(x, w) + b)`. If **fn** is not present the layer is linear. Note that **fn** must expose the keyword/named argument `name`, this is compatible with the tensorflow API. The returned `tensorbuilder.builders.Builder` has **b** and **w** stored inside `tensorbuilder.builders.Builder.variables`. **Parameters** * `fn`: a function of type `tensor -> tensor`. If `fn` is `None` then its not applied, resulting in just a linear trasformation. (default: None) * `size`: an `int` representing the size of the layer (number of neurons) * `name`: the name of the tensor (default: `"layer"`) * `bias`: determines where to use a bias **b** or not (default: `True`) * `weights_name`: the name of the `w` tensor of type `tf.Variable` (default: `None`) * `bias_name`: the name of the `w` tensor of type `tf.Variable` (default: `None`) **Return** * `tensorbuilder.builders.Builder` **Examples** The following builds the computation `tf.nn.sigmoid(tf.matmul(x, w) + b)` import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) h = ( x.builder() .connect_layer(3, fn=tf.nn.sigmoid) ) The previous is equivalent to using h = ( x.builder() .connect_weights(3) .connect_bias() .map(tf.nn.sigmoid) ) You can chain various `connect_layer`s to get deeper neural networks import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 40]) h = ( x.builder() .connect_layer(100, fn=tf.nn.tanh) .connect_layer(30, fn=tf.nn.softmax) ) """ builder = builder.connect_weights(size, weights_name=weights_name) if bias: builder = builder.connect_bias(bias_name=bias_name) if fn: builder.tensor = fn(builder.tensor, name=name) return builder @_immutable def map(builder, fn, *args, **kwargs): """ `@_immutable` Let **x** be `tensorbuilder.builders.Builder.tensor` and **fn** be a function from a tensor to a tensor. Then `builder.map(fn)` computes `fn(x)`. All extra positional and named arguments are forwarded to **fn** such that builder.map(fn, arg1, arg2, ..., kwarg1=kwarg1, kwarg2=kwarg2, ...) internally results in builder.tensor = fn(builder.tensor, arg1, arg2, ..., kwarg1=kwarg1, kwarg2=kwarg2, ...) **Parameters** * `fn`: a function of type `tensor -> tensor`. **Return** * `tensorbuilder.builders.Builder` **Examples** The following constructs a neural network with the architecture `[40 input, 100 tanh, 30 softmax]` and and applies `dropout` to the tanh layer import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 40]) keep_prob = tf.placeholder(tf.float32) h = ( x.builder() .connect_layer(100, fn=tf.nn.tanh) .map(tb.nn.dropout, keep_prob) .connect_layer(30, fn=tf.nn.softmax) ) """ builder.tensor = fn(builder.tensor, *args, **kwargs) return builder @_immutable def then(builder, fn): """ `@_immutable` Expects a function **fn** with type `builder -> builder`. This method is used primarily to manupilate the Builder with very fine grain control through the fluent immutable API. **Parameters** * `fn`: a function of type `builder -> builder`. **Return** * `tensorbuilder.builders.Builder` ** Example ** The following *manually* constructs the computation `tf.nn.sigmoid(tf.matmul(x, w) + b)` while updating the `tensorbuilder.tensorbuiler.Builder.variables` dictionary. import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 40]) keep_prob = tf.placeholder(tf.float32) def sigmoid_layer(builder, size): m = int(builder.tensor.get_shape()[1]) n = size w = tf.Variable(tf.random_uniform([m, n], -1.0, 1.0)) b = tf.Variable(tf.random_uniform([n], -1.0, 1.0)) builder.variables[w.name] = w builder.variables[b.name] = b builder.tensor = tf.nn.sigmoid(tf.matmul(builder.tensor, w) + b) return builder h = ( x.builder() .then(lambda builder: sigmoid_layer(builder, 3)) ) Note that the previous if equivalent to h = ( x.builder() .connect_layer(3, fn=tf.nn.sigmoid) ) """ return fn(builder) @_immutable def branch(builder, fn): """ `@_immutable` Expects a function **fn** with type `Builder -> list( Builder | BuilderTree )`. This method enables you to *branch* the computational graph so you can easily create neural networks with more complex topologies. You can later **Parameters** * `fn`: a function of type `Builder -> list( Builder | BuilderTree )`. **Return** * `tensorbuilder.builders.BuilderTree` ** Example ** The following will create a sigmoid layer but will branch the computation at the logit (z) so you get both the output tensor `h` and `trainer` tensor. Observe that first the logit `z` is calculated by creating a linear layer with `connect_layer(1)` and then its branched out import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) y = tf.placeholder(tf.float32, shape=[None, 1]) [h, trainer] = ( x.builder() .connect_layer(1) .branch(lambda z: [ z.map(tf.nn.sigmoid) , z.map(tf.nn.sigmoid_cross_entropy_with_logits, y) .map(tf.train.AdamOptimizer(0.01).minimize) ]) .tensors() ) Note that you have to use the `tensorbuilder.builders.BuilderTree.tensors` method from the `tensorbuilder.builders.BuilderTree` class to get the tensors back. Remember that you can also contain `tensorbuilder.builders.BuilderTree` elements when you branch out, this means that you can keep branching inside branch. Don't worry that the tree keep getting deeper, `tensorbuilder.builders.BuilderTree` has methods that help you flatten or reduce the tree. The following example will show you how create a (overly) complex tree and then connect all the leaf nodes to a single `sigmoid` layer import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) keep_prob = tf.placeholder(tf.float32) h = ( x.builder() .connect_layer(10) .branch(lambda base: [ base .connect_layer(3, fn=tf.nn.relu) , base .connect_layer(9, fn=tf.nn.tanh) .branch(lambda base2: [ base2 .connect_layer(6, fn=tf.nn.sigmoid) , base2 .map(tf.nn.dropout, keep_prob) .connect_layer(8, tf.nn.softmax) ]) ]) .connect_layer(6, fn=tf.nn.sigmoid) ) ** See Also ** * `tensorbuilder.builders.BuilderTree` * `tensorbuilder.builders.BuilderTree.connect_layer` * `tensorbuilder.builders.BuilderTree.builders` * `tensorbuilder.builders.BuilderTree.tensors` """ branches = fn(builder) return BuilderTree(branches) @_immutable def _leafs(builder): """A generator function that yields the builder, used by `tensorbuilder.builders.BuilderTree.leafs` of `tensorbuilder.builders.BuilderTree`""" yield builder class BuilderTree(object): """ BuilderTree is a class that enable you to perform computations over a complex branched builder. It contains methods to get all the leaf `tensorbuilder.builders.Builder` nodes, connect all the leaf nodes to a single layer, etc. """ def __init__(self, branches): super(BuilderTree, self).__init__() self.branches = branches """ A list that can contain elements that are of type `tensorbuilder.builders.Builder` or `tensorbuilder.builders.BuilderTree`. """ def copy(self): branches = self.branches[:] return BuilderTree(branches) def builders(self): """ Returns a flattened list `tensorbuilder.builders.Builder`s contained by this tree. The whole result is flattened in case of sub-elements are also `tensorbuilder.builders.BuilderTree`s. **Return** * `list( tensorbuilder.builders.Builder )` ** Example ** import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) y = tf.placeholder(tf.float32, shape=[None, 1]) [h_builder, trainer_builder] = ( x.builder() .connect_layer(1) .branch(lambda z: [ z.map(tf.nn.sigmoid) , z.map(tf.nn.sigmoid_cross_entropy_with_logits, y) .map(tf.train.AdamOptimizer(0.01).minimize) ]) .builders() ) """ return [builder for builder in self._leafs() ] def tensors(self): """ Same as `tensorbuilder.builders.BuilderTree.builders` but extracts the tensor from each `tensorbuilder.builders.Builder`. **Return** * `list( tf.Tensor )` ** Example ** import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) y = tf.placeholder(tf.float32, shape=[None, 1]) [h_tensor, trainer_tensor] = ( x.builder() .connect_layer(1) .branch(lambda z: [ z.map(tf.nn.sigmoid) , z.map(tf.nn.sigmoid_cross_entropy_with_logits, y) .map(tf.train.AdamOptimizer(0.01).minimize) ]) .tensors() ) """ return [builder.tensor for builder in self._leafs() ] @_immutable def connect_layer(tree, size, fn=None, name="layer", bias=True, bias_name=None): """ Connects all the leaf `tensorbuilder.builders.Builder` nodes of this tree to a single layer. The order of computation is done as follows 1. Each leaf builder node is linearly connected to a layer of size `size` with no bias. 2. All these layers of size `size` are added together (reduced with +) 3. If `bias` is `True` then a bias added 4. If `fn` is not `None` then the function `fn` is mapped ** Parameters ** * `fn`: a function of type `tensor -> tensor`. If `fn` is `None` then its not applied, resulting in just a linear trasformation. (default: None) * `name`: the name of the tensor (default: `"layer"`) * `bias`: determines where to use a bias **b** or not (default: `True`) * `bias_name`: the name of the `w` tensor of type `tf.Variable` (default: `None`) ** Examples ** # The following example shows you how to connect two tensors (rather builders) of different shapes to a single `softmax` layer of shape [None, 3] import tensorflow as tf import tensorbuilder as tb a = tf.placeholder(tf.float32, shape=[None, 8]).builder() b = tf.placeholder(tf.float32, shape=[None, 5]).builder() h = ( tb.branches([a, b]) .connect_layer(3, fn=tf.nn.softmax) ) The next example show you how you can use this to pass the input layer directly through one branch, and "analyze" it with a `tanh layer` filter through the other, both of these are connect to a single `softmax` output layer import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) h = ( x.builder() .branch(lambda x: [ x , x.connect_layer(10, fn=tf.nn.tanh) ]) .connect_layer(3, fn=tf.nn.softmax) ) """ builders = [ builder.connect_weights(size) for builder in tree._leafs() ] builder = _add_builders(builders) if bias: builder = builder.connect_bias(bias_name=bias_name) if fn: builder.tensor = fn(builder.tensor, name=name) return builder @_immutable def _leafs(tree): """A generator function that lazily returns all the Builders contianed by this tree""" for branch in tree.branches: for builder in branch._leafs(): yield builder ## Module Funs def builder(tensor): """ Takes a tensor and returns a `tensorbuilder.builders.Builder` that contians it. If function is also used to monkey-patch tensorflow's Tensor class with a method of the same name. ** Parameters ** * `tensor`: a tensorflow Tensor #### Example The following example shows you how to construct a `tensorbuilder.builders.Builder` from a tensorflow Tensor. import tensorflow as tf import tensorbuilder as tb a = tf.placeholder(tf.float32, shape=[None, 8]) a_builder = tb.builder(a) The previous is the same as a_builder = tf.placeholder(tf.float32, shape=[None, 8]).builder() since tensorbuilder monkey-patches tensorflow's Tensor with this function as method. """ return Builder(tensor) def branches(builder_list): """ Takes a list with elements of type `tensorbuilder.builders.Builder` or `tensorbuilder.builders.BuilderTree` and returns a `tensorbuilder.builders.BuilderTree` ** Parameters ** * `builder_list`: list of type `list( Builder | BuilderTree)` #### Example Given a list of Builders and/or BuilderTrees you construct a `tensorbuilder.builders.BuilderTree` like this import tensorflow as tf import tensorbuilder as tb a = tf.placeholder(tf.float32, shape=[None, 8]).builder() b = tf.placeholder(tf.float32, shape=[None, 8]).builder() tree = tb.branches([a, b]) `tensorbuilder.builders.BuilderTree`s are usually constructed using `tensorbuilder.builders.Builder.branch` of the `tensorbuilder.builders.Builder` class, but you can use this for special cases """ return BuilderTree(builder_list) def _add_builders(builders): tensor = None variables = {} for builder in builders: if tensor == None: tensor = builder.tensor else: tensor += builder.tensor variables.update(builder.variables) return Builder(tensor, variables=variables)
Functions
def branches(
builder_list)
Takes a list with elements of type Builder
or BuilderTree
and returns a BuilderTree
Parameters
builder_list
: list of typelist( Builder | BuilderTree)
Example
Given a list of Builders and/or BuilderTrees you construct a BuilderTree
like this
import tensorflow as tf import tensorbuilder as tb a = tf.placeholder(tf.float32, shape=[None, 8]).builder() b = tf.placeholder(tf.float32, shape=[None, 8]).builder() tree = tb.branches([a, b])
BuilderTree
s are usually constructed using branch
of the Builder
class, but you can use this for special cases
def branches(builder_list): """ Takes a list with elements of type `tensorbuilder.builders.Builder` or `tensorbuilder.builders.BuilderTree` and returns a `tensorbuilder.builders.BuilderTree` ** Parameters ** * `builder_list`: list of type `list( Builder | BuilderTree)` #### Example Given a list of Builders and/or BuilderTrees you construct a `tensorbuilder.builders.BuilderTree` like this import tensorflow as tf import tensorbuilder as tb a = tf.placeholder(tf.float32, shape=[None, 8]).builder() b = tf.placeholder(tf.float32, shape=[None, 8]).builder() tree = tb.branches([a, b]) `tensorbuilder.builders.BuilderTree`s are usually constructed using `tensorbuilder.builders.Builder.branch` of the `tensorbuilder.builders.Builder` class, but you can use this for special cases """ return BuilderTree(builder_list)
def builder(
tensor)
Takes a tensor and returns a Builder
that contians it. If function is also used to monkey-patch tensorflow's Tensor class with a method of the same name.
Parameters
tensor
: a tensorflow Tensor
Example
The following example shows you how to construct a Builder
from a tensorflow Tensor.
import tensorflow as tf import tensorbuilder as tb a = tf.placeholder(tf.float32, shape=[None, 8]) a_builder = tb.builder(a)
The previous is the same as
a_builder = tf.placeholder(tf.float32, shape=[None, 8]).builder()
since tensorbuilder monkey-patches tensorflow's Tensor with this function as method.
def builder(tensor): """ Takes a tensor and returns a `tensorbuilder.builders.Builder` that contians it. If function is also used to monkey-patch tensorflow's Tensor class with a method of the same name. ** Parameters ** * `tensor`: a tensorflow Tensor #### Example The following example shows you how to construct a `tensorbuilder.builders.Builder` from a tensorflow Tensor. import tensorflow as tf import tensorbuilder as tb a = tf.placeholder(tf.float32, shape=[None, 8]) a_builder = tb.builder(a) The previous is the same as a_builder = tf.placeholder(tf.float32, shape=[None, 8]).builder() since tensorbuilder monkey-patches tensorflow's Tensor with this function as method. """ return Builder(tensor)
Classes
class Builder
The Builder class is a wrapper around a Tensor. Most of its method are immutable in the sense that they don't modify the caller object but rather always make a copy, they also tend to return a Builder so you you can keep fluently chaining methods.
To create a builder from a method you have these options:
-
Use the
builder
functiontb.builder(tensor)
-
Use the monkey-patched method on the Tensor class
tensor.builder()
class Builder(object): """ The Builder class is a wrapper around a Tensor. Most of its method are immutable in the sense that they don't modify the caller object but rather always make a copy, they also tend to return a Builder so you you can keep fluently chaining methods. To create a builder from a method you have these options: 1. Use the `tensorbuilder.builders.builder` function tb.builder(tensor) 2. Use the monkey-patched method on the Tensor class tensor.builder() """ def __init__(self, tensor, variables={}): super(Builder, self).__init__() self.tensor = tensor """A `tensorflow` Tensor.""" self.variables = variables """A dictionary that accumulates the **tf.Variable** tensors generated during the building process, it has the form {tensor_name: String -> tensor: tf.Variable}. Since functions like `tensorbuilder.builders.Builder.connect_layer` hides you the complexity of creating the **bias** and **weights** of your network, `tensorbuilder.builders.Builder` stores them in this field. The methods of this class enables you to set the names of these variables, but take into account that the final name is actually the name of the tensor, with is set by `tensorflow`. Check their documentation to see how the name is defined.""" def copy(self): """Returns a copy of this Builder""" return Builder(self.tensor, self.variables.copy()) @_immutable def connect_weights(builder, size, name=None, weights_name=None): """ `@_immutable` Let **x** be `tensorbuilder.builders.Builder.tensor` of shape **[m, n]**, and let **w** be a **tf.Variable** of shape **[n, size]**. Then `builder.connect_weights(size)` computes `tf.matmul(x, w)`. The returned `tensorbuilder.builders.Builder` has **w** stored inside `tensorbuilder.builders.Builder.variables`. **Parameters** * `size`: an `int` representing the size of the layer (number of neurons) * `name`: the name of the tensor (default: "connect_weights") * `weights_name`: the name of the **w** tensor of type `tf.Variable`. **Return** * `tensorbuilder.builders.Builder` **Examples** The following builds `tf.matmul(x, w)` import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) z = x.builder().connect_weights(3, weights_name="weights") """ m = int(builder.tensor.get_shape()[1]) n = size w = tf.Variable(tf.random_uniform([m, n], -1.0, 1.0), name=weights_name) var_name = weights_name if weights_name else w.name builder.variables[var_name] = w builder.tensor = tf.matmul(builder.tensor, w, name=name) return builder @_immutable def connect_bias(builder, name=None, bias_name=None): """ `@_immutable` Let **x** be `tensorbuilder.builders.Builder.tensor` of shape **[m, n]**, and let **b** be a **tf.Variable** of shape **[n]**. Then `builder.connect_bias()` computes `tf.add(x, b)`. The returned `tensorbuilder.builders.Builder` has **b** stored inside `tensorbuilder.builders.Builder.variables`. **Parameters** * `name`: the name of the tensor (default: "connect_bias") * `bias_name`: the name of the `w` tensor of type `tf.Variable` (default: "b"). **Return** * `tensorbuilder.builders.Builder` **Examples** The following builds `tf.matmul(x, w) + b` import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) z = ( x.builder() .connect_weights(3, weights_name="weights") .connect_bias(bias_name="bias") ) Note, the previous is equivalent to using `tensorbuilder.builders.Builder.connect_layer` like this z = ( x.builder() .connect_layer(3, weights_name="weights", bias_name="bias") ) """ m = int(builder.tensor.get_shape()[1]) b = tf.Variable(tf.random_uniform([m], -1.0, 1.0), name=bias_name) var_name = bias_name if bias_name else b.name builder.variables[var_name] = b builder.tensor = tf.add(builder.tensor, b, name=name) return builder @_immutable def connect_layer(builder, size, fn=None, name=None, weights_name=None, bias=True, bias_name=None): """ `@_immutable` Let **x** be `tensorbuilder.builders.Builder.tensor` of shape **[m, n]**, let **w** be a **tf.Variable** of shape **[n, size]**, let **b** be a **tf.Variable** of shape **[n]**, and **fn** be a function from a tensor to a tensor. Then `builder.connect_layer(size, fn=fn)` computes `fn(tf.matmul(x, w) + b)`. If **fn** is not present the layer is linear. Note that **fn** must expose the keyword/named argument `name`, this is compatible with the tensorflow API. The returned `tensorbuilder.builders.Builder` has **b** and **w** stored inside `tensorbuilder.builders.Builder.variables`. **Parameters** * `fn`: a function of type `tensor -> tensor`. If `fn` is `None` then its not applied, resulting in just a linear trasformation. (default: None) * `size`: an `int` representing the size of the layer (number of neurons) * `name`: the name of the tensor (default: `"layer"`) * `bias`: determines where to use a bias **b** or not (default: `True`) * `weights_name`: the name of the `w` tensor of type `tf.Variable` (default: `None`) * `bias_name`: the name of the `w` tensor of type `tf.Variable` (default: `None`) **Return** * `tensorbuilder.builders.Builder` **Examples** The following builds the computation `tf.nn.sigmoid(tf.matmul(x, w) + b)` import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) h = ( x.builder() .connect_layer(3, fn=tf.nn.sigmoid) ) The previous is equivalent to using h = ( x.builder() .connect_weights(3) .connect_bias() .map(tf.nn.sigmoid) ) You can chain various `connect_layer`s to get deeper neural networks import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 40]) h = ( x.builder() .connect_layer(100, fn=tf.nn.tanh) .connect_layer(30, fn=tf.nn.softmax) ) """ builder = builder.connect_weights(size, weights_name=weights_name) if bias: builder = builder.connect_bias(bias_name=bias_name) if fn: builder.tensor = fn(builder.tensor, name=name) return builder @_immutable def map(builder, fn, *args, **kwargs): """ `@_immutable` Let **x** be `tensorbuilder.builders.Builder.tensor` and **fn** be a function from a tensor to a tensor. Then `builder.map(fn)` computes `fn(x)`. All extra positional and named arguments are forwarded to **fn** such that builder.map(fn, arg1, arg2, ..., kwarg1=kwarg1, kwarg2=kwarg2, ...) internally results in builder.tensor = fn(builder.tensor, arg1, arg2, ..., kwarg1=kwarg1, kwarg2=kwarg2, ...) **Parameters** * `fn`: a function of type `tensor -> tensor`. **Return** * `tensorbuilder.builders.Builder` **Examples** The following constructs a neural network with the architecture `[40 input, 100 tanh, 30 softmax]` and and applies `dropout` to the tanh layer import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 40]) keep_prob = tf.placeholder(tf.float32) h = ( x.builder() .connect_layer(100, fn=tf.nn.tanh) .map(tb.nn.dropout, keep_prob) .connect_layer(30, fn=tf.nn.softmax) ) """ builder.tensor = fn(builder.tensor, *args, **kwargs) return builder @_immutable def then(builder, fn): """ `@_immutable` Expects a function **fn** with type `builder -> builder`. This method is used primarily to manupilate the Builder with very fine grain control through the fluent immutable API. **Parameters** * `fn`: a function of type `builder -> builder`. **Return** * `tensorbuilder.builders.Builder` ** Example ** The following *manually* constructs the computation `tf.nn.sigmoid(tf.matmul(x, w) + b)` while updating the `tensorbuilder.tensorbuiler.Builder.variables` dictionary. import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 40]) keep_prob = tf.placeholder(tf.float32) def sigmoid_layer(builder, size): m = int(builder.tensor.get_shape()[1]) n = size w = tf.Variable(tf.random_uniform([m, n], -1.0, 1.0)) b = tf.Variable(tf.random_uniform([n], -1.0, 1.0)) builder.variables[w.name] = w builder.variables[b.name] = b builder.tensor = tf.nn.sigmoid(tf.matmul(builder.tensor, w) + b) return builder h = ( x.builder() .then(lambda builder: sigmoid_layer(builder, 3)) ) Note that the previous if equivalent to h = ( x.builder() .connect_layer(3, fn=tf.nn.sigmoid) ) """ return fn(builder) @_immutable def branch(builder, fn): """ `@_immutable` Expects a function **fn** with type `Builder -> list( Builder | BuilderTree )`. This method enables you to *branch* the computational graph so you can easily create neural networks with more complex topologies. You can later **Parameters** * `fn`: a function of type `Builder -> list( Builder | BuilderTree )`. **Return** * `tensorbuilder.builders.BuilderTree` ** Example ** The following will create a sigmoid layer but will branch the computation at the logit (z) so you get both the output tensor `h` and `trainer` tensor. Observe that first the logit `z` is calculated by creating a linear layer with `connect_layer(1)` and then its branched out import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) y = tf.placeholder(tf.float32, shape=[None, 1]) [h, trainer] = ( x.builder() .connect_layer(1) .branch(lambda z: [ z.map(tf.nn.sigmoid) , z.map(tf.nn.sigmoid_cross_entropy_with_logits, y) .map(tf.train.AdamOptimizer(0.01).minimize) ]) .tensors() ) Note that you have to use the `tensorbuilder.builders.BuilderTree.tensors` method from the `tensorbuilder.builders.BuilderTree` class to get the tensors back. Remember that you can also contain `tensorbuilder.builders.BuilderTree` elements when you branch out, this means that you can keep branching inside branch. Don't worry that the tree keep getting deeper, `tensorbuilder.builders.BuilderTree` has methods that help you flatten or reduce the tree. The following example will show you how create a (overly) complex tree and then connect all the leaf nodes to a single `sigmoid` layer import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) keep_prob = tf.placeholder(tf.float32) h = ( x.builder() .connect_layer(10) .branch(lambda base: [ base .connect_layer(3, fn=tf.nn.relu) , base .connect_layer(9, fn=tf.nn.tanh) .branch(lambda base2: [ base2 .connect_layer(6, fn=tf.nn.sigmoid) , base2 .map(tf.nn.dropout, keep_prob) .connect_layer(8, tf.nn.softmax) ]) ]) .connect_layer(6, fn=tf.nn.sigmoid) ) ** See Also ** * `tensorbuilder.builders.BuilderTree` * `tensorbuilder.builders.BuilderTree.connect_layer` * `tensorbuilder.builders.BuilderTree.builders` * `tensorbuilder.builders.BuilderTree.tensors` """ branches = fn(builder) return BuilderTree(branches) @_immutable def _leafs(builder): """A generator function that yields the builder, used by `tensorbuilder.builders.BuilderTree.leafs` of `tensorbuilder.builders.BuilderTree`""" yield builder
Ancestors (in MRO)
- Builder
- __builtin__.object
Instance variables
var tensor
A tensorflow
Tensor.
var variables
A dictionary that accumulates the tf.Variable tensors generated during the building process, it has the form {tensor_name: String -> tensor: tf.Variable}. Since functions like connect_layer
hides you the complexity of creating the bias and weights of your network, Builder
stores them in this field. The methods of this class enables you to set the names of these variables, but take into account that the final name is actually the name of the tensor, with is set by tensorflow
. Check their documentation to see how the name is defined.
Methods
def __init__(
self, tensor, variables={})
def __init__(self, tensor, variables={}): super(Builder, self).__init__() self.tensor = tensor """A `tensorflow` Tensor.""" self.variables = variables """A dictionary that accumulates the **tf.Variable** tensors generated during the building process, it has the form {tensor_name: String -> tensor: tf.Variable}. Since functions like `tensorbuilder.builders.Builder.connect_layer` hides you the complexity of creating the **bias** and **weights** of your network, `tensorbuilder.builders.Builder` stores them in this field. The methods of this class enables you to set the names of these variables, but take into account that the final name is actually the name of the tensor, with is set by `tensorflow`. Check their documentation to see how the name is defined."""
def branch(
builder, fn)
@_immutable
Expects a function fn with type Builder -> list( Builder | BuilderTree )
. This method enables you to branch the computational graph so you can easily create neural networks with more complex topologies. You can later
Parameters
fn
: a function of typeBuilder -> list( Builder | BuilderTree )
.
Return
Example
The following will create a sigmoid layer but will branch the computation at the logit (z) so you get both the output tensor h
and trainer
tensor. Observe that first the logit z
is calculated by creating a linear layer with connect_layer(1)
and then its branched out
import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) y = tf.placeholder(tf.float32, shape=[None, 1]) [h, trainer] = ( x.builder() .connect_layer(1) .branch(lambda z: [ z.map(tf.nn.sigmoid) , z.map(tf.nn.sigmoid_cross_entropy_with_logits, y) .map(tf.train.AdamOptimizer(0.01).minimize) ]) .tensors() )
Note that you have to use the tensors
method from the BuilderTree
class to get the tensors back. Remember that you can also contain BuilderTree
elements when you branch out, this means that you can keep branching inside branch. Don't worry that the tree keep getting deeper, BuilderTree
has methods that help you flatten or reduce the tree.
The following example will show you how create a (overly) complex tree and then connect all the leaf nodes to a single sigmoid
layer
import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) keep_prob = tf.placeholder(tf.float32) h = ( x.builder() .connect_layer(10) .branch(lambda base: [ base .connect_layer(3, fn=tf.nn.relu) , base .connect_layer(9, fn=tf.nn.tanh) .branch(lambda base2: [ base2 .connect_layer(6, fn=tf.nn.sigmoid) , base2 .map(tf.nn.dropout, keep_prob) .connect_layer(8, tf.nn.softmax) ]) ]) .connect_layer(6, fn=tf.nn.sigmoid) )
See Also
@_immutable def branch(builder, fn): """ `@_immutable` Expects a function **fn** with type `Builder -> list( Builder | BuilderTree )`. This method enables you to *branch* the computational graph so you can easily create neural networks with more complex topologies. You can later **Parameters** * `fn`: a function of type `Builder -> list( Builder | BuilderTree )`. **Return** * `tensorbuilder.builders.BuilderTree` ** Example ** The following will create a sigmoid layer but will branch the computation at the logit (z) so you get both the output tensor `h` and `trainer` tensor. Observe that first the logit `z` is calculated by creating a linear layer with `connect_layer(1)` and then its branched out import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) y = tf.placeholder(tf.float32, shape=[None, 1]) [h, trainer] = ( x.builder() .connect_layer(1) .branch(lambda z: [ z.map(tf.nn.sigmoid) , z.map(tf.nn.sigmoid_cross_entropy_with_logits, y) .map(tf.train.AdamOptimizer(0.01).minimize) ]) .tensors() ) Note that you have to use the `tensorbuilder.builders.BuilderTree.tensors` method from the `tensorbuilder.builders.BuilderTree` class to get the tensors back. Remember that you can also contain `tensorbuilder.builders.BuilderTree` elements when you branch out, this means that you can keep branching inside branch. Don't worry that the tree keep getting deeper, `tensorbuilder.builders.BuilderTree` has methods that help you flatten or reduce the tree. The following example will show you how create a (overly) complex tree and then connect all the leaf nodes to a single `sigmoid` layer import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) keep_prob = tf.placeholder(tf.float32) h = ( x.builder() .connect_layer(10) .branch(lambda base: [ base .connect_layer(3, fn=tf.nn.relu) , base .connect_layer(9, fn=tf.nn.tanh) .branch(lambda base2: [ base2 .connect_layer(6, fn=tf.nn.sigmoid) , base2 .map(tf.nn.dropout, keep_prob) .connect_layer(8, tf.nn.softmax) ]) ]) .connect_layer(6, fn=tf.nn.sigmoid) ) ** See Also ** * `tensorbuilder.builders.BuilderTree` * `tensorbuilder.builders.BuilderTree.connect_layer` * `tensorbuilder.builders.BuilderTree.builders` * `tensorbuilder.builders.BuilderTree.tensors` """ branches = fn(builder) return BuilderTree(branches)
def connect_bias(
builder, name=None, bias_name=None)
@_immutable
Let x be tensor
of shape [m, n], and let b be a tf.Variable of shape [n]. Then builder.connect_bias()
computes tf.add(x, b)
.
The returned Builder
has b stored inside variables
.
Parameters
name
: the name of the tensor (default: "connect_bias")bias_name
: the name of thew
tensor of typetf.Variable
(default: "b").
Return
Examples
The following builds tf.matmul(x, w) + b
import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) z = ( x.builder() .connect_weights(3, weights_name="weights") .connect_bias(bias_name="bias") )
Note, the previous is equivalent to using connect_layer
like this
z = ( x.builder() .connect_layer(3, weights_name="weights", bias_name="bias") )
@_immutable def connect_bias(builder, name=None, bias_name=None): """ `@_immutable` Let **x** be `tensorbuilder.builders.Builder.tensor` of shape **[m, n]**, and let **b** be a **tf.Variable** of shape **[n]**. Then `builder.connect_bias()` computes `tf.add(x, b)`. The returned `tensorbuilder.builders.Builder` has **b** stored inside `tensorbuilder.builders.Builder.variables`. **Parameters** * `name`: the name of the tensor (default: "connect_bias") * `bias_name`: the name of the `w` tensor of type `tf.Variable` (default: "b"). **Return** * `tensorbuilder.builders.Builder` **Examples** The following builds `tf.matmul(x, w) + b` import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) z = ( x.builder() .connect_weights(3, weights_name="weights") .connect_bias(bias_name="bias") ) Note, the previous is equivalent to using `tensorbuilder.builders.Builder.connect_layer` like this z = ( x.builder() .connect_layer(3, weights_name="weights", bias_name="bias") ) """ m = int(builder.tensor.get_shape()[1]) b = tf.Variable(tf.random_uniform([m], -1.0, 1.0), name=bias_name) var_name = bias_name if bias_name else b.name builder.variables[var_name] = b builder.tensor = tf.add(builder.tensor, b, name=name) return builder
def connect_layer(
builder, size, fn=None, name=None, weights_name=None, bias=True, bias_name=None)
@_immutable
Let x be tensor
of shape [m, n], let w be a tf.Variable of shape [n, size], let b be a tf.Variable of shape [n], and fn be a function from a tensor to a tensor. Then builder.connect_layer(size, fn=fn)
computes fn(tf.matmul(x, w) + b)
. If fn is not present the layer is linear.
Note that fn must expose the keyword/named argument name
, this is compatible with the tensorflow API.
The returned Builder
has b and w stored inside variables
.
Parameters
fn
: a function of typetensor -> tensor
. Iffn
isNone
then its not applied, resulting in just a linear trasformation. (default: None)size
: anint
representing the size of the layer (number of neurons)name
: the name of the tensor (default:"layer"
)bias
: determines where to use a bias b or not (default:True
)weights_name
: the name of thew
tensor of typetf.Variable
(default:None
)bias_name
: the name of thew
tensor of typetf.Variable
(default:None
)
Return
Examples
The following builds the computation tf.nn.sigmoid(tf.matmul(x, w) + b)
import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) h = ( x.builder() .connect_layer(3, fn=tf.nn.sigmoid) )
The previous is equivalent to using
h = ( x.builder() .connect_weights(3) .connect_bias() .map(tf.nn.sigmoid) )
You can chain various connect_layer
s to get deeper neural networks
import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 40]) h = ( x.builder() .connect_layer(100, fn=tf.nn.tanh) .connect_layer(30, fn=tf.nn.softmax) )
@_immutable def connect_layer(builder, size, fn=None, name=None, weights_name=None, bias=True, bias_name=None): """ `@_immutable` Let **x** be `tensorbuilder.builders.Builder.tensor` of shape **[m, n]**, let **w** be a **tf.Variable** of shape **[n, size]**, let **b** be a **tf.Variable** of shape **[n]**, and **fn** be a function from a tensor to a tensor. Then `builder.connect_layer(size, fn=fn)` computes `fn(tf.matmul(x, w) + b)`. If **fn** is not present the layer is linear. Note that **fn** must expose the keyword/named argument `name`, this is compatible with the tensorflow API. The returned `tensorbuilder.builders.Builder` has **b** and **w** stored inside `tensorbuilder.builders.Builder.variables`. **Parameters** * `fn`: a function of type `tensor -> tensor`. If `fn` is `None` then its not applied, resulting in just a linear trasformation. (default: None) * `size`: an `int` representing the size of the layer (number of neurons) * `name`: the name of the tensor (default: `"layer"`) * `bias`: determines where to use a bias **b** or not (default: `True`) * `weights_name`: the name of the `w` tensor of type `tf.Variable` (default: `None`) * `bias_name`: the name of the `w` tensor of type `tf.Variable` (default: `None`) **Return** * `tensorbuilder.builders.Builder` **Examples** The following builds the computation `tf.nn.sigmoid(tf.matmul(x, w) + b)` import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) h = ( x.builder() .connect_layer(3, fn=tf.nn.sigmoid) ) The previous is equivalent to using h = ( x.builder() .connect_weights(3) .connect_bias() .map(tf.nn.sigmoid) ) You can chain various `connect_layer`s to get deeper neural networks import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 40]) h = ( x.builder() .connect_layer(100, fn=tf.nn.tanh) .connect_layer(30, fn=tf.nn.softmax) ) """ builder = builder.connect_weights(size, weights_name=weights_name) if bias: builder = builder.connect_bias(bias_name=bias_name) if fn: builder.tensor = fn(builder.tensor, name=name) return builder
def connect_weights(
builder, size, name=None, weights_name=None)
@_immutable
Let x be tensor
of shape [m, n], and let w be a tf.Variable of shape [n, size]. Then builder.connect_weights(size)
computes tf.matmul(x, w)
.
The returned Builder
has w stored inside variables
.
Parameters
size
: anint
representing the size of the layer (number of neurons)name
: the name of the tensor (default: "connect_weights")weights_name
: the name of the w tensor of typetf.Variable
.
Return
Examples
The following builds tf.matmul(x, w)
import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) z = x.builder().connect_weights(3, weights_name="weights")
@_immutable def connect_weights(builder, size, name=None, weights_name=None): """ `@_immutable` Let **x** be `tensorbuilder.builders.Builder.tensor` of shape **[m, n]**, and let **w** be a **tf.Variable** of shape **[n, size]**. Then `builder.connect_weights(size)` computes `tf.matmul(x, w)`. The returned `tensorbuilder.builders.Builder` has **w** stored inside `tensorbuilder.builders.Builder.variables`. **Parameters** * `size`: an `int` representing the size of the layer (number of neurons) * `name`: the name of the tensor (default: "connect_weights") * `weights_name`: the name of the **w** tensor of type `tf.Variable`. **Return** * `tensorbuilder.builders.Builder` **Examples** The following builds `tf.matmul(x, w)` import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) z = x.builder().connect_weights(3, weights_name="weights") """ m = int(builder.tensor.get_shape()[1]) n = size w = tf.Variable(tf.random_uniform([m, n], -1.0, 1.0), name=weights_name) var_name = weights_name if weights_name else w.name builder.variables[var_name] = w builder.tensor = tf.matmul(builder.tensor, w, name=name) return builder
def copy(
self)
Returns a copy of this Builder
def copy(self): """Returns a copy of this Builder""" return Builder(self.tensor, self.variables.copy())
def map(
builder, fn, *args, **kwargs)
@_immutable
Let x be tensor
and fn be a function from a tensor to a tensor. Then builder.map(fn)
computes fn(x)
. All extra positional and named arguments are forwarded to fn such that
builder.map(fn, arg1, arg2, ..., kwarg1=kwarg1, kwarg2=kwarg2, ...)
internally results in
builder.tensor = fn(builder.tensor, arg1, arg2, ..., kwarg1=kwarg1, kwarg2=kwarg2, ...)
Parameters
fn
: a function of typetensor -> tensor
.
Return
Examples
The following constructs a neural network with the architecture [40 input, 100 tanh, 30 softmax]
and and applies dropout
to the tanh layer
import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 40]) keep_prob = tf.placeholder(tf.float32) h = ( x.builder() .connect_layer(100, fn=tf.nn.tanh) .map(tb.nn.dropout, keep_prob) .connect_layer(30, fn=tf.nn.softmax) )
@_immutable def map(builder, fn, *args, **kwargs): """ `@_immutable` Let **x** be `tensorbuilder.builders.Builder.tensor` and **fn** be a function from a tensor to a tensor. Then `builder.map(fn)` computes `fn(x)`. All extra positional and named arguments are forwarded to **fn** such that builder.map(fn, arg1, arg2, ..., kwarg1=kwarg1, kwarg2=kwarg2, ...) internally results in builder.tensor = fn(builder.tensor, arg1, arg2, ..., kwarg1=kwarg1, kwarg2=kwarg2, ...) **Parameters** * `fn`: a function of type `tensor -> tensor`. **Return** * `tensorbuilder.builders.Builder` **Examples** The following constructs a neural network with the architecture `[40 input, 100 tanh, 30 softmax]` and and applies `dropout` to the tanh layer import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 40]) keep_prob = tf.placeholder(tf.float32) h = ( x.builder() .connect_layer(100, fn=tf.nn.tanh) .map(tb.nn.dropout, keep_prob) .connect_layer(30, fn=tf.nn.softmax) ) """ builder.tensor = fn(builder.tensor, *args, **kwargs) return builder
def then(
builder, fn)
@_immutable
Expects a function fn with type builder -> builder
. This method is used primarily to manupilate the Builder with very fine grain control through the fluent immutable API.
Parameters
fn
: a function of typebuilder -> builder
.
Return
Example
The following manually constructs the computation tf.nn.sigmoid(tf.matmul(x, w) + b)
while updating the tensorbuilder.tensorbuiler.Builder.variables
dictionary.
import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 40]) keep_prob = tf.placeholder(tf.float32) def sigmoid_layer(builder, size): m = int(builder.tensor.get_shape()[1]) n = size w = tf.Variable(tf.random_uniform([m, n], -1.0, 1.0)) b = tf.Variable(tf.random_uniform([n], -1.0, 1.0)) builder.variables[w.name] = w builder.variables[b.name] = b builder.tensor = tf.nn.sigmoid(tf.matmul(builder.tensor, w) + b) return builder h = ( x.builder() .then(lambda builder: sigmoid_layer(builder, 3)) )
Note that the previous if equivalent to
h = ( x.builder() .connect_layer(3, fn=tf.nn.sigmoid) )
@_immutable def then(builder, fn): """ `@_immutable` Expects a function **fn** with type `builder -> builder`. This method is used primarily to manupilate the Builder with very fine grain control through the fluent immutable API. **Parameters** * `fn`: a function of type `builder -> builder`. **Return** * `tensorbuilder.builders.Builder` ** Example ** The following *manually* constructs the computation `tf.nn.sigmoid(tf.matmul(x, w) + b)` while updating the `tensorbuilder.tensorbuiler.Builder.variables` dictionary. import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 40]) keep_prob = tf.placeholder(tf.float32) def sigmoid_layer(builder, size): m = int(builder.tensor.get_shape()[1]) n = size w = tf.Variable(tf.random_uniform([m, n], -1.0, 1.0)) b = tf.Variable(tf.random_uniform([n], -1.0, 1.0)) builder.variables[w.name] = w builder.variables[b.name] = b builder.tensor = tf.nn.sigmoid(tf.matmul(builder.tensor, w) + b) return builder h = ( x.builder() .then(lambda builder: sigmoid_layer(builder, 3)) ) Note that the previous if equivalent to h = ( x.builder() .connect_layer(3, fn=tf.nn.sigmoid) ) """ return fn(builder)
class BuilderTree
BuilderTree is a class that enable you to perform computations over a complex branched builder. It contains methods to get all the leaf Builder
nodes, connect all the leaf nodes to a single layer, etc.
class BuilderTree(object): """ BuilderTree is a class that enable you to perform computations over a complex branched builder. It contains methods to get all the leaf `tensorbuilder.builders.Builder` nodes, connect all the leaf nodes to a single layer, etc. """ def __init__(self, branches): super(BuilderTree, self).__init__() self.branches = branches """ A list that can contain elements that are of type `tensorbuilder.builders.Builder` or `tensorbuilder.builders.BuilderTree`. """ def copy(self): branches = self.branches[:] return BuilderTree(branches) def builders(self): """ Returns a flattened list `tensorbuilder.builders.Builder`s contained by this tree. The whole result is flattened in case of sub-elements are also `tensorbuilder.builders.BuilderTree`s. **Return** * `list( tensorbuilder.builders.Builder )` ** Example ** import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) y = tf.placeholder(tf.float32, shape=[None, 1]) [h_builder, trainer_builder] = ( x.builder() .connect_layer(1) .branch(lambda z: [ z.map(tf.nn.sigmoid) , z.map(tf.nn.sigmoid_cross_entropy_with_logits, y) .map(tf.train.AdamOptimizer(0.01).minimize) ]) .builders() ) """ return [builder for builder in self._leafs() ] def tensors(self): """ Same as `tensorbuilder.builders.BuilderTree.builders` but extracts the tensor from each `tensorbuilder.builders.Builder`. **Return** * `list( tf.Tensor )` ** Example ** import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) y = tf.placeholder(tf.float32, shape=[None, 1]) [h_tensor, trainer_tensor] = ( x.builder() .connect_layer(1) .branch(lambda z: [ z.map(tf.nn.sigmoid) , z.map(tf.nn.sigmoid_cross_entropy_with_logits, y) .map(tf.train.AdamOptimizer(0.01).minimize) ]) .tensors() ) """ return [builder.tensor for builder in self._leafs() ] @_immutable def connect_layer(tree, size, fn=None, name="layer", bias=True, bias_name=None): """ Connects all the leaf `tensorbuilder.builders.Builder` nodes of this tree to a single layer. The order of computation is done as follows 1. Each leaf builder node is linearly connected to a layer of size `size` with no bias. 2. All these layers of size `size` are added together (reduced with +) 3. If `bias` is `True` then a bias added 4. If `fn` is not `None` then the function `fn` is mapped ** Parameters ** * `fn`: a function of type `tensor -> tensor`. If `fn` is `None` then its not applied, resulting in just a linear trasformation. (default: None) * `name`: the name of the tensor (default: `"layer"`) * `bias`: determines where to use a bias **b** or not (default: `True`) * `bias_name`: the name of the `w` tensor of type `tf.Variable` (default: `None`) ** Examples ** # The following example shows you how to connect two tensors (rather builders) of different shapes to a single `softmax` layer of shape [None, 3] import tensorflow as tf import tensorbuilder as tb a = tf.placeholder(tf.float32, shape=[None, 8]).builder() b = tf.placeholder(tf.float32, shape=[None, 5]).builder() h = ( tb.branches([a, b]) .connect_layer(3, fn=tf.nn.softmax) ) The next example show you how you can use this to pass the input layer directly through one branch, and "analyze" it with a `tanh layer` filter through the other, both of these are connect to a single `softmax` output layer import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) h = ( x.builder() .branch(lambda x: [ x , x.connect_layer(10, fn=tf.nn.tanh) ]) .connect_layer(3, fn=tf.nn.softmax) ) """ builders = [ builder.connect_weights(size) for builder in tree._leafs() ] builder = _add_builders(builders) if bias: builder = builder.connect_bias(bias_name=bias_name) if fn: builder.tensor = fn(builder.tensor, name=name) return builder @_immutable def _leafs(tree): """A generator function that lazily returns all the Builders contianed by this tree""" for branch in tree.branches: for builder in branch._leafs(): yield builder
Ancestors (in MRO)
- BuilderTree
- __builtin__.object
Instance variables
Methods
def __init__(
self, branches)
def __init__(self, branches): super(BuilderTree, self).__init__() self.branches = branches """ A list that can contain elements that are of type `tensorbuilder.builders.Builder` or `tensorbuilder.builders.BuilderTree`. """
def builders(
self)
Returns a flattened list Builder
s contained by this tree. The whole result is flattened in case of sub-elements are also BuilderTree
s.
Return
list( tensorbuilder.builders.Builder )
Example
import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) y = tf.placeholder(tf.float32, shape=[None, 1]) [h_builder, trainer_builder] = ( x.builder() .connect_layer(1) .branch(lambda z: [ z.map(tf.nn.sigmoid) , z.map(tf.nn.sigmoid_cross_entropy_with_logits, y) .map(tf.train.AdamOptimizer(0.01).minimize) ]) .builders() )
def builders(self): """ Returns a flattened list `tensorbuilder.builders.Builder`s contained by this tree. The whole result is flattened in case of sub-elements are also `tensorbuilder.builders.BuilderTree`s. **Return** * `list( tensorbuilder.builders.Builder )` ** Example ** import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) y = tf.placeholder(tf.float32, shape=[None, 1]) [h_builder, trainer_builder] = ( x.builder() .connect_layer(1) .branch(lambda z: [ z.map(tf.nn.sigmoid) , z.map(tf.nn.sigmoid_cross_entropy_with_logits, y) .map(tf.train.AdamOptimizer(0.01).minimize) ]) .builders() ) """ return [builder for builder in self._leafs() ]
def connect_layer(
tree, size, fn=None, name='layer', bias=True, bias_name=None)
Connects all the leaf Builder
nodes of this tree to a single layer. The order of computation is done as follows
- Each leaf builder node is linearly connected to a layer of size
size
with no bias. - All these layers of size
size
are added together (reduced with +) - If
bias
isTrue
then a bias added - If
fn
is notNone
then the functionfn
is mapped
Parameters
fn
: a function of typetensor -> tensor
. Iffn
isNone
then its not applied, resulting in just a linear trasformation. (default: None)name
: the name of the tensor (default:"layer"
)bias
: determines where to use a bias b or not (default:True
)bias_name
: the name of thew
tensor of typetf.Variable
(default:None
)
Examples
The following example shows you how to connect two tensors (rather builders) of different shapes to a single softmax
layer of shape [None, 3]
import tensorflow as tf import tensorbuilder as tb a = tf.placeholder(tf.float32, shape=[None, 8]).builder() b = tf.placeholder(tf.float32, shape=[None, 5]).builder() h = ( tb.branches([a, b]) .connect_layer(3, fn=tf.nn.softmax) )
The next example show you how you can use this to pass the input layer directly through one branch, and "analyze" it with a tanh layer
filter through the other, both of these are connect to a single softmax
output layer
import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) h = ( x.builder() .branch(lambda x: [ x , x.connect_layer(10, fn=tf.nn.tanh) ]) .connect_layer(3, fn=tf.nn.softmax) )
@_immutable def connect_layer(tree, size, fn=None, name="layer", bias=True, bias_name=None): """ Connects all the leaf `tensorbuilder.builders.Builder` nodes of this tree to a single layer. The order of computation is done as follows 1. Each leaf builder node is linearly connected to a layer of size `size` with no bias. 2. All these layers of size `size` are added together (reduced with +) 3. If `bias` is `True` then a bias added 4. If `fn` is not `None` then the function `fn` is mapped ** Parameters ** * `fn`: a function of type `tensor -> tensor`. If `fn` is `None` then its not applied, resulting in just a linear trasformation. (default: None) * `name`: the name of the tensor (default: `"layer"`) * `bias`: determines where to use a bias **b** or not (default: `True`) * `bias_name`: the name of the `w` tensor of type `tf.Variable` (default: `None`) ** Examples ** # The following example shows you how to connect two tensors (rather builders) of different shapes to a single `softmax` layer of shape [None, 3] import tensorflow as tf import tensorbuilder as tb a = tf.placeholder(tf.float32, shape=[None, 8]).builder() b = tf.placeholder(tf.float32, shape=[None, 5]).builder() h = ( tb.branches([a, b]) .connect_layer(3, fn=tf.nn.softmax) ) The next example show you how you can use this to pass the input layer directly through one branch, and "analyze" it with a `tanh layer` filter through the other, both of these are connect to a single `softmax` output layer import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) h = ( x.builder() .branch(lambda x: [ x , x.connect_layer(10, fn=tf.nn.tanh) ]) .connect_layer(3, fn=tf.nn.softmax) ) """ builders = [ builder.connect_weights(size) for builder in tree._leafs() ] builder = _add_builders(builders) if bias: builder = builder.connect_bias(bias_name=bias_name) if fn: builder.tensor = fn(builder.tensor, name=name) return builder
def copy(
self)
def copy(self): branches = self.branches[:] return BuilderTree(branches)
def tensors(
self)
Same as builders
but extracts the tensor from each Builder
.
Return
list( tf.Tensor )
Example
import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) y = tf.placeholder(tf.float32, shape=[None, 1]) [h_tensor, trainer_tensor] = ( x.builder() .connect_layer(1) .branch(lambda z: [ z.map(tf.nn.sigmoid) , z.map(tf.nn.sigmoid_cross_entropy_with_logits, y) .map(tf.train.AdamOptimizer(0.01).minimize) ]) .tensors() )
def tensors(self): """ Same as `tensorbuilder.builders.BuilderTree.builders` but extracts the tensor from each `tensorbuilder.builders.Builder`. **Return** * `list( tf.Tensor )` ** Example ** import tensorflow as tf import tensorbuilder as tb x = tf.placeholder(tf.float32, shape=[None, 5]) y = tf.placeholder(tf.float32, shape=[None, 1]) [h_tensor, trainer_tensor] = ( x.builder() .connect_layer(1) .branch(lambda z: [ z.map(tf.nn.sigmoid) , z.map(tf.nn.sigmoid_cross_entropy_with_logits, y) .map(tf.train.AdamOptimizer(0.01).minimize) ]) .tensors() ) """ return [builder.tensor for builder in self._leafs() ]


--- layout: post title: switch/case statements in cfscript tags: [] status: publish type: post published: true comments: true meta: {} ---
Just recently I have been moving away from using the tag based development methodology of ColdFusion and been doing most of my coding using cfscript. Of course this doesn't natively give me access to the wealth of functionality that ColdFusion has [note: there are ways around that and I hope I'll be bale to write something up soon], however it does give you a more traditional feel for coding and can certainly be a bit tighter. Here I just wanted to highlight how <cfswitch> and <cfcase> differ slightly when used as swicth/case in <cfscript>
OK let's take a look at the code traditional CF style side by side with cfscript stylee:
<cfswitch expression="#yourExpression#">
<cfcase value="a,b,c">
...something...
</cfcase>
<cfcase value="d">
...something else...
</cfcase>
<cfdefaultcase>
... default behaviour goes here ...
</cfdefaultcase>
</cfswitch>
<cfscript>
switch(yourExpression) {
case "a": case "b": case "c":
...something...
break;
case "d":
...something else...
break;
default:
... default behaviour goes here ...
break;
}
</cfscript>
I guess that side by side the comparison in terms of neatness and tightness does not really shine through. However this is just for swicth/case, once you come to work with cfscript you'll truly appreciate the scripting approach. So what differs:
And that's about it. A useful reference hopefully.


ACboy needs your help again!
问题描述 :
ACboy was kidnapped!!
he miss his mother very much and is very scare now.You can’t image how dark the room he was put into is, so poor :(.
As a smart ACMer, you want to get ACboy out of the monster’s labyrinth.But when you arrive at the gate of the maze, the monste say :" I have heard that you are very clever, but if can’t solve my problems, you will die with ACboy."
The problems of the monster is shown on the wall:
Each problem’s first line is a integer N(the number of commands), and a word "FIFO" or "FILO".(you are very happy because you know "FIFO" stands for "First In First Out", and "FILO" means "First In Last Out").
and the following N lines, each line is "IN M" or "OUT", (M represent a integer).
and the answer of a problem is a passowrd of a door, so if you want to rescue ACboy, answer the problem carefully!
he miss his mother very much and is very scare now.You can’t image how dark the room he was put into is, so poor :(.
As a smart ACMer, you want to get ACboy out of the monster’s labyrinth.But when you arrive at the gate of the maze, the monste say :" I have heard that you are very clever, but if can’t solve my problems, you will die with ACboy."
The problems of the monster is shown on the wall:
Each problem’s first line is a integer N(the number of commands), and a word "FIFO" or "FILO".(you are very happy because you know "FIFO" stands for "First In First Out", and "FILO" means "First In Last Out").
and the following N lines, each line is "IN M" or "OUT", (M represent a integer).
and the answer of a problem is a passowrd of a door, so if you want to rescue ACboy, answer the problem carefully!
输入:
The input contains multiple test cases.
The first line has one integer,represent the number oftest cases.
And the input of each subproblem are described above.
The first line has one integer,represent the number oftest cases.
And the input of each subproblem are described above.
输出:
For each command "OUT", you should output a integer depend on the word is "FIFO" or "FILO", or a word "None" if you don’t have any integer.
。。。 */ #include <iostream> #include <stack> #include <queue> using namespace std; int main() { int t, m; string str; cin >> t; stack<int> p; queue<int> q; while (t--) { /** * 因为STL中的stack、queue没有专门的clear()函数 * 所以每次都要自己写清空stack、queue的函数 */ while(!p.empty()){ p.pop(); } while(!q.empty()){ q.pop(); } cin >> m >> str; if (str == "FIFO") { for (int i = 0; i < m; ++i) { string cmd; int n; cin >> cmd; if (cmd == "IN") { cin >> n; q.push(n); } else if (cmd == "OUT") { if (!q.empty()) { int a = q.front(); q.pop(); cout << a << endl; } else { cout << "None" << endl; } } } } else { for (int i = 0; i < m; ++i) { string cmd; int n; cin >> cmd; if (cmd == "IN") { cin >> n; p.push(n); } else if (cmd == "OUT") { if (!p.empty()) { int a = p.top(); p.pop(); cout << a << endl; } else { cout << "None" << endl; } } } } } }
解题报告转自：http://blog.csdn.net/hjd_love_zzt/article/details/9821917
(adsbygoogle = window.adsbygoogle || []).push({});
真心不错，力顶作者
这个问题别太纠结了，不同的编译器结果不一
抱歉，确实是弄错了，原来的代码是错误的，删掉了
根本过不了。。。。
写的好····· Pollard Rh
是O(m+n)，不小心打错了
for(int i=1; i<=m; i++){
for(int j=1; j<=n; j++){
dp = dp [j-1] + 1;
if(s1.charAt(i-1) == s3.charAt(i+j-1))
dp = dp[i-1] + 1;
if(s2.charAt(j-1) == s3.charAt(i+j-1))
dp = Math.max(dp [j - 1] + 1, dp );
}
}
这里的代码似乎有点问题？ dp(i)(j) = dp(i)(j-1) + 1;这个例子System.out.println(ils.isInterleave("aa","dbbca", "aadbbcb"));返回的应该是false
约瑟夫也用说这么长……很成熟的一个问题了，分治的方法解起来o(n)就可以了，有兴趣可以看看具体数学的第一章，关于约瑟夫问题推导出了一系列的结论，很漂亮
对于第一题，我的直觉告诉我换不换都一样，求指教
不错，赞一个！
这道题目的核心一句话是：取还是不取。
如果当前取，则index+1作为参数。如果当前不取，则任用index作为参数。
网站不错
学习了
正确


| « Prev 5.2 Example Build Scripts | Table of Contents | Next » 5.4 YARD API Documentation |
You can make your Rscons-based project more familiar to users of
autoconf-generated projects by creating a configure
script and a Makefile
for the user.
Such users may be used to executing:
./configure make
to build a project.
To do this, create a configure
script with contents similar to the following:
#!/bin/sh exec "$(dirname "$0")"/rscons configure "$@"
and make it executable with chmod +x configure
.
If you want your users to be able to build/clean a project with make
but
still make use of Rscons under the hood, you can create a Makefile
with
contents something like this:
.PHONY: all all: ./rscons build .PHONY: clean clean: ./rscons clean
| « Prev 5.2 Example Build Scripts | Table of Contents | Next » 5.4 YARD API Documentation |


Caroline Kennedy christens new aircraft carrier USS John F. Kennedy
by IrishCentral StaffJFK's daughter Caroline Kennedy christened the new US Navy aircraft carrier, the USS John F. Kennedy, at a ceremony on Dec 7.
The former ambassador to Japan and President John F. Kennedy's daughter, Caroline Kennedy, christened the US Navy aircraft carrier, previously known as CVN 79, with its official name the USS John F. Kennedy on Sat, Dec 7, 2019.
The former president's daughter smashed a bottle of sparkling wine against the ship's hull, as is tradition, at Huntington Ingalls Industries, in Newport, Virginia, where the USS John F. Kennedy was built.
Here is the US Navy's latest video of the dry dock where the USS John F Kennedy's construction was completed being flooded on Oct 29, 2019:
Over 3,000 shipbuilders and 2,000 suppliers have been involved in the construction of the ship since its keel was laid in 2015. The USS John F Kennedy is a nuclear-powered ship and is expected to enter service in the US Navy in 2020.
On May 29, 2019, the aircraft carrier's 588 metric ton island was landed. Symbolically, on what would have been President John F Kennedy's 102nd birthday.
While his daughter was unable to attend that ceremony she did send a message which read "I know how proud my father would be of the ship that will bear his name and the patriotism and dedication of all who will sail in her."
WATCH: Color footage of President Kennedy’s historic 1963 visit to Ireland
Caroline also christened the first aircraft carrier named in honor of her father in 1967. Nicknamed "Big John," the ship was decommissioned in 2007 and was the last American aircraft carrier to be powered by conventional fuel.
In November 2019, the official seal of the USS John F Kennedy was revealed. At the time it was noted that in addition to honoring JFK's own naval service and vision for space exploration it also featured a drawing of the aircraft carriers including aircraft on its deck, giving a clue to what this vessel would be used for.
Cmdr. Jennifer Cragg of Naval Air Force Atlantic confirmed that the seal "displays future naval aviation capabilities that the aircraft carrier will likely support throughout its estimated 50-year service life."
John F Kennedy's US Navy career
At the age of 23, having graduated from Harvard with a Bachelor of Arts in government and international affairs, John F. Kennedy was medically disqualified from entering the Army Officer Candidate School due to lower back problems.
After a year of exercising and strengthening his back and with the help of the director of the Office of Naval Intelligence (ONI), Kennedy joined the United States Naval Reserve, on September 24, 1941.
He went on to serve in World War II and went on to be awarded the Navy and Marine Corps Medal for heroism and the Purple Heart Medal for injuries.
Read more: President John F Kennedy's US Navy career
H/T: UPI


--- layout: post title: "Programming in the small" published: 1 category: - Programming ---
I've really enjoyed reading Ivan Moore's Programming in the Small series. Lots of really nifty observations on how to improve your code a little bit at a time. My favorite quote thus far:
Another "programming in the small" style thing that I often see, even by people who understand non-interacting fermions, are conditionals that are more complicated than they need to be....
public boolean isWhatever() {
if ( someCondition() == true ) {
return true;
} else {
return false;
}
}
Must control fist of death.