transformers
Contents
1.9. transformers#
1.9.1. papers#
<IP_ADDRESS>. high-performing#
nlp (see also this link)
early papers
attention is all you need (vaswani et al. 2017) - initial transformer
encoder-decoder transformer for seq-to-seq (most new models don’t have special encoder-decoder structure for translation)
Semi-supervised Sequence Learning (dai & quoc le, 2015)
context vector is weighted sum of context vector at each word
ULMFiT (howard & ruder, 2018)
BERT (devlin et al. 2018) - semi-supervised learning (predict masked word - this is bidirectional) + supervised finetuning
roberta (liu et al. 2019)
BART (lewis et al. 2019) - generalizes BERT with sequence-to-squence training: train by (1) corrupting text then (2) reconstruct the original text
ELMo (peters…zettlemoyer, 2018) - no word embeddings - train embeddings w/ bidirectional lstm (on language modeling)
XLNet (yang…quoc le, 2020)
GPT-4 (openai, 2023) - adds multimodal understanding + boosts context length to 32k
GPT-3 (brown et al. 2020) - identitical to GPT-2 except larger and replaces dense attention with sparse attention
sizes: largest has 175B params, 96 layers, 96 heads in each layer, head with dim 128, vocab size ~50k
InstructGPT (ouyang…lowe, 2022)
GPT-2 (radford et al. 2018)
GPT (radford et al. 2018)
Gopher (deepmind, 2021) - basically gpt-3 with slight mods (replace layernorm by RMSnorm, different positional embeddings)
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (clark…quoc le, chris manning, 2020)
more efficient: rather than standard masked training, use generator-discriminator setup for “token detection”
generator replaces many masked tokens with plausible samples - train with MLM
discriminator tries to guess which tokens were the masked ones - this is the main model that gets used
Longformer: The Long-Document Transformer (beltagy, peters, & cohan, 2020) - processes very long contexts
PaLM: Scaling Language Modeling with Pathways (google, 2022) - 540 Billion params
pathways hardware center allows for fast/efficient training
discontinuous improvements - at some point large model improves
prompt engineering: “Explain yourself” - lets it explain jokes
Chinchilla: Training Compute-Optimal Large Language Models (deepmind, 2022)
“chinchilla scaling laws” - for compute-optimal training, the model size and the number of training tokens should be scaled equally
T0 (sanh…rush, 2022) - multitask training enables better zero-shot generalization
T5 (raffel…liu, 2020) – text-to-text transfer transformer
UL2: Unifying Language Learning Paradigms (tay…metzler, 2022) - open-source 20B model, beats GPT-3 at zero-shot
more effective training
instruction following
FLAN-PaLM: Scaling Instruction-Finetuned Language Models (chung, …, quoc le, jason wei, 2022) - finetune with datasets phrased as instructions
instructGPT / FLAN - finetune on instructions to follows instructions
human feedback
Learning to summarize with human feedback (OpenAI, 2020)
Can language models learn from explanations in context? (lampinen et al. 2022)
natural language feedback (scheurer et al. 2022) - makes training more efficient
other
text-vision models
CLIP (radford et al. 2021) - jointly train text/images
batch-based loss: encodings from same image/text pair should be close while encodings across different examples in the batch should be different
note: empirically works better with very large batch size
DALL-E 2 (OpenAI, 2022)
clip is foundation as generative model
generates text + image embeddings
“prior network” maps text embedding to image embedding
adds diffusion model
Stable diffusion (stability.ai, 2022) - open-source recreation, now highly optimized for speed
Imagen (google, 2022)
BLIP-2 (salesforce, 2023) - Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
BEiT-3 (2022) - treat vision as language and large-scale multimodal training
outperforms Flamingo: a Visual Language Model for Few-Shot Learning (2022), which uses more domain knowledge to connect vision & language
video
Text-To-4D Dynamic Scene Generation (meta, 2023)
vision
VIT: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (dosoviskiy, …, houlsby, 2020)
attention augmentation to resnet for vision (bello…quoc le, 2020)
here, people call image patches “tokens”
DINO Emerging Properties in Self-Supervised Vision Transformers (caron…joulin, 2021)
Masked Autoencoders Are Scalable Vision Learners (he…dollar, girshick, 2021) - BERT-style training
speed up by not applying encoder to mask tokens + adding mask to a lot of the data (like 75%)
really good results without much data
spatial transformers networks (deepmind, 2015)
rl
AdA: Human-Timescale Adaptation in an Open-Ended Task Space (deepmind, 2023)
GATO: A Generalist Agent (deepmind, 2022) - single agent plays many different video games
different modalities are converted to tokens differently (e.g. image patches are fed through resnet)
In-context Reinforcement Learning with Algorithm Distillation (laskin, wang, …, sahni, satinder singh, mnih, 2022, deepmind) - learn to improve an RL algorithm
put history of (observation, action, reward) sequences into context and then use them to predict new action given new observation
Decision Transformer: Reinforcement Learning via Sequence Modeling (chen, lu, …abbeel, srinivas, mordatch, 2021) - transformer that predicts what the next highest reward step is instead of the next word
biomedical
BioGPT: Generative pre-trained transformer for biomedical text generation and mining (luo…poon, liu, 2022)
PubMedGPT (2.7B): (bolton, hall, …, manning, liang, 2022)
BioBERT: A pre-trained biomedical language representation model for biomedical text mining (2019)
PubMedBERT: Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing (gu…gao, poon, 2021)
Large Language Models Encode Clinical Knowledge (singhal, …, natarajan, 2022, google/deepmind) - introduce MultiMedQA dataset + derive Med-PaLM, a prompt-tuned version of PaLM
question-answering (now just done with generic LLMs)
UnifiedQA: Crossing Format Boundaries With a Single QA System (khashabi…hajishirzi, 2020)
tabular
TabLLM: Few-shot Classification of Tabular Data with Large Language Models (hegelsmann…, sontag, 2022)
metalearning
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second (hollman, …, hutter, 2022) - transformer takes in train + test dataset then outputs predictions
builds on prior-data fitted networks (PFNs) (muller, …, hutter, 2021)
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes (garg, tsipras, liang, & valiant, 2022) - models can succesfully metalearn functions like OLS
e.g. during training, learn inputs-outputs from different linear functions
during testing, have to predict outputs for inputs from a different linear function
also test on slightly harder functions, like decision trees and 2-layer nets
Looped Transformers as Programmable Computers (giannou, …, jason lee, papailiopoulos, 2023 - use transformers as universal computers by programming them with specific weights
dialog
ChatGPT
GODEL: Large-Scale Pre-Training for Goal-Directed Dialog (baolin peng, galley, …, gao , 2022) - add grounded pre-training
Deal or No Deal? End-to-End Learning for Negotiation Dialogues (lewis…batra, 2017, Meta) - controversial paper where agents “make up their own language”
this is pre-transformers
MINERVA: Solving Quantitative Reasoning Problems with Language Models (google, 2022) - train on well-parsed, domain-specific data (math arxiv) to solve math-reasoning problems
autoformalization (wu…, szegedy, 2022) - translating from natural language math to formal language
produce sql/python that then finds an answer (cheng…zettlemoyer, smith, yu, 2022)
CODEX: Evaluating Large Language Models Trained on Code (2021, openai)
Repair Is Nearly Generation: Multilingual Program Repair with LLMs (joshi et al. 2022)
Improving automatically generated code from Codex via Automated Program Repair (fan et al. 2022) - use automated program repair to tweak codex outputs to make them better
Generating Question Titles for Stack Overflow from Mined Code Snippets (gao et al. 2020)
Automatic Program Repair with OpenAI’s Codex: Evaluating QuixBugs (prenner & robbes, 2021)
use prompt like:
#### fix the bug in the following function <buggy function and/or docstring here> #### fixed function
program synthesis arxiv.org/abs/2108.07732 - formalize natural language into runnable code
embeddings
Instructor: One Embedder, Any Task: Instruction-Finetuned Text Embeddings (su, …, smith, zettlemoyer, yu, 2022) - embedding is contextualized to eaach task
science
Galactica: A Large Language Model for Science (taylor…, stojnic, 2022, meta ai) - trained on mostly papers + some knowledge bases (e.g. proteins)
music
MusicLM: Generating Music From Text (google, 2023)
Jukebox: A Generative Model for Music (openai, 2020)
summarization / keywords
KeyBERT: Minimal keyword extraction with BERT (grootendorst, 2020)
<IP_ADDRESS>. external knowledge / tool use / grounding#
private
https://www.perplexity.ai/ - nice demo adding citation to each fact
langchain library
https://www.fixie.ai/ - provide tools for wrapping APIs in LLM + interaction through router (also default modules for stateful storage, user identity, etc.)
review
Augmented Language Models: a Survey (meta, 2023) – 3 categories: reasoning, tools, action
Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP (khattab, …, liang, potts, & zaharia, 2022) - use high-level programs to use multiple steps between retrieving and reading
Toolformer: Language Models Can Teach Themselves to Use Tools (meta, 2023) - model trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction
Given input, sample position and API call candidates, try them all, and filter out ones which do not reduce next-token loss
put correct API calls into prompt, e.g. Pittsburgh is also known as
[QA(What ...?→ Steel City)]
the Steel City.
Training
start with few human-written examples of API use
LLM generates more uses
self-supervised loss determines which calls help with future-token prediction
Atlas: Few-shot Learning with Retrieval Augmented Language Models (meta, 2022)
retreival-augmented in-context learning (put retrieved into into context, or something very similar)
REALM (guu, …, chang, 2020) - retrieves document chunks from corpus and adds them to context, for open-domain QA
RETRO (deepmind, 2022) - nearest neighbors to model’s input are retrieved, encoded, and conditioned on with chunked cross-attention
Decomposed prompting (khot et al., 2022) - decompose tasks via prompting which are delegated to a shared library of prompting-based LLMs dedicated to these sub-tasks
knowledge base triplets
Relational Memory-Augmented Language Models (liu, yogatama, & blunsom, 2022) - integrate knowledge base triplets with LLM
DRAGON: Deep Bidirectional Language-Knowledge Graph Pretraining (yasanaga, …, manning, liang, leskovec, 2022)
webgpt (nakano, …, schulman, 2022, OpenAI) - allows google search to add world info
Internet-augmented language models Lazaridou et al., 2022
GopherCite (menick, …, mcaleese, 2022, Deepmind) - generate answers + link/relevant snippet when making predictions (trained with RL from human preferences )
LaMDA (thoppilan, …, quoc le, 2022, google) - allows google search to add world info (in a dialog model)
this was the model that sparked the controversy about consciousness 🤔
A Neural Corpus Indexer for Document Retrieval (wang…yang, 2022) - train model to directly spit out document IDs given queries
RLPG (shrivastava, larochelle, & tarlow, 2022) - for code-completion, retrieves functions from a repo
memorizing transformers (wu…szegedy, 2022) - knn-based learned indexing + retrieval at training time
at test time, you just need to index the entire context and the model will be able to use it
kNN Prompting: Learning Beyond the Context with Nearest Neighbor Inference (xu…zhang, 2023)
kNN-Prompt: Nearest Neighbor Zero-Shot Inference (shi…zettlemoyer, 2022)
ACT-1: Transformer for Actions (2022, Adept) - transformer directly interacts with computer
ReAct: Synergizing Reasoning and Acting in Language Models (yao…cao, 2022) - use LLMs to generate reasoning traces + task-specific actions in interleaved manner
<IP_ADDRESS>. adaptation / transfer#
These are transformer-specific. For more general notes, see 📌 transfer learning or 📌 uncertainty. Most of these approaches can be combined with metalearning.
finetuning
finetune all DNN params
finetune linear layer on activations
standard - train linear model on the embedding of the first token (usually an added
[CLS]
token) (peters et al. 2018)finetune linear model on all the activations
e.g. evci, et al. 2022 - learn linear layer (using group-lasso) on features extracted from all layers
finetune specific DNN params (e.g. just the bias terms)
Cutting Down on Prompts and Parameters (logan…sameer singh, riedel, 2021) - finetune only the bias terms; works even with null prompts
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (zaken, ravfogel, & goldberg, 2021) - finetune only bias terms
adapter - finetune lightweight layers on top of pre-trained layers (between finetuning all layers, and just finetuning a new layer)
add some new layers and retrain some specific things (all human choices)
side-tuning (zhang, sax…malik, 2020) - train a “side” network that is fused with the pretrained model via summation
Combining Modular Skills in Multitask Learning (ponti, sordoni, bengio, & reddy, 2022) - learn adaptor with disentangled inventory of skills
predict a mask
ablate some model weights by training a binary mask over model parameters (Zhao et al., 2020; Radiya-Dixit and Wang, 2020)
predict mask over attention heads
prompting = few-shot learning = priming = in-context learning (starts with GPT)
prompting without changing any model parameters
limitation: can’t exploit sets longer than the training window
MetaICL: Learning to Learn In Context (min et al. 2022) - tune LLM to do in-context learning on a large set of training tasks (few-show prompting and training time and at test-time)
Visual Prompting via Image Inpainting (bar…darrell, globerson, efros, 2022)
PatternExploiting Training (PET) – Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference (schick & schutze, 2021)
cloze questions - same as masked language modeling: task is to replace some missing words
use cloze-question templates (e.g. it was “good” or “bad”) to get soft labels for unlabeled data and then finetune on theses
prompt-tuning (also see next section on autoprompting)
Attentional Mixtures of Soft Prompt Tuning for Parameter-efficient Multi-task Knowledge Sharing
Mixture of Soft Prompts for Controllable Data Generation (chen, … yu, 203) - LLMs as Synthetic Data Generators for Training Smaller Models
mt-dnn line of work
Multi-Task Deep Neural Networks for Natural Language Understanding (xiaodong liu … gao 2019) - multi-task learning on the 9 glue tasks (first layers are shared, then some task-specific layers at top)
RAdam: On the Variance of the Adaptive Learning Rate and Beyond (liyuan liu…gao, han, 2020)
usually need to do learning-rate warmup when trainin (e.g. with Adam)
RAdam = add a term to rectify the variance of the adaptive learning rate in Adam
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization (jiang…gao, zhao, 2020)
Smoothness-inducing regularization, which effectively manages the complexity of the model
Bregman proximal point optimization to prevent aggressive updating
Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding (xiaodong liu…gao, 2020)
Posterior Differential Regularization with f-divergence for Improving Model Robustness (hao cheng, …, gao 2021)
regularize model posterior difference between clean + noisy inputs (e.g. adversarially attacked inputs)
<IP_ADDRESS>. pruning#
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (frantar & alistarh, 2023) - prune GPT-style models to atleast 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy
Cramming: Training a Language Model on a Single GPU in One Day (geiping & goldstein, 2022) - tricks for training BERT
1.9.2. prompting#
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (liu…neubig, 2021)
from feature-engineering -> architecture engineering -> prompt engineering
LAMA Language Models as Knowledge Bases? (petroni…riedel, 2019) - Proposes using fill-in-the-blank (cloze) prompts for extracting knowledge from large language models
create LAMA probe - dataset of (subject, relation, object) triplets with templates – find that BERT can recall these relations
How to Query Language Models? (adolphs et al. 2021) - query LLMs by example (e.g. “Ronaldo plays for Portugal. Who does Neuer play for?”)
How Can We Know What Language Models Know? (jiang … neubig, 2020)
mining-based and paraphrasing-based methods to automatically generate high-quality diverse prompts
ensemble methods to combine answers from different prompts (e.g. avg logits and more)
Noisy Channel Language Model Prompting for Few-Shot Text Classification (min et al. 2022)
Querying \(P(question|answer)\) with Bayes rule outperforms standard querying \(P(answer|question)\)
memory-assisted prompt-editing (madaan…yang, 2022) - allows model to “save things to memory” that get added to prompt when needed
Prompting Is Programming: A Query Language For Large Language Models (Beurer-Kellner, Fischer, & Vechev, 2022)
<IP_ADDRESS>. autoprompting#
natural-language prompting
iPrompt: Explaining Patterns in Data with Language Models via Interpretable Autoprompting (singh, morris, …gao, 2022)
APE: Large Language Models Are Human-Level Prompt Engineers (zhou…ba, 2022)
similar to iPrompt, (1) propose prompt candidates with an LLM, (2) score the prompts by the accuracy they yield when using another LLM and (3) regenerate similar prompt candidates
experiments on instruction induction datasets + truthful QA
FluentPrompt: Toward Human Readable Prompt Tuning (shi, …, zettlemoyer, 2022) - use langevin sampling + fluency constraint to generate prompt
experiments relatively weak: 3 sentiment datasets + autoprompt is the only baseline
discrete prompting
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts (shin…sameer singh, 2020)
select prompts from a fixed set of tokens (resulting prompts are not coherent)
only work on MLM
elicit sentiment / factual knowledge
Universal Adversarial Triggers for Attacking and Analyzing NLP (wallace…sameer singh, 2019) - find input-agnostic sequences of tokens that trigger a model to produce a specific prediction when concatenated to any input from a dataset
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (deng…hu, 2022)
LM-BFF: Making Pre-trained Language Models Better Few-shot Learners (gao et al. 2020) - uses T5 to generate (i) template for the task (which might include a whole example or two) + (i) appropropriate label tokens in the vocabulary for the task (suffers from computationally intensive search + sub-optimal discrete space search)
PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains (ben-david, …, reichart, 2022)
prompt ensembles
PromptBoosting: Black-Box Text Classification with Ten Forward Passes (hou, …, jacob andreas, …, zhang, 2022) - get a small pool of prompts, learn a verbalizer (final classification layer) for each, then ensemble them with AdaBoost on LLM output
people have studied many works on prompt ensembling (e.g. lester et al. 2021)
PRBOOST: Prompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learning (zhang…zhang, 2022) - iteratively (1) select high-error examples, (2) have human label them as rules, and (3) use boosting to train model on the new rules + ensemble
typical rule generation
Snuba (Varma and Ré, 2018) generates heuristics based on a small labeled dataset with pre-defined rule types
TALLOR (Li et al. 2021a) & GLaRA (Zhao et al. 2021) study rule expansion for NER problem based on lexical information and then select rules based on a hand-tuned threshold
PTR: Prompt Tuning with Rules for Text Classification (han et al. 2021) – use logic rules to construct prompts with sub-prompts for many-class text classification (prompt is constructed hierarchically, but only one call is made to the LLM for inference)
Prefix-Tuning: Optimizing Continuous Prompts for Generation (li & percy liang, 2021) – optimizes in continuous space for language generation tasks
learn to map some parameters \(\theta\) through and MLP to generate a starting hidden state \(h_i\) – never actually sends the prefix through the network
Control Prefixes for Parameter-Efficient Text Generation (clive, cao, & rei, 2022) - allow for adapting the prefix to each input example
DART Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners (zhang…chen, 2022)
reformulating NLP task into differentially optimizing the prompt template + target label (given a pre-trained model)
focus on smaller models (Roberta-large + GPT-2) + few training shots
fluency constraint to ensure association among prompt embeddings
P-Tuning – GPT Understands, Too (liu et al. 2021) – use LSTM to generate prompt embeddings (don’t map to tokens)
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification (hu et al. 2021) – add knowledge-base info into the prompt search
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts (qin & eisner, 2021); github
use continuous tokens and ensemble (don’t map back to words)
WARP: Word-level Adversarial ReProgramming (Hambardzumyan et al. 2021) - add continous tokens (don’t map back to words) + some task-specific parameters for better generalization
KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction (chen et al. 2021) – incorporate relations, visualize learned prompt vectors with t-SNE
Calibrate Before Use: Improving Few-Shot Performance of Language Models (zhao, …, dan klein, sameer singh, 2021) - in order to make prompting easier, first calibrate output distr by making it uniform when given null inputs, e.g. “N/A”
misc
SentiPrompt: Sentiment Knowledge Enhanced Prompt-Tuning for Aspect-Based Sentiment Analysis – use sentiment knowledge penalties in the prompt
Meta-learning via Language Model In-context Tuning (Chen et al. 2022) – Given new task with new instruction
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm (Reynolds & McDonell, 2021) – define metaprompts as general wrappers around tasks e.g. “This problem asks us to”
Re3: Generating Longer Stories With Recursive Reprompting and Revision (yang, …, klein, 2022) - generate summaries, then expand and revise with prompts
Directional Stimulus Prompting (li, baoling peng, …jianfeng gao, xifeng yan, 2023) - generate hint keywords using small LLM that are put into the prompt when calling large LLM
critiques of prompting
Do Prompt-Based Models Really Understand the Meaning of their Prompts? (webson & pavlick, 2022) - models can learn fine with prompts that are intentionally irrelevant
Are Language Models Worse than Humans at Following Prompts? It’s Complicated (webson, …, pavlick, 2023)
can benefit from training for promptability
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (zhong…klein, 2021)
Continued Pretraining for Better Zero- and Few-Shot Promptability (wu…sameer singh, beltagy, 2022)
What learning algorithm is in-context learning? Investigations with linear models - investigate prompting through synthetic experiments with transformers trained for linear regression
Transformers as Algorithms: Generalization and Implicit Model Selection in In-context Learning (li, …, oymak, 2023) - generalization bounds for in-context learning when the input prompt is (1) a sequence of i.i.d. (input, label) pairs or (2) a trajectory arising from a dynamical system
<IP_ADDRESS>. llm chaining / decoding#
many notes are from this thread on chaining models together
steering
overviews
Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts (wu, terry, & cai, 2022) - chaining LLM steps together: output of one step becomes the input for the next
interactive system where users can modify chains + their intermediate results – improves performance + human experience
Language Model Cascades (dohan…sutton, 2022) - treat chaining models as probabilistic programs
use a probabilistic-programming language (PPL) to define a joint probability model on string-valued random variables, parameterized using LMs, and then condition this model on string-valued observations in order to compute a posterior over string-valued unknowns
self-PPLs extend probabilistic graphical models to support more complex joint distributions whose size and “shape” can itself be stochastic
e.g., a graph unrolled for a random number of iterations, until a data-dependent stopping criterion is met
variables are all text: questions \(Q\), answers \(A\), and intermediate thoughts \(T\)
posthoc
2023
Faithful Chain-of-Thought Reasoning (lyu et al. 2023)
Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks (chen et al. 2022)
PAL: Program-aided Language Models (gao…neubig, 2023)
Chain of Thought Prompting (wei et al. 2022)
in few-shot prompts, don’t just provide answer but also reasoning
model output then provides reasoning + answer
Self-Consistency Improves Chain of Thought Reasoning in Language Models (wang, wei, schuurmans, quoc le, … zhou, 2022) - use output samples rather than greedy and return the most consistent final answer in the set
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (suzgun, …, quoc le, …, jason wei, 2022)
self-ask (Press et al., 2022) - LLM asks itself (and then answers) follow-up questions before answering the initial question
scratchpads Show Your Work: Scratchpads for Intermediate Computation with Language Models (nye et al. 2021)
selection inference (creswell et al. 2022) - generate set of facts, then iteratively generate inferences from the facts to yield the final answer
least-to-most prompting (zhou…quoc le et al. 2022) - prompt LLM with context showing how to reduce into subproblems; then LLM sequentially solves the subproblems, using the previous answers
Generated Knowledge Prompting for Commonsense Reasoning (liu…hasjishirzi, 2021) - generate knowledge from an LLM then provide it as additional input when answering a question
maieutic prompting (jung et al. 2022) - generate a tree of all explanation of the form “True, because…”, “False, because…” then query LLM with these as prompts
then use Max-SAT to try to satisfy as many relations between the model explanations as possible to come up with the true answer
training
verifiers (cobbe et al. 2021) - train model to judge whether an answer and thought are likely to be “valid”
subgoal search (czechowski et al. 2021) - train model to generate subgoals then solve them in a graph
STaR “Self-taught reasoner” (zelikman…goodman, 2022)
first, finetune on observed \((Q, T, A)\) triplets, where \(T\) is a rationale
then, impute unknown \(T_i\) given dataset of pairs \((Q_i, A_i)\) by sampling until finding a \(T_i\) which leads to the correct answer
robotics-specific
zero-shot planning (huang, abbeel, pathak, & mordatch, 2022)
1.9.3. misc#
<IP_ADDRESS>. direct weight inspection#
nice paper list here
Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors (yun, chen, olshausen, lecun, 2021) - investigate LLM embeddings of different words using dictionary learning
LLMs produce interesting contextualized word embeddings
dictionary elements (of activations across layers) correspond to meaningful things
dictionary element has size \(d\), the embedding size
given list of sentences \(S\), training matrix has size \(\left(\underbrace{\text{num\_layers}}_{\text{12 for BERT}} \cdot \sum_{s \in S} \text{len(s)}\right) \times \underbrace{d}_{\text{768 for BERT}}\)
dictionary coefficient is for a specific position: maps (text, sequence_index, layer) \(\to\) coefficient
Neuron-level Interpretation of Deep NLP Models: A Survey (sajjad et al. 2022)
previous works generally use pre-specified concepts, and focus on
concept search - given a neuron find its concept(s)
neuron search - (ii) given a concept find its matching neuron(s)
concept search
visualization, e.g. karpathy, johnson, fei-fei li, 2015 visualize LSTM head response in text
elicit top-k ngram responses on a corpus, which are then labelled manually (kadar et al. 2017)
elicit top-k activating sentences from a corpus, which are then summarized using a parse tree into a synthetic explanation (na…kim, 2019)
limitation: the explanation may be ungrammatical and biased towards something arbitrary (like reptition)
input maximization (e.g. textattack, poerner et al. 2018)
Evaluating Neuron Interpretation Methods of NLP Models (fan…sajjad, 2023) - metric is how well evaluation from one method matches the other ones
A Circuit for Indirect Object Identification in GPT-2 small (wang, …, steinhardt, 2022)
explanation encompasses 26 attention heads grouped into 7 main classes
task: indirect object identification - “When Mary and John went to the store, John gave a drink to ___” should be “Mary”
circuit
identify all previous names
remove duplicated names
output remaining name
MILAN: Natural Language Descriptions of Deep Visual Features (hernandez…david bau…torallba, andreas, 2022) - given a neuron, generates a natural-language string that maximizes pointwise mutual information with the image regions in which the neuron is active
Finding Skill Neurons in Pre-trained Transformer-based Language Models - some individual neurons are predictive of the final task (dubbed “skill neurons’)
thread (elhage…olah, 2021)
all layers are same dimension and each attention block adds a vector to it
Although they’re parameterized as separate matrices, \(W_O W_V\) and \(W_Q^T W_K\) can always be thought of as individual, low-rank matrices
\(x \in \mathbb R^{d_{embed} \times d_{sequence}}\): \(d_{embed}\) can be hundreds - tens of thousands
\(W_Q, W_K, W_V \in \mathbb R^{d_{attn} \times d_{embed}}\)
\(W_Q^TW_k \in \mathbb R ^{d_{embed} \times d_{embed}}\)
\(W_O \in \mathbb R^{d_{embed} \times d_{attn}}\): projects attention values back to embedding dimention
\(W_O W_V \in \mathbb R ^{d_{embed} \times d_{embed}}\)
\(W_E \in \mathbb R^{d_{embed} \times d_{vocab}}\) embeds initial tokens and \(W_U \in \mathbb R^{d_{vocab} \times d_{embed}}\) undoes the embedding
\(d_{vocab}\) can be very large, e.g. 50k
\(A = \text{softmax}(x^TW_Q^TW_kx) \in \mathbb R^{d_{sequence} \times d_{sequence}}\)
if we have a 0-layer net (e.g. predict next token with linear layer given current token), we just learn bigram log-likelihood
2 circuits
QK circuit determines which “source” token the present “destination” token attends back to and copies information from
\(W_{E}^{T} W_{Q}^{T} W_{K} W_{E} \in \mathbb R ^{d_{vocab} \times d_{vocab}}\)
OV circuit describes what the resulting effect on the “out” predictions for the next token is
\(W_{U} W_{O} W_{V} W_{E} \in \mathbb R ^{d_{vocab} \times d_{vocab}}\)
if a single head increases the probability of both
keep… in mind
andkeep… at bay
, it must also increase the probability ofkeep… in bay
andkeep… at mind
induction heads search previous examples of present token
If they don’t find it, they attend to the first token and do nothing
if they do find it, they then look at the next token and copy it. This allows them to repeat previous sequences of tokens, both exactly and approximately
sometimes can do some kind of “fuzzy” matching
tensor/kronecker product \(\bigotimes\):
Left-right multiplying: Multiplying \(x\) by a tensor product \(A \otimes W\) is equivalent to simultaneously left and right multiplying: \((A \otimes W) x=A x W^{T}\)
When we add them, it is equivalent to adding the results of this multiplication: \(\left(A_{1} \otimes W_{1}+A_{2} \otimes W_{2}\right) x=A_{1} x W_{1}^{T}+A_{2} x W_{2}^{T}\) Softmax Linear Units
replacing activation function with softmax linear unit increases fraction of MLP neurons which are “interpretable”, i.e. correspond to meaningful features
however, may “hide” some non-neuron-aligned features by decreasing their magnitude and then later recovering it with LayerNorm
the presence of nonlinear activation functions createse an incentive for features to align with this basis and not get superposed
if the gains to sparse coding are large enough, this incentive will get overwhelmed
ways to combat polysemanticity
activation sparsity
lateral inhibition / co-occurrence sparsity
weight sparsity
superlinear activation functions
increase neurons per param
\(\text{SoLU}(x) = x \cdot \text{softmax}(x)\)
adds lateral inhibition, superlinearity, approximate sparsity
changes GeLU, which is approximately \(\text{sigmoid}(1.7x) \cdot x\)
just changing to SoLU decrease performance, had to add LayerNorm afterwards
<IP_ADDRESS>. attention variants#
Tree Transformer: Integrating Tree Structures into Self-Attention (wang, .., chen, 2019)
Waveformer: Linear-Time Attention with Forward and Backward Wavelet Transform (zhuang…shang, 2022)
<IP_ADDRESS>. editing#
Locating and Editing Factual Associations in GPT (meng, bau et al. 2022 )
localize factual associations - causal intervention for identifying neuron activations that are decisive in a model’s factual predictions
“causal traces” - run net multiple times, introducing corruptions and then restore states from original non-corrupted forward pass to see which states can restore the original results
a small number of states contain info that can flip the model from one state to another
change factual associations - modify feedforward weights to update specific factual associations using Rank-One Model Editing (ROME)
Mass Editing Memory in a Transformer (meng…, bau, 2022)
Knowledge Neurons in Pretrained Transformers (dai et al. 2021) - integrated gradients wrt to each neuron in BERT
Memory-Based Model Editing at Scale (mitchell…manning, finn, 2022)
keep track of list of edits in external memory and use them as appropriate context at test time (don’t finetune the model)
Fast model editing at scale (mitchell…finn, manning, 2022)
a collection of small auxiliary editing networks that use a single desired input-output pair to edit a pre-trained model
MEND learns to transform the gradient obtained by standard fine-tuning, using a low-rank decomposition of the gradient
<IP_ADDRESS>. debugging / interpretation#
TalkToModel: Understanding Machine Learning Models With Open Ended Dialogues (slack…lakkaraju, sameer singh, 2022) - natural language interface to query model (by converting to commands such as filtering the data / calculating importance)
Rethinking Explainability as a Dialogue: A Practitioner’s Perspective (lakkaraju, slack, …, sameer singh, 2022) - interviews with high-stakes users suggest they would like to be able to interact with systems via dialog
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning (ye & durrett, 2022)
AdaTest Adaptive Testing and Debugging of NLP Models (ribeiro & lundberg, 2022)
goal: easily specify, discover, and fix undesirable behaviors in an NLP model
2-step iterative algorithm
LLM generates many tests targeting the model’s failures
example of a test:
f(“I am a black woman”) ≠ neg
user selects and organizes the tests and reprompts the LLM to find more
User fixes the tests (e.g. via finetuning)
Checklist Beyond Accuracy: Behavioral Testing of NLP models with CheckList (ribeiro…sameer singh, 2020)
matrix of general linguistic capabilities + test types
Fixing Model Bugs with Natural Language Patches (murty, manning, lundberg, & ribeiro 2022)
specify patches with natural language rather than hard rule, allowing them to better handle text
finetune a model to combine original model output with output from a patch-conditioned interpreter head
augmenting interpretable models
Emb-GAM: an Interpretable and Efficient Predictor using Pre-trained Language Models (singh & gao, 2022)
use language model to extract embeddings which are then used to fit a better GAM (focuses on NLP)
CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with Large Language Models (mcinerney, ..wallace, 2023)
extract interpretable feature (e.g. “Does this patient have a chronic illness?”) and use in a linear model
<IP_ADDRESS>. symbolic reasoning#
See also notes on 📌 comp neuro.
GPT-3 Large Language Models are Zero-Shot Reasoners - simply adding “Let’s think step by step” before each answer increases the accuracy on MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with GPT-3
Compositional processing emerges in neural networks solving math problems (russin, roland fernandez, …, smolensky, gao, 2021)
Modular Deep Learning (pfeiffer, ruder, .., ponti, 2023) - overview of different modular architectures
neurocompositional computing (smolensky…gao, 2022)
longer tutorial (smolensky, …, gao, 2022)
central paradox of cognition is that brain both uses continuous neural symbols but is compositional (smolensky et al. 1992)
Compositionality
Continuity - the encoding and processing of information is formalized with real numbers that vary continuously
3 challenges
compositional generalization
data efficiency
comprehensibility
solution - NECST: Neurally-Encoded Compositionally-Structured Tensor computing (smolensky & legendre, 2006) - basically leverages TPR
TPR roles and fillers can both be made continuous
neural space vs symbolic space (many different things (e.g. sentences) can mean the same thing)
word vectors can be thought of as “soft symbols”
want to move from symbolic repr. to neural repr. while keeping interpretability
system should output intermediate steps in addition to answer
thinking fast (system 1: fast, intuitive) + slow (system 2: slower, logical, derivative)
concrete proposals
transformer activation vector should encode graph of flow through the network
ex. task: regurgitate a sequence
TPR: Tensor product variable binding and the representation of symbolic structures in connectionist systems (paul smolensky, 1990) - activation patterns are “symbols” and internal structure allows them to be processed like symbols
tensor product representation = TPR
TPR of a structure is the sum of the TPR of its constituents
tensor product operation allows constituents to be uniquely identified, even after the sum (if roles are linearly independent)
filler - one vector that embeds the content of the constituent
role - second vector that embeds the structural role it fills
NECSTransformer: Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving (schlag, …, gao, 2019)
TP-attention
beat SOAon free-form math word-problems
in addition to K, Q, V, also add a role-vector
do element-wise multiplication of outputted vector with role-vector
TPR built as tensor product of 2 vectors:
filler - the vector returned by attention
ex. one head learns “second-argument-of”
role - a relation conceptually labeling an edge of the attention graph
TP-N2F: Tensor Product Representation for Natural To Formal Language Generation - Microsoft Research (chen…gao, 2019)
<IP_ADDRESS>. sparse experts / ensembles / mixture of experts (MoE)#
mixture of experts models have become popular because of the need for (1) fast speed / low memory at test time while still (2) having a large model during training
note: nowadays often the “experts” are different MLPs following the self-attention layers
A Review of Sparse Expert Models in Deep Learning (fedus, jeff dean, zoph, 2022)
sparsity decouples the parameter count from the compute per example allowing for extremely large, but efficient models
routing algorithm - determines where to send examples
discreteness makes it difficult
some works use RL to learn routing
standard approach uses gumbel-softmax
usually get matrix of similarities between input tokens and experts and route based on these
sometimes route to topk experts rather than top1
load balancing - usually add an auxiliary loss to encourage equal tokens being sent to different experts
non-specialized experts
Early versions (Jacobs, michael jordan, nowlan, & hinton, 1991) had independent feed-forward networks serving as experts
Sparsely-gated MOE layer (Shazeer…quoc le, hinton, dean, 2017) have been studied with token-based routing with backprop
replace FFN in transformers with expert layers
GShard Lepikhin et al. (2021), which appplies this concept to machine translation
Switch transformers (Fedus et al. (2022)) simplifies the architecture to activation of only one expert per layer
BASE Layers Lewis et al. (2021) - find an alternative approach to routing by formulating it as a linear assignment problem
Hash layers Roller et al. (2021) use a fixed hash as the gating function
routing notes - make hard decision but still want to learn probabilities
straight-through estimator (STE) - take the argmax during the forward pass, while considering the orig- inal probabilities in the backward pass
highly biased
gumbel-softmax- allows for better sampling
specialized experts as fully independent models (sometimes for multi-task learning)
DEmix Layers Gururangan et al. (2022) – DEMix layers – placed in the feedforward layers of the Transformer – contain experts which specialize on specific domains. Routing at train time is determined only by the domain label, but all experts are activated at inference time and mixed according to weights estimated from a validation set
Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners (gupta…awadallah, gao, 2022) - use task description to improve routing
Pfeiffer et al. (2022) - multilingual expert model with language-specific routing
task-level MoE Kudugunta et al. (2021) – multi-task expert model with task-specific routing
ELMS – Branch-Train-Merge (li et al. 2022)
parallel language model of smaller expert LMs
each can be added/removed, ensembled, or parameter-averaged at any time for efficient scaling and rapid customization
improves perplexities, when controlling for training cost
require expert domain specialization
scaling up
OPT-MOE (artetxe et al. 2021)
AutoMoE (jawahar, mukherjee, liu…gao, 2022)
Interpretable entity representations through large-scale typing (onoe & durrett, 2020) - embedding is interpretable predictions for different entities/
Towards Understanding Mixture of Experts in Deep Learning (chen…gu, li, 2022)
ensembles (some of these are non-transformer papers)
model soups (wortsman…schmidt, 20221) - average weights of finetuned models
snapshot ensembles - average different checkpoints during training (huang et al. 2017)
stochastic weight averaging (izmailov, …, wilson, 2019) - average multiple checkpoints during training
batch ensemble (wen et al. 2020) - have several rank-1 keys that index different weights hidden within one neural net
Editing Models with Task Arithmetic (ilharco, rebeiro, wortsman…farhadi, 2022) - add task vectors (difference between weights before/after finetuning on a task) for different tasks to induce different multi-task behaviors
fit many models into one
superposition of many models into one (cheung…olshausen, 2019) - both during training/testing models are indexed via a high-dim key for each task
supermasks in superposition (wortsman, …, yosinski, farhadi, 2020) - randomly fixed based net + for each task finds subnet that chieves good performance
if task identity not given, correct subnet inferred by minimizing output entropy
Git Re-Basin: Merging Models modulo Permutation Symmetries (ainsworth, hayase, & srinivasa, 2022) - algo to merge models even when they haven’t been pretrained together
early exit - popular way to speed up inference
Multi-exit vision transformer for dynamic inference (Bakhtiarnia, A., Zhang, Q. and Iosifidis, A., 2021)
early layers have large activation map so early exist classifier must be complex
solution: ViT class token allows early-exit classifier to have constant complexity
DeeBERT: Dynamic early exiting for accelerating BERT inference (xin…lin, 2020)
<IP_ADDRESS>. llm querying / causal inference#
decoding
greedy - iteratively pick highest-probability token
nucleus sampling: The Curious Case of Neural Text Degeneration (holtzman…choi, 2019)
contrastive decoding (li et al. 2022) - decode based on the difference between a large and small LLM
Semantic Uncertainty (kuhn, gal, & farquhar, 2023) - yields uncertainties by incorporating linguistic invariances created by shared meanings
Discovering Latent Knowledge in Language Models Without Supervision (burns, ye, klein, & steinhardt, 2022) - identify whether text is true or false directly from a model’s unlabeled activations
InferBERT: A Transformer-Based Causal Inference Framework for Enhancing Pharmacovigilance (wang…liu, 2021) - learn + test feature relationships from attention weights
CausaLM: Causal Model Explanation Through Counterfactual Language Models (2021) - produce example-level causal model explanations using models finetuned on auxiliary adversarial tasks derived from the causal graph of the problem
Investigating Gender Bias in Language Models Using Causal Mediation Analysis (vig, …, shieber, 2020)
Applies causal mediation analysis to identify decisive neurons and attention heads responsible for gender bias in large language models
Identifies a small handful of decisive attention heads in this case
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (elazar, …, goldberg, 2021) - measure the importance of specific info within a model by introducing a causal intervention to erase that information, then observing the causal effects
<IP_ADDRESS>. dataset explanation#
iPrompt: Explaining Patterns in Data with Language Models via Interpretable Autoprompting (singh, morris, …gao, 2022 ) - prompting approach
Instruction Induction: From Few Examples to Natural Language Task Descriptions (honovich…bowman, levy 2022) - directly query model with prompt to search for task description
D3: Describing Differences between Text Distributions with Natural Language (zhong, snell, klein, & steinhardt, 2022) - finetune an LLM to directly describe difference between 2 text distrs
D5: Goal Driven Discovery of Distributional Differences via Language Descriptions (zhong, zhang, …, klein, & steinhardt, 2023) - add dataset-specific prompt + evaluation on larger set of 675 datasets
technically this is just learning a classifier, where the classifier is a natural-language string
method
proposer network generates hypotheses
verifier networks looks at all samples in the dataset (since proposer couldn’t fit them all in context) and returns how accurate the hypotheses were
some tricks
select samples which are “representative” of a class by predicting with another LLM
have a pool of 302 manual hypotheses they usefor seeding
GSCLIP : A Framework for Explaining Distribution Shifts in Natural Language (zhu…james zou, 2022) - automatically explain dataset-level distribution shifts (in image datasets) with natural language
<IP_ADDRESS>. connecting with rules#
Automatic Rule Extraction from Long Short Term Memory Networks (murdoch & szlam, 2017) - extract out phrases using feature importance
A Comparative Study of Rule Extraction for Recurrent Neural Networks (wang et al. 2018) - create automata based on interpretable states to track RNNs
<IP_ADDRESS>. cool tasks#
Forecasting Future World Events with Neural Networks (zou…hendrycks, 2022) - takes tasks from metaculus
forecasting paper titles (blog post)
Shortcut Learning of Large Language Models in Natural Language Understanding: A Survey (du et al. 2022)
Neurosymbolic Programming for Science (sun…costilla-reyes, 2022)
Learning from learning machines: a new generation of AI technology to meet the needs of science (berkeley+lbnl+, 2021)
do more than predict what will happen, they attempt to offer insight into how or why
AI-based language models powering drug discovery and development (liu et al. 2021)
BioTranslator: Multilingual translation for zero-shot biomedical classification (xu, woicik, poon, altman, & wang, 2023) - takes a user- written textual description of a new concept and then translates this description to a non-text biological data instance
results for biological data, e.g. genes, proteins
enables the identification of novel cell types using only a textual description
scientific organization (galactica)
related but smaller models
SciBERT (beltagy…cohan, 2019)
BioLM (lewis…stoyanov, 2020)
ScholarBERT (hong…foster, 2022) - large dataset, 770M-param model
all data is processed in a common markdown format
task-specific tokens to support different types of knowledge (e.g. citations, step-by-step reasoning, different modalities, e.g. proteins)
chemical compounds (train on 2 mil / 110 mil from PubChem Compound, authors still want it to focus on text)
predict IUPAC name from SMILES formula e.g.
CC(C)(C)C(=O)N(CC1=NC(=CS1)C(=O)OC)C2CCCCC2
->methyl 2-[[cyclohexyl-(2,2-dimethylpropanoyl)]amino] methyl]thiazole-4-
moleculenet (wu et al. 2017) classification benchmark (6 tasks)
training set examples are trained as text during fitting
HIV - classify whether comopund inhibits HIV replication
BACE C - binding results (classification + regression) for BACE
BBBP - blood-brain barrier penetration(permeability) (binary classification)
Tox21 - qualitative toxicity on 12 targets (12-class multilabel binary)
SIDER - 27-class multi-class disorders in different organ systems
ClinTox - binary toxicity classification
ex. for BBBP (one of the 6 tasks) - question is posed in different ways during training
Here is a SMILES formula: [START_I_SMILES]O=C(O)CCCC1=CC=C(N(CCCl)CCCl)C=C1[END_I_SMILES] Question: Will the chemical compound penetrate the blood-brain barrier? Answer: No
protein sequences
from 227 million in UniProt, look at only 0.5 million subset (called Swiss-Prot)
evaluate protein sequence perplexity
protein keyword prediction (predict keywords in UniProt, like “ATP-Binding”, “Cell membrane”)
protein function description - compare free-form description to GT UniProt function description
<IP_ADDRESS>. llm limitations / perspectives#
Dissociating language and thought in large language models: a cognitive perspective (mahowald, …, tenenbaum, fedorenko, 2023)
2 competences: (1) formal & (2) functional linguistic competence
speculative foundation models paper (stanford, 2022)
1.9.4. basics#
attention = vector of importance weights
to predict or infer one element, such as a pixel in an image or a word in a sentence, we estimate using the attention vector how strongly it is correlated with (or “attends to” other elements and take the sum of their values weighted by the attention vector as the approximation of the target
vanilla transformer: multihead attention, add + norm, position-wise ffn, add + norm
self-attention layer implementation, mathematics, and chandan’s self-attention cheat-sheet
<IP_ADDRESS>. mathematical overview of transformers (Formal Algorithms for Transformers)#
tasks
sequence modeling: learn \(p(x)\), usually factorized as \(p(x_i|x_1,...,x_{i-1})\)
sequence-to-sequence: learn \(p(z|x)\), e.g. transalation, speech-to-text, question answering
preprocessing
embedding matrix takes in one-hot tokens and linearly maps them to a vector
positional embedding of a token is usually added to the token embedding to form a token’s initial embedding
attention types
Bidirectional / unmasked self-attention - primary/context vectors are the same
Unidirectional / masked self-attention - mask scores from before a given word
Cross-attention - primary/context vectors can come from different places
non-attention
layernorm: controls mean/variance of activations
RMSnorm: simpler version, sets mean/offset to zero
unembedding
linear layer (with softmax) that outputs size of original vocab
sometimes fixed to be transpose of the embedding matrix
predictions
predict next word using single linear layer on hidden state from previous word
finetune classification head often only using linear layer on first token from sequence
architectures
initially, encoder-decoder was common, but now often no decoder
<IP_ADDRESS>. visual explanation (notes on article by jay allamar)#
**self-attention ** - layer that lets word learn its relation to other layers
for each word, want score telling how much importance to place on each other word (queries \(\cdot\) keys)
we get an encoding for each word
the encoding of each word returns a weighted sum of the values of the words (the current word gets the highest weight)
softmax this and use it to do weighted sum of values
(optional) implementation details
multi-headed attention - just like having many filters, get many encodings for each word
each one can take input as the embedding from the previous attention layer
position vector - add this into the embedding of each word (so words know how far apart they are) - usually use sin/cos rather than actual position number
padding mask - add zeros to the end of the sequence
look-ahead mask - might want to mask to only use previous words (e.g. if our final task is decoding)
residual + normalize - after self-attention layer, often have residual connection to previous input, which gets added then normalized
decoder - each word only allowed to attend to previous positions
3 components
queries
keys
values
attention
encoder reads input and ouputs context vector after each word
decoder at each step uses a different weighted combination of these context vectors
specifically, at each step, decoder concatenates its hidden state w/ the attention vector (the weighted combination of the context vectors)
this is fed to a feedforward net to output a word
at a high level we have \(Q, K, V\) and compute \(\text{softmax}(QK^T)V\)
instead could simplify it and do \(\text{softmax}(XX^T)V\) - this would then be based on kernel
transformer
uses many self-attention layers
many stacked layers in encoder + decoder (not rnn: self-attention + feed forward)
details
initial encoding: each word -> vector
each layer takes a list of fixed size (hyperparameter e.g. length of longest sentence) and outputs a list of that same fixed size (so one output for each word)
can easily train with a masked word to predict the word at the predicted position in the encoding
multi-headed attention has several of each of these (then just concat them)
<IP_ADDRESS>. huggingface tutorial#
Broadly, models can be grouped into three categories:
GPT-like (also called auto-regressive Transformer models)
BERT-like (also called auto-encoding Transformer models)
BART/T5-like (also called sequence-to-sequence Transformer models)
Handling multiple sequences - Hugging Face Course
pad sequences to have the same length (need to modify attention masks to ignore the padded values)
<IP_ADDRESS>.1. pre-transformer nlp models#
rnns
when training rnn, accumulate gradients over sequence and then update all at once
stacked rnns have outputs of rnns feed into another rnn
bidirectional rnn - one rnn left to right and another right to left (can concatenate, add, etc.)
standard seq2seq
encoder reads input and outputs context vector (the hidden state)
decoder (rnn) takes this context vector and generates a sequence


Just answer a few questions to see if you fit the criteria for our private club:
Woman
Man
If you are looking for a serious relationship, you are on wrong community!
Observe the following rules of our club:


|
|||||||||||||||||||
| Source file | Conditionals | Statements | Methods | TOTAL | |||||||||||||||
| RuleManager.java | 44,4% | 67,4% | 76,9% | 63,6% |
|
| 1 | /* | |
| 2 | * Copyright 2001-2005 (C) MetaStuff, Ltd. All Rights Reserved. | |
| 3 | * | |
| 4 | * This software is open source. | |
| 5 | * See the bottom of this file for the licence. | |
| 6 | */ | |
| 7 | ||
| 8 | package org.dom4j.rule; | |
| 9 | ||
| 10 | import java.util.HashMap; | |
| 11 | ||
| 12 | import org.dom4j.Document; | |
| 13 | import org.dom4j.Element; | |
| 14 | import org.dom4j.Node; | |
| 15 | import org.dom4j.rule.pattern.NodeTypePattern; | |
| 16 | ||
| 17 | /** | |
| 18 | *
 | |
| 19 | * RuleManager manages a set of rules such that a rule can be | |
| 20 | * found for a given DOM4J Node using the XSLT processing model. | |
| 21 | *
 | |
| 22 | * | |
| 23 | * @author <a<EMAIL_ADDRESS>Strachan  | |
| 24 | * @version $Revision: 1.9 $ | |
| 25 | */ | |
| 26 | public class RuleManager { | |
| 27 | /** Map of modes indexed by mode */ | |
| 28 | private HashMap modes = new HashMap(); | |
| 29 | ||
| 30 | /** | |
| 31 | * A counter so that rules can be ordered by the order in which they were | |
| 32 | * added to the rule base | |
| 33 | */ | |
| 34 | private int appearenceCount; | |
| 35 | ||
| 36 | /** Holds value of property valueOfAction. */ | |
| 37 | private Action valueOfAction; | |
| 38 | ||
| 39 | 5 | public RuleManager() { |
| 40 | } | |
| 41 | ||
| 42 | /** | |
| 43 | * DOCUMENT ME! | |
| 44 | * | |
| 45 | * @param modeName | |
| 46 | * DOCUMENT ME! | |
| 47 | * | |
| 48 | * @return the Mode instance for the given mode name. If one does not exist | |
| 49 | * then it will be created. | |
| 50 | */ | |
| 51 | 25 | public Mode getMode(String modeName) { |
| 52 | 25 | Mode mode = (Mode) modes.get(modeName); |
| 53 | ||
| 54 | 25 | if (mode == null) { |
| 55 | 4 | mode = createMode(); |
| 56 | 4 | modes.put(modeName, mode); |
| 57 | } | |
| 58 | ||
| 59 | 25 | return mode; |
| 60 | } | |
| 61 | ||
| 62 | 14 | public void addRule(Rule rule) { |
| 63 | 14 | rule.setAppearenceCount(++appearenceCount); |
| 64 | ||
| 65 | 14 | Mode mode = getMode(rule.getMode()); |
| 66 | 14 | Rule[] childRules = rule.getUnionRules(); |
| 67 | ||
| 68 | 14 | if (childRules != null) { |
| 69 | 0 | for (int i = 0, size = childRules.length; i < size; i++) { |
| 70 | 0 | mode.addRule(childRules[i]); |
| 71 | } | |
| 72 | } else { | |
| 73 | 14 | mode.addRule(rule); |
| 74 | } | |
| 75 | } | |
| 76 | ||
| 77 | 0 | public void removeRule(Rule rule) { |
| 78 | 0 | Mode mode = getMode(rule.getMode()); |
| 79 | 0 | Rule[] childRules = rule.getUnionRules(); |
| 80 | ||
| 81 | 0 | if (childRules != null) { |
| 82 | 0 | for (int i = 0, size = childRules.length; i < size; i++) { |
| 83 | 0 | mode.removeRule(childRules[i]); |
| 84 | } | |
| 85 | } else { | |
| 86 | 0 | mode.removeRule(rule); |
| 87 | } | |
| 88 | } | |
| 89 | ||
| 90 | /** | |
| 91 | * Performs an XSLT processing model match for the rule which matches the | |
| 92 | * given Node the best. | |
| 93 | * | |
| 94 | * @param modeName | |
| 95 | * is the name of the mode associated with the rule if any | |
| 96 | * @param node | |
| 97 | * is the DOM4J Node to match against | |
| 98 | * | |
| 99 | * @return the matching Rule or no rule if none matched | |
| 100 | */ | |
| 101 | 0 | public Rule getMatchingRule(String modeName, Node node) { |
| 102 | 0 | Mode mode = (Mode) modes.get(modeName); |
| 103 | ||
| 104 | 0 | if (mode != null) { |
| 105 | 0 | return mode.getMatchingRule(node); |
| 106 | } else { | |
| 107 | 0 | System.out.println("Warning: No Mode for mode: " + mode); |
| 108 | ||
| 109 | 0 | return null; |
| 110 | } | |
| 111 | } | |
| 112 | ||
| 113 | 0 | public void clear() { |
| 114 | 0 | modes.clear(); |
| 115 | 0 | appearenceCount = 0; |
| 116 | } | |
| 117 | ||
| 118 | // Properties | |
| 119 | // ------------------------------------------------------------------------- | |
| 120 | ||
| 121 | /** | |
| 122 | * DOCUMENT ME! | |
| 123 | * | |
| 124 | * @return the default value-of action which is used in the default rules | |
| 125 | * for the pattern "text()|@" | |
| 126 | */ | |
| 127 | 4 | public Action getValueOfAction() { |
| 128 | 4 | return valueOfAction; |
| 129 | } | |
| 130 | ||
| 131 | /** | |
| 132 | * Sets the default value-of action which is used in the default rules for | |
| 133 | * the pattern "text()|@" | |
| 134 | * | |
| 135 | * @param valueOfAction | |
| 136 | * DOCUMENT ME! | |
| 137 | */ | |
| 138 | 4 | public void setValueOfAction(Action valueOfAction) { |
| 139 | 4 | this.valueOfAction = valueOfAction; |
| 140 | } | |
| 141 | ||
| 142 | // Implementation methods | |
| 143 | // ------------------------------------------------------------------------- | |
| 144 | ||
| 145 | /** | |
| 146 | * A factory method to return a new {@link Mode}instance which should add | |
| 147 | * the necessary default rules | |
| 148 | * | |
| 149 | * @return DOCUMENT ME! | |
| 150 | */ | |
| 151 | 4 | protected Mode createMode() { |
| 152 | 4 | Mode mode = new Mode(); |
| 153 | 4 | addDefaultRules(mode); |
| 154 | ||
| 155 | 4 | return mode; |
| 156 | } | |
| 157 | ||
| 158 | /** | |
| 159 | * Adds the default stylesheet rules to the given {@link Mode}instance | |
| 160 | * | |
| 161 | * @param mode | |
| 162 | * DOCUMENT ME! | |
| 163 | */ | |
| 164 | 4 | protected void addDefaultRules(final Mode mode) { |
| 165 | // add an apply templates rule | |
| 166 | 4 | Action applyTemplates = new Action() { |
| 167 | 6 | public void run(Node node) throws Exception { |
| 168 | 6 | if (node instanceof Element) { |
| 169 | 5 | mode.applyTemplates((Element) node); |
| 170 | 1 | } else if (node instanceof Document) { |
| 171 | 1 | mode.applyTemplates((Document) node); |
| 172 | } | |
| 173 | } | |
| 174 | }; | |
| 175 | ||
| 176 | 4 | Action valueOf = getValueOfAction(); |
| 177 | ||
| 178 | 4 | addDefaultRule(mode, NodeTypePattern.ANY_DOCUMENT, applyTemplates); |
| 179 | 4 | addDefaultRule(mode, NodeTypePattern.ANY_ELEMENT, applyTemplates); |
| 180 | ||
| 181 | 4 | if (valueOf != null) { |
| 182 | 3 | addDefaultRule(mode, NodeTypePattern.ANY_ATTRIBUTE, valueOf); |
| 183 | 3 | addDefaultRule(mode, NodeTypePattern.ANY_TEXT, valueOf); |
| 184 | } | |
| 185 | } | |
| 186 | ||
| 187 | 14 | protected void addDefaultRule(Mode mode, Pattern pattern, Action action) { |
| 188 | 14 | Rule rule = createDefaultRule(pattern, action); |
| 189 | 14 | mode.addRule(rule); |
| 190 | } | |
| 191 | ||
| 192 | 14 | protected Rule createDefaultRule(Pattern pattern, Action action) { |
| 193 | 14 | Rule rule = new Rule(pattern, action); |
| 194 | 14 | rule.setImportPrecedence(-1); |
| 195 | ||
| 196 | 14 | return rule; |
| 197 | } | |
| 198 | } | |
| 199 | ||
| 200 | /* | |
| 201 | * Redistribution and use of this software and associated documentation | |
| 202 | * ("Software"), with or without modification, are permitted provided that the | |
| 203 | * following conditions are met: | |
| 204 | * | |
| 205 | * 1. Redistributions of source code must retain copyright statements and | |
| 206 | * notices. Redistributions must also contain a copy of this document. | |
| 207 | * | |
| 208 | * 2. Redistributions in binary form must reproduce the above copyright notice, | |
| 209 | * this list of conditions and the following disclaimer in the documentation | |
| 210 | * and/or other materials provided with the distribution. | |
| 211 | * | |
| 212 | * 3. The name "DOM4J" must not be used to endorse or promote products derived | |
| 213 | * from this Software without prior written permission of MetaStuff, Ltd. For | |
| 214 | * written permission, please contact<EMAIL_ADDRESS>| |
| 215 | * | |
| 216 | * 4. Products derived from this Software may not be called "DOM4J" nor may | |
| 217 | * "DOM4J" appear in their names without prior written permission of MetaStuff, | |
| 218 | * Ltd. DOM4J is a registered trademark of MetaStuff, Ltd. | |
| 219 | * | |
| 220 | * 5. Due credit should be given to the DOM4J Project - http://www.dom4j.org | |
| 221 | * | |
| 222 | * THIS SOFTWARE IS PROVIDED BY METASTUFF, LTD. AND CONTRIBUTORS ``AS IS'' AND | |
| 223 | * ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE | |
| 224 | * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE | |
| 225 | * ARE DISCLAIMED. IN NO EVENT SHALL METASTUFF, LTD. OR ITS CONTRIBUTORS BE | |
| 226 | * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR | |
| 227 | * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF | |
| 228 | * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS | |
| 229 | * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN | |
| 230 | * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) | |
| 231 | * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE | |
| 232 | * POSSIBILITY OF SUCH DAMAGE. | |
| 233 | * | |
| 234 | * Copyright 2001-2005 (C) MetaStuff, Ltd. All Rights Reserved. | |
| 235 | */ |
|


Exercice 1
Exercice 2
Exercice 3
Exercice 4
Exercice 5
Exercice 6
Exercice 7
Exercice 8
Exercice 9
Exercice 10
Exercice 11
Exercice 12
Exercice 13
Exercice 14


.sim-anim-1
(7 pictures)
.sim-anim-2
(5 pictures)
.sim-anim-3
(9 pictures)
.sim-anim-4
(9 pictures)
.sim-anim-5
(5 pictures)
.sim-anim-6
(9 pictures)
.sim-anim-7
(4 pictures)
.sim-anim-8
(9 pictures)
.sim-anim-9
(9 pictures)
动漫
动漫
动漫


Hot-keys on this page
r m x p toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
|
# Modified version of https://github.com/ikostrikov/pytorch-a2c-ppo-acktr/envs.py
""" Get from Gym, the module where the environment is stored :param env_id: (str) :return: (module, str, str) module_env, class_name, env_module_path """ # Get from the env_id, the entry_point, and distinguish if it is a callable, or a string entry_point = registry.spec(env_id)._entry_point if callable(entry_point): class_name = entry_point.__name__ env_module_path = entry_point.__module__ else: class_name = entry_point.split(':')[1] env_module_path = entry_point.split(':')[0] # Lets try and dynamically load the module_env, in order to fetch the globals. # If it fails, it means that it was unable to load the path from the entry_point # should this occure, it will mean that some parameters will not be correctly saved. try: module_env = importlib.import_module(env_module_path) except ImportError: raise AssertionError("Error: could not import module {}, ".format(env_module_path) + "Halting execution. Are you sure this is a valid environement?")
return module_env, class_name, env_module_path
""" Instantiate gym env :param env_id: (str) :param seed: (int) :param rank: (int) :param log_dir: (str) :param allow_early_resets: (bool) Allow reset before the enviroment is done :param env_kwargs: (dict) The extra arguments for the environment """
# define a place holder function to be returned to the caller. def _thunk(): local_env_kwargs = dict(env_kwargs) # copy this to avoid altering the others local_env_kwargs["env_rank"] = rank env = _make(env_id, env_kwargs=local_env_kwargs) env.seed(seed + rank) if log_dir is not None: env = bench.Monitor(env, os.path.join(log_dir, str(rank)), allow_early_resets=allow_early_resets) return env
return _thunk
""" Recreating the gym make function from gym/envs/registration.py as such as it can support extra arguments for the environment :param id_: (str) The environment ID :param env_kwargs: (dict) The extra arguments for the environment """ if env_kwargs is None: env_kwargs = {}
# getting the spec from the ID we want spec = registry.spec(id_)
# Keeping the checks and safe guards of the old code assert spec._entry_point is not None, 'Attempting to make deprecated env {}. ' \ '(HINT: is there a newer registered version of this env?)'.format(spec.id_)
if callable(spec._entry_point): env = spec._entry_point(**env_kwargs) else: cls = load(spec._entry_point) # create the env, with the original kwargs, and the new ones overriding them if needed env = cls(**{**spec._kwargs, **env_kwargs})
# Make the enviroment aware of which spec it came from. env.unwrapped.spec = spec
# Keeping the old patching system for _reset, _step and timestep limit if hasattr(env, "_reset") and hasattr(env, "_step") and not getattr(env, "_gym_disable_underscore_compat", False): patch_deprecated_methods(env) if (env.spec.timestep_limit is not None) and not spec.tags.get('vnc'): from gym.wrappers.time_limit import TimeLimit env = TimeLimit(env, max_episode_steps=env.spec.max_episode_steps, max_episode_seconds=env.spec.max_episode_seconds) return env |


Portfolio
-
2D N-Body Simulation
Gravity simulation of objects in a plane.
-
Monospace ASACII Art Generator
Generates text from a monospace font to display an image.
-
Multicycle Load/Store Architecture Processor
Design for a load/store architecture processor, assembler, and compiler, which runs the a program that calculates relative primes.
-
Non-Monospace ASACII Art Generator
Generates text from a non-monospace font to display an image.


--- description: Goodbye templateLanguage: liquid templateEngineOverride: html typing: - 68-210 deleteDelay: 0 funMode: true --- {% layout "./layouts/liquid-layout.html" %} {% block content %}


--- layout: default title: Jacob Fosso Tande, Computational Scientist ---
You are welcome to use my deployment script as much as you want Please make sure you give credit as due. Check here for some of the scripts


--- layout: default ---
March 1, 2018
Attendees: OB1 team – 4 Guests – 5 Upcoming releases: Verified moderators program – 45 day auto-release – After 45 days there’s an auto-release to sellers on offline and moderated payments & we’ve made it easier to release these funds. This...


For testing you can have the application call a custom API by including the URL to the endpoint after the #. For example, adding /app.html#https://<IP_ADDRESS>/api will POST the JPEG binary data, in a formdata field called image, to an endpoint on the localhost. Any parameters, including API keys, will have to be passed in the query string. For example /app.html#https://<IP_ADDRESS>/api?apikey=XXXXXXX. For more sophisticated calling conventions, including those requiring custom headers, you will have to modify the behaviour of the post function in app.js.


Providing solutions with infinite possibilities
Focussing on Embedded & Mobile technologies
New product development & deployments
Generate new revenue streams
Platform Customization
Infotainment and Automotive Solutions
Embracing open source
Migrating & maintaining the legacy systems


Brent and Shannon both was born the same year and went to Samuel J. Tilden High School. Both continued thier friendships throught the years and now they will continue in marital bliss. THEY ALSO LIKE HEAVY METAL.
Here you can buy clothing for the wedding.
Please participate in the survey. We would highly appreciate it. It is also mandatory.
Born and rasied in Brooklyn, Shannon has experienced different careers from working in a gym to law enforcement. Now she is currently on her destined path to success in the proffesional culinary arts. She is currently certified as a bartender and mixologist and is currently in school studying to become a world class chef.
Brent's carrer started as a mechanic in the Army. Destined for excitement in his life, he decided to save lives in a different manner by becoming a firefighter. Currently he is retired from both professions.
The wedding will be held on a beach at Gurney's Montauk Resort & Seawater Spa. Shoes are optional but if you decide to wear shoes must be gold colored. Floswer girls dresses will be mostly gold with burgundy. The ring bearers suit is burgundy with a gold colored shirt. Dresses must be champagne or burgundy. Men will wear Kilts with the colors champagne, burgundy, and or black.


culturespaces transforms french WWII submarine base into the largest digital art center
by bryan shimfrance’s leading museum and exhibition operating studio, culturespaces adapted bordeaux’s former WWII submarine base into the largest digital art center in the world — the ‘bassins de lumières’. the immersive exhibition will be open to public starting april 17, 2020 and will operate all year, seven days a week. similar to its prior multimedia exhibition, atelier des lumieres in paris, culturespaces will also devote its main exhibition spaces at bassins de lumières to iconic artists in art history, such as gustav klimt and paul klee.
the submarine base’s surface area is three times the size of that of the carrières de lumières in les baux-de-provence, and five times that of the atelier des lumières in paris. the total projection surface area will be 12,000 square meters, requiring up to 90 video projectors and 80 speakers.
‘following the creation of the atelier des lumières, we are pleased that we have created this enormous digital art centre for the city of bordeaux,’ explains bruno monnier, president of culturespaces. ‘the bassins de lumières will offer visitors unforgettable visual and audio experiences in a unique setting, a place for sharing culture that is open to all types of visitors.’
adapting the architecture of the submarine base to its advantage, the exhibition will consist of four 16-meter deep basins for the main installations along with six other segments. each highlighting different aspects of the project, the six complementary areas include le cube — highlighting artists who specialize in immersive art; le citerne — creating links between museums and original works; les grands nenuphars — projecting exhibition into water; museum space — dedicated to the history of the submarine base; an educational area — information on the featured artists; and mezzanine floor — stage and seating offering different viewing points.
project info:
exhibition: bassins de lumières
company: culturespaces
location: bordeaux, france
opening date: april 17, 2020


Just because I know how to make a website doesn't mean I know what to put inside one.
Content is hard!
Come visit the Orb.


Date: September 16th, 2010
Ask To Myself is
an online/offline platform operated by Badlydone under the internet
domain www.asktomyself.com which you can use to learn new words in
the easier way. Ask To Myself is available over your internet browser
and PC software.
With your registration with Ask To Myself you
agree to these general terms and conditions. The registration is a
valid contract between the user and Badlydone. Please read these T&C
carefully, as they make up the legal framework applying to the use of
Ask To Myself.
Ask To Myself is a cost-free but it will may offer in the future few addictional services no cost-free. Now it's totally free but it has the follows limits in the client software:
Max 10 connections per day
Max 100 questions per day
You can add max 4 categories per day
You can add max 100 words per day
1.1 The T&C apply to all users of Ask To Myself. Any deviating or supplementary regulations shall only apply if drawn up in writing and signed by the user and Ask To Myself. The T&C apply all the services provided by Ask To Myself.
1.2 Badlydone reserves the right to modify these T&C at any time and without indicating the reason for such modification. The modified T&C shall be sent to the user in advance by email. If the user does not object to the modified T&C within a period of four weeks, they are deemed to be accepted. If the user objects, Badlydone may close the user’s account. Badlydone shall separately point out these legal consequences in the email with the modified T&C.
2.1 Each user may only register once. The user is obligated to provide complete and truthful information in the sections provided on the registration form. The user shall be liable without restriction for loss or damage resulting from an incomplete or untruthful registration. Badlydone is entitled to verify the user's details on a random basis.
2.2 The contract between the user and Badlydone regarding the use of Ask To Myself is concluded when the user completes and sends the registration form. The form can only be sent if the option "I agree the terms and conditions" is selected and thus the provisions of the contract are accepted.
2.3 On receipt of the application, Badlydone will send the user an automatic confirmation by email.
3.1 The user is entitled to withdraw in any time. If the user doesn't sign in on the web site for more the one year, the account will be automatically deleted and the T&C will be ceases.
4.1 All the data insert by the user in Ask To Myself's database is private, only the user can see them throw the web site or the PC application accessing by the email and password.
4.2
If in the future, Badlydone will implement a new tool to share the
private data, only the user, by own chose, accessing by email and
password can do it.


Hi,
I would like to know how to update emonhub on my Pi.
I'm using the ready-to-go SD card image (emonSD-13-08-14.img.zip) with emonhub rc1.0.
Do a simple "git pull" is enough ? from what directory ?
Do my configuration file (/boot/emonhub.conf) will be erased ?
Something to change in that configuration file with the new version rc1.x ?
Thank's !
Eric
Re: How to update emonhub ?
Hi Eric
The git pull would need to be done from the emonhub directory which is located in /home/pi so
should do it, however there were some small changes made to the SD card version and I am far from confident about predicting what git is likely to do at the best of times. For this reason I have not documented or automated the update process yet.
It's always wise to create a copy of the settings file "just in case" personally I've had no issues at all when I upgrade using git but I'm not using the SD card image,
Glyn reported he had no issues when he updated, but I can't say with any certainty if that means "it worked first time with no intervention" or "it was easy enough for someone well versed in git".
I'm more than happy to be "on standby" in case something doesn't go as planned but I would hope it would work ok.
I would of liked a "emonhub --update" command or similar but I was out voted when I pushed for this at the planning stage, maybe a future version will have some commands to ease updates and also looking at logfiles and opening the conf for editing, who knows?
Paul
Re: How to update emonhub ?
Hi Paul,
Thank's for your clear reply. Now, I know the possible drawback of a "git pull". What about the advantages ? I mean, what does the last version of emonhub (rc1.2 ?) bring compare to rc 1.0 ?
Eric
Re: How to update emonhub ?
Hi, Eric. I've PM'd you as I couldn't post here.
Re: How to update emonhub ?
Like Paul, I'm not using a read-only file system, but just updated emonhub as per above using git, and all went well with no difficulties encountered.
Paul


This function takes an areal average over an X-Y region. The syntax is:
aave(expr, xdim1, xdim2, ydim1, ydim2)
where:
expr
- any valid GrADS expression xdim1
- starting X dimension expression xdim2
- ending X dimension expression ydim1
- starting Y dimension expression ydim2
- ending Y dimension expression For global averaging, a shorthand may be used:
aave(expr, global)
or aave(expr, g)
aave(expr, lon=0, lon=360, lat=-90, lat=90)
aave
gives the
same result as nested ave
functions in the X and Y dimensions. The expression
will produce the same numerical result as
aave(expr,x=1,x=72,y=1,y=46)
but the aave
function is faster more efficient.
aave
function
does not return the same result as nested ave
functions. To
see this, consider the small grid:
6 18 3 5 10 10 10 10 12 U U Uwhere U represents the missing data value. If we apply nested
ave
functions, the inner
ave
will provide row
averages of 8, 10, and 12. When the outside
ave
is applied,
the result will be an average of 10. When aave
is used, all
the values participate equally (in this case, we are assuming no weights
applied to the final average), and the result is 84/9 or about 9.33.
aave
function assumes that the world
coordinates are longitude in the X dimension and latitude in the Y
dimension, and does weighting in the latitude dimension by the
difference between the sines of the latitude at the northern and
southern edges of the grid box. For areal averaging without latitude weighting,
use the amean
function.
aave
and amean
functions use
appropriate weighting to account for unevenly spaced grids.
aave
function always does its average to the exact
boundaries specified, in world coordinates. This is somewhat different
from the ave
function, where
the -b
flag is used
to get this behavior. If the boundaries specified via the dimension
expressions do not fall on grid boundaries, then the boundary values are
weighted appropriately in the average.
aave(expr,x=1,x=72,y=1,y=46)
Here the boundary would be determined by using the grid values 0.5, 72.5,
0.5, and 46.5 which would be converted to world coordinates. If we assume
that x=1
is 0 degrees longitude and x=72
is 355
degrees longitude, then the averaging boundary would be -2.5 to 357.5
degrees, which would cover the earth. In the Y dimension, when the
boundary is beyond the pole, the aave
function recognizes
this and weights appropriately.
tloop
function
for an example of creating a time
series of area averages.
aave(maskout(p,mask.3(t=1)),x=1,x=72,y=1,y=46)
In this case, it is assumed the mask grid has negative values at ocean points.


0x00 Design Requirement
| Format
The final design of the logo should
be in digital format. Photoshop and Adobe Illustrator are
good tools if you don't have any prefer. You MUST
submit a digital design to participate the
competition. If you don't have the skill, participate as a
team or ask someone else for help. If you want to deisgn
the logo, you need to submit two type of logo. One
rectangular long logo and one square logo. You can see
both of the on team website.
| The
Looking
Any design should stick with SSFS
and BeestBot. Be relevant, but don't force yourself
to use school color if you think it ugly. Also, please use
as less different color as possible because the company
charge for the number of color we used.
|
Record Process
Record all your design process.
Take photos. Be sure to take photos of ALL your doodling
even they look ugly. This will help the team to write
their Engineering Notebook. Record how you get the idea of
your design.
| Award
You get a free t-shirt with your
name on it.
|
Symbolic Meaning
You need to explain why your design
have symbolic meaning (ie. How do you think it
relevant to the team.) The logo and the t-shirt design
should fit together, but you can choose to design
only the t-shirt or the logo, too.
|
Presentation
Have a good presentation on Judge day
is recommended. You can even discuss with other teammates
to see if they like your design before the day. You
can present more than one design.


|
Elektra
0.8.16
|
Methods to do various operations on Key values. More...
Functions | |
| const void * | keyValue (const Key *key) |
Return a pointer to the real internal key value. More... | |
| const char * | keyString (const Key *key) |
| Get the c-string representing the value. More... | |
| ssize_t | keyGetValueSize (const Key *key) |
| Returns the number of bytes needed to store the key value, including the NULL terminator. More... | |
| ssize_t | keyGetString (const Key *key, char *returnedString, size_t maxSize) |
| Get the value of a key as a string. More... | |
| ssize_t | keySetString (Key *key, const char *newStringValue) |
Set the value for key as newStringValue . More... | |
| ssize_t | keyGetBinary (const Key *key, void *returnedBinary, size_t maxSize) |
| Get the value of a key as a binary. More... | |
| ssize_t | keySetBinary (Key *key, const void *newBinary, size_t dataSize) |
| Set the value of a key as a binary. More... | |
Methods to do various operations on Key values.
A key can contain a value in different format. The most likely situation is, that the value is interpreted as text. Use keyGetString() for that. You can save any Unicode Symbols and Elektra will take care that you get the same back, independent of your current environment.
In some situations this idea fails. When you need exactly the same value back without any interpretation of the characters, there is keySetBinary(). If you use that, its very likely that your Configuration is not according to the standard. Also for Numbers, Booleans and Date you should use keyGetString(). To do so, you might use strtod() strtol() and then atol() or atof() to convert back.
To use them:
| ssize_t keyGetBinary | ( | const Key * | key, |
| void * | returnedBinary, | ||
| size_t | maxSize | ||
| ) |
Get the value of a key as a binary.
If the type is not binary -1 will be returned.
When the binary data is empty (this is not the same as ""!) 0 will be returned and the returnedBinary will not be changed.
For string values see keyGetString() and keyIsString().
When the returnedBinary is to small to hold the data (its maximum size is given by maxSize), the returnedBinary will not be changed and -1 is returned.
| key | the object to gather the value from |
| returnedBinary | pre-allocated memory to store a copy of the key value |
| maxSize | number of bytes of pre-allocated memory in returnedBinary |
returnedBinary
| 0 | if the binary is empty |
| -1 | on NULL pointers |
| -1 | if maxSize is 0 |
| -1 | if maxSize is too small for string |
| -1 | if maxSize is larger than SSIZE_MAX |
| -1 | on type mismatch: binary expected, but found string |
| ssize_t keyGetString | ( | const Key * | key, |
| char * | returnedString, | ||
| size_t | maxSize | ||
| ) |
Get the value of a key as a string.
When there is no value inside the string, 1 will be returned and the returnedString will be empty "" to avoid programming errors that old strings are shown to the user.
For binary values see keyGetBinary() and keyIsBinary().
| key | the object to gather the value from |
| returnedString | pre-allocated memory to store a copy of the key value |
| maxSize | number of bytes of allocated memory in returnedString |
returnedString
, including final NULL | 1 | if the string is empty |
| -1 | on any NULL pointers |
| -1 | on type mismatch: string expected, but found binary |
| -1 | maxSize is 0 |
| -1 | if maxSize is too small for string |
| -1 | if maxSize is larger than SSIZE_MAX |
| ssize_t keyGetValueSize | ( | const Key * | key | ) |
Returns the number of bytes needed to store the key value, including the NULL terminator.
It returns the correct size, independent of the Key Type. If it is a binary there might be '\0' values in it.
For an empty string you need one byte to store the ending NULL. For that reason 1 is returned. This is not true for binary data, so there might be returned 0 too.
A binary key has no '\0' termination. String types have it, so to there length will be added 1 to have enough space to store it.
This method can be used with elektraMalloc() before keyGetString() or keyGetBinary() is called.
| key | the key object to work with |
| 1 | when there is no data and type is not binary |
| 0 | when there is no data and type is binary |
| -1 | on null pointer |
| ssize_t keySetBinary | ( | Key * | key, |
| const void * | newBinary, | ||
| size_t | dataSize | ||
| ) |
Set the value of a key as a binary.
A private copy of newBinary
will allocated and saved inside key
, so the parameter can be deallocated after the call.
Binary values might be encoded in another way then string values depending on the plugin. Typically character encodings should not take place on binary data. Consider using a string key instead.
When newBinary is a NULL pointer the binary will be freed and 0 will be returned.
| key | the object on which to set the value |
| newBinary | is a pointer to any binary data or NULL to free the previous set data |
| dataSize | number of bytes to copy from newBinary |
| 0 | when the internal binary was freed and is now a null pointer |
| -1 | if key is a NULL pointer |
| -1 | when dataSize is 0 (but newBinary not NULL) or larger than SSIZE_MAX |
| ssize_t keySetString | ( | Key * | key, |
| const char * | newStringValue | ||
| ) |
Set the value for key
as newStringValue
.
The function will allocate and save a private copy of newStringValue
, so the parameter can be freed after the call.
String values will be saved in backend storage, when kdbSetKey() will be called, in UTF-8 universal encoding, regardless of the program's current encoding, when iconv plugin is present.
| key | the key to set the string value |
| newStringValue | NULL-terminated text string to be set as key's value |
| 1 | if newStringValue is a NULL pointer, this will make the string empty (string only containing null termination) |
| -1 | if key is a NULL pointer |
| const char* keyString | ( | const Key * | key | ) |
Get the c-string representing the value.
Will return (null) on null pointers. Will return (binary) on binary data not ended with a null byte.
It is not checked if it is actually a string, only if it terminates for security reasons.
| (null) | on null keys |
| "" | if no data found |
| (binary) | on binary keys |
| key | the key object to get the string from |
| const void* keyValue | ( | const Key * | key | ) |
Return a pointer to the real internal key
value.
This is a much more efficient version of keyGetString() keyGetBinary(), and you should use it if you are responsible enough to not mess up things. You are not allowed to modify anything in the returned string. If you need a copy of the Value, consider to use keyGetString() or keyGetBinary() instead.
If key
is string (keyIsString()), you may cast the returned as a "char *"
because you'll get a NULL terminated regular string.
keyValue() returns "" in string mode when there is no value. The reason is
If the data is binary, the size of the value must be determined by keyGetValueSize(), any strlen() operations are not suitable to determine the size.
keyValue() returns 0 in binary mode when there is no value. The reason is
| key | the key object to work with |
| "" | when there is no data and key is not binary |
| 0 | where there is no data and key is binary |
| 0 | on NULL pointer |


Hi {{ FullName . }},
A request was made to recover your account. Unfortunately, this account has been disabled. Please first contact an administrator to restore functionality.
If you did not make this request, you can ignore this email.
Thanks,
The ASHIRT Team


FEATURED STORIES
The 73 People Pardoned by Donald Trump on His Lats Day As President
Notable people pardoned by Trump today include former White House chief strategist Steve Bannon and rapper Lil Wayne.
Donald Trump Can Issue Secret Pardons, but They're Risky
Trump chose not to issue pardons for himself of his personal attorney, former New York Mayor Rudy Giuliani.
Newsweek Announces Launch of the Newsweek Expert Forum
Newsweek Expert Forum creates cohort of thought leaders, pioneering thinkers to share insights, expertise from range of professions, industries.
Can Joe Biden Rescind Donald Trump's Pardons?
The president did not issue pardons for himself, his children or for former New York Mayor Rudy Giuliani.
Joe Biden Immigration Orders Upend Donald Trump's Legacy
The Biden Administration will issue an order stopping construction of the border wall on its first day.
Hokkaido: Japan’s Northern Frontier
Featuring a unique history, uninhibited wildness and at times otherworldly natural spectacles, Hokkaido offers an experience that is completely different from the rest of Japan.
What Makes International Schools so Special?
 See why so many parents are turning to International Schools.
America's Most Responsible Companies 2021
We partnered with Statista to rank the country's Most Responsible Companies. They share the values that we all prioritize, and that's worth recognizing in 2021
This Startup Is Making Long Distance Real Estate Investing Possible for Millennials
75% of Roofstock users are first-time real estate investors, and more than half are under 35.
TOP STORY
Joe Biden's First Executive Actions Undo Many of Donald Trump's
Actions related to the COVID-19 crisis, climate change and racial equality are to be among the first taken as he begins his tenure.
THE DEBATE
OPINION
Is the European Union Combating or Whitewashing Anti-Semitism?
BY ABRAHAM COOPER AND MANFRED GERSTENFELDHungary: From Manufacturing Hub to Innovation and R&D; Center
With its pro-business agenda and open-border policy for investors, the Central European nation is hoping to bounce back from the COVID-19 crisis, while making greater strides toward innovation, research and development.
Panama - Central America's Leading Investment Destination
Panama is expected to rebound strongly in 2021, with growth forecasted between 4% and 7%.
Why are Private Schools the Best Option for Your Child’s Future?
If academic results, average class size, teacher proficiency, pupil diversity and religious orthodoxy of your child’s school are things that are important to you, you should be looking at a private school.
America's Best Nursing Homes 2021
In partnership with Statista, we established a ranking providing information and insights to help you during this challenging time.
IN THE MAGAZINE
EDITOR'S PICK
Israel Security Officials Reveal Top Threats Faced on Four Fronts This Year
Newsweek spoke exclusively with four Israeli security officials charged with frontiers facing Syria, Lebanon, Gaza and the southern borders with Egypt and Jordan, revealing their primary security concerns for 2021.
Donald Trump's Pardon List Doesn't Include Himself or His Children
Trump released a list of pardons on Tuesday, as last-minute presidential pardons have been a long-standing custom for outgoing presidents.
More Stimulus Checks Largely Wanted for Bills, Debts and Essentials
The coronavirus crisis continues to cause economic hardship across the nation, with continued calls for further direct payments to assist those struggling.
Gavin Newsom, Facing Recall, Accelerates Sluggish Vaccine Rollout
Slightly more than 1.3 million people in California had received a COVID-19 vaccine by Saturday.
Paris Climate Agreement Explained As Joe Biden Plans to Rejoin
The incoming Biden administrate plans to the the U.S. up to the according on inauguration day.
Trump's Facebook Gets Almost 200,000 Likes Despite Being Blocked after Riot
The president—who will exit the White House on Wednesday following the inauguration of Joe Biden—was effectively exiled from mainstream social media platforms this month after the violence.
Faith in Governors' COVID-19 Plans Drops to New Low During Vaccine Rollout
Polling has shown growing dissatisfaction in how action has been communicated, as cases continue to rise and the number of deaths linked to COVID-19 in the U.S. nears 400,000.
What Joe Biden Has Promised To Do on His First Day as President
The president-elect plans to reverse the so-called "Muslim ban" and the exit from the Paris Climate Accords.
SUBSCRIBE
In-depth Stories You Don't See Elsewhere About Topics You Want to Read!
All Access + Weekly Delivery. Choose A Membership That's Perfect for You!


The schedule for the 2019 Abu Dhabi Grand Prix weekend
by Nicolás Quarles van UffordThe championships might have been decided coming into the final Grand Prix of the 2019 F1 season, but there is still plenty to play for at the Abu Dhabi Grand Prix - here is the schedule for the full weekend at the Yas Marina Circuit!
Mileage on the new rubber
Abu Dhabi is a springboard into next season, as it hosts the final test of the calendar year. The results from the final race weekend are important too in terms of data as it's the final time teams can measure themselves against the rest of the grid. After that, everything is behind closed doors until the unveils ahead of the 2020 winter tests.
2020 will have more or less the same regulations, meaning testing with new parts will be more valuable than usual.
Championship implications
The top four places in the constructors' championship are already set - McLaren secured fourth place last weekend in Brazil after Carlos Sainz' incredible podium finish.
Behind the Woking-based team, Renault will now have to check their rear-view mirrors as they're just eight points ahead of Toro Rosso now. The development team have been in red-hot form recently and should be competitive again in Abu Dhabi. With the difference between fifth and sixth place being several millions in prize money, both sides should be throwing everything they have at this weekend's race.
Abu Dhabi Grand Prix Schedule
| Starting time | Ending time | |
| Friday | ||
| Free Practice 1 | 10: 00 AM (UK) 5: 00 AM (EST) |
11: 30 AM (UK) 6: 30 AM (EST) |
| Free Practice 2 | 1: 00 PM (UK) 8: 00 AM (EST) |
2: 30 PM (UK) 9: 30 AM (EST) |
| Saturday | ||
| Free Practice 3 | 10: 00 AM (UK) 05: 00 AM (EST) |
11: 00 AM (UK) 06: 00 AM (EST) |
| Qualifying | 1: 00 PM (UK) 8: 00 AM (EST) |
2: 00 PM (UK) 9: 00 AM (EST) |
| Sunday | ||
| Grand Prix | 1: 10 PM (UK) 8: 10 AM (EST) |
3: 10 PM (UK) 10: 10 AM (EST) |


A monad, providing information about validation result. It wraps either a valid TARGET or a set of ERROR.
Defines validation behaviour for TARGET value and given OPTIONS, raising ERRORs if rules are violated.