AUFS ストレージドライバーの利用
読む時間の目安: 8 分AUFS とは ユニオンファイルシステム（union filesystem）の一種です。
The aufs
storage driver was previously the default
storage driver used for managing images and layers on Docker for Ubuntu, and for
Debian versions prior to Stretch. If your Linux kernel is version 4.0 or higher,
and you use Docker Engine - Community, consider using the newer
overlay2, which has
potential performance advantages over the aufs
storage driver.
メモ
ディストリビューションや Docker エディションの中には AUFS をサポートしていないものがあります。 サポートされているプラットフォームなどの詳しい情報は 前提条件 を参照してください。
前提条件
- For Docker Engine - Community, AUFS is supported on Ubuntu, and on Debian versions prior to Stretch.
- For Docker EE, AUFS is supported on Ubuntu.
- If you use Ubuntu, you need to add the AUFS module to the kernel. If you do
not install these packages, you need to use
overlay2
. - AUFS cannot use the following backing filesystems:
aufs
,btrfs
, orecryptfs
. This means that the filesystem which contains/var/lib/docker/aufs
cannot be one of these filesystem types.
aufs
ストレージドライバーの設定
Docker の起動時に AUFS ドライバーがカーネルにロードされていて、その他のス。トレージドライバーの設定がされていなければ、Docker はデフォルトでこの AUFS ドライバーを利用します。
-
以下のコマンドを実行して、利用するカーネルが AUFS に対応していることを確認します。
$ grep aufs /proc/filesystems nodev aufs
-
Docker がどのストレージドライバーを利用しているかを確認します。
$ docker info <truncated output> Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 0 Dirperm1 Supported: true <truncated output>
-
If you are using a different storage driver, either AUFS is not included in the kernel (in which case a different default driver is used) or that Docker has been explicitly configured to use a different driver. Check
/etc/docker/daemon.json
or the output ofps auxw | grep dockerd
to see if Docker has been started with the--storage-driver
flag.
aufs
ストレージドライバーはどのように動作するか
AUFS is a union filesystem, which means that it layers multiple directories on a single Linux host and presents them as a single directory. These directories are called branches in AUFS terminology, and layers in Docker terminology.
The unification process is referred to as a union mount.
The diagram below shows a Docker container based on the ubuntu:latest
image.
Each image layer, and the container layer, are represented on the Docker host as
subdirectories within /var/lib/docker/
. The union mount provides the unified
view of all layers. The directory names do not directly correspond to the IDs
of the layers themselves.
AUFS uses the Copy-on-Write (CoW) strategy to maximize storage efficiency and minimize overhead.
 These files and directories are managed by Docker.
All of the information about the image and container layers is stored in
subdirectories of /var/lib/docker/aufs/
.
diff/
: the contents of each layer, each stored in a separate subdirectorylayers/
: metadata about how image layers are stacked. This directory contains one file for each image or container layer on the Docker host. Each file contains the IDs of all the layers below it in the stack (its parents).mnt/
: Mount points, one per image or container layer, which are used to assemble and mount the unified filesystem for a container. For images, which are read-only, these directories are always empty.
The container layer
If a container is running, the contents of /var/lib/docker/aufs/
change in the
following ways:
diff/
: Differences introduced in the writable container layer, such as new or modified files.layers/
: Metadata about the writable container layer’s parent layers.mnt/
: A mount point for each running container’s unified filesystem, exactly as it appears from within the container.
How container reads and writes work with aufs
Reading files
Consider three scenarios where a container opens a file for read access with aufs.
-
The file does not exist in the container layer: If a container opens a file for read access and the file does not already exist in the container layer, the storage driver searches for the file in the image layers, starting with the layer just below the container layer. It is read from the layer where it is found.
-
The file only exists in the container layer: If a container opens a file for read access and the file exists in the container layer, it is read from there.
-
The file exists in both the container layer and the image layer: If a container opens a file for read access and the file exists in the container layer and one or more image layers, the file is read from the container layer. Files in the container layer obscure files with the same name in the image layers.
Modifying files or directories
Consider some scenarios where files in a container are modified.
-
Writing to a file for the first time: The first time a container writes to an existing file, that file does not exist in the container (
upperdir
). Theaufs
driver performs a copy_up operation to copy the file from the image layer where it exists to the writable container layer. The container then writes the changes to the new copy of the file in the container layer.However, AUFS works at the file level rather than the block level. This means that all copy_up operations copy the entire file, even if the file is very large and only a small part of it is being modified. This can have a noticeable impact on container write performance. AUFS can suffer noticeable latencies when searching for files in images with many layers. However, it is worth noting that the copy_up operation only occurs the first time a given file is written to. Subsequent writes to the same file operate against the copy of the file already copied up to the container.
-
Deleting files and directories:
-
When a file is deleted within a container, a whiteout file is created in the container layer. The version of the file in the image layer is not deleted (because the image layers are read-only). However, the whiteout file prevents it from being available to the container.
-
When a directory is deleted within a container, an opaque file is created in the container layer. This works in the same way as a whiteout file and effectively prevents the directory from being accessed, even though it still exists in the image layer.
-
-
Renaming directories: Calling
rename(2)
for a directory is not fully supported on AUFS. It returnsEXDEV
(“cross-device link not permitted”), even when both of the source and the destination path are on a same AUFS layer, unless the directory has no children. Your application needs to be designed to handleEXDEV
and fall back to a “copy and unlink” strategy.
AUFS and Docker performance
To summarize some of the performance related aspects already mentioned:
-
The AUFS storage driver is less performant than the
overlay2
driver, but is a good choice for PaaS and other similar use-cases where container density is important. This is because AUFS efficiently shares images between multiple running containers, enabling fast container start times and minimal use of disk space. -
The underlying mechanics of how AUFS shares files between image layers and containers uses the page cache very efficiently.
-
The AUFS storage driver can introduce significant latencies into container write performance. This is because the first time a container writes to any file, the file needs to be located and copied into the containers top writable layer. These latencies increase and are compounded when these files exist below many image layers and the files themselves are large.
Performance best practices
The following generic performance best practices also apply to AUFS.
-
Solid State Devices (SSD) provide faster reads and writes than spinning disks.
-
Use volumes for write-heavy workloads: Volumes provide the best and most predictable performance for write-heavy workloads. This is because they bypass the storage driver and do not incur any of the potential overheads introduced by thin provisioning and copy-on-write. Volumes have other benefits, such as allowing you to share data among containers and persisting even when no running container is using them.


Contact Us!
Direct any questions or concerns to<EMAIL_ADDRESS>Angela Chen
UI/UX Instructor
UI/UX Instructor
Chris Lam
Program Lead/Web Development Instructor
Program Lead/Web Development Instructor
Justin Fong
Data Science Instructor
Data Science Instructor
Anne Le
Marketing Lead
Marketing Lead
Brian Liu
Administrative Assistant
Administrative Assistant
Renz Lane
Technology Chair
Technology Chair
Vincent Hoang
Marketing - Communications
Marketing - Communications
Minh Ton
Marketing - Graphics
Marketing - Graphics
Jatnael Montes
Marketing - Research
Marketing - Research
Jonathan Chu
Curriculum Development
Curriculum Development
Albert Dang
Curriculum Development
Curriculum Development
Zeven Vidmar-Barker
Curriculum Development/Teaching Assistant
Curriculum Development/Teaching Assistant


Highly Secured
All information are stored at the world's most trusted datacentres.
Web Based
There is no need to install any kind of software on client machines.
User Friendly
Just about anyone working knowledge of computers will able to handle Brume CMS.
Core Modules


Niraj Pant
Hey! I'm Niraj, and I'm a freshman at the University of Illinois at Urbana-Champaign. I love building and learning about new technologies.
Email - Resume - Twitter - LinkedIn - GitHub
Projects
I love to work on side projects in my free time. Here are some things I have worked on:
- LetsGit is an accessible, cloud-based Git GUI client and code editor in the browser. Lets you easily code and collaborate from anywhere without needing to setup a new development environment.
- Twitch Plays Tinder a web app that streams a port of Tinder onto twitch.tv and allows collaborative matchmaking on the platform. Originally inspired by Twitch Plays Pokémon.
- ScandIn is a Google Glass application to enhance networking events using facial recognition. The Glass application takes a picture, sends it to an Express.js backend, matches the image to a user in the training set, and returns the recognized user's information to a Glass card.
- Limit is a 20x20 multiplayer Tic-Tac-Toe game using Firebase's real-time data platform.
Work
In the past I've previously worked at Besso Labs, ShipBob, Lookout, and am currently a undergraduate researcher at the Stanford HCI Group.
Get In Touch
Let's chat! I can be contacted at hi (at) niraj (dot) io.


Survey Data
/r/Anime Seasonal Surveys organized by /u/jiecut
2016 Q1 (Mid-Winter)
2016 Q2 (Pre-Spring)
2016 Q2 (Mid-Spring)
2016 Q3 (Mid-Summer)
Check out the /r/Anime
Survey Index!
Home
|
About
|
Survey Data
|
Graphs


My name is Dionne Smith and I am 17 years old, I was born and raised in Indianpolis, IN. I started coding my Junior year of high school and Scott Dooley was my teacher that inspired me to learn how to code by using CodeAcademy. Learning to code was the best decision I've ever made other than joining Nextech. My first webpage was about Pandas and it was our first big assignment in class and learning HTML and CSS had helped me a lot when it came to finishing the project.
My favorite hobby is painting, and my first ever painting was a dragon and it turned out very nicely, and the cardinal in the picture is the painting that I painted my mother for her birthday in 2015. Her favorite bird is a cardinal and so I decided that I would do this for her. I love painting and I will alway continue to do it as my favorite hobby.
At my school, Christel House Academy, I was the president of Rotary. We needed grant money to fund a project we were doing with people at the Day Spring Shelter, and we received a total of $877.50 from United Way. We were so proud of ourselves and funded free and new books to everyone at the shelter each time we went and volunteered to serve in their kitchen.


--- date: "2019-07-30T13:17:51+02:00" title: Orientation to RStudio weight: 5 ---
First, open RStudio. As their icons can look very similar, be sure you are opening RStudio and not R.
For RStudio to function you must also have R installed on the computer (see the FAQ page for installation instructions). RStudio is an interface (GUI) for easier use of R. You can think of R as being the engine of a vehicle, doing the crucial work, and RStudio as the body of the vehicle (with seats, accessories, etc.) that helps you actually use the engine to move forward!
By default RStudio displays four rectangle panes.
{{% notice tip %}} If your RStudio displays only one left pane it is because you have no scripts open yet. {{% /notice %}}
The R Console, by default the left or lower-left pane in R Studio, is the home of the R “engine”. This is where the commands are actually run and non-graphic outputs and error/warning messages appear. You can directly enter and run commands in the R Console, but realize that these commands are not saved as they are when running commands from a script.
If you are familiar with Stata, the R Console is like the Command Window and also the Results Window.
This pane, by default in the upper-left, is space to edit and run your scripts, including the RMarkdown outbreak and survey templates. This pane can also display datasets (data frames) for viewing.
For Stata users, this pane is similar to your Do-file and Data Editor windows.
This pane, by default the upper-right, is most often used to see brief summaries of objects in the R Environment in the current session. These objects could include imported, modified, or created datasets, parameters you have defined (e.g. a specific epi week for the analysis), or vectors or lists you have defined during analysis (e.g. names of regions). Click on the arrow next to a dataframe name to see its variables.
In Stata, this is most similar to Variables Manager window.
The lower-right pane includes several tabs including plots (display of graphics including maps), help, a file library, and available R packages (including installation/update options).
This pane contains the Stata equivalents of the Plots Manager and Project Manager windows.
Change RStudio settings and appearance in the Tools drop-down menu, by selecting Global Options


#include <stGenericPriorityQueue.h>
Public Types | |
| typedef ObjectType | tObject |
| typedef stGenericEntry< ObjectType > | tGenericEntry |
Public Member Functions | |
| stGenericPriorityQueue () | |
| ~stGenericPriorityQueue () | |
| void | Add (tObject *object, stPageID pageID, stDistance distanceRepQuery, stDistance distanceRep, stDistance radius, int type) |
| void | Add (tObject *object, stPageID pageID, stDistance distanceQuery, stDistance radius, int type) |
| void | Add (tObject *object, stDistance distanceRep, stDistance distanceRepQuery, int type) |
| void | Add (tObject *object, stDistance distanceQuery, int type) |
| tGenericEntry * | Get () |
| stCount | GetSize () |
| bool | IsEmpty () |
This class implements a global priority queue based on chained list used for the Incremental Nearest Neighbor.
This priority queue is implemented using a simple chained list. This implementation of priority queue is based on the paper "Incremental Similarity Search in Multimedia Databases" - G. R. Hjaltason & Hanan Samet - 2000.
Definition at line 342 of file stGenericPriorityQueue.h.
| typedef stGenericEntry< ObjectType > stGenericPriorityQueue< ObjectType >::tGenericEntry |
This type defines the stGenericEntry used by this class.
Definition at line 353 of file stGenericPriorityQueue.h.
| typedef ObjectType stGenericPriorityQueue< ObjectType >::tObject |
Type of the object.
Definition at line 348 of file stGenericPriorityQueue.h.
|
inline |
Creates a new priority queue.
Definition at line 358 of file stGenericPriorityQueue.h.
| stGenericPriorityQueue< ObjectType >::~stGenericPriorityQueue | ( | ) |
Disposes this queue.
| void stGenericPriorityQueue< ObjectType >::Add | ( | tObject * | object, |
| stPageID | pageID, | ||
| stDistance | distanceRepQuery, | ||
| stDistance | distanceRep, | ||
| stDistance | radius, | ||
| int | type | ||
| ) |
Adds a new approximate node to the queue.
| object | The object to be inserted. |
| pageID | The pageID to be inserted. |
| distanceRepQuery | The distance from the representative and the query object. |
| distanceRep | The distance from the object and the representative. |
| radius | The radius value of the subtree to be inserted. |
| type | The type of this entry. |
| void stGenericPriorityQueue< ObjectType >::Add | ( | tObject * | object, |
| stPageID | pageID, | ||
| stDistance | distanceQuery, | ||
| stDistance | radius, | ||
| int | type | ||
| ) |
Adds a new node to the queue.
| object | The object to be inserted. |
| pageID | The pageID to be inserted. |
| distanceQuery | The distance from this entry and the query object. |
| radius | The radius value of the subtree to be inserted. |
| type | The type of this entry. |
| void stGenericPriorityQueue< ObjectType >::Add | ( | tObject * | object, |
| stDistance | distanceRep, | ||
| stDistance | distanceRepQuery, | ||
| int | type | ||
| ) |
Adds a new approximate object to the queue.
| object | The object to be inserted. |
| distanceRep | The distance from the object and the representative. |
| distanceRepQuery | The distance from the representative and the query object. |
| type | The type of this entry. |
| void stGenericPriorityQueue< ObjectType >::Add | ( | tObject * | object, |
| stDistance | distanceQuery, | ||
| int | type | ||
| ) |
Adds a new object to the queue.
| object | The object to be inserted. |
| distanceQuery | The distance from this entry and the query object. |
| type | The type of this entry. |
| tGenericEntry* stGenericPriorityQueue< ObjectType >::Get | ( | ) |
Gets the next node. This node is removed from the queue.
|
inline |
Returns the size of this queue.
Definition at line 423 of file stGenericPriorityQueue.h.
|
inline |
Returns if this queue is empty.
Definition at line 430 of file stGenericPriorityQueue.h.


GEAR
We ranked the Nintendo Switch 11.0.0 features from worst to best
by Callum Booth - 36 minutes ago
INSIGHTS
3 cliché pieces of startup advice I never want to hear again
by Jason VandeBoom - 1 hour ago
TECH
Boston's Mayor has big plans for EV in the city
by Cities Today - 2 hours ago
Latest news >
3 cliché pieces of startup advice I never want to hear again
Jason VandeBoom - 3 hours ago
Boston's Mayor has big plans for EVs in the city
Cities Today - 4 hours ago
WhatsApp now lets you assign separate wallpapers for your favorite people
Napiez Lopez - 5 hours ago
After Apple, Samsung is reportedly working on an AirTag rival to help find your stuff
Ivan Mehta - 7 hours ago
Dear pandemic, please go away so I can visit Super Nintendo World
Rachel Kaser - 15 hours ago
Review: Coway's sleek Airmega air purifier killed bad odors in my home
Napier Lopez - 16 hours ago
Study: Security flaw could allow hackers to trick lab scientists into making viruses
Tristen Greene - 17 hours ago
AI reveals vast racial inequities inNYC homeownership
Thomas Macaulay - 19 hours ago
Latest funding rounds on Index.co >
See more |
Get updates by email |
Add info
LambdaTest
$6M
SERIES A
Cloud Computing, +3
Entree Capital, +5
Sequoia's Surge leads $6 Mn round in LambdaTest on Entrackr
BuildBuddy
$3.2M
UNDISCLOSED
Information Technology
YC-backed BuildBuddy raises $3.15M to help developers build software more quickly on TechCrunch
Infogrid
$15.5M
SERIES A
Information Technology, +2
Northzone, +4
Infogrid raises $15.5M from Northzone to retrofit buildings with 'smart' IoT on TechCrunch
Tazapay
$3.2M
VENTURE CAPITAL
Financial Services
Surge
Tazapay snags US$3.2M to expand cross-border SMB commerce platform in Southeast Asia | e27 on e27: Asia tech news
Hard Fork >
Most popular
Square stock on fire: Jack Dorsey's beard sets 3 share price records in 3 days
David Canellis - 3 days ago
LATEST
Elon Musk eclipses Bill Gates to become the world's second richest person
- Tesla makes its S&P 500 debut in exactly 4.20 weeks
- Big Pharma's COVID-19 vaccines injected $1 trillion into the S&P 500
- Elon who? Tesla's Chinese rival NIO is up 1,200% this year
- Tencent bounces hard after President Xi spooks market into $200B tech selloff
- Warren Buffett might've just sold $5B worth of Apple stock
Apps >
Most popular
Snapchat is ripping off TikTok — the cycle is now complete
Rachel Kaser - 5 days ago
LATEST
This handy site tells you if specific apps work on Apple Silicon yet
- Twitter now warns you before liking a tweet labeled for misinformation
- Microsoft Edge adds autofill coupon codes and more annotation tools
- Google Chrome gets a new icon in Big Sur — and it wants your help choosing the next
- Google Pay gets a complete makeover to try to handle all your finances
- Strava finally reinstates support for Bluetooth heart rate monitors — after dropping it last year
Gear >
Most popular
How do you build a pet-friendly gadget? We asked experts and animal owners
Andrea Hak - 2 days ago
LATEST
Stop your search! Here are cool gifts for every budget
- The Spurcycle bike bell is so nice, it might just be worth paying $49 for
- How do you design an underwater scooter? Easy, start with a drone
- Battle of the affordables: Pixel 4a vs OnePlus Nord vs Poco X3
- The Mini Urbanaut concept is a TARDIS on wheels for day-trippers of the future
- Blessed be the relaxation powers of a CBD bath bomb
Tech >
Most popular
Facebook said It would ban holocaust deniers. Instead, its algorithm provided a network for them
The Markup - 21 hours ago
LATEST
Maserati commits to going all-electric by 2025, COO says
- Facebook said It would ban holocaust deniers. Instead, its algorithm provided a network for them
- Volvo thinks gaming tech can help make self-driving cars safer
- Understand adversarial attacks by doing one yourself with this tool
- How CAPTCHAs could show if an algorithm’s getting closer to AGI
- Researchers taught AI how to judge a video game by its cover
Creative >
Most popular
Graphic design apps are now as fundamental as Excel — here's how non-designers get up to speed
Alexis Ong - 9 weeks ago
LATEST
This real-world multiplayer game lets you spend other people’s money — really
- Warzone is infinitely more fun without vehicles (even though it's just a bug)
- The Winamp Skin Museum is a beautiful homage to an iconic piece of software
- Why skin tone modifiers don't work for 🤝, explained by an emoji historian
- Why you need Design Thinking and Proofs of Concept to level up your business
- How an AI graphic designer convinced clients it was human
Podium >
Most popular
Google still abuses its search monopoly in the EU — regulators wake up!
Christian Kroll - 4 weeks ago
LATEST
Mixed reality is perfect for our innate multitasking habits
- 6 big data blunders businesses should avoid
- The AI landscape is shifting from 'data' to 'knowledge.' Here's why that matters
- Disruptive fintech is our best bet to economic recovery post COVID-19
- Tech in Latin America: An entrepreneur’s guide to June’s startup news
- 4 ways to improve your latest product’s landing page to guarantee conversion
Insights >
Most popular
The long-term impacts COVID-19 will have on startups seeking capital
Built In - 6 hours ago
LATEST
Periodic table: Scientists propose new way of ordering the elements
- What the hell is the Pomodoro Technique and why is it good for productivity?
- Square stock on fire: Jack Dorsey's beard sets 3 share price records in 3 days
- We're already living in an era of unlimited technology
- It's not just you — work psychologist confirms COVID-19 made jobs suckier
- How COVID and an unexpected 17-hour time difference forced us to master asynchronous work
Launch >
Most popular
I’ve fallen for the electric Hummer and I hate myself for it
Matthew Beedham - 5 weeks ago
LATEST
Amazon's electric delivery vans made by Rivian are here, and they're pretty cute
- This adorable little electric car is made of trash
- Hyundai's first fuel cell-powered trucks arrive in Switzerland — bye bye, diesel
- Lynk & Co.'s car sharing-owning scheme will never cost more than €500 a month
- Lucid unveils the Air, its luxury EV with up to 517-miles of range costing as much as $169K
- Say hello to SHIFT, our new publication about the future of mobility tech
Distract >
Most popular
Stop embarrassing yourself in Zoom calls — use this app to check your face
Georgina Ustik - 3 days ago
LATEST
Suck at gifting?


--- layout: page title: About Me description: background: '/img/cover-1.jpg' ---
As a data practitioner and aficionado, I am dedicated to utilizing data to drive business success. My years of experience in machine learning and analytics have honed my skills in transforming complex datasets into actionable insights that can inform strategic decision-making.
Throughout my career, I have had the opportunity to wear many hats, from plumbing data pipelines to configuring cloud infrastructure to support data architecture. While my primary focus has been on creating ML-powered solutions, I am equally comfortable building the data models that support advanced analytics.
Borderline millenial, fond of all kinds of teas, covers a song with guitar every once in a blue moon, loves powerlifting and rucking for fitness, enjoys a good book, and convinced of the efficacy of ketogenic diet with intermittent fasting.


5Hz 2.5Hz: Sets the Live Sweep
repeat rate at 200ms for AC waveform sweeps and 400ms for Slow Sweeps.
Use this if you want a relatively uniform sample rate and your computer
is slow.
10Hz 5Hz:Sets the Live
Sweep repeat rate at 100ms for AC waveform sweeps and 200ms for Slow
Sweeps. Use this if you want a relatively uniform sample
rate and your computer is fast.
Max: Sets
the Live Sweep repeat rate as fast as the computer can process the
sweeps. At maximum, the sampling period is less uniform than for the
fixed rates.
For best throughput performance, set up the USB interface through Device Manager. Use the following Advanced Settings:
Note: COM Port Number may differ. Leave as set.


Emperor is a python package hosted in GitHub that relies on NumPy, scikit-bio and qcli.
Download Stable Version Download Development Version
To clone the git repository directly into your machine:
git clone git://github.com/biocore/emperor.git
To perform a global installation of Emperor, execute the following command from a terminal session:
python setup.py install
If you do not want to do a global installation, you will have to add the Emperor scripts and libraries to the PATH
and PYTHONPATH
environment variables. To add these variables to your .bash_profile
issue the following terminal commands:
echo "export PATH=$HOME/emperor_bin/:$PATH" >> ~/.bash_profile
echo "export PYTHONPATH=$HOME/emperor_lib/:$PYTHONPATH" >> ~/.bash_profile
python setup.py install --install-scripts=~/emperor_bin/ --install-purelib=~/emperor_lib/ --install-lib=~/emperor_lib/
Alternatively Emperor can be installed from the Python Package Index (PyPi) using either conda (note that this method will be notably faster):
conda install scikit-bio
pip install emperor
or using pip:
pip install numpy
pip install emperor
To test for a correct installation, open a new terminal window and issue the following command to see the help of make_emperor.py
:
make_emperor.py -h
Usage: make_emperor.py [options] {-i/--input_coords INPUT_COORDS -m/--map_fp MAP_FP}
[] indicates optional input (order unimportant)
{} indicates required input (order unimportant)
This script automates the creation of three-dimensional PCoA plots to be visualized with Emperor using Google Chrome.
Example usage:
Print help message and exit
make_emperor.py -h
...


Replicating
Archive
Tags
Categories
Pages
About me and this blog
Categories
R
4
finance
3
tutorial
2
R (4)
Replicating Goyal/Welch (2008)
How to produce nice tables in PDFs using knitr/Sweave and R
Vector autoregression (VAR) in R
Replicating Cochrane (2008)
finance (3)
Replicating Goyal/Welch (2008)
Vector autoregression (VAR) in R
Replicating Cochrane (2008)
tutorial (2)
How to produce nice tables in PDFs using knitr/Sweave and R
How to set up a new blog with ruhoh on github


New York City Green Festival launches its third year April 26-27 at
a new venue, Pier 94! At Green Festival®, there is something for everyone interested in living a
more sustainable and healthier life. Experience the widest selection of products and services to
work green, play green and live green – from food, fashion and health, to energy, construction and
design. Enjoy vegan and vegetarian cooking demos, educational activities for kids and families,
panels featuring inspirational speakers, and live music and entertainment. Shop in our unique
marketplace of more than 300 eco-friendly businesses - everything from all-natural body care
products and organic clothing to Fair Trade gifts, beautiful home renovations made from renewable
resources, plus vegan and vegetarian offerings based on organic, non-GMO or local, artisanal foods.


Concepter and manufacturer since 1979, AMN doesn't need to build its reputation anymore.
History has proved its talent regarding innovation. Yesterday's most advanced technologies are today's references.
Our exceptional resources for scientific investigations fit into this dynamic.
This is how we keep progress at the heart of our actions.
Our quality priority did not emerge overnight. It has been enrolled in the foundation of our philosophy.
Our particularity is based on the importance given to the connection between our commitments and the real performance of our high quality products.
We are worldwide specialist on design and manufactured tools for pelletizing polymer.
Die Plates, Knives, Knife-Holders, Die adaptors.
And the best technical assistance & training.


Comments
-
Greg Hurrell
I was going to say that perhaps the existing page model could be abused for this purpose.
We currently have the ability to pull wikitext markup or HTML out of the database for the product pages.
A third option could be to somehow reference the content on the disk (sendfile style or something, ugh).
But really, the whole point of having this kind of static content is to allow nginx to serve it without Rails ever being touched.
Not really sure though if the rewrite trickery that would be required is worth it.
-
Greg Hurrell
nginx notes
Will need to check this, but I believe nginx will only hit Rails if the requested filename does not exist:
if (-f $request_filename) { break; }
ie. if the directory exists, but no file does, it will still hit Rails.
This is actually what we want.
That is, given these directories
/public/products/synergy/
, we don't want nginx to try to serve those itself and effectively mask theproducts#show
action for URLs like https://wincent.com/products/synergy.But we do want it to serve a static file like
/public/products/synergy/docs/index.html
if one is uploaded.Still, like I said, must test it. The config does have this set up:
index index.html;
So I don't know whether the presence of the directory alone will be enough to make it look for
index.html
and if it can't find it, return a 404 rather than forwarding the request to Rails.Static page types
There are really two types of static content that we might wish to serve:
- stand-alone HTML pages: we want these to be served by nginx without hitting Rails at all if possible
- HTML fragments, to be embedded in the existing site layout: by necessity, these need to be mediated by Rails itself
The alternative
If the nginx thing doesn't work out for the first page type, there is an alternative.
We can use the "asset host" trick to give us URLs which at least in part look similar to the product URLs on the main site; eg.
- use: http://assets.wincent.com/products/synergy/doc (or whatever other subdomain may seem more appropriate than "assets")
- matching: https://wincent.com/products/synergy/
So will do some more investigation and see which option we have to pursue.
-
Greg Hurrell
Ah, silly me. I just remembered that the way page caching works under nginx, I already know the answer to my doubts above.
For example, we have a bunch of static cache files like:
/twitter.html
/twitter.atom
/twitter/1.html
So, yes, in the absence of an
/twitter/index.html
file, requests for/twitter
still wind up at the Railstwitter#index
action whenever the cache files are cleared.The only question remaining, then, for this type of static file, is how do we preserve such files across deployments (following the Capistrano pattern, everything in
public
is blown away (including all the cache files) except for stuff underpublic/system
which is actually a symlink elsewhere.The question about what to do about the other type of static file (the one we want to render embedded in the Rails-provided layout) still remains. But I have to pop out now, so no time to think it through.
-
Greg Hurrell
Thinking about it, switching to an "assets" subdomain would have a pretty big downside: would lose the ability to enter "relative" URLs in links (that is,
/products/foo/baz
rather thanhttp://assets.wincent.com/products/foo/baz
). -
Greg Hurrell
One idea to make the deployments easier would be to have a parallel hierarchy which nginx checks first. This would slow it down, but the thing is so fast that it might not even be measurable.
ie. given Rails app at
/rails/current
and static files at/rails/static
, nginx would do something like:# check static area first if (-f $static_root/$request_filename) { break; } # serve static content without hitting Rails location ~ ^/(images|javascripts|stylesheets)/ { expires 10y; } if (-f $request_filename) { break; } # show maintenance page (added by deploy script) if present if (-f $document_root/system/maintenance.html) { rewrite ^(.*)$ /system/maintenance.html last; break; } # cached pages: only for GET requests set $cache_extension ''; if ($request_method = GET) { set $cache_extension '.html'; } # the above is a hack because nginx doesn't allow nested or ANDed ifs if (-f $request_filename$cache_extension) { rewrite (.*) $1.html break; } # everything else goes to Unicorn if (!-f $request_filename) { proxy_pass http://unicorn; break; }
The above is based on quite an old config, and I suspect it could be simplified at least in part by using the
try_files
directive. -
Greg Hurrell
Ok, have a first cut at this working now:
location / { set $cache_extension ''; if ($request_method = GET) { set $cache_extension '.html'; } if (-f $request_filename) { break; } if (-f $request_filename$cache_extension) { rewrite (.*) $1.html break; } if (-f $static_root$uri) { root $static_root; break; } if (-f $static_root$uri$cache_extension) { root $static_root; rewrite (.*) $1.html break; } if (!-f $request_filename) { proxy_pass http://unicorn; break; } }
My initial stab at this didn't work because I tried to incorporate
try_files
and that seems to interact horribly withif
blocks (see "If Is Evil" on the nginx wiki). So it seems there is no choice but to have this string ofif
blocks testing one thing after another.Note that this will give us access to, for example,
/products/wikitext/rdoc/index.html
but not/products/wikitext/rdoc
(with automatic fallback toindex.html
). I am not sure whether this can be done safely:- test for directory at
$static_root$uri
- if $uri is just
/
, then test succeeds (not confirmed whether trailing slash matters yet) - if we then set
root
, we rule out the possibility of a match happening in the real root (although, not sure I'll ever want anindex.html
there anyway)
Perhaps I could just add another test:
if (-f $static_root$uri/index.html) { root $static_root; rewrite $static_root$uri/index.html break; }
Again, want to find out what happens with trailing slashes here.
And looking at it, can probably drop the
$cache_extension
block when looking under$static_root
. - test for directory at
-
Greg Hurrell
Status changed:
- From: new
- To: closed
Add a comment
Comments are now closed for this issue.


| Top |
| GIcon * | background-icon | Read / Write |
| gchar * | background-icon-name | Read / Write |
| gint | count | Read / Write |
| gchar * | label | Read / Write |
| GtkStyleContext * | style-context | Read / Write |
GtkNumerableIcon is a subclass of GEmblemedIcon that can show a number or short string as an emblem. The number can be overlayed on top of another emblem, if desired.
It supports theming by taking font and color information
from a provided GtkStyleContext; see
gtk_numerable_icon_set_style_context()
.
Typical numerable icons:
GIcon *
gtk_numerable_icon_new (GIcon *base_icon
);
gtk_numerable_icon_new
has been deprecated since version 3.14 and should not be used in newly-written code.
Creates a new unthemed GtkNumerableIcon.
Since: 3.0
GIcon * gtk_numerable_icon_new_with_style_context (GIcon *base_icon
,GtkStyleContext *context
);
gtk_numerable_icon_new_with_style_context
has been deprecated since version 3.14 and should not be used in newly-written code.
Creates a new GtkNumerableIcon which will themed according
to the passed GtkStyleContext. This is a convenience constructor
that calls gtk_numerable_icon_set_style_context()
internally.
Since: 3.0
GIcon *
gtk_numerable_icon_get_background_gicon
(GtkNumerableIcon *self
);
gtk_numerable_icon_get_background_gicon
has been deprecated since version 3.14 and should not be used in newly-written code.
Returns the GIcon that was set as the base background image, or
NULL
if there’s none. The caller of this function does not own
a reference to the returned GIcon.
Since: 3.0
void gtk_numerable_icon_set_background_gicon (GtkNumerableIcon *self
,GIcon *icon
);
gtk_numerable_icon_set_background_gicon
has been deprecated since version 3.14 and should not be used in newly-written code.
Updates the icon to use icon
as the base background image.
If icon
is NULL
, self
will go back using style information
or default theming for its background image.
If this method is called and an icon name was already set as
background for the icon, icon
will be used, i.e. the last method
called between gtk_numerable_icon_set_background_gicon()
and
gtk_numerable_icon_set_background_icon_name()
has always priority.
Since: 3.0
const gchar *
gtk_numerable_icon_get_background_icon_name
(GtkNumerableIcon *self
);
gtk_numerable_icon_get_background_icon_name
has been deprecated since version 3.14 and should not be used in newly-written code.
Returns the icon name used as the base background image,
or NULL
if there’s none.
Since: 3.0
void gtk_numerable_icon_set_background_icon_name (GtkNumerableIcon *self
,const gchar *icon_name
);
gtk_numerable_icon_set_background_icon_name
has been deprecated since version 3.14 and should not be used in newly-written code.
Updates the icon to use the icon named icon_name
from the
current icon theme as the base background image. If icon_name
is NULL
, self
will go back using style information or default
theming for its background image.
If this method is called and a GIcon was already set as
background for the icon, icon_name
will be used, i.e. the
last method called between gtk_numerable_icon_set_background_icon_name()
and gtk_numerable_icon_set_background_gicon()
has always priority.
Since: 3.0
gint
gtk_numerable_icon_get_count (GtkNumerableIcon *self
);
gtk_numerable_icon_get_count
has been deprecated since version 3.14 and should not be used in newly-written code.
Returns the value currently displayed by self
.
Since: 3.0
void gtk_numerable_icon_set_count (GtkNumerableIcon *self
,gint count
);
gtk_numerable_icon_set_count
has been deprecated since version 3.14 and should not be used in newly-written code.
Sets the currently displayed value of self
to count
.
The numeric value is always clamped to make it two digits, i.e.
between -99 and 99. Setting a count of zero removes the emblem.
If this method is called, and a label was already set on the icon,
it will automatically be reset to NULL
before rendering the number,
i.e. the last method called between gtk_numerable_icon_set_count()
and gtk_numerable_icon_set_label()
has always priority.
Since: 3.0
const gchar *
gtk_numerable_icon_get_label (GtkNumerableIcon *self
);
gtk_numerable_icon_get_label
has been deprecated since version 3.14 and should not be used in newly-written code.
Returns the currently displayed label of the icon, or NULL
.
Since: 3.0
void gtk_numerable_icon_set_label (GtkNumerableIcon *self
,const gchar *label
);
gtk_numerable_icon_set_label
has been deprecated since version 3.14 and should not be used in newly-written code.
Sets the currently displayed value of self
to the string
in label
. Setting an empty label removes the emblem.
Note that this is meant for displaying short labels, such as roman numbers, or single letters. For roman numbers, consider using the Unicode characters U+2160 - U+217F. Strings longer than two characters will likely not be rendered very well.
If this method is called, and a number was already set on the
icon, it will automatically be reset to zero before rendering
the label, i.e. the last method called between
gtk_numerable_icon_set_label()
and gtk_numerable_icon_set_count()
has always priority.
Since: 3.0
GtkStyleContext *
gtk_numerable_icon_get_style_context (GtkNumerableIcon *self
);
gtk_numerable_icon_get_style_context
has been deprecated since version 3.14 and should not be used in newly-written code.
Returns the GtkStyleContext used by the icon for theming,
or NULL
if there’s none.
a GtkStyleContext, or NULL
.
This object is internal to GTK+ and should not be unreffed.
Use g_object_ref()
if you want to keep it around.
[nullable][transfer none]
Since: 3.0
void gtk_numerable_icon_set_style_context (GtkNumerableIcon *self
,GtkStyleContext *style
);
gtk_numerable_icon_set_style_context
has been deprecated since version 3.14 and should not be used in newly-written code.
Updates the icon to fetch theme information from the given GtkStyleContext.
Since: 3.0
“background-icon”
property“background-icon” GIcon *
The icon for the number emblem background.
Flags: Read / Write
“background-icon-name”
property“background-icon-name” gchar *
The icon name for the number emblem background.
Flags: Read / Write
Default value: NULL
“count”
property“count” gint
The count of the emblem currently displayed.
Flags: Read / Write
Allowed values: [-99,99]
Default value: 0
“label”
property“label” gchar *
The label to be displayed over the icon.
Flags: Read / Write
Default value: NULL
“style-context”
property“style-context” GtkStyleContext *
The style context to theme the icon appearance.
Flags: Read / Write


WITNESS LIST Appropriations-S/C on Article IX Committee February 23, 1999-12:00P Article IX Provisions On: Majek, Mark (Small Agency Task Force) Pereyra, Ernest (Midsize Agency Coordinating Committee) White, Ed (State Agency Coordinating Committee)


No setup fees. Pay as you go.
Use our point of sale (POS) system
See your order summary and documentation
Create menus and order forms (max. 6 products)
Get access to revenue reports
Send notification updates to your customers
Coming soon!
Customize the branding of your order forms
Personalize your order form link
Add as many products as you want!
Get access to data analytics
Compute for your net profit!


In a previous article in this series, we learnt what is the Semantic Analysis module of a Compiler.
The key points to remember are:
- Semantic Analysis is the third step in front-end compilation (after Lexical Analysis and Parsing).
- It’s meant to catch all errors that weren’t caught by the first two steps.
- It’s the last step in front-end compilation, so it’s also the last point where a source code can be rejected.
In the first article about Semantic Analysis (see the references at the end) we saw what types of errors can still be out there after Parsing. These are the ones that Semantic Analysis must catch.
Now it’s time to go deeper into the details! We need to learn how to prototype and design the Semantic Analysis module within a Compiler. In order to do so, we must understand:
- What data structures and algorithms are commonly used for these objectives.
- How symbols are handled within Semantic Analysis.
- What’s typing, and why is important.
- How to handle complex language constructs, such as the ones in OOP languages.
Data Structures in Semantic Analysis Algorithms
In my opinion, an accurate design of data structures counts for the most part of any algorithm. In different words, your strategy may be brilliant, but if your data storage is bad the overall result will be bad too.
Luckily, choosing a data structure is not too difficult in most cases. A few principles are worth remembering:
- Your algorithm needs to do a lot of searches? Think about hash tables.
- Your algorithm is recursive? You may need a stack, somewhere.
- Your data is naturally ordered? Consider a balanced binary tree (maybe a red-black tree).
In my experience, if you truly master Arrays, Lists, Hash Maps, Trees (of any form) and Stacks, you are well ahead of the game. If you also know a few famous algorithms on Graphs then you’re definitely good to go.
Now, back to Semantic Analysis.
The first data structure we’ll need is something to store information about symbols. Please be aware that I will use the words “identifier” and “variable” interchangeably. Both are examples of “symbols”. To be clear, in the following line
int xyz = 0;
the string int
is a type, the string xyz
is the variable name, or identifier. It’s also a symbol in this language. In fact, both int
and xyz
are symbols.
What do we need to know about symbols, when performing Semantic Analysis? Mainly, we need to know that a symbol was defined, before being used. For example, the line
int k = i + j;
is not valid unless both symbols j
and i
were defined earlier.
Thus, all we need to start is a data structure that allows us to check if a symbol was already defined. Historically, this is called Symbol Table.
Let’s think about the operation that we will make most commonly on a Symbol Table:
- Each time we encounter a symbol, we add it to the table. Optionally, we may want to check whether that symbol was already defined. This can be a useful information to debug, such as “Warning: Variable Already Defined!“.
- Every time we see an expression (that is a computation), or an assignment, or pretty much any type of statement, we must verify that all symbols within it are well defined.
Therefore, we understand that insertion and search are the two most common operations we’ll make on the Symbol Table.
There are many valid solutions to the problem of how to implement a Symbol Table. As I said earlier, when lots of searches have to be done, a hash table is the most obvious solution (as it gives constant search time, on average).
However, we will also have to do a lot of insertion operations. Hence, an alphabetically ordered Linked List also comes to mind, so that we can use binary search (that’s logarithmic search time) followed by insertion (that’s also loogatithmic time operation, in a ordered Linked List). Clearly, if you don’t care about performance at this time, then a standard Linked List would also work.
By now you have hopefully spot a problem. In modern languages, symbols are repeated all the time! What about these two methods:
int foo1 (int x) {
...
}
void foo2 (int x) {
...
}
The symbol x
is used here twice, and in a totally correct way. Semantic Analysis should NOT return a compilation error because of that.
What is going on here? You’ve probably heard the word scope, especially if you read my previous article on the differences between programming languages.
A scope is a subsection of the source code that has some local information. These are pieces of information that exist only in that scope. For instance, each function/method defines its own scope.
Furthermore, variables declaration and symbols definition do not generate conflicts between scopes. That is, the same symbol can be used for two totally different meanings in two distinct functions. This is how pretty much all programming languages work.
The most basic programming languages, such as experimental, configuration or DS (Domain Specific) languages lack the possibility to define multiple scopes. They have only one global scope, therefore a single Symbol Table is enough to compile them. But what about more complex languages, where multiple scopes are very commonly used?
In such cases, a single Symbol Table would not work. Why? Because the same symbol would be overwritten multiple times even if it’s used in different scopes (for example, in different functions), and that’s definitely not what we want.
Let’s think about this issue more deeply. Actually, let’s put ourselves in the Semantic Analysis’ shoes, while it’s scanning the source code top-down. Let’s imagine we meet a function definition, or some other statement that defines a new scope. What has to be done at this point?
One thing we know for sure. This new scope will have to be terminated before the outer scope (the one that contains the new scope) is closed. For example, a class in Java defines a new scope that is inside the scope of the file (let’s call it global scope, for simplicity). On the other hand, any method inside that class defines a new scope, that is inside the class scope.
Thus, a method’s scope must be terminated before the class scope ends. Similarly, the class scope must be terminated before the global scope ends. However, two methods scopes are independent. More exactly, a method’s scope cannot be started before the previous method scope ends (this depends on the language though; for example, Python accepts functions inside functions).
This type of reasoning should ring a bell in our head: Stack! A stack data structure works with the Last-In First-Out policy. It’s exactly like a stack, where everything you put on the top for last, is also the first thing to be taken from the top of the stack. The operation on a Stack are called push
(insert on top), and pop
(extract from top).
Let’s see how this would work in Semantic Analysis, with a realistic step by step example, for Python.
- Start of the file, push on the stack the global scope.
- Start of a class, push on the stack a new scope for this class.
- Start of a method, push on the stack a new scope.
- End of a method, pop from the stack. As you can see, the top of the stack is the scope that was created at step 3, so we are correctly terminating the scope for this method.
- Start of another method, push on the stack a new scope.
- Start of a new (nested) method, push yet again a new scope.
- End of a method. This has got to be the method defined at step 6, so we are good simply popping from the top of the stack.
- End of a method. This must be the method started at step 5, which now is also what’s on the top of the stack, because step 7 removed the nested one. So, pop again.
- End of the class. Guess what, this class’ scope is exactly found at the top of stack. So, pop.
- End of the file. Pop. Nicely done.
At this point, the only question left is what are we actually pushing on the top of the stack? What kind of data do we need there?
For sure we need a Symbol Table, because each scope must have its own. There may be need for more information, and these will depend on the language specification. Therefore, the best thing to do is to define a new class, or some type of container, and use that to save information for a scope. It will have at least a Symbol Table, but maybe more.
In other words, we use a stack of scopes as data structure, where each scope is a container (struct-like) with a Symbol Table, and possibly more. This data structure should support the following five operations:
enter_scope()
, initialize a new scope on the top of the stack.exit_scope()
, terminates the current scope (pop).find_symbol(x)
, searches the symbolx
in any scope, and returns the first found.add_symbol(x)
, adds the new symbol to the current top-scope.check_scope(x)
, returnsTrue
is the symbolx
is defined in the current top-scope.
Before continuing…
Before going further, and getting into the intricacies of typing, I want to clarify one point.
I have been saying so far that the Semantic Analysis reads the source code from the top to the bottom. This is not true, and I said it just to ease the reading. Now it’s time to clarify this point.
Semantic Analysis analyzes the output of the Parsing stage.
The three front-end steps are like steps down a staircase. The first, Lexical Analysis, gets the output from the external word, that is the source code. It produces a sequence of Tokens. After that, the original source code is almost forgotten.
The second step, the Parser, takes the output of the first step and produces a tree-like data structure, called Parse Tree.
Thus, the third step (Semantic Analysis) gets as input the output of the Parser, precisely the Parse Tree so hardly built. All Semantic Analysis work is done on the Parse Tree, not on the source code.
Now we are ready to continue!
Handling Symbols in Advanced Programming Languages (OOP)
The general strategy I described in the previous section can be summarized in the following points:
- At the beginning of the Semantic Analysis, create a stack of scopes, and push onto it the global scope. In such scope, put all global symbols definitions, such as for types (
int
,float
, etc.). - Every time a new scope starts in the Parse Tree (for example when a function starts), use the method
enter_scope()
on the stack of scopes. - Every time a new symbol is defined (for example in a variable declaration), use the method
add_symbol(...)
on the stack of scopes. The symbol will be added in the current scope (at the top of the stack). - Every time we require to check whether an expression is well-defined, use
check_scope(...)
for every symbol used in the expression. - Everytime a scope ends (for example, when the current node in the Parse Tree is the end of a function) use
exit_scope()
on the stack of scopes.
This is in fact a sound strategy. It’s also the basic version of strategies implemented in many real compilers. Why basic? Quite simply, many adjustments have to be made to handle the specification of each particular language. That said, these are the core principles of all Semantic Analysis algorithms.
One of the main adjustments is about Object Oriented Programming Languages. In many (if not all) of them, class names can be used before they are defined. This clashes against the simple fact that symbols must be defined before being used.
In fact, this reasoning leads us to a very important and generic question. How to deal with symbols that can be used before being fully defined?
The solutions to this problem is very instructive, and that’s why I am focusing on it. The trick is to perform multiple passes over the Parse Tree.
For example, during the first pass, Semantic Analysis would gather all classes definition, without spending time checking much, not even if it’s correct. It would simply gather all class names and add those symbols to the global scope (or the appropriate scope).
The take-home message here is that multiple passes over the Parse Tree, or over the source code, are the recommended way to handle complicated dependencies.
Once again, the focus is on code modularity, easiness to debug and extend. To split a complex task, such as understanding class definitions, into multiple passes over the Parse Tree is a good solution.
Another common problem to solve in Semantic Analysis is how to analyze the “dot notation”. This exists both in old languages and in more modern OOP languages. For example, in C the dot notation is used to access a struct elements. In Java, dot notation is used to access class members, as well as to invoke methods on objects.
In fact, this is less complicated than it may seem. When Semantic Analysis gets the first part of the expression, the one before the dot, it will already know in what context the second part has to be evaluated. What this really means is that we must add additional information in the Symbol Table, and in the stack of Scopes. There isn’t a unique recipe for all cases, it does depend on the language specification.
Let’s try to understand this better with an example. Let’s use Java.
class Person {
String name;
String lastName;
int age;
public Person(String name, String lastName, int age) {
... // assign objects fields
}
public int getAge() {
return this.age;
}
};
// in the main
Person p1 = new Person("Jane"', "Doe", 30);
int a = p1.age();
How does the Compiler know that the final line (the assignment) is well-typed?
This is an overview of what will happen during compilation (and Semantic Analysis, in this case), for that line:
- The Compiler sees the declaration of the identifier
a
. It checks the Symbol Table for the current scope to see if this symbol was already defined. As it’s not, it add it into the table. - The Compiler also sees that the identifier is declared to be
int
. Therefore, it adds this information in the Symbol Table. - It then sees that the remainder of the line is an expression. Therefore it starts evaluating it.
- It will see that is an expression with a
.
. This is the first thing that it will see because the dot is the root of the subtree. Imagine a subtree where the dot is the root, then on the left branch we havep1
and on the left we haveage()
. - It will analyze the left subtree and understand the context is about the symbol
p1
. It will search in the Symbol Table within the current scope and find it (if not found, then an error must be reported). It will also get access to all additional information that this symbol has in the Table. For example, it will know that it’s an Object of type Person. - Then it will go search the symbol
Person
, and it will find it in some outer scope (because class names are defined in higher-level scopes). - It will get access to all additional information for the symbol
Person
. For example, it will know that another symbolage
is defined for allPerson
objects, and that is a method without argument that has return typeint
. - It will conclude that the code is correctly typed!
How Semantic Analysis Checks Types
The other big task of Semantic Analysis is about ensuring types were used correctly by whoever wrote the source code. Used, and understood correctly, I dare to say. In this respect, modern and “easy-to-learn” languages such as Python, Javascript, R really do no help. Let me tell you more about this point, starting with clarifying what such languages have different from the more robust ones.
Static vs Dynamic Typing
You’ve certainly heard about scripting languages. What are they? What does it mean to write a script? In fact, there’s no exact definition of it, but in most cases a script is a software program written to be executed in a special run-time environment.
Some examples: Javascript programs are written to be run in the web browser (well, before NodeJS was created); Python programs are written to be run in the Python interpreter; Bash programs are written to be run in the Unix shell environment.
Pretty much always, scripting languages are interpreted, instead of compiled. And this simple fact makes all the difference from our point of view. Generally, a language is interpreted when it’s lines of code are run into a special environment without being translated into code machine.
Some examples, again: C is a compiled language; So are C++ and Java. Perl, instead, is an interpreted language. The same holds true for Javascript and Python.
Unfortunately, things are not so much black and white. It turns out most programming languages are both interpreted and compiled. The classic example is Java. A Java source code is first compiled, but not into machine code, rather into a special code called bytecode, which is then interpreted by a special interpreter program, famously known as Java Virtual Machine.
I’ve already written a lot about compiled versus interpreted languages, in a previous article. Let’s go back to typing, instead.
The reason why I said above that types have to be “understood” is because many programming languages, in particular interpreted languages, totally hide the types specification from the eyes of the developer. This often results in misunderstanding and, unavoidably, low-quality code.
I really don’t mean to be frigid or overly critical, but let me give you one example. I’ve often seen code like this written in Python
# WARNING: This is code is not well written
def foo(arg1):
if arg1:
return [1,2,3] # list
else:
return -1 # integer
Now, this code may be correct, may do what you want, may be fast to type, and can be a lot of other nice things. But why on earth your function sometimes returns a List type, and other times returns an Integer type?! You’re leaving your “customer”, that is whoever would like to use your code, dealing with all issues generated by not knowing the type.
In the good old days, functions used to have a complete signature. Not just a def
keyword, but a complete signature such as:
<return-type> <function-name> ( <list of arguments with types> )
For example, the previous Python example would be written somehow like the following:
list foo (boolean arg1)
and you would know for sure that the function named foo
returns a List
type, and receives as input only one object, and that has got to be a boolean
.
Why do I like so much signatures? Because in compiled languages, the Compiler would give an error if the implementation didn’t match the signature. So for example, if you did something like this
list foo (boolean arg1):
if arg1:
return [1,2,3]
else:
return -1
your code would not compile!  You would stop during compilation, with a message like the following (this one is from the Java Compiler):
error: incompatible types: int cannot be converted to int[]
return -1;
At some point in Computer Science history, someone thought to remove typing because it allows faster prototyping (??!).
I have quite a harsh opinion on this point. What does that even mean? Does writing weak and possibly buggy code allow faster prototyping? I really don’t think so.
And I am not the only one to think that. In fact, starting from the version 3.5, with the Python programming language you can use something very similar to signature (see the references at the end)! The official name is Type Hints, and you would write this way:
def foo (arg1: bool) -> list :
if arg1:
return [1,2,3]
else:
return -1
I like that. A great step in the good direction.
In my opinion, programming languages should be designed as to encourage to write good and high-quality code, not just some code that maybe works.
Now, to tell you the full story, Python still is an interpreted language, so there’s no compiler which would generate an error for the above function. But I believe many IDE would at least show a red warning, and that’s already something.
At this point it should be pretty clear that I like static typing! Static typing, roughly said, is just another way of saying typing that is checked by the Compiler, before the program is run.
Basically, the Compiler can know the type of each object just by looking at the source code. The other side of the coin is dynamic typing, when the type of an object is fully known only at runtime.
Even though I like static typing, I must say it has some drawbacks. In particular, it’s clear that static typing imposes very strict constraints and therefore some program that would in fact run correctly is disabled by the compiler before it’s run. In simpler terms, programs that are not correctly typed don’t even get a chance to prove they are good during runtime! They are aborted long before that (during Semantic Analysis, in fact!).
Before continuing…
I have already been talking about typing and types for a little while. And yet I didn’t answer the most obvious question: what is a type?
Such question does not have a universally accepted answer. It may change from one language to another. Generally speaking though, the most common answer is that a type is:
- A set of values, along with
- A set of operations that are allowed on those values.
Let me give an example to clarify.
In many languages, int
is a type. That’s fine with the definition above, because the set of values is clearly defined (for example, from -2^31
to 2^31 - 1
) as well as the valid operations on them. Most commonly, addition, subtraction, multiplication, division, modulus, but there might be others depending on the language. For example, Python does not define increment and decrement, while Java, C and C++ do, with the operands ++
and --
.
It’s worth noting that the second point in the definition, about the set of valid operation, is extremely important. And it’s actually even trickier to check for Semantic Analysis. Let’s jump back straight into that.
Understanding Types in Semantic Analysis
We should now have a pretty good understanding of what a type is, and what is the difference between a static and a dynamic type. The question is, how does Semantic Analysis understand all of this?
After I studied these concepts, I now believe that the overall concept can be split into two subtasks to accomplish:
- Understanding the type of an expression.
- Understand the type of a declaration.
Arguably, a declaration can be an assignment, and in such case an expression is assigned to a variable. If that’s the case, then point 2 collapses onto point 1. Let’s talk about them separately anyway, starting by the easiest: declarations.
In statically typed languages, declaration are no-tasks for Semantic Analysis: the type is declared before the variable name, such as in
int x = 0;
which is a valid line in a lot of languages, for instance C, Java and C++. In this case, Semantic Analysis doesn’t need to do complicated stuff:
- To handle the Symbol
x
. Probably, check in the Symbol Table, within the current scope, if that Symbol already exists. In many languages you cannot declare the same symbol twice. Then, if it’s all good, add that Symbol into the Table. - Simply assign the type int to the Symbol
x
. There’s really nothing else to decide here, because that’s what the source code says. - Check that the right-hand side of the declaration, if present, has the same type (
int
, in this case). This boils down to understanding expressions type; I’ll talk about it in a minute.
Overall, this is a fairly easy task. The scenario becomes more interesting if the language is not explicitly typed. Consider the following line of valid Python code.
x = 3 + 5 >= 7
How should we change the steps above, in this case? Let’s see:
- To handle the Symbol
x
. I don’t know any non-typed language that forbids to declare the same symbol twice. So, in this case you would still need to check the Symbol Table and, ifx
was already defined, override the definition in the current scope. I would also print a warning message at screen! - Now, we can’t actually assign any type to the Symbol
x
. We must first analyze the remainder of the line, and figure out what typex
is. This procedure is called type inference.
I find type inference very interesting, and this is why I designed a new programming language that is not explicitly typed: mainly because I wanted to practice types inference in Semantic Analysis and Compilation!
Type inference is best shown when we have to figure out the type of a complex expression (the original point 1 of this discussion), so let’s get to it.
To infer the type of an expression we have to master the algorithm to walk down the Parse Tree. I explained to them in detail in a previous article, so I can only recommend you to read that, or to google for the following algorithm: “deep-first search”, “tree visit”.
Why do we need to walk through the Parse Tree? The reason is that, no matter how complex an expression is, the real content is in the leaves of the tree. For example, consider the following expression:
x = ((1 + 4) > y) or (5 + (3 * z) % 7 < 3)
The subtree that stores this expression is likely to be quite large. Still, the nodes that are at the deepest level (leaves) will contain the Tokens
<1, int>
<4, int>
<y, identifier>
<5, int>
...
Thus, when we get down to a leaf, for example <1, int>
, that’s the leftmost leaf in the tree, it takes nothing to infer that this token’s type is int
.
The most difficult case for a leaf is when it is an identifier. In such scenario, we must look up in the Symbol Table for the current scope, and get the type of the symbol from there. If the identifier is not in the Symbol Table, then we should reject the code and display an error, such as Undefined Variable
.
When we’ve done that for all leaves, we start walking backwards in the tree. In other words, we uplift by one level the types inferred at the last level in the tree. In the second to last level we should meet all operators Tokens. For the previous example these would be
<'+', operator>
<'>', conditionalOperator>
<'or', logicalOperator>
...
Each operator, by definition itself, it’s applied to a given number of operands. Most commonly two operands. Given that the operands are the types that we computed at the leafs, how can we compute now the type resulting from the application of each operator?
There’s really nothing magic here. We simply must check for each operator, and for each type of the first operand type, and for each type of the second operand, what’s the result type.
This can be realized, for instance, with a cascade of if-then-else. Something like the following snippet:
if (operator is '+')
if (operand-1 is int)
if (operand-2 is int)
return "int";
else if (operand-2 is double)
return "double";
// and so on...
However, this would be a pretty stupid way to implement it. Tedious and slow too. I believe the best way is with a set of lookup tables. We would create a look-up table for each operator. The columns of these tables are the possible types for the first operand, and the rows for the second operand. If the operator works with more than two operands, we would simply use a multi-dimensional array.
For example, the result types of the ‘+’ operator would be given by the following table:
int plusOperatorTypes[6][6] = {
/* int float string bool null list */
/* int */ { _int, _float, _undef, _undef, _undef, _undef},
/* float */ { _float, _float, _undef, _undef, _undef, _undef},
/* string */ { _undef, _undef, _undef, _undef, _undef, _undef},
/* bool */ { _undef, _undef, _undef, _undef, _undef, _undef},
/* null */ { _undef, _undef, _undef, _undef, _undef, _undef},
/* list */ { _undef, _undef, _undef, _undef, _undef, _undef}
}
This snippet was taken from the Semantic Analysis implementation in the Compiler for my own programming language.
The Grammar I designed defines as basic types int
, float
, null
, string
, bool
and list
. I am using symbolic names, implemented like an enum
object, but with integer values to easily access the lookup table. For instance, _int
, to refer to the integer type.
Let’s take a look at one example, to clarify things.
We could use the look-up table when we are to decide the type of the following expression:
x = 4 + true
The type for the token 4
is _int
(that’s implemented as 0
), while the type for the token true
is _bool
(that’s 3
). Thus, the fast lighting lookup operation
plusOperatorType[_int][_bool] // equal to plusOperatorType[0][3]
would give the result type of the operation (4 + true)
. In this example, the result is _undef
, a special value that I used to say the operation is not allowed.
If the lookup operation says that the operation is not allowed, then again we should reject the source code and give an error message as clear as possible.
When we have done that for all operators at the second to last level in the Parse Tree, we simply have to repeat the procedure recursively. Uplift the newly computed types to the above level in the tree, and compute again types.
This has to be done over and over until we reach the root of the subtree for the expression. At that point, we will have the result type of the overall expression, or an error code if the expression wasn’t typed correctly. That’s it!
Extensions to static typing check systems
You can easily imagine what a debate has taken place, over many years, between sustainers of static typing on one side, and supporters of dynamic typing on the other. As of today, in my opinion, there’s no winner.
The main criticism moved to statically enforced typing is that many source code may be correct at runtime, but the compiler kills them before they are ever run. Why is that?
The thing is that source code can get very tricky, especially when the developer plays with high-level semantic constructs, such as the ones available in OOP. For example, let’s take a look at the following Java snippet.
class A {
... // something
}
class B extends A {
... // something more than A
}
class MainClass {
public static void main (String[] args) {
A a1; // from now on, the static type of the identifier a1 will always be A
a1 = new B(); // this should be allowed according to inheritance rules
// (and indeed is valid, in Java)
}
}
The code above is a classic example that highlights the difference between the static and dynamic types, of the same identifier.
When the variable a1
is first declared, the compiler must add it into the Symbol Table and assign the type A
to it. Thus, the type A
will be the static type of the identifier a1
for the rest of the program.
However, according to the inheritance rule is-a, the assignment a1 = new B();
is valid.
How to use the is-a rule? It’s very simple, just answer the following question: Is new B()
an A
object? Yes, it is, because the class B
extends the class A
. Therefore, the assignment is valid. And if we are building an OOP, robust, programming language, then our compiler has to understand all of what I just said.
Another problem that static typing carries with itself is about the type assigned to an object when a method is invoked on it. Let me clarify with an example, again, in Java.
class A {
public A method1 () {
... // some operation
return this;
}
}
This type of code where the object itself is returned is actually quite common, for example in many API calls, or in the Builder Design Pattern (see the references at the end).
Let’s now say that we want to extend class A.
class B extends A {
... // something here, without overriding method1
}
So far so good. Now, let’s say we want to initialize a B
object using the builder method1. What we should (and would like to) do is:
B bObject = (new B()).method1();
This is a conceptually correct line of code. We instantiate a bare-bone B
object, using the normal new B()
, and then call the method1
on it, because we know it will do some operations and then return this.
If you start being a bit confused, then don’t worry. This stuff is tricky! Just take your time and read slowly.
The point is that line of code give a compilation error! Why is that? The problem lies in the fact that the return type of method1
is declared to be A. And even though we can assign a B
object to a variable of type A
, the other way around is not true.
In other words, while a B
object is-a A
object, the inverse is not true. Therefore, the compiler will see a wrongly-typed assignment, and will reject the source code. What a setback!
To solve this specific issue, some languages define “self-type”. Practically speaking, this is a keyword (e.g., SELF_TYPE
). What’s that? The important thing to know is that self-type is a static concept, NOT dynamic, which means the compiler knows how to handle it.
Unfortunately Java does not support self-type, but let’s assume for a moment it does, and let’s see how to rewrite the previous method.
// inside class A
public SELF_TYPE method1() {
... // some operation;
return this;
}
With that, a Java Compiler modified to handle SELF_TYPE
would know that the return type of method1
is-a A
object. And although this is a static check, it practically means that at runtime it can be any subtype of A
.
Thus, the line that previously gave compilation error
B bObject = (new B()).method1();
would now pass all checks, including Semantic Analysis’ ones.
Why?
The compiler will see that the return type of method1
is SELF_TYPE
, then it will see the dot notation <object>.<method>
, in this case (new B()).method1()
, and will understand that in this context SELF_TYPE = B
. So, everything will go fine.
Summing up
So far we have seen in detail static and dynamic typing, as well as self-type. These are just two examples, among many, of what extensions have been made over the years to static typing check systems.
To complicate things further, there’s a great deal of other, creative, things that happen in modern languages. I can’t possibly mention all of them, and even if I did the list would become incomplete in a day.
Thus, to wrap up this article, I just want to give a partial list of things that have been tried in one or more programming languages. It will look like a random list of words, but you may recognize some names, and I warmly recommend you to do your own research about them (Wikipedia is a good starting point).
- Generics.
- Self-Type.
- Agent Oriented Programming (the new frontier of OOP).
- Point-Free Style.
- Metaclasses (Read this amazing StackOverflow answer).
- Asynchronous Programming.
What’s next
In this article, I went through a lot of advanced content. It wasn’t easy for me at first place to study it, and I do have a good background in Computer Science, so don’t worry if you feel overwhelmed.
What I want to do next is to avoid leaving all these concepts lost in the wind. This is what I always try to avoid in my self-assigned classes. I found that the best way to do so is to assign myself a real, and quite complex project. Not at the industrial-strength level, but far more advanced than the typical MOOC assignments.
If you have seen my previous articles then you know that for this class about Compilers I decided to build a new programming language. It’s not too fancy, but I am building it from the ground, and without using any automatic tool.
I have already designed the Grammar (by hand), implemented the Lexical Analysis (by hand, in C) and implemented the Parser (by hand, in C). Therefore, I now want to implement all those concepts that I learnt about Semantic Analysis and that apply to my language. And will tell you about it in the next article!
References
Language Implementation Patterns. A book by Prof. Terrence Parr.
My article Why There Are So Many Programming Languages?. (Talks about front-end, back-end compilation).
My article How to Design A Programming Language Parser (If you don’t know yet what a Parse Tree is).
Alex Aiken’s class Compilers at Stanford University. Available as free MOOC online.
Wikipedia page about the Builder Software Design Pattern.
Python PEP 0484 about Type Hints, https://www.python.org/dev/peps/pep-0484.


bitwise
Multi base interactive calculator supporting dynamic base conversion and bit manipulation.
More information: home page
- Run using interactive mode:
bitwise
- Convert from decimal:
bitwise 12345
- Convert from hexadecimal:
bitwise 0x563d
- Convert a C-style calculation:
bitwise "0x123 + 0x20 - 30 / 50"


Razor FavIcon Support v3.0
Background & Goal
This is based on information from sources like
Goal: Create a simple command for the default use case, and provide more complex options for advanced stuff.
Basic Default Use Case
The default use case should
Take just 1 PNG image - to keep it simple. Ideally it could also contain placeholders like {h} & {w} or similar, which if provided would add the size properties to the url, for scenarios where you have an image resizer. This would allow you to do both:
- "/portals/0/favicon.png"
- "/portals/0/favicon.png?h={h}&w={w}" for ImageResizer.net
- "/portals/0/favicon.png?height={h}&width={w}" for Thorstons image resizer
- "/portals/0/favicons/icon-{h}x{w}.png" for pre-resized images
Generate the important headers, but not bloat the header with all possible sizes and variations. These would be:
favicon for IE 11 - rel="shortcut icon", as png (supported in IE 11)
This will cover any IE11 favicon when an html-page is shown. Note that when the browser shows a pdf/image only, it doesn't have html to tell it about favicons, so that will still pick up favicon.ico in the root.general icon - rel="icon", as png, ideally at least 250x250 this is for the general case, but actually I'm not even sure what case that is...
touch-icons for mobile devices rel="apple-touch-icon" for 196x196
this supposedly works for iOS and Android.
Specifically, we would NOT support the following for now
- IE 6-10, as these are fully deprecated
- Mac Desktop icons
- Windows 10 desktop icons
- we won't implement rel="apple-touch-icon-precomposed" as that's only for ios 6 and below
So basically it is a command like:
HtmlPage.AddIcon("/portals/0/favicon.png")
Advanced Use Cases
Advanced use cases should allow the following customizations, which limits or extends the icons added.
- specific list of sizes
- specific list of known types (icon, shortcut icon, apple-touch...)
Not Supported Use Cases
Basically anybody can always add more headers to the page using HtmlPage.AddHeader(...)
, so it's not necessary to support every exotic variation. This command should make the 80% use cases super easy (just mention 1 file name) and the remaining 15% easy (with addition of parameters), but exotic stuff should not be covered until it's a common standard. These are things probably not supported in the basic API
- meta tag for msapplication-TileColor and msapplication-TileImage
- rel="apple-touch-icon-precomposed" as this is for older iOS and iOS devices usually update very often - see also https://mathiasbynens.be/notes/touch-icons