奇安信数据安全研究院
Home
Talks
Publications
Members
PaperCollection
Contact
Wei Chen
Visualization
Visual Analytics
Biomedical Image Computing
Latest
ConceptExplorer: Visual Analysis of Concept Drifts in Multi-source Time-series Data
IEEE VAST
|
IEEE Conference on Visual Analytics Science and Technology
Time-series data is widely studied in various scenarios, like weather forecast, stock market, customer behavior analysis. To …
Xumeng Wang
,
Wei Chen
,
Jiazhi Xia
,
Zexian Chen
,
Dongshi Xu
,
Xiangyang Wu
,
Mingliang Xu
,
Tobias Schreck
PDF
Cite
Video
DRGraph: An Efficient Graph Layout Algorithm for Large-scale Graphs by Dimensionality Reduction
TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Efficient layout of large-scale graphs remains a challenging problem: the force-directed and dimensionality reduction-based methods …
Minfeng Zhu
,
Wei Chen
,
Yuanzhe Hu
,
Yuxuan Hou
,
Liangjun Liu
,
Kaiyuan Zhang
PDF
Cite
Code
Video
Exemplar-based Layout Fine-tuning for Node-link Diagrams
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
We design and evaluate a novel layout fine-tuning technique for node-link diagrams that facilitates exemplar-based adjustment of a …
Jiacheng Pan
,
Wei Chen
,
Xiaodong Zhao
,
Shuyue Zhou
,
Wei Zeng
,
Minfeng Zhu
,
Jian Chen
,
Siwei Fu
,
Yingcai Wu
PDF
Cite
Video
Video
Suppl.
EEMEFN: Low-Light Image Enhancement via Edge-Enhanced Multi-Exposure Fusion Network
AAAI
|
Proceedings of the 34th AAAI Conference on Artificial Intelligence
This work focuses on the extremely low-light image enhance-ment, which aims to improve image brightness and revealhidden information in …
Minfeng Zhu
,
Pingbo Pan
,
Wei Chen
,
Yi Yang
PDF
Cite
RSATree: Distribution-AwareData Representation of Large-Scale Tabular Datasets for Flexible Visual Query
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Analysts commonly investigate the data distributions derived from statistical aggregations of data that are represented by charts, such …
Honghui Mei
,
Wei Chen
,
Yating Wei
,
Yuanzhe Hu
,
Shuyue Zhou
,
Bingru Lin
,
Ying Zhao
,
Jiazhi Xia
PDF
Cite
Code
A Natural-language-based Visual Query Approach of Uncertain Human Trajectories
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Visual querying is essential for interactively exploring massive trajectory data. However, the data uncertainty imposes profound …
Zhaosong Huang
,
Ye Zhao
,
Wei Chen
,
Shengjie Gao
,
Kejie Yu
,
Weixia Xu
,
Mingjie Tang
,
Minfeng Zhu
,
Mingliang Xu
PDF
Cite
Slides
Evaluating Perceptual Bias During Geometric Scaling of Scatterplots
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Scatterplots are frequently scaled to fit display areas in multi-view and multi-device data analysis environments. A common method used …
Yating Wei
,
Honghui Mei
,
Ying Zhao
,
Shuyue Zhou
,
Bingru Lin
,
Haojing Jiang
,
Wei Chen
PDF
Cite
Slides
Visual Analytics for Electromagnetic Situation Awareness in Radio Monitoring and Management
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Traditional radio monitoring and management largely depend on radio spectrum data analysis, which requires considerable domain …
Ying Zhao
,
Xiaobo Luo
,
Xiaoru Lin
,
Hairong Wang
,
Xiaoyan Kui
,
Fangfang Zhou
,
Jinsong Wang
,
Yi Chen
,
Wei Chen
PDF
Cite
DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis.
IEEE CVPR
|
IEEE/CVF Conference on Computer Vision and Pattern Recognition
In this paper, we focus on generating realistic images from text descriptions. Current methods first generate an initial image with …
Minfeng Zhu
,
Pingbo Pan
,
Wei Chen
,
Yi Yang
PDF
Cite
location2vec: a situation-aware representation for visual exploration of urban locations
IEEE TITS
|
IEEE Transactions on Intelligent Transportation Systems
Understanding the relationship between urban locations is an essential task in urban planning and transportation management. Whereas …
Minfeng Zhu
,
Wei Chen
,
Jiazhi Xia
,
Yuxin Ma
,
Yankong Zhang
,
Yuetong Luo
,
Zhaosong Huang
,
Liangjun Liu
PDF
Cite
Urban Structure Accessibility Modeling and Visualization for Joint Spatiotemporal Constraints.
IEEE TITS
|
IEEE Transactions on Intelligent Transportation Systems
In modern cities, service providers want to identify the regions that are hard to reach from multiple fire stations, a citizen wants to …
Farah Kamw
,
Shamal AL-Dohuki
,
Ye Zhao
,
Thomas Eynon
,
David Sheets
,
Jing Yang
,
Xinyue Ye
,
Wei Chen
PDF
Cite
DOI
Exploring the Sensitivity of Choropleths under Attribute Uncertainty.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
The choropleth map is an essential tool for spatial data analysis. However, the underlying attribute values of a spatial unit greatly …
Zhaosong Huang
,
Yafeng Lu
,
Elizabeth Mack
,
Wei Chen
,
Ross Maciejewski
PDF
Cite
Code
Slides
Video
Semantics-Space-Time Cube: A Conceptual Framework for Systematic Analysis of Texts in Space and Time.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
We propose an approach to analyzing data in which texts are associated with spatial and temporal references with the aim to understand …
Jie Li
,
Siming Chen
,
Wei Chen
,
Gennady Andrienko
,
Natalia Andrienko
PDF
Cite
DOI
VAUD: A visual analysis approach for exploring spatio-temporal urban data.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Urban data is massive, heterogeneous, and spatio-temporal, posing a substantial challenge for visualization and analysis. In this …
Wei Chen
,
Zhaosong Huang
,
Feiran Wu
,
Minfeng Zhu
,
Huihua Guan
,
Ross Maciejewski
PDF
Cite
Video
A User Study on the Capability of Three Geo-Based Features in Analyzing and Locating Trajectories.
IEEE TITS
|
IEEE Transactions on Intelligent Transportation Systems
Visual analysis is widely applied to study human mobility due to the ability in integrating contextual information multiple data …
Xumeng Wang
,
Tianlong Gu
,
Xiwen Cai
,
Tianyi Lao
,
Wenlong Chen
,
Yingcai Wu
,
Jinhui Yu
,
Wei Chen
PDF
Cite
ScatterNet: A Deep Subjective Similarity Model for Visual Analysis of Scatterplots.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Similarity measuring methods are widely adopted in a broad range of visualization applications. In this work, we address the challenge …
Yuxin Ma
,
Anthony K. H. Tung
,
Wei Wang
,
Xiang Gao
,
Zhigeng Pan
,
Wei Chen
PDF
Cite
DOI
Cluster-based Visual Abstraction for Multivariate Scatterplots
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
The use of scatterplots is an important method for multivariate data visualization. The point distribution on the scatterplot, along …
Hongsen Liao
,
Yingcai Wu
,
Li Chen
,
Wei Chen
PDF
Cite
DOI
Evaluating Multi-Dimensional Visualizations for Understanding Fuzzy Clusters.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Fuzzy clustering assigns a probability of membership for a datum to a cluster, which veritably reflects real-world clustering scenarios …
Ying Zhao
,
Feng Luo
,
Minghui Chen
,
Yingchao Wang
,
Jiazhi Xia
,
Fangfang Zhou
,
Yunhai Wang
,
Yi Chen
,
Wei Chen
PDF
Cite
DOI
GraphProtector: A Visual Interface for Employing and Assessing Multiple Privacy Preserving Graph Algorithms.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Analyzing social networks reveals the relationships between individuals and groups in the data. However, such analysis can also lead to …
Xumeng Wang
,
Wei Chen
,
Jia-Kai Chou
,
Chris Bryan
,
Huihua Guan
,
Wenlong Chen
,
Rusheng Pan
,
Kwan-Liu Ma
PDF
Cite
Slides
Video
Video
Visual Abstraction of Large Scale Geospatial Origin-Destination Movement Data.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
A variety of human movement datasets are represented in an Origin-Destination(OD) form, such as taxi trips, mobile phone locations, …
Zhiguang Zhou
,
Linhao Meng
,
Cheng Tang
,
Ying Zhao
,
Zhiyong Guo
,
Miaoxin Hu
,
Wei Chen
PDF
Cite
DOI
ForVizor: Visualizing Spatio-Temporal Team Formations in Soccer.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Regarded as a high-level tactic in soccer, a team formation assigns players different tasks and indicates their active regions on the …
Yingcai Wu
,
Xiao Xie
,
Jiachen Wang
,
Dazhen Deng
,
Hongye Liang
,
Hui Zhang
,
Shoubin Cheng
,
Wei Chen
PDF
Cite
Video
DOI
Structure-Based Suggestive Exploration: New Approach for Effective Exploration of Large Networks
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
When analyzing a visualized network, users need to explore different sections of the network to gain insight. However, effective …
Wei Chen
,
Fangzhou Guo
,
Dongming Han
,
Jiacheng Pan
,
Xiaotao Nie
,
Jiazhi Xia
,
Xiaolong Zhang
PDF
Cite
Slides
Video
Structuring Mobility Transition With an Adaptive Graph Representation.
IEEE TCSS
|
IEEE Transactions on Computational Social Systems
Modeling human mobility is a critical task in fields such as urban planning, ecology, and epidemiology. Given the current use of mobile …
Tianlong Gu
,
Minfeng Zhu
,
Wei Chen
,
Zhaosong Huang
,
Ross Maciejewski
,
Liang Chang
PDF
Cite
Video
An Efficient Direct Volume Rendering Algorithm for Dichromats.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Color vision deficiency (CVD) affects a high percentage of the population worldwide. When seeing a volume visualization result, persons …
Weifeng Chen
,
Wei Chen
,
Hujun Bao
PDF
Cite
Video
ECharts: A Declarative Framework for Rapid Construction of Web-based Visualization.
Visual Informatics
|
Visual Informatics
While there have been a dozen of authoring systems and programming toolkits for visual design and development, users who do not have …
Deqing Li
,
Honghui Mei
,
Yi Shen
,
Shuang Su
,
Wenli Zhang
,
Junting Wang
,
Ming Zu
,
Wei Chen
PDF
Cite
Efficient Volume Exploration Using the Gaussian Mixture Model.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
The multidimensional transfer function is a flexible and effective tool for exploring volume data. However, designing an appropriate …
Yunhai Wang
,
Wei Chen
,
Jian Zhang
,
Tingxing Dong
,
Guihua Shan
,
Xuebin Chi
PDF
Cite
Video
Image-Space Texture-Based Output-Coherent Surface Flow Visualization.
VIEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Image-space line integral convolution (LIC) is a popular scheme for visualizing surface vector fields due to its simplicity and high …
Jin Huang
,
Zherong Pan
,
Guoning Chen
,
Wei Chen
,
Hujun Bao
PDF
Cite
Perception-Based Transparency Optimization for Direct Volume Rendering.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
The semi-transparent nature of direct volume rendered images is useful to depict layered structures in a volume. However, obtaining a …
Ming-Yuen Chan
,
Yingcai Wu
,
JWai-Ho Mak
,
Wei Chen
,
Huamin Qu
PDF
Cite
Video
Video
Video
Video
Video
Video
RankExplorer: Visualization of Ranking Changes in Large Time Series Data.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
For many applications involving time series data, people are often interested in the changes of item values over time as well as their …
Conglei Shi
,
Weiwei Cui
,
Shixia Liu
,
Panpan Xu
,
Wei Chen
,
Huamin Qu
PDF
Cite
Visual Exploration of Air Quality Data with A Time-Correlation Partitioning Tree Based on Information Theory.
ACM TiiS
|
ACM Transactions on Interactive Intelligent Systems
Discovering the correlations among variables of air quality data is challenging, because the correlation time series are long-lasting, …
Fangzhou Guo
,
Tianlong Gu
,
Wei Chen
,
Feiran Wu
,
Qi Wang
,
Lei Shi
,
Huamin Qu
PDF
Cite
DOI
Relationlines: Visual Reasoning of Egocentric Relations from Heterogeneous Urban Data
ACM TIST
|
ACM Transactions on Intelligent Systems and Technology
The increased accessibility of urban sensor data and the popularity of social network applications is enabling the discovery of crowd …
Wei Chen
,
Jing Xia
,
Xumeng Wang
,
Yi Wang
,
Jun Chen
,
Tianlong Gu
PDF
Cite
LDSScanner: Exploratory Analysis of Low-Dimensional Structures in High-Dimensional Datasets
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Many approaches for analyzing a high-dimensional dataset assume that the dataset contains specific structures, e.g., clusters in linear …
Jiazhi Xia
,
Fenjin Ye
,
Wei Chen
,
Yusi Wang
,
Weifeng Chen
,
Yuxin Ma
,
Anthony K.H. Tung
PDF
Cite
Video
DOI
A Utility-aware Visual Approach for Anonymizing Multi-attribute Tabular Data.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Sharing data for public usage requires sanitization to prevent sensitive information from leaking. Previous studies have presented …
Xumeng Wang
,
Jia-Kai Chou
,
Wei Chen
,
Huihua Guan
,
Wenlong Chen
,
Tianyi Lao Kwan-Liu Ma
PDF
Cite
Video
Video
Adaptively Exploring Population Mobility Patterns in Flow Visualization
IEEE TITS
|
IEEE Transactions on Intelligent Transportation Systems
Thanks to the ubiquitous cell phone use, we have never been so close to uncover population mobility patterns in urban area. While some …
Fei Wang
,
Wei Chen
,
Ye Zhao
,
Tianyu Gu
,
Siyuan Gao
,
Hujun Bao
PDF
Cite
DOI
Visual Analytics of Mobility and Transportation: State of the Art and Further Research Directions.
IEEE TITS
|
IEEE Transactions on Intelligent Transportation Systems
Many cities and countries are now striving to create intelligent transportation systems that utilize the current abundance of …
Gennady Andrienko
,
Natalia Andrienko
,
Wei Chen
,
Ross Maciejewski
,
Ye Zhao
PDF
Cite
DOI
DimScanner: A Relation-based Visual Exploration Approach Towards Data Dimension Inspection.
IEEE VAST
|
2016 IEEE Conference on Visual Analytics Science and Technology (VAST)
Exploring multi-dimensional datasets can be cumbersome if data analysts have little knowledge about the data. Various dimension …
Jing Xia
,
Wei Chen
,
Yumeng Hou
,
Xinxin Huang
,
Wanqi Hu
,
David S.Ebert
PDF
Cite
Visualizing Rank Time Series of Wikipedia Top Viewed Pages.
IEEE CG&A
|
IEEE Computer Graphics and Applications
Visual clutter is a common challenge when visualizing large rank time series data. WikiTopReader, a reader of Wikipedia page rank, lets …
Jing Xia
,
Yumeng Hou
,
Victor Chen
,
Cheryl Qian
,
David S. Ebert
,
Wei Chen
PDF
Cite
DOI
ViDX: Visual Diagnostics of Assembly Line Performance in Smart Factories
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Visual analytics plays a key role in the era of connected industry (or industry 4.0, industrial internet) as modern machines and …
Panpan Xu
,
Honghui Mei
,
Liu Ren
,
Wei Chen
PDF
Cite
Video
Video
Video
DOI
GameFlow: Narrative Visualization of NBA Basketball Games
IEEE TMM
|
IEEE Transactions on Multimedia
Although basketball games have received broad attention, the forms of game reports and webcast are purely content-based cross-media: …
Wei Chen
,
Tianyi Lao
,
Jing Xia
,
Xinxin Huang
,
Biao Zhu
,
Wanqi Hu
,
Huihua Guan
PDF
Cite
SemanticTraj: A New Approach to Interacting with Massive Taxi Trajectories
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Massive taxi trajectory data is exploited for knowledge discovery in transportation and urban planning. Existing tools typically …
Shamal Al-Dohuki
,
Yingyu Wu
,
Farah Kamw
,
Jing Yang
,
Xin Li
,
Ye Zhao
,
Xinyue Ye
,
Wei Chen
,
Chao Ma
,
Fei Wang
PDF
Cite
Visual Analysis of Cloud Computing Performance Using Behavioral Lines.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Cloud computing is an essential technology to Big Data analytics and services. A cloud computing system is often comprised of a large …
Chris Muelder
,
Biao Zhu
,
Wei Chen
,
Hongxin Zhang
,
Kwan-Liu Ma
PDF
Cite
DOI
Big Data in Social Transportation.
IEEE TITS
|
IEEE Transactions on Intelligent Transportation Systems
Big data for social transportation brings us unprecedented opportunities for resolving transportation problems for which traditional …
Xinhu Zheng
,
Wei Chen
,
Pu Wang
,
Dayong Shen
,
Songhang Chen
,
Xiao Wang
,
Qingpeng Zhang
,
Liuqin Yang
PDF
Cite
DOI
A Visual Voting Framework for Weather Forecast Calibration.
IEEE SciVis
|
IEEE Scientific Visualization Conference
Numerical weather predictions have been widely used for weather forecasting. Many large meteorological centers are producing highly …
Hongsen Liao
,
Li Chen
,
Yingcai Wu
,
Yunhai Wang
,
Huizhang
,
Wei Chen
PDF
Cite
DOI
Uncertainty-Aware Multidimensional Ensemble Data Visualization and Exploration
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
This paper presents an efficient visualization and exploration approach for modeling and characterizing the relationships and …
Haidong Chen
,
Song Zhang
,
Wei Chen
,
Honghui Mei
,
Jiawei Zhang
,
Huamin Qu
PDF
Cite
Video
DOI
A Visual Reasoning Approach for Data-driven Transport Assessment on Urban Road.
IEEE VAST
|
IEEE Conference on Visual Analytics Science and Technology
Transport assessment plays a vital role in urban planning and traffic control, which are influenced by multi-faceted traffic factors …
Fei Wang
,
Wei Chen
,
Feiran Wu
,
Ye Zhao
,
Han Hong
,
Tianyu Gu
,
Long Wang
,
Ronghua Liang
,
Hujun Bao
PDF
Cite
Slides
Video
DOI
BoundarySeer: Visual Analysis of 2D Boundary Changes.
IEEE VAST
|
IEEE Conference on Visual Analytics Science and Technology
Boundary changes exist ubiquitously in our daily life. From the Antarctic ozone hole to the land desertification, and from the …
Wenchao Wu
,
Yixian Zheng
,
Huamin Qu
,
Wei Chen
,
Eduard Groeller
,
Lionei Ni
PDF
Cite
Video
DOI
Visual Abstraction and Exploration of Multi-class Scatterplots
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Scatterplots are widely used to visualize scatter dataset for exploring outliers, clusters, local trends, and correlations. Depicting …
Haidong Chen
,
Wei Chen
,
Honghui Mei
,
Zhiqi Liu
,
Kun Zhou
,
Weifeng Chen
,
Wentao Gu
,
Kwan-Liu Ma
PDF
Cite
Slides
Video
DOI
VAET: A Visual Analytics Approach for E-transactions Time-series.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Previous studies on E-transaction time-series have mainly focused on finding temporal trends of transaction behavior. Interesting …
Cong Xie
,
Wei Chen
,
Xinxin Hunag
,
Yueqi Hu
,
Scott Barlowe
,
Jing Yang
PDF
Cite
Slides
Video
DOI
Automatic Shader Simplification using Surface Signal Approximation.
ACM TOG
|
ACM Transactions on Graphics
In this paper, we present a new automatic shader simplification method using surface signal approximation. We regard the entire …
Rui Wang
,
Hujun Bao
,
Karla Bala
,
Xianjin Yang
,
Yazhen Yuan
,
Wei Chen
PDF
Cite
DOI
Real-Time Shape Illustration Using Laplacian Lines
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
This paper presents a novel object-space line drawing algorithm that can depict shapes with view-dependent feature lines in real time. …
Long Zhang
,
Ying He
,
Jiazhi Xia
,
Xuexiang Xie
,
Wei Chen
PDF
Cite
Video
Structuring Feature Space: A Non-Parametric Method for Volumetric Transfer Function Generation.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
The use of multi-dimensional transfer functions for direct volume rendering has been shown to be an effective means of extracting …
Ross Maciejewski
,
Insoo Wu
,
Wei Chen
,
David S. Ebert
PDF
Cite
Video
DOI
A Novel Interface for Interactive Exploration of DTI Fibers.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Visual exploration is essential to the visualization and analysis of densely sampled 3D DTI fibers in biological speciments, due to the …
Wei Chen
,
Zi’ang Ding
,
Song Zhang
,
Anna MacKay-Brandt
,
Stephen Correia
,
Huamin Qu
,
John Allen Crow
,
David F. Tate
,
Zhicheng Yan
,
Qunsheng Peng
PDF
Cite
Video
Video
Video
Video
Video
Video
DOI
Volume Illustration of Muscle from Diffusion Tensor Images.
IEEE TVCG
|
IEEE Transactions on Visualization and Computer Graphics
Medical illustration has demonstrated its effectiveness to depict salient anatomical features while hiding the irrelevant details. …
Wei Chen
,
Zhicheng Yan
,
Song Zhang
,
John Allen Crow
,
David S. Ebert
,
R. McLaughlin
,
K. Mullins
,
R. Cooper
,
Zi’ang Ding
,
Jun Liao
PDF
Cite
Video
Hardware Accelerated Adaptive EWA Volume Splatting
IEEE VIS
|
IEEE Visualization
We present a hardware-accelerated adaptive EWA (elliptical weighted average) volume splatting algorithm. EWA splatting combines a …
Wei Chen
,
Liu Ren
,
Matthias Zwicker
,
Hanspter Pfister
PDF
Cite
Video
Video
DOI
Cite
×


Asks to keep RForm above (if set to true), so that it stays on top.
Note that you shouldn't assume the RForm is definitely above afterward, because other entities (e.g. the user or window manager) could not keep it above, and not all window managers support keeping windows above. But normally the window will end kept above. Just don't write code that crashes if not.
It's permitted to call this function before showing a window, in which case the window will be kept above when it appears onscreen initially.
|


| Version | Editor | Date |
| 1.0 - 1.14 | Paul Resnick, University of Michigan |
last revised 04-Aug-99 |
Status of This DocumentThis Frequently Asked Question (FAQ) white paper answers common questions about PICS in the context of intellectual freedom and censorship. It has no techincal standing what-so-ever, such information is found in the specifications. This document has no official W3C policy standing beyond describing the position the W3C has taken to date. Please send questions to<EMAIL_ADDRESS>. |
The published articles describing PICS (Communications of the ACM, Scientific American) have focused on individual controls over the materials that are received on a computer. While those articles also mention the possibility of more centralized controls (e.g., by employers or governments), they describe only briefly the technical details and the intellectual freedom implications of such centralized controls. The civil liberties community has raised some alarms about those intellectual freedom implications. The goals for this Frequently Asked Questions (FAQ) document are to:
In 1995, policies were proposed in several countries, including the USA, to restrict the distribution of certain kinds of material over the Internet. In many but not all cases, protection of children was the stated goal for such policies (see, for example, CIEC: Citizens Internet Empowerment Coalition).
The focus on restricting inappropriate materials at their source is not well suited to the international nature of the Internet, where an information source may be in a different legal jurisdiction than the recipient. Moreover, materials may be legal and appropriate for some recipients but not others, so that any decision about whether to block at the source will be incorrect for some audiences.
PICS, the Platform for Internet Content Selection, is a set of technical specifications that facilitate recipient-centered controls on Internet content, rather than sender-centered controls. The following diagram illustrates recipient-centered controls:
Filtering software sits between a child (or any Internet user) and the available content. It allows access to some materials, and blocks access to other materials. Some filtering software directly analyzes content, typically looking for particular keywords. This FAQ, however, does not deal with that kind of software; it deals, instead, with filtering software that decides what to allow and what to block based on two information sources.
PICS was not the first technology based on the idea of recipient-centered controls. For example, SurfWatch was already on the market in the summer of 1995 when PICS development began. It is based on a particularly simple set of labels: a list of URLs to avoid. As another example, some firewalls that corporations had introduced for security purposes blocked access to certain IP addresses. PICS provides a set of technical specifications so that pieces of the picture could be provided by different entities, yet still work together.
The first and most important distinction that PICS introduced is a separation between labeling and filtering. A label describes the content of something. A filter makes the content inaccessible to some audience. While both labeling and filtering may introduce social concerns, the concerns are somewhat different. More generally, there are six roles that could all be filled by different entities:
PICS itself actually fills none of the six roles listed above! PICS is a set of technical specifications that makes it possible for these roles to be played by independent entities.
For example, RSACi and SafeSurf have each defined labeling vocabulary and criteria for rating. They each wrote down a vocabulary in a machine-readable format that PICS specifies. RSACi has four categories in its vocabulary, language, nudity, sex, and violence; SafeSurf has more categories. Because they write down their vocabularies in the PICS format, label distribution software (e.g., from IBM and Net Shepherd) and filtering software (e.g., from Microsoft, IBM, and others) can process labels based on those vocabularies. Even though RSACi and SafeSurf have each specified a labeling vocabulary and criteria for assigning labels, neither of them actually assigns labels: they leave it up to the authors of documents to apply to criteria to their own documents, or self-label as PICS documents call it. Other services, such as CyberPatrol and Net Shepherd, take on both of the first two roles, choosing the labeling vocabulary and employing people to actually assign labels. The PICSRules specification provides a common format for expressing filtering criteria, which makes it easy for one entity to set filtering criteria which are then installed and run by someone else. For example, a parent might, with one mouse click install filtering criteria suggested by some trusted organization, say a local church group, even though that organization provides neither rating labels nor the filtering software.
Yes. While the motivation for PICS was concern over children accessing inappropriate materials, it is a general "meta-data" system, meaning that labels can provide any kind of descriptive information about Internet materials. For example, a labeling vocabulary could indicate the literary quality of an item rather than its appropriateness for children. Most immediately, PICS labels could help in finding particularly desirable materials (see, for example, NetShepherd's label-informed Alta Vista search), and this is the main motivation for the ongoing work on a next generation label format that can include arbitrary text strings. More generally, the W3C is working to extend Web meta-data capabilities generally and is applying them specifically in the following projects:
Regardless of content control, meta-data systems such as PICS are going to be an important part of the Web, because they enable more sophisticated commerce (build and manage trust relationships), communication, indexing, and searching services.
"The promise of digital commerce is that it will allow you to use the Internet to purchase the services of the best organic gardening advisors or mad cow disease specialists, whether they live in Santa Clara or Timbuktu. To do this, you need to do more than verify that the person at the other end of the wire is who he says he is. You need to assess competence, reliability, judgment. In other words, you need a system of branding, but applied much more widely for highly specialized and hard-to-evaluate services and products. You need value-added services that will not only lead you to the right product or service but also rate its quality or otherwise vouch for it."
Francis Fukayama
(Forbes ASAP 12/96 p 69)
This seemingly straightforward question, upon closer inspection, turns out to be many different questions when asked by different people. Many people are concerned about governments assuming one or more of the roles described in the answer to the previous question. Others are concerned about employers setting filtering rules, abuse of power by independent labelers, or a chilling effect on speech even if speech is not banned outright. People also employ different definitions of censorship. The most expansive definition is, "any action by one person that makes otherwise available information unavailable to another person." Under this expansive definition, even a parent setting filtering rules for a child would count as censorship. PICS documents have adopted the more restrictive definition of censorship as actions that limit what an individual can distribute, and use the term "access controls" for restrictions on what individuals can receive. But the distinction blurs if a central authority restricts access for a set of people. Finally, people have different definitions of "enable." Some would say that PICS enables any application that uses PICS-compatible components, while we reserve the term "enables" for applications that can easily be implemented with PICS-compatible components but could not be easily implemented otherwise.
Given the variety of implicit questions, it doesn't make sense to provide a blanket answer to the question of whether PICS enables censorship. This FAQ answers many of the specific questions that people often mean when they ask the more general question. For example, we ask questions about whether PICS makes it easier or harder for governments to impose labeling and filtering requirements. If you believe there's another specific question that should be addressed, please send it to<EMAIL_ADDRESS>for possible inclusion in a later version.
Yes. A government could try to assume any or all of the six roles described above, although some controls might be harder than others to enforce. As described below, governments could assume some of these roles even without PICS, while other roles would be harder to assume if PICS had not been introduced. It's important to note that W3C does not endorse any particular government policy. The purpose of this FAQ is to explain the range of potential policies and to explore some of the impacts of those policies on both the climate of intellectual freedom and the technical infrastructure of the World Wide Web. Potential government policies:
Yes; all other things being equal, when the price of something drops, more of it will be consumed.
Yes; for example, a national proxy-server/firewall combination that blocks access to a government-provided list of prohibited sites does not depend on interoperability of labels and filters provided by different organizations. While such a setup could use PICS-compatible technology, a proprietary technology provided by a single vendor would be just as effective. Other controls, based on individual or local choices, benefit more from mixing and matching filtering software and labels that come from different sources, which PICS enables. Thus, there should be some substitution of individual or local controls for centralized controls, although it is not obvious how strong this substitution effect will be. In Europe initial calls for centralized controls gave way to government reports calling for greater reliance on individual recipient controls; the end results of these political processes, however, are yet to be determined.
An IP address identifies the location of a computer on the Internet. A URL identifies the location of a document. To simplify a little, a URL has the form http://<domain-name>/<filename>. A web browser first resolves (translates) the domain-name into an IP address. It then contacts the computer at that address and asks it to send the particular filename. Thus, a label that applies to an IP address is a very broad label: it applies to every document that can be retrieved from that machine. Labeling of URLs permits more flexibility: different documents or directories of documents can be given different labels.
This difference of granularity will, naturally, have an impact on filtering. Filters based on IP addresses will be cruder: if some but not all of the documents available at a particular IP address are undesirable, the filter will have to either block all or none of those documents. PICS, by contrast, permits labeling of individual URLs, and hence permits finer grain filters as well.
Yes. Without a common format for labels, authors could not label themselves in a way that filtering programs could make use of. PICS provides that format.
It enables such a requirement to have more impact. A government requirement of self-labeling would have little impact if the labels were not usable by filtering programs. PICS provides the common format so that filtering software from one source can use labels provided by other sources (authors in this case).
Although universal agreement is not necessary, there does need to be some harmonization of vocabulary and labeling criteria, so that labels provided by different authors can be meaningfully compared.
Yes. PICS provides a language-independent format for expressing labels. If governments agreed on a common set of criteria for assigning labels, the criteria could be expressed in multiple languages, yet still be used to generate labels that can be compared to each other.
Both kinds of labeling could be effective, but only if a high percentage of the materials of a particular type are labeled. If the inappropriate materials are labeled, then a filter can block access to the labeled items. If the appropriate materials are labeled, then a filter can block access to all the unlabeled items.
Yes. Anyone can create a PICS label that describes any URL, and then distribute that label to anyone who wants to use that label. This is analogous to someone publishing a review of your web site in a newspaper or magazine.
If a lot of people use a particular organization's labels for filtering, that organization will indeed wield a lot of power. Such an organization could, for example, arbitrarily assign negative labels to materials from its commercial or political competitors. The most effective way to combat this danger is to carefully monitor the practices of labeling services, and to ensure diversity in the marketplace for such services, so that consumers can stop using services that abuse their power.
PICS documents use the term "label" broadly to refer to any machine-readable information that describes other information. Even information that merely classifies materials by topic or author (traditional card catalog information) would qualify as labels if expressed in a machine-readable format. The PICS developers recognized that the term "label" has a narrower meaning, with negative connotations, for librarians and some other audiences, but it was the most generic term the PICS creators could find without reverting to technical jargon like "metadata."
In media with centralized distribution channels, such as movies, labeling and filtering are not easily separated. For example, unrated movies are simply not shown in many theaters in the USA. In addition to its technical contribution, PICS makes an intellectual contribution by more clearly separating the ideas of labeling and filtering. Many of the negative connotations associated with "labeling" really should be associated with centralized filtering instead. There are, however, some subtle questions about the impact of labeling itself, as articulated in the next two questions.
Matt Blaze (personal communication) describes this concern with an analogy to discussions at local school board meeting about books to be read in a high school English class. Ideally, the discussion about a particular book should focus on the contents of the book, and not on the contents of a review of the book, or, worse yet, a label that says the book contains undesirable words.
There will always be a tradeoff, however, between speed of decision-making and the ability to take into account subtleties and context. When a large number of decisions need to be made in a short time, some will have to be made based on less than full information. The challenge for society, then, will be to choose carefully which decisions merit full discussion, in which case labels should be irrelevant, and which decisions can be left to the imperfect summary information that a label can provide. The following excerpt from Filtering the Internet summarizes this concern and the need for eternal vigilance:
"Another concern is that even without central censorship, any widely adopted vocabulary will encourage people to make lazy decisions that do not reflect their values. Today many parents who may not agree with the criteria used to assign movie ratings still forbid their children to see movies rated PG-13 or R; it is too hard for them to weigh the merits of each movie by themselves.
Labeling organizations must choose vocabularies carefully to match the criteria that most people care about, but even so, no single vocabulary can serve everyone's needs. Labels concerned only with rating the level of sexual content at a site will be of no use to someone concerned about hate speech. And no labeling system is a full substitute for a thorough and thoughtful evaluation: movie reviews in a newspaper can be far more enlightening than any set of predefined codes."
This is indeed a serious concern, explored in detail by Jonathan Weinberg in his law review article, Rating the Net. The following excerpt from Filtering the Internet acknowledges that materials of limited appeal may not reach even the audiences they would appeal to, but argues that labeling is merely a symptom rather than a cause of this underlying problem:
"Perhaps most troubling is the suggestion that any labeling system, no matter how well conceived and executed, will tend to stifle noncommercial communication. Labeling requires human time and energy; many sites of limited interest will probably go unlabeled. Because of safety concerns, some people will block access to materials that are unlabeled or whose labels are untrusted. For such people, the Internet will function more like broadcasting, providing access only to sites with sufficient mass-market appeal to merit the cost of labeling.
While lamentable, this problem is an inherent one that is not caused by labeling. In any medium, people tend to avoid the unknown when there are risks involved, and it is far easier to get information about material that is of wide interest than about items that appeal to a small audience."
PICSRules is a language for expressing filtering rules (profiles) that allow or block access to URLs based on PICS labels that describe those URLs. The purposes for a common profile-specification language are:
Sharing and installation of profiles. Sophisticated profiles
may be difficult for end-users to specify, even through
well-crafted user interfaces. An organization can create a recommended
profile for children of a certain age. Users who
trust that organization can install the profile rather than specifying
one from scratch. It will be easy to change the active
profile on a single computer, or to carry a profile to a new computer.
Communication to agents, search engines, proxies, or other servers.
Servers of various kinds may wish to tailor
their output to better meet users' preferences, as expressed in a
profile. For example, a search service can return only
links that match a user's profile, which may specify criteria based on
quality, privacy, age suitability, or the safety of
downloadable code.
Portability betwen filtering products. The same profile will
work with any PICSRules-compatible product.
No, but an effective national firewall would make it possible for a government to impose PICS-based filtering rules (or non PICS-based filtering rules) on its citizens. A firewall partitions a network into two components and imposes rules about what information flow between the two components. The goal of a national firewall is to put all the computers in the country into one component, and all computers outside the country into the other component. This is difficult to do, especially if people deliberately try to find out connections (e.g., telephone lines) between computers inside the country and those outside the country. Given a successful partition, however, PICS could be used to implement the filtering rules for a firewall. In particular, the government could identify prohibited sites outside the country that people inside the country could not access; such a filtering could be implemented based on PICS-formatted labels or, without relying on PICS-compatible technology, with a simple list of prohibited URLs.
No. PICSRules can provide a way to express filtering preferences, but has no impact on the ability of a government to partition the computers inside a country from those outside the country.
ISP compliance with government prohibition lists is already practical, even without
PICS. It would also be possible to comply using PICS-based technologies. PICS does make it
easier for ISPs to comply with a government requirement to block access to sites labeled
by non-governmental entities (including those that are self-labeled by the authors of the
sites).
Governments can make such requirements with or without PICSRules. PICSRules does make it possible for governments to precisely state filtering requirements, and perhaps simplify ISP compliance with changing government requirements, if the ISP implements a general interpreter for the PICSRules language.
Yes. PICS enables mixing and matching of the five roles. In particular, a service provider could install and run filtering software on a proxy server, but allow individuals to choose what filtering rules will be executed for each account. AOL already offers a primitive version of this idea, not based on PICS; parents can turn the preset filtering rules on or off for each member of the family.
No. Governments could require the use of filters on clients. The city of Boston, for example, requires public schools to install a client-based filtering product on all computers with Internet access, and requires public libraries to install a client-based filtering product on all computers designated for children.
W3C leaves this question to the political and legal processes of each country. Some people argue that unrestricted access to information is a fundamental human rights question that transcends national sovereignty. W3C has not adopted that position.
W3C leaves this question to the political and legal processes of each country.
W3C's mission is to "realize the full potential of the Web." The following two points are taken from a talk by Jim Miller at the WWW6 conference:
Thus, for example, when discussing the CDA-type legislation with government officials in the U.S. or abroad, it is appropriate for W3C to point out that sender-based restrictions are not likely to be effective at keeping all materials of a particular kind away from children, and that there could be unintended consequences in terms of chilling free speech or keeping the Web from reaching its full potential as a medium for communication and cultural exchange. W3C does not, however, debate with government officials about their perceived policy requirements. For example, Germany has a policy requirement of restricting access to hate speech while the U.S. does not: W3C does not try to convince either country that the other country's choice of policy requirements is better.
Some people(see The Net Labeling Delusion) have criticized the following paragraph from the CACM article on PICS:
Not everyone needs to block reception of the same materials. Parents may not wish to expose their children to sexual or violentimages. Businesses may want to prevent their employees from visiting recreational sites during hours of peak network usage. Governments may want to restrict reception of materials that are legal in other countries but not in their own. The off button (or disconnecting from the entire Net) is too crude: there should be some way to block only the inappropriate material. Appropriateness, however, is neither an objective nor a universal measure. It depends on at least three factors:
1. The supervisor: parenting styles differ, as do philosophies of management and
government.
2. The recipient: whats appropriate for one fifteen year old may not be for an
eight-year-old, or even all fifteen-year-olds.
3. The context: a game or chat room that is appropriate to access at home may be
inappropriate at work or school.
The main point of this section is to underscore the fact that people disagree about what materials are appropriate in what contexts. This point is illustrated at several levels of granularity: invidual children, organizations, and governments. The critcism focuses on the mention of possible government blocking, which did not appear in an earlier draft of the paper. We believe the example about differences in laws between countries is useful in explaining why there is a need for flexible, receiver-based controls rather than the kind of sender-based controls (e.g., the CDA) that most policy discussions were focusing on at the time.
The objection to the use of this example rests on an argument that governments should never designate any content as illegal. That argument is not widely accepted (in the U.S., for example, "obscenity" laws have been deemed constitutional, even though the CDA's "indecency" provisions were not). A more widely held position is that governments should not restrict political materials as a means of controlling their citizens. W3C leaves discussions about which materials should be illegal in a particular country to the political realm rather than the technological realm. W3C does, however, point out to policy makers, however, that it's not necessary to make materials illegal if they are offensive to some people but not others: end-user controls are a more flexible method of handling such materials.
Licensing such a technology was not considered to be a feasible option during the time of the CDA. Not only would it have undercut the "neutrality" and appeal of the technology, the W3C then would have had to be in the position of determining who should and should not use it; this is not a role the W3C is competent to play.
Yes. W3C is pleased that IBM has introduced a proxy server that can filter based on PICS labels, and encourages the development of other PICS-compatible servers. As discussed above, filter processing can be centralized at a proxy server while still permitting individuals to choose the filtering rules.
In addition to acting in the political arena, it would probably be helpful to implement positive uses of labels, such as searching applications. It is surpassingly difficult for people unfamiliar with computers to imagine new applications. By building prototypes and demonstrating them, it may be possible to focus policy-makers' energies on those uses of technology that accord with your political values.


Options HUD screen
Enable when using the ZAP Desktop
When set the HUD will be injected into HTML responses when the ZAP Desktop is used. This defaults to true.
Enable when using ZAP in daemon mode
When set the HUD will be injected into HTML responses when ZAP is started in daemon or 'headless' mode.
This defaults to false as many people use ZAP for automation, injecting the HUD would not help and could cause
existing tests to break.
Enable anonymous telemetry
When set will send pings to a bit.ly URL when:
- The HUD is used
- The HUD tutorial first page is visited
- The HUD tutorial last page is visited
Each ping is sent at most once per ZAP run and the only information available to the ZAP team is the number
of pings made and the countries they have come from.
We get very little feedback and so information like this really helps us understand how ZAP is being used.
We have limited time to spend and so want to focus on features that people are actually using.
Show the HUD welcome screen when a browser is opened
When set the HUD Welcome screen, which includes a link to the tutorial, will be shown whenever a browser that is proxying through ZAP is opened.
The welcome screen includes an option to not show the screen again, if that is selected then this is the only
way to make the welcome screen appear again.
Enable the HUD only for URLs that are in scope
If selected then only URLs flagged as 'in scope' in ZAP will have the HUD injected into them.
The plan is to expose this option via the HUD, but until that time if the option is enabled then you will
need to add URLs to the scope via the ZAP Desktop.
When this option is selected then the HUD icon in the desktop toolbar will have a small 'target' overlay on the
bottom right hand side.
Enable on-domain messages
Allow messages to be sent from the target domain to the ZAP domain. This is needed for features such as showing the
number of hidden fields and displaying the alert associated with a specific field.
If you think that the target domain could be malicious then you can turn this option off.
The HUD does not trust any messages from the target domain but a malicious site could potentially do mildly annoying things such as
causing ZAP alerts to be displayed unexpectedly.
You will need to restart your browser after changing this setting in order for it to take effect.
Remove CSP from target pages
The HUD will not currently work if a strong CSP policy is used in target web sites.
By default ZAP will strip off the CSP, turning this option off may well cause the HUD to break.
Development mode
If set then various options are set that can be useful when debugging the HUD, including:
- Setting the client side debug level to DEBUG
- Sending the client side logging to the ZAP Output tab
- Adding the 'HUD Errors' tool which shows any errors logged
Base directory
This should not be changed at this time.
Allow unsafe-evals
This is currently required in order for the HUD to work.
In the future the plan is to only require this when actually developing the HUD.
Skip tutorial tasks
Setting this option allows you to view all of the tutorial pages without passing the tasks.
Reset tutorial tasks
Clicking on this button will reset the tasks so that you will be able to take them again.
See also
| |
HUD |
for an overview of the HUD add-on |


Florida Woman Gets 15 Days in Prison for Throwing Slushie Cup at Rep. Matt Gaetz
by Associated Press(PENSACOLA, Fla.) — A woman has been sentenced to 15 days in federal custody for throwing a sports drink at a Florida congressman who frequently appears on television supporting President Donald Trump.
Federal court records show that 35-year-old Amanda Kondrat’yev was sentenced last Monday in Pensacola. Her confinement is scheduled to begin Dec. 2.
Kondrat’yev pleaded guilty to assault in August. She had faced up to a year in prison.
Authorities say Kondrat’yev was part of a group protesting in June outside a town hall meeting attended by U.S. Rep. Matt Gaetz. The second-term Republican was leaving a coffee house when a cup struck him in the back. Several witnesses identified Kondrat’yev as the assailant, and the throw was caught on video.


For an optimal functioning of any system, we often need a mixture of two opposing qualities.
We need to be data driven, but we need to think out of the box too. We need to listen to the users, but we also need to make things that would benefit the users, even if they do not say they want it.
In technology, we need to invest time in learning new ways doing old things in more efficient ways, but at times we just need to get job done in the most dependable way.
When I was an intern at Intel, Shekhar Borkar, who was the head of our reasearch wing, said a very interesting thing in a meeting with his team (which I imagine were more than hundred in number). He said that overall, he would like one third of his team's research work to succeed and other two thirds to fail. If all the projects were succeeding, that means the team was not taking enough risks. If all the projects were failing, that means the research team is working on unrealistic problems. The right proportion, for his research group, was that one third of the projects should succed. For production teams, perhaps more projects should succeed. I think for production teams, one in four projects should fail.
Long ago I read the book "The selfish gene" by Richard Dawkins, which had a significant impact on the way I think. One of the problems discussed in the book is the following.
Suppose, in a jungle, there is a population of jackals. Jackals, as you might know, feed on the food left over by predators like lions and tigers. Let's say there are two types of jackals: the hawkish jackals, which are combative and love to fight, and the dovish jackals, which are peace loving. If there is a piece of meat and two jackals approach it simultaneously, then what happens next depends on the tendencies of the jackals, whether they are hawkish or dovish.
If two hawkish jackals happen to approach the meat, then they fight over it. Ultimately, both of them get bruised quite badly, but one of them gets bruised more than the other and is forced to give up. The victor, though bruised, finally get the meat. If a hawkish and a dovish jackals happen to have a contest, there is no contest really. Hawkish jackal is ready to fight, and dovish jackal does not want a contest. Dovish jackal leaves the meat and goes away. If two dovish jackals meet, then they threaten each other, but no one actual hurts the other. Finally, one of the jackals gets bored and leaves the other to eat the meat.
If you are to be born in a jungle, is it better to be a hawkish jackal, or a dovish jackal? That depends on the proportion of hawks and doves in the current population. For instance, if all the jackals in the jungle are hawks, it is beneficial for you to be dove: there is no point being a hawk getting nearly fatally bruised in every encounter, just for a piece of meat. On the other hand, if all the jackals in the jungle are doves, it is beneficial for you to be a hawk: you will win every encounter. In a realistic scenario, where there is a mix of hawks and doves, the correct answer would depend the exact payoffs in various scenarios. Perhaps, when two hawks meet, the payoff for the loser is -100 (since he is very bruised), and the payoff for the victor is -90 (he is also bruised, but gets the meet). When a hawk and a dove meet, the payoff is perhaps 10 for the hawk and 0 for the dove. When two doves meet, the payoff could be 5 for the winner (he gets the meat worth 10 units, but loses 5 units in posturing) and -5 for the loser (he also loses 5 units in posturing). Using above numbers, one can derive whether being a hawk or dove is more beneficial in a given situation.
Thus, in a jackal population that evolves naturally, there will be a mix of hawks and doves.
Is your organisation completely data driven? Then perhaps you need some creative thinkers. Or is it mostly driven by creative thinkers? They perhaps can get out of touch with reality and need a data driven approach.
Are some projects of your team or company speculative enought that they fail? If not, time to do something wild. Are most projects of your company failing? Perhaps then you are out of touch with market.


“High Court orders six Irish internet providers to block The Pirate Bay” - thejournal.ie
“Turkey blocks YouTube days after Twitter crackdown” - cnn.com
“Censorship worldwide”:


class: center, middle, inverse, title-slide # 2.4 — Costs of Production ## ECON 306 • Microeconomic Analysis • Spring 2021 ### Ryan Safner
Assistant Professor of Economics
<EMAIL_ADDRESS>ryansafner/microS21
microS21.classes.ryansafner.com
--- class: inverse # Outline ### [Opportunity Costs in Production](#6) ### [Costs in the Short Run](#20) ### [Costs in the Long Run](#42) --- # Recall: The Firm's Two Problems .pull-left[ .smallest[ 1st Stage: .hi[firm's profit maximization problem]: 1. **Choose:** .hi-blue[ < output >] 2. **In order to maximize:** .hi-green[< profits >] 2nd Stage: .hi[firm's cost minimization problem]: 1. **Choose:** .hi-blue[ < inputs >] 2. **In order to _minimize_:** .hi-green[< cost >] 3. **Subject to:** .hi-red[< producing the optimal output >] - Minimizing costs `\(\iff\)` maximizing profits ] ] .pull-right[ .center[ ![](../images/management.jpg) ] ] --- # A *Competitive* Market .pull-left[ - We assume (for now) the firm is in a .hi[competitive] industry: 1. Firms’ products are .hi-purple[perfect substitutes] 2. Firms are .hi-purple[“price-takers”], no one firm can affect the *market price* 3. Market .hi-purple[entry and exit are free].magenta[†] ] .pull-right[ .center[ ![](../images/commodities.png) ] ] .footnote[.magenta[†] Remember this feature. It turns out to be the most important feature that distinguishes different types of industries!] --- # Profit .pull-left[ - Recall that profit is is: `$$\pi=\underbrace{pq}_{revenues}-\underbrace{(wl+rk)}_{costs}$$` - We’ll first take a closer look at costs today, then at revenues - Next class we'll put them together to find `\(q^*\)` that maximizes `\(\pi\)` (the first stage problem) ] .pull-right[ .center[ ![](../images/profitloss2.jpg) ] ] --- class: inverse, center, middle # Opportunity Costs in Production --- # Costs in Economics are Opportunity Costs .pull-left[ - Remember, .hi[economic costs] are different from common conception of “cost” - .hi-purple[Accounting cost]: monetary cost - .hi[Economic cost]: value of next best alternative use of resources given up (i.e. .hi[opportunity cost]) ] .pull-right[ .center[ ![:scale 70%](../images/costrevenue.jpg) ![:scale 70%](../images/disappear.jpg) ] ] --- # Costs in Economics are Opportunity Costs .pull-left[ - This leads to the difference between - .hi-purple[Accounting profit]: revenues minus accounting costs - .hi-purple[Economic profit]: revenues minues *opportunity* costs - One of the most difficult concepts to think about! ] .pull-right[ .center[ ![:scale 70%](../images/costrevenue.jpg) ![:scale 70%](../images/disappear.jpg) ] ] --- # Costs in Economics are Opportunity Costs .pull-left[ - Another helpful perspective: - .hi-purple[Accounting cost]: what you **historically** paid for a resource - .hi[Economic cost]: what you can **currently** get in the market for a selling a resource - Resource's value in *alternative* uses ] .pull-right[ .center[ ![:scale 70%](../images/costrevenue.jpg) ![:scale 70%](../images/disappear.jpg) ] ] --- # Costs in Economics are Opportunity Costs .pull-left[ - Because resources are scarce, and have rivalrous uses, .hi-turquoise[how do we know we are using resources efficiently??] - In functioning markets, .hi-purple[the market price measures the opportunity cost of using a resource for an alternative use] - Firms not only pay for direct use of a resource, but also indirectly compensate society for *"pulling the resource out"* of alternate uses in the economy! ] .pull-right[ .center[ ![:scale 70%](../images/costrevenue.jpg) ![:scale 70%](../images/disappear.jpg) ] ] --- # Opportunity Costs in Production .pull-left[ .center[ ![](../images/disappear.jpg) ] ] .pull-right[ - Every choice incurs an opportunity cost .bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[ .green[**Examples**]: .smallest[ - If you start a business, you may give up your salary at your current job - If you invest in a factory, you give up other investment opportunities - If you use an office building you own, you cannot rent it to other people - If you hire a skilled worker, you must pay them a high enough salary to deter them from working for other firms ] ] ] --- # Opportunity Cost is Hard for People .center[ ![](../images/oppcostpizza.jpg) ] --- # Opportunity Costs vs. Sunk Costs .pull-left[ - Opportunity cost is a *forward-looking* concept - Choices made in the *past* with *non-recoverable* costs are called .hi[sunk costs] - Sunk costs *should not* enter into future decisions - Many people have difficulty letting go of unchangeable past decisions: .hi-purple[sunk cost fallacy] ] .pull-left[ .center[ ![](../images/sunkcost.jpg) ] ] --- # Sunk Costs: Examples .pull-left[ .center[ ![:scale 50%](../images/sharknado.jpg) ] ] -- .pull-right[ .center[ ![](../images/paperwritingfrustration.jpg) ] ] --- # Sunks Costs: Examples .center[ ![:scale 70%](../images/concorde.jpg) ] --- # Common Sunk Costs in Business .pull-left[ - Licensing fees, long-term lease contracts - Specific capital (with no alternative use): uniforms, menus, signs - Research & Development spending - Advertising spending ] .pull-right[ .center[ ![](../images/sunkcost.jpg) ] ] --- # The Accounting vs. Economic Point of View I .pull-left[ - Helpful to consider two points of view: 1. .hi-purple[“Accounting point of view”]: are you taking in more cash than you are spending? 2. .hi-purple[“Economic point of view”]: is your product you making the *best social* use of your resources - i.e. are there higher-valued uses of your resources you are keeping them away from? ] .pull-right[ .center[ ![:scale 70%](../images/costrevenue.jpg) ![:scale 70%](../images/disappear.jpg) ] ] --- # The Accounting vs. Economic Point of View II .pull-left[ - .hi-turquoise[Social implications]: are consumers *best* off with you using scarce resources (with alternative uses!) to produce your current product? - Remember: .hi-turquoise[this is an *economics* course, not a *business* course]! - What might be good/bad for one business might have bad/good *consequences* for society! - e.g. monopoly vs. competition ] .pull-right[ .center[ ![:scale 70%](../images/costrevenue.jpg) ![:scale 70%](../images/disappear.jpg) ] ] --- class: inverse, center, middle # Costs in the Short Run --- # Costs in the Short Run - .hi[Total cost function, `\\(C(q)\\)`] relates output `\(q\)` to the total cost of production `\(C\)` `$$C(q)=f+VC(q)$$` -- - Two kinds of short run costs: **1.** .hi[Fixed costs, `\\(f\\)`] are costs that do not vary with output - Only true in the short run! (Consider this the cost of maintaining your capital) -- **2.** .hi[Variable costs, `\\(VC(q)\\)`] are costs that vary with output (notice the variable in them!) - Typically, the more production of `\(q\)`, the higher the cost - e.g. firm is hiring *additional* labor --- # Fixed vs. Variable costs: Examples .pull-left[ .center[ ![](../images/airline.jpg) ] ] .pull-right[ .bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[.hi-green[Example]: Airlines **Fixed costs**: the aircraft **Variable costs**: getting one more customer in a seat ] ] --- # Fixed vs. Variable costs: Examples .pull-left[ .center[ ![:scale 100%](../images/carfactory.jpg) ] ] .pull-right[ .bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[.hi-green[Example]: Car Factory **Fixed costs**: the factory, machines in the factory **Variable costs**: producing one more car ] ] --- # Fixed vs. Variable costs: Examples .pull-left[ .center[ ![](../images/starbuckscounter.jpg) ] ] .pull-right[ .bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[.hi-green[Example]: Starbucks **Fixed costs**: the retail space **Variable costs**: selling one more cup of coffee ] ] --- # Fixed vs. Sunk costs .pull-left[ .smallest[ - Diff. between .hi[fixed] vs. .hi-purple[sunk] costs? - .hi-purple[Sunk costs] are a *type* of .hi[fixed cost] that are *not* avoidable or recoverable - Many .hi[fixed costs] can be avoided or changed in the long run - Common .hi[fixed], but *not* .hi-purple[sunk], costs: - rent for office space, durable equipment, operating permits (that are renewed) - When deciding to *stay* in business, .hi[fixed costs] matter, .hi-purple[sunk costs] do not! ] ] .pull-right[ .center[ ![](../images/sunkcost.jpg) ] ] --- # Cost Functions: Example .bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[ .green[**Example**]: Suppose your firm has the following total cost function: `$$C(q)=q^2+q+10$$` ] 1. Write a function for the fixed costs, `\(f\)`. 2. Write a function for the variable costs, `\(VC(q)\)`. --- # Cost Functions: Example, Visualized .pull-left[ .tiny[ | `\(q\)` | `\(f\)` | `\(VC(q)\)` | `\(C(q)\)` | |----:|----:|--------:|-------:| | `\(0\)` | `\(10\)` | `\(0\)` | `\(10\)` | | `\(1\)` | `\(10\)` | `\(2\)` | `\(12\)` | | `\(2\)` | `\(10\)` | `\(6\)` | `\(16\)` | | `\(3\)` | `\(10\)` | `\(12\)` | `\(22\)` | | `\(4\)` | `\(10\)` | `\(20\)` | `\(30\)` | | `\(5\)` | `\(10\)` | `\(30\)` | `\(40\)` | | `\(6\)` | `\(10\)` | `\(42\)` | `\(52\)` | | `\(7\)` | `\(10\)` | `\(56\)` | `\(66\)` | | `\(8\)` | `\(10\)` | `\(72\)` | `\(82\)` | | `\(9\)` | `\(10\)` | `\(90\)` | `\(100\)` | | `\(10\)` | `\(10\)` | `\(110\)` | `\(120\)` | ] ] .pull-right[  ] --- # Average Costs - .hi[Average Fixed Cost]: fixed cost per unit of output: `$$AFC(q)=\frac{f}{q}$$` -- - .hi[Average Variable Cost]: variable cost per unit of output: `$$AVC(q)=\frac{VC(q)}{q}$$` -- - .hi[Average (Total) Cost]: (total) cost per unit of output: `$$AC(q)=\frac{C(q)}{q}$$` --- # Marginal Cost - .hi[Marginal Cost] is the change in cost for each additional unit of output produced: `$$MC(q) = \frac{\Delta C(q)}{\Delta q} \approx \frac{C_2-C_1}{q_2-q_1}$$` - Calculus: first derivative of the cost function - .hi-purple[Marginal cost is the *primary* cost that matters in making decisions] - All other costs are driven by marginal costs - This is the main cost that firms can "see" --- # The Importance of Marginal Cost .center[ ![](../images/qinrebellion.jpg) [Dazexiang Rebellion against the Qin Dynasty (209 B.C.)](https://en.wikipedia.org/wiki/Dazexiang_uprising) ] --- # Average and Marginal Costs: Example .bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[ .smallest[ .green[**Example**]: A small farm grows strawberries on 5 acres of land that it rents for $200 a week. The farm can hire workers at a wage of $250/week for each worker. The table below shows how the output of strawberries (in truckloads) varies with the number of workers hired:] .pull-left[ .smaller[ | Output | Labor | |--------|------:| | 0 | 0 | | 1 | 1 | | 2 | 3 | | 3 | 7 | | 4 | 12 | | 5 | 18 | ] ] ] .pull-right[ 1. If labor is the only variable cost, calculate the `\(MC(q)\)` and `\(AC(q)\)` for each of the first 5 truckloads. ] --- # Average and Marginal Costs: Visualized .pull-left[ .tiny[ | `\(q\)` | `\(C(q)\)` | `\(MC(q)\)` | `\(AFC(q)\)` | `\(AVC(q)\)` | `\(AC(q)\)` | |----:|----:|--------:|-------:| | `\(0\)` | `\(10\)` | `\(-\)` | `\(-\)` | `\(-\)` | `\(-\)` | | `\(1\)` | `\(12\)` | `\(2\)` | `\(10.00\)` | `\(2\)` | `\(12.00\)` | | `\(2\)` | `\(16\)` | `\(4\)` | `\(5.00\)` | `\(3\)` | `\(8.00\)` | | `\(3\)` | `\(22\)` | `\(6\)` | `\(3.33\)` | `\(4\)` | `\(7.30\)` | | `\(4\)` | `\(30\)` | `\(8\)` | `\(2.50\)` | `\(5\)` | `\(7.50\)` | | `\(5\)` | `\(40\)` | `\(10\)` | `\(2.00\)` | `\(6\)` | `\(8.00\)` | | `\(6\)` | `\(52\)` | `\(12\)` | `\(1.67\)` | `\(7\)` | `\(8.70\)` | | `\(7\)` | `\(66\)` | `\(14\)` | `\(1.43\)` | `\(8\)` | `\(9.40\)` | | `\(8\)` | `\(82\)` | `\(16\)` | `\(1.25\)` | `\(9\)` | `\(10.25\)` | | `\(9\)` | `\(100\)` | `\(18\)` | `\(1.11\)` | `\(10\)` | `\(11.10\)` | | `\(10\)` | `\(120\)` | `\(20\)` | `\(1.00\)` | `\(11\)` | `\(12.00\)` | ] ] .pull-right[  ] --- # Relationship Between Marginal and Average .pull-left[ .smallest[ - Relationship between a marginal and an average value: - .red[marginal] `\(<\)` .orange[average], .orange[average] `\(\downarrow\)` ] ] .pull-right[  ] --- # Relationship Between Marginal and Average .pull-left[ .smallest[ - Relationship between a marginal and an average value: - .red[marginal] `\(<\)` .orange[average], .orange[average] `\(\downarrow\)` - .red[marginal] `\(>\)` .orange[average], .orange[average] `\(\uparrow\)` ] ] .pull-right[  ] --- # Relationship Between Marginal and Average .pull-left[ .smallest[ - Relationship between a marginal and an average value: - .red[marginal] `\(<\)` .orange[average], .orange[average] `\(\downarrow\)` - .red[marginal] `\(>\)` .orange[average], .orange[average] `\(\uparrow\)` - When .red[marginal] `\(=\)` .orange[average], .orange[average] is **maximized/minimized** - .hi-purple[When `\\(MC=AC\\)`, `\\(AC\\)` is at a *minimum*] - .hi-purple[When `\\(MC=AVC\\)`, `\\(AVC\\)` is at a *minimum*] - Economic importance (later): Break-even price and shut-down price ] ] .pull-right[  ] --- # Short Run Costs: Example .bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[ .green[**Example**:] Suppose a firm's cost structure is described by: `$$\begin{align*} C(q)&=15q^2+8q+45\\ MC(q)&=30q+8\\ \end{align*}$$` ] 1. Write expressions for the firm's **fixed costs**, **variable costs**, **average fixed costs**, **average variable costs**, and **average (total) costs**. 2. Find the minimum average (total) cost. 3. Find the minimum average variable cost. --- # Costs: Example: Visualized  --- class: inverse, center, middle # Costs in the Long Run --- # Costs in the Long Run .pull-left[ - .hi[Long run]: firm can change all factors of production & vary scale of production - .hi-purple[Long run average cost, LRAC(q)]: cost per unit of output when the firm can change *both* `\(l\)` and `\(k\)` to make more `\(q\)` - .hi-purple[Long run marginal cost, LRMC(q)]: change in long run total cost as the firm produce an additional unit of `\(q\)` (by changing *both* `\(l\)` and/or `\(k\)`) ] .pull-right[ .center[ ![](../images/chooselocations.jpg) ] ] --- # Average Cost in the Long Run .pull-left[ - .hi[Long run]: firm can choose `\(k\)` (factories, locations, etc) - Separate short run average cost (SRAC) curves for each amount of `\(k\)` potentially chosen ] .pull-right[  ] --- # Average Cost in the Long Run .pull-left[ - .hi[Long run]: firm can choose `\(k\)` (factories, locations, etc) - Separate short run average cost (SRAC) curves for each amount of `\(k\)` potentially chosen - .purple[Long run average cost (LRAC)] curve "envelopes" the lowest (optimal) parts of all the SRAC curves! .smallest[ > "Subject to producing the optimal amount of output, choose l and k to minimize cost" ] ] .pull-right[  ] --- # Long Run Costs & Scale Economies I .pull-left[ .center[ ![](../images/scaleup.png) ] ] .pull-right[ - Further properties about costs based on .hi[scale economies] of production: - .hi[Economies of scale]: costs fall with output - `\(AFC > AVC(q)\)` - .hi[Diseconomies of scale]: costs rise with output - `\(AFC < AVC(q)\)` - .hi[Constant economies of scale]: costs don't change with output - Firm at minimum average cost ] --- # Long Run Costs & Scale Economies I .pull-left[ .center[ ![](../images/scaleup.png) ] ] .pull-right[ - Note .hi[economies of scale] `\(\neq\)` .hi-purple[returns to scale]! - .hi-purple[Returns to Scale] (last class): a **technological** relationship between inputs & output - .hi[Economies of Scale] (this class): an **economic** relationship between output and average costs ] --- # Long Run Costs & Scale Economies II .pull-left[ - .hi[Minimum Efficient Scale]: `\(q\)` with the lowest `\(AC(q)\)` - .hi-green[Economies of Scale]: `\(\uparrow q\)`, `\(\downarrow AC(q)\)` - .hi-red[Diseconomies of Scale]: `\(\uparrow q\)`, `\(\uparrow AC(q)\)` ] .pull-right[  ] --- # Long Run Costs and Scale Economies: Example .bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt5[ .green[**Example**]: A firm's long run cost structure is as follows: `$$\begin{align*} LRC(q)&= 32000q-250q^2+q^3\\ LRMC(q)&=32000-500q+3q^2\\ \end{align*}$$` ] 1. At what levels of output will the firm face economies of scale and diseconomies of scale? (Hint: This firm has a `\(U\)`-shaped LRAC.) --- # Long Run Costs and Scale Economies: Example


A simple comment form with submit validation and default messages
Name (required, at least 2 characters)
E-Mail (required)
URL (optional)
Your comment (required)
Remove focus handler


Disclaimer
This is a student project. The information on this website is intended to be educational, not diagnostic.
It is not a substitute for the expertise of a trained medical professional.


home
registry
event
wedding party
accommodations
photos
RSVP for You & Your Guest
Loading...
If you have any issues or questions please reach out directly to
rsvp@haasunkrich.com


N - Countries
Namibia
Nepal
Netherlands
New Caledonia
New Zealand
Nicaragua
Niger
Nigeria
Niue
North Korea
Northern Ireland
Northern Mariana Islands
Norway
Back


Sup title
Here you may describe the contents of the chart below in a few carefully crafted lines of text. Try your best not to be too wordy, though, as readers have short attention spans.
Data: Ask Jeeves
theglobalmail.org% if (projectType == 'webapp') { %> <% } %> <% if (includeModernizr && projectType == 'webapp') { %> <% } %> <% if (projectType == 'webapp') { %> <% } %>
<% if (projectType == 'webapp') { %>Here you may describe the contents of the chart below in a few carefully crafted lines of text. Try your best not to be too wordy, though, as readers have short attention spans.
Data: Ask Jeeves
theglobalmail.org


Innovative technological way to stop the Child Labour
Every one of us know what is the child labour, Most from us have seen one or more child labours at Tea Store, Vegetable shops, etc. In most of us, we worry about them, we donate them. Some of the active people from us would contact NGO and tell information about these child. These are the normal senarioes. If every people just send the information about child to various NGOs nearby the area. Atleast some of them would be saved. We should worry more about the childerns which are focefully bonded in various Factories. It incudes suger mills, Automotive industry and Water purifying and pacakging plant.
We need to Stop this and Everyone should contribute for it.
SCLApp - An app that can submit child victim's information to nearest NGOs. The NGOs which are not aware about the child, They will help and respond by their way. So, You need to just take a Photo of the Victim, You can add information about the victim and the Pricise location where NGO can reach and take action on their own.
Your little help can save a Child. And this help is just sharing the information about child victim to NGO by SCLApp
You can download the application and Keep it in your phone.
Simple and Stright way to submit Child labour information
Main page
First two steps to register child labout information
Last two steps to register child labout information
Child photo and information is attched to mail and send automatically.


{% extends 'base.html' %}
{% block content %}
{{ post.title }}
Comment {{ forloop.counter }} by {{ comment.name }} | {{ comment.created }}
{{ comment.body|linebreaks }}There are no comments yet. Be the first to comment!
{% endfor %}Your comment has been added.
{% else %} {% if user.is_authenticated %}You need to be logged in to comment!
{% endif %} {% endif %} {% endblock content %}


Salmon Cookie Sales Data
HOME
NEW STORE
Input New Store
Name
Minimum Number of Customers per Hour
Maximum Number of Customers per Hour
Average Number of Cookies Per Customer
Submit


HaikuYou
About
Compose
Read
Add Your Haiku
Title
5-Syllables
Count:
7-Syllables
Count:
5-Syllables
Count:
Your Pen Name
Generate Random Name
Poem Category
Select One
Action
Humor
Nature
Romantic
NSFW
Submit
User Submitted Haiku


@@include('./components/head.html')
@@include('./components/header.html')
Homepage
This is the homepage content
@@include('./components/footer.html')


DSL overview¶
DSL stands for domain-specific language: it's language particular to Miller which you can use to write expressions to specify customer transformations to your data. (See Miller programming language for an introduction.)
Verbs compared to DSL¶
While put
and filter
are verbs, they're different
from the rest in that they let you use the DSL -- so we often contrast DSL
(things you can do in the put
and filter
verbs), and verbs (things you
can do using the other verbs besides put
and filter
.)
Here's comparison of verbs and put
/filter
DSL expressions:
Example:
mlr stats1 -a sum -f x -g a data/small
a=pan,x_sum=0.346791 a=eks,x_sum=1.140078 a=wye,x_sum=0.777891
- Verbs are coded in Go
- They run a bit faster
- They take fewer keystrokes
- There is less to learn
- Their customization is limited to each verb's options
Example:
mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small
a=pan,x_sum=0.346791 a=eks,x_sum=1.140078 a=wye,x_sum=0.777891
- You get to write your own DSL expressions
- They run a bit slower
- They take more keystrokes
- There is more to learn
- They are highly customizable
Please see Verbs Reference for information on verbs other than put
and filter
.
Implicit loop over records for main statements¶
The most important point about the Miller DSL is that it is designed for streaming operation over records.
DSL statements include:
func
andsubr
for user-defined functions and subroutines, which we'll look at later in the separate page about them;begin
andend
blocks, for statements you want to run before the first record, or after the last one;- everything else, which collectively are called main statements.
The feature of streaming operation over records is implemented by the main statements getting invoked once per record. You don't explicitly loop over records, as you would in some dataframes contexts; rather, Miller loops over records for you, and it lets you specify what to do on each record: you write the body of the loop, not the loop itself.
(But you can, if you like, use those per-record statements to grow a list of
records, then loop over them all in an end
block. This is described in the
page on operating on all records.)
To see this in action, let's take a look at the data/short.csv file:
cat data/short.csv
word,value apple,37 ball,28 cat,54
There are three records in this file, with word=apple
, word=ball
, and
word=cat
, respectively. Let's print something in a begin
statement, add a
field in a main statement, and print something else in an end
statement:
mlr --csv --from data/short.csv put ' begin { print "begin"; } $nr = NR; end { print "end"; } '
begin word,value,nr apple,37,1 ball,28,2 cat,54,3 end
The print
statements for begin
and end
went out before the first record
was seen and after the last was seen; the field-creation statement $nr = NR
was invoked three times, once for each record. We didn't explicitly loop over
records, since Miller was already looping over records, and invoked our main
statement on each loop iteration.
For almost all simple uses of the Miller programming language, this implicit looping over records is probably all you will need. (For more involved cases you can see the pages on operating on all records, out-of-stream variables, and two-pass algorithms.)
Essential use: record-selection and record-updating¶
The essential usages of mlr filter
and mlr put
are for record-selection and
record-updating expressions, respectively. For example, given the following
input data:
cat data/small
a=pan,b=pan,i=1,x=0.346791,y=0.726802 a=eks,b=pan,i=2,x=0.758679,y=0.522151 a=wye,b=wye,i=3,x=0.204603,y=0.338318 a=eks,b=wye,i=4,x=0.381399,y=0.134188 a=wye,b=pan,i=5,x=0.573288,y=0.863624
you might retain only the records whose a
field has value eks
:
mlr filter '$a == "eks"' data/small
a=eks,b=pan,i=2,x=0.758679,y=0.522151 a=eks,b=wye,i=4,x=0.381399,y=0.134188
or you might add a new field which is a function of existing fields:
mlr put '$ab = $a . "_" . $b ' data/small
a=pan,b=pan,i=1,x=0.346791,y=0.726802,ab=pan_pan a=eks,b=pan,i=2,x=0.758679,y=0.522151,ab=eks_pan a=wye,b=wye,i=3,x=0.204603,y=0.338318,ab=wye_wye a=eks,b=wye,i=4,x=0.381399,y=0.134188,ab=eks_wye a=wye,b=pan,i=5,x=0.573288,y=0.863624,ab=wye_pan
Differences between put and filter¶
The two verbs mlr filter
and mlr put
are essentially the same. The only differences are:
-
Expressions sent to
mlr filter
should contain a boolean expression, which is the filtering criterion. (If not, all records pass through.) -
mlr filter
expressions may not reference thefilter
keyword within them.
Location of boolean expression for filter¶
You can define and invoke functions and subroutines to help produce the bare-boolean statement, and record fields may be assigned in the statements before or after the bare-boolean statement. For example:
mlr --c2p --from example.csv filter ' # Bare-boolean filter expression: only records matching this pass through: $quantity >= 70; # For records that do pass through, set these: if ($rate > 8) { $description = "high rate"; } else { $description = "low rate"; } '
color shape flag k index quantity rate description red square true 2 15 79.2778 0.0130 low rate red square false 4 48 77.5542 7.4670 low rate purple triangle false 5 51 81.2290 8.5910 high rate red square false 6 64 77.1991 9.5310 high rate purple triangle false 7 65 80.1405 5.8240 low rate purple square false 10 91 72.3735 8.2430 high rate
mlr --c2p --from example.csv filter ' # Bare-boolean filter expression: only records matching this pass through: $shape =~ "^(...)(...)$"; # For records that do pass through, capture the first "(...)" into $left and # the second "(...)" into $right $left = "\1"; $right = "\2"; '
color shape flag k index quantity rate left right red square true 2 15 79.2778 0.0130 squ are red circle true 3 16 13.8103 2.9010 cir cle red square false 4 48 77.5542 7.4670 squ are red square false 6 64 77.1991 9.5310 squ are yellow circle true 8 73 63.9785 4.2370 cir cle yellow circle true 9 87 63.5058 8.3350 cir cle purple square false 10 91 72.3735 8.2430 squ are
There are more details and more choices, of course, as detailed in the following sections.


[t]You can automatically enter a candidate into our system using the options below. First, enter the
candidate's e-mail address. Then, find the candidate's CV on your harddisk, using the file upload box. If we allready have the candidate
in our system, we will notify you. If now, we will extract the CV automatically, and you can check and correct this after.[/t]


SQL injection is one of the most classic hacks for SQL databases. Basically, hackers take advantage of weaknesses in SQL statements to manipulate parts of the database they're not meant to see. Since a lot of online databases include at least an email address (and in many cases, more sensitive information like credit card numbers), it can have pretty serious reprecussions.
This is an example of an unsecure SQL statement (from Wikipedia):
statement = "SELECT * FROM users WHERE name ='" + userName + "';"
It's pretty clear that the developer intended this to return all the information associated with a certain username (given by the user), but someone could insert "' OR '1'='1"
instead of a username to make this statement:
SELECT * FROM users WHERE name='' OR '1'='1';
Since '1' is always going to equal '1' (thanks, math), the above statement will always return "true." Consequently, the malicious user will recieve the entirety of the users table. In the same type of unsecure entry field, an attacker can insert a DROP TABLE
command, meaning that all the table information would be permanently lost.
Luckily, there are a few simple ways to avoid SQL injection:
With parameter binding, a developer can minimize user input. The variable used in the statement (like userName
, above) is modified into a placeholder that won't accept an arbitrary SQL statement. Instead, it only deals with a given, developer-controlled type of value. As a result, the malicious SQL code would be treated as incorrect input.
Pattern check is similar to parameter binding. It checks to see if the given input matches a certain type - if the input is supposed to be an integer, float, or boolean. Strings that are supposed to follow a strict pattern (like a date) can also be checked. Since the malicious code is unlikely to match the pretedermined pattern,
Escaping refers to making the SQL statement a "string literal" - for instance, by replacing all the single quotes with two single quotes. The malicious SQL code won't fit into the string literal SQL statement, and subsequently won't be executed. Escaping is more error-prone, however, and should be used in conjunction with something like parameter binding.
Finally, a developer could implement database permissions to reduce the effects of maliciously inserted SQL statements. Database permissions can mitigate which tables are accessible by certain statements, so while a SQL Inject statement would pass, it wouldn't affect all tables in one swoop.
These preventative measures are best used together. No one method is going to give you perfect security against malicious coders, but at least together they can minimize the effects!
8.30.2014


libs6dns
s6-dns
Software
skarnet.org
The following functions are declared in the s6-dns/s6dns-engine.h header, and implemented in the libs6dns.a or libs6dns.so library.
s6dns_engine is the nitty-gritty of DNS query management. These are the low-level asynchronous primitives sending DNS queries over the network, and getting answers.
s6dns_engine has been inspired by Dan J. Bernstein's dns_transmit library, but does not borrow any code from it. Unlike dns_transmit, s6dns_engine does not assume that network send operations are instant successes; s6dns_engine descriptors can be selected for writing as well as for reading. Also, if the underlying skalibs has been compiled with IPv6 support, s6dns_engine supports native IPv6 transport.
The s6dns_engine functions are used in the implementation of the s6dns_resolven_loop function - which is nothing more than a simple event loop around the s6dns_engine primitives - and the skadnsd daemon. Both pieces of code are good examples of how to use s6dns_engine.
However, unless you're implementing a DNS cache, you probably should not call the s6dns_engine functions directly: they are very low-level. If you need synchronous resolution, use the s6dns_resolve functions. If you need asynchronous DNS operations, use the skadns functions, which are designed to provide a higher level interface to multiple asynchronous DNS queries.
A s6dns_engine_t structure holds all the data necessary to manage a DNS query (and response). It must be initialized to S6DNS_ENGINE_ZERO when first declared, and recycled with s6dns_engine_recycle() after each use. It contains a stralloc, so it must be freed with s6dns_engine_free() before being discarded, to avoid memory leaks.
int s6dns_engine_init (s6dns_engine_t *dt, s6dns_ip46list_t const *servers, uint32_t options, char const *q, unsigned int qlen, uint16_t qtype, tain_t const *deadline, tain_t const *stamp)
Initializes dt with query q of length qlen and type qtype. If d is an encoded s6dns_domain_t, then d.s and d.len are appropriate candidates for arguments q and qlen respectively.
options can be 0 or an OR'ed combination of the following, defined in s6-dns/s6dns-constants.h:
servers must point to a list of IP addresses as defined in s6-dns/s6dns-ip46.h. Such a list can be obtained from the /etc/resolv.conf file via the s6dns_rci_fill() call when performing a recursive query, or it must be constructed from a list of relevant NS addresses when performing an iterative query.
stamp must be an accurate enough timestamp. deadline sets up a deadline for the query: if the query hasn't been satisfactorily answered by deadline - no matter how many round-trips to network servers the library performs internally - then it will be considered failed, and a timeout will be reported.
The function returns 1 if all went well, and 0 if an error occurred. It returns instantly; it does not perform any network operation, it just prepares dt to send a query. The actual data sending will take place on the next s6dns_engine_event() call.
void s6dns_engine_recycle (s6dns_engine_t *dt)
Recycles dt, making it ready for another use. This function does not deallocate the heap memory used by dt, so it's faster than s6dns_engine_free() and does not cause heap fragmentation.
void s6dns_engine_free (s6dns_engine_t *dt)
Frees the heap memory used by dt. Also makes dt ready for another use. It's advised to only use this function when certain that dt will not be reused.
The descriptor to select on is available as the fd field in the s6dns_engine_t structure. dt→fd should be read every iteration, because it can change between iterations even if no event or timeout is reported for dt.
void s6dns_engine_nextdeadline (s6dns_engine_t const *dt, tain_t *a)
If dt needs handling before the absolute date *a, then *a is updated so it contains the earlier date. This is useful to compute the next deadline in an iopause() loop.
int s6dns_engine_isreadable (s6dns_engine_t const *dt)
Returns nonzero iff dt→fd is to be selected for reading. Should be called in every iteration.
int s6dns_engine_iswritable (s6dns_engine_t const *dt)
Returns nonzero iff dt→fd is to be selected for writing. Should be called in every iteration.
int s6dns_engine_timeout (s6dns_engine_t *dt, tain_t const *stamp)
This function should be called if your selecting function returned 0, which means that an event timed out. stamp should contain the current time. The function returns -1 if an error occurred, 1 if dt actually timed out, and 0 if nothing special happened to dt (and your iopause timeout was caused by something else). If the return value is not 0, dt is automatically recycled.
int s6dns_engine_event (s6dns_engine_t *dt, tain_t const *stamp)
This function should be called if your selecting function returned a positive number, which means that some event got triggered. stamp should contain the current time. The function returns -1 if an error occurred (and dt is automatically recycled). It returns 0 if no answer has arrived yet, and 1 if an answer is available.
The s6dns_engine_timeout() and s6dns_engine_event() functions, when returning -1, make use of the following error codes:
char *s6dns_engine_packet (s6dns_engine_t const *dt)
Points to the response packet received from the network, if s6dns_engine_event() returned 1.
unsigned int s6dns_engine_packetlen (s6dns_engine_t const *dt)
Gives the length of the response packet, if s6dns_engine_event() returned 1.
You should recycle or free dt after reading the response packet.


Content Security Policy (CSP) is a new header for securing your web application against common scripting attacks. It works by limiting content sources a web page is allowed to access. For example, the header can declare that all content (images, scripts, frames) must originate from a single domain. The browser will block any requests that violate this policy.
CSP defaults to also blocking inline styles, inline Javascript, and the use of eval
in Javascript. These defaults have a high risk of breaking existing sites. Specifically, the Javascript code snippet that Google Analytics asks you to include on your website implementations break when using Content Security Policy.
Content Security Policy Header
CSP is implemented by returning the Content-Security-Policty
header to web requests. The header specifies the allowed domains for content sources. The default-src
applies to all unspecified content. The 'self'
value represents the domain which served this response.
The header below allows all content from the current domain, and allows scripts from one additional domain, https://ssl.google-analytics.com
.
Content-Security-Policy: "default-src 'self'; script-src 'self' https://ssl.google-analytics.com"
With this header in place, let’s see what happens when we load our page with a Google Analytics tracking snippet.
Uh-oh.
Javascript Changes
Google Analytics suggests you add the following snippet to all you pages.
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-22222222-1']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
This snippet will violate the above content policy as it includes inline Javascript. We could change our content policy to allow inline Javascript, but that would defeat the entire purpose of adding the header. Instead, we’ll break apart the above snippet into two script tags. First, create a new Javascript file named tracking.js
with the following content:
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-22222222-1']);
_gaq.push(['_trackPageview']);
Remove the Google Analytics tracking snippet and replace it with these two scripts.
<script type="text/javascript" src="/js/tracking.js"></script>
<script src="https://ssl.google-analytics.com/ga.js" async="true"></script>
Make sure that these scripts appear at the bottom of the page right before the closing body tag. Older browsers don’t honor the async
attribute, so you want to make sure it’s the last script to load.
Refresh the page, and you should see that Google Analytics is working as intended.
Please remember that CSP is just another tool to help secure your web application and does not replace proper HTML escaping. Browsers without CSP support need to be protected as well.
For a deeper look into CSP, the fine folks at GitHub wrote a fantastic post about their experience implementing CSP.


This is one of my first contributions / remakes to any of Smirf123's work. I hope somebody finds some use out of this at one point, thank you for reviewing! Please follow me on GitHub!


This is an experimental call graph visualiser. Calls are extracted at compile time. For this demonstration, the call graph is generated by passing "-fdump-rtl-expand" to GCC, and the subsequent RTL files generated are imported into a graph database, currently Neo4J. Importing calls to the database from other sources is possible.
The aims of this project are to make it easier to understand which functions and groups of functions are heavily used and the extent to which one piece of code depends on another. The first demonstration we have here does this by representing functions as blocks on the screen, with arrowed lines connecting them and external packages. The initial graph may look too dense to be readable, but you can highlight smaller parts of the call chain by clicking on function boxes. In all, you can interact with the graph in three ways:
Our next tool will be one that shows the dependency chains between two objects. The use of a graph database makes this easy to do.
We group functions in two ways: function are members of objects, which in most projects will mean a single source file. Objects are then members of packages. 'Package' is an arbitrary grouping, but we expect this to be analogous to the packages used by most Linux-based operating systems. In the example here, we've created artificial packages using the first letter of each object's name, so all objects beginning with 'a' are in one package.
The test data presented here is generated from NetHack 3.60.
Further details, including instructions on how to set up your own copy, can be found on Github: https://github.com/CodethinkLabs/software-dependency-visualizer. Please note that all of the project, including the documentation, is a work in progress and there are likely to be important steps missing.


Main Program
We can get any famous speaker from around the world to speak at your event.
We can get any influencer to hang out at your event and promote your event on their platform.
We are in contact with top artists and DJs. We can get big recording artists from all over the globe to play at your next event.
We have all aspects of your event covered. We can cater your event with anything from McDonalds to Michelin star meals.
We work with any venue that is right for the occation form small backyard affairs to Football stadims.
Main Program
Lecture
We can get any famous speaker from around the world to speak at your event.
Influencers
We can get any influencer to hang out at your event and promote your event on their platform.
Music
We are in contact with top artists and DJs. We can get big recording artists from all over the globe to play at your next event.
Catering
We have all aspects of your event covered. We can cater your event with anything from McDonalds to Michelin star meals
Venue
We work with any venue that is right for the occation form small backyard affairs to Football stadims.