PERF_EVENT_OPEN
Section: Linux Programmer's Manual (2)
Updated: 2014-08-19
Index
Return to Main Contents
NAME
perf_event_open - set up performance monitoring
SYNOPSIS
#include <linux/perf_event.h>
#include <linux/hw_breakpoint.h>
int perf_event_open(struct perf_event_attr *attr,
pid_t pid, int cpu, int group_fd,
unsigned long flags);
Note:
There is no glibc wrapper for this system call; see NOTES.
DESCRIPTION
Given a list of parameters,
perf_event_open()
returns a file descriptor, for use in subsequent system calls
(read(2), mmap(2), prctl(2), fcntl(2), etc.).
A call to
perf_event_open()
creates a file descriptor that allows measuring performance
information.
Each file descriptor corresponds to one
event that is measured; these can be grouped together
to measure multiple events simultaneously.
Events can be enabled and disabled in two ways: via
ioctl(2)
and via
prctl(2).
When an event is disabled it does not count or generate overflows but does
continue to exist and maintain its count value.
Events come in two flavors: counting and sampled.
A
counting
event is one that is used for counting the aggregate number of events
that occur.
In general, counting event results are gathered with a
read(2)
call.
A
sampling
event periodically writes measurements to a buffer that can then
be accessed via
mmap(2).
Arguments
The
pid
and
cpu
arguments allow specifying which process and CPU to monitor:
- pid == 0 and cpu == -1
-
This measures the calling process/thread on any CPU.
- pid == 0 and cpu >= 0
-
This measures the calling process/thread only
when running on the specified CPU.
- pid > 0 and cpu == -1
-
This measures the specified process/thread on any CPU.
- pid > 0 and cpu >= 0
-
This measures the specified process/thread only
when running on the specified CPU.
- pid == -1 and cpu >= 0
-
This measures all processes/threads on the specified CPU.
This requires
CAP_SYS_ADMIN
capability or a
/proc/sys/kernel/perf_event_paranoid
value of less than 1.
- pid == -1 and cpu == -1
-
This setting is invalid and will return an error.
The
group_fd
argument allows event groups to be created.
An event group has one event which is the group leader.
The leader is created first, with
group_fd = -1.
The rest of the group members are created with subsequent
perf_event_open()
calls with
group_fd
being set to the file descriptor of the group leader.
(A single event on its own is created with
group_fd = -1
and is considered to be a group with only 1 member.)
An event group is scheduled onto the CPU as a unit: it will
be put onto the CPU only if all of the events in the group can be put onto
the CPU.
This means that the values of the member events can be
meaningfully compared---added, divided (to get ratios), and so on---with each
other, since they have counted events for the same set of executed
instructions.
The
flags
argument is formed by ORing together zero or more of the following values:
- PERF_FLAG_FD_CLOEXEC (since Linux 3.14).
-
This flag enables the close-on-exec flag for the created
event file descriptor,
so that the file descriptor is automatically closed on
execve(2).
Setting the close-on-exec flags at creation time, rather than later with
fcntl(2),
avoids potential race conditions where the calling thread invokes
perf_event_open()
and
fcntl(2)
at the same time as another thread calls
fork(2)
then
execve(2).
- PERF_FLAG_FD_NO_GROUP
-
This flag allows creating an event as part of an event group but
having no group leader.
It is unclear why this is useful.
- PERF_FLAG_FD_OUTPUT
-
This flag reroutes the output from an event to the group leader.
- PERF_FLAG_PID_CGROUP (since Linux 2.6.39).
-
This flag activates per-container system-wide monitoring.
A container
is an abstraction that isolates a set of resources for finer-grained
control (CPUs, memory, etc.).
In this mode, the event is measured
only if the thread running on the monitored CPU belongs to the designated
container (cgroup).
The cgroup is identified by passing a file descriptor
opened on its directory in the cgroupfs filesystem.
For instance, if the
cgroup to monitor is called
test,
then a file descriptor opened on
/dev/cgroup/test
(assuming cgroupfs is mounted on
/dev/cgroup)
must be passed as the
pid
parameter.
cgroup monitoring is available only
for system-wide events and may therefore require extra permissions.
The
perf_event_attr
structure provides detailed configuration information
for the event being created.
struct perf_event_attr {
__u32 type; /* Type of event */
__u32 size; /* Size of attribute structure */
__u64 config; /* Type-specific configuration */
union {
__u64 sample_period; /* Period of sampling */
__u64 sample_freq; /* Frequency of sampling */
};
__u64 sample_type; /* Specifies values included in sample */
__u64 read_format; /* Specifies values returned in read */
__u64 disabled : 1, /* off by default */
inherit : 1, /* children inherit it */
pinned : 1, /* must always be on PMU */
exclusive : 1, /* only group on PMU */
exclude_user : 1, /* don't count user */
exclude_kernel : 1, /* don't count kernel */
exclude_hv : 1, /* don't count hypervisor */
exclude_idle : 1, /* don't count when idle */
mmap : 1, /* include mmap data */
comm : 1, /* include comm data */
freq : 1, /* use freq, not period */
inherit_stat : 1, /* per task counts */
enable_on_exec : 1, /* next exec enables */
task : 1, /* trace fork/exit */
watermark : 1, /* wakeup_watermark */
precise_ip : 2, /* skid constraint */
mmap_data : 1, /* non-exec mmap data */
sample_id_all : 1, /* sample_type all events */
exclude_host : 1, /* don't count in host */
exclude_guest : 1, /* don't count in guest */
exclude_callchain_kernel : 1,
/* exclude kernel callchains */
exclude_callchain_user : 1,
/* exclude user callchains */
mmap2 : 1, /* include mmap with inode data */
comm_exec : 1, /* flag comm events that are due to exec */
__reserved_1 : 39;
union {
__u32 wakeup_events; /* wakeup every n events */
__u32 wakeup_watermark; /* bytes before wakeup */
};
__u32 bp_type; /* breakpoint type */
union {
__u64 bp_addr; /* breakpoint address */
__u64 config1; /* extension of config */
};
union {
__u64 bp_len; /* breakpoint length */
__u64 config2; /* extension of config1 */
};
__u64 branch_sample_type; /* enum perf_branch_sample_type */
__u64 sample_regs_user; /* user regs to dump on samples */
__u32 sample_stack_user; /* size of stack to dump on
samples */
__u32 __reserved_2; /* Align to u64 */
};
The fields of the
perf_event_attr
structure are described in more detail below:
- type
-
This field specifies the overall event type.
It has one of the following values:
-
- PERF_TYPE_HARDWARE
-
This indicates one of the "generalized" hardware events provided
by the kernel.
See the
config
field definition for more details.
- PERF_TYPE_SOFTWARE
-
This indicates one of the software-defined events provided by the kernel
(even if no hardware support is available).
- PERF_TYPE_TRACEPOINT
-
This indicates a tracepoint
provided by the kernel tracepoint infrastructure.
- PERF_TYPE_HW_CACHE
-
This indicates a hardware cache event.
This has a special encoding, described in the
config
field definition.
- PERF_TYPE_RAW
-
This indicates a "raw" implementation-specific event in the
config field.
- PERF_TYPE_BREAKPOINT (since Linux 2.6.33)
-
This indicates a hardware breakpoint as provided by the CPU.
Breakpoints can be read/write accesses to an address as well as
execution of an instruction address.
- dynamic PMU
-
Since Linux 2.6.39,
perf_event_open()
can support multiple PMUs.
To enable this, a value exported by the kernel can be used in the
type
field to indicate which PMU to use.
The value to use can be found in the sysfs filesystem:
there is a subdirectory per PMU instance under
/sys/bus/event_source/devices.
In each subdirectory there is a
type
file whose content is an integer that can be used in the
type
field.
For instance,
/sys/bus/event_source/devices/cpu/type
contains the value for the core CPU PMU, which is usually 4.
- size
-
The size of the
perf_event_attr
structure for forward/backward compatibility.
Set this using
sizeof(struct perf_event_attr)
to allow the kernel to see
the struct size at the time of compilation.
The related define
PERF_ATTR_SIZE_VER0
is set to 64; this was the size of the first published struct.
PERF_ATTR_SIZE_VER1
is 72, corresponding to the addition of breakpoints in Linux 2.6.33.
PERF_ATTR_SIZE_VER2
is 80 corresponding to the addition of branch sampling in Linux 3.4.
PERF_ATR_SIZE_VER3
is 96 corresponding to the addition
of
sample_regs_user
and
sample_stack_user
in Linux 3.7.
- config
-
This specifies which event you want, in conjunction with
the
type
field.
The
config1 and config2
fields are also taken into account in cases where 64 bits is not
enough to fully specify the event.
The encoding of these fields are event dependent.
The most significant bit (bit 63) of
config
signifies CPU-specific (raw) counter configuration data;
if the most significant bit is unset, the next 7 bits are an event
type and the rest of the bits are the event identifier.
There are various ways to set the
config
field that are dependent on the value of the previously
described
type
field.
What follows are various possible settings for
config
separated out by
type.
If
type
is
PERF_TYPE_HARDWARE,
we are measuring one of the generalized hardware CPU events.
Not all of these are available on all platforms.
Set
config
to one of the following:
-
- PERF_COUNT_HW_CPU_CYCLES
-
Total cycles.
Be wary of what happens during CPU frequency scaling.
- PERF_COUNT_HW_INSTRUCTIONS
-
Retired instructions.
Be careful, these can be affected by various
issues, most notably hardware interrupt counts.
- PERF_COUNT_HW_CACHE_REFERENCES
-
Cache accesses.
Usually this indicates Last Level Cache accesses but this may
vary depending on your CPU.
This may include prefetches and coherency messages; again this
depends on the design of your CPU.
- PERF_COUNT_HW_CACHE_MISSES
-
Cache misses.
Usually this indicates Last Level Cache misses; this is intended to be
used in conjunction with the
PERF_COUNT_HW_CACHE_REFERENCES
event to calculate cache miss rates.
- PERF_COUNT_HW_BRANCH_INSTRUCTIONS
-
Retired branch instructions.
Prior to Linux 2.6.34, this used
the wrong event on AMD processors.
- PERF_COUNT_HW_BRANCH_MISSES
-
Mispredicted branch instructions.
- PERF_COUNT_HW_BUS_CYCLES
-
Bus cycles, which can be different from total cycles.
- PERF_COUNT_HW_STALLED_CYCLES_FRONTEND (since Linux 3.0)
-
Stalled cycles during issue.
- PERF_COUNT_HW_STALLED_CYCLES_BACKEND (since Linux 3.0)
-
Stalled cycles during retirement.
- PERF_COUNT_HW_REF_CPU_CYCLES (since Linux 3.3)
-
Total cycles; not affected by CPU frequency scaling.
-
If
type
is
PERF_TYPE_SOFTWARE,
we are measuring software events provided by the kernel.
Set
config
to one of the following:
-
- PERF_COUNT_SW_CPU_CLOCK
-
This reports the CPU clock, a high-resolution per-CPU timer.
- PERF_COUNT_SW_TASK_CLOCK
-
This reports a clock count specific to the task that is running.
- PERF_COUNT_SW_PAGE_FAULTS
-
This reports the number of page faults.
- PERF_COUNT_SW_CONTEXT_SWITCHES
-
This counts context switches.
Until Linux 2.6.34, these were all reported as user-space
events, after that they are reported as happening in the kernel.
- PERF_COUNT_SW_CPU_MIGRATIONS
-
This reports the number of times the process
has migrated to a new CPU.
- PERF_COUNT_SW_PAGE_FAULTS_MIN
-
This counts the number of minor page faults.
These did not require disk I/O to handle.
- PERF_COUNT_SW_PAGE_FAULTS_MAJ
-
This counts the number of major page faults.
These required disk I/O to handle.
- PERF_COUNT_SW_ALIGNMENT_FAULTS (since Linux 2.6.33)
-
This counts the number of alignment faults.
These happen when unaligned memory accesses happen; the kernel
can handle these but it reduces performance.
This happens only on some architectures (never on x86).
- PERF_COUNT_SW_EMULATION_FAULTS (since Linux 2.6.33)
-
This counts the number of emulation faults.
The kernel sometimes traps on unimplemented instructions
and emulates them for user space.
This can negatively impact performance.
- PERF_COUNT_SW_DUMMY (since Linux 3.12)
-
This is a placeholder event that counts nothing.
Informational sample record types such as mmap or comm
must be associated with an active event.
This dummy event allows gathering such records without requiring
a counting event.
-
If
type
is
PERF_TYPE_TRACEPOINT,
then we are measuring kernel tracepoints.
The value to use in
config
can be obtained from under debugfs
tracing/events/*/*/id
if ftrace is enabled in the kernel.
-
If
type
is
PERF_TYPE_HW_CACHE,
then we are measuring a hardware CPU cache event.
To calculate the appropriate
config
value use the following equation:
-
(perf_hw_cache_id) | (perf_hw_cache_op_id << 8) |
(perf_hw_cache_op_result_id << 16)
where
perf_hw_cache_id
is one of:
-
- PERF_COUNT_HW_CACHE_L1D
-
for measuring Level 1 Data Cache
- PERF_COUNT_HW_CACHE_L1I
-
for measuring Level 1 Instruction Cache
- PERF_COUNT_HW_CACHE_LL
-
for measuring Last-Level Cache
- PERF_COUNT_HW_CACHE_DTLB
-
for measuring the Data TLB
- PERF_COUNT_HW_CACHE_ITLB
-
for measuring the Instruction TLB
- PERF_COUNT_HW_CACHE_BPU
-
for measuring the branch prediction unit
- PERF_COUNT_HW_CACHE_NODE (since Linux 3.0)
-
for measuring local memory accesses
and
perf_hw_cache_op_id
is one of
-
- PERF_COUNT_HW_CACHE_OP_READ
-
for read accesses
- PERF_COUNT_HW_CACHE_OP_WRITE
-
for write accesses
- PERF_COUNT_HW_CACHE_OP_PREFETCH
-
for prefetch accesses
and
perf_hw_cache_op_result_id
is one of
-
- PERF_COUNT_HW_CACHE_RESULT_ACCESS
-
to measure accesses
- PERF_COUNT_HW_CACHE_RESULT_MISS
-
to measure misses
If
type
is
PERF_TYPE_RAW,
then a custom "raw"
config
value is needed.
Most CPUs support events that are not covered by the "generalized" events.
These are implementation defined; see your CPU manual (for example
the Intel Volume 3B documentation or the AMD BIOS and Kernel Developer
Guide).
The libpfm4 library can be used to translate from the name in the
architectural manuals to the raw hex value
perf_event_open()
expects in this field.
If
type
is
PERF_TYPE_BREAKPOINT,
then leave
config
set to zero.
Its parameters are set in other places.
- sample_period, sample_freq
-
A "sampling" counter is one that generates an interrupt
every N events, where N is given by
sample_period.
A sampling counter has
sample_period > 0.
When an overflow interrupt occurs, requested data is recorded
in the mmap buffer.
The
sample_type
field controls what data is recorded on each interrupt.
sample_freq
can be used if you wish to use frequency rather than period.
In this case, you set the
freq
flag.
The kernel will adjust the sampling period
to try and achieve the desired rate.
The rate of adjustment is a
timer tick.
- sample_type
-
The various bits in this field specify which values to include
in the sample.
They will be recorded in a ring-buffer,
which is available to user space using
mmap(2).
The order in which the values are saved in the
sample are documented in the MMAP Layout subsection below;
it is not the
enum perf_event_sample_format
order.
-
- PERF_SAMPLE_IP
-
Records instruction pointer.
- PERF_SAMPLE_TID
-
Records the process and thread IDs.
- PERF_SAMPLE_TIME
-
Records a timestamp.
- PERF_SAMPLE_ADDR
-
Records an address, if applicable.
- PERF_SAMPLE_READ
-
Record counter values for all events in a group, not just the group leader.
- PERF_SAMPLE_CALLCHAIN
-
Records the callchain (stack backtrace).
- PERF_SAMPLE_ID
-
Records a unique ID for the opened event's group leader.
- PERF_SAMPLE_CPU
-
Records CPU number.
- PERF_SAMPLE_PERIOD
-
Records the current sampling period.
- PERF_SAMPLE_STREAM_ID
-
Records a unique ID for the opened event.
Unlike
PERF_SAMPLE_ID
the actual ID is returned, not the group leader.
This ID is the same as the one returned by
PERF_FORMAT_ID.
- PERF_SAMPLE_RAW
-
Records additional data, if applicable.
Usually returned by tracepoint events.
- PERF_SAMPLE_BRANCH_STACK (since Linux 3.4)
-
This provides a record of recent branches, as provided
by CPU branch sampling hardware (such as Intel Last Branch Record).
Not all hardware supports this feature.
See the
branch_sample_type
field for how to filter which branches are reported.
- PERF_SAMPLE_REGS_USER (since Linux 3.7)
-
Records the current user-level CPU register state
(the values in the process before the kernel was called).
- PERF_SAMPLE_STACK_USER (since Linux 3.7)
-
Records the user level stack, allowing stack unwinding.
- PERF_SAMPLE_WEIGHT (since Linux 3.10)
-
Records a hardware provided weight value that expresses how
costly the sampled event was.
This allows the hardware to highlight expensive events in
a profile.
- PERF_SAMPLE_DATA_SRC (since Linux 3.10)
-
Records the data source: where in the memory hierarchy
the data associated with the sampled instruction came from.
This is only available if the underlying hardware
supports this feature.
- PERF_SAMPLE_IDENTIFIER (since Linux 3.12)
-
Places the
SAMPLE_ID
value in a fixed position in the record,
either at the beginning (for sample events) or at the end
(if a non-sample event).
This was necessary because a sample stream may have
records from various different event sources with different
sample_type
settings.
Parsing the event stream properly was not possible because the
format of the record was needed to find
SAMPLE_ID,
but
the format could not be found without knowing what
event the sample belonged to (causing a circular
dependency).
This new
PERF_SAMPLE_IDENTIFIER
setting makes the event stream always parsable
by putting
SAMPLE_ID
in a fixed location, even though
it means having duplicate
SAMPLE_ID
values in records.
- PERF_SAMPLE_TRANSACTION (Since Linux 3.13)
-
Records reasons for transactional memory abort events
(for example, from Intel TSX transactional memory support).
The
precise_ip
setting must be greater than 0 and a transactional memory abort
event must be measured or no values will be recorded.
Also note that some perf_event measurements, such as sampled
cycle counting, may cause extraneous aborts (by causing an
interrupt during a transaction).
- read_format
-
This field specifies the format of the data returned by
read(2)
on a
perf_event_open()
file descriptor.
-
- PERF_FORMAT_TOTAL_TIME_ENABLED
-
Adds the 64-bit
time_enabled
field.
This can be used to calculate estimated totals if
the PMU is overcommitted and multiplexing is happening.
- PERF_FORMAT_TOTAL_TIME_RUNNING
-
Adds the 64-bit
time_running
field.
This can be used to calculate estimated totals if
the PMU is overcommitted and multiplexing is happening.
- PERF_FORMAT_ID
-
Adds a 64-bit unique value that corresponds to the event group.
- PERF_FORMAT_GROUP
-
Allows all counter values in an event group to be read with one read.
- disabled
-
The
disabled
bit specifies whether the counter starts out disabled or enabled.
If disabled, the event can later be enabled by
ioctl(2),
prctl(2),
or
enable_on_exec.
When creating an event group, typically the group leader is initialized
with
disabled
set to 1 and any child events are initialized with
disabled
set to 0.
Despite
disabled
being 0, the child events will not start until the group leader
is enabled.
- inherit
-
The
inherit
bit specifies that this counter should count events of child
tasks as well as the task specified.
This applies only to new children, not to any existing children at
the time the counter is created (nor to any new children of
existing children).
Inherit does not work for some combinations of
read_formats,
such as
PERF_FORMAT_GROUP.
- pinned
-
The
pinned
bit specifies that the counter should always be on the CPU if at all
possible.
It applies only to hardware counters and only to group leaders.
If a pinned counter cannot be put onto the CPU (e.g., because there are
not enough hardware counters or because of a conflict with some other
event), then the counter goes into an 'error' state, where reads
return end-of-file (i.e.,
read(2)
returns 0) until the counter is subsequently enabled or disabled.
- exclusive
-
The
exclusive
bit specifies that when this counter's group is on the CPU,
it should be the only group using the CPU's counters.
In the future this may allow monitoring programs to
support PMU features that need to run alone so that they do not
disrupt other hardware counters.
Note that many unexpected situations may prevent events with the
exclusive
bit set from ever running.
This includes any users running a system-wide
measurement as well as any kernel use of the performance counters
(including the commonly enabled NMI Watchdog Timer interface).
- exclude_user
-
If this bit is set, the count excludes events that happen in user space.
- exclude_kernel
-
If this bit is set, the count excludes events that happen in kernel-space.
- exclude_hv
-
If this bit is set, the count excludes events that happen in the
hypervisor.
This is mainly for PMUs that have built-in support for handling this
(such as POWER).
Extra support is needed for handling hypervisor measurements on most
machines.
- exclude_idle
-
If set, don't count when the CPU is idle.
- mmap
-
The
mmap
bit enables generation of
PERF_RECORD_MMAP
samples for every
mmap(2)
call that has
PROT_EXEC
set.
This allows tools to notice new executable code being mapped into
a program (dynamic shared libraries for example)
so that addresses can be mapped back to the original code.
- comm
-
The
comm
bit enables tracking of process command name as modified by the
exec(2)
and
prctl(PR_SET_NAME)
system calls as well as writing to
/proc/self/comm.
If the
comm_exec
flag is also successfully set (possible since Linux 3.16),
then the misc flag
PERF_RECORD_MISC_COMM_EXEC
can be used to differentiate the
exec(2)
case from the others.
- freq
-
If this bit is set, then
sample_frequency
not
sample_period
is used when setting up the sampling interval.
- inherit_stat
-
This bit enables saving of event counts on context switch for
inherited tasks.
This is meaningful only if the
inherit
field is set.
- enable_on_exec
-
If this bit is set, a counter is automatically
enabled after a call to
exec(2).
- task
-
If this bit is set, then
fork/exit notifications are included in the ring buffer.
- watermark
-
If set, have a sampling interrupt happen when we cross the
wakeup_watermark
boundary.
Otherwise, interrupts happen after
wakeup_events
samples.
- precise_ip (since Linux 2.6.35)
-
This controls the amount of skid.
Skid is how many instructions
execute between an event of interest happening and the kernel
being able to stop and record the event.
Smaller skid is
better and allows more accurate reporting of which events
correspond to which instructions, but hardware is often limited
with how small this can be.
The values of this are the following:
-
- 0 -
-
SAMPLE_IP
can have arbitrary skid.
- 1 -
-
SAMPLE_IP
must have constant skid.
- 2 -
-
SAMPLE_IP
requested to have 0 skid.
- 3 -
-
SAMPLE_IP
must have 0 skid.
See also
PERF_RECORD_MISC_EXACT_IP.
- mmap_data (since Linux 2.6.36)
-
The counterpart of the
mmap
field.
This enables generation of
PERF_RECORD_MMAP
samples for
mmap(2)
calls that do not have
PROT_EXEC
set (for example data and SysV shared memory).
- sample_id_all (since Linux 2.6.38)
-
If set, then TID, TIME, ID, STREAM_ID, and CPU can
additionally be included in
non-PERF_RECORD_SAMPLEs
if the corresponding
sample_type
is selected.
If
PERF_SAMPLE_IDENTIFIER
is specified, then an additional ID value is included
as the last value to ease parsing the record stream.
This may lead to the
id
value appearing twice.
The layout is described by this pseudo-structure:
struct sample_id {
{ u32 pid, tid; } /* if PERF_SAMPLE_TID set */
{ u64 time; } /* if PERF_SAMPLE_TIME set */
{ u64 id; } /* if PERF_SAMPLE_ID set */
{ u64 stream_id;} /* if PERF_SAMPLE_STREAM_ID set */
{ u32 cpu, res; } /* if PERF_SAMPLE_CPU set */
{ u64 id; } /* if PERF_SAMPLE_IDENTIFIER set */
};
- exclude_host (since Linux 3.2)
-
Do not measure time spent in VM host.
- exclude_guest (since Linux 3.2)
-
Do not measure time spent in VM guest.
- exclude_callchain_kernel (since Linux 3.7)
-
Do not include kernel callchains.
- exclude_callchain_user (since Linux 3.7)
-
Do not include user callchains.
- mmap2 (since Linux 3.16)
-
Generate an extended executable mmap record that contains enough
additional information to uniquely identify shared mappings.
The
mmap
flag must also be set for this to work.
- comm_exec (since Linux 3.16)
-
This is purely a feature-detection flag, it does not change
kernel behavior.
If this flag can successfully be set, then, when
comm
is enabled, the
PERF_RECORD_MISC_COMM_EXEC
flag will be set in the
misc
field of a comm record header if the rename event being
reported was caused by a call to
exec(2).
This allows tools to distinguish between the various
types of process renaming.
- wakeup_events, wakeup_watermark
-
This union sets how many samples
(wakeup_events)
or bytes
(wakeup_watermark)
happen before an overflow signal happens.
Which one is used is selected by the
watermark
bit flag.
wakeup_events
only counts
PERF_RECORD_SAMPLE
record types.
To receive a signal for every incoming
PERF_RECORD
type set
wakeup_watermark
to 1.
- bp_type (since Linux 2.6.33)
-
This chooses the breakpoint type.
It is one of:
-
- HW_BREAKPOINT_EMPTY
-
No breakpoint.
- HW_BREAKPOINT_R
-
Count when we read the memory location.
- HW_BREAKPOINT_W
-
Count when we write the memory location.
- HW_BREAKPOINT_RW
-
Count when we read or write the memory location.
- HW_BREAKPOINT_X
-
Count when we execute code at the memory location.
The values can be combined via a bitwise or, but the
combination of
HW_BREAKPOINT_R
or
HW_BREAKPOINT_W
with
HW_BREAKPOINT_X
is not allowed.
- bp_addr (since Linux 2.6.33)
-
bp_addr
address of the breakpoint.
For execution breakpoints this is the memory address of the instruction
of interest; for read and write breakpoints it is the memory address
of the memory location of interest.
- config1 (since Linux 2.6.39)
-
config1
is used for setting events that need an extra register or otherwise
do not fit in the regular config field.
Raw OFFCORE_EVENTS on Nehalem/Westmere/SandyBridge use this field
on 3.3 and later kernels.
- bp_len (since Linux 2.6.33)
-
bp_len
is the length of the breakpoint being measured if
type
is
PERF_TYPE_BREAKPOINT.
Options are
HW_BREAKPOINT_LEN_1,
HW_BREAKPOINT_LEN_2,
HW_BREAKPOINT_LEN_4,
HW_BREAKPOINT_LEN_8.
For an execution breakpoint, set this to
sizeof(long).
- config2 (since Linux 2.6.39)
-
config2
is a further extension of the
config1
field.
- branch_sample_type (since Linux 3.4)
-
If
PERF_SAMPLE_BRANCH_STACK
is enabled, then this specifies what branches to include
in the branch record.
The first part of the value is the privilege level, which
is a combination of one of the following values.
If the user does not set privilege level explicitly, the kernel
will use the event's privilege level.
Event and branch privilege levels do not have to match.
-
- PERF_SAMPLE_BRANCH_USER
-
Branch target is in user space.
- PERF_SAMPLE_BRANCH_KERNEL
-
Branch target is in kernel space.
- PERF_SAMPLE_BRANCH_HV
-
Branch target is in hypervisor.
- PERF_SAMPLE_BRANCH_PLM_ALL
-
A convenience value that is the three preceding values ORed together.
In addition to the privilege value, at least one or more of the
following bits must be set.
- PERF_SAMPLE_BRANCH_ANY
-
Any branch type.
- PERF_SAMPLE_BRANCH_ANY_CALL
-
Any call branch.
- PERF_SAMPLE_BRANCH_ANY_RETURN
-
Any return branch.
- PERF_SAMPLE_BRANCH_IND_CALL
-
Indirect calls.
- PERF_SAMPLE_BRANCH_COND (since Linux 3.16)
-
Conditional branches.
- PERF_SAMPLE_BRANCH_ABORT_TX (since Linux 3.11)
-
Transactional memory aborts.
- PERF_SAMPLE_BRANCH_IN_TX (since Linux 3.11)
-
Branch in transactional memory transaction.
- PERF_SAMPLE_BRANCH_NO_TX (since Linux 3.11)
-
Branch not in transactional memory transaction.
- sample_regs_user (since Linux 3.7)
-
This bit mask defines the set of user CPU registers to dump on samples.
The layout of the register mask is architecture-specific and
described in the kernel header
arch/ARCH/include/uapi/asm/perf_regs.h.
- sample_stack_user (since Linux 3.7)
-
This defines the size of the user stack to dump if
PERF_SAMPLE_STACK_USER
is specified.
Reading results
Once a
perf_event_open()
file descriptor has been opened, the values
of the events can be read from the file descriptor.
The values that are there are specified by the
read_format
field in the
attr
structure at open time.
If you attempt to read into a buffer that is not big enough to hold the
data
ENOSPC
is returned
Here is the layout of the data returned by a read:
- *
-
If
PERF_FORMAT_GROUP
was specified to allow reading all events in a group at once:
struct read_format {
u64 nr; /* The number of events */
u64 time_enabled; /* if PERF_FORMAT_TOTAL_TIME_ENABLED */
u64 time_running; /* if PERF_FORMAT_TOTAL_TIME_RUNNING */
struct
u64 value; /* The value of the event */
u64 id; /* if PERF_FORMAT_ID */
} values[nr];
};
- *
-
If
PERF_FORMAT_GROUP
was
not
specified:
struct read_format {
u64 value; /* The value of the event */
u64 time_enabled; /* if PERF_FORMAT_TOTAL_TIME_ENABLED */
u64 time_running; /* if PERF_FORMAT_TOTAL_TIME_RUNNING */
u64 id; /* if PERF_FORMAT_ID */
};
The values read are as follows:
- nr
-
The number of events in this file descriptor.
Only available if
PERF_FORMAT_GROUP
was specified.
- time_enabled, time_running
-
Total time the event was enabled and running.
Normally these are the same.
If more events are started,
then available counter slots on the PMU, then multiplexing
happens and events run only part of the time.
In that case, the
time_enabled
and
time running
values can be used to scale an estimated value for the count.
- value
-
An unsigned 64-bit value containing the counter result.
- id
-
A globally unique value for this particular event, only there if
PERF_FORMAT_ID
was specified in
read_format.
MMAP layout
When using
perf_event_open()
in sampled mode, asynchronous events
(like counter overflow or
PROT_EXEC
mmap tracking)
are logged into a ring-buffer.
This ring-buffer is created and accessed through
mmap(2).
The mmap size should be 1+2^n pages, where the first page is a
metadata page
(struct perf_event_mmap_page)
that contains various
bits of information such as where the ring-buffer head is.
Before kernel 2.6.39, there is a bug that means you must allocate a mmap
ring buffer when sampling even if you do not plan to access it.
The structure of the first metadata mmap page is as follows:
struct perf_event_mmap_page {
__u32 version; /* version number of this structure */
__u32 compat_version; /* lowest version this is compat with */
__u32 lock; /* seqlock for synchronization */
__u32 index; /* hardware counter identifier */
__s64 offset; /* add to hardware counter value */
__u64 time_enabled; /* time event active */
__u64 time_running; /* time event on CPU */
union {
__u64 capabilities;
struct {
__u64 cap_usr_time / cap_usr_rdpmc / cap_bit0 : 1,
cap_bit0_is_deprecated : 1,
cap_user_rdpmc : 1,
cap_user_time : 1,
cap_user_time_zero : 1,
};
};
__u16 pmc_width;
__u16 time_shift;
__u32 time_mult;
__u64 time_offset;
__u64 __reserved[120]; /* Pad to 1k */
__u64 data_head; /* head in the data section */
__u64 data_tail; /* user-space written tail */
}
The following list describes the fields in the
perf_event_mmap_page
structure in more detail:
- version
-
Version number of this structure.
- compat_version
-
The lowest version this is compatible with.
- lock
-
A seqlock for synchronization.
- index
-
A unique hardware counter identifier.
- offset
-
When using rdpmc for reads this offset value
must be added to the one returned by rdpmc to get
the current total event count.
- time_enabled
-
Time the event was active.
- time_running
-
Time the event was running.
- cap_usr_time / cap_usr_rdpmc / cap_bit0 (since Linux 3.4)
-
There was a bug in the definition of
cap_usr_time
and
cap_usr_rdpmc
from Linux 3.4 until Linux 3.11.
Both bits were defined to point to the same location, so it was
impossible to know if
cap_usr_time
or
cap_usr_rdpmc
were actually set.
Starting with 3.12 these are renamed to
cap_bit0
and you should use the new
cap_user_time
and
cap_user_rdpmc
fields instead.
- cap_bit0_is_deprecated (since Linux 3.12)
-
If set, this bit indicates that the kernel supports
the properly separated
cap_user_time
and
cap_user_rdpmc
bits.
If not-set, it indicates an older kernel where
cap_usr_time
and
cap_usr_rdpmc
map to the same bit and thus both features should
be used with caution.
- cap_user_rdpmc (since Linux 3.12)
-
If the hardware supports user-space read of performance counters
without syscall (this is the "rdpmc" instruction on x86), then
the following code can be used to do a read:
u32 seq, time_mult, time_shift, idx, width;
u64 count, enabled, running;
u64 cyc, time_offset;
do {
seq = pc->lock;
barrier();
enabled = pc->time_enabled;
running = pc->time_running;
if (pc->cap_usr_time && enabled != running) {
cyc = rdtsc();
time_offset = pc->time_offset;
time_mult = pc->time_mult;
time_shift = pc->time_shift;
}
idx = pc->index;
count = pc->offset;
if (pc->cap_usr_rdpmc && idx) {
width = pc->pmc_width;
count += rdpmc(idx - 1);
}
barrier();
} while (pc->lock != seq);
- cap_user_time (since Linux 3.12)
-
This bit indicates the hardware has a constant, nonstop
timestamp counter (TSC on x86).
- cap_user_time_zero (since Linux 3.12)
-
Indicates the presence of
time_zero
which allows mapping timestamp values to
the hardware clock.
- pmc_width
-
If
cap_usr_rdpmc,
this field provides the bit-width of the value
read using the rdpmc or equivalent instruction.
This can be used to sign extend the result like:
pmc <<= 64 - pmc_width;
pmc >>= 64 - pmc_width; // signed shift right
count += pmc;
- time_shift, time_mult, time_offset
-
If
cap_usr_time,
these fields can be used to compute the time
delta since time_enabled (in nanoseconds) using rdtsc or similar.
u64 quot, rem;
u64 delta;
quot = (cyc >> time_shift);
rem = cyc & ((1 << time_shift) - 1);
delta = time_offset + quot * time_mult +
((rem * time_mult) >> time_shift);
Where
time_offset,
time_mult,
time_shift,
and
cyc
are read in the
seqcount loop described above.
This delta can then be added to
enabled and possible running (if idx), improving the scaling:
enabled += delta;
if (idx)
running += delta;
quot = count / running;
rem = count % running;
count = quot * enabled + (rem * enabled) / running;
- time_zero (since Linux 3.12)
-
If
cap_usr_time_zero
is set, then the hardware clock (the TSC timestamp counter on x86)
can be calculated from the
time_zero, time_mult, and time_shift values:
time = timestamp - time_zero;
quot = time / time_mult;
rem = time % time_mult;
cyc = (quot << time_shift) + (rem << time_shift) / time_mult;
And vice versa:
quot = cyc >> time_shift;
rem = cyc & ((1 << time_shift) - 1);
timestamp = time_zero + quot * time_mult +
((rem * time_mult) >> time_shift);
- data_head
-
This points to the head of the data section.
The value continuously increases, it does not wrap.
The value needs to be manually wrapped by the size of the mmap buffer
before accessing the samples.
On SMP-capable platforms, after reading the
data_head
value,
user space should issue an rmb().
- data_tail
-
When the mapping is
PROT_WRITE,
the
data_tail
value should be written by user space to reflect the last read data.
In this case, the kernel will not overwrite unread data.
The following 2^n ring-buffer pages have the layout described below.
If
perf_event_attr.sample_id_all
is set, then all event types will
have the sample_type selected fields related to where/when (identity)
an event took place (TID, TIME, ID, CPU, STREAM_ID) described in
PERF_RECORD_SAMPLE
below, it will be stashed just after the
perf_event_header
and the fields already present for the existing
fields, that is, at the end of the payload.
That way a newer perf.data
file will be supported by older perf tools, with these new optional
fields being ignored.
The mmap values start with a header:
struct perf_event_header {
__u32 type;
__u16 misc;
__u16 size;
};
Below, we describe the
perf_event_header
fields in more detail.
For ease of reading,
the fields with shorter descriptions are presented first.
- size
-
This indicates the size of the record.
- misc
-
The
misc
field contains additional information about the sample.
The CPU mode can be determined from this value by masking with
PERF_RECORD_MISC_CPUMODE_MASK
and looking for one of the following (note these are not
bit masks, only one can be set at a time):
-
- PERF_RECORD_MISC_CPUMODE_UNKNOWN
-
Unknown CPU mode.
- PERF_RECORD_MISC_KERNEL
-
Sample happened in the kernel.
- PERF_RECORD_MISC_USER
-
Sample happened in user code.
- PERF_RECORD_MISC_HYPERVISOR
-
Sample happened in the hypervisor.
- PERF_RECORD_MISC_GUEST_KERNEL
-
Sample happened in the guest kernel.
- PERF_RECORD_MISC_GUEST_USER
-
Sample happened in guest user code.
-
In addition, one of the following bits can be set:
- PERF_RECORD_MISC_MMAP_DATA
-
This is set when the mapping is not executable;
otherwise the mapping is executable.
- PERF_RECORD_MISC_COMM_EXEC
-
This is set for a
PERF_RECORD_COMM
record on kernels more recent than Linux 3.16
if a process name change was caused by an
exec(2)
system call.
It is an alias for
PERF_RECORD_MISC_MMAP_DATA
since the two values would not be set in the same record.
- PERF_RECORD_MISC_EXACT_IP
-
This indicates that the content of
PERF_SAMPLE_IP
points
to the actual instruction that triggered the event.
See also
perf_event_attr.precise_ip.
- PERF_RECORD_MISC_EXT_RESERVED
-
This indicates there is extended data available (currently not used).
- type
-
The
type
value is one of the below.
The values in the corresponding record (that follows the header)
depend on the
type
selected as shown.
-
- PERF_RECORD_MMAP
-
The MMAP events record the
PROT_EXEC
mappings so that we can correlate
user-space IPs to code.
They have the following structure:
struct {
struct perf_event_header header;
u32 pid, tid;
u64 addr;
u64 len;
u64 pgoff;
char filename[];
};
-
- pid
-
is the process ID.
- tid
-
is the thread ID.
- addr
-
is the address of the allocated memory.
len
is the length of the allocated memory.
pgoff
is the page offset of the allocated memory.
filename
is a string describing the backing of the allocated memory.
- PERF_RECORD_LOST
-
This record indicates when events are lost.
struct {
struct perf_event_header header;
u64 id;
u64 lost;
struct sample_id sample_id;
};
-
- id
-
is the unique event ID for the samples that were lost.
- lost
-
is the number of events that were lost.
- PERF_RECORD_COMM
-
This record indicates a change in the process name.
struct {
struct perf_event_header header;
u32 pid;
u32 tid;
char comm[];
struct sample_id sample_id;
};
-
- pid
-
is the process ID.
- tid
-
is the thread ID.
- comm
-
is a string containing the new name of the process.
- PERF_RECORD_EXIT
-
This record indicates a process exit event.
struct {
struct perf_event_header header;
u32 pid, ppid;
u32 tid, ptid;
u64 time;
struct sample_id sample_id;
};
- PERF_RECORD_THROTTLE, PERF_RECORD_UNTHROTTLE
-
This record indicates a throttle/unthrottle event.
struct {
struct perf_event_header header;
u64 time;
u64 id;
u64 stream_id;
struct sample_id sample_id;
};
- PERF_RECORD_FORK
-
This record indicates a fork event.
struct {
struct perf_event_header header;
u32 pid, ppid;
u32 tid, ptid;
u64 time;
struct sample_id sample_id;
};
- PERF_RECORD_READ
-
This record indicates a read event.
struct {
struct perf_event_header header;
u32 pid, tid;
struct read_format values;
struct sample_id sample_id;
};
- PERF_RECORD_SAMPLE
-
This record indicates a sample.
struct {
struct perf_event_header header;
u64 sample_id; /* if PERF_SAMPLE_IDENTIFIER */
u64 ip; /* if PERF_SAMPLE_IP */
u32 pid, tid; /* if PERF_SAMPLE_TID */
u64 time; /* if PERF_SAMPLE_TIME */
u64 addr; /* if PERF_SAMPLE_ADDR */
u64 id; /* if PERF_SAMPLE_ID */
u64 stream_id; /* if PERF_SAMPLE_STREAM_ID */
u32 cpu, res; /* if PERF_SAMPLE_CPU */
u64 period; /* if PERF_SAMPLE_PERIOD */
struct read_format v; /* if PERF_SAMPLE_READ */
u64 nr; /* if PERF_SAMPLE_CALLCHAIN */
u64 ips[nr]; /* if PERF_SAMPLE_CALLCHAIN */
u32 size; /* if PERF_SAMPLE_RAW */
char data[size]; /* if PERF_SAMPLE_RAW */
u64 bnr; /* if PERF_SAMPLE_BRANCH_STACK */
struct perf_branch_entry lbr[bnr];
/* if PERF_SAMPLE_BRANCH_STACK */
u64 abi; /* if PERF_SAMPLE_REGS_USER */
u64 regs[weight(mask)];
/* if PERF_SAMPLE_REGS_USER */
u64 size; /* if PERF_SAMPLE_STACK_USER */
char data[size]; /* if PERF_SAMPLE_STACK_USER */
u64 dyn_size; /* if PERF_SAMPLE_STACK_USER */
u64 weight; /* if PERF_SAMPLE_WEIGHT */
u64 data_src; /* if PERF_SAMPLE_DATA_SRC */
u64 transaction;/* if PERF_SAMPLE_TRANSACTION */
};
-
- sample_id
-
If
PERF_SAMPLE_IDENTIFIER
is enabled, a 64-bit unique ID is included.
This is a duplication of the
PERF_SAMPLE_ID
id
value, but included at the beginning of the sample
so parsers can easily obtain the value.
- ip
-
If
PERF_SAMPLE_IP
is enabled, then a 64-bit instruction
pointer value is included.
- pid, tid
-
If
PERF_SAMPLE_TID
is enabled, then a 32-bit process ID
and 32-bit thread ID are included.
- time
-
If
PERF_SAMPLE_TIME
is enabled, then a 64-bit timestamp
is included.
This is obtained via local_clock() which is a hardware timestamp
if available and the jiffies value if not.
- addr
-
If
PERF_SAMPLE_ADDR
is enabled, then a 64-bit address is included.
This is usually the address of a tracepoint,
breakpoint, or software event; otherwise the value is 0.
- id
-
If
PERF_SAMPLE_ID
is enabled, a 64-bit unique ID is included.
If the event is a member of an event group, the group leader ID is returned.
This ID is the same as the one returned by
PERF_FORMAT_ID.
- stream_id
-
If
PERF_SAMPLE_STREAM_ID
is enabled, a 64-bit unique ID is included.
Unlike
PERF_SAMPLE_ID
the actual ID is returned, not the group leader.
This ID is the same as the one returned by
PERF_FORMAT_ID.
- cpu, res
-
If
PERF_SAMPLE_CPU
is enabled, this is a 32-bit value indicating
which CPU was being used, in addition to a reserved (unused)
32-bit value.
- period
-
If
PERF_SAMPLE_PERIOD
is enabled, a 64-bit value indicating
the current sampling period is written.
- v
-
If
PERF_SAMPLE_READ
is enabled, a structure of type read_format
is included which has values for all events in the event group.
The values included depend on the
read_format
value used at
perf_event_open()
time.
- nr, ips[nr]
-
If
PERF_SAMPLE_CALLCHAIN
is enabled, then a 64-bit number is included
which indicates how many following 64-bit instruction pointers will
follow.
This is the current callchain.
- size, data[size]
-
If
PERF_SAMPLE_RAW
is enabled, then a 32-bit value indicating size
is included followed by an array of 8-bit values of length size.
The values are padded with 0 to have 64-bit alignment.
This RAW record data is opaque with respect to the ABI.
The ABI doesn't make any promises with respect to the stability
of its content, it may vary depending
on event, hardware, and kernel version.
- bnr, lbr[bnr]
-
If
PERF_SAMPLE_BRANCH_STACK
is enabled, then a 64-bit value indicating
the number of records is included, followed by
bnr
perf_branch_entry
structures which each include the fields:
-
- from
-
This indicates the source instruction (may not be a branch).
- to
-
The branch target.
- mispred
-
The branch target was mispredicted.
- predicted
-
The branch target was predicted.
- in_tx (since Linux 3.11)
-
The branch was in a transactional memory transaction.
- abort (since Linux 3.11)
-
The branch was in an aborted transactional memory transaction.
The entries are from most to least recent, so the first entry
has the most recent branch.
Support for
mispred
and
predicted
is optional; if not supported, both
values will be 0.
The type of branches recorded is specified by the
branch_sample_type
field.
- abi, regs[weight(mask)]
-
If
PERF_SAMPLE_REGS_USER
is enabled, then the user CPU registers are recorded.
The
abi
field is one of
PERF_SAMPLE_REGS_ABI_NONE, PERF_SAMPLE_REGS_ABI_32 or
PERF_SAMPLE_REGS_ABI_64.
The
regs
field is an array of the CPU registers that were specified by
the
sample_regs_user
attr field.
The number of values is the number of bits set in the
sample_regs_user
bit mask.
- size, data[size], dyn_size
-
If
PERF_SAMPLE_STACK_USER
is enabled, then the user stack is recorded.
This can be used to generate stack backtraces.
size
is the size requested by the user in
sample_stack_user
or else the maximum record size.
data
is the stack data (a raw dump of the memory pointed to by the
stack pointer at the time of sampling).
dyn_size
is the amount of data actually dumped (can be less than
size).
- weight
-
If
PERF_SAMPLE_WEIGHT
is enabled, then a 64-bit value provided by the hardware
is recorded that indicates how costly the event was.
This allows expensive events to stand out more clearly
in profiles.
- data_src
-
If
PERF_SAMPLE_DATA_SRC
is enabled, then a 64-bit value is recorded that is made up of
the following fields:
-
- mem_op
-
Type of opcode, a bitwise combination of:
-
- PERF_MEM_OP_NA
-
Not available
- PERF_MEM_OP_LOAD
-
Load instruction
- PERF_MEM_OP_STORE
-
Store instruction
- PERF_MEM_OP_PFETCH
-
Prefetch
- PERF_MEM_OP_EXEC
-
Executable code
- mem_lvl
-
Memory hierarchy level hit or miss, a bitwise combination of
the following, shifted left by
PERF_MEM_LVL_SHIFT:
-
- PERF_MEM_LVL_NA
-
Not available
- PERF_MEM_LVL_HIT
-
Hit
- PERF_MEM_LVL_MISS
-
Miss
- PERF_MEM_LVL_L1
-
Level 1 cache
- PERF_MEM_LVL_LFB
-
Line fill buffer
- PERF_MEM_LVL_L2
-
Level 2 cache
- PERF_MEM_LVL_L3
-
Level 3 cache
- PERF_MEM_LVL_LOC_RAM
-
Local DRAM
- PERF_MEM_LVL_REM_RAM1
-
Remote DRAM 1 hop
- PERF_MEM_LVL_REM_RAM2
-
Remote DRAM 2 hops
- PERF_MEM_LVL_REM_CCE1
-
Remote cache 1 hop
- PERF_MEM_LVL_REM_CCE2
-
Remote cache 2 hops
- PERF_MEM_LVL_IO
-
I/O memory
- PERF_MEM_LVL_UNC
-
Uncached memory
- mem_snoop
-
Snoop mode, a bitwise combination of the following, shifted left by
PERF_MEM_SNOOP_SHIFT:
-
- PERF_MEM_SNOOP_NA
-
Not available
- PERF_MEM_SNOOP_NONE
-
No snoop
- PERF_MEM_SNOOP_HIT
-
Snoop hit
- PERF_MEM_SNOOP_MISS
-
Snoop miss
- PERF_MEM_SNOOP_HITM
-
Snoop hit modified
- mem_lock
-
Lock instruction, a bitwise combination of the following, shifted left by
PERF_MEM_LOCK_SHIFT:
-
- PERF_MEM_LOCK_NA
-
Not available
- PERF_MEM_LOCK_LOCKED
-
Locked transaction
- mem_dtlb
-
TLB access hit or miss, a bitwise combination of the following, shifted
left by
PERF_MEM_TLB_SHIFT:
-
- PERF_MEM_TLB_NA
-
Not available
- PERF_MEM_TLB_HIT
-
Hit
- PERF_MEM_TLB_MISS
-
Miss
- PERF_MEM_TLB_L1
-
Level 1 TLB
- PERF_MEM_TLB_L2
-
Level 2 TLB
- PERF_MEM_TLB_WK
-
Hardware walker
- PERF_MEM_TLB_OS
-
OS fault handler
- transaction
-
If the
PERF_SAMPLE_TRANSACTION
flag is set, then a 64-bit field is recorded describing
the sources of any transactional memory aborts.
The field is a bitwise combination of the following values:
-
- PERF_TXN_ELISION
-
Abort from an elision type transaction (Intel-CPU-specific).
- PERF_TXN_TRANSACTION
-
Abort from a generic transaction.
- PERF_TXN_SYNC
-
Synchronous abort (related to the reported instruction).
- PERF_TXN_ASYNC
-
Asynchronous abort (not related to the reported instruction).
- PERF_TXN_RETRY
-
Retryable abort (retrying the transaction may have succeeded).
- PERF_TXN_CONFLICT
-
Abort due to memory conflicts with other threads.
- PERF_TXN_CAPACITY_WRITE
-
Abort due to write capacity overflow.
- PERF_TXN_CAPACITY_READ
-
Abort due to read capacity overflow.
-
In addition, a user-specified abort code can be obtained from
the high 32 bits of the field by shifting right by
PERF_TXN_ABORT_SHIFT
and masking with
PERF_TXN_ABORT_MASK.
- PERF_RECORD_MMAP2
-
This record includes extended information on
mmap(2)
calls returning executable mappings.
The format is similar to that of the
PERF_RECORD_MMAP
record, but includes extra values that allow uniquely identifying
shared mappings.
struct {
struct perf_event_header header;
u32 pid;
u32 tid;
u64 addr;
u64 len;
u64 pgoff;
u32 maj;
u32 min;
u64 ino;
u64 ino_generation;
u32 prot;
u32 flags;
char filename[];
struct sample_id sample_id;
};
-
- pid
-
is the process ID.
- tid
-
is the thread ID.
- addr
-
is the address of the allocated memory.
- len
-
is the length of the allocated memory.
- pgoff
-
is the page offset of the allocated memory.
- maj
-
is the major ID of the underlying device.
- min
-
is the minor ID of the underlying device.
- ino
-
is the inode number.
- ino_generation
-
is the inode generation.
- prot
-
is the protection information.
- flags
-
is the flags information.
- filename
-
is a string describing the backing of the allocated memory.
Signal overflow
Events can be set to deliver a signal when a threshold is crossed.
The signal handler is set up using the
poll(2),
select(2),
epoll(2)
and
fcntl(2),
system calls.
To generate signals, sampling must be enabled
(sample_period
must have a nonzero value).
There are two ways to generate signals.
The first is to set a
wakeup_events
or
wakeup_watermark
value that will generate a signal if a certain number of samples
or bytes have been written to the mmap ring buffer.
In this case, a signal of type
POLL_IN
is sent.
The other way is by use of the
PERF_EVENT_IOC_REFRESH
ioctl.
This ioctl adds to a counter that decrements each time the event overflows.
When nonzero, a
POLL_IN
signal is sent on overflow, but
once the value reaches 0, a signal is sent of type
POLL_HUP
and
the underlying event is disabled.
Note: on newer kernels (since at least as early as Linux 3.2),
a signal is provided for every overflow, even if
wakeup_events
is not set.
rdpmc instruction
Starting with Linux 3.4 on x86, you can use the
rdpmc
instruction to get low-latency reads without having to enter the kernel.
Note that using
rdpmc
is not necessarily faster than other methods for reading event values.
Support for this can be detected with the
cap_usr_rdpmc
field in the mmap page; documentation on how
to calculate event values can be found in that section.
perf_event ioctl calls
Various ioctls act on
perf_event_open()
file descriptors:
- PERF_EVENT_IOC_ENABLE
-
This enables the individual event or event group specified by the
file descriptor argument.
If the
PERF_IOC_FLAG_GROUP
bit is set in the ioctl argument, then all events in a group are
enabled, even if the event specified is not the group leader
(but see BUGS).
- PERF_EVENT_IOC_DISABLE
-
This disables the individual counter or event group specified by the
file descriptor argument.
Enabling or disabling the leader of a group enables or disables the
entire group; that is, while the group leader is disabled, none of the
counters in the group will count.
Enabling or disabling a member of a group other than the leader
affects only that counter; disabling a non-leader
stops that counter from counting but doesn't affect any other counter.
If the
PERF_IOC_FLAG_GROUP
bit is set in the ioctl argument, then all events in a group are
disabled, even if the event specified is not the group leader
(but see BUGS).
- PERF_EVENT_IOC_REFRESH
-
Non-inherited overflow counters can use this
to enable a counter for a number of overflows specified by the argument,
after which it is disabled.
Subsequent calls of this ioctl add the argument value to the current
count.
A signal with
POLL_IN
set will happen on each overflow until the
count reaches 0; when that happens a signal with
POLL_HUP
set is sent and the event is disabled.
Using an argument of 0 is considered undefined behavior.
- PERF_EVENT_IOC_RESET
-
Reset the event count specified by the
file descriptor argument to zero.
This resets only the counts; there is no way to reset the
multiplexing
time_enabled
or
time_running
values.
If the
PERF_IOC_FLAG_GROUP
bit is set in the ioctl argument, then all events in a group are
reset, even if the event specified is not the group leader
(but see BUGS).
- PERF_EVENT_IOC_PERIOD
-
This updates the overflow period for the event.
Since Linux 3.7 (on ARM) and Linux 3.14 (all other architectures),
the new period takes effect immediately.
On older kernels, the new period did not take effect until
after the next overflow.
The argument is a pointer to a 64-bit value containing the
desired new period.
Prior to Linux 2.6.36 this ioctl always failed due to a bug
in the kernel.
- PERF_EVENT_IOC_SET_OUTPUT
-
This tells the kernel to report event notifications to the specified
file descriptor rather than the default one.
The file descriptors must all be on the same CPU.
The argument specifies the desired file descriptor, or -1 if
output should be ignored.
- PERF_EVENT_IOC_SET_FILTER (since Linux 2.6.33)
-
This adds an ftrace filter to this event.
The argument is a pointer to the desired ftrace filter.
- PERF_EVENT_IOC_ID (since Linux 3.12)
-
This returns the event ID value for the given event file descriptor.
The argument is a pointer to a 64-bit unsigned integer
to hold the result.
Using prctl
A process can enable or disable all the event groups that are
attached to it using the
prctl(2)
PR_TASK_PERF_EVENTS_ENABLE
and
PR_TASK_PERF_EVENTS_DISABLE
operations.
This applies to all counters on the calling process, whether created by
this process or by another, and does not affect any counters that this
process has created on other processes.
It enables or disables only
the group leaders, not any other members in the groups.
perf_event related configuration files
Files in
/proc/sys/kernel/
-
- /proc/sys/kernel/perf_event_paranoid
-
The
perf_event_paranoid
file can be set to restrict access to the performance counters.
-
- 2
-
only allow user-space measurements.
- 1
-
allow both kernel and user measurements (default).
- 0
-
allow access to CPU-specific data but not raw tracepoint samples.
- -1
-
no restrictions.
-
The existence of the
perf_event_paranoid
file is the official method for determining if a kernel supports
perf_event_open().
- /proc/sys/kernel/perf_event_max_sample_rate
-
This sets the maximum sample rate.
Setting this too high can allow
users to sample at a rate that impacts overall machine performance
and potentially lock up the machine.
The default value is
100000 (samples per second).
- /proc/sys/kernel/perf_event_mlock_kb
-
Maximum number of pages an unprivileged user can
mlock(2).
The default is 516 (kB).
Files in
/sys/bus/event_source/devices/
-
Since Linux 2.6.34, the kernel supports having multiple PMUs
available for monitoring.
Information on how to program these PMUs can be found under
/sys/bus/event_source/devices/.
Each subdirectory corresponds to a different PMU.
- /sys/bus/event_source/devices/*/type (since Linux 2.6.38)
-
This contains an integer that can be used in the
type
field of
perf_event_attr
to indicate that you wish to use this PMU.
- /sys/bus/event_source/devices/*/rdpmc (since Linux 3.4)
-
If this file is 1, then direct user-space access to the
performance counter registers is allowed via the rdpmc instruction.
This can be disabled by echoing 0 to the file.
- /sys/bus/event_source/devices/*/format/ (since Linux 3.4)
-
This subdirectory contains information on the architecture-specific
subfields available for programming the various
config
fields in the
perf_event_attr
struct.
The content of each file is the name of the config field, followed
by a colon, followed by a series of integer bit ranges separated by
commas.
For example, the file
event
may contain the value
config1:1,6-10,44
which indicates that event is an attribute that occupies bits 1,6-10, and 44
of
perf_event_attr::config1.
- /sys/bus/event_source/devices/*/events/ (since Linux 3.4)
-
This subdirectory contains files with predefined events.
The contents are strings describing the event settings
expressed in terms of the fields found in the previously mentioned
./format/
directory.
These are not necessarily complete lists of all events supported by
a PMU, but usually a subset of events deemed useful or interesting.
The content of each file is a list of attribute names
separated by commas.
Each entry has an optional value (either hex or decimal).
If no value is specified, then it is assumed to be a single-bit
field with a value of 1.
An example entry may look like this:
event=0x2,inv,ldlat=3.
- /sys/bus/event_source/devices/*/uevent
-
This file is the standard kernel device interface
for injecting hotplug events.
- /sys/bus/event_source/devices/*/cpumask (since Linux 3.7)
-
The
cpumask
file contains a comma-separated list of integers that
indicate a representative CPU number for each socket (package)
on the motherboard.
This is needed when setting up uncore or northbridge events, as
those PMUs present socket-wide events.
RETURN VALUE
perf_event_open()
returns the new file descriptor, or -1 if an error occurred
(in which case,
errno
is set appropriately).
ERRORS
The errors returned by
perf_event_open()
can be inconsistent, and may
vary across processor architectures and performance monitoring units.
- E2BIG
-
Returned if the
perf_event_attr
size
value is too small
(smaller than
PERF_ATTR_SIZE_VER0),
too big (larger than the page size),
or larger than the kernel supports and the extra bytes are not zero.
When
E2BIG
is returned, the
perf_event_attr
size
field is overwritten by the kernel to be the size of the structure
it was expecting.
- EACCES
-
Returned when the requested event requires
CAP_SYS_ADMIN
permissions (or a more permissive perf_event paranoid setting).
Some common cases where an unprivileged process
may encounter this error:
attaching to a process owned by a different user;
monitoring all processes on a given CPU (i.e., specifying the
pid
argument as -1);
and not setting
exclude_kernel
when the paranoid setting requires it.
- EBADF
-
Returned if the
group_fd
file descriptor is not valid, or, if
PERF_FLAG_PID_CGROUP
is set,
the cgroup file descriptor in
pid
is not valid.
- EFAULT
-
Returned if the
attr
pointer points at an invalid memory address.
- EINVAL
-
Returned if the specified event is invalid.
There are many possible reasons for this.
A not-exhaustive list:
sample_freq
is higher than the maximum setting;
the
cpu
to monitor does not exist;
read_format
is out of range;
sample_type
is out of range;
the
flags
value is out of range;
exclusive
or
pinned
set and the event is not a group leader;
the event
config
values are out of range or set reserved bits;
the generic event selected is not supported; or
there is not enough room to add the selected event.
- EMFILE
-
Each opened event uses one file descriptor.
If a large number of events are opened the per-user file
descriptor limit (often 1024) will be hit and no more
events can be created.
- ENODEV
-
Returned when the event involves a feature not supported
by the current CPU.
- ENOENT
-
Returned if the
type
setting is not valid.
This error is also returned for
some unsupported generic events.
- ENOSPC
-
Prior to Linux 3.3, if there was not enough room for the event,
ENOSPC
was returned.
In Linux 3.3, this was changed to
EINVAL.
ENOSPC
is still returned if you try to add more breakpoint events
than supported by the hardware.
- ENOSYS
-
Returned if
PERF_SAMPLE_STACK_USER
is set in
sample_type
and it is not supported by hardware.
- EOPNOTSUPP
-
Returned if an event requiring a specific hardware feature is
requested but there is no hardware support.
This includes requesting low-skid events if not supported,
branch tracing if it is not available, sampling if no PMU
interrupt is available, and branch stacks for software events.
- EPERM
-
Returned on many (but not all) architectures when an unsupported
exclude_hv, exclude_idle, exclude_user, or exclude_kernel
setting is specified.
It can also happen, as with
EACCES,
when the requested event requires
CAP_SYS_ADMIN
permissions (or a more permissive perf_event paranoid setting).
This includes setting a breakpoint on a kernel address,
and (since Linux 3.13) setting a kernel function-trace tracepoint.
- ESRCH
-
Returned if attempting to attach to a process that does not exist.
VERSION
perf_event_open()
was introduced in Linux 2.6.31 but was called
perf_counter_open().
It was renamed in Linux 2.6.32.
CONFORMING TO
This
perf_event_open()
system call Linux- specific
and should not be used in programs intended to be portable.
NOTES
Glibc does not provide a wrapper for this system call; call it using
syscall(2).
See the example below.
The official way of knowing if
perf_event_open()
support is enabled is checking
for the existence of the file
/proc/sys/kernel/perf_event_paranoid.
BUGS
The
F_SETOWN_EX
option to
fcntl(2)
is needed to properly get overflow signals in threads.
This was introduced in Linux 2.6.32.
Prior to Linux 2.6.33 (at least for x86), the kernel did not check
if events could be scheduled together until read time.
The same happens on all known kernels if the NMI watchdog is enabled.
This means to see if a given set of events works you have to
perf_event_open(),
start, then read before you know for sure you
can get valid measurements.
Prior to Linux 2.6.34, event constraints were not enforced by the kernel.
In that case, some events would silently return "0" if the kernel
scheduled them in an improper counter slot.
Prior to Linux 2.6.34, there was a bug when multiplexing where the
wrong results could be returned.
Kernels from Linux 2.6.35 to Linux 2.6.39 can quickly crash the kernel if
"inherit" is enabled and many threads are started.
Prior to Linux 2.6.35,
PERF_FORMAT_GROUP
did not work with attached processes.
In older Linux 2.6 versions,
refreshing an event group leader refreshed all siblings,
and refreshing with a parameter of 0 enabled infinite refresh.
This behavior is unsupported and should not be relied on.
There is a bug in the kernel code between
Linux 2.6.36 and Linux 3.0 that ignores the
"watermark" field and acts as if a wakeup_event
was chosen if the union has a
nonzero value in it.
From Linux 2.6.31 to Linux 3.4, the
PERF_IOC_FLAG_GROUP
ioctl argument was broken and would repeatedly operate
on the event specified rather than iterating across
all sibling events in a group.
From Linux 3.4 to Linux 3.11, the mmap
cap_usr_rdpmc
and
cap_usr_time
bits mapped to the same location.
Code should migrate to the new
cap_user_rdpmc
and
cap_user_time
fields instead.
Always double-check your results!
Various generalized events have had wrong values.
For example, retired branches measured
the wrong thing on AMD machines until Linux 2.6.35.
EXAMPLE
The following is a short example that measures the total
instruction count of a call to
printf(3).
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <sys/ioctl.h>
#include <linux/perf_event.h>
#include <asm/unistd.h>
static long
perf_event_open(struct perf_event_attr *hw_event, pid_t pid,
int cpu, int group_fd, unsigned long flags)
{
int ret;
ret = syscall(__NR_perf_event_open, hw_event, pid, cpu,
group_fd, flags);
return ret;
}
int
main(int argc, char **argv)
{
struct perf_event_attr pe;
long long count;
int fd;
memset(&pe, 0, sizeof(struct perf_event_attr));
pe.type = PERF_TYPE_HARDWARE;
pe.size = sizeof(struct perf_event_attr);
pe.config = PERF_COUNT_HW_INSTRUCTIONS;
pe.disabled = 1;
pe.exclude_kernel = 1;
pe.exclude_hv = 1;
fd = perf_event_open(&pe, 0, -1, -1, 0);
if (fd == -1) {
fprintf(stderr, "Error opening leader %llx\n", pe.config);
exit(EXIT_FAILURE);
}
ioctl(fd, PERF_EVENT_IOC_RESET, 0);
ioctl(fd, PERF_EVENT_IOC_ENABLE, 0);
printf("Measuring instruction count for this printf\n");
ioctl(fd, PERF_EVENT_IOC_DISABLE, 0);
read(fd, &count, sizeof(long long));
printf("Used %lld instructions\n", count);
close(fd);
}
SEE ALSO
fcntl(2),
mmap(2),
open(2),
prctl(2),
read(2)
Index
- NAME
-
- SYNOPSIS
-
- DESCRIPTION
-
- Arguments
-
- Reading results
-
- MMAP layout
-
- Signal overflow
-
- rdpmc instruction
-
- perf_event ioctl calls
-
- Using prctl
-
- perf_event related configuration files
-
- RETURN VALUE
-
- ERRORS
-
- VERSION
-
- CONFORMING TO
-
- NOTES
-
- BUGS
-
- EXAMPLE
-
- SEE ALSO
-
This document was created by
man2html,
using the manual pages.
Time: 02:54:46 GMT, September 18, 2014


Cookie Sales
Home
Sales
Sales Per Store (in cookies)
New Store Data
Store Location:
Minimum Customers Per Hour:
Maximum Customers Per Hour:
Average Cookies Per Hour:
Submit Store Data


--- layout: default ---
I have told you these things so that My joy may be in you and your joy may be complete. — John 15:11


Shocker is an easy level machine based on Linux, the initial approach is based on a infamous vulnerability called ShellShock.
The privilege escalation is pretty straightforward and doesn’t require any special skills.
Recon
The port 80 has a page with a simple but enigmatic image:
After that, enumerating web directories using the gobuster tool give us the following output:
We noticed that there’s a folder named cgi-bin, this directory comes from a tipical vulnerability named ShellShock, where it’s possible to inject system commands through a vulnerable script inside the cgi-bin directory.
Now we have to look for any script inside that directory in order to exploit it, we have to enumerate any kind of script with these extensions:
.cgi
.php
.py
.sh
.rb
The big.txt wordlist from dirb doesn’t returned anything, so after a couple tries with different wordlists, we found a very interesting script file named user.sh:
Exploiting - Analyzing the vulnerable script and exploiting
This script file seems to be vulnerable to ShellShock vulnerability, a simple GET request returns the following output:
The server is vulnerable to ShellShock vulnerability using the sleep command to verify the server’s response:
The server waits 10 seconds with the sleep command and then we got our response, confirming that we actually can inject system commands!
After that we can send a reverse shell stager to the server and set up a netcat on listening mode to receive the reverse shell connection:
Exploiting - Using a custom exploit
For knowledge purposes I made a custom exploit that exploits the ShellShock vulnerability and spawn an interactive shell to us, it’s available on my GitHub :)


Note: if you set the division to 12 (or any divisor of 12), you won't actually get microtonal results ;).
"N-div" what?!
In 12-div equal temperament (the "normal" one, used to tune pianos), you divide the octave in 12 equally sized intervals (equal in terms of frequency ratios, not absolute frequencies). Here you can simply choose any other number (call it N).
Why would you do this?
Why not?
How do I play?
In order to play, press/touch the buttons or use the number keys and letter keys to control the keyboard.
What do the colors of the keys mean, why do they change when I change the octave division?
Octaves of the base frequency are shown in orange. Notes of the 12-div chromatic scale (or very close to it) are shown in white. The rest (in grey) are "microtonal" with respect to the base frequency.
Which browsers are compatible?
At least recent versions of Firefox, Chrome and Safari. Try those first and then see if your more exotic browsers work the same.
Will it work on my tablet/phone?
That is the goal, but no guarantees yet. So far I have only tested it on my Android phone with Google Chrome and it sort of works (the sound is really choppy but it might be my cheap phone's hardware).
There's no sound? What to do?
Try refreshing the page. Make sure the browser window has focus and that the tab is active. Especially if you have the web console open, sounds may sometimes disappear (at least on Firefox).
Also, disable any script blocking extensions or browser settings (the app obviously needs javascript).
Aaarrrgh, a key got stuck / sound won't stop playing!
Press and release the key again (or refresh the page). This happens sometimes if you are quick enough or if you switch tabs while sound is playing.
The sound is terribly choppy / clicky / distorted!
It could be your speakers, your hardware, your OS, your browser etc., or it could just be the waveform you've chosen. Or, you might be playing a dissonant chord (note: all chords will be slightly dissonant when the octave is divided like this, some of course more than others).
I have tried to limit the distortion and clipping by using highpass/lowpass filters, maybe my cutoff frequencies are too permissive (but on the other hand, I don't want to loose too much information by filtering).
Try reducing the gain on the waves in Waveform Settings or the output volume of your speakers.
What libraries did you use for this?
No external libs besdies Web Audio API (https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html), which should be implemented by most decent browsers. So, pretty much plain old javascript.
Can I use this for classroom demonstrations, homework, etc.?
Of course!
Can I use this for a musical perfomrance (either live or recorded)?
Sure!


I've never done drugs, nor am I interested in them. When I turned 21 I thought I would try something. I decided to stay awake all night and see how I felt the next day. I had always heard that sleep deprivation would make my mind insane from a couple of people I knew, but I did it anyway. I did it, and I felt quite sane.
After staying up all night a couple of times I decided that I would make a lifestyle of it, for I believed that if I pushed my body in that way I would accomplish more in my life. I felt that I studied better and retained information easier by doing this.
After a few months of this I realized that I had made a horrible mistake. I couldnt sleep anymore. My heartrate was very fast and hard. When I did sleep I would wake up a few hours later, my heart racing and my mind in such a state of chaos that I did not know what to do. Im my mind I thought, in a week I will be fine. The week passed and I felt the same or worse. I was in this state of sleeplessness for a year. I dropped out of school and didnt work. My heart still pounded. I didnt have a moment of peace, in the day or night. My bed was like a fire, if that makes any sense. I couldnt stay in one place for any length of time.
A year after that I could barely walk, I was delusional, and paranoid. My heart was weak from sleeplessness. I knew that I was dieing. To make a long story short, Im 24 now and feeling much stronger mentally and physically. My advice, if anyone wants it, is to never ever do what I did. Rest your body and mind as much as possible. Sleep deprivation will make you insane, for a while at least. In my case a damn eternity. I almost died. I did die in a way. Nearly four years of my life were in hell, and at the end of it I have nothing but the experience of it all and the knowledge to never do it again.
Experience Reports are the writings and opinions of the individual authors who submit them.
Some of the activities described are dangerous and/or illegal and none are recommended by Erowid.


The DSF record stores a flag that indicates if the workbook is a double stream file.
Record Data — BIFF8
|
Offset |
Flag Name |
Size |
Contents |
|
4 |
fDSF |
2 |
=1 if the workbook is a double stream file |


The Macy's Thanksgiving Day Parade might not fly its iconic balloons this year
by CNNIt's a time-honored tradition for many to watch the Macy's Thanksgiving Day Parade, but this year it could be missing a key element: its 16 giant character balloons.
Officials in New York City may have to ground the iconic balloons due to windy conditions. They cannot be flown when sustained winds exceed 23 mph and gusts exceed 34 mph, per city regulations .
In 1997, the iconic Cat in the Hat balloon injured four people after being overtaken by strong winds. The following year, city officials instituted new regulations for grounding the giant balloons.
The National Weather Service is currently forecasting sustained winds of 22 mph and wind gusts of 39 mph during the parade. The sustained winds do not exceed the regulations, but the gusts do. Officials will have to monitor wind conditions and decide whether to allow the balloons to be flown.
They'll likely have to make that call on Thursday, as wind conditions can quickly change. The last time the parade's balloons were grounded due to inclement weather was 1971, according to the New York Times .
The good news is this year will be much warmer than last year's parade . The National Weather Service confirmed a low of 19°F that morning, which made it the second-coldest Thanksgiving Day in the city's history. This year, it's forecast to be 44-47°F.


--- layout: default --- Your browser does not support the audio element. {{ content }} {% include sns-share.html %}


Text:
Text (alpha-numeric):
Number (int):
Number (float):
Date:
List:
Multi Select:
Text Area:
Text:
Text:
Text:
Text:


class: center, middle, inverse, title-slide # ETC1010: Data Modelling and Computing ## Lecture 5: Reading different data formats ### Di Cook (
<EMAIL_ADDRESS>,
@visnut
) ### Week 5 --- # Overview - Shape files for maps - Excel spreadsheets - Googlesheets - SPSS format (PISA data) - Audio files - `read_csv` vs `read.csv` - feather for large binary files - Handling large data sets by constructing a small database: `sqlite` - Web format (json) data, `jsonlite` (crossrates) - Web scraping? --- # Shape files Download the Australian electorate shape files from [http://www.aec.gov.au/Electorates/gis/gis_datadownload.htm], 2016 national, mapinfo format. Its 11Mb. ``` OGR data source with driver: MapInfo File Source: "data/national-midmif-09052016/COM_ELB.TAB", layer: "COM_ELB" with 150 features It has 9 fields Formal class 'SpatialPolygonsDataFrame' [package "sp"] with 5 slots ..@ data :'data.frame': 150 obs. of 9 variables: .. ..$ Elect_div : Factor w/ 150 levels "Adelaide","Aston",..: 90 135 7 17 40 56 93 12 14 16 ... .. ..$ State : Factor w/ 8 levels "ACT","NSW","NT",..: 3 3 6 6 6 6 6 4 4 4 ... .. ..$ Numccds : int [1:150] 335 180 208 226 197 179 256 233 216 199 ... .. ..$ Actual : int [1:150] 0 0 0 0 0 0 0 0 0 0 ... .. ..$ Projected : int [1:150] 0 0 0 0 0 0 0 0 0 0 ... .. ..$ Total_Population : int [1:150] 0 0 0 0 0 0 0 0 0 0 ... .. ..$ Australians_Over_18: int [1:150] 0 0 0 0 0 0 0 0 0 0 ... .. ..$ Area_SqKm : num [1:150] 1352034 337 7379 20826 289 ... .. ..$ Sortname : Factor w/ 150 levels "Adelaide","Aston",..: 90 135 7 17 40 56 93 12 14 16 ... ..@ polygons :List of 150 .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. ..$ :Formal class 'Polygons' [package "sp"] with 5 slots .. .. [list output truncated] ..@ plotOrder : int [1:150] 139 1 78 145 28 24 119 25 99 18 ... ..@ bbox : num [1:2, 1:2] 96.82 -43.74 159.11 -9.14 .. ..- attr(*, "dimnames")=List of 2 ..@ proj4string:Formal class 'CRS' [package "sp"] with 1 slot ``` --- class: inverse middle # Your turn How many Federal electorates in Australia? --- # Thinning out space - The shape object created is 46Mb. Too big! - Want smaller data set, that still effectively describes the spatial domain. - Thinning a map object can be tricky, want to thin long straight areas but keep twisty boundaries detailed. ```r library(rmapshaper) sFsmall <- ms_simplify(sF, keep=0.05) ``` --- # Plot it  Map is still defined in shapefile structure. Difficult to work with, in conjunction with any other data. --- # Extract information on each electorate ```r nat_data <- sF@data nat_data$id <- row.names(nat_data) head(nat_data) Elect_div State Numccds Actual Projected Total_Population 1 Lingiari NT 335 0 0 0 2 Solomon NT 180 0 0 0 3 Bass TAS 208 0 0 0 4 Braddon TAS 226 0 0 0 5 Denison TAS 197 0 0 0 6 Franklin TAS 179 0 0 0 Australians_Over_18 Area_SqKm Sortname id 1 0 1352034.0451 Lingiari 1 2 0 336.6861 Solomon 2 3 0 7378.7516 Bass 3 4 0 20826.1840 Braddon 4 5 0 288.7177 Denison 5 6 0 6514.2083 Franklin 6 ``` --- # Get map into tidy form ```r nat_map <- ggplot2::fortify(sFsmall) head(nat_map) long lat order hole piece id group 1 137.9982 -23.52089 1 FALSE 1 1 1.1 2 137.9984 -23.58319 2 FALSE 1 1 1.1 3 137.9984 -23.66652 3 FALSE 1 1 1.1 4 137.9985 -23.74985 4 FALSE 1 1 1.1 5 137.9985 -23.83318 5 FALSE 1 1 1.1 6 137.9985 -23.91651 6 FALSE 1 1 1.1 ``` --- # Be clear about id variables - Ensure group and piece variables are treated as factors, not numbers - Add electorate names to the polygons ```r nat_map$group <- paste("g",nat_map$group,sep=".") nat_map$piece <- paste("p",nat_map$piece,sep=".") nms <- sFsmall@data %>% select(Elect_div, State) nms$id <- as.character(1:150) nat_map <- left_join(nat_map, nms, by="id") head(nat_map) long lat order hole piece id group Elect_div State 1 137.9982 -23.52089 1 FALSE p.1 1 g.1.1 Lingiari NT 2 137.9984 -23.58319 2 FALSE p.1 1 g.1.1 Lingiari NT 3 137.9984 -23.66652 3 FALSE p.1 1 g.1.1 Lingiari NT 4 137.9985 -23.74985 4 FALSE p.1 1 g.1.1 Lingiari NT 5 137.9985 -23.83318 5 FALSE p.1 1 g.1.1 Lingiari NT 6 137.9985 -23.91651 6 FALSE p.1 1 g.1.1 Lingiari NT ``` --- # Map it, using area of the electorate to colour. With joins to data from other sources, e.g. census, other variables could be mapped to colour. ```r ggplot(aes(map_id=id), data=nat_data) + geom_map(aes(fill=Area_SqKm), map=nat_map) + expand_limits(x=nat_map$long, y=nat_map$lat) + theme_map() ```  --- # Interactivity Mouseover names more effective ```r p <- ggplot(aes(map_id=id), data=nat_data) + geom_map(aes(fill=Area_SqKm, label=Elect_div), map=nat_map) + expand_limits(x=nat_map$long, y=nat_map$lat) + theme_map() ggplotly(p) ``` --- # Add centroids Using the geographic centroid for each electorate is an alternative. It can also be extracted from the shape files. ```r centroid <- function(i, polys) { ctr <- Polygon(polys[i])@labpt data.frame(long_c=ctr[1], lat_c=ctr[2]) } centroids <- seq_along(polys) %>% purrr::map_df(centroid, polys=polys) head(centroids) long_c lat_c 1 133.3706 -19.48052 2 130.9355 -12.42392 3 147.5081 -41.15828 4 145.4985 -41.75995 5 147.2439 -42.88836 6 146.6272 -43.24309 ``` --- joined to the other information about each electorate... ```r nat_data <- bind_cols(nat_data, centroids) head(nat_data) Elect_div State Numccds Actual Projected Total_Population 1 Lingiari NT 335 0 0 0 2 Solomon NT 180 0 0 0 3 Bass TAS 208 0 0 0 4 Braddon TAS 226 0 0 0 5 Denison TAS 197 0 0 0 6 Franklin TAS 179 0 0 0 Australians_Over_18 Area_SqKm Sortname id long_c lat_c 1 0 1352034.0451 Lingiari 1 133.3706 -19.48052 2 0 336.6861 Solomon 2 130.9355 -12.42392 3 0 7378.7516 Bass 3 147.5081 -41.15828 4 0 20826.1840 Braddon 4 145.4985 -41.75995 5 0 288.7177 Denison 5 147.2439 -42.88836 6 0 6514.2083 Franklin 6 146.6272 -43.24309 ``` --- and plotted as is, or spread out ```r p1 <- ggplot(aes(map_id=id), data=nat_data) + geom_map(aes(fill=Area_SqKm), map=nat_map) + expand_limits(x=nat_map$long, y=nat_map$lat) + theme_map() + theme(legend.position="none") + geom_point(data=nat_data, aes(x=long_c, y=lat_c), colour="orange") p2 <- ggplot(aes(map_id=id), data=nat_data) + geom_map(aes(fill=Area_SqKm), map=nat_map) + expand_limits(x=nat_map$long, y=nat_map$lat) + theme_map() + theme(legend.position="none") + geom_jitter(data=nat_data, aes(x=long_c, y=lat_c), colour="orange", width=2, height=2) grid.arrange(p1, p2, ncol=2) ```  --- # Excel spreadsheets - Often data comes in multiple excel format files - It it tedious, and inefficient to manually convert each to csv and read - Easier to automate reading multiple files, in the original format - Example: Rental market in Tasmania from [data.gov.au](http://data.gov.au/dataset/rental-bond-and-rental-data-tasmania-2016-to-2017) --- ```r library(readxl) library(sawfish) # devtools::install_github("AnthonyEbert/sawfish") url<-"http://data.gov.au/dataset/rental-bond-and-rental-data-tasmania-2016-to-2017" fls <- find_files(url, "xlsx") f1 <- tempfile() download.file(fls[1], f1, mode="wb") t1 <- read_xlsx(path=f1, sheet=1) t1 # A tibble: 1,368 x 11 `Street Name` Suburb State Postcode `Bond Amount` `Weekly Rent` <chr> <chr> <chr> <dbl> <dbl> <dbl> 1 BANGALEE STREET LAUDERDALE TAS 7021 1440 360.0 2 WILMOT ROAD HUONVILLE TAS 7109 1180 295.0 3 FOREST ROAD WEST HOBART TAS 7000 350 87.5 4 FOREST ROAD WEST HOBART TAS 7000 350 87.5 5 CENTRAL AVE MOONAH TAS 7009 800 200.0 6 CHARLES ST MOONAH TAS 7009 1200 300.0 7 SPINIFEX ROAD RISDON VALE TAS 7016 1080 270.0 8 GARDENIA ROAD RISDON VALE TAS 7016 780 195.0 9 BRITTEN STREET NEW NORFOLK TAS 7140 1140 285.0 10 BOXHILL RD CLAREMONT TAS 7011 600 150.0 # ... with 1,358 more rows, and 5 more variables: `Bond Lodgement date # (DD/MM/YYYY)` <dttm>, `Bond Activation date (DD/MM/YYYY)` <dttm>, `No # of Bedrooms` <dbl>, `Dwelling/Premises Type` <chr>, `Length of Tenancy # (In Months)` <dbl> ``` --- # Now pull all and merge ```r rentals <- NULL for (i in 1:length(fls)) { download.file(fls[i], f1, mode="wb") t1 <- read_xlsx(path=f1, sheet=1) rentals <- bind_rows(rentals, t1) } dim(rentals) [1] 18263 12 ``` --- # How have rental rates changed over time ```r rentals %>% mutate(month=month(`Bond Lodgement date (DD/MM/YYYY)`), year=year(`Bond Lodgement date (DD/MM/YYYY)`)) %>% group_by(Postcode, month, year, `No of Bedrooms`) %>% summarise(rent=mean(`Weekly Rent`, na.rm=TRUE)) %>% mutate(time=dmy(paste("01", month, year, sep="-"))) %>% ggplot(aes(x=time, y=rent)) + geom_line(aes(group=Postcode)) + facet_wrap(~`No of Bedrooms`, ncol = 3) + ylab("Weekly rent") ``` ---  --- # Clean data and re-plot ```r rentals %>% mutate(month=month(`Bond Lodgement date (DD/MM/YYYY)`), year=year(`Bond Lodgement date (DD/MM/YYYY)`)) %>% group_by(Postcode, month, year, `No of Bedrooms`) %>% summarise(rent=mean(`Weekly Rent`, na.rm=TRUE)) %>% mutate(time=dmy(paste("01", month, year, sep="-"))) %>% filter(!is.na(`No of Bedrooms`)) %>% filter(`No of Bedrooms`<6, `No of Bedrooms`>0) %>% ggplot(aes(x=time, y=rent)) + geom_line(aes(group=Postcode), alpha=0.5) + facet_wrap(~`No of Bedrooms`, ncol = 3) + ylab("Weekly rent") + ylim(c(0, 1000)) + geom_smooth(se=FALSE) ``` ---  --- # Googlesheets - Google sheets are effectively excel spreadsheets - We can read these directly also - More efficient than download and read in ```r library(readxl) # Read from moodle excel sheet class <- read_xlsx("ETC1010 - S2 2017 Grades.xlsx") library(googlesheets) # Now get lab scores gs_ls() Waiting for authentication in browser... Press Esc/Ctrl + C to abort lss <- gs_title("ETC1010") labs <- gs_read(lss, col_types=c("ccccddc")) lab_scores <- full_join(class, labs[,c(1,5,6,7)], by=c("ID number"="ID number")) %>% filter(!is.na(Surname)) %>% rename("Lab 1"="Lab 1(Out of 10)") %>% replace_na(list(`Lab 1`=0)) ``` --- class: inverse middle # Share and share alike
This work is licensed under a Creative Commons Attribution 4.0 International License.


| Lightning Sale: Kitchen Textiles! | View email in web browser. |
| LIMITED TIME: UP TO 50% OFF SELECT KITCHEN TEXTILES! |
|
||||||
You're receiving this email because you subscribed toThe Diabetes Site reminder service.
One-click unsubscribe here or click here to manage your subscription(s).
To ensure delivery to your inbox, add<EMAIL_ADDRESS>to your address book/contacts.
The Diabetes Site, One Union Square, 600 University Street, Suite 1000, Seattle, WA 98101
1-888-355-4321 Customer Service Hours: M-F 7am - 3pm PT


rtcwake - manual page for rtcwake 1.31.1
rtcwake [-a | -l | -u] [-d DEV] [-m MODE] [-s SEC
| -t TIME]
BusyBox v1.31.1 (2020-04-30 13:38:01 EDT) multi-call binary.
Enter a system sleep state until specified wakeup time
- -a,--auto
- Read clock mode from adjtime
- -l,--local
- Clock is set to local time
- -u,--utc
- Clock is set to UTC time
-d,--device DEV Specify the RTC device
- -m,--mode MODE
- Set sleep state (default: standby)
-s,--seconds SEC Set timeout in SEC seconds from now
- -t,--time TIME
- Set timeout to TIME seconds from epoch


Shoot Short Videos
According to one recent analysis, YouTube videos under 2 minutes
in length get almost 50% of all views. That should be no surprise.
No one wants a long slog through the entire history of how your store
was founded or the every detail of where your product was sourced.
Don’t rush what you have to say, but it’s better to err on the side of brevity when possible.
Create a Branded Channel
Unless your video is truly a one-off, you should set up a dedicated store channel.
Not only does it look more professional for a video to be coming from an account
with your store name (rather than, say, frednelson86), you’ll have a more active web presence.
Having your own YouTube channel will give you yet another way for people to find your products.
Upload a banner, and your logo, to make it even more professional.
Be Consistent With Content Production
A well-designed channel isn’t much use if there’s no content.
If you choose to do YouTube marketing, it’s best to be committed to the long term.
 Once you decide that it’s worth it, be consistent about posting.
Remember, YouTube is a social media channel, and just like other forms of social media, long periods of silence are no fun.
You can decide on the frequency of posting. We recommend a video once a month, or once every two weeks if you can manage it.
And once you figure out something that works, commit to it.
Source: www.shopify.com


Recover username
Your e-mail
Recover username
If you remember your username
Sign in
,
otherwise
Create account
.